Voice Input Modes — choosing and building open-mic vs. push-to-talk

Design guidance for app builders deciding how a viewer's input reaches the agent. Three modes:

* In the default micStartMode: 'immediate' the permission prompt runs alongside the connect handshake. connect() never waits for it and never fails because of it: a denied, missing or busy mic emits one warning (mic_permission_denied, mic_not_found, mic_in_use) and the session connects mic-less, with typed turns working. Or deferred: micStartMode: 'deferred' connects the avatar session with no mic at all. The app calls startMic() later from a real user click, so the permission prompt is gesture-anchored instead of firing on page load. Until then, typed turns work, but startTapToTalk() throws mic_not_started. See README.md.

The first two are voice-capture modes on the live avatar transport. Most of this doc is about choosing between them and building their UI.

Chat mode is a full text-only transport, with the same brain, same tools, and same thread as the voice modes. It's covered in README.md, including mid-conversation switching between avatar and chat via KalturaAgentSession (switching section below).

For the wire mechanics of startTapToTalk()/endTapToTalk(), see README.md. For the exact socket events, see Wire Protocol (tapToTalkStart/tapToTalkEnd, isTapToTalk).

Text input is not voice-mode-exclusive: an avatar session accepts typed turns too (session.speak(text) — see Client-Side Commands and README). Offering a text box alongside either voice mode is how the SDK satisfies EN 301 549 §6.2.1.2 (concurrent voice and text) — see README.md.

The one rule that overrides everything else here

Pick one VOICE mode per agent, at configuration time. Never offer both voice modes live in the same session. (Chat text-only input is exempt — typed turns don't touch the VAD/capture machinery this rule protects, so text can coexist with either voice mode, or replace voice entirely.)

This is not a UI-polish preference. It is a correctness requirement.

Don't rely on anything else to catch a mismatch for you. The SDK's own client-side gate (startTapToTalk() throws capability_disabled unless session.capabilities.tapToTalk) stops your UI from opening a tap-to-talk window on an agent configured for open-mic. Build your UI to match the agent's single configured mode. Never render a tap-to-talk control against an open-mic agent, or vice versa. Mixing them produces double-cut or missed turns (see Wire Protocol's tapToTalkStart/tapToTalkEnd row for the event pair involved).

Every push-to-talk/open-mic product draws the same line — one active capture mechanism, chosen once, not a live per-session toggle exposing both:

Deciding which mode fits your app

If your viewers will regularly speak in full sentences or ask multi-part questions, open-mic is the better default. Neither mode is inherently better. Press-and-hold can strain a viewer's hand on a long dictation, and VAD-based toggle can cut a speaker off mid-thought on long unstructured reasoning. Which complaint you'll hear from your own viewers tracks utterance length, not a fixed advantage of one mode over the other.

UX pattern: click-to-toggle, not press-and-hold

Prefer a single click/tap to open the capture window and a second click/tap to close it, over press-and-hold-to-record. Reasons, in order of weight:

• Accessibility. Toggle satisfies WCAG 2.5.2 (Pointer Cancellation, Level A) on its own — the down-event never fires the action, so an accidental touch can be dragged away before commit. Press-and-hold can satisfy it too (release-to-stop counts as an "up reversal"), but only if release doesn't also irreversibly submit the utterance.
• Screen readers. Sustained-hold gestures are poorly supported by mobile screen readers (Android TalkBack has no native single-tap "hold" gesture — it requires double-tap-then-hold). Toggle needs only a single activation gesture, which every screen reader supports natively.
• Usability at length. Holding a button for the duration of a multi-sentence question is physically awkward and blocks other interaction (scrolling, reading) with the holding hand/thumb.

If your product genuinely needs walkie-talkie-style short bursts (not this SDK's typical investor-deck or knowledge-avatar use case), a press-and-hold-with-slide-to-lock hybrid (WhatsApp's pattern) is the documented middle ground. Start from toggle, and only move to hold-based capture if you have evidence your utterances are consistently short.

Visual and non-visual state feedback

Give the viewer three redundant signals that the mic is live, not one:

1. Icon/color change on the button itself: swap an idle-mic icon for a recording icon on a tap-to-talk button, not just an aria-pressed attribute change.
2. An animated level indicator: a waveform, pulsing glow, or (simplest, and already available) this SDK's localMicLevel event ({level}, 0–1, ~50ms tick) driving a CSS custom property. A single static icon alone is not enough (NN/g's critique of Amazon Echo's single light ring as "a far cry from rich textual feedback").
3. A live-region text or caption update (aria-live) confirming state changes ("Listening…" / "Sent") — necessary for screen-reader users who can't see the icon/waveform at all. Pair with an audio cue (a short start/stop tone) as an additional non-visual channel if your app's audio design allows it.

Safety: don't let a capture window hang open forever

A tap window that never closes (tab closed mid-recording, app crash, network drop) must not leave the server's tap-mode state stuck open. Layer these on top of startTapToTalk()/endTapToTalk():

• Silence-based auto-stop — close the window automatically after a period of continuous silence, so a forgotten "open" mic doesn't stay live indefinitely.
• A hard max-duration cap — an absolute ceiling regardless of speech/silence, as a backstop.
• Treat disconnect/pagehide/visibilitychange as an implicit close — the browser's own getUserMedia/socket teardown on tab close is the only mechanism that reliably fires when the viewer abandons the tab mid-recording. Call endTapToTalk() (or just let disconnect naturally end the session) from those events, rather than assuming a clean call will always arrive.

Implementation checklist

1. Decide the agent's mode at provisioning time — isTapToTalk is a fixed per-agent deployment choice (set wherever your app builds its intellect/session config), not something your UI code branches on live.
2. Build the UI conditionally on session.capabilities.tapToTalk (derived from clientConfiguration.isTapToTalk). Render a tap-to-talk control only when true. Never render both a tap-to-talk control and an open-mic affordance for the same session.
3. Wire click-to-toggle calling session.startTapToTalk()/session.endTapToTalk(), updating aria-pressed and an icon/level indicator on tapToTalkStarted/tapToTalkEnded — see the conditional-UI example in README.md.
4. Add the silence/max-duration/abandonment safeguards above; the SDK does not impose them for you.
5. Update any UI copy that currently assumes continuous listening (e.g. a text-input placeholder like "...or just speak naturally") — that copy is wrong for a tap-to-talk agent and should describe the button instead.

Switching between avatar and chat mid-conversation

KalturaAgentSession runs one conversation over either transport and switches between them with switchMode('avatar' | 'chat') — the thread, the request_vars context, and every onToolCall handler carry over automatically. One state machine:

modeChanged.threadContinuity tells you which happened. true means the new transport was seeded with the live thread (show "conversation restored"). false means no turn had happened yet, so there was no thread to carry.

Two UX rules for the switch:

• Switching INTO a voice mode prompts for the mic. Browsers require a live user gesture for getUserMedia, and a prior grant in chat mode doesn't exist to reuse. Route switchMode('avatar') through a real click target (a "Continue with video" button), never a state-change callback. A programmatic call outside a gesture gets auto-denied. The session still connects (mic acquisition never blocks or fails connect()), but it arrives mic-less with a warning (mic_permission_denied), and the viewer has to call startMic() from a real click to get voice.
• Expect a brief reconnect blip. Switching is tear-down-and-reconstruct by design — show a transient "switching…" state on the facade's stateChange {state:'switching'} event rather than hiding it. sendText() calls during the blip are buffered (up to 8) and delivered in order on the new transport.

Related docs