Socket.IO events — developer-facing catalog

← Back to Wire Protocol

Direction: → client emits, ← server emits. "Captured" = seen in a live session. This covers the events a client integration uses. The server also defines internal, binary-relay events (e.g. userVideoBinaryData/agentVideoBinaryData, contactCollector) that aren't part of the developer-facing surface.

4a. Client → Server (emit)

tapToTalkStart/tapToTalkEnd in detail

tapToTalkEnd schedules a 300ms timer that mints the turn from whatever was buffered during the tap window. The SDK emits this pair from KalturaAvatarSession#startTapToTalk()/#endTapToTalk(). The resulting turn arrives via the existing agentTurnToTalk handler like any open-mic turn.

Do not use for typed-text barge-in. Bracketing onTextEntered inside this pair mints a duplicate turn, since neither tapToTalkStart nor tapToTalkEnd invalidates a turn already in flight.

Why isTapToTalk:true is required, not optional. Mixing tap-to-talk with an open-mic (isTapToTalk:false) agent produces unreliable turn-taking — the two capture modes aren't meant to run together. The SDK enforces this client-side: startTapToTalk() throws capability_disabled unless capabilities.tapToTalk is set.

For the app-level decision of when to use this mode and how to design its UI, see Voice Input Modes.

4b. Server → Client (on) — handshake/session phase

4c. Server → Client (on) — ASR signaling (relayed over the socket)

4d. Server → Client (on) — conversation phase

These fire once approvedPermissions is sent. Several are server-emitted but only the built-in client OR the embed client OR a debug panel subscribes — noted per row. Many of these payloads carry a speechId, the per-utterance identifier explained in §4f below.

4e. agent_raw_text.delta — the brain stream (parsed)

delta is a JSON string. Parsed shape:

{ role:"assistant", type, content, segmentNumber, et,            // always present
  threadId?, messageId?, segmentStart?, segmentEnd?, isFinal?,    // conditional (in `extra`)
  metadata?:{widgetName?,runtimeName?} }                          // unisphere-tool segments

Always-present fields are role (always "assistant"), type, content, segmentNumber, and et (elapsed seconds) — set by the server on every segment it sends. The rest are conditional extra keys. For example, threadId/messageId ride the first think segment, and metadata rides unisphere-tool.

How type is determined. The brain emits markdown. The server sets each segment's type from the code-fence language tag the LLM writes, and defaults to text for un-fenced prose. So the fence-driven types are open-ended — the LLM chooses them. On top of that, a fixed set of control types is emitted by hardcoded response-formatting calls.

Three kinds of tool segments

Parse a tool call with the SDK's parseToolCall(seg), session.onToolCall(name), or collectConverse().toolCalls. Author the tool with tools.client(...). See External API Integrations § Don't skip kaltura_genie_experiences: 'off' for why a command-driven intellect (the agent's brain configuration — its prompts, tools, and capabilities) must turn that capability off, and only at creation time.

Fused multi-tool tool segments

When a turn calls two or more tools, the server can emit one type:"tool" segment. Its content concatenates every called tool's JSON args back-to-back, but names only the last one — e.g. open_filing {"quarters": [...], "metric": "total_revenue"}{"quarter": "q1_2026", "docType": "press_release"}. The highlight_chart call that preceded open_filing rides in the same string, unnamed.

How the response echoes the missing name

The tool_response segments that follow still echo every called tool by name, in call order (highlight_chart responded with size 113 then open_filing responded with size 104). This is the only reliable client-side signal for attributing the earlier, unnamed blob to its real tool.

How the SDK recovers it

The SDK's parseToolCall(seg) recovers the named tool's own args correctly (the last JSON object), and surfaces earlier blobs as call.fusedArgs (an array, in arrival order). parseToolResponseName(seg) extracts a tool_response's echoed name. KalturaAvatarSession pairs the two automatically, using an ASR-sub-turn-scoped queue of un-attributed fusedArgs blobs. The queue drains on the next tool_response name not already dispatched in this sub-turn, so every onToolCall(name) handler fires with correct args even on a fused turn. No app-level change is needed.

Queue reset boundary

The queue and its dispatched-names guard reset on every agent_start_speech, whether isNewTurn is true or not. A name dispatched directly in one ASR sub-turn must not block that same name's fused recovery in the next sub-turn of the same turnId — it's a distinct call with distinct args, not a repeat. This is narrower than the SDK's own cross-turn _firedToolCalls dedup, which drops an exact repeat (same name, same args) and only clears on a real isNewTurn boundary — a separate mechanism from this turn-scoped fused-recovery queue.

Headless caveat

Headless collectConverse() gets the corrected named-tool args for free, but it does not run this pairing recovery. An earlier fused blob is only reachable via fusedArgs on that one ToolCall, not as its own toolCalls entry.

The wait_for_response ACK — one wire contract, two transports

A tools.client tool built with waitForResponse:true blocks the model's turn until the host app supplies a result. The server polls up to timeout seconds (default 30) for an ACK. The ACK is not a socket event. On both transports it is the same plain HTTPS POST, authorized by the session's own conversation KS — the model speaks the acked value in the same turn:

POST {genieUrl}/assistant/tool_response
Content-Type: application/json
Authorization: KS <conversation ks>

{ "tool_name": "<toolName>", "tool_id": "<toolMetadata.id>",
  "tool_invocation_id": "<toolMetadata.id>", "response": { …your JSON result… } }

→ 200 {}

tool_id and tool_invocation_id are both the toolMetadata.id from the parsed tool segment. This is exactly what KalturaAvatarSession#respondToTool (SDK:session.js) and KalturaChatSession#respondToTool (SDK:chat-session.js) send. The tool segment may arrive over the socket (agent_raw_text, above) or over the HTTP /assistant/converse chat stream, but the ACK path is identical. So one waitForResponse:true tool definition works unmodified on both transports. See Client-Side Commands for the app-level contract (onToolCall → respondToTool).

init_response is NOT an HTTP-converse segment. It is a WebSocket-only event. In the live runtime it arrives as the delta of the first agent_raw_text socket event (carrying openingPhrase/threadId/messageId). It never appears in an /assistant/converse HTTP stream. openingPhrase is the intellect's opening_phrase, rendered server-side (Jinja2 over request_vars and sys__*) for this session. With a silent opening phrase (SILENT_OPENING) the opening turn still fires stvStartedTalking/stvFinishedTalking (about 0.5 s apart) but no audible speech, and the SDK surfaces that turn as SILENT_OPENING_LABEL ([silence]) on transcript/speechChunk/avatarStopTalking. A configured kickoff is sent on that stvFinishedTalking; its first think delta fires responsePending. See Start the Conversation.

Stored-only message types. Two message types never stream. They only appear when you read the thread back (mgmt.messages/mgmt.threads, and the thread transcript as opening: ...). opening is the rendered opening_phrase, stored as the thread's first assistant message: { type: 'opening', content: [{ type: 'avatar', content: '<rendered phrase>', metadata: {} }] }. summary is written once when an avatar session ends and the intellect has an avatar_summary_config (or the default). Its stored shape is { type: 'summary', content: [{ type: 'text', content: '<rendered template>', metadata: {} }] }, alongside session_duration on the stored row. It is skipped when the thread has no human messages.

Avatar-runtime segment handling. The type values above are the brain's raw types. On the avatar runtime, a type === 'unisphere-tool' segment already arrives rewritten to its metadata.runtimeName (minus the -tool suffix). So a runtime client sees that normalized type, not the raw unisphere-tool name. Every segment type is delivered (none are dropped) along with start/end/final/delta. Because an avatar session always runs as force_experience: 'avatar_only', the brain's spoken content arrives primarily as avatar (streamed) / avatar-filler segments. Control/structured types (think/tool/unisphere-tool/share/thread/error) stream alongside for the transcript/UI. Note: grouping avatar-filler under "spoken" here describes wire mechanics only. Unlike avatar/text, its phrasing is server-generated per turn and NOT reliably steerable via base_directive (see the avatar_filler capability note in GenUI · Authoring and Consuming Widgets).

The built-in client's text-assembly logic only assembles text | unisphere-tool | error into the transcript, and treats a start+end share as message-complete. It ignores avatar | think | tool | tool_response (those drive the live runtime).

Session-completion signal — tell the backend a conversation is truly over

Same auth model as the ACK above (conversation KS, no elevation), but fire-and-forget in the opposite direction — client tells server, no response payload to parse:

POST {genieUrl}/thread/session_completed
Content-Type: application/json
Authorization: KS <conversation ks>

{ "id": "<threadId>" }

→ 200 {}

The SDK itself is idempotent about sending it. A sent flag means a repeat call for the same session is a client-side no-op. On a page-unload path (pagehide, hidden-grace) the SDK never applies a timeout and never awaits the response. It fires with fetch(url, {keepalive:true}), never navigator.sendBeacon (which can't carry Authorization).

On other paths (disconnect(), idle auto-logoff) the SDK aborts the request after sessionCompleteTimeoutMs (default 5000). KalturaAvatarSession/KalturaChatSession/KalturaAgentSession send this automatically on disconnect() and on tab-close/backgrounding/bfcache-freeze. See System Internals Reference § Session-completion signal for the full trigger table, and README.md § Ending a conversation cleanly for the config surface.

4f. speechId — the per-utterance key (and the barge-in mechanism)

speechId appears on agent_raw_text, agent_start_speech, agent_end_turn, generatingSpeech, stvSpeechChunk, debug_stvTaskGenerated, and stvFinishedGenerating. It is the session server's identifier for one agent utterance, and it's how you group a turn's events. Do not group by timestamp — barge-ins interleave turns.

• Format: a 4-char nonce, then the trigger, then its content, joined with -: <nonce>-<trigger>-<payload>. For example: 4nkM-transcript-Hey, what's up?, 1Yev-approved-permissions.

  Known triggers:

  Trigger	Meaning
  transcript	A user speech/text turn — the same path onTextEntered feeds
  approved-permissions	The opening greeting. Typed text cannot interrupt it; speak() holds text until it ends
  tap-to-talk	—
  resume-replay	The replayed last line after a resume(). Same hold as the greeting
  wake-up	The server's own "are you still there?" check-in after a silence. Typed text cannot interrupt it; speak() holds text until it ends
  begin-agent-conversation	—
  contact-info-received / contact-info-rejected	—
  html-element-click	—
  iframe-completed	—
  code-block-completed	—
  hangup-message	The server's goodbye line before it ends the session. Typed text cannot interrupt it; speak() holds, then resolves false when the session ends

• Minted per utterance, and it maps 1:1 to the brain's request uuid. stvStartedTalking/stvFinishedTalking carry no speechId in their payload. Attribute them to the speechId of the surrounding stvSpeechChunks.

• The staleness guard is what makes barge-in work. The server tracks a single active speechId per session. Any TTS/STV event whose speechId doesn't match the current one is dropped server-side. When a new user turn arrives, the server mints a new transcript speechId and makes it the active one, instantly invalidating the prior utterance's in-flight audio. That is exactly what agentInterrupted reflects. The stvSpeechChunk speechId switches at each agentInterrupted (e.g. 4nkM-transcript-… → agentInterrupted → d1qD-transcript-…).

Related docs