Socket.IO events — developer-facing catalog

← Back to Wire Protocol

Direction: → the client emits, ← the server emits. This catalog lists the events KalturaAvatarSession sends and receives. A row says so when the SDK has no handler for an event. The golden fixture holds one real capture.

4a. Client → Server (emit)

tapToTalkStart/tapToTalkEnd in detail

endTapToTalk() ends the capture window, and the resulting turn arrives through agentTurnToTalk like any open-mic turn.

Do not use this pair for typed-text barge-in. Wrapping onTextEntered in it creates a duplicate turn.

Use it only on an agent configured with isTapToTalk: true. Mixing tap-to-talk with an open-mic agent gives unreliable turn-taking. The SDK enforces this: startTapToTalk() throws capability_disabled unless capabilities.tapToTalk is set.

For the app-level decision of when to use this mode and how to design its UI, see Voice Input Modes.

4b. Server → Client (on): handshake/session phase

4c. Server → Client (on): ASR signaling (relayed over the socket)

4d. Server → Client (on): conversation phase

These fire once approvedPermissions is sent. Many payloads carry a speechId, the per-utterance identifier (§4f).

The four debug_* events arrive because the SDK connects with debugMode: true (§2). The SDK has no handler for them. Its captions and transcripts do not read them.

4e. agent_raw_text.delta: the agent stream (parsed)

delta is a JSON string. Parsed shape:

{ role:"assistant", type, content, segmentNumber, et,            // always present
  threadId?, messageId?, segmentStart?, segmentEnd?, isFinal?,    // conditional
  metadata?:{widgetName?,runtimeName?} }                          // unisphere-tool segments

Every segment has role (always "assistant"), type, content, segmentNumber and et (elapsed seconds). The rest are conditional. For example, threadId and messageId ride the first think segment, and metadata rides unisphere-tool.

The set of type values is open-ended. Ignore types you don't handle.

On the avatar socket, the SDK resolves a unisphere-tool segment to its metadata.runtimeName without the -tool suffix, so runtime clients work with the normalized name. Every segment type is delivered. The avatar session joins with force_experience: 'avatar_only', so the spoken content arrives mainly as avatar and avatar-filler segments, and think, tool, unisphere-tool, share, thread and error stream alongside for the transcript and UI. The avatar-filler phrasing can't be reliably steered with base_directive. See the avatar_filler note in GenUI · Authoring and Consuming Widgets.

Three kinds of tool segments

Parse a tool call with parseToolCall(seg), session.onToolCall(name), or collectConverse().toolCalls. Author the tool with tools.client(...). See External API Integrations § Don't skip kaltura_genie_experiences: 'off' for why a command-driven intellect (the agent's configuration: its prompts, tools and capabilities) must turn that capability off, and only at creation time.

Fused multi-tool tool segments

When a turn calls two or more tools, one type:"tool" segment can carry all the JSON args back to back but name only the last tool. Example: open_filing {"quarters": [...], "metric": "total_revenue"}{"quarter": "q1_2026", "docType": "press_release"}. The earlier call rides in the same string, unnamed.

The tool_response segments that follow name every called tool, in call order (highlight_chart responded with size 113, then open_filing responded with size 104).

What the SDK does:

• parseToolCall(seg) returns the last JSON object as the named tool's args. Earlier objects appear as call.fusedArgs (an array, in arrival order).
• parseToolResponseName(seg) extracts the name a tool_response names.
• KalturaAvatarSession pairs the two and dispatches, so each onToolCall(name) handler fires with the right args. No app change is needed. It keeps the pairing state per ASR sub-turn and resets it on every agent_start_speech.
• Headless collectConverse() returns the named tool's args but does not pair. Reach the earlier blob through fusedArgs on that one ToolCall.

OAuth consent redirect (interruption / subtype:"oauth_required")

When a tool call needs the end user to authorize access (an MCP server or an api tool wired to OAuth2), the turn doesn't fail. It pauses. The stream carries a type:"interruption" segment with a consent-redirect URL instead of a normal tool_response:

{ role: "assistant", type: "interruption", segmentNumber, et,
  content: { auth_url: "https://…/authorize?…" },   // an object, not a plain string
  metadata: { tool_name: "jira_search", tool_display_name: "Jira", subtype: "oauth_required" } }

content is {auth_url} here, unlike every other segment type where content is a plain string. The segment rides agent_raw_text deltas on the live socket and NDJSON/SSE lines on the HTTP /assistant/converse stream, with no dedicated event. It reaches text and avatar sessions identically.

Parse it with parseOAuthRequired(seg) (src/core/stream.js). It returns null for anything else, including a user-interruption, and never throws:

import { parseOAuthRequired } from '@kaltura/intelligent-agents/experience';

const oauth = parseOAuthRequired(seg);
// { authUrl: "https://…/authorize?…", toolName: "jira_search", toolDisplayName: "Jira" } | null

Or register a handler with session.onOAuthRequired(handler). It exists on KalturaAvatarSession, KalturaChatSession and the KalturaAgentSession facade, with one signature, and survives a mode switch. A code sample is in MCP Integrations § OAuth-gated servers.

Unlike onToolCall(name, handler), onOAuthRequired takes no name: there is one OAuth event shape, not one per tool. Each handler runs isolated (a throw in one doesn't block the others) and fires once per distinct authUrl per turn. The session also re-emits a plain oauthRequired event, and an oauthRequiredResult event carrying each handler's return value. Headless callers get the same data from collectConverse().oauthRequired, an array in arrival order.

To show the link, build a show-link GenUI widget card client-side ({kind:'show-link', data:{url, label, description, safe}}) from authUrl and toolDisplayName.

The wait_for_response ACK: one wire contract, two transports

A tools.client tool built with waitForResponse:true blocks the agent's turn until the host app supplies a result. The wait lasts up to the tool's timeout seconds (default 30). The ACK is not a socket event. On both transports it is the same plain HTTPS POST, authorized by the session's own conversation KS. The agent speaks the acked value in the same turn:

POST {genieUrl}/assistant/tool_response
Content-Type: application/json
Authorization: KS <conversation ks>

{ "tool_name": "<toolName>", "tool_id": "<toolMetadata.id>", "tool_invocation_id": "<toolMetadata.id>", "response": { …your JSON result… } }

→ 200 {}

tool_id and tool_invocation_id both carry the toolMetadata.id from the parsed tool segment. KalturaAvatarSession#respondToTool and KalturaChatSession#respondToTool send exactly this. The tool segment may arrive over the socket (agent_raw_text) or over the HTTP /assistant/converse stream, but the ACK path is identical. One waitForResponse:true tool definition works unchanged on both transports.

respondToTool(id, response) returns a result object:

It throws invalid_state when the session is not connected, on a network failure, and bad_request for an empty id or a response that is not a plain object. See Client-Side Commands for the app-level contract (onToolCall → respondToTool).

init_response is a WebSocket-only event. It arrives as the delta of the first agent_raw_text event (carrying openingPhrase, threadId and messageId). It never appears in an /assistant/converse HTTP stream. openingPhrase is the intellect's opening_phrase, rendered for this session (a template over request_vars and sys__*). With a silent opening phrase (SILENT_OPENING) the opening turn still fires stvStartedTalking and stvFinishedTalking (about 0.5 s apart) but no audible speech, and the SDK surfaces that turn as SILENT_OPENING_LABEL ([silence]) on transcript, speechChunk and avatarStopTalking. A configured kickoff is sent on that stvFinishedTalking, and its first think delta fires responsePending. See Start the Conversation.

Stored-only message types. Two message types never stream. They appear only when you read a thread back (mgmt.messages / mgmt.threads, and the thread transcript as opening: ...). opening is the rendered opening_phrase, the thread's first assistant message: { type: 'opening', content: [{ type: 'avatar', content: '<rendered phrase>', metadata: {} }] }. summary is the rendered avatar_summary_config template, present after an avatar session ends: { type: 'summary', content: [{ type: 'text', content: '<rendered template>', metadata: {} }] }.

Session-completion signal: tell the backend a conversation is truly over

The request shape and auth are in Conversation & Analytics § Session-Completion Signal. The response has no payload to parse. Triggers and config: README.md § Ending a conversation cleanly.

4f. speechId: the per-utterance key (and the barge-in mechanism)

speechId appears on agent_raw_text, agent_start_speech, agent_end_turn, generatingSpeech, stvSpeechChunk, debug_stvTaskGenerated and stvFinishedGenerating. It identifies one agent utterance. Group a turn's events by speechId, never by timestamp, because barge-ins interleave turns.

• stvStartedTalking and stvFinishedTalking carry no speechId. Attribute them to the speechId of the surrounding stvSpeechChunk events.
• Barge-in. When the user interrupts, agentInterrupted fires and the following events carry a new speechId. The SDK tracks the latest speechId and drops captions for older ones, so it never shows stale captions after an interruption. Do the same in a custom client.
• Uninterruptible turns. Some turns can't be interrupted: the opening greeting, the replay after resume(), and the server's own status lines. speak() holds text sent during them and sends it as one turn when they end. If the session ends first, the held call resolves false.

Related docs