Connection Basics

← Back to Wire Protocol

The SDK's committed fixture at test/fixtures/golden-session.json shows one full real session's events, redacted.

1. Channels at a glance

These are two separate peer connections by design. WHEP is receive-only and ASR is send-only, and they use different ICE policies (§5/§6). Separating them gives independent negotiation and failure isolation.

2. Socket.IO connection

This package (referred to as SDK throughout this reference) opens the socket the same way Kaltura's other clients do — its built-in player client and its embed widget client, both used here for comparison. Captured connection args:

io("https://conversation.avatar.us.kaltura.ai", {
  path: "/socket.io",
  transports: ["websocket"],
  auth:  { token: "<enriched KS from appInit>" },          // KSv2 string
  query: { partnerId: "<pid>", billed_client: "", stickyId: "<16 chars>",
           level: "published", debugMode: true }
})

Query params:

Auth/tenant scope: the KS in auth.token scopes the session to a partner and agent. Entitlement stays on for end-user sessions.

3. Connect sequence (state-machine order)

Order from the built-in client's connecting-state machine (steps 0–9, 11) plus the SDK/embed client video-ready gate (step 10). → = client emits, ← = client receives. Steps 1–5 are serial: each waits for its inbound event before advancing. After step 5 the SDK runs two lanes in parallel: steps 6→7→9 (agent, ready, ASR uplink) and step 10 (WHEP), which needs only the step 5 result. Step 11 runs once both lanes are done. The first lane to fail rejects connect() at once. Step 0 is never awaited: the mic prompt runs alongside the whole sequence and a denied mic emits a warning, never a failure.

Top-level machine states (the built-in client's connection state machine): preparing → connecting → connected → (disconnecting / disconnected / error). Overall connecting timeout 30s. Step timeouts are from the built-in client's connecting state (30e3 overall, 10000 server-connect, 5e3 join-room, 10000 agent, ASR 30s). Every wait in the table, including the two ASR waits and the WHEP answer, is also bounded by the 30s overall deadline: an event or WHEP answer that lands after it rejects connect() with ConnectTimeout.

Why joinComplete gets 20s, not 5s (deliberate deviation from the built-in client's single 5s join-room budget): clientConfiguration typically arrives right after join, but joinComplete can take longer than 5s under load. This SDK therefore budgets the two waits separately: clientConfiguration 5s, joinComplete 20s (SDK:session.js TIMEOUTS.joinRoom / TIMEOUTS.joinComplete). A client that reuses the built-in client's single 5s budget for both will see spurious JoinRoomTimeout failures on loaded rooms.

Steps 10–11 are a client-side refinement, not part of the built-in client's machine. The bare built-in client connecting-state machine approves on connectToASR onDone (sendApprovedPermissions → done → #connected). Its STV video is subscribed later, in the player layer.

The SDK and embed client instead gate approvedPermissions on STV video being playable first. (SDK:session.js _approve gates on the same canplay/HAVE_FUTURE_DATA settle logic; the embed client's own permission-approval check requires _micReady && _videoReady.)

Why it matters: approvedPermissions is what makes the server speak the opening line. ICE connected fires ~2s before the first frame decodes, so approving before <video> canplay (readyState ≥ HAVE_FUTURE_DATA, plus a ~300ms jitter settle) clips the greeting.

This wait is not unconditional. A 6s hard cap settles the gate anyway if canplay never fires (a stalled or dropped video track), so approval isn't blocked forever on a video that never decodes. This SDK applies the same 6s fallback; do the same in your client.

Related docs