Connection and Handshake

← Back to Architecture Reference

Endpoints & Credentials

conversationManagerUrl, srsBaseUrl, turnServerUrl, turnCredentials and the conversation ks come from POST https://api.avatar.us.kaltura.ai/v1/application/appInit (see Backend API Reference). The agent is identified by partnerId (from the KS) and the KS itself.

TURN configuration

Pass turnServerUrl and turnCredentials from appInit to the session. The TURN URLs, credentials and ICE policy are in Wire Protocol · Audio Channels.

Socket.IO Connection

The SDK opens the socket with the URL, auth token and query params in Socket.IO connection.

Full Connect Sequence (state-machine order)

connect() runs these steps in order (the step numbers match the comments in src/experience/session.js). Steps 1-5 are serial: each waits for the named inbound event. After step 5 the SDK runs two lanes in parallel:

• Lane A: steps 6→7→9 (agent, ready, ASR uplink).
• Lane B: step 10 (WHEP), which needs only the step 5 result.

Step 11 runs once both lanes are done. The first lane to fail rejects connect() at once. Step 0 is never awaited: the mic prompt runs alongside the whole sequence, and a denied mic emits a warning, never a failure. Timeouts appear in the last column.

Step 8 is unused. In audio/phone mode there is no STV session, so lane B is skipped (Wire Protocol · Audio Channels §5b).

Overall connecting timeout: 30s (timeouts.overall). The deadline is set once, at the start of connect(). It bounds every wait in the table, including the two ASR waits, the WHEP answer and the capacity queue, and it is not paused or extended when the queue activates. An event or WHEP answer that lands after it rejects connect() with timeout. To wait longer for a slot, call waitForCapacity({maxWaitMs, pollIntervalMs}) before connect(). It is a separate, opt-in poll, and maxWaitMs defaults to 300000ms. See Capacity & the queue.

Step 10: the playable-video gate

At timeouts.firstFrame with no frame, the SDK re-subscribes once. A second miss continues with warning media_no_video and mediaReady {degraded:true}. If ICE never connected, the second miss rejects with ice_failed instead.

Timeout errors. Every wait in the table rejects with code: 'timeout'. The names in the table (JoinRoomTimeout and the rest) are labels in detail, not codes. Branch on error.phase, which names the step that ran out:

error.retryable is true on timeout, capacity_unavailable, whep_timeout, a whep_failed caused by the network, stv_session_gone, ice_failed and reconnect_failed, because the same call can succeed on a later try. A whep_failed that carries an HTTP status does not set it. Both fields are also in error.toJSON().

// Retry a connect that can succeed later. Give up on anything else.
for (let attempt = 1; ; attempt++) {
  try { await session.connect(); break; }
  catch (err) {
    if (!err.retryable || attempt === 3) throw err;
    await new Promise((r) => setTimeout(r, 1000 * attempt));
    session.disconnect();
    session.setToken(await mintFreshToken());   // disconnect() clears the token
  }
}

Why step 3 has two timeouts. joinComplete can arrive later than clientConfiguration under load. The SDK budgets the two waits separately: clientConfiguration gets 5s (TIMEOUTS.joinRoom), joinComplete gets 20s (TIMEOUTS.joinComplete). A single 5s budget for both causes spurious timeout failures with phase join on loaded rooms.

Why approval follows the video gate. approvedPermissions is what makes the agent start speaking. ICE connected fires about 2s before the first frame decodes, so step 11 sends it only after lane B has passed its gate. The gate ends at timeouts.firstFrame, so a stalled video track cannot block approval forever. Do the same in a custom client.

The opening line itself can't be interrupted. Typed text sent during it is held (speak()) until stvFinishedTalking. For the fastest interruptible start, give the avatar a silent opening phrase (SILENT_OPENING) and let the session's kickoff option send the first turn on that event. See Start the Conversation.

Connecting again. disconnect() clears the token. Call setToken() with a fresh conversation KS before the next connect() or prepare(), or they reject with invalid_state. From the error state, call disconnect() first.

Start faster

prepare(). await session.prepare() does steps 1 to 3 ahead of time: it opens the socket, sends join and waits for the join ack. Call it when the user is likely to start soon (page load, hover over the start button). The later connect() skips those steps. prepare() never asks for the microphone and never creates an avatar session, so a prepared page costs nothing until connect().

Resource hints. Add these to the page <head> so the browser opens the connections while the page loads. The origins are the ones you pass as conversationManagerUrl and srsBaseUrl, and your TURN host.

<link rel="preconnect" href="https://MESSAGING_HOST">
<link rel="preconnect" href="https://MESSAGING_HOST" crossorigin>
<link rel="preconnect" href="https://SRS_HOST" crossorigin>
<link rel="dns-prefetch" href="https://TURN_HOST">

Connect timings. session.timings returns the phases of the current or last connect() in ms from the start of that call. A phase that was not reached is absent. After a successful connect the session emits connectTimings once with the same object. A failed connect() attaches the phases it reached to error.timings.

The rows group related phases and are not in time order. firstFrame is also absent when there is no videoEl, when the browser has no requestVideoFrameCallback, or when no frame arrived in time (mediaReady then has degraded: true).

Tuning. timeouts overrides any wait in the table above (overall, serverConnect, joinRoom, joinComplete, agent, asr, firstFrame, prepareIdle) and the WHEP, recovery and watchdog limits (whepTry, whepTries, whepBackoff, whepRelease, recover, healthTick, videoStall, iceStuck, coldAttempts, coldBackoff; defaults in Resilience). reconnectionDelay (default 250 ms) and reconnectionDelayMax (default 2000 ms) set how soon and how often a dropped socket retries.

The join payload (step 2): carries the agent config

Field-by-field shape: events catalog.

• Which intellect loads. The geniegpcid:<configId> in the KS selects the intellect (the agent's configuration). An agent token also carries agentid:<agentId>.
• Always send kaltura.ks. A join without it never gets clientConfiguration or joinComplete, and the connect stalls until the timeout.
• force_experience is always avatar_only. buildJoin sets it on every call. See Wire Protocol · Client Configuration for what that means for widgets.
• capabilities and request_vars are client-controlled. They are sent at join. You can update them mid-session with the updateGenieContext event (events catalog).
• The socket carries the same agent protocol as the HTTP API. It streams agent_raw_text in the same envelope as HTTP /assistant/converse, documented in Backend API Reference. Headless or text-only integrations use that HTTP path. The live avatar runtime uses the socket.

Related docs