Connection and Handshake

← Back to System Internals Reference

Endpoints & Credentials

All of conversationManagerUrl, srsBaseUrl, turnServerUrl, and the enriched ks come from POST https://api.avatar.us.kaltura.ai/v1/application/appInit (see Backend API Reference). The agent is identified by partnerId (from the KS) and the KS itself, not by clientId or flowId. Both of those are optional and unused by Kaltura agents.

TURN configuration

Set explicit ports and transports on the TURN address. A bare turn:host gives no relay candidate. When that happens, packetsSent stays at 0 and the avatar can't hear you. Use all four forms:

• turn:HOST:80?transport=udp
• turn:HOST:443?transport=udp
• turn:HOST:80?transport=tcp
• turns:HOST:443?transport=tcp

iceTransportPolicy (SDK:wire.js iceConfig()) is per-channel:

• STV: 'relay', except on Firefox (isFirefox constructor option) where it's 'all'.
• ASR: always 'all', on every browser.

The setting matters less than it looks: the ASR server advertises only a private host candidate, so the pair relays through TURN either way, regardless of policy. This means the TURN URLs must be correct.

Full per-client matrix: Wire Protocol · Audio Channels §5.

Socket.IO Connection

import { io } from 'socket.io-client';

const socket = io(conversationManagerUrl, {   // from appInit
  path: '/socket.io',
  transports: ['websocket'],
  auth: { token: enrichedKs },                // from appInit
  query: {
    partnerId: '<your_partner_id>',           // derived from the KS
    clientId: undefined,                       // optional; unused for Kaltura agents
    flowId: undefined,                         // optional; unused for Kaltura agents
    billed_client: '',
    stickyId: '<random-16>',
    level: 'published',
    debugMode: true
  }
});

Full Connect Sequence (state-machine order)

Exact order from the platform's built-in client's connection state machine. Steps 1–5 are serial: each waits for the named inbound event before advancing. After step 5 the SDK runs two lanes in parallel: lane A is steps 6→7→9 (agent, ready, ASR uplink), lane B is step 10 (WHEP), which needs only the step 5 result. Step 11 runs once both lanes are done. The first lane to fail rejects connect() at once. Step 0 is never awaited: the mic prompt runs alongside the whole sequence and a denied mic emits a warning, never a failure. Timeouts appear in the last column.

Overall connecting timeout: 30s. It bounds every wait in the table, including the two ASR waits and the WHEP answer: an event or WHEP answer that lands after the deadline rejects connect() with ConnectTimeout.

Why step 3 has two timeouts, not one. The server emits clientConfiguration immediately on join. It emits joinComplete only after an awaited context-update call, which can take more than 5s under load. The SDK budgets the two waits separately: clientConfiguration gets 5s, joinComplete gets 20s. See Wire Protocol · Connection Basics §3 for the full rationale. Conflating them into one 5s budget causes spurious JoinRoomTimeout failures on loaded rooms.

This 30s deadline is set once, at the start of connect(). It keeps running through every step below, including the capacity queue, and is not paused or extended when the queue activates. If the account is queued (throwToNoAgent / availabilityResult{available:false}) and no slot frees up before the 30s runs out, connect() rejects with ConnectTimeout. To wait longer than that for a slot, call waitForCapacity({maxWaitMs, pollIntervalMs}) before connect(). This is a separate, opt-in poll with its own bound: maxWaitMs defaults to 300000ms. See Capacity & the queue.

Why approvedPermissions waits for playable video. Sending it too early clips the greeting: ICE connected fires about 2s before the first frame decodes, and approvedPermissions is what makes the server start speaking. _approve (SDK:session.js) gates on <video> reaching canplay/HAVE_FUTURE_DATA, with a 6s hard cap so a stalled video track can't block approval forever. See Wire Protocol · Connection Basics §3 for the full rationale. The opening line itself can't be interrupted; typed text sent during it is held (speak()) until stvFinishedTalking. For the fastest interruptible start, give the avatar a silent opening phrase (SILENT_OPENING) and let the session's kickoff option send the first turn on that event. See Start the Conversation.

The join payload (step 2): carries the agent/brain config

socket.emit('join', {
  client: clientId,            // optional
  room: roomId,                // a client-generated room id (also sent as 'channel')
  channel: roomId,
  kaltura: {
    entryId: <entryId>,            // only if context is a media entry
    contextId: <contextId>,        // category/entry the KB is scoped to
    contextType: <contextType>,    // the type of contextId (e.g. 'entry' vs 'category')
    threadId: <existingThreadId>,  // to resume a conversation thread
    force_experience: 'avatar_only',
    capabilities: {                // brain capabilities, same enum as intellect config
      avatar: 'on',
      generate_followup_questions: 'on',
      use_knowledge_base: 'off',   // forced off when an entryId is set
      // use_content_search, include_sources, etc.
    }
  },
  userAgent, userAgentHints, isMobile,
  channel_password: null, peer_name: 'unknown',
  peer_video: false, peer_audio: true
});

• Which intellect loads. The geniegpcid:<configId> in the KS tells the server which intellect (brain) to load.
• kaltura.ks is required. Omit it and the session server accepts the socket and emits onServerConnected, but never responds to join: no clientConfiguration/joinComplete arrives, and the connect stalls.
• force_experience in join is always avatar_only. buildJoin hardcodes it on every call, so the live avatar runtime never requests flashcards or summarization experiences through this path.
• capabilities and request_vars are genuinely client-controlled. By contrast, these two are read at join time. They can also be updated mid-session via the updateGenieContext socket event. The server merges them over defaults before forwarding them to the brain.
• The live socket carries the same brain protocol as the HTTP API. It streams agent_raw_text back in the same envelope as HTTP /assistant/converse, documented in Backend API Reference. Headless or text-only integrations use that HTTP path; the live avatar runtime uses the socket instead.

Related docs