Platform Architecture — Agentic Avatar System

This page is the map for platform developers. It shows how the SDK reaches the agentic backend, end to end. It gives enough detail to reimplement any layer with only a Socket.IO client and standard WebRTC, with no dependency on Kaltura's apps, widgets or libraries.

The SDK works across four layers:

• the management API: agents, avatars, catalog, intellects (an agent's prompts, tools and knowledge), application
• the conversation API: conversations, threads, messages, feedback, followups
• the live-avatar control plane: the Socket.IO session plus the ASR/STV WebRTC media
• the scripted-video control API: /v1/avatar-session/*

Symbol names below are the stable contracts to navigate by. Exact details live in Wire Protocol.

Contents

• The Three Planes
• Backend Services Map
• Text Conversation Flow
• Video Runtime Protocol: The Big Picture
• Two Session Modes (choose the right one)
• SDK Entry Points
• Resilience & Failure Handling: Overview

The Three Planes

An app uses only the planes it needs.

A live conversation also runs three flows at once: conversation control, agent orchestration and your own expertise. See Inside a Live Conversation.

Backend Services Map

The SDK talks to six endpoints. Each has an option that overrides it.

The scripted-video control API (avatar-session/*) is a separate API from the management API, reached through agenticUrl. Details: Scripted-Video Sessions.

REGIONS[region] and the explicit *Url options are described in Regions and base URLs. The socket, video and TURN values always come from appInit.

The SDK checks every WHEP URL with whepUrlHasPrivateIp(). A URL that points at a private address makes connect() throw whep_private_ip, because a browser can't reach it.

Text Conversation Flow

The simplest path: no video, fully headless. The client calls POST {genieUrl}/assistant/converse with a conversation token or an agent token (token kinds). The response is an NDJSON (or SSE) stream of segments. Segment type values and parsing rules are the same as the avatar's agent_raw_text stream (see the events catalog). Endpoint details: Backend API Reference.

Video Runtime Protocol: The Big Picture

The live talking avatar: the full bidirectional protocol.

A full interactive agentic avatar is three concurrent channels over one Socket.IO connection plus two WebRTC peer connections:

                          ┌───────────────────────────────────────────────┐
                          │   conversationManagerUrl                      │
                          │   (Socket.IO control plane)                   │
   ┌──────────┐  socket   │                                               │
   │          │◄─────────►│  • handshake / join / session                 │
   │  YOUR    │           │  • agent_raw_text  (agent output, NDJSON)     │
   │  BROWSER │           │  • stvStartedTalking / stvFinishedTalking     │
   │  CLIENT  │           │  • ASR WebRTC signaling relay                 │
   │          │           └───────────────────────────────────────────────┘
   │          │  WebRTC (ASR, mic→server)   via socket-relayed SDP/ICE
   │          │═════════════════════════════════════════►  speech-to-text
   │          │
   │          │  WebRTC (STV, server→video) via WHEP (HTTP SDP)
   │  <video> │◄═════════════════════════════════════════  avatar video (srsBaseUrl)
   └──────────┘

• Control plane: one Socket.IO connection. It carries the handshake, the agent's streaming text, talking state, and ASR signaling.
• ASR channel (uplink): a WebRTC peer connection that publishes your microphone. SDP offer/answer and ICE are relayed through the Socket.IO connection (custom asr-webrtc-* events). Your speech comes back as the agent's reply.
• STV channel (downlink): a WebRTC peer connection that receives the avatar video and audio. It uses standard WHEP (plain SDP over HTTP), independent of the socket.

The client never calls an LLM. It only publishes audio, receives video, and receives the agent's text as agent_raw_text deltas, in the same format as the /assistant/converse NDJSON.

For the exact connect sequence, wire shapes, endpoints, and scaling model, see Architecture Reference.

For the exhaustive map, see Wire Protocol. It covers every socket event with its payload shape, the exact ICE/SDP/WHEP config, the parsed agent_raw_text delta types, and a turn-by-turn event trace.

This section is the orientation. Those docs are the reference.

Two Session Modes (choose the right one)

There are two session modes, and they are NOT interchangeable. Scripted sessions render speech you author, line by line. Interactive sessions run all three conversation flows for you: conversation control, agent orchestration, and your plugged-in expertise. Interactive agentic is the product experience. Scripted (puppet) is a narrow authoring tool.

The protocol above describes the interactive path. The scripted path has no text-in of its own: the SDK wraps only say-audio. You provide pre-rendered speech audio (for example, from your own TTS call) and its duration. Full auth/lifecycle details: Scripted-Video sessions.

Agents with no avatar run in audio/phone mode, which this SDK does not support end to end. See Wire Protocol · Audio Channels §5b.

SDK Entry Points

Management checks the token kind before any network call whenever the kind is known client-side. A minted Token records its kind (admin, conversation, agent, widget). A raw encrypted KS string can't be inspected, so Management sends the call without that check. KalturaAvatarSession is the live socket and WHEP runtime described in "Video Runtime Protocol" above. Socket.IO is injected into it, never bundled. Entry points, modules and how-tos: README.md § Architecture.

Resilience & Failure Handling: Overview

How the system behaves under network failures, disconnects, and device problems. KalturaAvatarSession recovers in three layers: the control socket (Socket.IO reconnect, then a cold reconnect if state was lost), the WebRTC media peers (ICE restart for ASR, WHEP re-subscribe for STV), and a cold reconnect of the whole session as the last step (two attempts). A health watchdog catches peers that die without an ICE event, and closing the tab releases the server side. A custom client that skips KalturaAvatarSession must implement these itself.

Recovery layers, the failure-mode matrix, device-permission handling and the tool-call spiral breaker: Resilience & Failure Handling.

A conversation ending cleanly is a separate concern from recovering from failure. On tab-close, backgrounding, bfcache freeze, or an explicit disconnect(), the SDK sends POST /thread/session_completed to mark the thread as over.

App-facing config and the trigger table: README.md § Ending a conversation cleanly.