SDK Reference

This page documents the @kaltura/intelligent-agents JavaScript SDK's own object-level API: its classes, functions and constructor options across the Management and Experience entry points. For the raw HTTP endpoints the SDK wraps, see the Backend API Reference.

Contents

Management

import { Management } from '@kaltura/intelligent-agents/management';

const mgmt = new Management({ partnerId, adminSecret });

// 1. provision a complete agent from a one-line brief
const admin = await mgmt.sessions.createAdminToken({ userId: 'admin@example.com' });   // admin KS, server-side only
const { agentId, configId, widgetId } = await mgmt.provision({
  brief: 'A helpful support agent for a video platform',
  ks: admin.ks,
});

// 2. headless streaming conversation (auto-mints a conversation token)
for await (const seg of mgmt.converse(configId, 'Hello!')) {
  if (seg.type === 'text') process.stdout.write(seg.content);
}

// 3. assembled result (text + toolCalls + threadId + _meta)
const result = await mgmt.converseOnce(configId, 'Hello!');
console.log(result.text, result.threadId);

provision() returns {name, configId, avatarId, agentId, widgetId, profile, personaLint, blocks?, _meta}. personaLint (see lintPersonaIdentity below) is a warning-only check for persona-name drift. It never fails provision(). Read personaLint.findings to surface or act on it. On a partial failure provision() throws a KalturaError with code:'provision_failed'. err.body.failedStep names the step that failed, and err.body.createdSoFar lists the ids already created ({configId?, avatarId?, agentId?}), so you can clean up an orphaned intellect, avatar or agent.

converseOnce returns { text, threadId, messageId, segments, toolCalls, experiences, experiencesList, kindCounts, spiralStopped, truncated, _meta }. spiralStopped:true means a tool spiral was detected and cut short. Check toolCalls[0] and re-prompt. truncated:true means the stream hit maxSegments (a runaway-segment guard, default 2000) before finishing. The gathered content is returned, but the turn is incomplete.

Pass recoverFromSpiral: true (to converseOnce or conversations.send) to recover from an empty spiral: a tool-call loop so long the agent never speaks in that turn (spiralStopped:true with text:''). The SDK resends the turn once with a recovery nudge. The result then carries spiralRecovered (true/false) and firstAttempt: {toolCalls, spiralStopped} from the discarded attempt. Off by default. The mechanism is the same as on a live session: see Resilience.

mgmt.converse()/converseOnce() auto-mint a plain conversation token. That is fine for talking to the intellect, but without an agentId the resulting thread's agent_id is "default", so it can never match a lifecycle rule scoped to object.agent_id. For that, pass { agentId, userId } (converseOnce(configId, msg, { agentId, userId })), or mint a token yourself and pass it as the ks argument. Which method to use: Conversation token or agent token?.

Session tokens

Conversation or agent token: which one to use.

Pass a per-user userId to every conversation or agent token. A token without one is a single identity shared by every holder. sessionType: 'admin' is the explicit opt-out to KS type 2. With a widget token, anyone who has a threadId can continue that thread, so store each threadId per user on your server. When users must stay separate, mint createAgentToken({ agentId, userId }) on your server, call appInit with it and send the returned KS to the browser. What each token can reach: Security & Compliance § Session type.

Experience

Browsers and mobile. The live runtime needs a browser with WebRTC and getUserMedia, served over HTTPS. Desktop Chromium, Firefox and WebKit are tested. iOS and Android browsers are not tested on real devices. There is no native iOS or Android SDK.

import { KalturaAvatarSession } from '@kaltura/intelligent-agents/experience';

const session = new KalturaAvatarSession({
  token,               // conversation KS, appInit.ks
  conversationManagerUrl,  // from appInit
  srsBaseUrl,          // from appInit
  turnServerUrl,       // from appInit
  // Markup: <video autoplay playsinline></video> and <audio autoplay></audio>.
  // A missing <audio> makes querySelector return null and the SDK falls back to merged mode.
  videoEl: document.querySelector('video'),
  audioEl: document.querySelector('audio'),     // recommended: voice on its own element, see below
  socketFactory: (url, opts) => io(url, opts),  // inject socket.io
  kickoff: 'Greet the user and briefly say how you can help.',  // first turn, sent once by the SDK
  // region: 'frp2',  // same value you gave Management, if your partner is not in the US
});

session.on('transcript', ({ text }) => console.log(text));
session.onToolCall('navigate_to_slide', ({ slide_num }) => deck.goTo(slide_num));

await session.connect();
session.speak('Tell me about onboarding.');

Attach every listener before connect(). Some events fire before connect() resolves: streamReady, capacityChanged, mediaReady, connectTimings, disclosure, and a warning (an early mic warning, or media_no_video). KalturaAgentSession also emits transportChanged before its connect() resolves and modeChanged before switchMode() resolves. A listener added after the await misses them, so call .on(...) first. On a KalturaAgentSession, mediaReady, connectTimings, mediaRecovering and mediaRecovered are forwarded, but streamReady, capacityChanged and disclosure stay on the transport: listen for them on session.transport inside a transportChanged handler.

How speak(text) works: it injects text into the conversation on the same path as the viewer's own voice transcript. The agent treats it as a new turn and replies in its own words, not a verbatim echo of text. Need exact scripted playback instead? See Scripted avatar sessions.

When it sends: speak(text) is safe to call at any moment after connect() resolves. If the agent is idle, thinking, or talking a normal reply, the text goes out now (a talking avatar stops mid-sentence and answers it). If the agent is in a turn typed text can't interrupt (its opening line right after connect()/resume(), or its own "are you still there?" check-in), the text is held and sent the instant that turn ends. Several speak() calls during one held turn go out as one turn, one text per line. The promise resolves true once sent, false if the session ended first. Details: Dynamic Data Injection.

Starting the conversation: pass kickoff: 'text' (or { text, echo? }) and the SDK sends that text as the first turn, exactly once per session object, the moment the server accepts input. Pair it with a silent opening phrase on the intellect (SILENT_OPENING, exported from ./management and ./experience) and the agent's first words are its own interruptible reply, written by the model. Never re-sent on resume() or a reconnect. Scripted or silent opening: Start the Conversation § Choose an opening.

All transports are injected: socketFactory, rtcConstructor, fetch, getUserMedia. You can pass fakes, and the SDK stays zero-dependency.

The SDK binds the avatar's tracks to the elements you pass and applies no CSS of its own. Size the box yourself with object-fit: cover (aspect-agnostic, no letterbox/pillarbox bars). Pass both videoEl and audioEl so the picture and the voice live on separate elements: a UI re-render that replaces the <video> then costs you the picture for a moment, not the whole avatar. With videoEl alone the SDK merges both tracks onto that one element, and omitting videoEl gives you session.avatarStream for headless rendering. See Avatar audio and video rendering below.

Don't hide a loading spinner on 'streamReady'. It fires at the initial signaling handshake, before any video track exists. Listen for 'mediaReady' instead: it fires once per connect, unconditionally, with {mode:'video', videoWidth, videoHeight} when the first video frame is painted (dimensions are 0 when there is no videoEl; use 'videoMetadata' if you need real dimensions), {mode:'video', videoWidth:0, videoHeight:0, degraded:true} when no frame arrived (see warning media_no_video), or {mode:'audio'} immediately if the session falls back to audio-only (capacity limited).

session.on('mediaReady', ({ mode }) => spinner.hidden = true);   // covers both video and audio-only sessions

Avatar audio and video rendering

The avatar downlink delivers the avatar's video and audio as two separate tracks. The SDK combines them into streams it owns and binds them to the elements you give it. Three shapes, all on the same KalturaAvatarSession (and KalturaScriptedVideoSession):

// Recommended: picture on <video>, voice on its own <audio>.
new KalturaAvatarSession({ ...cfg, videoEl: document.querySelector('video'), audioEl: document.querySelector('audio') });

// Merged: one <video> plays picture and voice. Fine when you own the element's whole lifetime.
new KalturaAvatarSession({ ...cfg, videoEl: document.querySelector('video') });

// Headless: bind nothing, render the tracks yourself.
const session = new KalturaAvatarSession({ ...cfg, videoEl: null });
session.on('track', () => canvasRenderer.use(session.avatarStream));

Split is the recommended shape. The two elements fail independently, and the voice is the part users notice first:

• UI frameworks replace elements without asking. A React, Vue or Svelte re-render, a route change or a conditional block can unmount the <video> and mount a new one. In the merged shape that single element carries the voice too, so the avatar goes silent and blank at once. In the split shape, keep the <audio> on a stable node (the document body, an app-shell component that never re-renders) so the conversation keeps going, and call session.setVideoEl(newVideo) when the picture's element mounts again. If the framework moves the same <video> node instead of replacing it, the browser leaves it paused: call session.startPlayback() to resume it.
• The video is a texture source (chroma-key, canvas, WebGL) and the voice must still be audible. A muted source element cannot play sound, by browser design.
• The picture should start before the autoplay gesture. In the split shape videoEl carries no audio track, so browsers let it play at once with no muted needed, while audioEl waits for startPlayback() from a click.
• You want the voice on its own element for mixing, visualizers or per-element control.

The merged shape stays fully supported for apps that create the <video> once and keep it for the life of the session.

Every element gets one srcObject write and one play() per binding. In the merged layout videoEl.srcObject carries one video and one audio track; in the split layout the audio track lives on audioEl only. disconnect(), a terminal ended and a failed connect() set srcObject to null on every bound element. videoEl, audioEl, setVideoEl(el) and setAudioEl(el) accept null or an object with a play() function (a real media element, or a jsdom one in tests); anything else throws KalturaError bad_request. The SDK creates no DOM and adds no CSS.

Two more members speed up and measure connect() (Start faster):

Media recovery (a WHEP re-subscribe after a stall) never touches your elements: the new tracks are swapped into the same streams, and only an element that the browser paused meanwhile (Firefox does this) gets one play() call. The one exception is an element whose srcObject your app replaced itself (for example set to null to hide the avatar): recovery binds it again. On Chromium a remote audio track is only decoded while some media element plays it, so a headless app that mixes avatarStream through Web Audio must keep a muted <audio> bound to the track; Firefox and WebKit do not need it. Calling disconnect() from inside a 'track' listener is safe.

Multiple avatars on one page. Each avatar is its own session with its own elements. Sessions share nothing: mute, volume and output device are per session.

const sessions = configs.map((cfg) => new KalturaAvatarSession({
  ...cfg,
  videoEl: document.querySelector(`#avatar-${cfg.id} video`),
  audioEl: document.querySelector(`#avatar-${cfg.id} audio`),
}));

// Optional: also mix every voice into one Web Audio graph. Keep a muted <audio> bound
// per session so Chromium decodes the remote audio (see the note above).
const ctx = new AudioContext();
const mixer = ctx.createGain();
mixer.connect(ctx.destination);
for (const s of sessions) {
  s.on('mediaReady', () => ctx.createMediaStreamSource(s.avatarStream).connect(mixer));
}

await Promise.all(sessions.map((s) => s.connect()));   // after the listeners: mediaReady fires before connect() resolves

Mixing or recording the avatar. avatarStream (every track) or videoEl.captureStream() (default mode) feed MediaRecorder or a Web Audio graph. Chromium and Firefox report the mimeType as video/webm;codecs=vp8,opus for a stream with both tracks.

const recorder = new MediaRecorder(session.avatarStream);
recorder.ondataavailable = (e) => chunks.push(e.data);
recorder.start(1000);

A recording covers one connection. When the SDK recovers the media connection it replaces the tracks inside avatarStream, so the running recorder stops (onstop fires). Start a new segment from the mediaReady event.

Non-fatal problems on this path arrive as warning events, { code, message, ... }:

Text-only chat (KalturaChatSession)

The same agent, thread, and tool contract as the avatar, over plain HTTPS. No mic, no camera, no video element, no socket. The class never imports getUserMedia, WHEP, or a socket factory, so a chat-only page ships none of that and never triggers a browser permission prompt.

import { KalturaChatSession } from '@kaltura/intelligent-agents/experience';

const chat = new KalturaChatSession({ token });   // same conversation KS as the avatar
chat.on('transcript', ({ text, type }) => render(type, text));
chat.onToolCall('navigate_to_page', ({ page }) => router.go(page));

await chat.connect();                             // no network, marks the session live

const turn = await chat.sendText('What does the Presenter plugin do?');
console.log(turn.text, chat.threadId);

Each sendText() is one streamed POST /assistant/converse turn; turns are serialized, and segments are processed mid-stream. An onToolCall handler that calls respondToTool(call.toolMetadata.id, result) unblocks a waitForResponse:true tool within the same turn. It is the same wire ACK the avatar transport sends (see Wire Protocol). updateRequestVars() / setDynamicPrompt() carry the same merge semantics as the avatar session; the full map rides every turn.

If the server refuses a turn with an in-stream error segment (for example a threadId owned by another user), sendText() rejects with thread_access_denied or stream_error and emits error. Pass onErrorSegment: 'warn' to get a warning and a resolved empty turn instead. Details: Conversation & Analytics.

Thread continuity is symmetric in both directions: seed cfg.threadId with another session's threadId getter to continue that conversation here, or hand this session's threadId to a KalturaAvatarSession. When allow_client_variables is off, sendText() either returns a silent empty turn with a once-per-session empty_turn_with_request_vars warning, or rejects with client_variables_disabled (see the {{var}} section below).

KalturaChatSession carries the same agent-stall watchdog and dead-air masking as the avatar transport (brainStallMs/brainStalled, responsePending/responseSettled), plus the same soft tool-call-spiral signal (toolSpiralLimit/toolSpiralDetected; see Resilience below). It has no hard-limit cold-reconnect escalation, since there is no socket to recover through.

One conversation, switchable transports (KalturaAgentSession)

A facade that runs a single conversation over either transport and switches mid-conversation without losing the thread:

import { KalturaAgentSession } from '@kaltura/intelligent-agents/experience';

const agent = new KalturaAgentSession({
  token,
  mode: 'chat',                                  // start text-only: zero permission prompts
  avatar: { videoEl, conversationManagerUrl, srsBaseUrl, turnServerUrl, socketFactory },
  chat: {},                                      // per-transport cfg
});
agent.onToolCall('navigate_to_page', go);        // registered once, survives every switch
agent.on('modeChanged', ({ mode, threadContinuity }) => showBanner(mode, threadContinuity));
await agent.connect();

await agent.sendText('Hi!');
await agent.switchMode('avatar');                // call from a real click: the mic prompt needs a user gesture

The facade owns the state machine (idle → connecting → connected ⇄ switching → closed | failed), the canonical request_vars map, and the onToolCall registry. All three carry over on every switchMode(). Switching is tear-down-and-reconstruct: the old transport disconnects, the new one is constructed seeded with the live threadId and full context, and sendText() calls during the blip are buffered (up to 8). The facade forwards the events an app needs to render the conversation: transcript, turnStart, turnEnd, toolCall, toolCallResult, toolCallInvalid, error, warning, responsePending, responseSettled, agentActionDenied, ended on both transports, plus speechChunk, avatarStartTalking, avatarStopTalking, interrupted in avatar mode. Listeners on the facade survive every switch. Events that pair with a transport-only method (disclosure with acknowledgeDisclosure(), micStarted with startMic()) and the mic/video controls stay on the transport: use the transport getter and rewire on each transportChanged event. For the full state-transition table and switch UX rules, see Voice Input Modes.

{{var}} personalization (request_vars)

Request variables are the SDK's one channel for app-supplied context: values the agent's prompt reads via {{var}} templating: a viewer's name, an account tier, or a whole JSON context blob. Seed them at join time via cfg.requestVars, update them mid-session via updateRequestVars():

const session = new KalturaAvatarSession({ token, /* … */, requestVars: { user_name: 'Ada' } });
// later, once you learn more about the viewer:
session.updateRequestVars({ account_tier: 'enterprise' });

updateRequestVars(vars) merges what you pass into the session's variable map. Merge, persistence and reconnect rules: Dynamic Data Injection. session.setDynamicPrompt(data) is sugar over the same channel. It serializes data into the single variable page_context. Pair it with the PAGE_CONTEXT_PROMPT block exported from ./management so the prompt references {{page_context}}.

The intellect's allow_client_variables flag must be on. It is on by default. Pin it with intellects.setClientVariablesEnabled(configId, true, adminKs). If it is off, the turn fails in one of two ways:

• Silent empty turn. The turn comes back empty with no error on the wire. Both session classes detect it and emit a once-per-session warning event, { code: 'empty_turn_with_request_vars', message, requestVarKeys } (variable names only, never values).
• Loud error. The request gets a 403 and the call rejects with a KalturaError with code client_variables_disabled. KalturaChatSession.sendText() also emits error, and the session stays connected.

For merge and persistence semantics, size headroom, tool interpolation, the lintPrompts pre-flight, and a worked example composing request_vars with speak() and submitStructuredDataForm(), see Dynamic Data Injection.

Tap-to-talk (push-to-talk voice)

For the app-level decision of whether to use this at all, and the UI, accessibility and safety design around it, see Voice Input Modes. This section is the API reference.

startTapToTalk()/endTapToTalk() are a distinct voice-input mode from typed-text speak()/interrupt(). The mic uplink is connected once connect() resolves. Tapping opens a capture window (tapToTalkStart). Releasing closes it and produces the turn from what was captured (tapToTalkEnd). That turn arrives through the same agentTurnToTalk/transcript pipeline as any open-mic turn. There is no separate transcript path to wire up.

startTapToTalk() throws capability_disabled unless session.capabilities.tapToTalk (from clientConfiguration.isTapToTalk) is true. Treat isTapToTalk as a fixed, per-agent deployment choice, never a live per-session toggle. Build the UI conditionally on the flag:

if (session.capabilities.tapToTalk) {
  micButton.addEventListener('click', () => {
    if (session.tapToTalkActive) session.endTapToTalk();
    else session.startTapToTalk();
  });
  session.on('tapToTalkStarted', () => micButton.setAttribute('aria-pressed', 'true'));
  session.on('tapToTalkEnded', () => micButton.setAttribute('aria-pressed', 'false'));
} else {
  micButton.remove();   // open-mic agent: no tap control to show
}

speak()/interrupt() throw invalid_state while a tap is open, because they would create a duplicate turn. startTapToTalk()/endTapToTalk() throw invalid_state if called out of order, and use the same requireDisclosureAck disclosure gate as speak(). Add silence-based auto-stop and a hard max-duration cap so an abandoned tap (tab closed, navigation away) cannot leave a capture window open. Treat a disconnect or pagehide while tapToTalkActive as an implicit endTapToTalk().

Build the control as click-to-toggle, not press-and-hold: it suits longer utterances, and it satisfies WCAG 2.5.2 Pointer Cancellation on its own, since the down-event never fires the action.

Resilience: agent stalls and tool-call spirals

Both session classes watch for an agent that goes quiet instead of answering, and for one that loops instead of narrating. Only KalturaAvatarSession can recover from a hard loop, since it has a socket to cold-reconnect through. The full failure-mode matrix is in Resilience & Failure Handling.

• Agent-stall watchdog (brainStallMs, default on, KalturaAvatarSession + KalturaChatSession). It emits brainStalled ({afterMs, count}), repeating for as long as nothing perceivable (spoken/avatar content or a GenUI widget) follows a turn. On the chat transport, a bare keepalive segment does not count as perceivable output either.
• Dead-air masking (responsePending/responseSettled, both transports). responsePending ({}) fires the moment a turn starts awaiting the agent's first perceivable output (spoken/avatar/GenUI content). It fires again when the server starts a turn itself, so a kickoff reply shows "thinking" before the first word. responseSettled ({}) fires once that output arrives, the turn ends, an interruption occurs, or the session tears down. Use this pair to show and hide a "thinking…" indicator instead of leaving the avatar's face frozen. examples/browser-experience.html shows it working.
• Tool-call spiral circuit breaker: a guard against an agent that re-issues the same client command instead of narrating. Soft (toolSpiralLimit, default 10, per turn, on both KalturaAvatarSession and KalturaChatSession): emits toolSpiralDetected. This is signal only and does NOT call interrupt() (a mid-turn barge-in truncates the turn's own narration with no recovery, see Tool spirals starve the voice). Hard (hardToolSpiralLimit, default toolSpiralLimit * 3, session-scoped, immune to turn-boundary resets, KalturaAvatarSession only; the chat transport has no socket to reconnect through): emits toolSpiralRecovering ({count, limit, lastTurnText}) and forces a cold reconnect. It opens a new socket that replays threadId, so the conversation continues.
• Spiral recovery auto-resend (recoverFromSpiral, default true). A hard-spiral cold reconnect restores connectivity but would otherwise drop the turn that triggered it. With the default on, once the reconnect succeeds the SDK resends that turn's text once. The text is prefixed with the SPIRAL_RECOVERY_PREFIX instruction (the headless path uses the same one: Conversations#send({recoverFromSpiral:true}), see Management) and still passes through your onBeforeSend guardrail. The SDK emits spiralRecovered ({text}, the original un-prefixed text), for example to show "Let me get that for you". Set recoverFromSpiral: false to opt out and handle it yourself. toolSpiralRecovering's lastTurnText tells you what was abandoned.

const session = new KalturaAvatarSession({ token, /* … */, recoverFromSpiral: false });
session.on('toolSpiralRecovering', ({ lastTurnText }) => {
  if (lastTurnText) myOwnResend(lastTurnText);
});

The session also recovers without an ICE event: a health watchdog catches a peer closed from outside and a video stream that stops arriving, a WHEP 404 or 409 gets a new avatar session on the live socket, a failed cold reconnect is tried twice, and an online event retries a down socket at once. Each WHEP request has a deadline and bounded retries. Zero-candidates fail-fast has its own faster path. See ARCHITECTURE-REFERENCE.md § Resilience & Failure Handling for the exact timings and event payloads.

Devices and media quality

session.on('hardwareMuteChanged', ({ muted }) => micIndicator.set(muted));
session.on('localSpeakingChanged', ({ speaking }) => localSpeakerIndicator.set(speaking));
session.on('localMicLevel', ({ level }) => micButton.style.setProperty('--level', level)); // 0-1, every 50ms tick

const { mics, speakers } = await session.listDevices();
await session.switchMic(mics[1].deviceId);       // replaceTrack, no renegotiation
await session.setAudioOutput(speakers[1].deviceId); // setSinkId on the element carrying the audio track (audioEl if set, else videoEl)

await session.setAsrBandwidth(24); // kbps, applied live via RTCRtpSender.setParameters

• hardwareMuteChanged ({muted}): fires when the OS/hardware mutes or unmutes the active mic track (track.onmute/onunmute). Mute is debounced 5s (many platforms blip onmute during device switches); unmute fires immediately.
• localSpeakingChanged ({speaking}): an instant local speaking indicator from client-side volume analysis (AnalyserNode, 50ms sampling, threshold via localVadThreshold, default 300). It is independent of the server's turn-taking signals. It starts only while at least one listener is registered, so a session that never listens pays zero Web Audio cost. It stops when the last listener unsubscribes.
• localMicLevel ({level}, 0-1): the same 50ms sampler's continuous volume, emitted on every tick rather than only on threshold transitions. Use it for a real-time UI meter, such as a mic button that fills with live input volume. It shares the lifecycle above: a listener for either localMicLevel or localSpeakingChanged starts the sampler, and it stops once every listener for both has unsubscribed.
• listDevices(): {mics, speakers} from navigator.mediaDevices.enumerateDevices() (video input omitted, since an avatar session has no local camera). Returns empty lists, and does not throw, when headless or without permission.
• switchMic(deviceId): swaps the ASR uplink's sender track via replaceTrack, no renegotiation. It rewires the hardware-mute watch and VAD onto the new stream and stops the old one.
• micStartMode: 'deferred' (constructor option) + startMic() + micStarted (getter and event): connect with no mic at all. connect() skips getUserMedia entirely. Call startMic() later from a real user click, so the browser's permission prompt is anchored to the gesture instead of firing on page load. It acquires the mic, attaches it via replaceTrack (no renegotiation), honors any mute() issued before the mic existed, re-applies maxAsrBitrateKbps, and emits micStarted. Until then micStarted is false, typed turns (speak()) work normally, and startTapToTalk()/switchMic() throw mic_not_started. A denied prompt rejects startMic() with a typed mic error (mic_permission_denied, mic_not_found, mic_in_use) and leaves the session connected, so the viewer can retry or stay typed-only. In the default 'immediate' mode the same failure surfaces as a warning with that code instead (see the warning table above), because connect() never waits on or fails for the mic. startMic() is idempotent: it returns at once when a mic is attached and joins an acquire that is still in flight instead of prompting twice.
• setAudioOutput(deviceId): calls setSinkId on the element carrying the audio track (audioEl if set, else videoEl). Works before connect(): the id is stored and applied to whichever element is bound next. Returns false (never throws) if the platform lacks setSinkId, nothing is bound yet, or the browser rejects the id. Pass '' to return to the system default. It works in Chromium, Firefox and WebKit. The Chromium-only 'default' id is rejected by Firefox.
• preferredVideoCodec (constructor option): filters the avatar video transceiver to a single codec via setCodecPreferences. Leave this unset. The avatar video is H264. Any other value ('VP8', 'VP9', 'AV1') still connects and audio keeps working, but no video frame is ever decoded and no error is thrown.
• maxAsrBitrateKbps (constructor option) / setAsrBandwidth(kbps) (mid-session): caps the ASR mic uplink's bitrate via RTCRtpSender.setParameters(), no renegotiation.

Noise suppression (Tier-1 default + Tier-2 BYO-DSP)

Two independent layers:

// Tier 1 (always on by default): no code needed. To customize or opt out:
const session = new KalturaAvatarSession({ token, /* … */,
  micConstraints: { noiseSuppression: false },   // merge over the default, or...
  // micConstraints: false,                      // ...opt out entirely (bare audio:true)
});

// Tier 2 (opt-in), BYO-DSP: any lib or bespoke processor shaped (stream) => Promise<MediaStream|{stream,stop}>
import { createNoiseSuppressor } from '@kaltura/intelligent-agents/experience/noise-suppressor';

const session = new KalturaAvatarSession({ token, /* … */,
  noiseProcessor: createNoiseSuppressor({ thresholdDb: -50 }),   // the SDK's own lightweight AudioWorklet gate
  micConstraints: false,   // recommended when the DSP expects raw, unprocessed audio. Stacking Tier-1 browser
                            // suppression under a second denoiser double-processes the signal
});

• micConstraints (constructor option): MediaTrackConstraints merged into every getUserMedia({audio}) call this session makes (connect(), switchMic()). Default {echoCancellation:true, noiseSuppression:true, autoGainControl:true}, the standard browser-native Tier-1 baseline. Pass false to send bare audio:true. Pass a partial object to override individual fields.
• noiseProcessor (constructor option): pluggable Tier-2 DSP hook, (stream) => Promise<MediaStream|{stream,stop}>. It is called with the raw getUserMedia stream at connect() and every switchMic(). The stream it returns reaches the ASR uplink. Return {stream,stop} if the processor owns a resource that needs teardown, such as an AudioWorkletNode graph. The SDK bundles NO DSP library. Bring a third-party processor (import it dynamically so apps that don't use it never load it) or a bespoke one. Anything matching the shape works. A processor that throws fails mic acquisition closed (the raw stream is stopped) with the code noise_processor_failed: a warning during the background acquire in connect(), a thrown KalturaError from startMic()/switchMic().
• createNoiseSuppressor(opts) (./experience/noise-suppressor, importable on its own, no effect until you pass it as noiseProcessor): the SDK's own lightweight, dependency-free Tier-2 implementation. It is an adaptive RMS noise gate running as a browser-native AudioWorkletProcessor (attack/release-smoothed envelope, adaptive noise-floor tracking). It is NOT spectral or ML denoising. Options: thresholdDb (default -50), attackMs (default 5), releaseMs (default 150), floorAdaptMs (default 2000). audioContext, getAudioContext and audioWorkletNodeConstructor are injectable.

KAVA analytics (opt-in, client-only Application Events)

import { KavaAnalytics } from '@kaltura/intelligent-agents/experience/analytics';

const analytics = new KavaAnalytics({
  partnerId: AGENTIC_PARTNER_ID, sessionId: session.threadId,
  hostingKalturaApplication: 28,   // see HOSTING_APPLICATIONS. Avatar Videos in this example
});
analytics.pageLoad({ pageType: 'View', pageName: 'earnings-deck' });
btnFeedbackDismiss.onclick = () => analytics.buttonClicked({ buttonType: 'Open', buttonName: 'feedback-dismiss' });

KavaAnalytics (./experience/analytics, its own subpath so apps that don't report analytics never load it) reports KAVA (Kaltura Video Analytics) events to https://analytics.kaltura.com/api_v3/index.php (service=analytics&action=trackEvent). It sends two Application Events, pageLoad and buttonClicked, for interactions the server cannot see: a page or view landing, a UI-only click, a contact-form submit or skip, a widget dismiss. It is a best-effort write and NOT idempotent (each call records a new row; there is no dedup). It is fire-and-forget, so you do not need to await it.

It sends nothing else. Conversation events are tracked by the platform, not by this module.

Transport: prefers navigator.sendBeacon (survives page-unload); falls back to an injectable fetch with keepalive:true when unavailable or when the beacon queue is full. Never reads a response body. enabled: false makes every call a no-op with no network use. Use it for offline and mock test runs.

• pageLoad(fields): {pageType, pageName, pageValue, pageInfo}. pageType is validated against the closed enum PAGE_TYPES (View/Create/Edit/Participate/List/Analytics/Admin/Error/Login/Registration/Custom); an invalid value throws before any network call.
• buttonClicked(fields): {buttonType, buttonName, buttonValue, buttonInfo}. buttonType is free text (the spec leaves it open-ended, e.g. Create/Filter/Navigate/Open).
• Common params set once at construction and attached to every event: partnerId, ks, entryId, sessionId, referrer, userId, hostingKalturaApplication/hostingKalturaApplicationVer, customId1/customId2.
• buildPageLoadParams/buildButtonClickedParams are the pure param-builders behind the class. Use them directly if you want your own transport.

Reporting a GenUI widget interaction specifically (which chip/link/answer the learner picked)? See GenUI · Widget Analytics § Widget-interaction analytics for the recipe, plus the list of signals the platform already tracks, so you do not duplicate one client-side.

Connectivity beacon (opt-in)

const session = new KalturaAvatarSession({ token, /* … */, statsIntervalMs: 5000 });
session.on('connectionQuality', ({ channel, rttMs, packetLossPct, jitterMs, bitrateKbps }) => {
  metrics.gauge(`avatar.${channel}.rtt_ms`, rttMs);
});

• statsIntervalMs (constructor option): polls RTCPeerConnection.getStats() on both the ASR uplink and the avatar downlink at this interval and emits connectionQuality. It carries the raw numbers only, with no scoring or telemetry wiring, so you can feed them into any metrics pipeline. Unset (the default) disables the beacon, so a session that never opts in pays zero getStats() cost.
• connectionQuality ({channel: 'asr'|'stv', rttMs, packetLossPct, jitterMs, bitrateKbps}): rttMs comes from the active candidate-pair's currentRoundTripTime; packetLossPct/jitterMs come from the RTP stream stats (outbound-rtp for ASR, inbound-rtp for STV); bitrateKbps is a byte-count delta against the previous poll, so it is null on the first tick for each channel. Any field the browser did not report is null.

Ending a conversation cleanly (session_completed signal)

Both session classes tell the server the moment a conversation is genuinely over. That includes tab-close, backgrounding and bfcache, not just your own disconnect(). Lifecycle rules (summaries, insights, CRM pushes) then fire in seconds. On by default (sessionCompleteOnEnd); nothing to wire up.

session.on('sessionCompleted', ({ reason, sent, suppressed, peers }) => {
  // best-effort: on a real tab-close there may be no listener left alive to run this
});

// App-owned teardown (server-side logout hook, a cross-device "end session" button
// a same-device BroadcastChannel can't see): send the signal without tearing anything down:
await session.completeThread();

disconnect(opts) takes {final?: boolean, reason?: string}. final: false skips the signal for an SDK-initiated teardown that isn't a real end (e.g. KalturaAgentSession.switchMode() tearing down the old transport while keeping the same thread). A zero-arg call means "final". stop() is an alias for disconnect({reason:'stop'}). Both are safe at any point: called while connect() is still in flight, they cancel every pending wait (the media handshake, a reconnect's capacity wait) so connect() rejects with connect_failed at once, and no error event follows.

A visibilitychange to hidden fires when the whole page leaves the screen (tab switch, minimize, app switch, lock, close). It does not fire on scroll, resize, partial occlusion, devtools, or fullscreen video. It answers "did the user leave the page". For "is the avatar widget visible on screen", use IntersectionObserver.

Accessibility (WCAG 2.2 AA / captions) + AI-disclosure gate

For EN 301 549 §6.2.1.2 (real-time text alongside voice), offer a text input next to either voice mode: an avatar session accepts typed turns via speak(text), and KalturaChatSession / KalturaAgentSession give viewers a full text-only path: same agent, same thread, no mic permission at all. See Voice Input Modes for the input-mode decision table.

Live captions satisfy WCAG 1.2.4 (Live Captions). Render them from CaptionService. It handles segmentation, timing and barge-in invalidation:

import { CaptionService } from '@kaltura/intelligent-agents/experience';

const captions = new CaptionService(session, {
  replacements: { 'Kalturah': 'Kaltura' },  // optional term corrections
});
captions.onCaption(({ text, clear }) => {
  captionEl.textContent = clear ? '' : text;
});

For the EU AI Act Art. 50 interaction-disclosure obligation, construct the session with requireDisclosureAck:true. This blocks speak() with a typed disclosure_required error until your app calls acknowledgeDisclosure(), so a deployer has a provable gate rather than a policy document:

const session = new KalturaAvatarSession({ token, /* … */, requireDisclosureAck: true });
session.on('disclosure', (notice) => showDisclosureBanner(notice));
await session.connect();
// session.speak(...) throws `disclosure_required` here until:
session.acknowledgeDisclosure();

A configured kickoff respects the gate: it is sent once, right after acknowledgeDisclosure(), never before it.

The gate holds across recovery. If the session resumes from a pause or reconnects while the ack is still outstanding, the conversation stays parked until acknowledgeDisclosure() lands. Once acknowledged, it stays acknowledged for the life of the session object, so a later reconnect never re-asks the user.

Security posture

Token types, token lifetimes, the audit stream, transport rules and browser hygiene are in Security & Compliance, together with the full control matrix. For Kaltura's legal and compliance positions, see Kaltura's AI Principles and the subprocessors list.

Key design rules

• Keep disableentitlement (management) server-side. KalturaAvatarSession expects a user-session (conversation or agent, with a per-user userId) or widget token (entitlement ON). See Security & Compliance for the full guidance and the rare case where an app deliberately needs to hand a browser broader access.
• Destructive ops require { confirmPermanent: true }. Never a flag on a read operation.
• Change one capability with intellects.setCapability(configId, name, state, ks). It reads, merges and writes, so the other capabilities stay as they are.
• Set kaltura_genie_experiences to 'off' at creation for tool-driven intellects. Left on, it competes with your client tools. tools.clientToolReadiness(body) lints for this. Why: External API Integrations.
• The live socket does not produce flashcards or summarization widgets. Use the HTTP converse path for those. See Two delivery paths.
• Group turn events by speechId, never timestamp. A new utterance invalidates the prior one's in-flight captions (barge-in guard).

Client-side commands

The cleanest way for the agent to drive your UI: no custom JSON, no fragile text parsing, no server-side echo call. tools.client() builds a native type:"client" tool. The model calls it, and your client receives a silent type:"tool" segment carrying tool_metadata.id. See Client-Side Commands for the full mechanism and the deployment gotchas (tool spirals starving the voice, the model having no real-time clock, guardrail and arg-scrubbing defaults). The rest of this section is the SDK-side API for consuming a client tool once it is authored.

// author once (server, admin KS)
import { tools } from '@kaltura/intelligent-agents/management';

const navigate = tools.client({
  name: 'navigate_to_slide',
  description: 'Go to a slide. Call when the user asks about a deck topic.',
  args: { slide_num: { prompt: 'Slide number (1-N).', type: 'int', required: true } },
  waitForResponse: false,   // fire-and-forget. Omitting this blocks the turn
});

// tools are a SEPARATE, partner-level entity: create it, then reference the id.
const { id } = await mgmt.tools.add(navigate, ks);

const { configId } = await mgmt.intellects.create({
  capabilities: { kaltura_genie_experiences: 'off' },  // required, see design rules
  tool_ids: [id],
}, ks);

// consume at runtime (browser)
session.onToolCall('navigate_to_slide', ({ slide_num }) => deck.goTo(slide_num));

// or headless
const { toolCalls } = await mgmt.converseOnce(configId, 'tell me about pricing');

waitForResponse controls whether the model's turn blocks on a real client-supplied result. Omitting it is not the same as false, so pass it explicitly. Fire-and-forget tools (waitForResponse:false) have no response channel back to the model at all, so put any "call once, then narrate" guidance in the tool's description.

Native client tools with a real wire ACK

For a tool that must block the model's turn on a real client-supplied result (rather than fire-and-forget), set waitForResponse: true:

const pick = tools.client({
  name: 'ask_user_to_pick_a_slide',
  description: 'Ask the on-screen viewer to pick a slide, and wait for their answer.',
  waitForResponse: true,   // block the model's turn on a real client result
  timeout: 15,              // seconds to wait for the ACK (default 30)
});
const { id } = await mgmt.tools.add(pick, adminKs);

// live-socket host side
session.onToolCall('ask_user_to_pick_a_slide', async (call) => {
  const slide = await askViewer();
  await session.respondToTool(call.toolMetadata.id, { slide });
});

waitForResponse:false never populates call.toolMetadata with an id to ACK against. Only register respondToTool/onToolCall ACK logic for tools you built with waitForResponse:true. The respondToTool return values are in the events catalog. This ACK is a live-socket operation only. There is no headless (Management) equivalent.

Handler results (local only, unless waitForResponse:true)

A handler's return value (or thrown/rejected error) is captured and re-emitted as 'toolCallResult': {call, ok:true, value} for a non-undefined return or resolve, {call, ok:false, error} for a throw or reject. A handler returning undefined (the common case) emits nothing.

session.onToolCall('create_slide', (args) => ({ slideNumber: deck.append(args) }));
session.on('toolCallResult', ({ call, ok, value, error }) => console.log(call.name, ok, value ?? error));

For a waitForResponse:false tool this is local and app-observable only. It never reaches the model. To carry your handler's result back to the model, build the tool with waitForResponse:true and call session.respondToTool(call.toolMetadata.id, ...) (see above). That is the only channel back to the model.

Arg validation before dispatch

onToolCall takes an optional third argument, a per-key {type, required, enum} schema, checked against call.args before the handler runs:

session.onToolCall('navigate_to_slide', ({ slide_num }) => deck.goTo(slide_num), {
  slide_num: { type: 'int', required: true },
});
session.on('toolCallInvalid', ({ call, errors }) => console.warn('rejected', call.name, errors));

Validated fields, top-level keys only: type (one of str/int/float/bool/list/dict, the same ARG_TYPES as tools.client's args, so you can pass the object you already declared there), required, and enum (a closed set of legal values). On a mismatch the handler is not invoked. The SDK emits 'toolCallInvalid' {call, errors} instead of 'toolCall', so a malformed call (for example a bare string where an int was declared) never runs on bad data. With no schema registered there is no check. collectConverse has the same guard for headless use: pass opts.toolArgSchemas: {toolName: schema} and a mismatched call is diverted to the result's toolCallsInvalid array instead of toolCalls.

Fused multi-tool turns (handled automatically on the live session)

When a turn calls 2+ tools, the server can stream them as one type:"tool" segment that names only the last tool, with the earlier tools' JSON args concatenated into the same string (see Wire Protocol · Events Catalog §4e). parseToolCall(segment) returns the named tool's own args and exposes the earlier, unnamed blobs as call.fusedArgs (array, arrival order, absent when the segment was not fused).

• KalturaAvatarSession: nothing to do. It pairs each fusedArgs blob with the tool_response segment that echoes the real tool name (parseToolResponseName(segment)) and dispatches it through the normal onToolCall path, with the same dedup, schema validation and toolCallResult/toolCallInvalid events. The pairing is scoped to one turn.
• Headless collectConverse(): you get the named tool's args, but no pairing. An earlier fused blob is reachable only through fusedArgs on that one ToolCall, not as its own toolCalls entry. To recover it, pair each fusedArgs blob with the matching tool_response name yourself using parseToolResponseName.

AI-SDR / CRM lead capture

./management ships validated api tool builders for CRM capture: salesforceLeadUpsert, salesforceContactUpsert and hubspotContactUpsert. They are pure config builders, so an AI-SDR or concierge agent needs no hand-written HTTP tool. Defaults, result handling, token options and a customer-endpoint recipe are in External API Integrations.

GenUI

import { ExperienceRenderer } from '@kaltura/intelligent-agents/experience/genui';

// 2-line happy path: renders all first-class GenUI widgets
new ExperienceRenderer({
  session,
  mount: document.getElementById('widgets'),
  onAction: (action, payload) => { /* followup / submit / play / open / answer */ },
}).start();

mountWidget(descriptor, target, opts) is the zero-dep, never-innerHTML, accessible renderer. It ships zero styling. You theme the stable kgenui/kgenui__* class contract. onMount(root, descriptor) is the seam for host-injected libraries (Mermaid, Chart.js, KaTeX).

A widget interrupted mid-stream (for example by a barge-in) never mounts truncated. It falls back to a typed {kind:'error', ...} descriptor.

Two topics have their own write-ups:

• Markdown rendering. A summary widget's text renders as flat escaped text by default. Pass { markdown: true } to mountWidget for sanitized markdown DOM. See GenUI · Authoring and Consuming Widgets § Markdown rendering (opt-in).
• Turn boundaries. In live mode, ExperienceRenderer clears stale widgets on turnStart by default (clearOnTurnStart: true), so a widget from a previous turn never lingers. See GenUI · Authoring and Consuming Widgets § Live vs. headless dispatch.

graded-question (multiple-choice or free-text, with an optional answer key and explanation) is a host-registered widget: import { renderGradedQuestion } from '@kaltura/intelligent-agents/experience/genui', then new ExperienceRenderer({ renderers: { 'graded-question': renderGradedQuestion } }). Grading happens client-side. It is a comprehension check, not a tamper-proof assessment. Full shape, the onAction('answer', ...) event, and a runnable example are in GenUI · Per-Runtime Widget Detail and examples/genui-graded-question.mjs.

Presenter

The Presenter helper (./experience/presenter, its own subpath so apps that don't need it never load it) manages a deck walkthrough end to end. It handles per-slide context injection (the current slide goes to the agent as a structured page_context request variable, so it knows what is on screen), navigation through ONE deterministic, silent, idempotent mechanism (onToolCall('navigate_to_slide'), no speech-parsing fallback), duplicate-nav suppression, a sequential resume point (reason:'resume'), and session memory ("welcome back"). It runs over an injected session and storage.

Getters (read-only):

Methods:

App hooks (each extends one seam without forking the class):

The constructor option oneNavPerTurn: true guards against the agent firing a second navigate_to_slide call within the same spoken turn (same or different target). The second call is suppressed until the next turn.

The constructor option deckOutline: true adds a full-deck {slide_num, title}[] outline to every context payload as a top-level outline key. Use it instead of hand-rolling a topic-to-slide mapping into BASE_DIRECTIVE, which goes stale after a runtime appendSlide() because BASE_DIRECTIVE is static. Duplicate titles are disambiguated automatically with the colliding slide's first talking point, or its slide number if it has none. Default false: no outline key unless requested.

See examples/deck-presenter.html for a self-contained runnable demo: construct Presenter right after the session, before connect(), with requireDisclosureAck: true and the extendContext/extraMemory/restoreMemory hooks in action.

Site navigation (SiteNavigator)

./experience/site-nav lets the agent bring the visitor to the right page and section of any website while it answers. It is built on one fire-and-forget client tool, go_to(path, section?), so there is nothing to ACK, nothing to time out on, and no way for the tool to spiral. The full contract, manifest format, adapters and repurposing steps are in Site Navigation with go_to.

Provision (Node, ./management):

import { goToTool, siteMapPrompt, SITE_NAV_RULES_PROMPT, PAGE_CONTEXT_PROMPT, loadSectionsManifest } from '@kaltura/intelligent-agents/management';

const manifest = await loadSectionsManifest('https://docs.example.com/nova/sections.json');
const tool = await mgmt.tools.add(goToTool({ siteLabel: 'the Example docs' }), ks);
await mgmt.intellects.create({
  tool_ids: [tool.id],
  allow_client_variables: true,
  prompts: [identity, siteMapPrompt(manifest), SITE_NAV_RULES_PROMPT, PAGE_CONTEXT_PROMPT],
  // ...
}, ks);

Browser (./experience/site-nav), right after the session:

import { SiteNavigator } from '@kaltura/intelligent-agents/experience/site-nav';

const nav = new SiteNavigator({
  session,
  manifestUrl: '/nova/sections.json',
  navigate: (url) => router.push(url),   // or location.assign(url)
  point: (el) => el.classList.add('is-pointed'),
});
// ...
nav.destroy();

The site build publishes sections.json with buildSectionsManifest(pages): one entry per page, each heading reduced to a 2–3 word key (edge-case-dont, quick-start). The same file renders as the SITE MAP prompt the model reads and drives resolution in the browser, so the two can never disagree. Keys are derived from the heading text in any language, collisions are resolved deterministically, and a docs change re-keys only the page it touched.

Chroma-key Avatar Compositor

attachChromaKeyAvatar() (./experience/chroma-key, its own subpath so apps that don't composite the avatar never load it) wires a bring-your-own transparent-background compositor (any chroma-key-video-shaped class) directly onto a KalturaAvatarSession's own avatar <video> element, and keeps that compositor's lifecycle in lockstep with the session's. The SDK never bundles, imports, or depends on chroma-key-video (or any keying/matting library) itself. This is glue: the same constructor-injection pattern ./experience/noise-suppressor uses for audioWorkletNodeConstructor:

import { KalturaAvatarSession } from '@kaltura/intelligent-agents/experience';
import { attachChromaKeyAvatar } from '@kaltura/intelligent-agents/experience/chroma-key';
// YOUR dependency, not the SDK's. There is no npm package for chroma-key-video; load it by
// bundling https://github.com/kaltura/chroma-key-video locally, or straight from jsDelivr's
// GitHub-CDN mode, pinned to a released tag:
import { ChromaKeyVideo } from 'https://cdn.jsdelivr.net/gh/kaltura/chroma-key-video@v1.2.0/src/chromakey.js';

// The keyed source <video> is muted and off-screen, so give the voice its own element:
const session = new KalturaAvatarSession({ token, …appInit, videoEl, audioEl, socketFactory });
const player = attachChromaKeyAvatar({
  session,
  videoEl: session.videoEl,     // must be the SAME element the session itself renders into
  ChromaKeyVideo,
  options: { autoTune: true },
  container: document.getElementById('composited'),   // omit to skip .mount() entirely
});
await session.connect();
// No extra teardown needed: player.destroy() fires automatically on session 'ended', a
// fatal session 'error', or session.disconnect()/stop() (the normal "hang up" path).
// Calling session.disconnect() alone is enough.

Behavior:

• Construction is synchronous. attachChromaKeyAvatar() returns the live ChromaKeyVideo instance immediately, no Promise.
• videoEl must be session.videoEl, the session's read-only getter for the element its WHEP downlink assigns srcObject to. A different reference throws a KalturaError, so a stale or duplicated element cannot silently key the wrong stream.
• Returned unwrapped. The returned player is the exact instance ChromaKeyVideo constructed, with no proxy. It is a standard EventTarget. Listen on player with addEventListener for its own events (for example chroma-key-video's 'started'/'backend'/'error'). attachChromaKeyAvatar() never re-emits them onto session.
• Auto-cleanup. player.destroy() is called exactly once: on the session's 'ended' event, on any fatal 'error' (capacity_unavailable/tier_exceeded/bad_request/peer_removed/unsupported_client), or when the session reaches its 'disconnected' state. The last one is what session.disconnect()/session.stop() (for example a "leave call" button) triggers, and that path never emits 'ended'. A recoverable error (for example a socket hiccup the session reconnects from) does NOT destroy the player. The plugin checks the player's isDestroyed flag first, so a player.destroy() you called yourself never gets a second call. The three teardown paths are safe to fire in any order.
• Idempotent. A second attachChromaKeyAvatar() call against a session that already has a live compositor logs console.warn and returns the EXISTING instance instead of constructing (and WebGL-context-leaking) a second one. Never throws for this. If you destroyed the player yourself with player.destroy(), the next attachChromaKeyAvatar() call for that session builds a new player without a warning.
• No reconnect step. A WHEP reconnect swaps the new tracks into the same stream already bound to videoEl; srcObject is never reassigned, so the compositor keeps reading frames from the element it was constructed against and no re-attachChromaKeyAvatar() call is needed.

Non-goals: this plugin does not reimplement chroma-keying, matting, renderer fallback, or WebGL context-loss recovery. That is chroma-key-video's (or your chosen library's) job. If your app keys a URL-sourced clip with chroma-key-video directly, you must run that URL through safeUrl() first. This plugin never accepts or fetches a URL, only the session's own live video element.

See examples/chroma-key-avatar.html for a self-contained runnable demo.

Advanced / building-block exports

These are importable from their entry points and useful when composing custom pipelines or renderers outside the high-level helpers.

./management

./experience

./experience/presenter

./experience/site-nav

./experience/chroma-key

./experience/genui

./experience/noise-suppressor

./experience/analytics

Test your app offline

The package ships its test fakes. Import them to test your own app without a network or a browser:

import { FakeSocket } from '@kaltura/intelligent-agents/test/fakes/socket.js';
const session = new KalturaAvatarSession({ /* your config */ socketFactory: () => new FakeSocket() });

For a test against the real backend, build the session without socketFactory overrides, call connect(), wait for the ready state and assert on the resulting DOM.

Intellect configuration

The intellectConfig facade wraps the read-merge-write cycle:

// one call: reads current config, overlays your patch, writes
await mgmt.intellectConfig.patch(configId, { base_directive: 'Be concise.' }, ks);

// typed setters
await mgmt.intellects.setCapability(configId, 'use_knowledge_base', 'on', ks);
const { id: toolId } = await mgmt.tools.add(myTool, ks);       // tools are a separate, partner-level entity
await mgmt.intellectConfig.setToolIds(configId, [toolId], ks); // then link it
await mgmt.intellects.secrets.set(configId, { API_KEY: value }, ks);  // write-only
await mgmt.intellectConfig.setKnowledgeIds(configId, [knowledgeId], ks);
await mgmt.intellectConfig.setMcpServers(configId, { docs: { url: 'https://mcp.example.com/mcp' } }, ks);
await mgmt.intellectConfig.setModelConfiguration(configId, { model_id: 'gemini-3.5-flash', temperature: 0.3 }, ks);
await mgmt.intellectConfig.setOpeningPhrase(configId, '{% if user_name %}Hi {{ user_name }}, what can I help with?{% else %}Hi, what can I help with?{% endif %}', ks);
await mgmt.intellectConfig.setThreadStartTools(configId, [toolId], ks);
await mgmt.intellectConfig.setAvatarSummaryConfig(configId, {
  analysis: { summary: 'Two-sentence recap', next_step: 'The one action the user agreed to' },
  template: '{{ summary }}\n\nNext: {{ next_step }}',
  content_type: 'text',
}, ks);

setMcpServers writes the intellect's mcp_servers map ({"<name>": {url}}, pass {} to clear). A later get returns each entry expanded, with type:'mcp', a default transport of 'streamable_http' and null header and allow-list fields. Never diff your input against a get byte-for-byte. Full entry shape (transport, headers, allowedTools, per-attendee credentials, OAuth-gated servers): MCP Integrations.

A worked example with one MCP server, a per-attendee credential set at join time, then a live turn:

await mgmt.intellects.secrets.set(configId, { CRM_MCP_TOKEN: process.env.CRM_MCP_TOKEN }, ks);
await mgmt.intellectConfig.setMcpServers(configId, {
  crm: { url: 'https://mcp.example.com/mcp', headers: { Authorization: 'Bearer {{secrets.CRM_MCP_TOKEN}}' } },
}, ks);

const token = await mgmt.sessions.createAgentToken({ agentId, userId: attendeeId });  // server side, one per attendee
const session = new KalturaChatSession({ token });                                      // token (or token.ks) reaches the browser
await session.connect();
session.updateRequestVars({ account_id: 'acct_42' });  // this attendee's own scoping value
await session.sendText('Look up open tickets for this account.');

{{secrets.CRM_MCP_TOKEN}} is the shared credential for every attendee. account_id rides in as a per-attendee request var the server-side prompt/tool can reference. Give each attendee their own credential instead of a shared one by templating the header itself with a bare {{VAR}} and setting it per session via updateRequestVars. See MCP Integrations § A different credential per attendee. Per-attendee credentials need a per-attendee session. A widget-path session is one shared identity for every visitor, so give each attendee a token from createAgentToken({ agentId, userId }) (see Session tokens).

setModelConfiguration picks the chat model and its sampling limits. model_id must be one of MODEL_IDS, thinking_level one of THINKING_LEVELS ('low'/'high', Gemini only), max_output_tokens a positive integer, temperature 0..1. Pass null to return to the defaults. Which models answer depends on your partner's region, so run converseOnce once after switching.

setOpeningPhrase sets the intellect's opening_phrase, the line spoken as the first turn of every avatar join. It is a Jinja2 template. Pass null to clear it, or SILENT_OPENING for a silent opening so the browser can send the first turn with kickoff. How to choose, personalize the phrase and use kickoff: Start the Conversation § Personalize the opening.

setThreadStartTools lists tool ids that run once at the start of every thread, before the first turn. Only api and code tools are supported. Pass [] to clear.

setAvatarSummaryConfig shapes the summary written when an avatar session ends. analysis maps output keys to what the model should extract, template renders them (every {{ key }} must be an analysis key), content_type is one of SUMMARY_CONTENT_TYPES. The result is stored on the thread as a summary message. Pass null to return to the default one-paragraph summary.

intellectConfig.describe(configId, ks) returns every writable field under editable (null when the server did not echo it) plus capabilityNames. Wire it directly to a settings UI.

patch() only accepts keys in EDITABLE_FIELDS and rejects anything else before the network call. describe()'s editable map is exactly that set. For event-driven session and thread rules, see Lifecycle Rules.

intellectConfig.audit(configId, ks) checks a saved intellect for mistakes, lifecycle.audit(ks) does the same for lifecycle rules, and mgmt.doctor(ks) runs both across every agent. All three only read and return findings with a severity, code and fix. See Audit your rules.

Forcing the reply language (setForcedLanguage)

mgmt.setForcedLanguage() pins the language an agent speaks and listens in. Usage, options and how to clear it: Languages.

Skills and voice import

Skills (mgmt.skills) are standalone, partner-level reusable instruction entities: {id (uuid), name, description, instructions}. Full lifecycle including update:

const skill = await mgmt.skills.add({ name: 'greeter', description: 'Greets warmly.', instructions: 'Always say hi.' }, ks);
const page = await mgmt.skills.list(ks);          // async-iterable + awaitable first page
const one = await mgmt.skills.get(skill.id, ks);
await mgmt.skills.update(skill.id, { instructions: 'Always say hi, in one short sentence.' }, ks);  // idempotent; a name conflict returns 409
await mgmt.skills.delete(skill.id, ks, { confirmPermanent: true });

A name conflict returns 409. The same applies to Tools.

Before deleting, mgmt.skills.delete lists every intellect and refuses with a typed skill_in_use error naming each one still referencing the id in skill_ids, unless called with {confirmPermanent:true, force:true}. mgmt.tools.delete has the same tool_in_use guard.

Attach a Skill to an intellect via intellectConfig.setSkillIds. The intellect only holds a reference list ({id, mode, condition?} entries), the skill body itself lives in mgmt.skills. mode is one of the exported SKILL_MODES: 'preloaded' (instructions go in the system prompt every turn), 'adhoc' (the agent loads it on demand and does not remember it), or 'adhoc-save' (adhoc, plus auto-load on later turns once used). condition is an optional Jinja2 expression over thread variables that must be truthy for the skill to apply:

await mgmt.intellectConfig.setSkillIds(configId, [
  { id: skill.id, mode: 'adhoc' },
  { id: avatarOnlySkill.id, mode: 'preloaded', condition: '{{ sys__avatar_enabled }}' },
], ks);
// pass [] to detach every skill

A skill id owned by another partner is rejected with a 403 forbidden.

Provider voice import (mgmt.catalog) creates a catalog Voice item directly from an ElevenLabs or Cartesia voice id, with no audio upload:

const v = await mgmt.catalog.importVoiceFromElevenLabs('EXAVITQu4vr4xnSDxMaL', ks); // ElevenLabs' own public sample voice id, safe to use as-is
// or: await mgmt.catalog.importVoiceFromCartesia('<cartesia-voice-id>', ks);

An unknown provider id creates nothing and raises a typed voice_not_found_elevenlabs / voice_not_found_cartesia error.

Custom face works self-serve (mgmt.catalog, mgmt.avatars): upload a portrait image via catalog.createVisual, pass itemId as visualId in provision/avatars.create. The model animates the portrait at runtime. Video-clip ingest is not available through this API. Composing a face and background, and curated templates: Avatar and agent.

Embed snippet or SDK? There are two ways to put an agent on a page. Pick by how much you want to build.

Scripted-Video (STV-only) Sessions

Pre-rendered speech with no LLM, ASR or socket: Scripted avatar sessions.

RAG (knowledge base)

const rec = await mgmt.knowledge.addRecord({ name: 'Product Docs' }, ks);
const { configId } = await mgmt.intellects.create({
  knowledge_ids: [rec.id],
  capabilities: { use_knowledge_base: 'on' },
}, ks);
// isIndexed() reports the record's own lifecycle status (ready:true immediately,
// before any entry has indexed). It is NOT an indexing-completion check, see below.
const status = await mgmt.knowledge.isIndexed(rec.id, ks);

Content modalities indexed: captions, OCR, document attachments. Don't use knowledge.isIndexed()'s ready flag, knowledge.search()'s "couldn't find relevant information" reply, or knowledge.corpusStatus()'s populated flag, as an indexing-status signal. See Ground the Agent for why. Use knowledge.entryStatus() instead, the per-entry completion check.

Knowledge records have full lifecycle CRUD:

const got = await mgmt.knowledge.getRecord(rec.id, ks);            // read one
await mgmt.knowledge.updateRecord(rec.id, { name: 'Docs v2' }, ks); // rename/edit
await mgmt.knowledge.deleteRecord(rec.id, ks, { confirmPermanent: true });

deleteRecord lists every intellect and refuses with a typed knowledge_in_use error naming each one still carrying the id in knowledge_ids, unless called with {confirmPermanent:true, force:true}. tools.delete and skills.delete have the same guard. A deleted or unknown record id gives a typed not_found. Another partner's record gives forbidden.

Threads, messages, feedback and follow-ups

mgmt.threads, mgmt.messages, mgmt.feedback and mgmt.followups list, read, rate and clean up conversations. Methods, filters and PII handling are in Management Operations and Conversation & Analytics.

Related docs