Dynamic Data Injection — keeping the brain in sync with your app

Design guidance for app builders on getting your app's state into the conversation: who the viewer is, what's currently on screen, and what just happened in your UI. The goal is to keep the agent's context fresh and correct, instead of a stale snapshot from when the session first connected.

The SDK gives you three mechanisms, each solving a different problem. The most common integration mistake is reaching for the wrong one: using a passive mechanism when you needed the agent to react right now, or using the active one for something that should just be quiet background context. This doc explains why each mechanism exists, when to reach for it, and how they compose into one coherent update flow.

Three ways to get information to the agent

Request variables are passive: they update what the agent will see, whenever its next turn happens to occur. speak() is the only active one: it's the trigger that makes a turn happen right now.

Getting this distinction right is the key to reliable behavior. If you need the agent to react to something the instant it happens (a modal just closed, a background action just finished), a passive update alone will sit there unread until some other turn happens to occur. Sometimes that's much later. Sometimes it never happens in a short session.

The context channel: request variables

Request variables (request_vars) are the SDK's one channel for app-supplied context. Each variable is a scalar value (string, number, boolean, or null) the agent's prompt reads via {{var}} templating: a viewer's name, an account tier, or a whole JSON document (as a string) the prompt reasons over. Seed them at connect time, update them any time after:

const session = new KalturaAvatarSession({
  token, /* …other config… */
  requestVars: { user_name: 'Ada' },
});

// later, once you learn more about the viewer:
session.updateRequestVars({ account_tier: 'enterprise' });

Three properties make this channel do the heavy lifting:

• Updates merge. updateRequestVars(vars) merges what you pass into the session's canonical map. Send only the keys that changed; keys you omit keep their values. A headless converse call that sends only a delta behaves the same way.
• Values persist for the whole thread. Send a variable once and every later turn on that thread still sees it. You don't resend per turn. A new thread starts clean. On a warm reconnect the SDK re-sends the full map automatically. To turn a flag off, send it again as ''; omitting it keeps the old value.
• Values can be big. Each value must be a scalar (string, number, boolean, or null). A JSON document goes in as a string. Tens of kilobytes of JSON in one variable works. The SDK's live verification pushes a ~32 KB blob through and reads it back (see Runnable examples below).

updateRequestVars() and setDynamicPrompt() throw invalid_state until the session is connected. Attach your session.on(...) listeners before connect(), because some events fire before it resolves: README.md § Experience.

Page context: setDynamicPrompt()

For "what's on screen right now" (the current slide, page section, or task state) you don't hand-pick variables. setDynamicPrompt(data) serializes any JSON-safe object into one well-known request variable, page_context, and sends it through the same channel as updateRequestVars() (how the two relate):

session.setDynamicPrompt({
  current_section: 'pricing',
  section_summary: 'Enterprise tier: $499/mo, includes SSO and priority support.',
});

Each call replaces the previous page_context whole (it's one variable, with no deep merge), and it persists on the thread like any other variable. This is context, not speech. Calling it does not make the avatar say anything. It updates what the agent reads the next time it takes a turn.

The prompt must reference {{page_context}}, or the agent never sees it. The management entry point exports the canonical prompt block for this. Spread PAGE_CONTEXT_PROMPT into the prompts[] you provision instead of hand-rolling your own:

import { PAGE_CONTEXT_PROMPT } from '@kaltura/intelligent-agents/management';
await mgmt.intellects.setPrompts(configId, [PAGE_CONTEXT_PROMPT, ...otherBlocks]);

If you're using the SDK's Presenter plugin for a slide-deck-style walkthrough, it already manages per-slide page-context injection for you. See README.md → Presenter for the deck-specific API surface. Presenter.refreshContext() re-sends the current context outside of a navigation, which is exactly the building block the worked example below relies on.

The gate: allow_client_variables

The intellect must have allow_client_variables: true, or every request variable you send, including page_context, is rejected. It is on by default. Pin it once, server-side, with an admin KS:

await mgmt.intellects.setClientVariablesEnabled(configId, true, adminKs);

The rejection is silent on the live socket and in KalturaChatSession. The turn comes back as an empty reply: no HTTP error, no socket error. Both session classes (KalturaAvatarSession and KalturaChatSession) detect the pattern and emit a once-per-session warning event, { code: 'empty_turn_with_request_vars', message, requestVarKeys } (variable names only, never values), pointing at the gate. A management converse call that receives the 403 throws client_variables_disabled instead. KalturaChatSession.sendText() can fail either way when it sends request variables: the silent empty turn with the warning, or a rejected promise with client_variables_disabled.

session.on('warning', (w) => {
  if (w.code === 'empty_turn_with_request_vars') console.warn(w.message);
});

The mirror-image misconfiguration is also silent live: the gate is on, but no prompt block references any client {{variable}}, so your variables have no effect. Catch both before shipping with the lintPrompts pre-flight:

import { lintPrompts } from '@kaltura/intelligent-agents/management';
const lint = lintPrompts(prompts, { allowClientVariables: true, knownVars: ['page_context'] });
// findings to look for:
//   vars_gate_unreferenced: gate on, but NO block references any client {{variable}}
//   known_var_unreferenced: a var you send is never referenced by any block
//   client_variable_not_allowed: a block references a client var but the gate is off

Reserved sys__* variables

Reserved sys__* keys (like sys__user_id and sys__thread_id) are server-injected on every turn and rejected if you try to set them yourself, regardless of the gate. See Reserved Template Variables. The SDK's own pre-flight rejects them (and non-scalar values such as objects and arrays) client-side before anything hits the wire.

Server-side tools read them too

Request variables aren't limited to prompt text. A server-side api tool's request template can interpolate them ({{account_id}} in a URL, header, or body), so a value your app set turns into a parameter of a backend call the agent makes. This includes variables from earlier turns that were never mentioned in conversation. Arg placeholders in a tool follow the table in Tools and Secrets.

Security stance: request variables are client-suppliable and thread-persistent. Never treat one as an authorization claim; your endpoints must independently authorize every call. A poisoned value also outlives its turn: it keeps interpolating into prompts and tool calls for the rest of the thread. Don't pass unsanitized end-user text into setDynamicPrompt, and never put secrets in any request variable.

Runnable examples

Two scripts in this repo exercise every behavior above against the real API:

• examples/request-vars-live-context.mjs: a five-turn walkthrough of seed → persist → merge → large page_context → fresh-thread reset.
• npm run live-verify:request-vars: persistence, merge, tool interpolation, server-side sys__* injection, ~32 KB payload, thread isolation.

The active nudge: speak()

speak() is the one mechanism that actually provokes a new turn. It is handled like the viewer's own speech, so the agent reasons over it and responds, right then.

session.speak('Tell me about your pricing.');

You're not limited to putting the viewer's own words here. A common and effective pattern is to send a short, clearly-tagged app-generated message describing something that just happened in your UI, so the agent reacts to it immediately rather than waiting for whatever the next real user turn happens to be:

session.speak('[SECTION CHANGE] The viewer just opened the pricing section. Discuss THIS section only.');

A bracketed tag like [SECTION CHANGE] is not a wire-level feature. It's a convention. Write your system prompt to recognize a tag like this as an app-generated cue, as opposed to something the viewer said out loud. That gives you a clean, unambiguous signal to react to, without ever putting synthetic text in the viewer's own mouth. Design your own tag vocabulary to match whatever events your app needs the agent to react to instantly.

Pair it with a context update, in this order: call setDynamicPrompt() (or Presenter.refreshContext()) first, then speak() immediately after. That way the nudge that provokes the turn arrives after the context it needs to reason correctly about is already in place, not racing it.

When speak() actually sends

speak() is safe to call at any moment after connect() resolves. It decides the timing for you:

Several speak() calls during one held turn go out together as one turn, one text per line, in call order. The agent reads everything the user said while it was busy and answers once, like a person catching up.

// The first nudge: let the SDK send it. `kickoff` rides the same hold and goes out once,
// the moment the opening turn ends. Never re-sent on resume() or a reconnect.
const session = new KalturaAvatarSession({
  ...runtimeConfig,
  kickoff: '[SESSION START] The session just loaded. Begin now per your OPENING instructions.',
});
await session.connect();

// Later nudges: speak(). Resolves true once the text reached the server,
// false if the session ended while the text was still held.
const sent = await session.speak('[CONTEXT] The user just opened the pricing page.');

Guardrails (onBeforeSend, maxTurnsPerMinute, tap-to-talk, disclosure) run when you call speak(), not when the held text is sent, so a blocked call rejects immediately.

Answering an agent-initiated request

The mechanisms above all push data from your app to the agent. There's also a path in the other direction: your agent's configuration can require the agent to ask the viewer for specific structured fields at some point in the conversation (an email, a booking date, a support ticket's category). See Structured Data Forms for how to configure what it asks for. Once your UI collects the viewer's answer, hand it back with:

session.submitStructuredDataForm({ email: 'ada@example.com' });

This routes the value back into the conversation so the agent can act on it. Treat the moment your UI closes that form (submitted or explicitly declined) as a real, immediate event worth an active nudge too. See the worked example below.

How they work together: a worked example

Example, an interactive product walkthrough: viewer identity is known at connect, the UI advances through sections as the viewer explores, and partway through, the agent asks the viewer for an email address via a structured data form.

// 1. Connect. Personalization the prompt substitutes via {{user_name}}.
const session = new KalturaAvatarSession({ token, /* …other config… */ requestVars: { user_name: 'Ada' } });
await session.connect();

// 2. The viewer navigates to a new section. Refresh context, then actively nudge.
function onSectionChange(section) {
  session.setDynamicPrompt({ current_section: section.id, section_summary: section.summary });
  session.speak(`[SECTION CHANGE] Now viewing "${section.title}". Discuss THIS section only.`);
}

// 3. The agent asks for an email; your UI renders the form and the viewer submits it.
function onEmailFormClosed(email) {
  if (email) session.submitStructuredDataForm({ email });
  // The form closing either way is a real, immediate event. Update a variable, then
  // nudge, exactly like a section change, rather than waiting for whatever turn
  // happens next to pick it up.
  session.updateRequestVars({ email_form_status: email ? 'submitted' : 'declined' });
  session.speak(`[EMAIL FORM] The form just closed. The viewer ${email ? 'submitted an email' : 'declined'}.`);
}

The pattern that repeats: update context, then actively nudge, in that order, for anything that just happened and needs an immediate reaction.

Which one do I want?

Related but distinct: client-side commands

If instead you need the avatar to drive your UI (navigate, open a panel, highlight something), that's a different, silent channel (session.onToolCall()) and not a data-injection mechanism. See Client-Side Commands.

Related docs