Authoring & Consuming Widgets

← Back to GenUI Reference

Authoring: which capability turns each widget on

Set capabilities at intellect creation (the agent's configuration). Do not change them after creation, see Client-Side Commands. Source of truth: src/management/capabilities.js (CAPABILITY_INFO, CAPABILITY_DEFAULTS, OFF_BY_DEFAULT).

kind is tool, segment, mode, or prompt. This only names the mechanism that gates the capability on or off. It says nothing about whether the capability's content is persona-steerable.

avatar_filler (kind: 'prompt') is the exception to watch for. Its filler phrasing is NOT reliably steerable via base_directive, even though it streams as a "spoken" segment alongside avatar/text (see Wire Protocol · Events Catalog). Disable the capability if the default phrasing doesn't fit your persona.

The structured data-collection form (user-properties-form) is configured via the intellect's user_properties_forms field, not a boolean capability. On the wire this field is a LIST of {call_stage, properties:[{key,type}]} forms. setUserPropertiesForms accepts a single form object and wraps it into a one-element list.

Every capability and its default is listed in Configure an Intellect.

Consuming widgets in your app

The 2-line happy path: pass a DOM Element as mount and the SDK renders for you, live:

import { ExperienceRenderer } from '@kaltura/intelligent-agents/experience/genui';

// session: an active KalturaAvatarSession from your app
new ExperienceRenderer({
  session,
  mount: document.getElementById('widgets'),
  onAction: (action, payload) => console.log(action, payload),
}).start();

mountWidget (genui/renderers/mount.js, exported from ./experience/genui) is the SDK's last-mile descriptor→DOM renderer:

• Zero-dep and isomorphic. Returns null with no DOM.
• Never innerHTML.
• Accessible by construction.
• Ships zero styling. It emits the kgenui / kgenui__* class contract for you to theme.

Call it directly with mountWidget(descriptor, targetEl, { replace?, onAction? }), or let ExperienceRenderer call it when mount is an Element (or target).

ExperienceRenderer options

new ExperienceRenderer({ session?, mount, target?, onAction?, renderers?, replace?, onUnhandled?, urlPolicy?, partnerId?, uiConfId?, clearOnTurnStart?, maxRendered? }) (genui/renderer.js). partnerId alone enables video-gallery to build a player-embed iframe URL from each entryId. uiConfId (string or number) is optional and pins a specific player uiConf in that URL.

mount is a (descriptor)=>void function (full control) or a DOM Element (auto-rendered via mountWidget). maxRendered (default 100) caps the rendered history: the oldest descriptor is dropped when the cap is exceeded, so a long-lived session can't grow memory unboundedly.

Live vs. headless dispatch

In the live socket runtime, .start() subscribes to session.on('brainSegment') and flushes on turnEnd/avatarStopTalking, resetting on interrupted. clearOnTurnStart (default true) also resets the assembler and clear()s rendered/last on the session's turnStart event (re-emitted from the raw agent_start_speech socket event). So a widget from turn N never lingers into turn N+1. Set false for intentional cross-turn persistence.

Headless, call .render(runtimeName, widget) (or .render(segment)) per segment from a conversations.stream() feed. This is the reliable path.

Registration, fallback, and provenance

• .register(runtimeName, fn) adds/overrides a renderer (name normalized); .has, .runtimes, .last, .rendered, .clear(). replace:true keeps only the latest widget per turn.
• An unknown runtime renders {kind:'unknown', data:{runtime, model}} and fires onUnhandled (never throws). A throwing custom renderer degrades to {kind:'error', ...}.
• _meta receipt stamps {partnerId, source:'experience/genui', scope:'conversation (entitlement ON)', known, firstClass}. known means this instance has a renderer for it (a registered 10th runtime is known:true). firstClass means it's one of the built-in set.
• graded-question (GenUI · Per-Runtime Widget Detail § 10) is a concrete, shipped example of a registered 10th runtime. It's exported (renderGradedQuestion) but deliberately excluded from DEFAULT_RENDERERS/RUNTIMES, so it stays known:false until a host explicitly registers it.

onAction, WIDGET_KINDS, and the hand-rolled escape hatch

onAction(action, payload) surfaces interactions mountWidget can't fulfil itself:

• 'followup' {question} → session.speak
• 'submit' {values} → session.submitStructuredDataForm
• 'play' {entryId, url, embedUrl}
• 'open' {url}
• 'answer' {questionId, variant, correct, value, explanation, optionId?}: a graded-question was answered (see GenUI · Per-Runtime Widget Detail § 10)

WIDGET_KINDS (exported) is the frozen list of every kind (the nine first-class GenUI runtimes, plus unknown and error). Use it for an exhaustive host switch or a parity test.

You can still read seg.metadata.runtimeName and build DOM yourself (use renderSafeLink/safeText/safeUrl so nothing un-sanitized hits the DOM). Prefer mountWidget. The hand-rolled path is for a fully custom design system only.

Markdown rendering (opt-in)

mountWidget(descriptor, el, {markdown:true}) parses a summary widget's text (genui/renderers/markdown.js, renderMarkdown) as markdown instead of flat text. It supports:

• Headings (#-######)
• bold/bold and italic/italic
• Inline `code`
• Links: [text](https://example.com)
• Unordered and ordered lists
• Fenced code blocks (with a language-<token> class)
• GFM tables

A markdown table renders through the tableEl builder (genui/renderers/dom-helpers.js), one shared, safe <table> construction path with no duplicate table logic.

This is markdown-in-plain-text rendering, not a new wire segment type. The underlying descriptor is still {kind:'summary', data:{summary, ...}}. Only the render path changes.

It is never innerHTML. Every text run is built via textContent/createTextNode, so a raw <script> tag in the LLM output is inert text, not markup. Every extracted URL goes through safeUrl too. A javascript: or other unsafe-scheme link degrades to plain safe text, never a dead or unsafe href.

Without opts.markdown, the summary renders as flat escaped text.

Theming + a11y contract (kgenui classes)

mountWidget emits semantic, accessible DOM and a stable kgenui/kgenui__* class contract. Theme it in your CSS. The SDK ships none.

Built-in accessibility:

• Flashcards are <button aria-pressed> flip toggles.
• Followups are <button> chips in a labeled list.
• Links carry a visually-hidden "(opens in a new tab)" cue and rel=noopener noreferrer, and are dropped when safe:false.
• Images always have alt.
• Form fields are real <label for> and <input> elements, with type/inputmode from the field type and aria-required/aria-describedby from required/description.

The SDK ships zero CSS for these class names. Style the .kgenui__* block in your own stylesheet.

screen_share_analysis / sendScreenShot(data)

An avatar agent can look at the user's screen. Turn it on with the agent's screenShare setting (agent fields). Setting the screen_share_analysis capability on the intellect is not enough.

On a live session, call session.startScreenShare(). It opens the browser's screen picker and sends a still image every 2 seconds. session.stopScreenShare() ends it, and so does the user's own "stop sharing" control. See Screen share for options, events and error codes.

session.sendScreenShot(data) sends one still you captured yourself (data is an ArrayBuffer or Uint8Array). Use it only when you need your own capture path. Both calls throw capability_disabled unless the agent's screenShare setting is on.

Related docs