GenUI — Complete Capabilities Reference

Everything an agent can put on screen beyond spoken text: the unisphere-tool runtimes (flashcards, summaries, sources, forms, Kaltura/external video, images, links) the Genie brain backend emits as type:"unisphere-tool" stream segments, and the SDK renders natively via ExperienceRenderer.

This is the authoritative map — every runtime, its enabling capability, the exact wire shape, the SDK function/keys that parse and render it, the backend code flow it rides, and the restrictions that bite in practice.

All claims here are anchored to repo source (src/...) and WIRE-PROTOCOL.md; where a behavior is inferred rather than live-captured, it is marked INFERRED.

Naming note. unisphere-tool and unisphere.widget.genie below are the Genie brain's literal, on-the-wire constant values — carried over from a naming decision made outside this SDK, and preserved verbatim here because changing them would break real interoperability. They are unrelated to "GenUI," this doc's own name for the feature; don't read them as a reference to a different product.

The model in one paragraph

The brain emits a GenUI widget by writing a fenced block carrying a widgetName. The Genie brain backend's message_service converts that into a stream segment of type:"unisphere-tool" shaped { type, content, metadata:{ widgetName, runtimeName }, speechId?, threadId? }. All widgets share widgetName:"unisphere.widget.genie" — the host keys off metadata.runtimeName (stripping the -tool suffix) to pick a renderer. The SDK turns each segment into a framework-agnostic descriptor {kind, data} your app maps to DOM. Nothing here emits HTML; every string/URL is run through core/safety.js first.

The first-class runtimes

Backend tool key (defined in the Genie brain backend's experience-definitions module) → wire runtimeName → normalized dispatch key (the renderer registry key). Source: src/core/stream.js GENUI_RUNTIMES; src/experience/genui/parse.js RUNTIMES (derived from GENUI_RUNTIMES, so the two can never drift).

normalizeRuntime(name) (parse.js lines 54–58) strips a trailing -tool and trims; it tolerates an already-normalized name and a non-string (→ ''). isKnownRuntime(name) tests membership in this set. Any other runtime (e.g. the backend's gen-ui-composer-tool, gen-ui-components-tool, kaltura-video-player-tool — see Restrictions) is NOT in this set and falls through to a safe fallback.

How a widget reaches your screen (the data flow)

1. Author — at intellect creation, enable the gating capability (table below). The capability injects a Jinja block into the system prompt telling the model when to emit that fenced widget.
2. Emit — the brain writes a fenced block with widgetName:"unisphere.widget.genie" + runtimeName. message_service turns it into a unisphere-tool segment.
3. Stream — segments arrive as agent_raw_text deltas (live socket) or SSE/NDJSON lines (HTTP converse). A single widget can span multiple fragments.
4. Assemble — SegmentAssembler (genui/segments.js) buffers fragments and flushes a complete widget on a boundary change (different runtime or speechId, or turn end). If that boundary change interrupts a JSON body before it finishes writing, the fragment is flagged malformed (onMalformed) instead of flushed as a widget — ExperienceRenderer mounts the same typed {kind:'error', data:{runtime, message}} fallback it uses for a throwing custom renderer.
5. Parse — parseWidget(segment) (parse.js) → {widgetName, runtimeName, runtime, model}. parseContent is forgiving: object content is used as-is; a string is JSON-parsed, else parsed as a loose key: value block, else preserved under .raw. Never throws.
6. Render — ExperienceRenderer._renderWidget dispatches model to the runtime's renderer → {kind, data, runtime, runtimeName, _meta}. Your mount(descriptor) turns it into DOM.

Two delivery paths (this is the #1 gotcha)

• HTTP converse (Management.conversations.stream/send) — the reliable widget path. You pass force_experience and read widgets off the segment stream. collectConverse() (core/stream.js) separates them: experiences (keyed by wire runtimeName), experiencesList (arrival order), and kindCounts.experience. segmentKind(seg) classifies a unisphere-tool segment as 'experience' (vs 'spoken' for text/avatar/avatar-filler).
• Live avatar socket (KalturaAvatarSession) — the join payload hardcodes force_experience:'avatar_only' (src/experience/wire.js, EXPERIENCES join), so the socket emits structured widgets rarely. ExperienceRenderer.start() subscribes to brainSegment and tolerates zero widgets. Don't rely on rich widgets on the live face path — drive visuals yourself via the client-command channel, or read widgets from the HTTP path.

force_experience — a hint, not a contract

• Valid values (single source of truth, src/experience/wire.js line 17; EXPERIENCES): 'markdown', 'summarization', 'flashcards', 'avatar_only'.
• Parameters are validated on the first iteration (entering for await) in conversations.stream (conversations.js lines 120–121), NOT at call time — an invalid value throws a typed validation_error.
• It is a HINT: the brain decides which widget(s) to actually emit from the prompt + intellect. Asking for flashcards may yield flashcards-tool and followups-tool, or neither. The renderer renders whatever runtimeName arrives; tests are lenient by design.

Per-runtime detail (model keys → descriptor)

Each renderer lives in src/experience/genui/renderers/<name>.js, takes (model, ctx), and returns {kind, data}. ctx.urlPolicy ({allow:[schemes]}) is threaded from the ExperienceRenderer. Every renderer accepts multiple input key aliases (the model is untrusted LLM output) and clamps text via safeText(str, max) and URLs via safeUrl(url, policy).

1. flashcards (renderFlashcards)

Cards come from model.cards, model.items, or model.flashcards. Each card:

Descriptor: {kind:'flashcards', data:{title, cards:[{front, back, label}]}}.

2. followups (renderFollowups)

Descriptor: {kind:'followups', data:{questions:[string]}}. Server-side add_to_history:false — chips are suggestions, not replayed into history.

3. sources (renderSources)

Items come from model.sources, model.items, or model.citations. Each item:

Descriptor: {kind:'sources', data:{sources:[{title, url, snippet, score?}]}}. RAG-driven emission is unverified, so score's presence is NOT a claimed backend guarantee.

4. summary (renderSummary)

Descriptor: {kind:'summary', data:{title, summary, bullets:[string]}}. The summary stays untrusted (LLM output); by default mountWidget renders it as flat escaped text. Pass mountWidget(descriptor, el, {markdown:true}) to opt into a first-party, allow-listed markdown-to-DOM renderer instead — see "Markdown rendering" below. The SDK never emits raw HTML either way.

5. video-gallery (renderVideoGallery) — Kaltura clips

Items come from model.videos, model.entries, or model.items. Each item:

Descriptor: {kind:'video-gallery', data:{title, videos:[{entryId, title, thumbnailUrl, url, embedUrl, duration, description, alt}]}}. This is the in-platform video widget: the host renders the Kaltura player against entryId.

6. show-link (renderShowLink) — links

Descriptor: {kind:'show-link', data:{url, label, description, safe}} where safe:!!url — an unsafe scheme yields url:'' + safe:false so the host drops it (mirrors the earnings app's renderSafeLink null-drop).

7. external-video (renderExternalVideo) — non-Kaltura video embeds

Descriptor: {kind:'external-video', data:{url, title, provider, poster, description, safe}}, safe:!!url. The client check is defense-in-depth; the server-side media-URL validator is the primary guard (INFERRED — server validator not in this repo).

8. user-properties-form (renderUserPropertiesForm) — structured data collection

Fields come from model.fields, model.properties, or model.items. A field without a key is dropped. Each field:

required/description let a host wire aria-required/aria-describedby/inputmode.

Descriptor: {kind:'user-properties-form', data:{title, fields:[{key, type, label, knownValue, required, description}]}}.

• Report back: the default (user_properties_forms-configured) path has the host call session.submitStructuredDataForm(info) (session.js), which sanitizeJsons the object and emits the socket event setFormLeadInfo — a fire-and-forget emit with no durable server-side read-back. An app can instead take a different, durable path: never configure user_properties_forms at all, and reach this same user-properties-form widget purely as one enum value of its own show_widget client tool (kaltura_genie_experiences OFF), rendering the widget into its own dedicated host UI and, on submit, bridging the collected values into request_vars so the brain itself can call a server-side api tool that persists them wherever you point it — see STRUCTURED-DATA-FORMS.md.
• For the full picture — configuration, the exact mandatory prompt injection, where setFormLeadInfo actually persists server-side, and how to deliver the collected data somewhere durable — see STRUCTURED-DATA-FORMS.md and EXTERNAL-API-INTEGRATIONS.md.

9. content-gallery (renderContentGallery) — image/content cards

Items come from model.items, model.slides, or model.cards. Each item:

Descriptor: {kind:'content-gallery', data:{title, items:[{id, title, description, imageUrl, url, alt}]}}. This is the image-bearing widget (a deck/gallery of cards with thumbnails). Note the backend key is gallery_slides, and the video_gallery capability summary says it permits both video-gallery-tool and content-gallery-tool.

• Multi-item only. The renderer always wraps items in a CSS grid sized for several thumbnails (.kgenui__gallery, repeat(auto-fill, minmax(120px,1fr))) — it does not branch on item count, so a single, image-less item stretches to the grid's full row width inside the widget's full-slot frame and reads as an oversized, awkward card. A :has(> .kgenui__gallery > li:only-child) CSS rule can give that case a flex/centered treatment instead, and a show_widget tool description can steer the brain toward summary for a single text-only point. Prefer content-gallery for 2+ image-bearing items; use summary for one.

Authoring — which capability turns each widget on

Capabilities are set at intellect creation (partner config caches ~24h; set them up front). Source of truth: src/management/capabilities.js (CAPABILITY_INFO, CAPABILITY_DEFAULTS, OFF_BY_DEFAULT). kind is tool | segment | mode | prompt — this only names the mechanism that gates the capability on/off, not whether its content is persona-steerable. avatar_filler (kind: 'prompt') is the exception to watch for: its filler phrasing is server-generated per turn and NOT reliably steerable via base_directive, even though it streams as a "spoken" segment alongside avatar/text (see WIRE-PROTOCOL.md § 4e) — disable the capability if the default phrasing doesn't fit your persona.

The structured data-collection form (user-properties-form) is configured via the intellect's user_properties_forms (a LIST of {call_stage, properties:[{key,type}]} forms — the server rejects a bare object with 422), not a boolean capability. The eight OFF_BY_DEFAULT capabilities are: avatar, avatar_filler, avatar_show_content, video_gallery, external_video, show_link, use_web_search, screen_share_analysis.

Consuming widgets in your app

The 2-line happy path — pass a DOM Element as mount and the SDK renders for you, live:

import { ExperienceRenderer } from '@kaltura/intelligent-agents/experience/genui';
new ExperienceRenderer({ session, mount: document.getElementById('widgets'), onAction }).start();

mountWidget (genui/renderers/mount.js, exported from ./experience/genui) is the SDK's last-mile descriptor→DOM renderer — zero-dep, isomorphic (returns null with no DOM), never innerHTML, accessible by construction, and ships zero styling (it emits the kgenui / kgenui__* class contract for you to theme). Call it directly — mountWidget(descriptor, targetEl, { replace?, onAction? }) — or let ExperienceRenderer call it when mount is an Element (or target).

ExperienceRenderer options

new ExperienceRenderer({ session?, mount, target?, onAction?, renderers?, replace?, onUnhandled?, urlPolicy?, partnerId?, uiConfId?, clearOnTurnStart? }) (genui/renderer.js). uiConfId (string or number) enables video-gallery to build a player-embed iframe URL (requires partnerId). mount is a (descriptor)=>void function (full control) or a DOM Element (auto-rendered via mountWidget).

Live vs. headless dispatch

In the live socket runtime, .start() subscribes to session.on('brainSegment') and flushes on turnEnd/avatarStopTalking, resetting on interrupted. clearOnTurnStart (default true) also resets the assembler and clear()s rendered/last on the session's turnStart event (re-emitted from the raw agent_start_speech socket event), so a widget from turn N never lingers into turn N+1 — mirrors the Genie brain backend's own web client nulling content on AgentStartSpeechReceived. Set false for intentional cross-turn persistence.

Headless, call .render(runtimeName, widget) (or .render(segment)) per segment from a conversations.stream() feed — the reliable path.

Registration, fallback, and provenance

• .register(runtimeName, fn) adds/overrides a renderer (name normalized); .has, .runtimes, .last, .rendered, .clear(). replace:true keeps only the latest widget per turn.
• An unknown runtime renders {kind:'unknown', data:{runtime, model}} and fires onUnhandled (never throws). A throwing custom renderer degrades to {kind:'error', ...}.
• _meta receipt stamps {partnerId, source:'experience/genui', scope:'conversation (geniegpcid, entitlement ON)', known, firstClass} — known = this instance has a renderer (a registered 10th runtime is known:true); firstClass = one of the built-in set.

onAction, WIDGET_KINDS, and the hand-rolled escape hatch

onAction(action, payload) surfaces interactions mountWidget can't fulfil itself: 'followup' {question} (→ session.speak), 'submit' {values} (→ session.submitStructuredDataForm), 'play' {entryId,url,embedUrl}, 'open' {url}.

WIDGET_KINDS (exported) is the frozen list of every kind (the nine first-class GenUI runtimes + unknown + error) — use it for an exhaustive host switch or a parity test.

You can still read seg.metadata.runtimeName and build DOM yourself (use renderSafeLink/safeText/safeUrl so nothing un-sanitized hits the DOM). Prefer mountWidget — the hand-rolled path is for a fully custom design system only.

Markdown rendering (opt-in)

mountWidget(descriptor, el, {markdown:true}) parses a summary widget's text (genui/renderers/markdown.js, renderMarkdown) as markdown instead of flat text — headings (#-######), bold/bold, italic/italic, inline `code`, [link](url), unordered/ ordered lists, fenced code blocks (with a language-<token> class), and GFM tables. A markdown table renders through the tableEl builder (genui/renderers/dom-helpers.js) — one shared, safe <table> construction path, no duplicate table logic. This is markdown-in-plain-text rendering, not a new wire segment type: the underlying descriptor is still {kind:'summary', data:{summary, ...}}; only the render path changes. It is never innerHTML — every text run is built via textContent/createTextNode (so a raw <script> tag in the LLM output is inert text, not markup) and every extracted URL goes through safeUrl (a javascript:/unsafe-scheme link degrades to plain safe text, never a dead/unsafe href). Default behavior (no opts.markdown) is unchanged — flat escaped text — so no existing app regresses by upgrading.

Theming + a11y contract (kgenui classes)

mountWidget emits semantic, accessible DOM and these stable classes (theme them in your CSS — the SDK ships none): root kgenui kgenui--{kind} (role="group", aria-label); kgenui__title, kgenui__text, kgenui__list, kgenui__chip, kgenui__card, kgenui__flip, kgenui__back, kgenui__link, kgenui__img, kgenui__gallery, kgenui__form, kgenui__field, kgenui__label, kgenui__input, kgenui__help, kgenui__submit, kgenui__sr-only. Built-in a11y: flashcards are <button aria-pressed> flip toggles; followups are <button> chips in a labeled list; links carry a visually-hidden "(opens in a new tab)" cue + rel=noopener noreferrer and are dropped when safe:false; images always have alt; form fields are real <label for>+<input> with type/inputmode from the field type and aria-required/aria-describedby from required/description. The SDK ships zero CSS for these class names — style the .kgenui__* block in your own stylesheet.

Safety model (OWASP LLM05 — every widget passes through this)

src/core/safety.js:

• safeText(s, max=2000) — coerces to string, strips ASCII control chars (the C0 range U+0000-U+001F plus U+007F DEL), length-clamps.
• safeUrl(url, {allow}) — returns the URL only if its scheme is allow-listed (default https|http|mailto|tel); blocks javascript:/data:/vbscript:/unknown. A scheme-relative path (/foo, foo/bar) is allowed, but an authority-relative URL (//host, \\host) is rejected (open-redirect / embed-hijack vector). external-video additionally requires an absolute http(s) URL.
• renderSafeLink(info, opts) — builds a real <a> via createElement + textContent + scheme-checked href (never innerHTML); target=_blank + rel=noopener noreferrer; returns null outside a browser or for an unsafe URL.
• sanitizeJson(v) — drops __proto__/constructor/prototype (used by submitStructuredDataForm + setDynamicPrompt). clampInbound(s) — inbound text clamp.
• No renderer emits HTML; the host renders the {kind, data} descriptor with textContent / DOM APIs. A summary's markdown is the host's responsibility to sanitize.

Restrictions & gotchas (read before you build)

• Live socket rarely emits widgets — force_experience:'avatar_only' is hardcoded on the avatar join (wire.js). Use the HTTP converse path for reliable widgets.
• force_experience is a hint — never assume the requested experience arrived; the renderer parses whatever shows up.
• Only followups-tool + flashcards-tool are live-captured — the other seven runtimes and the multi-fragment SegmentAssembler boundary rules are INFERRED (unit-tested red/green, not live-verified). Source: genui/segments.js header.
• RAG-driven vs. config-driven emission is unverified — whether video_gallery / external_video / show_link fire from RAG hits or pure prompt tuning is not documented; the author-time lever is the capability + prompt, but the trigger is the brain's discretion.
• Backend may add runtimes outside this set — e.g. gen-ui-composer-tool, gen-ui-components-tool, kaltura-video-player-tool. They are NOT in RUNTIMES; the renderer routes them to {kind:'unknown'} + onUnhandled rather than faking a known kind.
• sources needs a knowledge base to cite from — ground a new agent via knowledge.addRecord
  • knowledge_ids (ungated; see API-REFERENCE.md § Ground the Agent). sources then renders the brain's real retrieved citations.
• entryId playback needs the Kaltura player — video-gallery preserves entryId; when embedUrl is present (requires partnerId+uiConfId), the SDK renders an inline player; without those the host renders by entryId.

Pointers (source of truth)