GenUI Model & Runtimes

← Back to GenUI Reference

The model in one paragraph

The brain emits a GenUI widget by writing a fenced block carrying a widgetName. The server converts that into a stream segment of type:"unisphere-tool", shaped { type, content, metadata:{ widgetName, runtimeName }, speechId?, threadId? }. All widgets share widgetName:"unisphere.widget.genie" — the host keys off metadata.runtimeName (stripping the -tool suffix) to pick a renderer.

The SDK turns each segment into a framework-agnostic descriptor {kind, data} that your app maps to DOM. Nothing here emits HTML. Every string and URL is run through core/safety.js first.

The first-class runtimes

Backend tool key (defined server-side) → wire runtimeName → normalized dispatch key (the renderer registry key). Source: src/core/stream.js GENUI_RUNTIMES; src/experience/genui/parse.js RUNTIMES (derived from GENUI_RUNTIMES, so the two can never drift).

normalizeRuntime(name) (parse.js) strips a trailing -tool and trims; it tolerates an already-normalized name and a non-string (→ ''). isKnownRuntime(name) tests membership in this set. Any other runtime (e.g. the backend's gen-ui-composer-tool, gen-ui-components-tool, kaltura-video-player-tool — see GenUI · Safety and Restrictions) is NOT in this set and falls through to a safe fallback.

How a widget reaches your screen (the data flow)

1. Author — at intellect creation (the agent's configuration), enable the gating capability (see GenUI · Authoring and Consuming Widgets). The capability injects a template block into the system prompt telling the model when to emit that fenced widget.
2. Emit — the brain writes a fenced block with widgetName:"unisphere.widget.genie" and runtimeName. The server turns it into a unisphere-tool segment.
3. Stream — segments arrive as agent_raw_text deltas (live socket) or SSE/NDJSON lines (HTTP converse). A single widget can span multiple fragments.
4. Assemble — SegmentAssembler (genui/segments.js) buffers fragments and flushes a complete widget on a boundary change (different runtime or speechId, or turn end). If that boundary change interrupts a JSON body before it finishes writing, the fragment is flagged malformed (onMalformed) instead of flushed as a widget. ExperienceRenderer then mounts the same typed {kind:'error', data:{runtime, message}} fallback it uses for a throwing custom renderer.
5. Parse — parseWidget(segment) (parse.js) → {widgetName, runtimeName, runtime, model}. parseContent is forgiving and never throws: object content is used as-is; string content is JSON-parsed first, else parsed as a loose key: value block, else preserved under .raw.
6. Render — ExperienceRenderer._renderWidget dispatches model to the runtime's renderer → {kind, data, runtime, runtimeName, _meta}. Your mount(descriptor) turns it into DOM.

Two delivery paths (this is the #1 gotcha)

• HTTP converse (Management.conversations.stream/send) — the reliable widget path. You pass force_experience and read widgets off the segment stream. collectConverse() (core/stream.js) separates them: experiences (keyed by wire runtimeName), experiencesList (arrival order), and kindCounts.experience. segmentKind(seg) classifies a unisphere-tool segment as 'experience' (vs 'spoken' for text/avatar/avatar-filler).
• Live avatar socket (KalturaAvatarSession) — the join payload hardcodes force_experience:'avatar_only' (src/experience/wire.js, EXPERIENCES join). So the socket emits structured widgets rarely. ExperienceRenderer.start() subscribes to brainSegment and tolerates zero widgets. Don't rely on rich widgets on the live face path. Drive visuals yourself via the client-command channel, or read widgets from the HTTP path.

force_experience — a hint, not a contract

• Valid values (single source of truth, src/experience/wire.js EXPERIENCES): 'markdown', 'summarization', 'flashcards', 'avatar_only'.
• Parameters are validated on the first iteration (entering for await) in conversations.stream (conversations.js), NOT at call time — an invalid value throws a typed validation_error.
• It is a HINT: the brain decides which widget(s) to actually emit, based on the prompt and the intellect. Asking for flashcards may yield flashcards-tool and followups-tool, or neither. The renderer renders whatever runtimeName arrives. Tests are lenient by design.

Related docs