Case Study: Nova
TL;DR: Nova, the avatar on this site, runs on
@kaltura/intelligent-agentsand is fully open source. Her repo is the reference implementation for two things customers ask about most: knowing when a knowledge base has finished indexing, and running evals against a live agent.
Nova isn't a demo built to look good in a screenshot. She's the same avatar you can talk to right now, on this site, running the same code you can read at github.com/kaltura/docs-site-avatar. That repo provisions her, redeploys her, and proves she behaves correctly before every release. It's public, so you can study the patterns or lift them directly into your own build.
Checking whether your knowledge base finished indexing
Ground a Kaltura Agentic Avatar in your own content, and there's a gap between uploading it and the avatar being able to cite it. Indexing runs asynchronously and can take 45-90 seconds or more on a cold knowledge base.
knowledge.isIndexed() is not the signal to wait on. It reads the knowledge
record's own container status, which reports ready the instant the record
exists, before any of the entries you just uploaded have indexed. The real
per-entry check is knowledge.entryStatus().
Nova's own provisioning script polls it, up to a bounded budget, before it ever creates or updates her intellect (the agent's AI configuration). This is the same pattern documented in Ground the Agent in Your Content (RAG).
Nova resolves that wait before the create/update call, not after, because
partner configuration is cached for up to 24 hours server-side. A capability
flip sent as a follow-up patch can miss that cache window entirely. See
server/provision.mjs
for the live version of this sequencing.
Running evals against your agent
Once an agent is live, "does it still behave correctly" is an ongoing question, not a one-time check. Nova's eval suite runs her through adversarial personas. It scores each turn against release-blocking and soft dimensions. It also repeats each persona across multiple independent trials, to catch reliability gaps a single pass would miss. This is pass^k, not pass@k: a customer-facing agent needs to hold up across many conversations, not just one lucky one.
The suite is documented in docs/EVALS.md.
Its design rationale is covered in docs/ARCHITECTURE.md:
why adversarial personas, why pass^k, and why some failures block a release
and others don't. Both were written to be lifted, not just read: the harness
has no dependency on Nova specifically, so you can point it at your own
agent's personas and probes.
Where to go next
- Ground the Agent in Your Content (RAG): the indexing recipe in full
- docs-site-avatar on GitHub: Nova's complete source, provisioning script, and eval suite
- Getting Started: build your first agent with this SDK