Audio Channels

← Back to Wire Protocol

What the SDK does on the two peer connections.

5. ASR uplink (pc1): microphone → server

A WebRTC peer connection that publishes the mic. SDP and ICE travel over the Socket.IO socket (the asr-webrtc-* events in §4a and §4c), not over HTTP. There is no separate signaling channel to manage.

ICE config (iceConfig() in src/experience/wire.js):

new RTCPeerConnection({
  iceServers: [{
    urls: [ "turn:<turnServerUrl>:80?transport=udp",
            "turn:<turnServerUrl>:443?transport=udp",
            "turn:<turnServerUrl>:80?transport=tcp",
            "turns:<turnServerUrl>:443?transport=tcp" ],
    username, credential }],
  iceTransportPolicy: "all",
  bundlePolicy: "max-bundle"
})

TURN servers

• If you pass turnServerUrl without turnCredentials, the SDK uses built-in fallback credentials and logs a security warning. Always pass turnCredentials in production.
• The STV downlink uses iceTransportPolicy: "relay", except on Firefox (isFirefox option), where it is "all".
• Keep explicit ports and transports on every TURN URL. A bare turn:host yields no relay candidate and the uplink sends no packets silently.
• WebKit rejects ?transport= query strings. The SDK retries once without them, which drops the plain-TCP port 80 URL on that engine only.

Signaling sequence:

→ asr-webrtc-init {sessionId}
← asr-webrtc-ready
  create offer
→ asr-webrtc-offer {offer, is_reconnect}
← asr-webrtc-answer {answer}
  setRemoteDescription
ICE trickles both ways: → asr-webrtc-ice-candidate, ← asr-ice-candidate

• Each wait (asr-webrtc-ready, asr-webrtc-answer) has a 30 s timeout. During connect() both are also bounded by the 30 s overall connect deadline. A timeout rejects connect() with timeout (error.phase names the step).
• The offer carries the mic track as Opus. With micStartMode: 'deferred' the offer carries an empty send-only audio slot, and startMic() attaches the track later without renegotiation.
• The SDK sets is_reconnect: true when it renegotiates this peer after an ICE failure (resilience).
• Healthy stats: outbound-rtp audio packetsSent climbs steadily, and the selected candidate-pair is nominated: true, state: succeeded.
• Once connected, the server transcribes this audio and feeds the agent. There is no "send transcript" call.

5b. Audio-mode WebRTC (separate from the ASR uplink)

Some agents run in audio/phone mode. Their stvNewSession reply carries status: "audio/phone mode - no STV session" and no session_id or webrtc_url (§4b).

KalturaAvatarSession handles that reply as follows:

• It sets session.mode to 'audio'.
• It skips the WHEP downlink (§6), so no videoMetadata fires.
• It emits mediaReady { mode: 'audio' } at once, so a loading UI can stop.
• It still runs the ASR uplink (§5).

This SDK does not implement an audio-mode WebRTC downlink. The mic uplink connects, but the agent's spoken reply does not reach you through this SDK. Audio/phone mode is not a supported feature. For video agents, use the §5 and §6 paths.

6. STV downlink (pc2): avatar video+audio → you

A receive-only WebRTC peer connection fed by WHEP (WebRTC-HTTP Egress Protocol). Signaling is plain SDP over HTTP, independent of the socket.

URL: the SDK POSTs to the webrtc_url from the stvNewSession reply, verbatim. If the reply has no webrtc_url, the SDK builds {srsBaseUrl}/rtc/v1/whep/?app=app&stream={session_id} from the srsBaseUrl session option (whepUrl() in wire.js). The SDK never sends cast_mode in stvNewSession.

ICE config:

iceTransportPolicy: "relay"     // 'all' on Firefox
bundlePolicy: "max-bundle"

It uses the same TURN URL block and options as §5.

• Transceivers: addTransceiver('video', {direction:'recvonly'}) and addTransceiver('audio', {direction:'recvonly'}).
• Private address guard: the SDK rejects connect() with whep_private_ip if the WHEP URL (or the Location header of the answer) points at a private, loopback or link-local host. A browser cannot reach one.

WHEP request:

POST {webrtc_url}
Content-Type: application/sdp
body: <client offer SDP>          → response body: <answer SDP>  (HTTP 201)

Timeout and retry: each POST gets 5 s, and reading the answer body counts toward it. A try that times out or fails on the network is retried after 1 s, up to 3 tries in all, inside the connect deadline. An HTTP status is never retried, because each status has its own meaning (below). If every try fails, connect() rejects with whep_timeout (the last try timed out) or whep_failed (the last try failed on the network). Both carry phase: 'whep' and retryable: true. Tune with timeouts.whepTry, whepTries and whepBackoff.

Teardown: the SDK sends DELETE to the WHEP resource named by the 201's Location header. The SDK builds the release URL as the POSTed subscribe URL plus the header's /viewer/… suffix (whepResourceUrl() in wire.js). An absolute Location is used as-is. Any other relative one resolves against the subscribe URL. A custom client must build the URL the same way, or the DELETE misses and the viewer slot stays held. The DELETE has a 3 s deadline (timeouts.whepRelease) and never throws. A teardown DELETE is sent with keepalive, so it outlives a page that closes right after (Resilience: page exit). A DELETE can only name a resource whose answer arrived, so a POST aborted in flight leaves nothing to release.

WHEP status codes: 404 and 409 mean the avatar session cannot take this subscribe (it is gone, or a viewer is already attached, for example after a timed-out try whose answer was lost). The SDK asks for a new avatar session once on the live socket and POSTs again, both at connect and during media recovery. A second 404 or 409 rejects with stv_session_gone. Any other non-2xx rejects connect() with whep_failed, the HTTP status, and a hint in detail for 415. If the re-subscribe during media recovery still fails, the SDK does a cold reconnect.

• Codec: the video stream is H264. If you set preferredVideoCodec to another codec, the request still returns 201 with a valid answer, but the video m= line comes back a=inactive. No frame is decoded and no error surfaces. Audio is unaffected. Leave the option unset.
• Healthy stats: inbound-rtp video frame dimensions stay stable, framesDecoded and bytesReceived climb steadily, and the selected pair is nominated: true, state: succeeded with both candidates relay.
• Greeting gate: send approvedPermissions only after the video is playable: the first painted frame, or <video> canplay plus 100 ms when no frame callback exists (a 300 ms settle when there is no videoEl). See the playable-video gate and steps 10 and 11 of the connect sequence.

Related docs