WebSocket API

Note: This is a consolidated document. For detailed specifications, refer to the individual documents under reference/websocket/.

Note: The URL used in this document (vas-poc.vurbo.ai) is the planned deployment address. A separate notice will be issued after the official launch.

Table of Contents

1. Connection Info
2. Authentication
3. Message Format
4. Health - Heartbeat Service
5. Voice Translation - start
6. Voice Translation - config
7. Voice Translation - audio
8. Voice Translation - pause
9. Voice Translation - resume
10. Voice Translation - stop
11. Voice Translation - retranslate
12. Voice Translation - switch_language
13. Voice Translation - set_name
14. Voice Translation - rename_speaker
15. Voice Translation - reassign_speaker
16. Voice Translation - merge_speakers
17. Voice Translation - tts_play
18. Voice Translation - tts_stop
19. Voice Translation - tts_mode
20. Voice Translation - set_tts
21. Voice Translation - start_speaking
22. Voice Translation - stop_speaking
23. Voice Translation - switch_conversation_mode
24. Voice Translation - set_speaker_language
25. Voice Translation - set_speaking_speed
26. Voice Translation - add_channel
27. Voice Translation - remove_channel
28. Voice Translation - set_channel_language
29. Voice Translation - broadcast_go_live
30. Voice Translation - broadcast_announcement
31. Voice Translation - set_standby_message
32. Response Events

Connection Info

Authentication

The VAS WebSocket uses a Ticket mechanism for authentication, passing a one-time Ticket via Sec-WebSocket-Protocol. For details, refer to Authentication.

Step 1: Obtain a Ticket

Exchange your API Key for a one-time Ticket via the REST API:

POST /api/v1/auth/ticket
X-API-Key: vas_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Response:

{
  "ticket": "aBcDeFgHiJkLmNoPqRsTuVwXyZ012345",
  "expires_in": 60
}

Step 2: Connect to the WebSocket using the Ticket

Place the Ticket into Sec-WebSocket-Protocol in the format ticket.{TICKET_VALUE}:

// Native browser support
const ws = new WebSocket('wss://vas-poc.vurbo.ai/ws', [`ticket.${ticket}`]);

ws.onopen = () => {
  console.log('Connected! Protocol:', ws.protocol);
  // Start using the WebSocket...
};

ws.onerror = (error) => {
  console.error('Connection failed:', error);
};

Node.js example:

const WebSocket = require('ws');

const ws = new WebSocket('wss://vas-poc.vurbo.ai/ws', [`ticket.${ticket}`]);

Ticket Characteristics

Ticket Error Codes

For the full API specification, refer to Auth Ticket API.

Message Format

All messages use a unified nested structure:

{
  "type": "service type",
  "data": { ... }
}

Service Types

Error Message Format

When an error occurs, the server returns a message with type: "error":

{
  "type": "error",
  "data": {
    "error_code": "auth_invalid_api_key",
    "severity": "fatal",
    "message": "Invalid API key",
    "context": "auth",
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-01-15T10:30:45.123Z"
  }
}

Sentence-level errors (such as a translation failure for one language of a sentence) additionally carry sid and details:

{
  "type": "error",
  "data": {
    "error_code": "llm_content_filtered",
    "severity": "warning",
    "message": "Content filtered",
    "context": "translation",
    "sid": 5,
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-01-15T10:30:45.123Z",
    "details": {
      "provider": "llm_service",
      "translation_language": "ja-JP"
    }
  }
}

Session-level translation service errors (escalated after consecutive failures reach a threshold) do not carry sid. The frontend should display a global notice but does not need to disconnect:

{
  "type": "error",
  "data": {
    "error_code": "translation_service_unavailable",
    "severity": "error",
    "message": "Translation service unavailable",
    "context": "translation",
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-01-15T10:30:45.123Z",
    "details": {
      "provider": "llm_service",
      "last_error_code": "llm_provider_error",
      "fail_count": 5
    }
  }
}

For the full trigger rules (consecutive failure threshold, error code classification), refer to the translation_service_unavailable section in Error Code Reference.

Plan Limit Errors (unlimited plans, v1.9.0)

API Keys on an "unlimited plan" may receive the following error messages (not applicable to credit-based keys). Use GET /api/v1/me/plan to look up the plan contents, current usage, and when a restriction lifts:

plan_feature_not_allowed has two occurrence points:

1. start rejected: when the plan does not include a feature enabled in the request, the start is rejected outright; the connection is not closed — adjust the parameters and start again.
2. Detected while recording: for example, a feature not in the plan is turned on mid-session; once the check before the next minute detects it, the current recording is stopped.

For real-time recording, usage limits are checked before each minute begins; once a limit is reached, the next minute does not start and is not counted toward usage. When the same API Key runs several recordings at once, after the periodic-stop threshold is reached, each recording stops before its own next minute begins.

In addition, details.max of too_many_languages may come from the plan's cap on simultaneously recognized transcription languages, in addition to the system-wide limit (10 transcription languages); details carries max and received. See Error Code Reference — Plan and Usage Limit Errors.

Single-Message Error (Message Failed, Connection Kept)

When the server encounters an unexpected internal error while handling a single WebSocket message (such as set_name, switch_language, tts_play, etc.), it returns internal_error. This error indicates only that the specific message failed to process; the connection is not terminated. The frontend should keep the connection open and may retry the operation:

{
  "type": "error",
  "data": {
    "error_code": "internal_error",
    "severity": "error",
    "message": "Internal server error",
    "context": "general",
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-05-08T10:30:45.123Z",
    "details": {
      "message_type": "voice-translation",
      "action": "set_name"
    }
  }
}

details Fields

What the Frontend Should Do

1. Keep the WebSocket connection open: Do not call ws.close(), navigate away, or return to a history page because of this error. The recording is still in progress.
2. Decide on follow-up handling based on details.action:

   Scenario	Recommended Action
   Idempotent operations such as set_name / switch_language / tts_mode / config	Simply resend the same message. These operations use a "last write wins" approach, so retrying has no side effects.
   tts_play / tts_stop / retranslate	Usually safe to retry directly. If the user is waiting for TTS playback, consider showing a transient toast indicating the retry is in progress.
   speaker.rename / speaker.merge	Before retrying, use the REST API (speakers) to confirm the current DB state and avoid duplicate operations (for example, the rename already succeeded and only the response frame failed).
   details.action is absent	The error occurred after the message payload was parsed, so the system cannot infer the specific operation. The frontend can infer it from "the most recent message the user sent," or display a generic error message such as "Operation failed, please retry."

3. User experience: Show a transient toast / inline error. Do not interrupt the user flow with a modal or a redirect.
4. Telemetry / reporting: Report request_id + details to your frontend error tracking (Sentry, Datadog, etc.) to make it easier to correlate with backend logs during troubleshooting.

What Will Not Happen (Guarantees)

• The recording will not be interrupted: segment_uploaded, result, origin, and other messages keep arriving.
• The connection will not be actively closed by the server.
• The session state will not be reset (session_id stays the same).
• State already written to the DB will not be rolled back (for example, if set_name was written to the DB successfully and only the response frame failed, the name still takes effect).

Client Handling Example

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type !== 'error') {
    handleNormalMessage(msg);
    return;
  }

  const { error_code, severity, request_id, details } = msg.data;

  // Single-message failure: keep the connection, decide whether to retry based on action
  if (error_code === 'internal_error') {
    console.warn('[ws] message handler failed (connection kept)', {
      request_id,
      message_type: details?.message_type,
      action: details?.action,
    });
    showTransientToast(`Failed to process "${details?.action ?? 'operation'}", please retry`);
    // Note: do not call ws.close() and do not navigate away from the current page
    return;
  }

  // Handle other errors with your existing logic (only fatal errors require disconnecting)
  handleErrorBySeverity(severity, msg.data);
};

For the full list of error codes, refer to Error Code Reference.

Health (Heartbeat Service)

Description

Used to confirm that the WebSocket connection is healthy. We recommend sending a ping every 30 seconds; if no pong is received, treat the connection as dropped and reconnect. While the previous recording is still being processed after it ends (for example, while its summary is being generated), pong can be delayed by a few seconds to a few tens of seconds; allow enough waiting time and do not treat the connection as dropped just because a pong has not arrived yet.

Use Cases

• Maintaining a long-lived connection
• Detecting connection status
• Preventing connection timeouts

Request - Ping

{
  "type": "health",
  "data": {
    "action": "ping"
  }
}

Response - Pong

{
  "type": "health",
  "data": {
    "action": "pong"
  }
}

Voice Translation - start (Start Voice Translation)

Description

Starts a new voice translation session and begins processing audio according to the configured parameters.

Use Cases

• Starting a meeting record
• Starting real-time translation
• Starting a voice memo

Request Parameters

options Sub-fields

options is the speech recognition options object; all fields are optional and use their respective defaults when omitted.

Note: These options apply to STT sentence segmentation; multi-speaker mode (multi_speaker) does not currently apply speaking_speed.

Recording Type Descriptions

Request Example (Basic)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "realtime_translation": false,
    "type": "transcribe",
    "audio_format": "pcm",
    "summary_template": "meeting",
    "options": {
      "speaking_speed": "normal",
      "profanity_handling": "mask"
    }
  }
}

Request Example (Initial Default Name)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "type": "transcribe",
    "audio_format": "pcm",
    "summary_template": "meeting",
    "name": "Product Planning Meeting"
  }
}

Recording Name Rules

Note: The name in start is the initial default name; the system may still override it when the session ends. If you need a fixed name, use set_name.

Default name format (fixed English):

N is the sequential number for that user's recordings of the same type. Name priority: user > llm > default. Once the user sets a name, the system will not override it when the session ends.

Request Example (With TTS)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "realtime_translation": true,
    "type": "transcribe",
    "tts_enabled": true,
    "tts_language": "en-US",
    "tts_voice": "en-US-JennyNeural",
    "tts_mode": "sync"
  }
}

Request Example (Two-Way Mode - Auto-Detect)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "conversation",
    "transcription_languages": ["zh-TW", "en-US"],
    "active_language": "zh-TW",
    "audio_format": "pcm",
    "speakers": [
      { "id": 1, "language": "zh-TW" },
      { "id": 2, "language": "en-US" }
    ],
    "tts_config": {
      "zh-TW": { "voice": "zh-TW-HsiaoChenNeural", "speaking_rate": 1.0 },
      "en-US": { "voice": "en-US-JennyNeural", "speaking_rate": 1.0 }
    }
  }
}

Request Example (Two-Way Mode - Manual Mode)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "conversation",
    "transcription_languages": ["zh-TW", "en-US"],
    "conversation_mode": "manual",
    "audio_format": "pcm",
    "speakers": [
      { "id": 1, "language": "zh-TW" },
      { "id": 2, "language": "en-US" }
    ],
    "tts_config": {
      "zh-TW": { "voice": "zh-TW-HsiaoChenNeural", "speaking_rate": 1.0 },
      "en-US": { "voice": "en-US-JennyNeural", "speaking_rate": 1.0 }
    }
  }
}

Request Example (Custom Summary Prompt - custom mode)

In mode=custom, your summary_prompt content completely replaces the built-in template rules, and the backend already adds prompt injection protection. The summary_prompt_slug is metadata for your own identification (stored in the backend record) and does not enter the prompt content.

If you want to keep the built-in template and add your own supplemental instructions afterward, use summary_mode=builtin + summary_template=<slug> + summary_prompt=<supplemental instructions> instead (in builtin mode, summary_prompt is treated as supplemental and appended after the built-in template).

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "type": "transcribe",
    "audio_format": "pcm",
    "summary_language": "zh-TW",
    "summary_mode": "custom",
    "summary_prompt": "You are a meeting-minutes assistant. List every amount and committed date discussed in bullet points, and note the responsible person for each.",
    "summary_prompt_slug": "client_x_finance_v3",
    "summary_plain_text": false
  }
}

Important — How to Retrieve the Summary Result: In WebSocket mode, summaries are non-streaming by design; final_content is not pushed back via a WebSocket event (the summary_done event only signals completion and does not contain the content). The client must retrieve it afterward over HTTP:

1. After receiving the summary_done event, call GET /api/v1/sse/history/transcribe/{taskId} to retrieve the summary (the init_summary event carries a top-level summary plain string + summary_mode / summary_template / summary_plain_text / summary_prompt_snapshot + the two content-filter fallback audit fields summary_fallback_level / summary_dropped_segments added in v1.5.5).
2. Or query the summary_mode / summary_template / summary_prompt_slug fields of the transcript record via the REST API.

v1.5.5 Content-Filter Automatic Downgrade: If your prompt or transcript content triggers the LLM service's content filter, the system automatically downgrades (standard mode → neutral mode → segment-omission mode). The summary_fallback_level field of the summary_done event (value 2 or 3; omitted when standard mode succeeds directly) tells the client which path was actually taken, so the frontend can display hints such as "neutral mode in use" / "N segments omitted." See reference/websocket/events.md – summary_done and the V1.5.5 changelog.

Two-Way Mode Special Rules:

speakers Field Descriptions:

conversation_mode Descriptions:

Multi-Channel Mode Description (recognition_mode: "multi_channel")

A single recording session takes input from multiple physical microphones at the same time. Speaker identity is determined by the channel (not inferred by AI): each transcript sentence is attributed to the speaker of the channel_id it came from. There are two sub-modes:

• per_channel: each microphone runs its own independent speech recognition; channels can speak at the same time and each can be bound to a different language.
• shared: channels take turns speaking and share one recognition stream, and the whole session uses the same set of languages. Suited to situations where only one person speaks at a time; see Shared Mode below.

Scope and limits:

channels field descriptions:

Multi-channel mode request example:

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "transcribe",
    "recognition_mode": "multi_channel",
    "channel_mode": "per_channel",
    "transcription_languages": ["zh-TW", "en-US"],
    "translation_languages": ["ja-JP"],
    "audio_format": "pcm",
    "channels": [
      { "channel_id": 1, "speaker_name": "Manager Wang", "transcription_languages": ["zh-TW"] },
      { "channel_id": 2, "speaker_name": "Alex", "transcription_languages": ["en-US"] },
      { "channel_id": 3, "speaker_name": "Lee", "transcription_languages": ["zh-TW"] }
    ]
  }
}

Successful response (session_started, multi-channel):

The top level of data additionally carries channel_mode and channels[] (each entry with channel_id, speaker_name, transcription_languages, status; transcription_languages is not present in shared mode). The frontend should use this to confirm the server actually started in multi-channel mode:

{
  "type": "voice-translation",
  "data": {
    "action": "session_started",
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "task_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "recording_type": "transcribe",
    "recognition_mode": "multi_channel",
    "channel_mode": "per_channel",
    "channels": [
      { "channel_id": 1, "speaker_name": "Manager Wang", "transcription_languages": ["zh-TW"], "status": "preparing" },
      { "channel_id": 2, "speaker_name": "Alex", "transcription_languages": ["en-US"], "status": "preparing" },
      { "channel_id": 3, "speaker_name": "Lee", "transcription_languages": ["zh-TW"], "status": "preparing" }
    ],
    "resume_token": "L0VBAwIy... (43 characters)",
    "resume_grace_seconds": 45,
    "server_time": 1749550000000,
    "message": "Speech recognition started"
  }
}

Multi-channel mode behavior summary:

• Every audio frame must include channel_id (see the audio section); we recommend one frame every 100ms, and every channel should keep sending audio even while silent. A single silent channel does not end the recording; see Automatic End After a Long Silence
• The origin of result events carries channel_id, speaker_id (format channel_{N}), and speaker_label (= speaker_name; after a rename_speaker, the new label); translations does not carry channel_id — use sid to match back to origin
• During recording you can dynamically add or disable channels and change a channel's language via add_channel / remove_channel / set_channel_language; switch_language always returns multichannel_switch_language_not_allowed
• rename_speaker works (a channel's speaker can be renamed before they say anything); reassign_speaker and merge_speakers do not apply to multi-channel (speaker identity is determined by the physical channel)
• Channel status changes are reported via the channel_status event

Shared Mode (Taking Turns)

channel_mode: "shared": the microphones take turns speaking and share one recognition stream. The speaker is labeled by which channel that stretch of audio came in on, which is highly accurate when people take turns; billing is always counted as 1 channel (see Pricing).

Request example:

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "transcribe",
    "recognition_mode": "multi_channel",
    "channel_mode": "shared",
    "transcription_languages": ["zh-TW", "en-US"],
    "audio_format": "pcm",
    "channels": [
      { "channel_id": 1, "speaker_name": "Host" },
      { "channel_id": 2, "speaker_name": "Guest A" },
      { "channel_id": 3, "speaker_name": "Guest B" }
    ]
  }
}

session_started returns channel_mode: "shared"; the entries in channels[] do not carry transcription_languages.

Client requirements:

Channel status:

• The status of the other channels follows the first channel: when the first channel turns ready, all channels turn ready together; when the first channel prepares recognition again (resume after pause, resume after disconnect, automatic reconnection), all channels return to preparing together. Each channel receives its own channel_status event, with the same reason as the first channel. After resuming from a disconnect, preparing is reported in the resume_ok snapshot rather than by a separate event.
• A channel added with add_channel during the recording takes on the first channel's current status directly.
• removed follows each channel's own status; a removed channel receives no further events.
• When the first channel turns error, the other channels turn error as well (with the same reason); when the first channel recovers, they recover together.
• The channels[] snapshots in session_started and resume_ok follow the same rules.

Known limitations:

• When the gap between speakers is shorter than about 0.8 seconds, the words of the two people may be merged into one sentence labeled with only one speaker.
• Speakers are determined by channel labeling; merging and reassigning speakers through the speaker APIs do not apply (same as the common multi-channel rule), so this cannot be corrected afterward.
• Retroactive transcription after resuming from a pause covers the whole last 60 seconds, for all channels together, with speakers labeled (see pause).

Multi-channel start-specific errors:

Broadcast Mode Description (type: "broadcast")

In broadcast mode, the language settings are automatically obtained from the broadcast channel settings and do not need to be sent in the WebSocket message.

Required parameters:

Optional parameters (override the broadcast channel settings):

Auto-configured parameters (can be omitted):

• transcription_languages: read automatically from the broadcast settings
• translation_languages: read automatically from the broadcast settings
• realtime_translation: always enabled in broadcast mode, and false is treated as true; translation is always billed at the real-time translation rate
• summary_template: read automatically from the broadcast settings (the value passed via WebSocket takes precedence)
• summary_language: read automatically from the broadcast settings (the value passed via WebSocket takes precedence)

Both the host and viewers receive interim translations: a sentence's translation first arrives with is_final: false, followed by the finalized version with is_final: true. Overwrite the display by sid plus language; to show only finalized translations, skip is_final: false.

The recording name does not reuse the channel name. If start omits name, a name such as Broadcast #1 is generated; naming otherwise works the same as for other types, see Recording Name Rules.

Broadcast Phase Descriptions:

Standby phase purpose: Lets the host warm up STT/translation before going live, confirm that equipment is working, and then switch to the live phase.

The standby phase has a time limit (30 minutes by default): when the accumulated standby time reaches the limit, the session ends automatically, with a warning about 2 minutes before. See Standby Time Limit.

broadcast_phase accepts only lowercase standby and live: an empty string is treated as live, and any other value is rejected (invalid_parameter).

Broadcast Mode Request Example:

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm"
  }
}

Broadcast Mode Request Example (Standby Phase + Override Summary Template):

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm",
    "broadcast_phase": "standby",
    "standby_message": "The talk is about to begin, please wait...",
    "summary_template": "lecture"
  }
}

Summary template priority: The value passed in the WebSocket start > the default set when the broadcast channel was created. If neither is set, no summary is automatically generated.

Broadcast Mode TTS Settings (tts_config):

Use the tts_config parameter to specify which translation languages should produce TTS audio for viewers.

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm",
    "tts_config": {
      "en-US": {
        "voice": "en-US-JennyNeural",
        "speaking_rate": 1.0
      },
      "ja-JP": {
        "voice": "ja-JP-NanamiNeural",
        "speaking_rate": 1.0
      }
    }
  }
}

Note:

• TTS languages must be valid languages in translation_languages; invalid languages are automatically ignored.
• The host (WebSocket) does not receive TTS audio; only SSE viewers receive the tts_ready event.
• TTS is sent only during the live phase; nothing is sent during the standby phase.

TTS Playback Mode Descriptions

Success Response

After a successful start, a session_started event is returned containing complete session initialization info. For real-time recording, the first minute is deducted first, and the event is returned once that deduction completes (see Pricing).

General recordings (transcribe / conversation / record):

{
  "type": "voice-translation",
  "data": {
    "action": "session_started",
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "task_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "recording_type": "transcribe",
    "recognition_mode": "single",
    "resume_token": "L0VBAwIy... (43 characters)",
    "resume_grace_seconds": 45,
    "server_time": 1749550000000,
    "message": "Speech recognition started"
  }
}

Broadcast mode (broadcast):

{
  "type": "voice-translation",
  "data": {
    "action": "session_started",
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "task_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "recording_type": "broadcast",
    "recognition_mode": "multi_speaker",
    "phase": "standby",
    "viewer_count": 0,
    "queue_count": 0,
    "peak_viewers": 0,
    "total_viewers": 0,
    "resume_token": "L0VBAwIy... (43 characters)",
    "resume_grace_seconds": 45,
    "server_time": 1749550000000,
    "message": "Speech recognition started"
  }
}

Multi-channel mode: the top level of data additionally carries channel_mode and channels[]; see "Multi-Channel Mode Description" above for an example and field descriptions.

Automatic End After a Long Silence

A recording that goes a certain length of time without recognizing any text (including interim results that are not final yet) ends automatically, so a recording someone forgot to stop does not keep being billed.

• The default threshold is 900 seconds (15 minutes) and can be changed per session with silenceTimeoutSeconds. About 2 minutes before the end, a warning is sent once.
• Any recognized text restarts the count from 0; resume and start_speaking also restart it.
• The count does not run in these cases:
  • While paused: the count restarts from 0 after resuming. A paused recording is still billed; to stop billing, end the recording.
  • While a disconnected session waits to be resumed: after resuming, the count continues from where it was.
  • Broadcasts (type: "broadcast"): never end for lack of speech. The standby phase has its own time limit; see the Broadcast Guide.
• Multi-channel: counts only when no channel recognizes any text; a single silent channel does not end the recording.
• Conversation manual mode: speech recognized while the speak button is not pressed also restarts the count.

Values of silenceTimeoutSeconds (top level of the start data, optional)

• The value must be a JSON integer: strings (for example "900"), decimal notations (for example 900.0 or 1e3), and booleans are all rejected.
• The parameter name is silenceTimeoutSeconds; sending silence_timeout_seconds is rejected (details.field is silence_timeout_seconds), not ignored.
• A valid value has no effect on broadcasts, but an invalid value is still rejected.
• With a threshold of 120 seconds or less, no warning is sent; the recording simply ends when the time is up.
• Resuming a disconnected session keeps the original setting; send it again when starting a new recording.

Events you receive

1. stt_silence_warning (an error event with severity: "warning"): a warning; the recording continues. details.silenceSeconds is how long the silence has lasted and details.remainingSeconds is how many seconds remain. Do not treat it as the end of the recording; recognized text or resuming the recording restarts the count.
2. stt_silence_timeout (severity: "fatal"): the recording has ended automatically. details.silence_seconds is the threshold in seconds.
3. Then status: "ended" and task_complete, in that order, just as when the client sends stop: the recording is saved and summarized as usual, and billing runs until the end.

{
  "type": "error",
  "data": {
    "error_code": "stt_silence_warning",
    "severity": "warning",
    "message": "No speech detected for a while; the recording will end automatically soon",
    "context": "stt",
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-09-25T10:28:00.000Z",
    "details": {
      "silenceSeconds": 780,
      "remainingSeconds": 120
    }
  }
}

After stt_silence_timeout, stop sending audio. Each audio message that arrives afterwards gets a session_not_started reply, which you can ignore.

Error Responses

Voice Translation - config (Set Terminology / Correction Rules)

Description

Before or during recording, pass in terminology, fuzzy-word correction rules, and translation dictionary settings. These settings improve STT accuracy, fix homophone errors, and ensure translation consistency.

Terminology also drives homophone correction: When terminology is provided, the terms become the reference for homophone matching — any span in the transcript that sounds the same but is written differently is corrected back to the spelling of the term. Providing terminology alone is therefore enough to get correction; you do not need to list possible misspellings by hand.

Limits

Per-block entry limits, length limits, and the matching error codes are collected in Terminology Guide → Limits.

The numbers are defaults; the limit actually in force can be tuned per environment — always treat max in the error response's details as authoritative rather than hard-coding the numbers.

Use Cases

• Pass in professional terminology (Phrase List) before recording starts
• Set fuzzy-word correction rules (homophone correction) - optional; terminology itself already provides homophone correction
• Set a translation dictionary (ensure consistent terminology translation)

Timing

Note: When you update terminology, fuzzy-word correction, or the translation dictionary during recording, the new settings take effect from the next utterance after they are sent; the utterance currently being recognized is not guaranteed to be covered. No reconnection is needed. When terminology is updated, the response includes a terminology_effective: "next_turn" field as a hint.

Request Parameters

Note: At least one setting item must be provided.

Note: All or nothing: all three blocks are validated before any of them is applied. If any block fails validation, an error is returned and none of the three blocks is applied — the settings stay as they were.

For example, sending a valid terminology list together with more than 3000 translation-dictionary entries returns config_too_many_dict_entries, and the terminology does not take effect either. Fix the problem and resend the complete config; all three blocks replace their previous value wholesale, so resending does not stack on top of earlier settings.

Terminology Format (terminology)

Keyed by language code, with an array of terms as the value:

{
  "zh-TW": [
    { "term": "語者分離" },
    { "term": "WebSocket" }
  ],
  "en-US": [
    { "term": "diarization" }
  ]
}

Limit: Up to 500 terms across all languages combined in one config — not 500 per language. Exceeding the limit returns config_too_many_entries (with count and max in details).

Language scope: Terms apply only to recognition in the language they are registered under. Single-language situations — each multi-channel track, speaker diarization, and file import — use only that language's terms; multi-language transcription and conversation mode use the terms for the languages declared for the session. Terms registered under a language not used in this session have no effect, but still count toward the 500 combined total above.

These numbers are defaults: the limit actually in force can be tuned per environment; always treat max in the error response's details as authoritative.

Fuzzy-Word Correction Format (fuzzy_correction)

Note: This field usually does not need to be set manually — terminology already corrects misspellings that sound the same or nearly the same. Use it in these three cases:

1. The misspelling is itself an ordinary word and is therefore blocked by common-word protection — 晶圓 heard as 金元, for example
2. The misspelling sounds very different from the correct term, such as a foreign brand name recognized as a phonetically unrelated word
3. Misspellings in Japanese, Korean or English, which do not participate in homophone matching

When correct is Chinese, it also becomes a reference for homophone matching — homophone misspellings not listed in incorrect are corrected to correct as well. case_insensitive applies only to the literal matching of incorrect; it does not affect homophone matching.

Keyed by language code, with an array of correction rules as the value:

{
  "zh-TW": [
    { "correct": "語者分離", "incorrect": ["語這分離", "語者分力"] },
    { "correct": "IPEVO", "incorrect": ["ltfo"], "case_insensitive": true }
  ]
}

Supplying only the correct term: when correct is Chinese (contains Han characters), incorrect may be omitted entirely — the system matches by pronunciation, and spellings in the transcript that sound the same or nearly the same are corrected back to correct.

{ "fuzzy_correction": { "zh-TW": [{ "correct": "艾思通" }] } }

No misspellings need to be listed above: 愛思通, 愛時通, 愛司東 and 愛似通 are all corrected. Only spellings that sound quite different (愛自動, say) or have a different number of syllables (愛松) still need to be listed in incorrect.

Note: Both conditions must hold: the language must be Chinese (zh-TW, zh-CN, zh-HK and so on) and correct must contain Han characters. Otherwise incorrect remains required — omitting it in those cases would have no effect at all, and accepting it would leave you believing the setting took. List misspellings explicitly for Japanese, Korean and English.

Case sensitivity: case_insensitive is optional and defaults to false (exact-case matching). When set to true, every incorrect variant in that rule matches regardless of case. The flag is per rule — the same correct term can be split across several rules with different settings, for example making variants that cannot collide with ordinary words case-insensitive while keeping variants that could hit a personal name exact. It has no effect on Chinese rules (Chinese has no letter case).

Note: Enabling it widens the false-positive surface: if ivo is case-insensitive, the personal name Ivo is replaced too.

When the same incorrect variant appears in more than one rule: collisions are resolved on incorrect (the variant), not on correct.

• When several rules point at different correct terms, which one actually takes effect is not guaranteed; do not rely on any ordering (registration order included)
• The case flag resolves to strict wins — if any rule leaves case_insensitive off, that variant is matched with exact case

"Strict wins" is a deliberately conservative choice: it prevents a permissive rule elsewhere in your vocabulary from silently loosening a brand-name rule you explicitly set to strict.

Splitting one correct term across several rules is therefore safe, as long as their incorrect variants do not overlap. But if your data contains the same incorrect variant mapped to different correct terms, the later one is dropped with no warning — check for duplicate variants before sending.

How Terminology Participates in Homophone Correction

Once terminology is provided, the terms become the reference for homophone matching. Any span in the transcript that sounds the same or nearly the same but is written differently is corrected back to the spelling of the term:

You do not need to list the possible misspellings in advance — matching is based on pronunciation, so coverage is not limited by how many variants you can think of.

Mixed Chinese-English terms: For a term like CVD製程, matching applies only to the Chinese portion; the Latin portion is left unchanged.

Applicable languages: Homophone matching applies to Chinese only (Traditional and Simplified are interchangeable, since they share pronunciation). Terms in Japanese, Korean or English do not participate in homophone matching — list their misspellings explicitly with fuzzy_correction.

Matching range: Both identical and near-identical pronunciations are covered, including accent differences such as jin vs jing (final -n vs -ng) and retroflex vs non-retroflex initials. For the term 晶圓廠, for example, 金圓廠 in the transcript is corrected.

Common-word protection: If the span in the transcript is itself an ordinary word (金元 or 反案, say), it is not changed even when it shares a pronunciation with a term — this prevents normal sentences from being altered. To force such a correction, list the misspelling explicitly with fuzzy_correction: explicitly listed misspellings are not subject to common-word protection.

Limit: Up to 4000 rules across all languages combined in one config. Exceeding it returns config_too_many_entries (with field: "fuzzy_correction", count and max in details).

This number is a default: the limit actually in force can be tuned per environment, so always treat max in the error response's details as authoritative. Do not hard-code the numbers in your integration — if you want to check before sending, read max and fill it back in. When you do hit a limit, details carries both count (what you sent) and max (the limit in force).

Language scope: A correction rule applies only to sentences in the language it is registered under — a rule under zh-TW will not alter an English sentence. When the sentence language cannot be determined, all rules are applied as a fallback (better to over-apply than to skip the sentence entirely). Homophone matching driven by terminology likewise follows the language each term is registered under.

Translation Dictionary Format (translation_dict)

Group entries by language code, giving each language its own dictionary:

{
  "en-US": [
    { "source": "語者分離", "target": "Speaker Diarization" },
    { "source": "晶圓", "target": "wafer", "case_sensitive": true }
  ],
  "ja-JP": [
    { "source": "語者分離", "target": "話者分離" }
  ]
}

Limit: Up to 3000 entries per language. Exceeding the limit returns config_too_many_dict_entries; the details in the response identify which language exceeded it.

Note: The entry count directly affects translation workload and cost. A single translation only carries entries whose source word actually appears in that piece of text, up to 100 of them; beyond that, longer source words are kept first. Also, the more entries there are, the smaller the share that is reliably honored — an inherent limit that a higher cap does not change.

The previous format is still supported: the earlier array-of-entries format ([{ "source": ..., "translations": { "language code": ... } }]) is still accepted, with identical content and behavior. Existing integrations need no changes. The same dictionary sent in either format produces identical results.

On resume, the translation_dict returned in resume_ok matches the format you last sent — send the old format and you get the old format back; send the new one and you get the new one.

Note: Clearing the whole dictionary is not supported: an empty object is treated as not sending this setting at all. Clearing a single language is supported — send that language an empty array (e.g. {"en-US": []}).

Case sensitivity: case_sensitive is optional and defaults to false (case-insensitive). When set to true, the entry applies only where the source text matches source exactly, including case. The flag is per entry.

Note: The translation dictionary guides the model through prompting rather than literal substitution, so it is best-effort, not deterministic — the case flag is likewise a hint and is not guaranteed to be honored. Use fuzzy_correction when you need deterministic replacement.

Case-Flag Comparison

fuzzy_correction and translation_dict each have a case switch. Their field names are opposites, and so is the behavior their default value produces:

Both default to false, yet one means strict and the other means permissive. Do not share a single variable between them, and do not mirror one value onto the other — getting it wrong produces no error at all, only matching behavior opposite to what you intended.

Request Example (Recommended: Terminology Only)

{
  "type": "voice-translation",
  "data": {
    "action": "config",
    "terminology": {
      "zh-TW": [
        { "term": "語者分離" },
        { "term": "CVD製程" },
        { "term": "wafer良率" }
      ]
    }
  }
}

Request Example (Full Settings, Including Manual Correction Rules)

{
  "type": "voice-translation",
  "data": {
    "action": "config",
    "terminology": {
      "zh-TW": [
        { "term": "語者分離" },
        { "term": "即時轉錄" }
      ]
    },
    "fuzzy_correction": {
      "zh-TW": [
        { "correct": "語者分離", "incorrect": ["語這分離", "語者分力"] }
      ]
    },
    "translation_dict": {
      "en-US": [{ "source": "語者分離", "target": "Speaker Diarization" }]
    }
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "config_updated",
    "updated": ["terminology", "fuzzy_correction", "translation_dict"],
    "message": "Settings updated"
  }
}

Error Responses

Important: Your client must also listen for type: "error" messages — do not wait only for config_updated.

When the server rejects a configuration it sends a type: "error" message and notconfig_updated. An integration that waits only for config_updated will hang until its own timeout and appear as "the server never responded", even though the error was delivered and data.error_code already states the reason.

Voice Translation - audio (Send Audio)

Description

Sends audio data to the server for speech recognition. The audio must be Base64-encoded before sending.

Use Cases

• Continuously sending microphone audio
• Sending recorded audio segments

Request Parameters

Audio Format Requirements

PCM format (default):

WebM/Opus format:

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "audio",
    "payload": "Base64-encoded PCM audio data"
  }
}

Request Example (Multi-Channel Mode)

In multi-channel mode every frame must include channel_id:

{
  "type": "voice-translation",
  "data": {
    "action": "audio",
    "channel_id": 2,
    "payload": "Base64-encoded PCM audio data"
  }
}

Multi-channel mode notes: we recommend sending one frame every 100ms; every channel should keep sending audio even while silent. Multi-channel supports only the pcm format (16kHz / 16-bit / mono / little-endian).

Error Responses

Voice Translation - pause (Pause Translation)

Description

Pauses speech recognition processing. Audio received during the pause is buffered and continues to be processed after resuming. Two-way translation is an exception: audio received while paused is not kept; for a sentence in progress at the moment of pausing, the server first waits for the recognizer to finish the last part (usually about 1 second, at most about 3 seconds), sends it with is_final: true, and only then returns status: "paused". Billing continues while paused; see Pricing.

Use Cases

• The user steps away temporarily
• You need to pause recording

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "pause"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "status": "paused",
    "message": "Speech recognition paused"
  }
}

Error Responses

Multi-channel mode: while paused, every channel's audio file is still saved as usual, but no text is produced (no transcript is generated). Adding or disabling channels, changing a channel's language, or adjusting the speaking speed while paused returns channel_action_while_paused — call resume first, then operate.

Voice Translation - resume (Resume Translation)

Description

Resumes paused speech recognition processing.

Use Cases

• The user returns to continue
• You need to continue recording

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "resume"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "status": "live",
    "message": "Speech recognition resumed"
  }
}

Error Responses

Multi-channel mode: after resuming, speech from the paused period is transcribed retroactively (timestamps land at the moment the words were actually spoken, not after the resume). Retroactive transcription has a limit: a trailing 60 seconds shared across the whole session (under per_channel, split evenly across channels when there are several; under shared, the whole last 60 seconds, for all channels together, with speakers labeled); anything earlier is kept only in the audio file and does not enter the transcript. A sentence cut off at the moment of pausing may not appear in the transcript (same as single-stream behavior); when it is indeed not kept, resuming also sends segment_discarded (reason: "resumed") to identify it. On resume, each channel prepares recognition again and sends its own channel_status event (reason: "resumed").

Voice Translation - stop (Stop Translation)

Description

Stops speech recognition and ends the session. The server first waits for the recognizer to finish the last sentence (usually about 1 second, at most about 3 seconds) so that it is included in the transcript; if it does not arrive in time, the last interim result shown is used as that sentence. The system automatically uploads the audio file and transcript, and generates a summary (if configured; when the available credits cannot cover the summary fee, no summary is generated and summary_error is sent instead).

Use Cases

• The meeting ends
• Recording is complete

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "stop"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "status": "ended",
    "message": "Speech recognition stopped"
  }
}

Task Complete Event

This event is sent after the audio file and transcript have been uploaded:

{
  "type": "voice-translation",
  "data": {
    "action": "task_complete",
    "task_id": "550e8400-e29b-41d4-a716-446655440000",
    "message": "Task processing complete"
  }
}

Error Codes

Sending stop a second time returns this error rather than another success response. task_complete is sent only once, after the first successful stop, when the audio and transcript have finished uploading.

Voice Translation - retranslate (Retranslate)

Description

Retranslates a specified sentence, useful when the original text has been corrected and the translation needs to be updated.

Use Cases

• The user edits the original text and the translation needs updating
• Correcting recognition errors

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "retranslate",
    "sid": 1,
    "translation_languages": ["en-US"],
    "text": "The user-corrected original text"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "result",
    "translations": {
      "en-US": {
        "sid": 1,
        "text": "The new translation result",
        "is_final": true,
        "is_retranslation": true
      }
    }
  }
}

Error Responses

If the translation service returns a specific failure code (for example llm_content_filtered when the content cannot be translated), that code is returned as-is instead of being wrapped in retranslate_llm_failed.

Voice Translation - switch_language (Switch Language)

Description

Switches or adjusts translation languages while real-time translation is in progress. The behavior varies by recording type and the number of translation languages:

• General mode, single language (translation_languages has 1 entry): replaces the translation target language and automatically batch-retranslates all already-translated sentences.
• General mode, multiple languages (2 or more entries, v1.6.7): redefined as "add or remove a single language". The op parameter (add / remove) is required; omitting it returns a switch_language_op_required error. add automatically backfills existing sentences (same response sequence as single-language replace); remove returns a translation_language_removed event and keeps existing translations. See the WebSocket reference
• Two-way mode (conversation): switches the STT source language (spoken language); the translation target automatically switches to the other language.
• Multi-channel mode (multi_channel, v1.10.0): not supported — always returns multichannel_switch_language_not_allowed (including op: add / op: remove). Languages are bound to channels; use set_channel_language instead.

Use Cases

• Switching the translation target language
• A change in language needs mid-meeting
• Adding / removing a translation language mid-session in multi-language sessions (v1.6.7)

Request Parameters

Request Example (General Mode, Single-Language Replace)

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "translation_languages": ["ja-JP"]
  }
}

Request Example (Multi-Language Add / Remove, v1.6.7)

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "op": "add",
    "translation_languages": ["de-DE"]
  }
}

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "op": "remove",
    "translation_languages": ["ko-KR"]
  }
}

Request Example (Two-Way Mode)

Specify the target to switch to:

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "transcription_languages": ["en-US"]
  }
}

Automatic toggle (no parameters):

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language"
  }
}

Two-Way Mode Special Behavior:

• Two-way mode uses automatic language detection and usually does not require manually switching the language.
• switch_language only updates the internal preference state.
• After a successful switch, a language_switched event is returned (not a language_switch_start/done sequence).
• Switching to the same language returns a conversation_same_language warning.

Response Sequence (General Mode)

After switching the language, you receive the following events in order:

1. language_switch_start: notifies that the switch has begun

{
  "type": "voice-translation",
  "data": {
    "action": "language_switch_start",
    "translation_language": "ja-JP",
    "translation_languages": ["en-US", "ja-JP"],
    "total_segments": 15
  }
}

2. batch_retranslation (multiple): returns retranslation results sentence by sentence

{
  "type": "voice-translation",
  "data": {
    "action": "batch_retranslation",
    "sid": 3,
    "translations": {
      "ja-JP": {
        "sid": 3,
        "text": "今日はプロジェクトの進捗について話し合いましょう",
        "is_final": true,
        "is_retranslation": true
      }
    }
  }
}

3. language_switch_done: notifies that the switch is complete

{
  "type": "voice-translation",
  "data": {
    "action": "language_switch_done",
    "translation_language": "ja-JP",
    "translation_languages": ["en-US", "ja-JP"],
    "success_count": 15,
    "failed_count": 0
  }
}

Error Responses

Voice Translation - set_name (Set Recording Name)

Description

Sets the name while recording is in progress. After it is set, this name is used when the recording ends and will not be auto-generated.

Tip: You can also set an initial default name via the name parameter at start, but that name may still be overridden by the system when the session ends. If you need a fixed name, use set_name.

Use Cases

• Customizing the recording title after recording starts
• Overriding an auto-generated name or a previously set name

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "set_name",
    "name": "Product Planning Meeting"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "event": "name_set",
    "name": "Product Planning Meeting",
    "message": "Recording name updated"
  }
}

Compatibility note: For backward compatibility, the set_name success response keeps action: "status" (unchanged). New clients should identify a successful set_name via event: "name_set" (together with the name field). Relying on action: "status" to detect set_name success is deprecated and may be removed in a future version.

Error Responses

Voice Translation - rename_speaker (Globally Rename a Speaker)

Description

In multi-speaker diarization mode (multi_speaker), globally renames a speaker. All sentences using that speaker ID are updated in sync.

Also available in multi-channel mode (multi_channel): speaker IDs use the format channel_{N} (such as channel_1); speakers are registered at start / add_channel, so a channel's speaker can be renamed before they say anything. After renaming, the speaker_label in result events and the transcript both reflect the new label.

Use Cases

• Changing a system-assigned speaker ID (such as Guest-1) to a meaningful name (such as Manager Wang)
• Naming a newly recognized speaker during a meeting

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "rename_speaker",
    "speaker_id": "Guest-1",
    "new_label": "Manager Wang"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "speaker_renamed",
    "speaker_id": "Guest-1",
    "new_label": "Manager Wang",
    "affected_sids": [1, 3, 5, 8]
  }
}

Error Responses

Voice Translation - reassign_speaker (Change the Speaker of a Single Sentence)

Description

Changes the speaker identity of a specific sentence, assigning the sentence to an existing speaker.

Not applicable in multi-channel mode (multi_channel): speaker identity is determined by the physical channel (each sentence's speaker_id is its source channel), so reassigning a sentence to another channel is not supported.

Use Cases

• Correcting a speaker identity that the system recognized incorrectly
• Reassigning a sentence to another known speaker

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "reassign_speaker",
    "sid": 5,
    "target_speaker_id": "Guest-2"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "speaker_reassigned",
    "sid": 5,
    "old_speaker_id": "Guest-1",
    "new_speaker_id": "Guest-2",
    "new_speaker_label": "Lee Hsiao-hua"
  }
}

Error Responses