Websocket Api

Note: This is a consolidated document. For detailed specifications, refer to the individual documents under reference/websocket/.

Note: The URL used in this document (vas-poc.vurbo.ai) is the planned deployment address. A separate notice will be issued after the official launch.

Table of Contents

1. Connection Info
2. Authentication
3. Message Format
4. Health - Heartbeat Service
5. Voice Translation - start
6. Voice Translation - config
7. Voice Translation - audio
8. Voice Translation - pause
9. Voice Translation - resume
10. Voice Translation - stop
11. Voice Translation - retranslate
12. Voice Translation - switch_language
13. Voice Translation - set_name
14. Voice Translation - rename_speaker
15. Voice Translation - reassign_speaker
16. Voice Translation - merge_speakers
17. Voice Translation - tts_play
18. Voice Translation - tts_stop
19. Voice Translation - tts_mode
20. Voice Translation - set_tts
21. Voice Translation - start_speaking
22. Voice Translation - stop_speaking
23. Voice Translation - switch_conversation_mode
24. Voice Translation - set_speaker_language
25. Voice Translation - broadcast_go_live
26. Voice Translation - broadcast_announcement
27. Voice Translation - set_standby_message
28. Response Events

Connection Info

Authentication

The VAS WebSocket uses a Ticket mechanism for authentication, passing a one-time Ticket via Sec-WebSocket-Protocol. For details, refer to Authentication.

Step 1: Obtain a Ticket

Exchange your API Key for a one-time Ticket via the REST API:

POST /api/v1/auth/ticket
X-API-Key: vas_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Response:

{
  "ticket": "aBcDeFgHiJkLmNoPqRsTuVwXyZ012345",
  "expires_in": 60
}

Step 2: Connect to the WebSocket using the Ticket

Place the Ticket into Sec-WebSocket-Protocol in the format ticket.{TICKET_VALUE}:

// Native browser support
const ws = new WebSocket('wss://vas-poc.vurbo.ai/ws', [`ticket.${ticket}`]);

ws.onopen = () => {
  console.log('Connected! Protocol:', ws.protocol);
  // Start using the WebSocket...
};

ws.onerror = (error) => {
  console.error('Connection failed:', error);
};

Node.js example:

const WebSocket = require('ws');

const ws = new WebSocket('wss://vas-poc.vurbo.ai/ws', [`ticket.${ticket}`]);

Ticket Characteristics

Ticket Error Codes

For the full API specification, refer to Auth Ticket API.

Message Format

All messages use a unified nested structure:

{
  "type": "service type",
  "data": { ... }
}

Service Types

Error Message Format

When an error occurs, the server returns a message with type: "error":

{
  "type": "error",
  "data": {
    "error_code": "auth_invalid_api_key",
    "severity": "fatal",
    "message": "Invalid API key",
    "context": "auth",
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-01-15T10:30:45.123Z"
  }
}

Sentence-level errors (such as a translation failure for one language of a sentence) additionally carry sid and details:

{
  "type": "error",
  "data": {
    "error_code": "llm_content_filtered",
    "severity": "warning",
    "message": "Content filtered",
    "context": "translation",
    "sid": 5,
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-01-15T10:30:45.123Z",
    "details": {
      "provider": "azure_openai",
      "translation_language": "ja"
    }
  }
}

Session-level translation service errors (escalated after consecutive failures reach a threshold) do not carry sid. The frontend should display a global notice but does not need to disconnect:

{
  "type": "error",
  "data": {
    "error_code": "translation_service_unavailable",
    "severity": "error",
    "message": "Translation service unavailable",
    "context": "translation",
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-01-15T10:30:45.123Z",
    "details": {
      "provider": "azure_openai",
      "last_error_code": "llm_provider_error",
      "fail_count": 5
    }
  }
}

For the full trigger rules (consecutive failure threshold, error code classification), refer to the translation_service_unavailable section in Error Code Reference.

Single-Message Error (per-message panic recovered)

When the server encounters an internal error (panic) while handling a single WebSocket message (such as set_name, switch_language, tts_play, etc.), it returns internal_error. This error indicates only that the specific message failed to process; the connection is not terminated. The frontend should keep the connection open and may retry the operation:

{
  "type": "error",
  "data": {
    "error_code": "internal_error",
    "severity": "error",
    "message": "Internal server error",
    "context": "general",
    "request_id": "req_abc123xyz789",
    "timestamp": "2026-05-08T10:30:45.123Z",
    "details": {
      "message_type": "voice-translation",
      "action": "set_name"
    }
  }
}

details Fields

What the Frontend Should Do

1. Keep the WebSocket connection open: Do not call ws.close(), navigate away, or return to a history page because of this error. The recording is still in progress.
2. Decide on follow-up handling based on details.action:

   Scenario	Recommended Action
   Idempotent operations such as set_name / switch_language / tts_mode / config	Simply resend the same message. These operations use a "last write wins" approach, so retrying has no side effects.
   tts_play / tts_stop / retranslate	Usually safe to retry directly. If the user is waiting for TTS playback, consider showing a transient toast indicating the retry is in progress.
   speaker.rename / speaker.merge	Before retrying, use the REST API (speakers) to confirm the current DB state and avoid duplicate operations (for example, the rename already succeeded and only the response frame failed).
   details.action is absent	The server panicked after parsing the message payload and cannot infer the specific operation. The frontend can infer it from "the most recent message the user sent," or display a generic error message such as "Operation failed, please retry."

3. User experience: Show a transient toast / inline error. Do not interrupt the user flow with a modal or a redirect.
4. Telemetry / reporting: Report request_id + details to your frontend error tracking (Sentry, Datadog, etc.) to make it easier to correlate with backend logs during troubleshooting.

What Will Not Happen (Guarantees)

• The recording will not be interrupted: segment_uploaded, result, origin, and other messages keep arriving.
• The connection will not be actively closed by the server.
• The session state will not be reset (session_id stays the same).
• State already written to the DB will not be rolled back (for example, if set_name was written to the DB successfully and only the response frame failed, the name still takes effect).

Client Handling Example

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);
  if (msg.type !== 'error') {
    handleNormalMessage(msg);
    return;
  }

  const { error_code, severity, request_id, details } = msg.data;

  // Single-message panic: keep the connection, decide whether to retry based on action
  if (error_code === 'internal_error') {
    console.warn('[ws] message handler panic recovered', {
      request_id,
      message_type: details?.message_type,
      action: details?.action,
    });
    showTransientToast(`Failed to process "${details?.action ?? 'operation'}", please retry`);
    // Note: do not call ws.close() and do not navigate away from the current page
    return;
  }

  // Handle other errors with your existing logic (only fatal errors require disconnecting)
  handleErrorBySeverity(severity, msg.data);
};

For the full list of error codes, refer to Error Code Reference.

Health (Heartbeat Service)

Description

Used to confirm that the WebSocket connection is healthy. We recommend sending a ping every 30 seconds; if no pong is received, treat the connection as dropped and reconnect.

Use Cases

• Maintaining a long-lived connection
• Detecting connection status
• Preventing connection timeouts

Request - Ping

{
  "type": "health",
  "data": {
    "action": "ping"
  }
}

Response - Pong

{
  "type": "health",
  "data": {
    "action": "pong"
  }
}

Voice Translation - start (Start Voice Translation)

Description

Starts a new voice translation session and begins processing audio according to the configured parameters.

Use Cases

• Starting a meeting record
• Starting real-time translation
• Starting a voice memo

Request Parameters

Recording Type Descriptions

Request Example (Basic)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "realtime_translation": false,
    "type": "transcribe",
    "audio_format": "pcm",
    "summary_template": "meeting",
    "options": {
      "speaking_speed": "normal",
      "segmentation_mode": "auto",
      "profanity_handling": "mask"
    }
  }
}

Request Example (Initial Default Name)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "type": "transcribe",
    "audio_format": "pcm",
    "summary_template": "meeting",
    "name": "Product Planning Meeting"
  }
}

Recording Name Rules

Note: The name in start is the initial default name; the system may still override it when the session ends. If you need a fixed name, use set_name.

Default name format (fixed English):

N is the sequential number for that user's recordings of the same type. Name priority: user > llm > default. Once the user sets a name, the system will not override it when the session ends.

Request Example (With TTS)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "realtime_translation": true,
    "type": "transcribe",
    "tts_enabled": true,
    "tts_language": "en-US",
    "tts_voice": "en-US-JennyNeural",
    "tts_mode": "sync"
  }
}

Request Example (Two-Way Mode - Auto-Detect)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "conversation",
    "transcription_languages": ["zh-TW", "en-US"],
    "active_language": "zh-TW",
    "audio_format": "pcm",
    "realtime_translation": true,
    "speakers": [
      { "id": 1, "language": "zh-TW" },
      { "id": 2, "language": "en-US" }
    ],
    "tts_config": {
      "zh-TW": { "voice": "zh-TW-HsiaoChenNeural", "speaking_rate": 1.0 },
      "en-US": { "voice": "en-US-JennyNeural", "speaking_rate": 1.0 }
    }
  }
}

Request Example (Two-Way Mode - Manual Mode)

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "conversation",
    "transcription_languages": ["zh-TW", "en-US"],
    "conversation_mode": "manual",
    "audio_format": "pcm",
    "realtime_translation": true,
    "speakers": [
      { "id": 1, "language": "zh-TW" },
      { "id": 2, "language": "en-US" }
    ],
    "tts_config": {
      "zh-TW": { "voice": "zh-TW-HsiaoChenNeural", "speaking_rate": 1.0 },
      "en-US": { "voice": "en-US-JennyNeural", "speaking_rate": 1.0 }
    }
  }
}

Request Example (Custom Summary Prompt - custom mode)

In mode=custom, your summary_prompt content completely replaces the built-in template rules, and the backend already adds prompt injection protection. The summary_prompt_slug is metadata for your own identification (stored in the backend record) and does not enter the prompt content.

If you want to keep the built-in template and add your own supplemental instructions afterward, use summary_mode=builtin + summary_template=<slug> + summary_prompt=<supplemental instructions> instead (in builtin mode, summary_prompt is treated as supplemental and appended after the built-in template).

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "transcription_languages": ["zh-TW"],
    "translation_languages": ["en-US"],
    "type": "transcribe",
    "audio_format": "pcm",
    "summary_language": "zh-TW",
    "summary_mode": "custom",
    "summary_prompt": "You are a meeting-minutes assistant. List every amount and committed date discussed in bullet points, and note the responsible person for each.",
    "summary_prompt_slug": "client_x_finance_v3",
    "summary_plain_text": false
  }
}

Important — How to Retrieve the Summary Result: In WebSocket mode, summaries are non-streaming by design; final_content is not pushed back via a WebSocket event (the summary_done event only signals completion and does not contain the content). The client must retrieve it afterward over HTTP:

1. After receiving the summary_done event, call GET /api/v1/sse/history/transcribe/{taskId} to retrieve the summary (the init_summary event carries a top-level summary plain string + summary_mode / summary_template / summary_plain_text / summary_prompt_snapshot + the two content-filter fallback audit fields summary_fallback_level / summary_dropped_segments added in v1.5.5).
2. Or query the summary_mode / summary_template / summary_prompt_slug columns of the recordings table via the REST API.

v1.5.5 Content-Filter Automatic Downgrade: If your prompt or transcript content triggers the LLM service's content filter, the system automatically downgrades (standard mode → neutral mode → segment-omission mode). The summary_fallback_level field of the summary_done event (value 2 or 3; omitted when standard mode succeeds directly) tells the client which path was actually taken, so the frontend can display hints such as "neutral mode in use" / "N segments omitted." See reference/websocket/events.md – summary_done and the V1.5.5 changelog.

Two-Way Mode Special Rules:

speakers Field Descriptions:

conversation_mode Descriptions:

Broadcast Mode Description (type: "broadcast")

In broadcast mode, the language settings are automatically obtained from the broadcast channel settings and do not need to be sent in the WebSocket message.

Required parameters:

Optional parameters (override the broadcast channel settings):

Auto-configured parameters (can be omitted):

• transcription_languages: read automatically from the broadcast settings
• translation_languages: read automatically from the broadcast settings
• realtime_translation: enabled by default in broadcast mode
• summary_template: read automatically from the broadcast settings (the value passed via WebSocket takes precedence)
• summary_language: read automatically from the broadcast settings (the value passed via WebSocket takes precedence)

Broadcast Phase Descriptions:

Standby phase purpose: Lets the host warm up STT/translation before going live, confirm that equipment is working, and then switch to the live phase.

Broadcast Mode Request Example:

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm"
  }
}

Broadcast Mode Request Example (Standby Phase + Override Summary Template):

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm",
    "broadcast_phase": "standby",
    "standby_message": "The talk is about to begin, please wait...",
    "summary_template": "lecture"
  }
}

Summary template priority: The value passed in the WebSocket start > the default set when the broadcast channel was created. If neither is set, no summary is automatically generated.

Broadcast Mode TTS Settings (tts_config):

Use the tts_config parameter to specify which translation languages should produce TTS audio for viewers.

{
  "type": "voice-translation",
  "data": {
    "action": "start",
    "type": "broadcast",
    "broadcast_token": "a3f9",
    "audio_format": "pcm",
    "tts_config": {
      "en-US": {
        "voice": "en-US-JennyNeural",
        "speaking_rate": 1.0
      },
      "ja-JP": {
        "voice": "ja-JP-NanamiNeural",
        "speaking_rate": 1.0
      }
    }
  }
}

Note:

• TTS languages must be valid languages in translation_languages; invalid languages are automatically ignored.
• The host (WebSocket) does not receive TTS audio; only SSE viewers receive the tts_ready event.
• TTS is sent only during the live phase; nothing is sent during the standby phase.

TTS Playback Mode Descriptions

Success Response

After a successful start, a session_started event is returned containing complete session initialization info.

General recordings (transcribe / conversation / record):

{
  "type": "voice-translation",
  "data": {
    "action": "session_started",
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "recording_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "recording_type": "transcribe",
    "recognition_mode": "single",
    "message": "Speech recognition started"
  }
}

Broadcast mode (broadcast):

{
  "type": "voice-translation",
  "data": {
    "action": "session_started",
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "recording_id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "recording_type": "broadcast",
    "recognition_mode": "multi_speaker",
    "phase": "standby",
    "viewer_count": 0,
    "queue_count": 0,
    "peak_viewers": 0,
    "total_viewers": 0,
    "message": "Speech recognition started"
  }
}

Error Responses

Voice Translation - config (Set Terminology / Correction Rules)

Description

Before or during recording, pass in terminology, fuzzy-word correction rules, and translation dictionary settings. These settings improve STT accuracy, fix homophone errors, and ensure translation consistency.

Auto-generated correction rules: When terminology is passed in, the system automatically generates fuzzy-word correction rules for each term (homophones, near-homophones, Traditional/Simplified variants). The frontend does not need to define fuzzy_correction manually, greatly simplifying the setup process.

Use Cases

• Pass in professional terminology (Phrase List) before recording starts
• Set fuzzy-word correction rules (homophone correction) - optional, the system generates them automatically
• Set a translation dictionary (ensure consistent terminology translation)

Timing

Note: When you update terminology during recording, the new terms automatically take effect at the next recognition turn boundary, with no need to reconnect. The response includes a terminology_effective: "next_turn" field as a hint.

Request Parameters

Note: At least one setting item must be provided.

Terminology Format (terminology)

Keyed by language code, with an array of terms as the value:

{
  "zh-TW": [
    { "term": "語者分離", "boost": 1.5 },
    { "term": "WebSocket", "boost": 2.0 }
  ],
  "en-US": [
    { "term": "diarization", "boost": 1.5 }
  ]
}

Limit: Up to 500 terms per language.

Fuzzy-Word Correction Format (fuzzy_correction)

Note: This field usually does not need to be set manually. The system automatically generates correction rules based on terminology. Use it only when you need custom special rules.

Keyed by language code, with an array of correction rules as the value:

{
  "zh-TW": [
    { "correct": "語者分離", "incorrect": ["語這分離", "語者分力"] }
  ]
}

Auto-Generated Correction Rule Description

When terminology is passed in, the system automatically generates fuzzy-word correction rules for each term:

Mixed Chinese-English term support: For mixed terms like "CVD製程," the system generates variants only for the Chinese portion and leaves the English unchanged.

Translation Dictionary Format (translation_dict)

Use an array of entries directly:

[
  {
    "source": "語者分離",
    "translations": {
      "en-US": "Speaker Diarization",
      "ja-JP": "話者分離"
    }
  }
]

Limit: We recommend no more than 50 entries (to avoid degrading processing performance).

Request Example (Recommended: Terminology Only)

{
  "type": "voice-translation",
  "data": {
    "action": "config",
    "terminology": {
      "zh-TW": [
        { "term": "語者分離", "boost": 1.5 },
        { "term": "CVD製程", "boost": 1.5 },
        { "term": "wafer良率", "boost": 1.5 }
      ]
    }
  }
}

Request Example (Full Settings, Including Manual Correction Rules)

{
  "type": "voice-translation",
  "data": {
    "action": "config",
    "terminology": {
      "zh-TW": [
        { "term": "語者分離", "boost": 1.5 },
        { "term": "即時轉錄", "boost": 1.5 }
      ]
    },
    "fuzzy_correction": {
      "zh-TW": [
        { "correct": "語者分離", "incorrect": ["語這分離", "語者分力"] }
      ]
    },
    "translation_dict": [
      { "source": "語者分離", "translations": { "en-US": "Speaker Diarization" } }
    ]
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "config_updated",
    "updated": ["terminology", "fuzzy_correction", "translation_dict"],
    "message": "Settings updated"
  }
}

Error Responses

Voice Translation - audio (Send Audio)

Description

Sends audio data to the server for speech recognition. The audio must be Base64-encoded before sending.

Use Cases

• Continuously sending microphone audio
• Sending recorded audio segments

Request Parameters

Audio Format Requirements

PCM format (default):

WebM/Opus format:

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "audio",
    "payload": "Base64-encoded PCM audio data"
  }
}

Error Responses

Voice Translation - pause (Pause Translation)

Description

Pauses speech recognition processing. Audio received during the pause is buffered and continues to be processed after resuming.

Use Cases

• The user steps away temporarily
• You need to pause recording

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "pause"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "message": "Speech recognition paused"
  }
}

Error Responses

Voice Translation - resume (Resume Translation)

Description

Resumes paused speech recognition processing.

Use Cases

• The user returns to continue
• You need to continue recording

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "resume"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "message": "Speech recognition resumed"
  }
}

Error Responses

Voice Translation - stop (Stop Translation)

Description

Stops speech recognition and ends the session. The system automatically uploads the audio file and transcript, and generates a summary (if configured).

Use Cases

• The meeting ends
• Recording is complete

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "stop"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "message": "Speech recognition stopped"
  }
}

Task Complete Event

This event is sent after the audio file and transcript have been uploaded:

{
  "type": "voice-translation",
  "data": {
    "action": "task_complete",
    "task_id": "550e8400-e29b-41d4-a716-446655440000",
    "message": "Task processing complete"
  }
}

Voice Translation - retranslate (Retranslate)

Description

Retranslates a specified sentence, useful when the original text has been corrected and the translation needs to be updated.

Use Cases

• The user edits the original text and the translation needs updating
• Correcting recognition errors

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "retranslate",
    "sid": 1,
    "translation_languages": ["en-US"],
    "text": "The user-corrected original text"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "result",
    "translations": {
      "en-US": {
        "sid": 1,
        "text": "The new translation result",
        "is_final": true,
        "is_retranslation": true
      }
    }
  }
}

Error Responses

Voice Translation - switch_language (Switch Language)

Description

Switches the language while real-time translation is in progress. The behavior varies by recording type:

• General mode (transcribe, etc.): switches the translation target language and automatically batch-retranslates all already-translated sentences.
• Two-way mode (conversation): switches the STT source language (spoken language); the translation target automatically switches to the other language.

Use Cases

• Switching the translation target language
• A change in language needs mid-meeting

Request Parameters

Request Example (General Mode)

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "translation_languages": ["ja-JP"]
  }
}

Request Example (Two-Way Mode)

Specify the target to switch to:

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language",
    "transcription_languages": ["en-US"]
  }
}

Automatic toggle (no parameters):

{
  "type": "voice-translation",
  "data": {
    "action": "switch_language"
  }
}

Two-Way Mode Special Behavior:

• Two-way mode uses automatic language detection and usually does not require manually switching the language.
• switch_language only updates the internal preference state.
• After a successful switch, a language_switched event is returned (not a language_switch_start/done sequence).
• Switching to the same language returns a conversation_same_language warning.

Response Sequence (General Mode)

After switching the language, you receive the following events in order:

1. language_switch_start: notifies that the switch has begun

{
  "type": "voice-translation",
  "data": {
    "action": "language_switch_start",
    "translation_language": "ja-JP",
    "total_segments": 15,
    "message": "Starting language switch and retranslation"
  }
}

2. batch_retranslation (multiple): returns retranslation results sentence by sentence

{
  "type": "voice-translation",
  "data": {
    "action": "batch_retranslation",
    "sid": 3,
    "translations": {
      "ja-JP": {
        "sid": 3,
        "text": "今日はプロジェクトの進捗について話し合いましょう",
        "is_final": true,
        "is_retranslation": true
      }
    }
  }
}

3. language_switch_done: notifies that the switch is complete

{
  "type": "voice-translation",
  "data": {
    "action": "language_switch_done",
    "translation_language": "ja-JP",
    "success_count": 15,
    "failed_count": 0,
    "message": "Language switch complete"
  }
}

Error Responses

Voice Translation - set_name (Set Recording Name)

Description

Sets the name while recording is in progress. After it is set, this name is used when the recording ends and will not be auto-generated.

Tip: You can also set an initial default name via the name parameter at start, but that name may still be overridden by the system when the session ends. If you need a fixed name, use set_name.

Use Cases

• Customizing the recording title after recording starts
• Overriding an auto-generated name or a previously set name

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "set_name",
    "name": "Product Planning Meeting"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "message": "Recording name updated"
  }
}

Error Responses

Voice Translation - rename_speaker (Globally Rename a Speaker)

Description

In multi-speaker diarization mode (multi_speaker), globally renames a speaker. All sentences using that speaker ID are updated in sync.

Use Cases

• Changing a system-assigned speaker ID (such as Guest-1) to a meaningful name (such as Manager Wang)
• Naming a newly recognized speaker during a meeting

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "rename_speaker",
    "speaker_id": "Guest-1",
    "new_label": "Manager Wang"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "speaker_renamed",
    "speaker_id": "Guest-1",
    "new_label": "Manager Wang",
    "affected_sids": [1, 3, 5, 8]
  }
}

Error Responses

Voice Translation - reassign_speaker (Change the Speaker of a Single Sentence)

Description

Changes the speaker identity (OriginalSpeakerID) of a specific sentence, assigning the sentence to an existing speaker.

Use Cases

• Correcting a speaker identity that the system recognized incorrectly
• Reassigning a sentence to another known speaker

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "reassign_speaker",
    "sid": 5,
    "target_speaker_id": "Guest-2"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "speaker_reassigned",
    "sid": 5,
    "old_speaker_id": "Guest-1",
    "new_speaker_id": "Guest-2",
    "new_speaker_label": "Lee Hsiao-hua"
  }
}

Error Responses

Voice Translation - merge_speakers (Merge Speakers)

Description

Merges all sentences of one speaker into another speaker. After the merge, future recognition results for that speaker are also automatically converted to the target speaker.

Use Cases

• The speech recognition engine sometimes misidentifies the same person's voice as multiple speakers (for example, Guest-1 and Guest-2 are actually the same person)
• Use this feature to merge all of Guest-2's sentences into Guest-1
• After the merge, future Guest-2 recognition results are automatically displayed as Guest-1

Difference from reassign_speaker

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "merge_speakers",
    "source_speaker_id": "Guest-2",
    "target_speaker_id": "Guest-1"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "speakers_merged",
    "source_speaker_id": "Guest-2",
    "target_speaker_id": "Guest-1",
    "target_speaker_label": "Manager Wang",
    "affected_sids": [3, 5, 7]
  }
}

Error Responses

Voice Translation - tts_play (Play TTS)

Description

In async mode, manually plays the TTS audio for a specified sentence.

Use Cases

• The user selects a specific sentence for TTS playback
• Playing multiple consecutive sentences

Request Parameters

Note: The maximum value of length is controlled by the backend environment variable TTS_SSE_MAX_LENGTH (default 20).

Request Example (Single Sentence)

{
  "type": "voice-translation",
  "data": {
    "action": "tts_play",
    "sid": 5
  }
}

Request Example (Multiple Sentences)

{
  "type": "voice-translation",
  "data": {
    "action": "tts_play",
    "sid": 5,
    "length": 3
  }
}

Error Responses

Voice Translation - tts_stop (Stop TTS)

Description

Stops the TTS audio that is currently playing.

Use Cases

• The user manually stops TTS playback
• Stopping the current playback before switching to another sentence

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "tts_stop"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "message": "TTS playback stopped"
  }
}

Voice Translation - tts_mode (Switch TTS Mode)

Description

Switches the TTS playback mode (synchronous/asynchronous) while recording is in progress.

Use Cases

• Switching from automatic playback to manual control
• Switching from manual control to automatic playback

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "tts_mode",
    "tts_mode": "async"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "tts_mode_changed",
    "tts_mode": "async"
  }
}

Error Responses

Voice Translation - set_tts (Two-Way TTS Settings)

Description

While a two-way mode (conversation) recording is in progress, dynamically toggles TTS on/off or updates the TTS voice settings. Available only in two-way mode.

Use Cases

• Turning the TTS audio response off/on mid-conversation in two-way mode
• Changing the TTS voice or speaking rate for a specific language

Request Parameters

Note: At least one of tts_enabled and tts_config must be provided. tts_config updates only the settings for the specified languages; unspecified languages remain unchanged.

Request Example (Disable TTS)

{
  "type": "voice-translation",
  "data": {
    "action": "set_tts",
    "tts_enabled": false
  }
}

Request Example (Update Voice Settings)

{
  "type": "voice-translation",
  "data": {
    "action": "set_tts",
    "tts_enabled": true,
    "tts_config": {
      "en-US": {
        "voice": "en-US-GuyNeural",
        "speaking_rate": 1.2
      }
    }
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "tts_updated",
    "tts_enabled": true,
    "tts_config": {
      "zh-TW": { "voice": "zh-TW-HsiaoChenNeural", "speaking_rate": 1.0 },
      "en-US": { "voice": "en-US-GuyNeural", "speaking_rate": 1.2 }
    }
  }
}

Error Responses

Voice Translation - start_speaking (Start Speaking / Manual Mode)

Description

In two-way manual mode (conversation_mode: "manual"), notifies the system that the user has started speaking. From this point on, audio is sent to STT for recognition, and all recognition results accumulate into a single sentence (no automatic segmentation).

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "start_speaking",
    "speaker": 1
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "status",
    "message": "Speaking started"
  }
}

Error Responses

Voice Translation - stop_speaking (Stop Speaking / Manual Mode)

Description

In two-way manual mode, notifies the system that the user has stopped speaking. The system merges the recognition results accumulated during the period into a single complete sentence and performs translation and TTS synthesis.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "stop_speaking"
  }
}

Success Response

After stopping speaking, the system sends a complete result event (containing origin and translations):

{
  "type": "voice-translation",
  "data": {
    "action": "result",
    "origin": {
      "sid": 1,
      "language": "zh-TW",
      "text": "The complete sentence merged from all recognition during this period",
      "is_final": true,
      "speaker_id": "Speaker-1",
      "start_time": "00:05"
    },
    "translations": {
      "en-US": {
        "sid": 1,
        "text": "The complete merged sentence from this speaking period",
        "is_final": true
      }
    }
  }
}

Error Responses

Voice Translation - switch_conversation_mode (Switch Conversation Mode)

Description

While two-way mode is in progress, switches between auto-detect mode (auto) and manual mode (manual). If the user is currently speaking when the switch happens, speaking is ended automatically.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "switch_conversation_mode",
    "conversation_mode": "manual"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "conversation_mode_changed",
    "conversation_mode": "manual"
  }
}

Error Responses

Voice Translation - set_speaker_language (Set User Language)

Description

While two-way mode is in progress, changes a specified user's language in real time. The system rebuilds the STT connection to accommodate the new language, and the translation target is also updated automatically. The transcript content before the change keeps its original language, and the timestamp continues to count without resetting.

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "set_speaker_language",
    "speaker": 1,
    "language": "ja-JP"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "speaker_language_changed",
    "speaker_language_map": {
      "1": "ja-JP",
      "2": "en-US"
    }
  }
}

Error Responses

Voice Translation - broadcast_go_live (Switch to the Live Phase)

Description

Switches from the broadcast standby phase (standby) to the live phase (live). After switching, STT/translation results begin broadcasting to viewers and start being written to the transcript.

Use Cases

• The host confirms the equipment is working and starts the official broadcast
• Switching from the warm-up phase to live streaming

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "broadcast_go_live"
  }
}

Success Response

{
  "type": "voice-translation",
  "data": {
    "action": "broadcast_phase_changed",
    "phase": "live",
    "message": "Broadcast started"
  }
}

Error Responses

Note: If already in the live phase, a status message "Broadcast is already in progress" is returned and is not treated as an error.

Voice Translation - broadcast_announcement (Send an Announcement)

Description

The host sends a custom message announcement to all viewers. Viewers receive an announcement event via SSE. The announcement message is automatically translated into all translation languages, and the SSE event viewers receive includes a translations field.

Use Cases

• Notifying viewers that the meeting is about to end
• Sending an important reminder or announcement
• One-way communication with viewers

Request Parameters

Request Example

{
  "type": "voice-translation",
  "data": {
    "action": "broadcast_announcement",
    "message": "The meeting will end in 5 minutes"
  }
}