|
|
@@ -25,82 +25,81 @@ asr/
|
|
|
|
|
|
## 端到端数据流
|
|
|
|
|
|
-```
|
|
|
-App 麦克风 ──Opus──► LiveKit Room
|
|
|
- │
|
|
|
- Worker (auto_subscribe)
|
|
|
- │
|
|
|
- ┌─ AudioStream (16kHz PCM) ─┐
|
|
|
- │ │
|
|
|
- Silero VAD AudioBuffer
|
|
|
- (ONNX 双阈值) (环形缓冲)
|
|
|
- │ │
|
|
|
- 语音开始/结束 │
|
|
|
- │ │
|
|
|
- └── VAD END ──► ASR ◄───────┘
|
|
|
- │
|
|
|
- Mimo ASR API
|
|
|
- (或 Qwen3 本地)
|
|
|
- │
|
|
|
- transcription
|
|
|
- │
|
|
|
- LLM (流式 SSE)
|
|
|
- Mimo v2.5 / vLLM
|
|
|
- │
|
|
|
- 分段 + TTS (并发)
|
|
|
- Mimo TTS API (24kHz)
|
|
|
- │
|
|
|
- play_q 顺序入队
|
|
|
- │
|
|
|
- _player 单轨播放
|
|
|
- │
|
|
|
- LiveKit AudioTrack
|
|
|
- │
|
|
|
- App 扬声器播放
|
|
|
+```mermaid
|
|
|
+flowchart TD
|
|
|
+ A[App 麦克风] -->|Opus| B[LiveKit Room]
|
|
|
+ B -->|auto_subscribe| C[Worker]
|
|
|
+
|
|
|
+ C --> D[AudioStream<br/>16kHz PCM]
|
|
|
+ D --> E[Silero VAD<br/>ONNX 双阈值]
|
|
|
+ D --> F[AudioBuffer<br/>环形缓冲]
|
|
|
+
|
|
|
+ E -->|语音结束| G[ASR]
|
|
|
+ F --> G
|
|
|
+
|
|
|
+ G -->|Mimo API / Qwen3| H[transcription]
|
|
|
+ H --> I[LLM 流式 SSE<br/>Mimo v2.5 / vLLM]
|
|
|
+ I -->|分段| J[TTS 并发生成<br/>Mimo API 24kHz]
|
|
|
+ J -->|play_q 顺序入队| K[_player 单轨播放]
|
|
|
+ K -->|AudioTrack| B
|
|
|
+ B -->|Opus| A
|
|
|
```
|
|
|
|
|
|
## 服务部署架构
|
|
|
|
|
|
-```
|
|
|
- ┌──────────────────────────┐
|
|
|
- │ 36.152.142.37 (外网) │
|
|
|
- └──────┬────────┬──────────┘
|
|
|
- │ │
|
|
|
- :10005 │ │ :10003
|
|
|
- │ │
|
|
|
- ┌──────▼──┐ ┌──▼──────────┐
|
|
|
- │ nginx │ │ nginx │
|
|
|
- │ HTTP │ │ WebSocket │
|
|
|
- └────┬───┘ └──┬───────────┘
|
|
|
- │ │
|
|
|
- :9100 │ │ :7880
|
|
|
- │ │
|
|
|
- ┌──────────▼──┐ ┌──▼───────────┐
|
|
|
- │ Dispatcher │ │ LiveKit │
|
|
|
- │ HTTP API │ │ Server │
|
|
|
- └──────┬──────┘ └──┬───────────┘
|
|
|
- │ │
|
|
|
- POST /connect Worker 加入房间
|
|
|
- │ │
|
|
|
- 创建独立房间 同一房间通信
|
|
|
- │ │
|
|
|
- 返回 token │
|
|
|
- │ │
|
|
|
- └──────┬───────────┘
|
|
|
- │
|
|
|
- App 连接
|
|
|
+```mermaid
|
|
|
+flowchart TD
|
|
|
+ subgraph External[36.152.142.37 外网]
|
|
|
+ P10005[":10005"]
|
|
|
+ P10003[":10003"]
|
|
|
+ end
|
|
|
+
|
|
|
+ subgraph Nginx[nginx]
|
|
|
+ N1[HTTP Proxy]
|
|
|
+ N2[WebSocket Proxy]
|
|
|
+ end
|
|
|
+
|
|
|
+ subgraph Services[内网服务]
|
|
|
+ D[Dispatcher<br/>:9100]
|
|
|
+ LK[LiveKit Server<br/>:7880]
|
|
|
+ end
|
|
|
+
|
|
|
+ P10005 --> N1
|
|
|
+ P10003 --> N2
|
|
|
+ N1 -->|proxy_pass| D
|
|
|
+ N2 -->|proxy_pass| LK
|
|
|
+
|
|
|
+ A2[App] -->|POST /connect| P10005
|
|
|
+ A2 -->|WebSocket| P10003
|
|
|
+
|
|
|
+ D -->|启动 Worker| W[Worker 进程]
|
|
|
+ W -->|加入房间| LK
|
|
|
+ A2 -->|同一房间| LK
|
|
|
```
|
|
|
|
|
|
## 多用户隔离
|
|
|
|
|
|
-每个 App 调用 `POST /connect` 获取**独立房间**：
|
|
|
-
|
|
|
-- App → Dispatcher `/connect` → 返回 `{room, url, token}`
|
|
|
-- Token 带 `roomCreate: true`，App 连接时自动创建房间
|
|
|
-- Dispatcher 为每个房间启动一个 Worker 进程
|
|
|
-- Worker 处理该房间的 VAD → ASR → LLM → TTS
|
|
|
-- App 断开 → `_on_off` 检测房间空 → Worker 退出
|
|
|
-- 多 App 之间完全隔离，互不干扰
|
|
|
+```mermaid
|
|
|
+sequenceDiagram
|
|
|
+ participant A as App A
|
|
|
+ participant D as Dispatcher :9100
|
|
|
+ participant LK as LiveKit
|
|
|
+ participant W as Worker
|
|
|
+
|
|
|
+ A->>D: POST /connect
|
|
|
+ D->>D: 创建房间 room-xxx
|
|
|
+ D->>W: spawn_worker(room-xxx)
|
|
|
+ D-->>A: {room, url, token}
|
|
|
+
|
|
|
+ A->>LK: WebSocket connect (room-xxx)
|
|
|
+ W->>LK: connect (room-xxx)
|
|
|
+
|
|
|
+ Note over A,W: A 说话 → VAD → ASR → LLM → TTS
|
|
|
+
|
|
|
+ A->>LK: disconnect
|
|
|
+ LK->>W: participant_disconnected
|
|
|
+ W->>W: 检测房间空 → exit
|
|
|
+ W-->>LK: disconnect
|
|
|
|
|
|
## Worker 流水线详解
|
|
|
|
