|
@@ -7,6 +7,7 @@ asr/
|
|
|
├── asr_agent/ # 服务端(Python)
|
|
├── asr_agent/ # 服务端(Python)
|
|
|
│ ├── dispatcher.py # HTTP Dispatcher — 多房间调度
|
|
│ ├── dispatcher.py # HTTP Dispatcher — 多房间调度
|
|
|
│ ├── worker.py # VAD → ASR → LLM → TTS 完整流水线
|
|
│ ├── worker.py # VAD → ASR → LLM → TTS 完整流水线
|
|
|
|
|
+│ ├── voiceprint.py # 声纹识别 + 对话记忆(Zvec + JSON)
|
|
|
│ ├── whisper_asr/ # ASR 核心引擎
|
|
│ ├── whisper_asr/ # ASR 核心引擎
|
|
|
│ │ ├── audio_processor.py # AudioBuffer 环形缓冲
|
|
│ │ ├── audio_processor.py # AudioBuffer 环形缓冲
|
|
|
│ │ ├── qwen_engine.py # Qwen3-ASR 本地引擎
|
|
│ │ ├── qwen_engine.py # Qwen3-ASR 本地引擎
|
|
@@ -111,6 +112,53 @@ sequenceDiagram
|
|
|
| **LLM** | Mimo v2.5 / vLLM | `LLM_PROVIDER` 切换;流式 SSE + `<think>` 过滤 |
|
|
| **LLM** | Mimo v2.5 / vLLM | `LLM_PROVIDER` 切换;流式 SSE + `<think>` 过滤 |
|
|
|
| **TTS** | Mimo v2.5 API | 24kHz PCM,并行生成 + 顺序入队播放 |
|
|
| **TTS** | Mimo v2.5 API | 24kHz PCM,并行生成 + 顺序入队播放 |
|
|
|
| **播放** | 单轨 `LocalAudioTrack` | `play_q` 队列 + `_player` 协程持续 drain |
|
|
| **播放** | 单轨 `LocalAudioTrack` | `play_q` 队列 + `_player` 协程持续 drain |
|
|
|
|
|
+| **声纹** | scipy MFCC + Zvec | 自动注册 / 识别说话人,识别结果注入 LLM 上下文 |
|
|
|
|
|
+| **记忆** | Zvec + JSON | 声纹向量存 Zvec;对话历史存 JSON,按用户隔离 |
|
|
|
|
|
+
|
|
|
|
|
+## 声纹识别
|
|
|
|
|
+
|
|
|
|
|
+```mermaid
|
|
|
|
|
+flowchart LR
|
|
|
|
|
+ A[音频输入] --> B[MFCC 特征提取<br/>120维]
|
|
|
|
|
+ B --> C[Zvec 向量检索<br/>余弦相似度]
|
|
|
|
|
+ C -->|匹配 >0.65| D[返回已知用户]
|
|
|
|
|
+ C -->|无匹配| E[自动注册新用户]
|
|
|
|
|
+ D --> F[注入 LLM 上下文]
|
|
|
|
|
+ E --> F
|
|
|
|
|
+ F --> G[个性化回复]
|
|
|
|
|
+
|
|
|
|
|
+ B --> H[对话记忆<br/>JSON 持久化]
|
|
|
|
|
+ H -->|同一说话人| I[恢复历史会话]
|
|
|
|
|
+ I --> F
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## 会话管理
|
|
|
|
|
+
|
|
|
|
|
+```mermaid
|
|
|
|
|
+sequenceDiagram
|
|
|
|
|
+ participant A as App
|
|
|
|
|
+ participant D as Dispatcher
|
|
|
|
|
+ participant W as Worker
|
|
|
|
|
+
|
|
|
|
|
+ Note over A,W: 首次连接
|
|
|
|
|
+ A->>D: POST /connect {identity}
|
|
|
|
|
+ D-->>A: {room: "room-xxx", token}
|
|
|
|
|
+ Note over A: 保存 room
|
|
|
|
|
+
|
|
|
|
|
+ Note over A,W: 暂停 → 继续
|
|
|
|
|
+ A->>D: POST /connect {identity, room: "room-xxx"}
|
|
|
|
|
+ D->>D: 检测 room-xxx 存活
|
|
|
|
|
+ D-->>A: {room: "room-xxx", token}
|
|
|
|
|
+ Note over A,W: 同房间 → Worker 保持 → 上下文不丢
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
|
|
+## 三层记忆系统
|
|
|
|
|
+
|
|
|
|
|
+| 层级 | 存储 | 键 | 生命周期 | 用途 |
|
|
|
|
|
+|------|------|-----|---------|------|
|
|
|
|
|
+| **房间会话** | `self.hist` (内存) | 房间号 | APP 暂停→继续 | 当前对话上下文 |
|
|
|
|
|
+| **声纹向量** | Zvec 嵌入式 DB | 声纹特征 | 永久 | 说话人识别 |
|
|
|
|
|
+| **对话记忆** | JSON 文件 | `speaker_id` | 永久 | 跨会话历史 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -131,6 +179,7 @@ python3 -m py_compile asr_agent/dispatcher.py
|
|
|
# 2. 上传文件到服务器
|
|
# 2. 上传文件到服务器
|
|
|
sshpass -p '123456' scp asr_agent/worker.py ubuntu@200.200.18.11:/tmp/asr_build/worker.py
|
|
sshpass -p '123456' scp asr_agent/worker.py ubuntu@200.200.18.11:/tmp/asr_build/worker.py
|
|
|
sshpass -p '123456' scp asr_agent/dispatcher.py ubuntu@200.200.18.11:/tmp/asr_build/dispatcher.py
|
|
sshpass -p '123456' scp asr_agent/dispatcher.py ubuntu@200.200.18.11:/tmp/asr_build/dispatcher.py
|
|
|
|
|
+sshpass -p '123456' scp asr_agent/voiceprint.py ubuntu@200.200.18.11:/tmp/asr_build/voiceprint.py
|
|
|
sshpass -p '123456' scp -r asr_agent/whisper_asr ubuntu@200.200.18.11:/tmp/asr_build/
|
|
sshpass -p '123456' scp -r asr_agent/whisper_asr ubuntu@200.200.18.11:/tmp/asr_build/
|
|
|
|
|
|
|
|
# 3. 服务器构建镜像
|
|
# 3. 服务器构建镜像
|
|
@@ -151,6 +200,7 @@ docker run -d --name asr-dispatcher --network host --restart unless-stopped \
|
|
|
-e LLM_PROVIDER=mimo \
|
|
-e LLM_PROVIDER=mimo \
|
|
|
-e ASR_PROVIDER=mimo \
|
|
-e ASR_PROVIDER=mimo \
|
|
|
-v /data/models:/data/models:ro \
|
|
-v /data/models:/data/models:ro \
|
|
|
|
|
+ -v /data/voiceprints:/data/voiceprints \
|
|
|
asr-dispatcher:v1
|
|
asr-dispatcher:v1
|
|
|
'
|
|
'
|
|
|
|
|
|
|
@@ -176,6 +226,8 @@ curl -s -X POST http://localhost:10005/connect -H "Content-Type: application/jso
|
|
|
| `MIMO_KEY` | — | Mimo API Key |
|
|
| `MIMO_KEY` | — | Mimo API Key |
|
|
|
| `MIMO_API_BASE` | `https://token-plan-cn.xiaomimimo.com/v1` | Mimo API 地址 |
|
|
| `MIMO_API_BASE` | `https://token-plan-cn.xiaomimimo.com/v1` | Mimo API 地址 |
|
|
|
| `VAD_MODEL_PATH` | `whisper_asr/silero_vad.onnx` | Silero VAD 模型路径 |
|
|
| `VAD_MODEL_PATH` | `whisper_asr/silero_vad.onnx` | Silero VAD 模型路径 |
|
|
|
|
|
+| `VP_DB_PATH` | `/data/voiceprints` | 声纹数据库目录 |
|
|
|
|
|
+| `VP_SIMILARITY_THRESHOLD` | `0.65` | 声纹匹配余弦相似度阈值 |
|
|
|
|
|
|
|
|
## 端口
|
|
## 端口
|
|
|
|
|
|
|
@@ -196,6 +248,16 @@ curl -s -X POST http://localhost:10005/connect -H "Content-Type: application/jso
|
|
|
5. App 断开 → Worker 退出 → 房间释放
|
|
5. App 断开 → Worker 退出 → 房间释放
|
|
|
```
|
|
```
|
|
|
|
|
|
|
|
|
|
+### 暂停 → 继续
|
|
|
|
|
+
|
|
|
|
|
+```
|
|
|
|
|
+1. App 暂停 → 保存 room 名称
|
|
|
|
|
+2. App 继续 → POST /connect {identity, room: "room-xxx"}
|
|
|
|
|
+3. Dispatcher 检测 room-xxx 是否存活
|
|
|
|
|
+4. 存活 → 返回相同 room,Worker 保持,上下文不丢
|
|
|
|
|
+5. 已释放 → 创建新 room + 新 Worker,声纹记忆恢复上下文
|
|
|
|
|
+```
|
|
|
|
|
+
|
|
|
## Flutter App 编译
|
|
## Flutter App 编译
|
|
|
|
|
|
|
|
```bash
|
|
```bash
|