Any-to-Any
Transformers
ONNX
Safetensors
English
Chinese
multimodal
audio
video
speech
streaming
full-duplex
long-video
custom-code
Instructions to use inclusionAI/Realtime-Venus with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use inclusionAI/Realtime-Venus with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("inclusionAI/Realtime-Venus", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Realtime-Venus local preparation commited on
Commit ·
7b362f0
1
Parent(s): cbf2133
Refine public model documentation
Browse files- README.md +108 -38
- README_zh.md +108 -38
README.md
CHANGED
|
@@ -131,11 +131,35 @@ new output directory when repeating an experiment.
|
|
| 131 |
|
| 132 |
## 6. 🛠️ Installation
|
| 133 |
|
| 134 |
-
|
| 135 |
|
| 136 |
-
|
| 137 |
|
| 138 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
|
| 140 |
```bash
|
| 141 |
conda create -n realtime-venus python=3.10 -y
|
|
@@ -147,9 +171,9 @@ mkdir -p Realtime-Venus
|
|
| 147 |
cd Realtime-Venus
|
| 148 |
```
|
| 149 |
|
| 150 |
-
### 6.
|
| 151 |
|
| 152 |
-
Fetch the unified downloader
|
| 153 |
|
| 154 |
```bash
|
| 155 |
MODEL_REVISION=03b5c2919dd684900bdf6852272ac6a8f30fab74
|
|
@@ -157,46 +181,32 @@ hf download inclusionAI/Realtime-Venus download_models.py \
|
|
| 157 |
--revision "$MODEL_REVISION" --local-dir .
|
| 158 |
```
|
| 159 |
|
| 160 |
-
Choose the
|
| 161 |
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
| `audio` | Realtime-Venus-Audio for audio understanding and conversation |
|
| 166 |
-
| `all` | Both models |
|
| 167 |
|
| 168 |
-
|
|
|
|
| 169 |
|
| 170 |
-
|
| 171 |
python download_models.py --model all --local-dir . --revision "$MODEL_REVISION"
|
| 172 |
```
|
| 173 |
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
The main files after downloading both models are shown below. Selecting one model creates only its corresponding model directory:
|
| 177 |
|
| 178 |
-
``
|
| 179 |
-
Realtime-Venus/
|
| 180 |
-
├── config.yaml
|
| 181 |
-
├── download_models.py
|
| 182 |
-
├── Realtime-Venus-Omni/
|
| 183 |
-
│ ├── config.json
|
| 184 |
-
│ ├── requirements.txt
|
| 185 |
-
│ └── … # Weights, custom code, tokenizer, processor, and resources
|
| 186 |
-
└── Realtime-Venus-Audio/
|
| 187 |
-
├── config.json
|
| 188 |
-
└── … # Weights, custom code, tokenizer, processor, and resources
|
| 189 |
-
```
|
| 190 |
|
| 191 |
-
### 6.
|
| 192 |
|
| 193 |
-
Both models share `Realtime-Venus-Omni/requirements.txt`. After downloading Omni or
|
| 194 |
|
| 195 |
```bash
|
| 196 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 197 |
```
|
| 198 |
|
| 199 |
-
For an Audio-only download, fetch this dependency file
|
| 200 |
|
| 201 |
```bash
|
| 202 |
hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt \
|
|
@@ -204,23 +214,83 @@ hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt \
|
|
| 204 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 205 |
```
|
| 206 |
|
| 207 |
-
### 6.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 208 |
|
| 209 |
-
|
|
|
|
|
|
|
| 210 |
|
| 211 |
```python
|
| 212 |
-
|
|
|
|
|
|
|
| 213 |
|
| 214 |
-
|
| 215 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 216 |
trust_remote_code=True,
|
| 217 |
local_files_only=True,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 218 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 219 |
```
|
| 220 |
|
| 221 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 222 |
|
| 223 |
-
|
| 224 |
|
| 225 |
## 7. 🎙️ Realtime-Venus-Omni Usages
|
| 226 |
|
|
|
|
| 131 |
|
| 132 |
## 6. 🛠️ Installation
|
| 133 |
|
| 134 |
+
### 6.1 Use the new repository layout
|
| 135 |
|
| 136 |
+
Use the repository root as your local working directory, keeping the root configuration alongside the complete selected model directories. Run the commands and examples below from that working directory.
|
| 137 |
|
| 138 |
+
```text
|
| 139 |
+
Realtime-Venus/
|
| 140 |
+
├── config.yaml # Complete model configurations and directory mapping
|
| 141 |
+
├── download_models.py # Unified downloader
|
| 142 |
+
├── Realtime-Venus-Omni/ # Audio-visual model; download as needed
|
| 143 |
+
│ ├── config.json
|
| 144 |
+
│ ├── requirements.txt
|
| 145 |
+
│ ├── assets/
|
| 146 |
+
│ └── … # Weights, custom code, tokenizer, processor, etc.
|
| 147 |
+
└── Realtime-Venus-Audio/ # Audio-only model; download as needed
|
| 148 |
+
├── config.json
|
| 149 |
+
├── assets/
|
| 150 |
+
└── … # Weights, custom code, tokenizer, processor, etc.
|
| 151 |
+
```
|
| 152 |
+
|
| 153 |
+
| File or directory | How to use it |
|
| 154 |
+
| --- | --- |
|
| 155 |
+
| Root `config.yaml` | Contains the complete model configurations; the downloader fetches it automatically and the custom configuration code reads it when loading a model |
|
| 156 |
+
| Each model's `config.json` | Identifies the model to Transformers and references a fixed root configuration revision |
|
| 157 |
+
| `Realtime-Venus-Omni/` | Load this subdirectory for audio-visual understanding, conversation, and full-duplex interaction |
|
| 158 |
+
| `Realtime-Venus-Audio/` | Load this subdirectory for audio understanding, conversation, and full-duplex interaction |
|
| 159 |
+
|
| 160 |
+
### 6.2 Prepare the environment
|
| 161 |
+
|
| 162 |
+
Inference requires an NVIDIA CUDA GPU. The following commands use Python 3.10, Conda, and FFmpeg. Use compatible versions of PyTorch, CUDA, and your GPU driver.
|
| 163 |
|
| 164 |
```bash
|
| 165 |
conda create -n realtime-venus python=3.10 -y
|
|
|
|
| 171 |
cd Realtime-Venus
|
| 172 |
```
|
| 173 |
|
| 174 |
+
### 6.3 Download the models you need
|
| 175 |
|
| 176 |
+
Fetch the unified downloader and set the model revision used in this section. Run the following commands in the same terminal:
|
| 177 |
|
| 178 |
```bash
|
| 179 |
MODEL_REVISION=03b5c2919dd684900bdf6852272ac6a8f30fab74
|
|
|
|
| 181 |
--revision "$MODEL_REVISION" --local-dir .
|
| 182 |
```
|
| 183 |
|
| 184 |
+
Choose one of the following three commands:
|
| 185 |
|
| 186 |
+
```bash
|
| 187 |
+
# Omni only: audio-visual interaction
|
| 188 |
+
python download_models.py --model omni --local-dir . --revision "$MODEL_REVISION"
|
|
|
|
|
|
|
| 189 |
|
| 190 |
+
# Audio only: audio interaction
|
| 191 |
+
python download_models.py --model audio --local-dir . --revision "$MODEL_REVISION"
|
| 192 |
|
| 193 |
+
# Both models
|
| 194 |
python download_models.py --model all --local-dir . --revision "$MODEL_REVISION"
|
| 195 |
```
|
| 196 |
|
| 197 |
+
The downloader fetches root `config.yaml` first, then uses its directory mapping to download the complete selected model directories, including weights, custom code, tokenizer, processor, TTS resources, and demo assets. Selecting one model does not download the other model's weights. These commands also do not download `data/HEARTBEAT_Challenge/`.
|
|
|
|
|
|
|
| 198 |
|
| 199 |
+
If interrupted, rerun the same download command to reuse completed files. Change `--local-dir` to use a different destination. When running examples, enter that directory or adjust the model and asset paths accordingly.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 200 |
|
| 201 |
+
### 6.4 Install inference dependencies
|
| 202 |
|
| 203 |
+
Both models share `Realtime-Venus-Omni/requirements.txt`. After downloading Omni or both models, install it directly:
|
| 204 |
|
| 205 |
```bash
|
| 206 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 207 |
```
|
| 208 |
|
| 209 |
+
For an Audio-only download, fetch this dependency file before installing. This adds only the Omni `requirements.txt` file and does not download Omni weights:
|
| 210 |
|
| 211 |
```bash
|
| 212 |
hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt \
|
|
|
|
| 214 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 215 |
```
|
| 216 |
|
| 217 |
+
### 6.5 Load and use the models
|
| 218 |
+
|
| 219 |
+
#### Omni: load the local model subdirectory
|
| 220 |
+
|
| 221 |
+
From the `Realtime-Venus/` working directory, use Transformers to load `./Realtime-Venus-Omni`:
|
| 222 |
+
|
| 223 |
+
```python
|
| 224 |
+
import torch
|
| 225 |
+
from transformers import AutoModel
|
| 226 |
+
|
| 227 |
+
model = AutoModel.from_pretrained(
|
| 228 |
+
"./Realtime-Venus-Omni",
|
| 229 |
+
trust_remote_code=True,
|
| 230 |
+
local_files_only=True,
|
| 231 |
+
attn_implementation="sdpa",
|
| 232 |
+
torch_dtype=torch.bfloat16,
|
| 233 |
+
).eval().cuda()
|
| 234 |
+
```
|
| 235 |
+
|
| 236 |
+
After loading, follow Section 7.2 for full-duplex video interaction or Section 7.3 for offline conversation over a complete video. Full-duplex interaction uses `as_duplex()`, `prepare()`, `streaming_prefill()`, and `streaming_generate()`; offline conversation uses `model.chat(...)`. Follow the corresponding example to read input files from `Realtime-Venus-Omni/assets/`.
|
| 237 |
|
| 238 |
+
#### Audio: run one audio question-answering example
|
| 239 |
+
|
| 240 |
+
Save the following code as `run_audio.py` in the same working directory. It loads the local Audio model, reads its bundled demo audio, and prints a text response:
|
| 241 |
|
| 242 |
```python
|
| 243 |
+
import librosa
|
| 244 |
+
import torch
|
| 245 |
+
from transformers import AutoModel, AutoTokenizer
|
| 246 |
|
| 247 |
+
model_dir = "./Realtime-Venus-Audio"
|
| 248 |
+
tokenizer = AutoTokenizer.from_pretrained(
|
| 249 |
+
model_dir,
|
| 250 |
+
trust_remote_code=True,
|
| 251 |
+
local_files_only=True,
|
| 252 |
+
fix_mistral_regex=True,
|
| 253 |
+
)
|
| 254 |
+
model = AutoModel.from_pretrained(
|
| 255 |
+
model_dir,
|
| 256 |
trust_remote_code=True,
|
| 257 |
local_files_only=True,
|
| 258 |
+
attn_implementation="sdpa",
|
| 259 |
+
torch_dtype=torch.bfloat16,
|
| 260 |
+
init_vision=False,
|
| 261 |
+
init_audio=True,
|
| 262 |
+
init_tts=False,
|
| 263 |
+
).eval().cuda()
|
| 264 |
+
|
| 265 |
+
audio, _ = librosa.load(
|
| 266 |
+
"Realtime-Venus-Audio/assets/case_offline.wav", sr=16000, mono=True
|
| 267 |
+
)
|
| 268 |
+
answer = model.chat(
|
| 269 |
+
msgs=[
|
| 270 |
+
{"role": "system", "content": "You are a helpful assistant."},
|
| 271 |
+
{"role": "user", "content": [audio, "What is the speaker asking about?"]},
|
| 272 |
+
],
|
| 273 |
+
tokenizer=tokenizer,
|
| 274 |
+
do_sample=False,
|
| 275 |
+
max_new_tokens=2048,
|
| 276 |
+
enable_thinking=False,
|
| 277 |
+
use_tts_template=True,
|
| 278 |
+
generate_audio=False,
|
| 279 |
)
|
| 280 |
+
print(answer)
|
| 281 |
+
```
|
| 282 |
+
|
| 283 |
+
```bash
|
| 284 |
+
python run_audio.py
|
| 285 |
```
|
| 286 |
|
| 287 |
+
This example outputs text only and therefore sets `init_tts=False`. For speech output or full-duplex interaction, enable `init_tts=True` and follow the loading and calling examples in Section 8.
|
| 288 |
+
|
| 289 |
+
### 6.6 Offline use and moving the models
|
| 290 |
+
|
| 291 |
+
After downloading the models and installing dependencies while online, load them using the local paths above. When moving to another machine, copy root `config.yaml` together with the complete model directories you need. Preserve the relative layout in Section 6.1 and install the dependencies on the target machine. The model option `local_files_only=True` uses local files.
|
| 292 |
|
| 293 |
+
Each model's `config.json` references root configuration revision `15d5b420cd8557e9582d41a57a80af3e4de18e47` through `venus_config_ref` and verifies its SHA-256 digest. The unified downloader fetches the corresponding root configuration. Point `from_pretrained` at the selected model subdirectory, such as `./Realtime-Venus-Audio`.
|
| 294 |
|
| 295 |
## 7. 🎙️ Realtime-Venus-Omni Usages
|
| 296 |
|
README_zh.md
CHANGED
|
@@ -91,11 +91,35 @@
|
|
| 91 |
|
| 92 |
## 6. 🛠️ 安装
|
| 93 |
|
| 94 |
-
|
| 95 |
|
| 96 |
-
|
| 97 |
|
| 98 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
|
| 100 |
```bash
|
| 101 |
conda create -n realtime-venus python=3.10 -y
|
|
@@ -107,9 +131,9 @@ mkdir -p Realtime-Venus
|
|
| 107 |
cd Realtime-Venus
|
| 108 |
```
|
| 109 |
|
| 110 |
-
### 6.
|
| 111 |
|
| 112 |
-
先
|
| 113 |
|
| 114 |
```bash
|
| 115 |
MODEL_REVISION=03b5c2919dd684900bdf6852272ac6a8f30fab74
|
|
@@ -117,46 +141,32 @@ hf download inclusionAI/Realtime-Venus download_models.py \
|
|
| 117 |
--revision "$MODEL_REVISION" --local-dir .
|
| 118 |
```
|
| 119 |
|
| 120 |
-
根据需要选择
|
| 121 |
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
| `audio` | Realtime-Venus-Audio:音频理解与语音对话 |
|
| 126 |
-
| `all` | 两款模型 |
|
| 127 |
|
| 128 |
-
|
|
|
|
| 129 |
|
| 130 |
-
|
| 131 |
python download_models.py --model all --local-dir . --revision "$MODEL_REVISION"
|
| 132 |
```
|
| 133 |
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
下载两款模型后的主要目录如下;只下载一款时,仅有对应的模型目录:
|
| 137 |
|
| 138 |
-
``
|
| 139 |
-
Realtime-Venus/
|
| 140 |
-
├── config.yaml
|
| 141 |
-
├── download_models.py
|
| 142 |
-
├── Realtime-Venus-Omni/
|
| 143 |
-
│ ├── config.json
|
| 144 |
-
│ ├── requirements.txt
|
| 145 |
-
│ └── … # 权重、自定义代码、Tokenizer、Processor 与资源
|
| 146 |
-
└── Realtime-Venus-Audio/
|
| 147 |
-
├── config.json
|
| 148 |
-
└── … # 权重、自定义代码、Tokenizer、Processor 与资源
|
| 149 |
-
```
|
| 150 |
|
| 151 |
-
### 6.
|
| 152 |
|
| 153 |
-
两款模型共用 `Realtime-Venus-Omni/requirements.txt`。下载了 Omni 或
|
| 154 |
|
| 155 |
```bash
|
| 156 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 157 |
```
|
| 158 |
|
| 159 |
-
如果只下载了 Audio,先
|
| 160 |
|
| 161 |
```bash
|
| 162 |
hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt \
|
|
@@ -164,23 +174,83 @@ hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt \
|
|
| 164 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 165 |
```
|
| 166 |
|
| 167 |
-
### 6.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 168 |
|
| 169 |
-
|
|
|
|
|
|
|
| 170 |
|
| 171 |
```python
|
| 172 |
-
|
|
|
|
|
|
|
| 173 |
|
| 174 |
-
|
| 175 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
trust_remote_code=True,
|
| 177 |
local_files_only=True,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 178 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
```
|
| 180 |
|
| 181 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 182 |
|
| 183 |
-
模型
|
| 184 |
|
| 185 |
## 7. 🎙️ Realtime-Venus-Omni 使用方法
|
| 186 |
|
|
|
|
| 91 |
|
| 92 |
## 6. 🛠️ 安装
|
| 93 |
|
| 94 |
+
### 6.1 新版仓库的使用方式
|
| 95 |
|
| 96 |
+
下载模型时,以仓库根目录为本地工作目录,保留根配置和所选模型的完整子目录。后续所有命令和示例均在这个工作目录中执行。
|
| 97 |
|
| 98 |
+
```text
|
| 99 |
+
Realtime-Venus/
|
| 100 |
+
├── config.yaml # 两款模型的完整配置及目录映射
|
| 101 |
+
├── download_models.py # 统一下载入口
|
| 102 |
+
├── Realtime-Venus-Omni/ # 音视频模型,按需下载
|
| 103 |
+
│ ├── config.json
|
| 104 |
+
│ ├── requirements.txt
|
| 105 |
+
│ ├── assets/
|
| 106 |
+
│ └── … # 权重、自定义代码、Tokenizer、Processor 等
|
| 107 |
+
└── Realtime-Venus-Audio/ # 纯音频模型,按需下载
|
| 108 |
+
├── config.json
|
| 109 |
+
├── assets/
|
| 110 |
+
└── … # 权重、自定义代码、Tokenizer、Processor 等
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
| 文件或目录 | 如何使用 |
|
| 114 |
+
| --- | --- |
|
| 115 |
+
| 根 `config.yaml` | 保存完整模型配置;统一下载器会自动获取,加载模型时由自定义配置代码读取 |
|
| 116 |
+
| 各模型目录的 `config.json` | 供 Transformers 识别模型类型,并引用固定版本的根配置 |
|
| 117 |
+
| `Realtime-Venus-Omni/` | 音视频理解、对话和全双工交互,从这个子目录加载模型 |
|
| 118 |
+
| `Realtime-Venus-Audio/` | 音频理解、语音对话和全双工交互,从这个子目录加载模型 |
|
| 119 |
+
|
| 120 |
+
### 6.2 准备环境
|
| 121 |
+
|
| 122 |
+
推理需要 NVIDIA CUDA GPU。以下使用 Python 3.10、Conda 和 FFmpeg;请使用相互兼容的 PyTorch、CUDA 和显卡驱动。
|
| 123 |
|
| 124 |
```bash
|
| 125 |
conda create -n realtime-venus python=3.10 -y
|
|
|
|
| 131 |
cd Realtime-Venus
|
| 132 |
```
|
| 133 |
|
| 134 |
+
### 6.3 按需下载模型
|
| 135 |
|
| 136 |
+
先下载统一下载脚本,并指定本节使用的模型版本。以下命令应在同一个终端中执行:
|
| 137 |
|
| 138 |
```bash
|
| 139 |
MODEL_REVISION=03b5c2919dd684900bdf6852272ac6a8f30fab74
|
|
|
|
| 141 |
--revision "$MODEL_REVISION" --local-dir .
|
| 142 |
```
|
| 143 |
|
| 144 |
+
根据需要,从下面三条命令中选择一条执行:
|
| 145 |
|
| 146 |
+
```bash
|
| 147 |
+
# 只下载 Omni:用于音视频交互
|
| 148 |
+
python download_models.py --model omni --local-dir . --revision "$MODEL_REVISION"
|
|
|
|
|
|
|
| 149 |
|
| 150 |
+
# 只下载 Audio:用于纯音频交互
|
| 151 |
+
python download_models.py --model audio --local-dir . --revision "$MODEL_REVISION"
|
| 152 |
|
| 153 |
+
# 同时下载两款模型
|
| 154 |
python download_models.py --model all --local-dir . --revision "$MODEL_REVISION"
|
| 155 |
```
|
| 156 |
|
| 157 |
+
下载器会先下载根 `config.yaml`,再根据其中的目录映射下载所选模型的完整目录,包括权重、自定义代码、Tokenizer、Processor、TTS 和演示资源。选择一款模型时,不会下载另一款模型的权重;以上命令也不会下载 `data/HEARTBEAT_Challenge/` 数据。
|
|
|
|
|
|
|
| 158 |
|
| 159 |
+
中断后,重新执行相同的下载命令即可复用已完成的文件。若需要更换本地保存位置,可修改 `--local-dir`;运行示例时,进入该目录,或相应调整模型和资源文件路径。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
|
| 161 |
+
### 6.4 安装推理依赖
|
| 162 |
|
| 163 |
+
两款模型共用 `Realtime-Venus-Omni/requirements.txt`。如果下载了 Omni 或两款模型,直接安装:
|
| 164 |
|
| 165 |
```bash
|
| 166 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 167 |
```
|
| 168 |
|
| 169 |
+
如果只下载了 Audio,先获取这一份依赖文件,再安装。该命令只会增加 Omni 目录下的 `requirements.txt`,不会下载 Omni 权重:
|
| 170 |
|
| 171 |
```bash
|
| 172 |
hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt \
|
|
|
|
| 174 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 175 |
```
|
| 176 |
|
| 177 |
+
### 6.5 加载并使用模型
|
| 178 |
+
|
| 179 |
+
#### Omni:从本地子目录加载
|
| 180 |
+
|
| 181 |
+
在 `Realtime-Venus/` 工作目录中,使用 Transformers 加载 `./Realtime-Venus-Omni`:
|
| 182 |
+
|
| 183 |
+
```python
|
| 184 |
+
import torch
|
| 185 |
+
from transformers import AutoModel
|
| 186 |
+
|
| 187 |
+
model = AutoModel.from_pretrained(
|
| 188 |
+
"./Realtime-Venus-Omni",
|
| 189 |
+
trust_remote_code=True,
|
| 190 |
+
local_files_only=True,
|
| 191 |
+
attn_implementation="sdpa",
|
| 192 |
+
torch_dtype=torch.bfloat16,
|
| 193 |
+
).eval().cuda()
|
| 194 |
+
```
|
| 195 |
+
|
| 196 |
+
加载后,第 7.2 节给出全双工视频交互示例,第 7.3 节给出完整视频的离线对话示例。全双工使用 `as_duplex()`、`prepare()`、`streaming_prefill()` 和 `streaming_generate()`;离线对话使用 `model.chat(...)`。按对应示例读取 `Realtime-Venus-Omni/assets/` 中的输入文件。
|
| 197 |
|
| 198 |
+
#### Audio:运行一次音频问答
|
| 199 |
+
|
| 200 |
+
在同一工作目录中,将下面代码保存为 `run_audio.py`。它加载本地 Audio 模型,读取随模型下载的演示音频,并打印文本回答:
|
| 201 |
|
| 202 |
```python
|
| 203 |
+
import librosa
|
| 204 |
+
import torch
|
| 205 |
+
from transformers import AutoModel, AutoTokenizer
|
| 206 |
|
| 207 |
+
model_dir = "./Realtime-Venus-Audio"
|
| 208 |
+
tokenizer = AutoTokenizer.from_pretrained(
|
| 209 |
+
model_dir,
|
| 210 |
+
trust_remote_code=True,
|
| 211 |
+
local_files_only=True,
|
| 212 |
+
fix_mistral_regex=True,
|
| 213 |
+
)
|
| 214 |
+
model = AutoModel.from_pretrained(
|
| 215 |
+
model_dir,
|
| 216 |
trust_remote_code=True,
|
| 217 |
local_files_only=True,
|
| 218 |
+
attn_implementation="sdpa",
|
| 219 |
+
torch_dtype=torch.bfloat16,
|
| 220 |
+
init_vision=False,
|
| 221 |
+
init_audio=True,
|
| 222 |
+
init_tts=False,
|
| 223 |
+
).eval().cuda()
|
| 224 |
+
|
| 225 |
+
audio, _ = librosa.load(
|
| 226 |
+
"Realtime-Venus-Audio/assets/case_offline.wav", sr=16000, mono=True
|
| 227 |
+
)
|
| 228 |
+
answer = model.chat(
|
| 229 |
+
msgs=[
|
| 230 |
+
{"role": "system", "content": "You are a helpful assistant."},
|
| 231 |
+
{"role": "user", "content": [audio, "What is the speaker asking about?"]},
|
| 232 |
+
],
|
| 233 |
+
tokenizer=tokenizer,
|
| 234 |
+
do_sample=False,
|
| 235 |
+
max_new_tokens=2048,
|
| 236 |
+
enable_thinking=False,
|
| 237 |
+
use_tts_template=True,
|
| 238 |
+
generate_audio=False,
|
| 239 |
)
|
| 240 |
+
print(answer)
|
| 241 |
+
```
|
| 242 |
+
|
| 243 |
+
```bash
|
| 244 |
+
python run_audio.py
|
| 245 |
```
|
| 246 |
|
| 247 |
+
该示例只输出文本,因此设置 `init_tts=False`。需要语音输出或全双工交互时,按照第 8 节的加载和调用示例启用 `init_tts=True`。
|
| 248 |
+
|
| 249 |
+
### 6.6 离线使用与目录搬移
|
| 250 |
+
|
| 251 |
+
在联网环境中完成模型下载和依赖安装后,可以按上述本地路径加载。复制到其他机器时,将根 `config.yaml` 与所需的完整模型目录一起复制,保持 6.1 中的相对位置,并为目标机器安装对应依赖。模型参数中的 `local_files_only=True` 使用本地文件。
|
| 252 |
|
| 253 |
+
各模型目录的 `config.json` 通过 `venus_config_ref` 引用根配置版本 `15d5b420cd8557e9582d41a57a80af3e4de18e47`,并校验其 SHA-256;统一下载器已获取对应的根配置。使用模型时,将 `from_pretrained` 指向模型子目录,例如 `./Realtime-Venus-Audio`。
|
| 254 |
|
| 255 |
## 7. 🎙️ Realtime-Venus-Omni 使用方法
|
| 256 |
|