Any-to-Any
Transformers
ONNX
Safetensors
English
Chinese
multimodal
audio
video
speech
streaming
full-duplex
long-video
custom-code
Instructions to use inclusionAI/Realtime-Venus with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use inclusionAI/Realtime-Venus with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("inclusionAI/Realtime-Venus", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Realtime-Venus local preparation commited on
Commit ·
a0d2f25
1
Parent(s): 7b362f0
Refine public model documentation
Browse files- README.md +37 -128
- README_zh.md +22 -128
README.md
CHANGED
|
@@ -131,166 +131,75 @@ new output directory when repeating an experiment.
|
|
| 131 |
|
| 132 |
## 6. 🛠️ Installation
|
| 133 |
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
```text
|
| 139 |
-
Realtime-Venus/
|
| 140 |
-
├── config.yaml # Complete model configurations and directory mapping
|
| 141 |
-
├── download_models.py # Unified downloader
|
| 142 |
-
├── Realtime-Venus-Omni/ # Audio-visual model; download as needed
|
| 143 |
-
│ ├── config.json
|
| 144 |
-
│ ├── requirements.txt
|
| 145 |
-
│ ├── assets/
|
| 146 |
-
│ └── … # Weights, custom code, tokenizer, processor, etc.
|
| 147 |
-
└── Realtime-Venus-Audio/ # Audio-only model; download as needed
|
| 148 |
-
├── config.json
|
| 149 |
-
├── assets/
|
| 150 |
-
└── … # Weights, custom code, tokenizer, processor, etc.
|
| 151 |
-
```
|
| 152 |
-
|
| 153 |
-
| File or directory | How to use it |
|
| 154 |
-
| --- | --- |
|
| 155 |
-
| Root `config.yaml` | Contains the complete model configurations; the downloader fetches it automatically and the custom configuration code reads it when loading a model |
|
| 156 |
-
| Each model's `config.json` | Identifies the model to Transformers and references a fixed root configuration revision |
|
| 157 |
-
| `Realtime-Venus-Omni/` | Load this subdirectory for audio-visual understanding, conversation, and full-duplex interaction |
|
| 158 |
-
| `Realtime-Venus-Audio/` | Load this subdirectory for audio understanding, conversation, and full-duplex interaction |
|
| 159 |
-
|
| 160 |
-
### 6.2 Prepare the environment
|
| 161 |
-
|
| 162 |
-
Inference requires an NVIDIA CUDA GPU. The following commands use Python 3.10, Conda, and FFmpeg. Use compatible versions of PyTorch, CUDA, and your GPU driver.
|
| 163 |
|
| 164 |
```bash
|
| 165 |
-
conda create -n realtime-venus python=3.10 -y
|
| 166 |
-
conda activate realtime-venus
|
| 167 |
-
conda install -c conda-forge ffmpeg -y
|
| 168 |
python -m pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
|
| 169 |
-
|
| 170 |
-
mkdir -p Realtime-Venus
|
| 171 |
-
cd Realtime-Venus
|
| 172 |
```
|
| 173 |
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
Fetch the unified downloader and set the model revision used in this section. Run the following commands in the same terminal:
|
| 177 |
|
| 178 |
-
``
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
``
|
| 183 |
|
| 184 |
-
|
| 185 |
|
| 186 |
```bash
|
| 187 |
-
|
| 188 |
-
python download_models.py --model omni --local-dir . --revision "$MODEL_REVISION"
|
| 189 |
-
|
| 190 |
-
# Audio only: audio interaction
|
| 191 |
-
python download_models.py --model audio --local-dir . --revision "$MODEL_REVISION"
|
| 192 |
-
|
| 193 |
-
# Both models
|
| 194 |
-
python download_models.py --model all --local-dir . --revision "$MODEL_REVISION"
|
| 195 |
```
|
| 196 |
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
|
|
|
|
|
|
|
|
|
| 200 |
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
Both models share `Realtime-Venus-Omni/requirements.txt`. After downloading Omni or both models, install it directly:
|
| 204 |
|
| 205 |
```bash
|
| 206 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 207 |
```
|
| 208 |
|
| 209 |
-
|
|
|
|
|
|
|
| 210 |
|
| 211 |
```bash
|
| 212 |
-
hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt
|
| 213 |
-
--revision "$MODEL_REVISION" --local-dir .
|
| 214 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 215 |
```
|
| 216 |
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
#### Omni: load the local model subdirectory
|
| 220 |
-
|
| 221 |
-
From the `Realtime-Venus/` working directory, use Transformers to load `./Realtime-Venus-Omni`:
|
| 222 |
|
| 223 |
```python
|
| 224 |
-
import
|
| 225 |
-
from transformers import AutoModel
|
| 226 |
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
trust_remote_code=True,
|
| 230 |
-
local_files_only=True,
|
| 231 |
-
attn_implementation="sdpa",
|
| 232 |
-
torch_dtype=torch.bfloat16,
|
| 233 |
-
).eval().cuda()
|
| 234 |
```
|
| 235 |
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
|
| 242 |
-
|
| 243 |
-
|
| 244 |
-
import torch
|
| 245 |
-
from transformers import AutoModel, AutoTokenizer
|
| 246 |
-
|
| 247 |
-
model_dir = "./Realtime-Venus-Audio"
|
| 248 |
-
tokenizer = AutoTokenizer.from_pretrained(
|
| 249 |
-
model_dir,
|
| 250 |
-
trust_remote_code=True,
|
| 251 |
-
local_files_only=True,
|
| 252 |
-
fix_mistral_regex=True,
|
| 253 |
-
)
|
| 254 |
-
model = AutoModel.from_pretrained(
|
| 255 |
-
model_dir,
|
| 256 |
-
trust_remote_code=True,
|
| 257 |
-
local_files_only=True,
|
| 258 |
-
attn_implementation="sdpa",
|
| 259 |
-
torch_dtype=torch.bfloat16,
|
| 260 |
-
init_vision=False,
|
| 261 |
-
init_audio=True,
|
| 262 |
-
init_tts=False,
|
| 263 |
-
).eval().cuda()
|
| 264 |
-
|
| 265 |
-
audio, _ = librosa.load(
|
| 266 |
-
"Realtime-Venus-Audio/assets/case_offline.wav", sr=16000, mono=True
|
| 267 |
-
)
|
| 268 |
-
answer = model.chat(
|
| 269 |
-
msgs=[
|
| 270 |
-
{"role": "system", "content": "You are a helpful assistant."},
|
| 271 |
-
{"role": "user", "content": [audio, "What is the speaker asking about?"]},
|
| 272 |
-
],
|
| 273 |
-
tokenizer=tokenizer,
|
| 274 |
-
do_sample=False,
|
| 275 |
-
max_new_tokens=2048,
|
| 276 |
-
enable_thinking=False,
|
| 277 |
-
use_tts_template=True,
|
| 278 |
-
generate_audio=False,
|
| 279 |
-
)
|
| 280 |
-
print(answer)
|
| 281 |
-
```
|
| 282 |
|
| 283 |
```bash
|
| 284 |
-
|
| 285 |
```
|
| 286 |
|
| 287 |
-
This
|
| 288 |
-
|
| 289 |
-
### 6.6 Offline use and moving the models
|
| 290 |
-
|
| 291 |
-
After downloading the models and installing dependencies while online, load them using the local paths above. When moving to another machine, copy root `config.yaml` together with the complete model directories you need. Preserve the relative layout in Section 6.1 and install the dependencies on the target machine. The model option `local_files_only=True` uses local files.
|
| 292 |
-
|
| 293 |
-
Each model's `config.json` references root configuration revision `15d5b420cd8557e9582d41a57a80af3e4de18e47` through `venus_config_ref` and verifies its SHA-256 digest. The unified downloader fetches the corresponding root configuration. Point `from_pretrained` at the selected model subdirectory, such as `./Realtime-Venus-Audio`.
|
| 294 |
|
| 295 |
## 7. 🎙️ Realtime-Venus-Omni Usages
|
| 296 |
|
|
|
|
| 131 |
|
| 132 |
## 6. 🛠️ Installation
|
| 133 |
|
| 134 |
+
Running the inference examples requires Python 3.10, CUDA, and FFmpeg. First,
|
| 135 |
+
install the download dependencies and fetch the unified downloader from this
|
| 136 |
+
Hugging Face repository:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
|
| 138 |
```bash
|
|
|
|
|
|
|
|
|
|
| 139 |
python -m pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
|
| 140 |
+
hf download inclusionAI/Realtime-Venus download_models.py --local-dir .
|
|
|
|
|
|
|
| 141 |
```
|
| 142 |
|
| 143 |
+
Then choose the models to download:
|
|
|
|
|
|
|
| 144 |
|
| 145 |
+
| `--model` | Download |
|
| 146 |
+
| --- | --- |
|
| 147 |
+
| `omni` | Realtime-Venus-Omni for audio-visual interaction |
|
| 148 |
+
| `audio` | Realtime-Venus-Audio for audio understanding and conversation |
|
| 149 |
+
| `all` | Both models |
|
| 150 |
|
| 151 |
+
For example, download both models into the current directory:
|
| 152 |
|
| 153 |
```bash
|
| 154 |
+
python download_models.py --model all --local-dir .
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 155 |
```
|
| 156 |
|
| 157 |
+
Use `--model omni` or `--model audio` to download only the model you need.
|
| 158 |
+
The downloader first fetches root `config.yaml`, which contains both models'
|
| 159 |
+
complete configurations and directory mapping, then downloads the selected
|
| 160 |
+
model directories in full, including weights, custom code, and assets.
|
| 161 |
+
Each download uses one repository revision and the Hugging Face Hub's
|
| 162 |
+
standard progress display and cache.
|
| 163 |
|
| 164 |
+
Install the inference dependencies after downloading. For Omni or `all`:
|
|
|
|
|
|
|
| 165 |
|
| 166 |
```bash
|
| 167 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 168 |
```
|
| 169 |
|
| 170 |
+
Audio uses the same published dependency list; it does not have a separate
|
| 171 |
+
`requirements.txt`. If you downloaded only Audio, fetch that small file first
|
| 172 |
+
without downloading the Omni weights:
|
| 173 |
|
| 174 |
```bash
|
| 175 |
+
hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt --local-dir .
|
|
|
|
| 176 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 177 |
```
|
| 178 |
|
| 179 |
+
To download from Python instead, run this once from the directory containing
|
| 180 |
+
`download_models.py`. It uses the same downloader as the command above:
|
|
|
|
|
|
|
|
|
|
| 181 |
|
| 182 |
```python
|
| 183 |
+
from download_models import download_models
|
|
|
|
| 184 |
|
| 185 |
+
paths = download_models(model="omni", local_dir=".") # "omni", "audio", or "all"
|
| 186 |
+
model_dir = paths["omni"] # pathlib.Path; use paths["audio"] for Audio
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 187 |
```
|
| 188 |
|
| 189 |
+
The new layout loads each model using root `config.yaml` together with
|
| 190 |
+
the model directory's `config.json`. Keep the root configuration alongside
|
| 191 |
+
the model directories when downloading or moving them. The examples below
|
| 192 |
+
load `./Realtime-Venus-Omni` or `./Realtime-Venus-Audio`; run them from the
|
| 193 |
+
download directory, with asset and output paths relative to that directory.
|
| 194 |
|
| 195 |
+
As an independent alternative, the ModelScope CLI (installed separately) can
|
| 196 |
+
download the entire mirror repository:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 197 |
|
| 198 |
```bash
|
| 199 |
+
modelscope download --model inclusionAI/Realtime-Venus --local_dir .
|
| 200 |
```
|
| 201 |
|
| 202 |
+
This mirror command is separate from the Hugging Face downloader above.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 203 |
|
| 204 |
## 7. 🎙️ Realtime-Venus-Omni Usages
|
| 205 |
|
README_zh.md
CHANGED
|
@@ -91,166 +91,60 @@
|
|
| 91 |
|
| 92 |
## 6. 🛠️ 安装
|
| 93 |
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
下载模型时,以仓库根目录为本地工作目录,保留根配置和所选模型的完整子目录。后续所有命令和示例均在这个工作目录中执行。
|
| 97 |
-
|
| 98 |
-
```text
|
| 99 |
-
Realtime-Venus/
|
| 100 |
-
├── config.yaml # 两款模型的完整配置及目录映射
|
| 101 |
-
├── download_models.py # 统一下载入口
|
| 102 |
-
├── Realtime-Venus-Omni/ # 音视频模型,按需下载
|
| 103 |
-
│ ├── config.json
|
| 104 |
-
│ ├── requirements.txt
|
| 105 |
-
│ ├── assets/
|
| 106 |
-
│ └── … # 权重、自定义代码、Tokenizer、Processor 等
|
| 107 |
-
└── Realtime-Venus-Audio/ # 纯音频模型,按需下载
|
| 108 |
-
├── config.json
|
| 109 |
-
├── assets/
|
| 110 |
-
└── … # 权重、自定义代码、Tokenizer、Processor 等
|
| 111 |
-
```
|
| 112 |
-
|
| 113 |
-
| 文件或目录 | 如何使用 |
|
| 114 |
-
| --- | --- |
|
| 115 |
-
| 根 `config.yaml` | 保存完整模型配置;统一下载器会自动获取,加载模型时由自定义配置代码读取 |
|
| 116 |
-
| 各模型目录的 `config.json` | 供 Transformers 识别模型类型,并引用固定版本的根配置 |
|
| 117 |
-
| `Realtime-Venus-Omni/` | 音视频理解、对话和全双工交互,从这个子目录加载模型 |
|
| 118 |
-
| `Realtime-Venus-Audio/` | 音频理解、语音对话和全双工交互,从这个子目录加载模型 |
|
| 119 |
-
|
| 120 |
-
### 6.2 准备环境
|
| 121 |
-
|
| 122 |
-
推理需要 NVIDIA CUDA GPU。以下使用 Python 3.10、Conda 和 FFmpeg;请使用相互兼容的 PyTorch、CUDA 和显卡驱动。
|
| 123 |
|
| 124 |
```bash
|
| 125 |
-
conda create -n realtime-venus python=3.10 -y
|
| 126 |
-
conda activate realtime-venus
|
| 127 |
-
conda install -c conda-forge ffmpeg -y
|
| 128 |
python -m pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
|
| 129 |
-
|
| 130 |
-
mkdir -p Realtime-Venus
|
| 131 |
-
cd Realtime-Venus
|
| 132 |
```
|
| 133 |
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
先下载统一下载脚本,并指定本节使用的模型版本。以下命令应在同一个终端中执行:
|
| 137 |
|
| 138 |
-
``
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
``
|
| 143 |
|
| 144 |
-
|
| 145 |
|
| 146 |
```bash
|
| 147 |
-
|
| 148 |
-
python download_models.py --model omni --local-dir . --revision "$MODEL_REVISION"
|
| 149 |
-
|
| 150 |
-
# 只下载 Audio:用于纯音频交互
|
| 151 |
-
python download_models.py --model audio --local-dir . --revision "$MODEL_REVISION"
|
| 152 |
-
|
| 153 |
-
# 同时下载两款模型
|
| 154 |
-
python download_models.py --model all --local-dir . --revision "$MODEL_REVISION"
|
| 155 |
```
|
| 156 |
|
| 157 |
-
下载器
|
| 158 |
-
|
| 159 |
-
中断后,重新执行相同的下载命令即可复用已完成的文件。若需要更换本地保存位置,可修改 `--local-dir`;运行示例时,进入该目录,或相应调整模型和资源文件路径。
|
| 160 |
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
两款模型共用 `Realtime-Venus-Omni/requirements.txt`。如果下载了 Omni 或两款模型,直接安装:
|
| 164 |
|
| 165 |
```bash
|
| 166 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 167 |
```
|
| 168 |
|
| 169 |
-
如果只下载了 Audio,先获取这
|
| 170 |
|
| 171 |
```bash
|
| 172 |
-
hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt
|
| 173 |
-
--revision "$MODEL_REVISION" --local-dir .
|
| 174 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 175 |
```
|
| 176 |
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
#### Omni:从本地子目录加载
|
| 180 |
-
|
| 181 |
-
在 `Realtime-Venus/` 工作目录中,使用 Transformers 加载 `./Realtime-Venus-Omni`:
|
| 182 |
|
| 183 |
```python
|
| 184 |
-
import
|
| 185 |
-
from transformers import AutoModel
|
| 186 |
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
trust_remote_code=True,
|
| 190 |
-
local_files_only=True,
|
| 191 |
-
attn_implementation="sdpa",
|
| 192 |
-
torch_dtype=torch.bfloat16,
|
| 193 |
-
).eval().cuda()
|
| 194 |
```
|
| 195 |
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
#### Audio:运行一次音频问答
|
| 199 |
-
|
| 200 |
-
在同一工作目录中,将下面代码保存为 `run_audio.py`。它加载本地 Audio 模型,读取随模型下载的演示音频,并打印文本回答:
|
| 201 |
|
| 202 |
-
|
| 203 |
-
import librosa
|
| 204 |
-
import torch
|
| 205 |
-
from transformers import AutoModel, AutoTokenizer
|
| 206 |
-
|
| 207 |
-
model_dir = "./Realtime-Venus-Audio"
|
| 208 |
-
tokenizer = AutoTokenizer.from_pretrained(
|
| 209 |
-
model_dir,
|
| 210 |
-
trust_remote_code=True,
|
| 211 |
-
local_files_only=True,
|
| 212 |
-
fix_mistral_regex=True,
|
| 213 |
-
)
|
| 214 |
-
model = AutoModel.from_pretrained(
|
| 215 |
-
model_dir,
|
| 216 |
-
trust_remote_code=True,
|
| 217 |
-
local_files_only=True,
|
| 218 |
-
attn_implementation="sdpa",
|
| 219 |
-
torch_dtype=torch.bfloat16,
|
| 220 |
-
init_vision=False,
|
| 221 |
-
init_audio=True,
|
| 222 |
-
init_tts=False,
|
| 223 |
-
).eval().cuda()
|
| 224 |
-
|
| 225 |
-
audio, _ = librosa.load(
|
| 226 |
-
"Realtime-Venus-Audio/assets/case_offline.wav", sr=16000, mono=True
|
| 227 |
-
)
|
| 228 |
-
answer = model.chat(
|
| 229 |
-
msgs=[
|
| 230 |
-
{"role": "system", "content": "You are a helpful assistant."},
|
| 231 |
-
{"role": "user", "content": [audio, "What is the speaker asking about?"]},
|
| 232 |
-
],
|
| 233 |
-
tokenizer=tokenizer,
|
| 234 |
-
do_sample=False,
|
| 235 |
-
max_new_tokens=2048,
|
| 236 |
-
enable_thinking=False,
|
| 237 |
-
use_tts_template=True,
|
| 238 |
-
generate_audio=False,
|
| 239 |
-
)
|
| 240 |
-
print(answer)
|
| 241 |
-
```
|
| 242 |
|
| 243 |
```bash
|
| 244 |
-
|
| 245 |
```
|
| 246 |
|
| 247 |
-
该
|
| 248 |
-
|
| 249 |
-
### 6.6 离线使用与目录搬移
|
| 250 |
-
|
| 251 |
-
在联网环境中完成模型下载和依赖安装后,可以按上述本地路径加载。复制到其他机器时,将根 `config.yaml` 与所需的完整模型目录一起复制,保持 6.1 中的相对位置,并为目标机器安装对应依赖。模型参数中的 `local_files_only=True` 使用本地文件。
|
| 252 |
-
|
| 253 |
-
各模型目录的 `config.json` 通过 `venus_config_ref` 引用根配置版本 `15d5b420cd8557e9582d41a57a80af3e4de18e47`,并校验其 SHA-256;统一下载器已获取对应的根配置。使用模型时,将 `from_pretrained` 指向模型子目录,例如 `./Realtime-Venus-Audio`。
|
| 254 |
|
| 255 |
## 7. 🎙️ Realtime-Venus-Omni 使用方法
|
| 256 |
|
|
|
|
| 91 |
|
| 92 |
## 6. 🛠️ 安装
|
| 93 |
|
| 94 |
+
运行推理示例需要 Python 3.10、CUDA 和 FFmpeg。首先安装下载依赖,并从本 Hugging Face 仓库获取统一下载脚本:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
|
| 96 |
```bash
|
|
|
|
|
|
|
|
|
|
| 97 |
python -m pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
|
| 98 |
+
hf download inclusionAI/Realtime-Venus download_models.py --local-dir .
|
|
|
|
|
|
|
| 99 |
```
|
| 100 |
|
| 101 |
+
然后选择需要下载的模型:
|
|
|
|
|
|
|
| 102 |
|
| 103 |
+
| `--model` | 下载内容 |
|
| 104 |
+
| --- | --- |
|
| 105 |
+
| `omni` | 用于音视频交互的 Realtime-Venus-Omni |
|
| 106 |
+
| `audio` | 用于音频理解与语音对话的 Realtime-Venus-Audio |
|
| 107 |
+
| `all` | 两款模型 |
|
| 108 |
|
| 109 |
+
例如,将两款模型下载到当前目录:
|
| 110 |
|
| 111 |
```bash
|
| 112 |
+
python download_models.py --model all --local-dir .
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
```
|
| 114 |
|
| 115 |
+
如只需一款模型,将参数改为 `--model omni` 或 `--model audio`。下载器先获取根 `config.yaml`(包含两款模型的完整配置和目录映射),再下载所选模型的完整子目录,包括权重、自定义代码和资源文件。单次下载中的所有文件均来自同一个仓库版本,并使用 Hugging Face Hub 的标准进度显示与缓存。
|
|
|
|
|
|
|
| 116 |
|
| 117 |
+
下载后安装推理依赖。选择 Omni 或 `all` 时运行:
|
|
|
|
|
|
|
| 118 |
|
| 119 |
```bash
|
| 120 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 121 |
```
|
| 122 |
|
| 123 |
+
Audio 使用相同的已发布依赖清单,没有独立的 `requirements.txt`。如果只下载了 Audio,先单独获取这份小文件,无需下载 Omni 权重:
|
| 124 |
|
| 125 |
```bash
|
| 126 |
+
hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt --local-dir .
|
|
|
|
| 127 |
python -m pip install -r Realtime-Venus-Omni/requirements.txt
|
| 128 |
```
|
| 129 |
|
| 130 |
+
也可以在包含 `download_models.py` 的目录中,通过 Python 完成一次下载准备。它与上述命令使用同一个下载入口:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
```python
|
| 133 |
+
from download_models import download_models
|
|
|
|
| 134 |
|
| 135 |
+
paths = download_models(model="omni", local_dir=".") # 可选 "omni"、"audio" 或 "all"
|
| 136 |
+
model_dir = paths["omni"] # pathlib.Path;Audio 使用 paths["audio"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 137 |
```
|
| 138 |
|
| 139 |
+
新版通过根 `config.yaml` 与模型子目录中的 `config.json` 配合加载。下载或搬移模型后,请保留根配置与模型目录的相对位置。下文示例分别从 `./Realtime-Venus-Omni` 或 `./Realtime-Venus-Audio` 加载模型;请在下载目录中运行,资源文件和输出路径均相对于该目录。
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
|
| 141 |
+
也可以独立使用 ModelScope CLI(需另行安装)下载整个镜像仓库:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 142 |
|
| 143 |
```bash
|
| 144 |
+
modelscope download --model inclusionAI/Realtime-Venus --local_dir .
|
| 145 |
```
|
| 146 |
|
| 147 |
+
该镜像命令与上述 Hugging Face 下载器相互独立。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 148 |
|
| 149 |
## 7. 🎙️ Realtime-Venus-Omni 使用方法
|
| 150 |
|