Realtime-Venus local preparation commited on
Commit
a0d2f25
·
1 Parent(s): 7b362f0

Refine public model documentation

Browse files
Files changed (2) hide show
  1. README.md +37 -128
  2. README_zh.md +22 -128
README.md CHANGED
@@ -131,166 +131,75 @@ new output directory when repeating an experiment.
131
 
132
  ## 6. 🛠️ Installation
133
 
134
- ### 6.1 Use the new repository layout
135
-
136
- Use the repository root as your local working directory, keeping the root configuration alongside the complete selected model directories. Run the commands and examples below from that working directory.
137
-
138
- ```text
139
- Realtime-Venus/
140
- ├── config.yaml # Complete model configurations and directory mapping
141
- ├── download_models.py # Unified downloader
142
- ├── Realtime-Venus-Omni/ # Audio-visual model; download as needed
143
- │ ├── config.json
144
- │ ├── requirements.txt
145
- │ ├── assets/
146
- │ └── … # Weights, custom code, tokenizer, processor, etc.
147
- └── Realtime-Venus-Audio/ # Audio-only model; download as needed
148
- ├── config.json
149
- ├── assets/
150
- └── … # Weights, custom code, tokenizer, processor, etc.
151
- ```
152
-
153
- | File or directory | How to use it |
154
- | --- | --- |
155
- | Root `config.yaml` | Contains the complete model configurations; the downloader fetches it automatically and the custom configuration code reads it when loading a model |
156
- | Each model's `config.json` | Identifies the model to Transformers and references a fixed root configuration revision |
157
- | `Realtime-Venus-Omni/` | Load this subdirectory for audio-visual understanding, conversation, and full-duplex interaction |
158
- | `Realtime-Venus-Audio/` | Load this subdirectory for audio understanding, conversation, and full-duplex interaction |
159
-
160
- ### 6.2 Prepare the environment
161
-
162
- Inference requires an NVIDIA CUDA GPU. The following commands use Python 3.10, Conda, and FFmpeg. Use compatible versions of PyTorch, CUDA, and your GPU driver.
163
 
164
  ```bash
165
- conda create -n realtime-venus python=3.10 -y
166
- conda activate realtime-venus
167
- conda install -c conda-forge ffmpeg -y
168
  python -m pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
169
-
170
- mkdir -p Realtime-Venus
171
- cd Realtime-Venus
172
  ```
173
 
174
- ### 6.3 Download the models you need
175
-
176
- Fetch the unified downloader and set the model revision used in this section. Run the following commands in the same terminal:
177
 
178
- ```bash
179
- MODEL_REVISION=03b5c2919dd684900bdf6852272ac6a8f30fab74
180
- hf download inclusionAI/Realtime-Venus download_models.py \
181
- --revision "$MODEL_REVISION" --local-dir .
182
- ```
183
 
184
- Choose one of the following three commands:
185
 
186
  ```bash
187
- # Omni only: audio-visual interaction
188
- python download_models.py --model omni --local-dir . --revision "$MODEL_REVISION"
189
-
190
- # Audio only: audio interaction
191
- python download_models.py --model audio --local-dir . --revision "$MODEL_REVISION"
192
-
193
- # Both models
194
- python download_models.py --model all --local-dir . --revision "$MODEL_REVISION"
195
  ```
196
 
197
- The downloader fetches root `config.yaml` first, then uses its directory mapping to download the complete selected model directories, including weights, custom code, tokenizer, processor, TTS resources, and demo assets. Selecting one model does not download the other model's weights. These commands also do not download `data/HEARTBEAT_Challenge/`.
198
-
199
- If interrupted, rerun the same download command to reuse completed files. Change `--local-dir` to use a different destination. When running examples, enter that directory or adjust the model and asset paths accordingly.
 
 
 
200
 
201
- ### 6.4 Install inference dependencies
202
-
203
- Both models share `Realtime-Venus-Omni/requirements.txt`. After downloading Omni or both models, install it directly:
204
 
205
  ```bash
206
  python -m pip install -r Realtime-Venus-Omni/requirements.txt
207
  ```
208
 
209
- For an Audio-only download, fetch this dependency file before installing. This adds only the Omni `requirements.txt` file and does not download Omni weights:
 
 
210
 
211
  ```bash
212
- hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt \
213
- --revision "$MODEL_REVISION" --local-dir .
214
  python -m pip install -r Realtime-Venus-Omni/requirements.txt
215
  ```
216
 
217
- ### 6.5 Load and use the models
218
-
219
- #### Omni: load the local model subdirectory
220
-
221
- From the `Realtime-Venus/` working directory, use Transformers to load `./Realtime-Venus-Omni`:
222
 
223
  ```python
224
- import torch
225
- from transformers import AutoModel
226
 
227
- model = AutoModel.from_pretrained(
228
- "./Realtime-Venus-Omni",
229
- trust_remote_code=True,
230
- local_files_only=True,
231
- attn_implementation="sdpa",
232
- torch_dtype=torch.bfloat16,
233
- ).eval().cuda()
234
  ```
235
 
236
- After loading, follow Section 7.2 for full-duplex video interaction or Section 7.3 for offline conversation over a complete video. Full-duplex interaction uses `as_duplex()`, `prepare()`, `streaming_prefill()`, and `streaming_generate()`; offline conversation uses `model.chat(...)`. Follow the corresponding example to read input files from `Realtime-Venus-Omni/assets/`.
237
-
238
- #### Audio: run one audio question-answering example
239
-
240
- Save the following code as `run_audio.py` in the same working directory. It loads the local Audio model, reads its bundled demo audio, and prints a text response:
241
 
242
- ```python
243
- import librosa
244
- import torch
245
- from transformers import AutoModel, AutoTokenizer
246
-
247
- model_dir = "./Realtime-Venus-Audio"
248
- tokenizer = AutoTokenizer.from_pretrained(
249
- model_dir,
250
- trust_remote_code=True,
251
- local_files_only=True,
252
- fix_mistral_regex=True,
253
- )
254
- model = AutoModel.from_pretrained(
255
- model_dir,
256
- trust_remote_code=True,
257
- local_files_only=True,
258
- attn_implementation="sdpa",
259
- torch_dtype=torch.bfloat16,
260
- init_vision=False,
261
- init_audio=True,
262
- init_tts=False,
263
- ).eval().cuda()
264
-
265
- audio, _ = librosa.load(
266
- "Realtime-Venus-Audio/assets/case_offline.wav", sr=16000, mono=True
267
- )
268
- answer = model.chat(
269
- msgs=[
270
- {"role": "system", "content": "You are a helpful assistant."},
271
- {"role": "user", "content": [audio, "What is the speaker asking about?"]},
272
- ],
273
- tokenizer=tokenizer,
274
- do_sample=False,
275
- max_new_tokens=2048,
276
- enable_thinking=False,
277
- use_tts_template=True,
278
- generate_audio=False,
279
- )
280
- print(answer)
281
- ```
282
 
283
  ```bash
284
- python run_audio.py
285
  ```
286
 
287
- This example outputs text only and therefore sets `init_tts=False`. For speech output or full-duplex interaction, enable `init_tts=True` and follow the loading and calling examples in Section 8.
288
-
289
- ### 6.6 Offline use and moving the models
290
-
291
- After downloading the models and installing dependencies while online, load them using the local paths above. When moving to another machine, copy root `config.yaml` together with the complete model directories you need. Preserve the relative layout in Section 6.1 and install the dependencies on the target machine. The model option `local_files_only=True` uses local files.
292
-
293
- Each model's `config.json` references root configuration revision `15d5b420cd8557e9582d41a57a80af3e4de18e47` through `venus_config_ref` and verifies its SHA-256 digest. The unified downloader fetches the corresponding root configuration. Point `from_pretrained` at the selected model subdirectory, such as `./Realtime-Venus-Audio`.
294
 
295
  ## 7. 🎙️ Realtime-Venus-Omni Usages
296
 
 
131
 
132
  ## 6. 🛠️ Installation
133
 
134
+ Running the inference examples requires Python 3.10, CUDA, and FFmpeg. First,
135
+ install the download dependencies and fetch the unified downloader from this
136
+ Hugging Face repository:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
137
 
138
  ```bash
 
 
 
139
  python -m pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
140
+ hf download inclusionAI/Realtime-Venus download_models.py --local-dir .
 
 
141
  ```
142
 
143
+ Then choose the models to download:
 
 
144
 
145
+ | `--model` | Download |
146
+ | --- | --- |
147
+ | `omni` | Realtime-Venus-Omni for audio-visual interaction |
148
+ | `audio` | Realtime-Venus-Audio for audio understanding and conversation |
149
+ | `all` | Both models |
150
 
151
+ For example, download both models into the current directory:
152
 
153
  ```bash
154
+ python download_models.py --model all --local-dir .
 
 
 
 
 
 
 
155
  ```
156
 
157
+ Use `--model omni` or `--model audio` to download only the model you need.
158
+ The downloader first fetches root `config.yaml`, which contains both models'
159
+ complete configurations and directory mapping, then downloads the selected
160
+ model directories in full, including weights, custom code, and assets.
161
+ Each download uses one repository revision and the Hugging Face Hub's
162
+ standard progress display and cache.
163
 
164
+ Install the inference dependencies after downloading. For Omni or `all`:
 
 
165
 
166
  ```bash
167
  python -m pip install -r Realtime-Venus-Omni/requirements.txt
168
  ```
169
 
170
+ Audio uses the same published dependency list; it does not have a separate
171
+ `requirements.txt`. If you downloaded only Audio, fetch that small file first
172
+ without downloading the Omni weights:
173
 
174
  ```bash
175
+ hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt --local-dir .
 
176
  python -m pip install -r Realtime-Venus-Omni/requirements.txt
177
  ```
178
 
179
+ To download from Python instead, run this once from the directory containing
180
+ `download_models.py`. It uses the same downloader as the command above:
 
 
 
181
 
182
  ```python
183
+ from download_models import download_models
 
184
 
185
+ paths = download_models(model="omni", local_dir=".") # "omni", "audio", or "all"
186
+ model_dir = paths["omni"] # pathlib.Path; use paths["audio"] for Audio
 
 
 
 
 
187
  ```
188
 
189
+ The new layout loads each model using root `config.yaml` together with
190
+ the model directory's `config.json`. Keep the root configuration alongside
191
+ the model directories when downloading or moving them. The examples below
192
+ load `./Realtime-Venus-Omni` or `./Realtime-Venus-Audio`; run them from the
193
+ download directory, with asset and output paths relative to that directory.
194
 
195
+ As an independent alternative, the ModelScope CLI (installed separately) can
196
+ download the entire mirror repository:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
197
 
198
  ```bash
199
+ modelscope download --model inclusionAI/Realtime-Venus --local_dir .
200
  ```
201
 
202
+ This mirror command is separate from the Hugging Face downloader above.
 
 
 
 
 
 
203
 
204
  ## 7. 🎙️ Realtime-Venus-Omni Usages
205
 
README_zh.md CHANGED
@@ -91,166 +91,60 @@
91
 
92
  ## 6. 🛠️ 安装
93
 
94
- ### 6.1 新版仓库的使用方式
95
-
96
- 下载模型时,以仓库根目录为本地工作目录,保留根配置和所选模型的完整子目录。后续所有命令和示例均在这个工作目录中执行。
97
-
98
- ```text
99
- Realtime-Venus/
100
- ├── config.yaml # 两款模型的完整配置及目录映射
101
- ├── download_models.py # 统一下载入口
102
- ├── Realtime-Venus-Omni/ # 音视频模型,按需下载
103
- │ ├── config.json
104
- │ ├── requirements.txt
105
- │ ├── assets/
106
- │ └── … # 权重、自定义代码、Tokenizer、Processor 等
107
- └── Realtime-Venus-Audio/ # 纯音频模型,按需下载
108
- ├── config.json
109
- ├── assets/
110
- └── … # 权重、自定义代码、Tokenizer、Processor 等
111
- ```
112
-
113
- | 文件或目录 | 如何使用 |
114
- | --- | --- |
115
- | 根 `config.yaml` | 保存完整模型配置;统一下载器会自动获取,加载模型时由自定义配置代码读取 |
116
- | 各模型目录的 `config.json` | 供 Transformers 识别模型类型,并引用固定版本的根配置 |
117
- | `Realtime-Venus-Omni/` | 音视频理解、对话和全双工交互,从这个子目录加载模型 |
118
- | `Realtime-Venus-Audio/` | 音频理解、语音对话和全双工交互,从这个子目录加载模型 |
119
-
120
- ### 6.2 准备环境
121
-
122
- 推理需要 NVIDIA CUDA GPU。以下使用 Python 3.10、Conda 和 FFmpeg;请使用相互兼容的 PyTorch、CUDA 和显卡驱动。
123
 
124
  ```bash
125
- conda create -n realtime-venus python=3.10 -y
126
- conda activate realtime-venus
127
- conda install -c conda-forge ffmpeg -y
128
  python -m pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
129
-
130
- mkdir -p Realtime-Venus
131
- cd Realtime-Venus
132
  ```
133
 
134
- ### 6.3 按需下载模型
135
-
136
- 先下载统一下载脚本,并指定本节使用的模型版本。以下命令应在同一个终端中执行:
137
 
138
- ```bash
139
- MODEL_REVISION=03b5c2919dd684900bdf6852272ac6a8f30fab74
140
- hf download inclusionAI/Realtime-Venus download_models.py \
141
- --revision "$MODEL_REVISION" --local-dir .
142
- ```
143
 
144
- 根据需要,从下面三条命令中选择一条执行:
145
 
146
  ```bash
147
- # 只下载 Omni:用于音视频交互
148
- python download_models.py --model omni --local-dir . --revision "$MODEL_REVISION"
149
-
150
- # 只下载 Audio:用于纯音频交互
151
- python download_models.py --model audio --local-dir . --revision "$MODEL_REVISION"
152
-
153
- # 同时下载两款模型
154
- python download_models.py --model all --local-dir . --revision "$MODEL_REVISION"
155
  ```
156
 
157
- 下载器会先下载根 `config.yaml`,再根据其中的目录映射下载所选模型的完整目录,包括权重、自定义代码、Tokenizer、Processor、TTS 和演示资源。选择一款模型时,不会下载另一款模型的权重;以上命令也不会下载 `data/HEARTBEAT_Challenge/` 数据。
158
-
159
- 中断后,重新执行相同的下载命令即可复用已完成的文件。若需要更换本地保存位置,可修改 `--local-dir`;运行示例时,进入该目录,或相应调整模型和资源文件路径。
160
 
161
- ### 6.4 安装推理依赖
162
-
163
- 两款模型共用 `Realtime-Venus-Omni/requirements.txt`。如果下载了 Omni 或两款模型,直接安装:
164
 
165
  ```bash
166
  python -m pip install -r Realtime-Venus-Omni/requirements.txt
167
  ```
168
 
169
- 如果只下载了 Audio,先获取这一份依赖文件,再安装。该命令只会增加 Omni 目录下的 `requirements.txt`,不会下载 Omni 权重:
170
 
171
  ```bash
172
- hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt \
173
- --revision "$MODEL_REVISION" --local-dir .
174
  python -m pip install -r Realtime-Venus-Omni/requirements.txt
175
  ```
176
 
177
- ### 6.5 加载并使用模型
178
-
179
- #### Omni:从本地子目录加载
180
-
181
- 在 `Realtime-Venus/` 工作目录中,使用 Transformers 加载 `./Realtime-Venus-Omni`:
182
 
183
  ```python
184
- import torch
185
- from transformers import AutoModel
186
 
187
- model = AutoModel.from_pretrained(
188
- "./Realtime-Venus-Omni",
189
- trust_remote_code=True,
190
- local_files_only=True,
191
- attn_implementation="sdpa",
192
- torch_dtype=torch.bfloat16,
193
- ).eval().cuda()
194
  ```
195
 
196
- 加载后,第 7.2 节给出全双工视频交互示例,第 7.3 节给出完整视频的离线对话示例。全双工使用 `as_duplex()`、`prepare()`、`streaming_prefill()` 和 `streaming_generate()`;离线对话使用 `model.chat(...)`。按对应示例读取 `Realtime-Venus-Omni/assets/` 中的输入文件。
197
-
198
- #### Audio:运行一次音频问答
199
-
200
- 在同一工作目录中,将下面代码保存为 `run_audio.py`。它加载本地 Audio 模型,读取随模型下载的演示音频,并打印文本回答:
201
 
202
- ```python
203
- import librosa
204
- import torch
205
- from transformers import AutoModel, AutoTokenizer
206
-
207
- model_dir = "./Realtime-Venus-Audio"
208
- tokenizer = AutoTokenizer.from_pretrained(
209
- model_dir,
210
- trust_remote_code=True,
211
- local_files_only=True,
212
- fix_mistral_regex=True,
213
- )
214
- model = AutoModel.from_pretrained(
215
- model_dir,
216
- trust_remote_code=True,
217
- local_files_only=True,
218
- attn_implementation="sdpa",
219
- torch_dtype=torch.bfloat16,
220
- init_vision=False,
221
- init_audio=True,
222
- init_tts=False,
223
- ).eval().cuda()
224
-
225
- audio, _ = librosa.load(
226
- "Realtime-Venus-Audio/assets/case_offline.wav", sr=16000, mono=True
227
- )
228
- answer = model.chat(
229
- msgs=[
230
- {"role": "system", "content": "You are a helpful assistant."},
231
- {"role": "user", "content": [audio, "What is the speaker asking about?"]},
232
- ],
233
- tokenizer=tokenizer,
234
- do_sample=False,
235
- max_new_tokens=2048,
236
- enable_thinking=False,
237
- use_tts_template=True,
238
- generate_audio=False,
239
- )
240
- print(answer)
241
- ```
242
 
243
  ```bash
244
- python run_audio.py
245
  ```
246
 
247
- 该示例只输出文本,因此设置 `init_tts=False`。需要语音输出或全双工交互时,按照第 8 节的加载和调用示例启用 `init_tts=True`。
248
-
249
- ### 6.6 离线使用与目录搬移
250
-
251
- 在联网环境中完成模型下载和依赖安装后,可以按上述本地路径加载。复制到其他机器时,将根 `config.yaml` 与所需的完整模型目录一起复制,保持 6.1 中的相对位置,并为目标机器安装对应依赖。模型参数中的 `local_files_only=True` 使用本地文件。
252
-
253
- 各模型目录的 `config.json` 通过 `venus_config_ref` 引用根配置版本 `15d5b420cd8557e9582d41a57a80af3e4de18e47`,并校验其 SHA-256;统一下载器已获取对应的根配置。使用模型时,将 `from_pretrained` 指向模型子目录,例如 `./Realtime-Venus-Audio`。
254
 
255
  ## 7. 🎙️ Realtime-Venus-Omni 使用方法
256
 
 
91
 
92
  ## 6. 🛠️ 安装
93
 
94
+ 运行推理示例需要 Python 3.10、CUDA 和 FFmpeg。首先安装下载依赖,并从本 Hugging Face 仓库获取统一下载脚本:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95
 
96
  ```bash
 
 
 
97
  python -m pip install 'huggingface_hub>=0.34' 'PyYAML>=6.0'
98
+ hf download inclusionAI/Realtime-Venus download_models.py --local-dir .
 
 
99
  ```
100
 
101
+ 然后选择需要下载的模型:
 
 
102
 
103
+ | `--model` | 下载内容 |
104
+ | --- | --- |
105
+ | `omni` | 用于音视频交互的 Realtime-Venus-Omni |
106
+ | `audio` | 用于音频理解与语音对话的 Realtime-Venus-Audio |
107
+ | `all` | 两款模型 |
108
 
109
+ 例如,将两款模型下载到当前目录:
110
 
111
  ```bash
112
+ python download_models.py --model all --local-dir .
 
 
 
 
 
 
 
113
  ```
114
 
115
+ 如只需一款模型,将参数改为 `--model omni` 或 `--model audio`。下载器先获取根 `config.yaml`(包含两款模型的完整配置和目录映射),再下载所选模型的完整子目录,包括权重、自定义代码和资源文件。单次下载中的所有文件均来自同一个仓库版本,并使用 Hugging Face Hub 的标准进度显示与缓存。
 
 
116
 
117
+ 下载后安装推理依赖。选择 Omni 或 `all` 时运行:
 
 
118
 
119
  ```bash
120
  python -m pip install -r Realtime-Venus-Omni/requirements.txt
121
  ```
122
 
123
+ Audio 使用相同的已发布依赖清单,没有独立的 `requirements.txt`。如果只下载了 Audio,先单独获取这份小文件,无需下载 Omni 权重:
124
 
125
  ```bash
126
+ hf download inclusionAI/Realtime-Venus Realtime-Venus-Omni/requirements.txt --local-dir .
 
127
  python -m pip install -r Realtime-Venus-Omni/requirements.txt
128
  ```
129
 
130
+ 也可以在包含 `download_models.py` 的目录中,通过 Python 完成一次下载准备。它与上述命令使用同一个下载入口:
 
 
 
 
131
 
132
  ```python
133
+ from download_models import download_models
 
134
 
135
+ paths = download_models(model="omni", local_dir=".") # 可选 "omni"、"audio" 或 "all"
136
+ model_dir = paths["omni"] # pathlib.Path;Audio 使用 paths["audio"]
 
 
 
 
 
137
  ```
138
 
139
+ 新版通过根 `config.yaml` 与模型子目录中的 `config.json` 配合加载。下载或搬移模型后,请保留根配置与模型目录的相对位置。下文示例分别从 `./Realtime-Venus-Omni` 或 `./Realtime-Venus-Audio` 加载模型;请在下载目录中运行,资源文件和输出路径均相对于该目录。
 
 
 
 
140
 
141
+ 也可以独立使用 ModelScope CLI(需另行安装)下载整个镜像仓库:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
142
 
143
  ```bash
144
+ modelscope download --model inclusionAI/Realtime-Venus --local_dir .
145
  ```
146
 
147
+ 该镜像命令与上述 Hugging Face 下载器相互独立。
 
 
 
 
 
 
148
 
149
  ## 7. 🎙️ Realtime-Venus-Omni 使用方法
150