Ming-3.0-audio-flash is a native audio-language model from inclusionAI. It accepts audio together with text instructions and supports speech recognition, spoken question answering, audio understanding, audio reasoning, and optional synthesized-speech responses.
## Model Summary
| Property | Description |
| --- | --- |
| Developer | inclusionAI |
| Model type | Native audio-language Mixture-of-Experts model |
| Parameters | 126B total parameters, 6.5B activated per inference step |
| Inputs | Audio and text instructions |
| Outputs | Text, or text with optional synthesized speech |
| Primary languages | Chinese and English, including varied accents and dialects |
| License | MIT |
## Highlights
- **Complex speech recognition:** Handles Chinese and English speech, dialects, accents, noisy environments, and multi-speaker recordings.
- **Long-audio understanding:** Comprehends and analyzes long-form recordings.
- **Multidimensional perception:** Understands speech, non-speech sounds, acoustic scenes, and music.
- **Audio reasoning:** Answers questions and performs reasoning over audio content.
- **Audio interaction:** Produces text responses and can optionally synthesize speech with built-in voices or a reference voice.
## Architecture
Ming-3.0-audio-flash combines an audio encoder and projector with a Mixture-of-Experts language backbone. An optional Talker generates speech responses from the model's output.
## Quick Start with Transformers
The model uses custom Transformers code. Review the repository code before enabling `trust_remote_code=True`.
```bash
git clone https://github.com/inclusionAI/Ming-3.0-audio-flash.git
cd Ming-3.0-audio-flash
python -m venv .venv-transformers
source .venv-transformers/bin/activate
pip install -r requirements-transformers.txt
```
### Audio Understanding
```bash
python examples/transformers_audio.py examples/audio/introduce_ant_group.wav \
--model inclusionAI/Ming-3.0-audio-flash \
--prompt "Please answer the question in the audio."
```
### Speech Recognition
```bash
python examples/transformers_audio.py examples/audio/introduce_ant_group.wav \
--model inclusionAI/Ming-3.0-audio-flash \
--task asr \
--max-new-tokens 256 \
--no-repeat-ngram-size 10
```
The validated ASR prompt asks the model to identify the spoken language and place a tab between the language label and transcription.
### Direct Python Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor
model_id = "inclusionAI/Ming-3.0-audio-flash"
audio_path = "examples/audio/introduce_ant_group.wav"
processor = AutoProcessor.from_pretrained(
model_id,
trust_remote_code=True,
use_fast=False,
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
).eval()
messages = [
{
"role": "user",
"content": [
{"type": "audio", "audio": audio_path},
{"type": "text", "text": "Please answer the question in the audio."},
],
}
]
inputs = processor(messages).to(model.device)
inputs["audio_feats"] = inputs["audio_feats"].to(model.dtype)
with torch.inference_mode():
output_ids = model.generate(
**inputs,
max_new_tokens=512,
do_sample=False,
)
generated_ids = output_ids[0, inputs["input_ids"].shape[1] :]
response = processor.decode(generated_ids, skip_special_tokens=True).strip()
print(response)
```
## Speech Output
The Talker example supports built-in voices and reference-audio voice prompting:
```bash
python examples/transformers_thinker_talker.py \
--model inclusionAI/Ming-3.0-audio-flash \
--audio examples/audio/introduce_ant_group.wav \
--voice US_Stella \
--output output.wav
```
See the [voice guide](https://github.com/inclusionAI/Ming-3.0-audio-flash/blob/main/docs/voices.md) for available voice names and reference-audio usage.
## Serving
### vLLM-Omni
```bash
export MODEL_PATH=inclusionAI/Ming-3.0-audio-flash
MING_OUTPUT_MODE=text ./scripts/serve_vllm_omni.sh "$MODEL_PATH"
```
The reference 128K serving profile uses tensor parallelism across eight 80 GB H800 GPUs. Select the deployment YAML that matches text-only or Talker output. See the [deployment guide](https://github.com/inclusionAI/Ming-3.0-audio-flash/blob/main/README.md#vllm-omni) for details.
For speech output, start the audio profile and select an IP or registered voice with the vLLM-Omni client's `--voice` option:
```bash
MING_OUTPUT_MODE=audio ./scripts/serve_vllm_omni.sh "$MODEL_PATH"
python examples/vllm_omni_audio_chat.py examples/audio/introduce_ant_group.wav \
--audio-response \
--voice US_Stella \
--output-audio outputs/answer.wav
```
The client sends this selection as Chat Completions `audio.voice: {"id": "..."}`. vLLM-Omni uses the built-in IP voice `灵小柔` when no voice is specified. The model directory also provides `灵楷` and `US_James` as preset reference voices under `voice_presets/voice_audio/`, with their matching transcripts in `voice_presets/voice_prompts.json`. Both names are also available as built-in IP voices.
### SGLang
```bash
export MODEL_PATH=inclusionAI/Ming-3.0-audio-flash
./scripts/serve_sglang.sh "$MODEL_PATH"
```
The validated SGLang profile uses four GPUs and provides text output. See the [SGLang example](https://github.com/inclusionAI/Ming-3.0-audio-flash/blob/main/examples/sglang_audio_chat.py) for client usage.
### SGLang-Omni
[SGLang-Omni](https://sgl-project.github.io/sglang-omni/) serves transcription, audio chat, and speech synthesis from one server. Serving uses two GPUs: `--thinker.tp_size 2` places the thinker on both visible devices, the audio encoder uses GPU 0, and speech synthesis uses GPU 1.
```bash
CUDA_VISIBLE_DEVICES=0,1 sgl-omni serve \
--model-path inclusionAI/Ming-3.0-audio-flash \
--thinker.tp_size 2
```
The server exposes OpenAI-compatible `/v1/audio/transcriptions`, `/v1/audio/speech`, and `/v1/chat/completions` endpoints. See the [SGLang-Omni cookbook](https://sgl-project.github.io/sglang-omni/cookbook/ming3_audio.html) for client examples, including voice cloning with `ref_audio` and `ref_text`.
## Evaluation
### Speech Recognition
### Audio Intelligence
### Text Benchmarks
All benchmark values are percentages. Higher is better unless the metric is marked with a downward arrow. Results marked with an asterisk are self-evaluated. Refer to the [GitHub README](https://github.com/inclusionAI/Ming-3.0-audio-flash#benchmarks) for the latest notes and evaluation details.
## License
This project is released under the [MIT License](https://github.com/inclusionAI/Ming-3.0-audio-flash/blob/main/LICENSE). See [LEGAL.md](https://github.com/inclusionAI/Ming-3.0-audio-flash/blob/main/LEGAL.md) for the legal disclaimer.
GitHub
