MiDashengLM-Gen

Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching

arXiv  HuggingFace Model  Demo Page  GitHub

English | δΈ­ζ–‡

MiDashengLM-Gen is an end-to-end framework that uses a pre-trained Large Language Model and audio tokenizer as the backbone, combined with per-token conditional flow matching for autoregressive, variable-length mixed-audio scene generation. It generates coherent 16 kHz audio scenes that simultaneously blend speech, music, sound effects, and environmental acoustics from structured text descriptions.

Architecture

Architecture

Left: training pipeline with flow matching loss. Right: autoregressive inference pipeline.

Key components:

Component Details
Audio Tokenizer DashengTokenizer β†’ 768-dim latents @ 25 Hz, downsampled to 5 Hz via audio projector
LLM Backbone Qwen3-1.7B, fully fine-tuned
Flow Matching DiT 16 layers, hidden dim 2048, 8 heads, MLP ratio 4.0
Inference 10-step Euler ODE solver, CFG scale 2.0
Stop Head Learned binary classifier for variable-length truncation
Output 16 kHz mono WAV, auto-truncated

Input Format

Input uses structured multi-view captions with special tokens to describe different aspects of an audio scene. Use <|unknown|> for absent elements.

Token Description Example
<|caption|> Overall scene description A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit
<|asr|> Speech transcript And that is why I never buy cheap luggage anymore!
<|speech|> Speaker characteristics (voice, emotion, style) expressive comedic male voice
<|sfx|> Sound effects uproarious crowd laughter
<|music|> Music description sudden upbeat jazz band sting
<|env|> Environment / ambience intimate comedy club

Installation

pip install torch torchaudio "transformers>=4.51" einops safetensors soundfile tqdm numpy x-transformers

Quick Start

from transformers import AutoModel
import soundfile as sf

model = AutoModel.from_pretrained("mispeech/midashenglm-gen", trust_remote_code=True)
model = model.cuda()

result = model.generate(
    "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit "
    "<|asr|> And that is why I never buy cheap luggage anymore! "
    "<|speech|> expressive comedic male voice "
    "<|music|> sudden upbeat jazz band sting "
    "<|sfx|> uproarious crowd laughter "
    "<|env|> intimate comedy club"
)

sf.write("output.wav", result["audio"], result["sample_rate"])

Batch Generation

texts = [
    "<|caption|> A comedian delivering a punchline followed by uproarious crowd laughter and an upbeat jazz band hit <|asr|> And that is why I never buy cheap luggage anymore! <|speech|> expressive comedic male voice <|music|> sudden upbeat jazz band sting <|sfx|> uproarious crowd laughter <|env|> intimate comedy club",
    "<|caption|> Upbeat instrumental music with occasional whistling and low audio quality. <|asr|> <|unknown|> <|speech|> <|unknown|> <|sfx|> Intermittent whistling emerges in later segment alongside persistent instrumental playback. <|music|> Bright melodic composition with guitar, piano, and rhythmic percussion elements, maintaining a consistently upbeat character. <|env|> Low-fidelity recording with compressed dynamics and limited frequency response.",
]
result = model.generate(texts)
for i, audio in enumerate(result["audio"]):
    sf.write(f"output_{i}.wav", audio, result["sample_rate"])

Generation Parameters

result = model.generate(
    "...",
    eval_cfg=2.0,           # Classifier-free guidance scale
    stop_threshold=0.5,     # Stop prediction threshold
    min_stop_step=5,        # Minimum steps before stopping
    seed=42,                # Random seed for reproducibility
)
Parameter Default Description
eval_cfg 2.0 Classifier-free guidance scale
stop_threshold 0.5 Stop probability threshold for audio truncation
min_stop_step 5 Minimum generation steps before stopping
seed None Random seed for reproducibility

Citation

@article{sun2026midashenglmgen,
  title={MiDashengLM-Gen: Unified Audio Scene Generation via LLM-Driven Autoregressive Flow Matching},
  author={Sun, Xingwei and Dinkel, Heinrich and Li, Gang and Mei, Jiahao and Niu, Yadong and Han, Zerui and Jiang, Yuepeng and Zhou, Jiahao and Fan, Lichun and Luan, Jian},
  journal={arXiv preprint},
  eprint={2608.11804},
  archivePrefix={arXiv},
  year={2026}
}

License

Apache 2.0

Use Restrictions

You are solely responsible for your use of MiDashengLM-Gen and any outputs, actions, or consequences arising therefrom, and you agree not to use MiDashengLM-Gen or any derivatives thereof:

  • For any unlawful, fraudulent, or malicious purpose, or in any manner that violates any applicable laws or regulations;
  • To infringe upon the intellectual property rights, privacy rights, publicity rights, or other lawful rights or interests of any third party;
  • To exploit, harm, harass, defame, unlawfully discriminate against, or otherwise adversely affect any individual or group, including minors or vulnerable persons;
  • For any military purpose or application.
Downloads last month
242
Safetensors
Model size
3B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for mispeech/midashenglm-gen

Adapters
1 model

Spaces using mispeech/midashenglm-gen 3

Collection including mispeech/midashenglm-gen

Paper for mispeech/midashenglm-gen