Tachyeon/diffusion-asr-30h

Diffusion ASR — trained adapter + LoRA (30h)

Trained weights for a speech-recognition system that pairs a frozen audio encoder with a frozen block-diffusion language model, bridged by a small trained adapter and LoRA.

These are deltas, not a standalone model. This repo contains only the ~5M-parameter adapter and the ~17M-parameter LoRA. To run anything you also need the two frozen base models (below) and the code from the project repo: https://github.com/ipritamdash/diffusion-asr-research

What's in this repo

File	Contents	Params	sha256 (first 16)
ctc_sft30h_stair_best.pt	audio adapter (incl. gain) + logit-scale log_s	5,250,050	fe982b73182c5516
lora30h_stair_best.pt	LoRA on the decoder (PEFT state dict, 392 tensors)	17,432,576	d80455b2c66e3f1e

ctc_sft30h_stair_best.pt keys: adapter, log_s, epoch (40), plus dev metrics. lora30h_stair_best.pt keys: lora, epoch (24), dev_ce (0.5147).

Base models (frozen — download separately)

Role	Repo	Pinned revision
Decoder (block-diffusion LM)	JetLM/SDAR-1.7B-Chat	97cf0dc9a7e85433f3f71a6bbac6fec9d381294f
Encoder (audio tower)	Qwen/Qwen3-ASR-0.6B	5eb144179a02acc5e5ba31e748d22b0cf3e303b0

Use these exact revisions — the weights here were trained against them.

The base decoder must be loaded through the project repo's patched loader (qwen_sdar/sdar.py::load_patched_sdar), which swaps the CUDA-only flash-attn for SDPA and adds the embeddings-input path. Loading stock JetLM/SDAR-1.7B-Chat with plain from_pretrained will not reproduce these results.

Architecture

audio → Qwen3-ASR-0.6B encoder → adapter → SDAR-1.7B decoder + LoRA → text
        frozen                   trained    frozen          trained

• Adapter (Adapter in qwen_sdar/adapter.py): Conv1d(1024→1024, k=3, stride=2, pad=1) → GELU → Linear(1024→2048) → LayerNorm(2048) → ×gain. Per-frame; the stride-2 conv pools the encoder's 12.5 Hz features to 6.25 Hz. gain is initialised to the RMS of the decoder's token embeddings so the prefix matches their scale.
• LoRA (PEFT): r=16, alpha=32, dropout=0, on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
• Trained with block-causal ("staircase") attention at every stage, so the model can be served on fast inference engines (see the project repo, engine/).

Training

• Data: a 30-hour subset of LibriSpeech train-clean-100, with a 300-clip held-out dev split.
• Staged, never joint: grounding (adapter via CTC, decoder detached) → SFT (adapter through the frozen decoder) → LoRA (adapter frozen).
• Only ~1% of the ~2.5B-parameter system is trained (encoder 0.78B + decoder 1.7B; ~22M trained).

Results (held-out dev-300)

WER %, lower is better. raw = macro WER; norm = normalized corpus WER (Whisper normalizer).

Decode	raw / norm
block-2 (best)	6.96 / 6.48
block-4	8.24 / 7.69

Also served on LMDeploy's PyTorch diffusion engine at parity (block-2: 7.33 / 6.71, within the decoder's kernel noise), at RTF 0.0013 (~800× real time, block-4 decode) under batching on one L40S.

Note: dev-300 is one in-domain held-out subset — not the standard test-clean/test-other benchmark, and not evaluated on multiple datasets.

How to use

Clone the project repo (github.com/ipritamdash/diffusion-asr-research) and run inference in two steps, in two environments — the encoder (qwen-asr) and the decoder (transformers) have conflicting dependencies, so they don't share one env:

# 1. encoder env — audio -> features
pip install -r requirements-encoder.txt
python featurize.py --audio clip.wav --out features.pt

# 2. decoder env — features -> transcript (loads the adapter + LoRA from this repo)
pip install -r requirements-decode.txt
python decode.py --features features.pt \
  --adapter ctc_sft30h_stair_best.pt --lora lora30h_stair_best.pt --block_length 2

Both steps need a CUDA GPU. For fast serving instead, see engine/ (merge the LoRA with engine/merge_lora.py, then run on LMDeploy).

Decode config: block_length=2 (best) or 4, denoising_steps=4, confidence_threshold=0.95, greedy, with top-1 fallback and EOS truncation.

Intended use & limitations

• Scope: a research model for English read speech (LibriSpeech domain). Not evaluated for streaming, other languages, or noisy/conversational audio.
• In-domain, tuned set: the 6.96 / 6.48 dev-300 numbers are on a held-out split of the same train-clean-100 pool the model was tuned on — read them as an in-domain result, not a public benchmark.
• Out-of-domain degrades sharply: the earlier 10h version scored ~16% on test-other and ~40% on VoxPopuli; expect large degradation off clean read speech.
• No standard-benchmark number for this 30h model: test-clean / test-other were run only for a separate 10h proof-of-concept, not this model — do not read 6.48 as a test-clean score.

Licensing

The adapter and LoRA weights here are trained deltas. Any use depends on the two frozen base models, which are governed by their own licenses — check JetLM/SDAR-1.7B-Chat and Qwen/Qwen3-ASR-0.6B before any redistribution, and do not redistribute a merged checkpoint (which would embed the base weights) without confirming their terms.