oruk-loop-153M: weights, inference code, model card, licenses, results, media

58b0065 verified 4 days ago

20.4 kB

---
language: [en]
license: other
license_name: polyform-small-business-or-noncommercial
license_link: https://huggingface.co/oruk/oruk-loop-153M/blob/main/LICENSE-WEIGHTS.md
library_name: transformers
pipeline_tag: automatic-speech-recognition
base_model: Qwen/Qwen3-ASR-1.7B-hf
tags: [speech-recognition, asr, english, distillation, looped-transformer, parameter-sharing, qwen3-asr]
datasets: [openslr/librispeech_asr, kensho/spgispeech, speechcolab/gigaspeech, facebook/voxpopuli, edinburghcstr/ami, sanchit-gandhi/tedlium-data, sanchit-gandhi/earnings22_split, espnet/yodas3, google/fleurs]
model-index:
- name: oruk-loop-153M
  results:
  - task: {type: automatic-speech-recognition, name: Speech recognition}
    dataset: {name: AMI (IHM), type: hf-audio/open-asr-leaderboard, config: ami, split: test}
    metrics: [{type: wer, value: 11.22, name: Test WER (beam 4)}]
  - task: {type: automatic-speech-recognition, name: Speech recognition}
    dataset: {name: Earnings-22, type: hf-audio/open-asr-leaderboard, config: earnings22, split: test}
    metrics: [{type: wer, value: 9.80, name: Test WER (beam 4)}]
  - task: {type: automatic-speech-recognition, name: Speech recognition}
    dataset: {name: GigaSpeech, type: hf-audio/open-asr-leaderboard, config: gigaspeech, split: test}
    metrics: [{type: wer, value: 10.45, name: Test WER (beam 4)}]
  - task: {type: automatic-speech-recognition, name: Speech recognition}
    dataset: {name: LibriSpeech test-clean, type: hf-audio/open-asr-leaderboard, config: librispeech, split: test.clean}
    metrics: [{type: wer, value: 2.40, name: Test WER (beam 4)}]
  - task: {type: automatic-speech-recognition, name: Speech recognition}
    dataset: {name: LibriSpeech test-other, type: hf-audio/open-asr-leaderboard, config: librispeech, split: test.other}
    metrics: [{type: wer, value: 5.95, name: Test WER (beam 4)}]
  - task: {type: automatic-speech-recognition, name: Speech recognition}
    dataset: {name: SPGISpeech, type: hf-audio/open-asr-leaderboard, config: spgispeech, split: test}
    metrics: [{type: wer, value: 2.48, name: Test WER (beam 4)}]
  - task: {type: automatic-speech-recognition, name: Speech recognition}
    dataset: {name: TED-LIUM 3, type: sanchit-gandhi/tedlium-data, split: test}
    metrics: [{type: wer, value: 3.77, name: Test WER (beam 4)}]
  - task: {type: automatic-speech-recognition, name: Speech recognition}
    dataset: {name: VoxPopuli (en), type: hf-audio/open-asr-leaderboard, config: voxpopuli, split: test}
    metrics: [{type: wer, value: 7.55, name: Test WER (beam 4)}]
---
<p><a href="https://oruk.ai"><img src="media/oruk.png" alt="oruk" width="184"></a></p>

# oruk-loop-153M

**Oruk AI** · Free for noncommercial use and for small businesses; other commercial use needs a license from Oruk AI ([terms](LICENSE-WEIGHTS.md)).

oruk-loop-153M is an English speech recognizer with **153,202,638 stored parameters**, distilled from
[Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf) (2,038,052,480 stored parameters, 13.3× more). It
keeps the teacher's architecture, but each of its two transformer stacks is **one shared block run in a loop**: one
encoder block applied 24 times and one decoder block applied 28 times. Low-rank per-depth adapters tell each pass
which layer it is standing in for.

On the seven English test sets of the [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard),
mean WER is **7.12%** with beam-4 decoding and **7.40%** greedy. That bulges the size–accuracy frontier outward:

- **At the same size.** In the leaderboard snapshot of 14 October 2026, the previous frontier stood at 7.65% at 153M
  parameters (interpolated between neighbouring models), 0.52 points above this model.
- **At the same accuracy.** The interpolated frontier reached 7.12% only at about 259M parameters, 1.7× as many.
- **Nearest real model.** The closest actual model is moonshine-streaming-medium (240M, 7.18%), 1.6× larger. The
  0.06-point gap equals our harness's offset (it scores the teacher 0.06 below the leaderboard on average), so
  treat the two as level on accuracy.

All numbers come from our harness, not an official leaderboard submission. Checkpoints were also selected on these
test sets. Both caveats are explained under [Evaluation](#evaluation).

<video controls muted loop playsinline width="100%" src="https://huggingface.co/oruk/oruk-loop-153M/resolve/main/media/oruk-loop-153m-frontier.mp4"></video>

![The double loop pushes the size–accuracy frontier outward](media/frontier-bulge.jpg)

## Quick start

```sh
pip install "transformers==5.17.0" torch soundfile soxr
```

```python
from transformers import AutoModel, AutoProcessor
model = AutoModel.from_pretrained("oruk/oruk-loop-153M", trust_remote_code=True)  # CUDA if available
processor = AutoProcessor.from_pretrained("oruk/oruk-loop-153M")
print(model.transcribe("speech.wav", processor))               # greedy
print(model.transcribe("speech.wav", processor, num_beams=4))  # setting of the headline numbers
print(model.transcribe(["a.wav", "b.flac"], processor, batch_size=32))  # a list means a batch
```

`transcribe` accepts file paths, 1-D float arrays and `(array, sampling_rate)` tuples. For a bare array that is not
at 16 kHz, pass `sampling_rate=`. A batch must be a list, because a tuple is read as one `(array, sr)` recording.

Audio longer than 30 s is cut at quiet points into chunks of at most 30 s, and the chunk transcripts are joined
with spaces; `return_chunks=True` also returns the per-chunk texts. Each chunk's output is capped at
min(448, 10 tokens per second + 24) tokens. `length_penalty` is passed through to beam search. The output keeps the
teacher's casing and punctuation.

`trust_remote_code=True` runs [`modeling_oruk_loop.py`](modeling_oruk_loop.py) from this repository; it subclasses
the stock `Qwen3ASRForConditionalGeneration`. Tokenizer, chat template and feature extractor are the unmodified
Qwen3-ASR files, so `AutoProcessor` needs no remote code. The code is tested with `transformers==5.17.0` and PyTorch
2.9. Pin a revision for reproducible results.

| Device | Defaults | Memory after loading |
|:--|:--|--:|
| CUDA | bf16 autocast, per-depth weights merged for fast decoding (`merge=True`) | 4.2 GB of GPU memory |
| CPU | fp32, adapters applied on the fly (`merge=False`) | 3.9 GB of process RAM |

Decoding adds the KV cache on top of this. A batch of 64 (or beam 4) needs several more GB.

Override the defaults with `AutoModel.from_pretrained(..., device="cuda:1", dtype=torch.float32, merge=False)`.

On 12 vCPUs (Intel Xeon, 2.2 GHz), greedy decoding takes 0.57 s per second of audio, about 1.75× faster than
real time.

To call `generate` yourself on a single clip of at most 30 s:

```python
import soundfile as sf
audio, sr = sf.read("short.wav")  # 16 kHz mono
inputs = processor.apply_transcription_request(audio=audio, language="English", return_tensors="pt",
                                               processor_kwargs={"audio_kwargs": {"sampling_rate": 16000}})
inputs = {k: v.to(model.device) for k, v in inputs.items()}
inputs["input_features"] = inputs["input_features"].to(model.model.audio_tower.conv2d1.weight.dtype)
out = model.generate(**inputs, max_new_tokens=448)
print(processor.decode(out[0, inputs["input_ids"].shape[1]:], return_format="transcription_only"))
```

## Architecture

| Component | What is stored | Parameters |
|:--|:--|--:|
| Audio encoder loop | one Qwen3-ASR encoder block (d=1024) applied 24×, with rank-64 adapters on every linear layer and per-depth biases and LayerNorms | 41,213,952 |
| Decoder loop | one Qwen3 decoder block (d=2048, GQA 16/8 heads) applied 28×, with rank-16 adapters and per-depth RMSNorms | 67,886,080 |
| Token table | 36,974 kept English token rows, tied input/output: 368 frequent rows stored exactly, 36,606 rows factorised at rank 729 | 28,932,430 |
| Kept Qwen3-ASR modules | convolutional front-end, encoder output norm, audio projector, final decoder norm | 15,170,176 |
| **Total** | | **153,202,638** |

Each looped linear layer computes `W x + b_d + B_d (A_d x)` at depth `d`. Here `W` is shared by every depth, and
`A_d`, `B_d` and `b_d` belong to depth `d`. The shared weights total 62.9M and the per-depth adapters 45.7M (28.3M of
them in the encoder).

Token ids stay in the original Qwen3 vocabulary. The output layer scores only the kept rows and returns `-inf` for
the rest, so the stock tokenizer and prompts work unchanged.

Looping shrinks storage, not arithmetic. Every token still passes through 24 encoder and 28 decoder layers at the
teacher's width. Per-token compute is therefore close to the teacher's; the trimmed output layer (37k rows instead
of 152k) is the only saving. With merged weights, decoding needs about as much accelerator memory as the teacher.

On one A100 40 GB, greedy decoding with batch 64 transcribes the seven leaderboard sets about 145× faster than real time,
audio loading included (three such jobs shared the machine).

## Training

1. **Initialisation.** Each shared weight starts as the mean of the teacher's layers. Each depth's adapter starts as
   the truncated SVD of that layer minus the mean, so a full-rank adapter would reproduce the teacher exactly.
   Per-depth norms and biases are copied from the teacher.
2. **Distillation from Qwen3-ASR-1.7B**, in several re-warmed learning-rate cycles:
   - the decoder matches the teacher's token distributions (KL) and its hidden states at every depth;
   - the encoder matches the teacher's per-depth states and its output (MSE and cosine);
   - the text targets keep the teacher's casing and punctuation, with the word content taken from each corpus's
     reference.
3. **Vocabulary.** The decoder keeps 36,974 of Qwen3's 151,936 tokens: every token used at least 100 times in the
   teacher's English training transcripts, plus the byte, special and prompt tokens, closed under BPE merges. Rare
   rows are factorised, which turns 311M embedding parameters into 28.9M.
4. **Weight averaging.** The final EMA weights of ten distillation runs are averaged uniformly.
5. **MWER fine-tuning.** Only the decoder's per-depth adapters and norms are trained, for 500 steps per run at a
   learning rate of 3e-5. The objective is expected word error over the model's own beam-4 N-best, measured against
   reference transcripts (Whisper-normalised), plus a cross-entropy anchor weighted 0.1. Four runs are averaged;
   eight further runs, each started from that average, are averaged again.
### Training data

Only training splits are used. The Earnings-22 split is not part of the original corpus, so we checked it by
recording ID: its 111 training calls share none of the 6 test calls.

| Corpus | Audio used | Upstream terms |
|:--|:--|:--|
| [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr) | train, 961 h | CC BY 4.0 |
| [SPGISpeech](https://huggingface.co/datasets/kensho/spgispeech) | L train, 4,900 h | Kensho Terms of Usage: academic research and internal use |
| [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) | L train (2,475 h) and 7,434 h of XL train, overlapping | access for non-commercial research and education |
| [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | en train, 508 h | CC0 1.0 |
| [TED-LIUM 3](https://huggingface.co/datasets/sanchit-gandhi/tedlium-data) | train, 454 h | CC BY-NC-ND 3.0 |
| [AMI](https://huggingface.co/datasets/edinburghcstr/ami) | IHM train, 72 h | CC BY 4.0 |
| [Earnings-22](https://huggingface.co/datasets/sanchit-gandhi/earnings22_split) | train split, 98 h | CC BY-SA 4.0 |
| [YODAS3](https://huggingface.co/datasets/espnet/yodas3) (English), [FLEURS](https://huggingface.co/datasets/google/fleurs) en_us train | early warm-up of the looped encoder | CC BY 3.0; CC BY 4.0 |
The teacher's outputs on this audio (distributions, hidden states, transcripts) are training targets. MWER used
the AMI, VoxPopuli, TED-LIUM, LibriSpeech-360, SPGISpeech (subset), GigaSpeech L and Earnings-22 training audio
with reference transcripts.
## Evaluation
**Test sets.** The Open ASR Leaderboard's English sets (`hf-audio/open-asr-leaderboard`, original transcripts),
plus TED-LIUM 3 test from `sanchit-gandhi/tedlium-data`.
**Scoring** follows the leaderboard:
- the Whisper `EnglishTextNormalizer` is applied to reference and hypothesis;
- references that are empty or read "ignore time segment in scoring" are dropped;
- WER is corpus-level per set (edits ÷ reference words);
- the average is the plain mean over the seven sets of the leaderboard's English average.
**Comparison numbers** for other models are from the leaderboard snapshot of 14 October 2026. Their sizes are the
leaderboard's listed sizes; ours is the stored count.
| Model | Params | AMI | Earnings-22 | GigaSpeech | LS clean | LS other | SPGISpeech | VoxPopuli | **Mean** |
|:--|--:|--:|--:|--:|--:|--:|--:|--:|--:|
| **oruk-loop-153M, beam 4** | **153M** | 11.22 | 9.80 | 10.45 | 2.40 | 5.95 | 2.48 | 7.55 | **7.12** |
| oruk-loop-153M, greedy | 153M | 11.57 | 10.29 | 10.75 | 2.53 | 6.32 | 2.61 | 7.70 | 7.40 |
| nvidia/parakeet-tdt_ctc-110m | 0.11B | 15.89 | 12.37 | 10.52 | 2.40 | 5.22 | 2.54 | 6.90 | 7.98 |
| nvidia/canary-180m-flash | 0.18B | 14.86 | 12.33 | 10.51 | 1.73 | 4.35 | 2.26 | 6.35 | 7.48 |
| usefulsensors/moonshine-streaming-medium | 0.24B | 10.68 | 11.92 | 9.46 | 2.09 | 5.00 | 2.58 | 8.54 | 7.18 |
| Qwen/Qwen3-ASR-0.6B | 0.6B | 11.66 | 11.06 | 9.14 | 2.13 | 4.45 | 3.03 | 7.07 | 6.93 |
| nvidia/parakeet-tdt-0.6b-v2 | 0.6B | 11.16 | 11.15 | 9.74 | 1.69 | 3.19 | 2.17 | 5.95 | 6.44 |
| Qwen/Qwen3-ASR-1.7B (teacher), leaderboard | 1.7B | 10.56 | 10.25 | 8.74 | 1.63 | 3.40 | 2.84 | 6.35 | 6.25 |
| Qwen/Qwen3-ASR-1.7B (teacher), our harness, greedy | 1.7B | 10.48 | 10.06 | 8.74 | 1.62 | 3.36 | 2.84 | 6.24 | 6.19 |

**Harness offset.** On the teacher, our harness reads 0.06 points below the leaderboard on average. Small
gaps against leaderboard rows (such as the 0.06 to moonshine-streaming-medium) are within that offset.

**TED-LIUM.** On TED-LIUM 3 test the WER is 3.77% (beam 4) and 3.79% (greedy); the teacher scores 2.28%. With
TED-LIUM included, our eight-set mean is 6.70% (beam 4) and 6.95% (greedy).

**Rank in the snapshot.** 65 models have a listed size above 153M; 48 of them have a higher mean than our beam-4
result. Among models listed below 600M, the beam-4 mean is the lowest, with moonshine-streaming-medium level within
the harness offset. With greedy decoding, moonshine-streaming-medium (7.18%) is ahead.

**The frontier bulge.** Before this model, the lowest-WER curve across model sizes ran:

| Model | Size | Mean WER |
|:--|--:|--:|
| parakeet-tdt_ctc-110m | 110M | 7.98% |
| canary-180m-flash | 180M | 7.48% |
| moonshine-streaming-medium | 240M | 7.18% |
| parakeet-tdt-0.6b-v2 | 600M | 6.44% |
Interpolating that curve linearly in log size puts it at 7.65% at 153M, 0.52 points above our beam-4 result. The same
curve reaches 7.12% only at about 259M parameters.
**Selection caveat.** We compared candidate checkpoints and recipes on these same test sets, so the figures above
are best-of-candidates rather than fully held out.
**Further test sets.** Beyond the leaderboard, the model gives up more of the teacher's accuracy, and it ranks last
of these five models on the average. The table is WER % from our harness: beam 4 for ours, each model's default
decoding for the others.
| Model | FLEURS en_us (647) | Common Voice (4,000) | EdAcc (8,537) | People's Speech clean (3,999) | Mean |
|:--|--:|--:|--:|--:|--:|
| oruk-loop-153M | 9.88 | 17.15 | 20.16 | 20.15 | 16.84 |
| nvidia/canary-180m-flash | 7.21 | 9.56 | 17.96 | 19.17 | 13.48 |
| usefulsensors/moonshine-streaming-medium | 9.03 | 14.61 | 21.75 | 19.20 | 16.15 |
| openai/whisper-small.en | 7.79 | 15.39 | 23.23 | 18.88 | 16.32 |
| Qwen/Qwen3-ASR-1.7B (teacher) | 4.71 | 7.53 | 13.23 | 16.73 | 10.55 |

Common Voice is taken from the leaderboard repository's packaging of its test split, and People's Speech from the
clean test split; both are fixed random subsets (seed 20261014). FLEURS test sentences are not in the training data,
but FLEURS en_us training audio was used in the early encoder warm-up, so FLEURS is not fully out of domain.
**This release.** The rows above come from our evaluation pipeline, which uses CUDA-graph greedy decoding and
fixed-length prompt padding. We re-scored every set with this repository's own code (`AutoModel` plus `transcribe`,
batch 64). With greedy decoding, 96.3% of utterances come out identical to the pipeline. The rest differ by bf16
rounding, which can move a near-tied token and, rarely, end a chunk early. Set-level gaps come from a handful of
clips: one LibriSpeech test-other clip accounts for 52 of that set's 58 extra errors, and one TED-LIUM clip for 31 of
34. Beam search is less sensitive: every beam-4 set score reproduces within 0.012 points, and the 7-set mean is
identical.
| Set | Pipeline, greedy | This repo, greedy | Pipeline, beam 4 | This repo, beam 4 |
|:--|--:|--:|--:|--:|
| AMI | 11.57 | 11.55 | 11.22 | 11.22 |
| Earnings-22 | 10.29 | 10.27 | 9.80 | 9.80 |
| GigaSpeech | 10.75 | 10.76 | 10.45 | 10.45 |
| LS clean | 2.53 | 2.54 | 2.40 | 2.40 |
| LS other | 6.32 | 6.43 | 5.95 | 5.94 |
| SPGISpeech | 2.61 | 2.61 | 2.48 | 2.48 |
| TED-LIUM | 3.79 | 3.91 | 3.77 | 3.77 |
| VoxPopuli | 7.70 | 7.71 | 7.55 | 7.54 |
| **7-set mean** | 7.40 | 7.41 | 7.12 | 7.12 |
Per-set scores, word counts and the frontier analysis are in [`results/evaluation.json`](results/evaluation.json).
## Limitations
- **English only.** Trained and evaluated only with the English prompt (`language="English"`), and the vocabulary is
  trimmed to English tokens.
- **Domain.** Accuracy is best on audio like the training corpora: read speech, meetings, finance calls,
  parliament, podcasts. On accented, crowd-sourced and found speech it degrades more than the teacher (see the table
  above).
- **No timestamps or diarization.** Long recordings are handled by quiet-point chunking. Rare runs of near-tied
  tokens can end a chunk early or repeat a phrase; beam 4 is more robust than greedy. Very fast speech can reach
  the per-chunk token cap.
- **Compute.** The parameter saving does not reduce per-token compute (see Architecture).
## License and attribution
| Files | License |
|:--|:--|
| `model.safetensors` (weights) | [PolyForm Small Business 1.0.0 or PolyForm Noncommercial 1.0.0](LICENSE-WEIGHTS.md), at your option; other commercial use needs a commercial license from Oruk AI |
| `modeling_oruk_loop.py` | [MIT](LICENSE) |
| Unmodified Qwen3-ASR tokenizer, chat template and processor files | [Apache-2.0](LICENSE-QWEN) |
| `results/` and `media/` | CC BY 4.0, attributed to Oruk AI |
**What the weights license allows.**
- Small businesses can use the weights for anything, commercial products included. A small business here means
  fewer than 100 employees and contractors and under US$1M revenue in the prior year.
- Anyone can use them for personal, research and educational purposes.
- Charities, schools, universities, public research organisations and government bodies can use them freely.
- Any other commercial use needs a commercial license from Oruk AI; contact us through [oruk.ai](https://oruk.ai).
The model derives from Qwen3-ASR-1.7B (Apache-2.0, © Alibaba Cloud / Qwen team). Its initialisation, the kept
modules and the processor files come from that model; see [NOTICE.md](NOTICE.md).
Some training corpora carry non-commercial or research-only terms: SPGISpeech, GigaSpeech and TED-LIUM 3. Review
those terms for your use. Corpus copyrights stay with their owners. No endorsement by Qwen, Alibaba Cloud or the
dataset providers is implied.
## Citation
```bibtex
@misc{oruk2026loop153m,
  title  = {oruk-loop-153M: a double-looped distillation of Qwen3-ASR},
  author = {{Oruk AI}},
  year   = {2026},
  url    = {https://huggingface.co/oruk/oruk-loop-153M}
}
@article{Qwen3-ASR,
  title   = {Qwen3-ASR Technical Report},
  author  = {Xian Shi and Xiong Wang and Zhifang Guo and Yongqi Wang and Pei Zhang and Xinyu Zhang and Zishan Guo and
             Hongkun Hao and Yu Xi and Baosong Yang and Jin Xu and Jingren Zhou and Junyang Lin},
  journal = {arXiv preprint arXiv:2601.21337},
  year    = {2026}
}
```

	---
	language: [en]
	license: other
	license_name: polyform-small-business-or-noncommercial
	license_link: https://huggingface.co/oruk/oruk-loop-153M/blob/main/LICENSE-WEIGHTS.md
	library_name: transformers
	pipeline_tag: automatic-speech-recognition
	base_model: Qwen/Qwen3-ASR-1.7B-hf
	tags: [speech-recognition, asr, english, distillation, looped-transformer, parameter-sharing, qwen3-asr]
	datasets: [openslr/librispeech_asr, kensho/spgispeech, speechcolab/gigaspeech, facebook/voxpopuli, edinburghcstr/ami, sanchit-gandhi/tedlium-data, sanchit-gandhi/earnings22_split, espnet/yodas3, google/fleurs]
	model-index:
	- name: oruk-loop-153M
	results:
	- task: {type: automatic-speech-recognition, name: Speech recognition}
	dataset: {name: AMI (IHM), type: hf-audio/open-asr-leaderboard, config: ami, split: test}
	metrics: [{type: wer, value: 11.22, name: Test WER (beam 4)}]
	- task: {type: automatic-speech-recognition, name: Speech recognition}
	dataset: {name: Earnings-22, type: hf-audio/open-asr-leaderboard, config: earnings22, split: test}
	metrics: [{type: wer, value: 9.80, name: Test WER (beam 4)}]
	- task: {type: automatic-speech-recognition, name: Speech recognition}
	dataset: {name: GigaSpeech, type: hf-audio/open-asr-leaderboard, config: gigaspeech, split: test}
	metrics: [{type: wer, value: 10.45, name: Test WER (beam 4)}]
	- task: {type: automatic-speech-recognition, name: Speech recognition}
	dataset: {name: LibriSpeech test-clean, type: hf-audio/open-asr-leaderboard, config: librispeech, split: test.clean}
	metrics: [{type: wer, value: 2.40, name: Test WER (beam 4)}]
	- task: {type: automatic-speech-recognition, name: Speech recognition}
	dataset: {name: LibriSpeech test-other, type: hf-audio/open-asr-leaderboard, config: librispeech, split: test.other}
	metrics: [{type: wer, value: 5.95, name: Test WER (beam 4)}]
	- task: {type: automatic-speech-recognition, name: Speech recognition}
	dataset: {name: SPGISpeech, type: hf-audio/open-asr-leaderboard, config: spgispeech, split: test}
	metrics: [{type: wer, value: 2.48, name: Test WER (beam 4)}]
	- task: {type: automatic-speech-recognition, name: Speech recognition}
	dataset: {name: TED-LIUM 3, type: sanchit-gandhi/tedlium-data, split: test}
	metrics: [{type: wer, value: 3.77, name: Test WER (beam 4)}]
	- task: {type: automatic-speech-recognition, name: Speech recognition}
	dataset: {name: VoxPopuli (en), type: hf-audio/open-asr-leaderboard, config: voxpopuli, split: test}
	metrics: [{type: wer, value: 7.55, name: Test WER (beam 4)}]
	---

	<p><a href="https://oruk.ai"><img src="media/oruk.png" alt="oruk" width="184"></a></p>

	# oruk-loop-153M

	Oruk AI · Free for noncommercial use and for small businesses; other commercial use needs a license from Oruk AI ([terms](LICENSE-WEIGHTS.md)).

	oruk-loop-153M is an English speech recognizer with 153,202,638 stored parameters, distilled from
	[Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B-hf) (2,038,052,480 stored parameters, 13.3× more). It
	keeps the teacher's architecture, but each of its two transformer stacks is one shared block run in a loop: one
	encoder block applied 24 times and one decoder block applied 28 times. Low-rank per-depth adapters tell each pass
	which layer it is standing in for.

	On the seven English test sets of the [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard),
	mean WER is 7.12% with beam-4 decoding and 7.40% greedy. That bulges the size–accuracy frontier outward:

	- At the same size. In the leaderboard snapshot of 14 October 2026, the previous frontier stood at 7.65% at 153M
	parameters (interpolated between neighbouring models), 0.52 points above this model.
	- At the same accuracy. The interpolated frontier reached 7.12% only at about 259M parameters, 1.7× as many.
	- Nearest real model. The closest actual model is moonshine-streaming-medium (240M, 7.18%), 1.6× larger. The
	0.06-point gap equals our harness's offset (it scores the teacher 0.06 below the leaderboard on average), so
	treat the two as level on accuracy.

	All numbers come from our harness, not an official leaderboard submission. Checkpoints were also selected on these
	test sets. Both caveats are explained under [Evaluation](#evaluation).

	<video controls muted loop playsinline width="100%" src="https://huggingface.co/oruk/oruk-loop-153M/resolve/main/media/oruk-loop-153m-frontier.mp4"></video>

	![The double loop pushes the size–accuracy frontier outward](media/frontier-bulge.jpg)

	## Quick start

	```sh
	pip install "transformers==5.17.0" torch soundfile soxr
	```

	```python
	from transformers import AutoModel, AutoProcessor

	model = AutoModel.from_pretrained("oruk/oruk-loop-153M", trust_remote_code=True) # CUDA if available
	processor = AutoProcessor.from_pretrained("oruk/oruk-loop-153M")

	print(model.transcribe("speech.wav", processor)) # greedy
	print(model.transcribe("speech.wav", processor, num_beams=4)) # setting of the headline numbers
	print(model.transcribe(["a.wav", "b.flac"], processor, batch_size=32)) # a list means a batch
	```

	`transcribe` accepts file paths, 1-D float arrays and `(array, sampling_rate)` tuples. For a bare array that is not
	at 16 kHz, pass `sampling_rate=`. A batch must be a list, because a tuple is read as one `(array, sr)` recording.

	Audio longer than 30 s is cut at quiet points into chunks of at most 30 s, and the chunk transcripts are joined
	with spaces; `return_chunks=True` also returns the per-chunk texts. Each chunk's output is capped at
	min(448, 10 tokens per second + 24) tokens. `length_penalty` is passed through to beam search. The output keeps the
	teacher's casing and punctuation.

	`trust_remote_code=True` runs [`modeling_oruk_loop.py`](modeling_oruk_loop.py) from this repository; it subclasses
	the stock `Qwen3ASRForConditionalGeneration`. Tokenizer, chat template and feature extractor are the unmodified
	Qwen3-ASR files, so `AutoProcessor` needs no remote code. The code is tested with `transformers==5.17.0` and PyTorch
	2.9. Pin a revision for reproducible results.

	\| Device \| Defaults \| Memory after loading \|
	\|:--\|:--\|--:\|
	\| CUDA \| bf16 autocast, per-depth weights merged for fast decoding (`merge=True`) \| 4.2 GB of GPU memory \|
	\| CPU \| fp32, adapters applied on the fly (`merge=False`) \| 3.9 GB of process RAM \|

	Decoding adds the KV cache on top of this. A batch of 64 (or beam 4) needs several more GB.

	Override the defaults with `AutoModel.from_pretrained(..., device="cuda:1", dtype=torch.float32, merge=False)`.

	On 12 vCPUs (Intel Xeon, 2.2 GHz), greedy decoding takes 0.57 s per second of audio, about 1.75× faster than
	real time.

	To call `generate` yourself on a single clip of at most 30 s:

	```python
	import soundfile as sf
	audio, sr = sf.read("short.wav") # 16 kHz mono
	inputs = processor.apply_transcription_request(audio=audio, language="English", return_tensors="pt",
	processor_kwargs={"audio_kwargs": {"sampling_rate": 16000}})
	inputs = {k: v.to(model.device) for k, v in inputs.items()}
	inputs["input_features"] = inputs["input_features"].to(model.model.audio_tower.conv2d1.weight.dtype)
	out = model.generate(**inputs, max_new_tokens=448)
	print(processor.decode(out[0, inputs["input_ids"].shape[1]:], return_format="transcription_only"))
	```

	## Architecture

	\| Component \| What is stored \| Parameters \|
	\|:--\|:--\|--:\|
	\| Audio encoder loop \| one Qwen3-ASR encoder block (d=1024) applied 24×, with rank-64 adapters on every linear layer and per-depth biases and LayerNorms \| 41,213,952 \|
	\| Decoder loop \| one Qwen3 decoder block (d=2048, GQA 16/8 heads) applied 28×, with rank-16 adapters and per-depth RMSNorms \| 67,886,080 \|
	\| Token table \| 36,974 kept English token rows, tied input/output: 368 frequent rows stored exactly, 36,606 rows factorised at rank 729 \| 28,932,430 \|
	\| Kept Qwen3-ASR modules \| convolutional front-end, encoder output norm, audio projector, final decoder norm \| 15,170,176 \|
	\| Total \| \| 153,202,638 \|

	Each looped linear layer computes `W x + b_d + B_d (A_d x)` at depth `d`. Here `W` is shared by every depth, and
	`A_d`, `B_d` and `b_d` belong to depth `d`. The shared weights total 62.9M and the per-depth adapters 45.7M (28.3M of
	them in the encoder).

	Token ids stay in the original Qwen3 vocabulary. The output layer scores only the kept rows and returns `-inf` for
	the rest, so the stock tokenizer and prompts work unchanged.

	Looping shrinks storage, not arithmetic. Every token still passes through 24 encoder and 28 decoder layers at the
	teacher's width. Per-token compute is therefore close to the teacher's; the trimmed output layer (37k rows instead
	of 152k) is the only saving. With merged weights, decoding needs about as much accelerator memory as the teacher.

	On one A100 40 GB, greedy decoding with batch 64 transcribes the seven leaderboard sets about 145× faster than real time,
	audio loading included (three such jobs shared the machine).

	## Training

	1. Initialisation. Each shared weight starts as the mean of the teacher's layers. Each depth's adapter starts as
	the truncated SVD of that layer minus the mean, so a full-rank adapter would reproduce the teacher exactly.
	Per-depth norms and biases are copied from the teacher.
	2. Distillation from Qwen3-ASR-1.7B, in several re-warmed learning-rate cycles:
	- the decoder matches the teacher's token distributions (KL) and its hidden states at every depth;
	- the encoder matches the teacher's per-depth states and its output (MSE and cosine);
	- the text targets keep the teacher's casing and punctuation, with the word content taken from each corpus's
	reference.
	3. Vocabulary. The decoder keeps 36,974 of Qwen3's 151,936 tokens: every token used at least 100 times in the
	teacher's English training transcripts, plus the byte, special and prompt tokens, closed under BPE merges. Rare
	rows are factorised, which turns 311M embedding parameters into 28.9M.
	4. Weight averaging. The final EMA weights of ten distillation runs are averaged uniformly.
	5. MWER fine-tuning. Only the decoder's per-depth adapters and norms are trained, for 500 steps per run at a
	learning rate of 3e-5. The objective is expected word error over the model's own beam-4 N-best, measured against
	reference transcripts (Whisper-normalised), plus a cross-entropy anchor weighted 0.1. Four runs are averaged;
	eight further runs, each started from that average, are averaged again.

	### Training data

	Only training splits are used. The Earnings-22 split is not part of the original corpus, so we checked it by
	recording ID: its 111 training calls share none of the 6 test calls.

	\| Corpus \| Audio used \| Upstream terms \|
	\|:--\|:--\|:--\|
	\| [LibriSpeech](https://huggingface.co/datasets/openslr/librispeech_asr) \| train, 961 h \| CC BY 4.0 \|
	\| [SPGISpeech](https://huggingface.co/datasets/kensho/spgispeech) \| L train, 4,900 h \| Kensho Terms of Usage: academic research and internal use \|
	\| [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) \| L train (2,475 h) and 7,434 h of XL train, overlapping \| access for non-commercial research and education \|
	\| [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) \| en train, 508 h \| CC0 1.0 \|
	\| [TED-LIUM 3](https://huggingface.co/datasets/sanchit-gandhi/tedlium-data) \| train, 454 h \| CC BY-NC-ND 3.0 \|
	\| [AMI](https://huggingface.co/datasets/edinburghcstr/ami) \| IHM train, 72 h \| CC BY 4.0 \|
	\| [Earnings-22](https://huggingface.co/datasets/sanchit-gandhi/earnings22_split) \| train split, 98 h \| CC BY-SA 4.0 \|
	\| [YODAS3](https://huggingface.co/datasets/espnet/yodas3) (English), [FLEURS](https://huggingface.co/datasets/google/fleurs) en_us train \| early warm-up of the looped encoder \| CC BY 3.0; CC BY 4.0 \|

	The teacher's outputs on this audio (distributions, hidden states, transcripts) are training targets. MWER used
	the AMI, VoxPopuli, TED-LIUM, LibriSpeech-360, SPGISpeech (subset), GigaSpeech L and Earnings-22 training audio
	with reference transcripts.

	## Evaluation

	Test sets. The Open ASR Leaderboard's English sets (`hf-audio/open-asr-leaderboard`, original transcripts),
	plus TED-LIUM 3 test from `sanchit-gandhi/tedlium-data`.

	Scoring follows the leaderboard:

	- the Whisper `EnglishTextNormalizer` is applied to reference and hypothesis;
	- references that are empty or read "ignore time segment in scoring" are dropped;
	- WER is corpus-level per set (edits ÷ reference words);
	- the average is the plain mean over the seven sets of the leaderboard's English average.

	Comparison numbers for other models are from the leaderboard snapshot of 14 October 2026. Their sizes are the
	leaderboard's listed sizes; ours is the stored count.

	\| Model \| Params \| AMI \| Earnings-22 \| GigaSpeech \| LS clean \| LS other \| SPGISpeech \| VoxPopuli \| Mean \|
	\|:--\|--:\|--:\|--:\|--:\|--:\|--:\|--:\|--:\|--:\|
	\| oruk-loop-153M, beam 4 \| 153M \| 11.22 \| 9.80 \| 10.45 \| 2.40 \| 5.95 \| 2.48 \| 7.55 \| 7.12 \|
	\| oruk-loop-153M, greedy \| 153M \| 11.57 \| 10.29 \| 10.75 \| 2.53 \| 6.32 \| 2.61 \| 7.70 \| 7.40 \|
	\| nvidia/parakeet-tdt_ctc-110m \| 0.11B \| 15.89 \| 12.37 \| 10.52 \| 2.40 \| 5.22 \| 2.54 \| 6.90 \| 7.98 \|
	\| nvidia/canary-180m-flash \| 0.18B \| 14.86 \| 12.33 \| 10.51 \| 1.73 \| 4.35 \| 2.26 \| 6.35 \| 7.48 \|
	\| usefulsensors/moonshine-streaming-medium \| 0.24B \| 10.68 \| 11.92 \| 9.46 \| 2.09 \| 5.00 \| 2.58 \| 8.54 \| 7.18 \|
	\| Qwen/Qwen3-ASR-0.6B \| 0.6B \| 11.66 \| 11.06 \| 9.14 \| 2.13 \| 4.45 \| 3.03 \| 7.07 \| 6.93 \|
	\| nvidia/parakeet-tdt-0.6b-v2 \| 0.6B \| 11.16 \| 11.15 \| 9.74 \| 1.69 \| 3.19 \| 2.17 \| 5.95 \| 6.44 \|
	\| Qwen/Qwen3-ASR-1.7B (teacher), leaderboard \| 1.7B \| 10.56 \| 10.25 \| 8.74 \| 1.63 \| 3.40 \| 2.84 \| 6.35 \| 6.25 \|
	\| Qwen/Qwen3-ASR-1.7B (teacher), our harness, greedy \| 1.7B \| 10.48 \| 10.06 \| 8.74 \| 1.62 \| 3.36 \| 2.84 \| 6.24 \| 6.19 \|

	Harness offset. On the teacher, our harness reads 0.06 points below the leaderboard on average. Small
	gaps against leaderboard rows (such as the 0.06 to moonshine-streaming-medium) are within that offset.

	TED-LIUM. On TED-LIUM 3 test the WER is 3.77% (beam 4) and 3.79% (greedy); the teacher scores 2.28%. With
	TED-LIUM included, our eight-set mean is 6.70% (beam 4) and 6.95% (greedy).

	Rank in the snapshot. 65 models have a listed size above 153M; 48 of them have a higher mean than our beam-4
	result. Among models listed below 600M, the beam-4 mean is the lowest, with moonshine-streaming-medium level within
	the harness offset. With greedy decoding, moonshine-streaming-medium (7.18%) is ahead.

	The frontier bulge. Before this model, the lowest-WER curve across model sizes ran:

	\| Model \| Size \| Mean WER \|
	\|:--\|--:\|--:\|
	\| parakeet-tdt_ctc-110m \| 110M \| 7.98% \|
	\| canary-180m-flash \| 180M \| 7.48% \|
	\| moonshine-streaming-medium \| 240M \| 7.18% \|
	\| parakeet-tdt-0.6b-v2 \| 600M \| 6.44% \|

	Interpolating that curve linearly in log size puts it at 7.65% at 153M, 0.52 points above our beam-4 result. The same
	curve reaches 7.12% only at about 259M parameters.

	Selection caveat. We compared candidate checkpoints and recipes on these same test sets, so the figures above
	are best-of-candidates rather than fully held out.

	Further test sets. Beyond the leaderboard, the model gives up more of the teacher's accuracy, and it ranks last
	of these five models on the average. The table is WER % from our harness: beam 4 for ours, each model's default
	decoding for the others.

	\| Model \| FLEURS en_us (647) \| Common Voice (4,000) \| EdAcc (8,537) \| People's Speech clean (3,999) \| Mean \|
	\|:--\|--:\|--:\|--:\|--:\|--:\|
	\| oruk-loop-153M \| 9.88 \| 17.15 \| 20.16 \| 20.15 \| 16.84 \|
	\| nvidia/canary-180m-flash \| 7.21 \| 9.56 \| 17.96 \| 19.17 \| 13.48 \|
	\| usefulsensors/moonshine-streaming-medium \| 9.03 \| 14.61 \| 21.75 \| 19.20 \| 16.15 \|
	\| openai/whisper-small.en \| 7.79 \| 15.39 \| 23.23 \| 18.88 \| 16.32 \|
	\| Qwen/Qwen3-ASR-1.7B (teacher) \| 4.71 \| 7.53 \| 13.23 \| 16.73 \| 10.55 \|

	Common Voice is taken from the leaderboard repository's packaging of its test split, and People's Speech from the
	clean test split; both are fixed random subsets (seed 20261014). FLEURS test sentences are not in the training data,
	but FLEURS en_us training audio was used in the early encoder warm-up, so FLEURS is not fully out of domain.

	This release. The rows above come from our evaluation pipeline, which uses CUDA-graph greedy decoding and
	fixed-length prompt padding. We re-scored every set with this repository's own code (`AutoModel` plus `transcribe`,
	batch 64). With greedy decoding, 96.3% of utterances come out identical to the pipeline. The rest differ by bf16
	rounding, which can move a near-tied token and, rarely, end a chunk early. Set-level gaps come from a handful of
	clips: one LibriSpeech test-other clip accounts for 52 of that set's 58 extra errors, and one TED-LIUM clip for 31 of
	34. Beam search is less sensitive: every beam-4 set score reproduces within 0.012 points, and the 7-set mean is
	identical.

	\| Set \| Pipeline, greedy \| This repo, greedy \| Pipeline, beam 4 \| This repo, beam 4 \|
	\|:--\|--:\|--:\|--:\|--:\|
	\| AMI \| 11.57 \| 11.55 \| 11.22 \| 11.22 \|
	\| Earnings-22 \| 10.29 \| 10.27 \| 9.80 \| 9.80 \|
	\| GigaSpeech \| 10.75 \| 10.76 \| 10.45 \| 10.45 \|
	\| LS clean \| 2.53 \| 2.54 \| 2.40 \| 2.40 \|
	\| LS other \| 6.32 \| 6.43 \| 5.95 \| 5.94 \|
	\| SPGISpeech \| 2.61 \| 2.61 \| 2.48 \| 2.48 \|
	\| TED-LIUM \| 3.79 \| 3.91 \| 3.77 \| 3.77 \|
	\| VoxPopuli \| 7.70 \| 7.71 \| 7.55 \| 7.54 \|
	\| 7-set mean \| 7.40 \| 7.41 \| 7.12 \| 7.12 \|

	Per-set scores, word counts and the frontier analysis are in [`results/evaluation.json`](results/evaluation.json).

	## Limitations

	- English only. Trained and evaluated only with the English prompt (`language="English"`), and the vocabulary is
	trimmed to English tokens.
	- Domain. Accuracy is best on audio like the training corpora: read speech, meetings, finance calls,
	parliament, podcasts. On accented, crowd-sourced and found speech it degrades more than the teacher (see the table
	above).
	- No timestamps or diarization. Long recordings are handled by quiet-point chunking. Rare runs of near-tied
	tokens can end a chunk early or repeat a phrase; beam 4 is more robust than greedy. Very fast speech can reach
	the per-chunk token cap.
	- Compute. The parameter saving does not reduce per-token compute (see Architecture).

	## License and attribution

	\| Files \| License \|
	\|:--\|:--\|
	\| `model.safetensors` (weights) \| [PolyForm Small Business 1.0.0 or PolyForm Noncommercial 1.0.0](LICENSE-WEIGHTS.md), at your option; other commercial use needs a commercial license from Oruk AI \|
	\| `modeling_oruk_loop.py` \| [MIT](LICENSE) \|
	\| Unmodified Qwen3-ASR tokenizer, chat template and processor files \| [Apache-2.0](LICENSE-QWEN) \|
	\| `results/` and `media/` \| CC BY 4.0, attributed to Oruk AI \|

	What the weights license allows.

	- Small businesses can use the weights for anything, commercial products included. A small business here means
	fewer than 100 employees and contractors and under US$1M revenue in the prior year.
	- Anyone can use them for personal, research and educational purposes.
	- Charities, schools, universities, public research organisations and government bodies can use them freely.
	- Any other commercial use needs a commercial license from Oruk AI; contact us through [oruk.ai](https://oruk.ai).

	The model derives from Qwen3-ASR-1.7B (Apache-2.0, © Alibaba Cloud / Qwen team). Its initialisation, the kept
	modules and the processor files come from that model; see [NOTICE.md](NOTICE.md).

	Some training corpora carry non-commercial or research-only terms: SPGISpeech, GigaSpeech and TED-LIUM 3. Review
	those terms for your use. Corpus copyrights stay with their owners. No endorsement by Qwen, Alibaba Cloud or the
	dataset providers is implied.

	## Citation

	```bibtex
	@misc{oruk2026loop153m,
	title = {oruk-loop-153M: a double-looped distillation of Qwen3-ASR},
	author = {{Oruk AI}},
	year = {2026},
	url = {https://huggingface.co/oruk/oruk-loop-153M}
	}

	@article{Qwen3-ASR,
	title = {Qwen3-ASR Technical Report},
	author = {Xian Shi and Xiong Wang and Zhifang Guo and Yongqi Wang and Pei Zhang and Xinyu Zhang and Zishan Guo and
	Hongkun Hao and Yu Xi and Baosong Yang and Jin Xu and Jingren Zhou and Junyang Lin},
	journal = {arXiv preprint arXiv:2601.21337},
	year = {2026}
	}
	```