File size: 7,182 Bytes
82fe555 b78cc62 82fe555 b78cc62 82fe555 b78cc62 82fe555 b78cc62 82fe555 b78cc62 fd3177c b78cc62 82fe555 fd3177c 82fe555 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 | ---
license: apache-2.0
base_model: CohereLabs/cohere-transcribe-03-2026
pipeline_tag: automatic-speech-recognition
library_name: transformers
language: [en, fr, de, es, it, pt, nl, pl]
tags: [asr, streaming, speech-recognition, cache-aware, fastconformer, distillation]
datasets: [openslr/librispeech_asr, facebook/multilingual_librispeech, facebook/voxpopuli, fixie-ai/common_voice_17_0,
google/fleurs, speechcolab/gigaspeech, edinburghcstr/ami, MLCommons/peoples_speech]
---
# Cohere Streaming (2B)
A **cache-aware streaming** version of [Cohere Transcribe](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026)
(2B, offline). Words appear while the person is speaking. When they stop, a second pass produces the final
transcript, close to offline accuracy.
- **One model, five latencies:** live-caption lookahead of 80, 160, 320, 560 or 1,120 ms, chosen at load time.
- **Final transcript doesn't depend on the latency setting:** at the end of speech the buffered utterance is
re-encoded at 1.12 s lookahead and decoded with a 4-beam search, about 0.3 s after the speaker stops.
- **8 languages:** English, French, German, Spanish, Italian, Portuguese, Dutch, Polish.
## Results
All Open ASR Leaderboard numbers below use the leaderboard's own data loading, normaliser and scoring
([huggingface/open_asr_leaderboard](https://github.com/huggingface/open_asr_leaderboard) @ `67e8bd6`). The other
systems' numbers are the leaderboard's published ones. Our final transcript is the same at every latency setting,
within ±0.05 average WER.
**English (8 public datasets, WER %)**
| | AMI | Earnings22 | GigaSpeech | LS clean | LS other | SPGISpeech | Monsoon | VoxPopuli | **Avg** |
|---|---|---|---|---|---|---|---|---|---|
| **Cohere Streaming (2B)** | **9.80** | 14.38 | 10.30 | 2.03 | **4.72** | 3.91 | 7.76 | 5.19 | **7.26** |
| Nemotron 3.5 ASR streaming (0.6B) | 13.43 | 15.30 | 9.86 | 2.83 | 6.79 | 3.27 | 7.30 | 4.24 | 7.88 |
| Voxtral Mini 4B Realtime | 13.34 | 9.31 | 8.80 | 1.62 | 4.94 | 2.23 | 8.86 | 2.60 | 6.46 |
| *Cohere Transcribe, offline (not streaming)* | *7.02* | *7.88* | *7.88* | *0.97* | *2.05* | *2.74* | *3.44* | *5.38* | *4.67* |
**Multilingual (average of FLEURS / Common Voice / MLS per language, WER %)**
| | fr | de | es | it | pt | nl | **Avg** |
|---|---|---|---|---|---|---|---|
| **Cohere Streaming (2B)** | **6.91** | 6.41 | 4.63 | **5.48** | 10.19 | 9.78 | **7.23** |
| Nemotron 3.5 ASR streaming (0.6B) | 9.40 | 8.96 | 5.43 | 8.66 | 8.25 | 14.50 | 9.20 |
| Voxtral Mini 4B Realtime | 7.78 | 6.34 | 4.18 | 6.41 | 6.22 | 9.19 | 6.69 |
| *Cohere Transcribe, offline* | *4.02* | *3.10* | *2.81* | *2.97* | *5.99* | *4.17* | *3.84* |
**Latency (LibriSpeech, real-time streaming harness, one A100, one stream)**
| Setting | WER clean / other | Median word delay (clean / other) | Final text after end of speech | Real-time factor, 1 stream |
|---|---|---|---|---|
| 80 ms | 1.36 / 4.85 | 303 / 356 ms | 364 ms | 0.68× |
| 320 ms | 1.36 / 4.85 | 263 / 296 ms | 302 ms | 1.69× |
| 560 ms | 1.36 / 4.85 | 387 / 429 ms | 296 ms | 2.46× |
| 1120 ms | 1.36 / 4.85 | 680 / 725 ms | 321 ms | 2.99× |
Word delay is measured against forced-aligned word ends, and includes compute time. At 80 ms the per-stream Python
loop is slower than real time on one A100. Use 320 ms or more, or a faster GPU, for live use.
## Usage
The checkpoint includes its streaming model code, processor, and generation settings. Load it directly with Transformers:
```bash
pip install "transformers==5.14.1" torch librosa soundfile
hf auth login
```
```python
from transformers import AutoModel
model = AutoModel.from_pretrained(
"Steveeeeeeen/cohere-streaming",
trust_remote_code=True,
lookahead=6,
)
model.reset(language="en")
for chunk in audio_chunks_80ms: # Mono float32 NumPy arrays: 16 kHz, 1,280 samples each.
print(model.step(chunk))
print(model.finish())
```
The loader chooses CUDA, Apple Metal, or CPU. Pass `device="cuda"` or `device="cpu"` to choose explicitly, and `dtype=torch.bfloat16` or `dtype=torch.float32` to choose precision. Streaming supports one device. `reset()` starts an utterance, `step()` updates its partial transcript, and `finish()` flushes the remaining audio and decodes the endpoint.
Inference loads the weights and processor from this public repository without an authentication token; it does not download the original Cohere weights. Access to the original gated Cohere model is still needed for teacher labeling and distillation.
`trust_remote_code=True` loads the streaming implementation shipped here. This adapter exposes the streaming methods above; standard ASR pipeline streaming and `AutoModel.from_config()` are not implemented.
To save a copy with its code and processor:
```python
model.save_pretrained("cohere-streaming-local")
```
The directory contains `student.safetensors`, `config.json`, `mel_stats.json`, `model.py`, processor/tokenizer files, and `generation_config.json`. The published repository also retains `log.json` from training. The local `CohereStreaming` adapter in the [smol-audio example](https://github.com/Deep-unlearning/smol-audio/tree/master/cohere_transcribe_streaming) remains compatible with this checkpoint.
## How it was made
1. **Exact conversion** of Cohere's 48-layer FastConformer encoder into transformers' cache-aware streaming encoder:
- BatchNorm folded into a per-channel affine;
- subsampling made causal in time;
- parity checked to 1e-4 in non-causal mode.
2. **Distillation from offline Cohere.** About 12.5k hours were transcribed by the offline model, and the streaming
student was trained on those transcripts. Losses: decoder cross-entropy, CTC, and cosine feature distillation at 4
encoder layers. Each batch samples one lookahead from {0, 1, 3, 6, 13}.
- **Read speech:** LibriSpeech, MLS, VoxPopuli, Common Voice, FLEURS-train.
- **Conversational speech:** about 1,460 h of AMI, GigaSpeech and People's Speech.
3. **Robustness:**
- causal running feature normalisation with a randomised warm-up prior;
- ±10 dB gain augmentation.
4. **Endpoint decoding, tuned on dev sets:**
- re-encode the utterance at 1.12 s lookahead;
- 4-beam search;
- bracket tokens suppressed.
## Limitations
- **The final text takes about 0.3 s after end of speech.** Nemotron takes about 50 ms.
- **Live partials can be rewritten** before they settle, especially at 80–320 ms on noisy audio.
- **Transcription style follows Cohere's:** lightly edited, not verbatim. This costs WER on verbatim references
(VoxPopuli-cleaned) and on fully edited ones (SPGISpeech, where it keeps "you know").
- **Weaker spots:** financial vocabulary (Earnings22), Portuguese, and Common Voice Dutch.
- **Long inputs:** utterances over 30 s fall back to the streamed states for the final pass, and the final decode
is capped at 256 tokens. Segment long audio with a VAD.
## License and attribution
Apache-2.0, like the base model [CohereLabs/cohere-transcribe-03-2026](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026).
The training audio comes from the datasets listed above, under their own licenses.
|