Card: name the training data, explain the CTC head, add profile analysis and figures

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+ctc_effect.png filter=lfs diff=lfs merge=lfs -text
+profile.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -13,6 +13,8 @@ tags:
 - speculative-decoding
 - ctc
 base_model: CohereLabs/cohere-transcribe-arabic-07-2026
+datasets:
+- oddadmix/lahgtna-v3-small
 pipeline_tag: automatic-speech-recognition
 ---
 
@@ -30,6 +32,94 @@ Two things are packaged here that the original does not have:
 
 Together: **158.5 s → 4.82 s** on a 33-second Arabic clip, on a six-core desktop CPU.
 
+![What the Arabic CTC draft head does](ctc_effect.png)
+
+---
+
+## The draft head, and why it helps
+
+The decoder here is autoregressive: one forward pass per token, ~130 per chunk. On a CPU at batch 1
+that is not an arithmetic problem — it is a *per-call* problem. Measured on this machine:
+
+| tokens per decoder forward | 1 | 4 | 8 | 16 | 32 |
+|---|---|---|---|---|---|
+| ms per forward | 22.06 | 24.55 | 27.38 | 27.17 | **28.72** |
+| ms per token | 22.06 | 6.14 | 3.42 | 1.70 | **0.90** |
+
+**Thirty-two tokens cost 1.30× what one token costs.** The per-token price falls 24-fold. Almost all
+of a decoder call is fixed overhead, and the arithmetic in between is nearly free.
+
+That is exactly the condition speculative decoding needs. If something cheap can *guess* the next K
+tokens, the decoder can check all of them in a single forward for roughly the price of checking one.
+
+**The CTC head is that cheap thing.** It sits on the encoder output and is frame-synchronous and
+non-autoregressive — CTC assumes tokens are conditionally independent given the audio, so the whole
+hypothesis comes out of **one** forward pass. That independence assumption makes it a poor
+transcriber and an excellent drafter. Measured cost: **22.8 ms** on a 10-second clip, against a
+1069 ms decoder — a 2.1% tax.
+
+The effect, measured: **2.66 tokens retired per decoder call** instead of 1, which is
+**1.29×–1.48×** wall clock depending on the clip, as shown in the figure above.
+
+### A wrong guess cannot produce a wrong word
+
+The verifier keeps only the prefix its own argmax agrees with, then appends one token taken straight
+from the decoder. **A drafted token reaches the output only if the decoder would have produced it
+anyway.** A completely wrong draft degrades to ordinary greedy decoding. What a bad draft costs is
+time, not accuracy — which is why the accuracy column below barely moves.
+
+---
+
+## Training data for the CTC head
+
+| | |
+|---|---|
+| dataset | [`oddadmix/lahgtna-v3-small`](https://huggingface.co/datasets/oddadmix/lahgtna-v3-small) — multi-dialect Arabic speech |
+| used for training | **4,256 clips · 18.33 h** from the `train` split (median clip 15.3 s, filtered to 1–30 s) |
+| held out | **1,024 clips · 4.58 h** from the `test` split, never trained on |
+| labels | **the base model's own transcripts** — self-distillation |
+| epochs | 12, AdamW, `F.ctc_loss` with blank = `pad_token_id` = 2 |
+
+**Only the audio from that dataset was used.** The targets are not its transcripts — they are what
+[CohereLabs/cohere-transcribe-arabic-07-2026](https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026) itself emits on that audio. The drafter's job is to predict
+*this model's* output, so training it on anything else would teach it to propose tokens the verifier
+would reject. It also means no human annotation was involved anywhere in this head.
+
+Labels were produced by an fp32 GPU replica of the deployed int8 model rather than by bf16, which
+cut label noise against the deployed model from 2.81% to 1.74% WER.
+
+**Vocabulary saturates almost immediately.** Only 274 of the 16,384 vocabulary entries contain an
+Arabic character at all — the rest belong to other scripts the multilingual base model supports — and
+the tokenizer is sub-word for Arabic at ~2.4 tokens per word. After **0.83 hours**, the types seen
+10+ times already covered **99.24% of all token occurrences**. Past that point more data buys
+acoustic quality, not new symbols.
+
+The head is **data-limited, not capacity-limited**: acceptance went 0.143 at 0.83 h to 0.331 at
+18.3 h, and test loss was still improving on the metric that matters when labelling stopped for
+time. More hours would help.
+
+---
+
+## Where the time goes
+
+![Latency profile](profile.png)
+
+Two independent measurements agree on this: forward hooks attributing time to architectural
+components, and `torch.profiler` attributing it to ATen kernels. The kernel view:
+
+| ATen kernel | self CPU | share | calls | what it is |
+|---|---|---|---|---|
+| `quantized::linear_dynamic` | 900.9 ms | 58.5% | 4,079 | the int8 GEMMs — the real work |
+| `aten::mkldnn_convolution` | 264.4 ms | 17.2% | 149 | conformer convs, still fp32 |
+| SDPA flash attention | 55.1 ms | 3.6% | 928 | attention proper |
+| `aten::aminmax` | 36.1 ms | 2.3% | 4,079 | the per-call activation scale — pure dynamic-quant overhead |
+| `aten::cat` | 34.9 ms | 2.3% | 952 | KV cache growing one token at a time |
+| view / empty / as_strided | ~74 ms | 4.8% | 48,600 | dispatch overhead, no arithmetic |
+
+`aminmax` fires exactly once per `linear_dynamic` — that is dynamic quantization recomputing an
+activation scale on every single call. And 48,600 calls to pure-metadata operations do no arithmetic
+at all. The decoder issues roughly 3,520 of the 4,079 GEMM calls, each on a 1×1024 vector.
+
 ---
 
 ## Performance
@@ -63,10 +153,9 @@ int8, batch 16, 12 threads, preallocated KV cache, draft head enabled:
 | 14-minute talk (ar) | 832.8 s | 98.36 s | 8.47× |
 | **total** | **954.4 s** | **117.3 s** | **8.14×** |
 
-### What the draft head is worth
+### What the draft head is worth, referenced to fp32
 
-Five clips, batch 1, referenced to **fp32** — the only reference that separates *different* from
-*worse*:
+fp32 is the only reference that separates *different* from *worse*:
 
 | configuration | wall | vs int8 | WER from fp32 |
 |---|---|---|---|
@@ -97,7 +186,8 @@ and the 14-minute recording — the only statistically meaningful one at 2,086 w
 
 A GPU is both faster **and** more accurate — bf16 is a far lighter perturbation than int8. This repo
 is for when there is no GPU. Worth knowing: the gap is 4.2× at batch 1 and 13.2× batched, because
-only the GPU gains from being fed more work.
+only the GPU gains from being fed more work. The draft head also transfers to the GPU, where it
+gives 1.40× at batch 1 — almost exactly what it gives on the CPU, for the same underlying reason.
 
 ---
 
@@ -129,9 +219,7 @@ Loading takes about 10 s and needs ~2.9 GB of RAM.
 
 ---
 
-## How it works
-
-### int8 dynamic quantization
+## How the quantization works
 
 Every `nn.Linear` — 1.81 B of the 2.07 B parameters — is stored as int8. Convolutions, layer norms
 and embeddings stay fp32. Weights use one per-tensor scale, `float32(max|W| / 127.5)`, with a zero
@@ -142,24 +230,11 @@ That last detail has a visible consequence: there is one activation scale for th
 handing a layer a wider tensor quantizes it more coarsely. This is why the model's output shifts
 slightly with batch size, and why speculative decoding cannot be bit-identical to greedy.
 
-### The Arabic CTC draft head
-
-At batch 1 the decoder is bound by per-call overhead rather than arithmetic — a 32-token forward
-costs only 1.30× a 1-token forward. A CTC head over the encoder output exploits that: it is
-frame-synchronous and non-autoregressive, so it emits an entire hypothesis in **one** pass
-(**22.8 ms** against a 1069 ms decoder), and the decoder then verifies up to K+1 tokens for roughly
-the price of one.
-
-Architecture: one `ParakeetEncoderBlock` copied from `encoder.layers[47]`, then a projection, a
-LayerNorm and an output matrix copied from the base model's `decoder.proj`, `decoder.norm` and
-`proj_out`. Blank is `pad_token_id` = 2, which never appears in a transcript. Trained by
-self-distillation on 18.3 hours of Egyptian Arabic labelled by the model itself — no annotation
-involved. Acceptance 0.331, **2.66 tokens per decoder forward**.
-
-**A wrong guess cannot produce a wrong word.** The verifier keeps only the prefix its own argmax
-agrees with, then appends one token taken straight from the decoder, so a drafted token reaches the
-output only if the decoder would have produced it anyway. A completely wrong draft degrades to
-ordinary greedy decoding. What a bad draft costs is time, not accuracy.
+The draft head architecture: one `ParakeetEncoderBlock` copied from `encoder.layers[47]`, then a
+projection, a LayerNorm and an output matrix copied from the base model's `decoder.proj`,
+`decoder.norm` and `proj_out`. Folding those three into a single matrix — the obvious "free"
+initialisation — measured **worse than random init**, because the real path contains a LayerNorm and
+a LayerNorm cannot be folded into a matrix product.
 
 ---
 
@@ -176,6 +251,7 @@ ordinary greedy decoding. What a bad draft costs is time, not accuracy.
   this model to CUDA raises `NotImplementedError`. For a GPU, use the base model in bf16.
 * **int8 costs accuracy.** 2.06% WER from fp32. If you have the RAM and can spare the speed, fp32 is
   more faithful; if you have a GPU, bf16 is both faster and more accurate.
+* The draft head saw 18.3 h of one dataset's dialect mix. Acoustics far from it will accept less.
 * Inherits every limitation of the base model, including its language and domain coverage.
 
 ---
@@ -207,11 +283,13 @@ the weights bit-perfect.
 
 ## License and attribution
 
-Apache 2.0, inherited from [CohereLabs/cohere-transcribe-arabic-07-2026]( https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026 ), copyright Cohere Labs.
+Apache 2.0, inherited from [CohereLabs/cohere-transcribe-arabic-07-2026](https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026), copyright Cohere Labs.
 
 **Changes made to the original** (Apache 2.0 §4b): weights of all `nn.Linear` modules quantized to
 int8 as described above; a separately trained CTC draft head added; no change to the architecture,
 the tokenizer, or the training data of the base model.
 
 The base repository is **gated**. This derivative is not, so if you need the original weights,
-please obtain them from Cohere Labs directly and accept their terms.
+please obtain them from Cohere Labs directly and accept their terms. Speech data for the draft head
+came from [`oddadmix/lahgtna-v3-small`](https://huggingface.co/datasets/oddadmix/lahgtna-v3-small); please observe that dataset's own
+licence and terms for any redistribution of the audio itself (none of it is included here).