Reader v2: 6 layers, LibriSpeech 960 h (test-clean WER 4.21 %, test-other 10.42 %)

NOTICE

@@ -1,9 +1,10 @@
 EmbeddingGemma-2-Transcribe-Reader — NOTICE
 
-This repository holds reader v1 of EmbeddingGemma 2 Transcribe, licensed under the Apache License, Version 2.0
+This repository holds reader v2 of EmbeddingGemma 2 Transcribe, licensed under the Apache License, Version 2.0
 (see LICENSE).
 
-- reader.pt: reader weights trained by mlboydaisuke (john-rocky), 2026-10, with CTC on LibriSpeech train-clean-100:
+- reader.pt: reader weights trained by mlboydaisuke (john-rocky), 2026-10, with CTC on LibriSpeech train-clean-100,
+  train-clean-360 and train-other-500:
   V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain audio books",
   ICASSP 2015, https://www.openslr.org/12, licensed under CC BY 4.0
   (https://creativecommons.org/licenses/by/4.0/). The audio is not redistributed here.

README.md

@@ -10,34 +10,38 @@ datasets:
 tags: [embeddinggemma, speech-recognition, ctc, word-embeddings, on-device, core-ai, litert]
 ---
 
-# EmbeddingGemma 2 Transcribe — reader v1
+# EmbeddingGemma 2 Transcribe — reader v2
 
-A 13.3 M-parameter reader that turns the audio soft tokens of Google's
+A 19.6 M-parameter reader that turns the audio soft tokens of Google's
 [EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) into English words: each 40 ms token is mapped
 into the model's 768-d text-embedding space and read as the closest of 200,000 words, or as blank, with greedy CTC
-decoding and no language model. The EmbeddingGemma 2 encoder stays frozen; only the reader is trained.
+decoding and no language model. The EmbeddingGemma 2 encoder stays frozen; only the reader is trained. Reader v2 is
+trained on the three LibriSpeech training sets (960 h): test-clean WER 4.21 %, test-other 10.42 %.
 
 The idea comes from Shahir's post (2026-10-13) and the *Embedding Gemma Transcribe* Mac app (eg2.haku.sh), whose
 code and weights are not public. This reader is an independent reproduction, trained from scratch on LibriSpeech. It
 is not affiliated with Google.
 
+Reader v1 (4 layers, 13.3 M parameters, trained on train-clean-100 alone; test-clean 6.81 %, test-other 15.10 %) is
+revision `1af9c5f5` of this repository.
+
 These weights are framework-neutral (a PyTorch state dict and a NumPy table). The converted builds that run them on
 a device:
 
 | Build | Repository |
 |---|---|
 | Core AI (iOS 27 / macOS 27): the EmbeddingGemma 2 audio encoder and this reader with its word table, as `.aimodel` bundles | [mlboydaisuke/EmbeddingGemma-2-Transcribe-CoreAI](https://huggingface.co/mlboydaisuke/EmbeddingGemma-2-Transcribe-CoreAI) |
-| LiteRT (Android) | [mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT](https://huggingface.co/mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT) |
+| LiteRT (Android), with reader v1 | [mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT](https://huggingface.co/mlboydaisuke/EmbeddingGemma-2-Transcribe-LiteRT) |
 
 ## Files
 
 | File | Bytes | SHA-256 | What |
 |---|---:|---|---|
-| `reader.pt` | 53,095,605 | `67a06a857b9aeb066347f8947e4faeed11d2ec3e89e0c23974be0a2c20243d5b` | the reader: a PyTorch `state_dict`, float32 |
-| `word_table.npy` | 307,200,128 | `b2d0b13cc70c492f1444b8a42deab437d25254eac47d752d777d27b65377faa8` | float16 `[200000, 768]`, one L2-normalized row per word |
-| `vocab.txt` | 1,737,588 | `bb3b3c70379082333e1bbd90d07ca69fe5c0096bd1c3b2b3f83edd65f1062a68` | the 200,000 words, one per line; the line number is the word index |
-| `reader_config.json` | 2,019 | `68efaabae4c2eaa8ae2a5abf8f85fab115791b13178ee3a22e57a0cec727b94e` | the contract for ports: input, output, logits, decoding, the window policy, the WER |
-| `RELEASE.md` | 12,584 | | the release note: WER details, the reader's output on the 20 fixture utterances, training log, weaknesses |
+| `reader.pt` | 78,323,637 | `d605f6fba8687497be6b5fd717ea6c51beaee5d5f69c30a64225da96ebe0d8c2` | the reader: a PyTorch `state_dict`, float32 |
+| `word_table.npy` | 307,200,128 | `b2d0b13cc70c492f1444b8a42deab437d25254eac47d752d777d27b65377faa8` | float16 `[200000, 768]`, one L2-normalized row per word (the file of v1) |
+| `vocab.txt` | 1,737,588 | `bb3b3c70379082333e1bbd90d07ca69fe5c0096bd1c3b2b3f83edd65f1062a68` | the 200,000 words, one per line; the line number is the word index (the file of v1) |
+| `reader_config.json` | 2,080 | `3aede371f7fc1b69bd948a3fce6f861e44eedb237d7b6dce360063ea007a90ed` | the contract for ports: input, output, logits, decoding, the window policy, the WER |
+| `RELEASE.md` | 15,575 | `9d0ac2a5d246d8b35f6456891e9a2ae2f199ef44a51087d1fa367f4cd1f35994` | the release note: WER details, the reader's output on the 20 fixture utterances, training log, weaknesses |
 
 `SHA256SUMS` lists every file. `RELEASE.md` is the release note of the `reader/` directory of the [eg2-transcribe](https://github.com/john-rocky/eg2-transcribe)
 repository; the paths in it (`train/`, `fixtures/`) are relative to that directory.
@@ -48,11 +52,11 @@ repository; the paths in it (`train/`, `fixtures/`) are relative to that directo
   `google/embeddinggemma-2` at revision `914f7f89142e33e77833254d9c9b90c3cef7303b`
   (`model.get_audio_features(input_features, input_features_mask).pooler_output`; transformers 5.19.0 with the
   default attention, not `eager`, whose audio mask is inverted in that version).
-- **Body**: LayerNorm → Linear 512→512, 4 pre-LayerNorm transformer layers (width 512, 8 heads, FFN 2048 with GELU,
+- **Body**: LayerNorm → Linear 512→512, 6 pre-LayerNorm transformer layers (width 512, 8 heads, FFN 2048 with GELU,
   rotary positions with base 10000), LayerNorm → Linear 512→768. Attention covers the whole input window in both
-  directions. 13,268,994 parameters.
+  directions. 19,573,762 parameters.
 - **Output per token**: a unit vector in the 768-d text space and a blank logit.
-- **Logits**: word `w` scores `scale × cos(vector, word_table[w])`, with `scale = exp(log_scale) = 91.218`
+- **Logits**: word `w` scores `scale × cos(vector, word_table[w])`, with `scale = exp(log_scale) = 216.351`
   (learned); blank is class 200,000 and scores `blank_logit`.
 - **Decoding**: per token, blank where `blank_logit > top-1 word logit` (a tie reads the word), else the top-1 word;
   merge repeats, drop blanks. No language model, no punctuation, no casing.
@@ -60,54 +64,46 @@ repository; the paths in it (`train/`, `fixtures/`) are relative to that directo
   one, keeping only its new tokens and dropping its last token: the encoder is causal, so the start of a window lacks
   its left context and the last token of a 30 s cut misses one input frame. `reader_config.json` → `window`.
 - **Weights file**: `reader.pt` keys `body.*` (the reader body), `blank.weight [1, 768]`, `blank.bias [1]`,
-  `log_scale`; the module is `Reader(arch, n_vocab, k)` of `reader/train/model.py` in the eg2-transcribe repository
-  (arch `a`).
+  `log_scale`; the module is `Reader(arch, n_vocab, k, layers)` of `reader/train/model.py` in the eg2-transcribe
+  repository (arch `a`, `layers` 6; `reader_config.json` → `architecture.layers`).
 
 ## WER
 
 Greedy, no language model, every utterance of each set. Reference and hypothesis are lower-cased with runs of spaces
 collapsed; words outside the vocabulary stay in the references and count as errors.
 
-| Set | Utterances | Reference words | WER | Substitutions | Deletions | Insertions |
-|---|---:|---:|---:|---:|---:|---:|
-| dev-clean (model selection) | 2,703 | 54,402 | 6.57 % | 2,823 | 375 | 374 |
-| dev-other | 2,864 | 50,948 | 14.93 % | 5,885 | 1,089 | 631 |
-| test-clean | 2,620 | 52,576 | 6.81 % | 2,789 | 412 | 378 |
-| test-other | 2,939 | 52,343 | 15.10 % | 6,040 | 1,141 | 724 |
-| fixtures `test-clean-20` | 20 | 541 | 4.81 % | 22 | 1 | 3 |
+| Set | Utterances | Reference words | WER | Substitutions | Deletions | Insertions | v1 WER |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| dev-clean (model selection) | 2,703 | 54,402 | 3.75 % | 1,621 | 210 | 211 | 6.57 % |
+| dev-other | 2,864 | 50,948 | 10.26 % | 4,109 | 679 | 441 | 14.93 % |
+| test-clean | 2,620 | 52,576 | 4.21 % | 1,739 | 247 | 226 | 6.81 % |
+| test-other | 2,939 | 52,343 | 10.42 % | 4,249 | 655 | 551 | 15.10 % |
+| fixtures `test-clean-20` | 20 | 541 | 3.70 % | 18 | 1 | 1 | 4.81 % |
 
-On a 3,257.4 s LibriVox recording (*The Adventure of the Speckled Band*), read in 125 windows of 30 s as above through
-the Core AI build on an Apple M4 Max, the WER against the Project Gutenberg text is 5.80 %, next to 5.78 % for the
-float32 PyTorch path on the same windows (9,889 reference words; the LibriVox intro and outro are in the audio only).
+Long audio, in the 30 s windows of the contract: *The Adventure of the Speckled Band* (LibriVox, 3,257.4 s, 125
+windows), scored against the Project Gutenberg text (9,889 words; the LibriVox intro and outro are in the audio only):
+3.81 % through the float32 PyTorch path, 3.80 % through the Core AI build (the encoder on the GPU and this reader in
+float16 on the Neural Engine of an Apple M4 Max, two windows in flight). Reader v1: 5.78 % and 5.80 %.
 
 ## Training
 
-- Data: LibriSpeech train-clean-100 (CC BY 4.0), 28,539 utterances, 100.65 h. The 2,117 utterances with a word
-  outside the vocabulary were left out: 26,422 utterances, 92.8 h, 8,351,376 tokens. Soft tokens computed once in
-  float32, stored as float16.
+- Data: LibriSpeech train-clean-100, train-clean-360 and train-other-500 (CC BY 4.0), 281,241 utterances, 961.6 h,
+  read as one set per epoch. The 19,755 utterances with a word outside the vocabulary were left out: 261,486
+  utterances, 890.2 h, 80,121,094 tokens. Soft tokens computed once in float32 (TF32 off), stored as float16.
 - dev-clean chose the reader body and the epoch; test-clean and test-other were decoded once, after training.
 - Objective: CTC over the 200,000 words and blank (`torch.nn.functional.ctc_loss`, float32, `zero_infinity`, mean of
   loss / target length).
 - Optimizer: AdamW, peak learning rate 0.0003, weight decay 0.01 on matrices, 500 warmup steps, then cosine to zero at
-  step 16,520; batches of 16 utterances bucketed by length; gradient clipping 1.0; float32 throughout.
-- 8 epochs, 13,216 steps; the best dev-clean WER came at epoch 8 (these weights). Training stopped on its active-time
-  budget (2.72 h of 3.0 h).
-- Hardware: one Apple M4 Max (128 GB), torch 2.11.0 on MPS, macOS 27.0 (26A428); 3.64 h of wall clock, 1.02 h of it
-  paused while other jobs measured on the same Mac.
-- Body chosen after one epoch each of three candidates on dev-clean: a 4-layer transformer from scratch (13.48 %, this
-  one), EmbeddingGemma 2 text layers 0–3 (14.60 %), a per-token MLP (25.21 %).
-
-The command that continued the chosen body, from the `reader/train/` directory of the eg2-transcribe repository:
-
-```sh
-mkdir -p runs/v1 && cp runs/cmp_a/ckpt_epoch1.pt runs/v1/
-for e in $(seq 2 10); do  # one process per epoch
-  .venv/bin/python -I train.py --arch a --cache cache --vocab ../vocab.txt --table ../word_table.npy \
-    --run-dir runs/v1 --resume runs/v1/ckpt_epoch$((e - 1)).pt --epochs $e --schedule-epochs 10 \
-    --max-hours 3 --patience 2 || break
-  [ "$(jq -r .stop_reason runs/v1/metrics.json)" = "reached $e epochs" ] || break  # early stop or budget
-done
-```
+  step 163,430; batches of 16 utterances bucketed by length; gradient clipping 1.0; float32 weights and activations,
+  the training matmuls in TF32 (A100); the dev, test and fixture WER are decoded in full float32.
+- 9 epochs, 147,087 steps; the best dev-clean WER came at epoch 7 (these weights); early stop after 2 epochs without
+  a better dev-clean WER.
+- Hardware: one NVIDIA A100-SXM4-80GB (Google Colab), torch 2.11.0+cu130, CUDA 13.0; 4.38 h of training plus 2 min of
+  dev decoding.
+- Body chosen on dev-clean between two bodies trained the same way: 4 layers (13.3 M parameters, 3.88 % at its best
+  epoch 8) and 6 layers (19.6 M, 3.75 % at epoch 7, these weights).
+
+The command of the released run is in `RELEASE.md`.
 
 ## Word table
 
@@ -115,17 +111,17 @@ done
   code point, apostrophes kept.
 - `word_table.npy`: the EmbeddingGemma 2 text embedding of each word with the prompt `task: search result | query: `
   (sentence-transformers, float32, then stored as float16 and L2-normalized). Each of 100 random words, embedded
-  again, comes back at rank 1.
+  again, comes back at rank 1. Both files are those of v1.
 - Out of vocabulary: test-clean 220 of 52,576 words (0.42 %), test-other 283 of 52,343 (0.54 %). These are always
   errors.
 
 ## Limits
 
-- By word length: on 2,518 aligned words of 100 test-clean utterances, the word error rate is 2.8 % for words of 1–2
-  tokens (up to 80 ms), 3.4 % for 3–5 tokens and 9.7 % for 6 or more.
-- The most frequent substitutions on test-clean are function words (in → and 41, and → in 39, the → a 20); homophones
-  are 12.4 % of all substitutions, which a reader without a language model cannot tell apart.
-- English read speech only (audiobooks); trained on utterances of up to 24.5 s and measured on utterances of up to
+- By word length: on 2,518 aligned words of 100 test-clean utterances, the word error rate is 1.8 % for words of 1–2
+  tokens (up to 80 ms), 2.0 % for 3–5 tokens and 6.1 % for 6 or more.
+- The most frequent substitutions on test-clean are function words (and → in 27, in → and 22, a → the 16); homophones
+  are 12.9 % of all substitutions, which a reader without a language model cannot tell apart.
+- English read speech only (audiobooks); trained on utterances of up to 29.7 s and measured on utterances of up to
   35.2 s; longer audio goes through the 30 s windows above.
 
 ## License and credits
@@ -133,7 +129,7 @@ done
 - This repository: Apache-2.0 (`LICENSE`); see `NOTICE`.
 - LibriSpeech: V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain
   audio books", ICASSP 2015, https://www.openslr.org/12, CC BY 4.0. The reader weights are trained on
-  train-clean-100.
+  train-clean-100, train-clean-360 and train-other-500.
 - LibriSpeech lexicon: https://www.openslr.org/11, public domain.
 - EmbeddingGemma 2 (`google/embeddinggemma-2`, Google DeepMind): Apache 2.0. It produces the soft tokens the reader
   reads and the word table. Deployments must adhere to the

RELEASE.md

@@ -1,8 +1,29 @@
-# Reader v1
+# Reader v2
 
-Released 2026-10-14. Reads EmbeddingGemma 2 audio soft tokens as words. Greedy decoding, no language model: **test-clean WER 6.81 %, test-other WER 15.10 %**.
+Released 2026-10-15. Reads EmbeddingGemma 2 audio soft tokens as words. Greedy decoding, no language model: **test-clean WER 4.21 %, test-other WER 10.42 %**. Trained on the three LibriSpeech training sets (960 h; 890 h after leaving out the utterances with a word outside the vocabulary); reader v1 was trained on train-clean-100 only.
 
-Pre-registered v1 line: test-clean <= 10 % and test-other <= 20 % (greedy, no LM). test-clean meets it (6.81 %), test-other meets it (15.10 %).
+Pre-registered v2 line: test-clean <= 5.5 % and test-other <= 12 % (greedy, no LM). test-clean meets it (4.21 %), test-other meets it (10.42 %).
+
+## v2 and v1
+
+Same encoder, vocabulary, word table, objective, optimizer settings and decoding; the training data and the machine differ (and the body depth, when the table says so).
+
+| Set | v1 WER | v2 WER | Change (points) |
+|---|---:|---:|---:|
+| dev-clean | 6.57 % | 3.75 % | -2.81 |
+| dev-other | 14.93 % | 10.26 % | -4.66 |
+| test-clean | 6.81 % | 4.21 % | -2.60 |
+| test-other | 15.10 % | 10.42 % | -4.68 |
+| fixtures `test-clean-20` | 4.81 % | 3.70 % | -1.11 |
+
+| | v1 | v2 |
+|---|---|---|
+| Training data | train-clean-100, 26,422 utterances, 92.8 h | 960 h set, 261,486 utterances, 890.2 h |
+| Body | arch a, 4 layers, 13,268,994 parameters | arch a, 6 layers, 19,573,762 parameters |
+| Epochs (released) | 8 (8) | 9 (7) |
+| Steps per epoch | 1,652 | 16,343 |
+| Machine | Apple M4 Max, torch 2.11.0 MPS | NVIDIA A100-SXM4-80GB, torch 2.11.0+cu130 CUDA 13.0 |
+| Precision | float32 (MPS) | training matmuls in TF32 (A100), evaluation in fp32 |
 
 ## Files
 
@@ -10,16 +31,16 @@ Pre-registered v1 line: test-clean <= 10 % and test-other <= 20 % (greedy, no LM
 |---|---:|---|
 | `vocab.txt` | 1,737,588 | `bb3b3c70379082333e1bbd90d07ca69fe5c0096bd1c3b2b3f83edd65f1062a68` |
 | `word_table.npy` | 307,200,128 | `b2d0b13cc70c492f1444b8a42deab437d25254eac47d752d777d27b65377faa8` |
-| `reader.pt` | 53,095,605 | `67a06a857b9aeb066347f8947e4faeed11d2ec3e89e0c23974be0a2c20243d5b` |
-| `reader_config.json` | 2,019 | `68efaabae4c2eaa8ae2a5abf8f85fab115791b13178ee3a22e57a0cec727b94e` |
+| `reader.pt` | 78,323,637 | `d605f6fba8687497be6b5fd717ea6c51beaee5d5f69c30a64225da96ebe0d8c2` |
+| `reader_config.json` | 2,080 | `3aede371f7fc1b69bd948a3fce6f861e44eedb237d7b6dce360063ea007a90ed` |
 
-`reader.pt` and `word_table.npy` are ignored by git (`*.pt`, `*.npy` in `.gitignore`); the SHA-256 above identifies them. `reader_config.json` is the contract for ports: input, output, logits, decoding and the 30 s window policy of the encoder.
+`vocab.txt` and `word_table.npy` are the files of v1, unchanged. `reader.pt` and `word_table.npy` are ignored by git (`*.pt`, `*.npy` in `.gitignore`); the SHA-256 above identifies them. `reader_config.json` is the contract for ports: input, output, logits, decoding and the 30 s window policy of the encoder.
 
 ## What the reader is
 
 - Input: EmbeddingGemma 2 audio soft tokens, float32 `[T, 512]`, 25 per second (`embed_audio` output of `google/embeddinggemma-2` revision 914f7f89; the encoder is frozen).
-- Body: a bidirectional transformer encoder trained from scratch: LayerNorm -> Linear 512->512, 4 pre-LayerNorm layers (d 512, 8 heads, FFN 2048 GELU, rotary (base 10000) positions), LayerNorm -> Linear 512->768. Attention covers the whole input. 13,268,994 parameters in total.
-- Output per token: a unit vector in the 768-d text space and a blank logit. Logits over the 200,000 words = scale x cosine to `word_table.npy`, scale = 91.218 (learned); blank is class N. Decoding: argmax per token, merge repeats, drop blanks.
+- Body: a bidirectional transformer encoder trained from scratch: LayerNorm -> Linear 512->512, 6 pre-LayerNorm layers (d 512, 8 heads, FFN 2048 GELU, rotary (base 10000) positions), LayerNorm -> Linear 512->768. Attention covers the whole input. 19,573,762 parameters in total.
+- Output per token: a unit vector in the 768-d text space and a blank logit. Logits over the 200,000 words = scale x cosine to `word_table.npy`, scale = 216.351 (learned); blank is class N. Decoding: argmax per token, merge repeats, drop blanks.
 
 ## WER
 
@@ -27,105 +48,114 @@ Greedy, no language model, every utterance of each set. Normalization of referen
 
 | Set | Utterances | Reference words | WER | Substitutions | Deletions | Insertions |
 |---|---:|---:|---:|---:|---:|---:|
-| dev-clean (model selection) | 2,703 | 54,402 | 6.57 % | 2,823 | 375 | 374 |
-| dev-other | 2,864 | 50,948 | 14.93 % | 5,885 | 1,089 | 631 |
-| test-clean | 2,620 | 52,576 | 6.81 % | 2,789 | 412 | 378 |
-| test-other | 2,939 | 52,343 | 15.10 % | 6,040 | 1,141 | 724 |
-| fixtures `test-clean-20` | 20 | 541 | 4.81 % | 22 | 1 | 3 |
+| dev-clean (model selection) | 2,703 | 54,402 | 3.75 % | 1,621 | 210 | 211 |
+| dev-other | 2,864 | 50,948 | 10.26 % | 4,109 | 679 | 441 |
+| test-clean | 2,620 | 52,576 | 4.21 % | 1,739 | 247 | 226 |
+| test-other | 2,939 | 52,343 | 10.42 % | 4,249 | 655 | 551 |
+| fixtures `test-clean-20` | 20 | 541 | 3.70 % | 18 | 1 | 1 |
 
 The fixture row decodes `fixtures/test-clean-20/<id>/soft_tokens.npy` (the float32 CPU reference soft tokens). The reader's output on each of them, for comparing ports of the reader:
 
 | Utterance | Word errors | Reader output |
 |---|---:|---|
-| `1089-134686-0000` | 2 / 28 | he hoped there would be stew for dinner turnips and carrots and bruised potatoes and fat muttered pieces to be ladled out in thick peppered flour fatten sauce |
-| `1089-134686-0002` | 3 / 18 | after early nightfall fall all the yellow lamps would light up here and there the squalid quarter of the brawls |
-| `1188-133604-0000` | 2 / 19 | you will find me continually speaking of four men tion sublime turner and tintoret in almost the same terms |
-| `1188-133604-0001` | 2 / 18 | the unite every quality and sometimes you will find me referring to them as colorists sometimes as curio |
-| `121-121726-0000` | 2 / 17 | also a popular can drives whereby love making may be suspended but not stopped during the picnic season |
-| `121-121726-0001` | 1 / 8 | hurrah the tiresome product of a tireless tongue |
-| `1221-135766-0000` | 1 / 37 | how strange it seemed to the sad woman as she watched the growth and the beauty that became every day more brilliant and the intelligence that through its quivering sunshine over the tiny features of this child |
-| `1221-135766-0001` | 2 / 48 | god as a direct consequence of the sin which man thus punished had given her a lovely child whose place was on that same dishonored bottom to connect her parent for ever with the race and descent of mortals and to be finally a blessed soul in heaven |
-| `1284-1180-0000` | 0 / 24 | he wore blue silk stockings blue knee pants with gold buckles a blue ruffled waist and a jacket of bright blue braided with gold |
-| `1284-1180-0001` | 1 / 26 | his hat had a peaked crown at a flat brim and around the brim was a row of tiny golden bells that tinkled when he moved |
-| `1320-122612-0000` | 1 / 41 | since the period of art tale the active spirit of the country has surrounded it with a belt of rich and thriving settlements though none but the hunter or the savage is ever known even now to penetrate its wild recesses |
-| `1320-122612-0001` | 0 / 30 | the dews were suffered to exhale and the sun had dispersed the mists and was shedding a strong and clear light in the forest when the travelers resumed their journey |
-| `1580-141083-0000` | 2 / 30 | i will endeavor in my statement to avoid such terms as would serve to limit the events to any particular place or give a clew as to the people concerned |
+| `1089-134686-0000` | 2 / 28 | he hoped there would bestow stew for dinner turnips and carrots and bruised potatoes and fat mutton pieces to be ladled out in thick peppered flour fatten sauce |
+| `1089-134686-0002` | 1 / 18 | after early nightfall the yellow lamps would light up here and there the squalid quarter of the broths |
+| `1188-133604-0000` | 2 / 19 | you will find me continually speaking of four men titian holborn turner and tintoret a in almost the same terms |
+| `1188-133604-0001` | 1 / 18 | they unite every quality and sometimes you will find me referring to them as colorists sometimes as kio |
+| `121-121726-0000` | 0 / 17 | also a popular contrivance whereby love making may be suspended but not stopped during the picnic season |
+| `121-121726-0001` | 0 / 8 | harangue the tiresome product of a tireless tongue |
+| `1221-135766-0000` | 0 / 37 | how strange it seemed to the sad woman as she watched the growth and the beauty that became every day more brilliant and the intelligence that threw its quivering sunshine over the tiny features of this child |
+| `1221-135766-0001` | 1 / 48 | god as a direct consequence of the sin which man thus punished had given her a lovely child whose place was on that same dishonoured bottom to connect her parent for ever with the race and descent of mortals and to be finally a blessed soul in heaven |
+| `1284-1180-0000` | 1 / 24 | he wore blue silk stockings blue knee pans with gold buckles a blue ruffled waist and a jacket of bright blue braided with gold |
+| `1284-1180-0001` | 1 / 26 | his hat had a peaked crown in a flat brim and around the brim was a row of tiny golden bells that tinkled when he moved |
+| `1320-122612-0000` | 1 / 41 | since the period of our the active spirit of the country has surrounded it with a belt of rich and thriving settlements though none but the hunter or the savage is ever known even now to penetrate its wild recesses |
+| `1320-122612-0001` | 2 / 30 | the dews were suffered to exhale and the sun had dispersed the mists and was shedding a strong and clear light in the forests when the travellers resumed their journey |
+| `1580-141083-0000` | 1 / 30 | i will endeavor in my statement to avoid such terms as would serve to limit the events to any particular place or give a clue as to the people concerned |
 | `1580-141083-0001` | 0 / 34 | i had always known him to be restless in his manner but on this particular occasion he was in such a state of uncontrollable agitation that it was clear something very unusual had occurred |
 | `1995-1826-0000` | 0 / 27 | in the debate between the senior societies her defence of the fifteenth amendment had been not only a notable bit of reasoning but delivered with real enthusiasm |
-| `1995-1826-0001` | 1 / 27 | the south she had not thought of seriously and yet knowing of its delightful hospitality and mild climate she was not averse to charlestown or new orleans |
-| `2094-142345-0001` | 0 / 21 | but the windows are patched with wooden panes and the door i think is like the gate it is never opened |
-| `2094-142345-0002` | 2 / 42 | for it is a solid heavy handsome door and must once have been in the habit of shedding with a sonorous bang behind the liveried lackey who had just seen his master and mistress off the grounds in a carriage and pair |
-| `2300-131720-0000` | 1 / 13 | the paris plant like that of the crystal palace was a temporary exhibit |
-| `2300-131720-0001` | 3 / 33 | the london plat was last temporary but not permanent supplying before it was torn out no fewer than three thousand lamps in hotels churches stores and dwellings in the vicinity of holborn |
+| `1995-1826-0001` | 0 / 27 | the south she had not thought of seriously and yet knowing of its delightful hospitality and mild climate she was not averse to charleston or new orleans |
+| `2094-142345-0001` | 1 / 21 | but the windows are patched with wooden panes and the door i think is like the gate it isn't never opened |
+| `2094-142345-0002` | 3 / 42 | for it is a solid heavy handsome door and must once have been in the habit of shedding with a sonorous bang behind the liveried lackey who had just seen his master and mistress off the grounds in a carriage and pear |
+| `2300-131720-0000` | 2 / 13 | but paris plant like that of the crystal palace was a temporary exhibit |
+| `2300-131720-0001` | 1 / 33 | the london plant was last temporary but not permanent supplying before it was torn out no fewer than three thousand lamps in hotels churches stores and dwellings in the vicinity of holborn viaduct |
 
 ## Training
 
-- Data: LibriSpeech train-clean-100 (CC BY 4.0), 28,539 utterances, 100.65 h. The 2,117 utterances with a word outside the vocabulary were left out: 26,422 utterances, 92.8 h, 8,351,376 tokens. Soft tokens computed once (float32, stored float16).
+- Data: LibriSpeech train-clean-100, train-clean-360 and train-other-500 (CC BY 4.0), 281,241 utterances, 961.6 h, read as one set per epoch. Utterances with a word outside the vocabulary were left out: 261,486 utterances, 890.2 h, 80,121,094 tokens in training.
+
+| Split | Utterances | Hours | Left out (word outside the vocabulary) | In training | Hours in training |
+|---|---:|---:|---:|---:|---:|
+| train-clean-100 | 28,539 | 100.65 | 2,117 | 26,422 | 92.79 |
+| train-clean-360 | 104,014 | 363.82 | 7,532 | 96,482 | 336.45 |
+| train-other-500 | 148,688 | 497.17 | 10,106 | 138,582 | 461.00 |
+
+- Soft tokens: computed once on the training machine in float32 with TF32 off (stored float16), 16 utterances of similar length per encoder call (checked against one at a time: per-token cosine min 0.999999994, max |difference| 0.00051). Check before use: the 20 fixture utterances (5,036 tokens) against the float32 CPU reference soft tokens: per-token cosine min 0.9999999978, max |difference| 0.00036.
+- The cache index (ids, samples, frames, tokens, transcripts) of dev-clean, dev-other, test-clean, test-other and train-clean-100 is identical to the v1 cache index.
 - Dev: dev-clean, used for the choice of the reader body and for early stopping. test-clean and test-other were decoded once, after training.
 - Objective: CTC over the 200,000 words + blank (`torch.nn.functional.ctc_loss`, float32, zero_infinity, mean of loss / target length).
-- Optimizer: AdamW, peak learning rate 0.0003, weight decay 0.01 on matrices, linear warmup over 500 steps, then cosine to zero at step 16,520, batch 16 utterances bucketed by length, gradient clipping 1.0, float32 throughout.
-- Length: 8 epochs, 13,216 steps; best dev-clean WER at epoch 8 (the released weights). Stop: active-time budget: 2.72 h used over 8 epochs, one more (~0.34 h) would pass 3.0 h.
-- Time: 3.64 h of training wall clock (1.02 h of it paused while other jobs measured on the same Mac) plus 6 min of dev decoding. Apple M4 Max (128 GB), torch 2.11.0 MPS, macOS 27.0 (26A428).
+- Optimizer: AdamW, peak learning rate 0.0003, weight decay 0.01 on matrices, linear warmup over 500 steps, then cosine to zero at step 163,430, batch 16 utterances bucketed by length, gradient clipping 1.0, float32 weights and activations; training matmuls in TF32 (A100), evaluation in fp32: the dev, test and fixture WER are decoded in full float32, as in v1.
+- Length: 9 epochs, 147,087 steps; best dev-clean WER at epoch 7 (the released weights). Stop: early stop: no better dev-clean WER for 2 epochs.
+- Time: 4.38 h of training plus 2 min of dev decoding on one NVIDIA A100-SXM4-80GB (Google Colab, torch 2.11.0+cu130, CUDA 13.0).
+- Colab compute units: 95.7 in total (both runtimes: soft tokens, training, evaluation), 6.77 per hour per A100 runtime; about 22 of them were the two runtimes waiting idle after the work, before they were deleted.
 
-| Epoch | Steps | Train loss (last 100) | dev-clean WER | Wall (min) | Paused (min) |
+| Epoch | Steps | Train loss (last 100) | dev-clean WER | Training (min) | Dev decoding (s) |
 |---:|---:|---:|---:|---:|---:|
-| 1 | 1,652 | 1.103 | 13.48 % | 22.2 | 3.0 |
-| 2 | 3,304 | 0.434 | 9.24 % | 22.2 | 2.0 |
-| 3 | 4,956 | 0.281 | 7.99 % | 45.7 | 21.9 |
-| 4 | 6,608 | 0.199 | 7.24 % | 22.0 | 3.4 |
-| 5 | 8,260 | 0.136 | 7.07 % | 34.5 | 16.7 |
-| 6 | 9,912 | 0.094 | 6.84 % | 20.8 | 2.8 |
-| 7 | 11,564 | 0.057 | 6.74 % | 20.1 | 1.0 |
-| 8 | 13,216 | 0.028 | 6.57 % | 31.1 | 10.2 |
-
-Command that continued the chosen body (paths as in `train/README.md`):
+| 1 | 16,343 | 0.318 | 6.08 % | 29.2 | 15 |
+| 2 | 32,686 | 0.242 | 5.16 % | 29.5 | 12 |
+| 3 | 49,029 | 0.203 | 4.57 % | 29.3 | 12 |
+| 4 | 65,372 | 0.170 | 4.38 % | 29.2 | 12 |
+| 5 | 81,715 | 0.154 | 4.10 % | 29.2 | 12 |
+| 6 | 98,058 | 0.103 | 3.83 % | 29.2 | 12 |
+| 7 | 114,401 | 0.082 | 3.75 % | 29.1 | 12 |
+| 8 | 130,744 | 0.052 | 3.77 % | 29.2 | 12 |
+| 9 | 147,087 | 0.037 | 3.79 % | 29.2 | 12 |
+
+Command of the released run (paths as on the training machine):
 
 ```sh
-mkdir -p runs/v1 && cp runs/cmp_a/ckpt_epoch1.pt runs/v1/
-for e in $(seq 2 10); do  # one process per epoch, see train/README.md "Memory on Apple silicon"
-  .venv/bin/python -I train.py --arch a --cache cache --vocab ../vocab.txt --table ../word_table.npy \
-    --run-dir runs/v1 --resume runs/v1/ckpt_epoch$((e - 1)).pt --epochs $e --schedule-epochs 10 \
-    --max-hours 3 --patience 2 || break
-  [ "$(jq -r .stop_reason runs/v1/metrics.json)" = "reached $e epochs" ] || break  # early stop or budget
-done
+python -u /content/r7/code/train/train.py --arch a --layers 6 --cache /content/r7/cache --vocab /content/r7/assets/reader/vocab.txt --table /content/r7/assets/reader/word_table.npy --run-dir /content/r7/runs/v2a6 --device cuda --train-split train-clean-100,train-clean-360,train-other-500 --dev-split dev-clean --epochs 10 --schedule-epochs 10 --patience 2 --max-hours 4.727 --matmul tf32 --log-every 500 --stop-file /content/r7/control/stop_v2a6 --resume /content/r7/runs/v2a6/ckpt_epoch1.pt
 ```
 
-### Candidates (one epoch each, the same data and schedule)
+### Candidates (epoch 1 of each, the same data and schedule)
 
-| Body | Parameters | dev-clean WER after 1 epoch | Train loss (last 100) | Epoch, min (paused) | Peak MPS (GB) |
-|---|---:|---:|---:|---:|---:|
-| a: transformer from scratch (4 layers) | 13,268,994 | 13.48 % | 1.103 | 19.3 (3.0) | 38.6 |
-| b: EmbeddingGemma 2 text layers 0..3 | 22,427,394 | 14.60 % | 2.104 | 23.6 (4.6) | 28.9 |
-| c: per-token MLP | 1,314,306 | 25.21 % | 1.788 | 16.2 (26.6) | 10.0 |
+| Body | Parameters | dev-clean WER after 1 epoch | Train loss (last 100) | Epoch (min) |
+|---|---:|---:|---:|---:|
+| a, 4 layers | 13,268,994 | 6.27 % | 0.322 | 27.5 |
+| a, 6 layers | 19,573,762 | 6.08 % | 0.318 | 29.2 |
+
+Both bodies, trained to the end (early stopping on dev-clean, patience 2) and decoded the same way:
 
-Epoch minutes count training only (no dev decoding); the paused time, while other jobs measured on the same Mac, is shown in parentheses and not counted.
+| Body | Epochs | Best epoch | dev-clean | dev-other | test-clean | test-other | fixtures |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| a, 4 layers | 10 | 8 | 3.88 % | 10.77 % | 4.32 % | 10.88 % | 3.14 % |
+| a, 6 layers (released) | 9 | 7 | 3.75 % | 10.26 % | 4.21 % | 10.42 % | 3.70 % |
 
-Chosen: `cmp_a` (lowest dev-clean WER (greedy, no LM) after one epoch; tie -> fewer parameters; test sets not used).
+Chosen: arch a with 6 layers. After one epoch its dev-clean WER was the lower one (6.08 % against 6.27 %), so both runs went on; each stopped early on dev-clean, the 6-layer run at epoch 9 (best epoch 7, 3.75 %) and the 4-layer run at epoch 10 (best epoch 8, 3.88 %). The choice uses dev-clean only. The 6-layer run trained epoch 1 with --epochs 1 and then continued from that checkpoint with --epochs 10 (the command above); the cosine schedule spans 10 epochs in both.
 
 ## Vocabulary and word table
 
-- `vocab.txt`: 200,000 words = every word of the LibriSpeech lexicon (openslr.org/resources/11, public domain), lower case, sorted by code point, apostrophes kept.
+- `vocab.txt`: 200,000 words = every word of the LibriSpeech lexicon (openslr.org/resources/11, public domain), lower case, sorted by code point, apostrophes kept (as in v1).
 - `word_table.npy`: EmbeddingGemma 2 text embedding of each word with the prompt `task: search result | query: ` (sentence-transformers, float32, then stored as float16, L2-normalized). Check: each of 100 random words, embedded again, comes back at rank 1 (top-1 1.000).
 - Out of vocabulary: test-clean 220 of 52,576 words (0.42 %), test-other 283 of 52,343 (0.54 %); fixtures 1 word (`chiaroscurists`). These are always errors.
 
 ## Known weaknesses
 
-- Long words err most: on the 2,518 aligned words of 100 test-clean utterances, the word error rate rises with word length: 2.8 % for words of 1-2 tokens (<= 80 ms, n 824), 3.4 % for 3-5 tokens (n 700) and 9.7 % for 6+ tokens (n 994). The frozen-embedding probe before training had the opposite profile (short words hardest); the trained reader reverses it.
-- Substitutions on test-clean: 2,789 in 2,234 distinct pairs; most frequent in -> and (41), and -> in (39), the -> a (20), an -> and (20), a -> the (14), is -> as (11), of -> a (10), this -> the (8). Homophones (same lexicon pronunciation): 2 of the top 20 pairs, 12.4 % of all substitutions; a reader without a language model cannot tell them apart.
+- By word length: on the 2,518 aligned words of 100 test-clean utterances, the word error rate is 1.8 % for words of 1-2 tokens (<= 80 ms, n 824), 2.0 % for 3-5 tokens (n 700) and 6.1 % for 6+ tokens (n 994).
+- Substitutions on test-clean: 1,739 in 1,420 distinct pairs; most frequent and -> in (27), in -> and (22), a -> the (16), the -> a (11), an -> and (9), this -> the (6), i've -> i (5), man -> men (5). Homophones (same lexicon pronunciation): 3 of the top 20 pairs, 12.9 % of all substitutions; a reader without a language model cannot tell them apart.
 - No language model, no punctuation or casing; trained only on read English audiobook speech.
-- Trained on whole LibriSpeech utterances of up to 24.5 s and measured on whole utterances of up to 35.2 s. Long audio goes through the encoder in 30 s windows (see `reader_config.json` window and "Long audio in 30 s windows" below).
+- Trained on whole LibriSpeech utterances of up to 29.7 s and measured on whole utterances of up to 35.2 s. Long audio goes through the encoder in 30 s windows (see `reader_config.json` window and "Long audio in 30 s windows" below).
 
 ## Long audio in 30 s windows
 
-The long fixture `fixtures/long-speckled-band` (a 3,257.4 s LibriVox recording) was read in 125 windows of 30 s, each stepping 653 tokens with a 96-token left overlap and its last token dropped, through the Core AI conversion of the encoder (GPU) and of this reader (fp16, Neural Engine) on an Apple M4 Max: WER 5.80 % against the book text (9,889 words; the LibriVox intro and outro are in the audio only), next to 5.78 % for the torch float32 path on the same windows. The labels of the 81,436 tokens are identical whether the windows run one at a time or two in flight. Records: `exports/pipeline/pipeline_long_fp16_ane_overlap.json` and `exports/pipeline/pipeline_long_torch.json` of the Core AI conversion work (not in this repository).
+The long fixture `fixtures/long-speckled-band` (a 3,257.4 s LibriVox recording) was read in 125 windows of 30 s, each stepping 653 tokens with a 96-token left overlap and its last token dropped, through the Core AI conversion of the encoder (GPU) and of this reader (fp16, Neural Engine) on an Apple M4 Max: WER 3.80 % against the book text (9,889 words; the LibriVox intro and outro are in the audio only), next to 3.81 % for the torch float32 path on the same windows. The labels of the 81,436 tokens are identical whether the windows run one at a time or two in flight. Records: `exports/pipeline_v2/pipeline_long_fp16_ane_overlap.json` and `exports/pipeline_v2/pipeline_long_torch.json` of the Core AI conversion work (not in this repository).
 
 ## Reproduce
 
-`train/README.md` lists the environment, the data downloads and every command; `train/model.py --self-test` checks the CTC objective.
+`train/README.md` lists the environment, the data downloads and every command; v2 ran the same scripts on CUDA with the three training splits joined by commas in `--train-split` (the command above). `train/model.py --self-test` checks the CTC objective.
 
 ## Licenses and credits
 
-- LibriSpeech: V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain audio books", ICASSP 2015, https://www.openslr.org/12, CC BY 4.0. The reader weights are trained on train-clean-100.
+- LibriSpeech: V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: an ASR corpus based on public domain audio books", ICASSP 2015, https://www.openslr.org/12, CC BY 4.0. The reader weights are trained on train-clean-100, train-clean-360 and train-other-500.
 - LibriSpeech lexicon: https://www.openslr.org/11, public domain.
 - EmbeddingGemma 2 (`google/embeddinggemma-2`, Google DeepMind): Apache 2.0; it produces the soft tokens and the word table. Deployments must adhere to the Gemma Prohibited Use Policy (https://ai.google.dev/gemma/prohibited_use_policy).
 - `reader.pt` and `word_table.npy` are this repository's own artifacts: the reader weights were trained with `train/train.py` on the LibriSpeech data above, and the table was computed with `train/make_table.py` from EmbeddingGemma 2 and the lexicon words.

SHA256SUMS

@@ -1,9 +1,9 @@
-# EmbeddingGemma-2-Transcribe-Reader, staged 2026-10-14. Verify with: shasum -a 256 -c SHA256SUMS
+# EmbeddingGemma-2-Transcribe-Reader, staged 2026-10-15 (reader v2; reader v1 is revision 1af9c5f5). Verify with: shasum -a 256 -c SHA256SUMS
 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  LICENSE
-941840e4078a82c5fe6c1c9a0254af231e1cb01d8455a49250565243523c1106  NOTICE
-55e63adcaff0822e40a3d800be4f7e9e2c53a77c24ae39fd016c9110a3f23dfc  README.md
-7ee442009c39d6942dbafdc4e7c004bd7a7f7447ca5407c7d22a8ff24e54f0c4  RELEASE.md
-67a06a857b9aeb066347f8947e4faeed11d2ec3e89e0c23974be0a2c20243d5b  reader.pt
-68efaabae4c2eaa8ae2a5abf8f85fab115791b13178ee3a22e57a0cec727b94e  reader_config.json
+f353028c569417fff6bcd8c1e2ee8d77efe765c2ad2238ec36ff64be574cda2c  NOTICE
+2224e017715355689bf5e5fcdb39c612d92939063c62d8633501c48613ffa867  README.md
+9d0ac2a5d246d8b35f6456891e9a2ae2f199ef44a51087d1fa367f4cd1f35994  RELEASE.md
+d605f6fba8687497be6b5fd717ea6c51beaee5d5f69c30a64225da96ebe0d8c2  reader.pt
+3aede371f7fc1b69bd948a3fce6f861e44eedb237d7b6dce360063ea007a90ed  reader_config.json
 bb3b3c70379082333e1bbd90d07ca69fe5c0096bd1c3b2b3f83edd65f1062a68  vocab.txt
 b2d0b13cc70c492f1444b8a42deab437d25254eac47d752d777d27b65377faa8  word_table.npy

reader.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:67a06a857b9aeb066347f8947e4faeed11d2ec3e89e0c23974be0a2c20243d5b
-size 53095605
+oid sha256:d605f6fba8687497be6b5fd717ea6c51beaee5d5f69c30a64225da96ebe0d8c2
+size 78323637

reader_config.json

@@ -1,5 +1,5 @@
 {
- "version": "v1",
+ "version": "v2",
  "input": {
   "shape": "[T, 512]",
   "dtype": "float32",
@@ -9,7 +9,7 @@
  "architecture": {
   "name": "a",
   "kind": "transformer encoder from scratch",
-  "layers": 4,
+  "layers": 6,
   "d_model": 512,
   "heads": 8,
   "ffn": 2048,
@@ -18,14 +18,14 @@
   "positions": "rotary (base 10000)",
   "input": "LayerNorm -> Linear 512->512",
   "output": "LayerNorm -> Linear 512->768",
-  "params": 13268994
+  "params": 19573762
  },
  "output": {
   "vector": "[T, 768] L2-normalized",
   "blank_logit": "[T]"
  },
  "logits": "scale * cos(vector, table[w]) for w in vocab; blank from blank_logit",
- "scale": 91.21830749511719,
+ "scale": 216.3509979248047,
  "vocab": "vocab.txt",
  "table": {
   "file": "word_table.npy",
@@ -43,34 +43,34 @@
  },
  "weights": {
   "file": "reader.pt",
-  "format": "torch state_dict, float32",
-  "code": "train/model.py Reader(arch, n_vocab, k)",
+  "format": "torch state_dict, float32; training matmuls in TF32 (A100), evaluation in fp32",
+  "code": "train/model.py Reader(arch, n_vocab, k, layers)",
   "keys": "body.* (reader body), blank.weight [1, 768], blank.bias [1], log_scale (scale = exp(log_scale))"
  },
  "n_vocab": 200000,
  "wer": {
   "dev-clean": {
-   "wer": 0.06566,
+   "wer": 0.03754,
    "utterances": 2703,
    "ref_words": 54402
   },
   "dev-other": {
-   "wer": 0.14927,
+   "wer": 0.10263,
    "utterances": 2864,
    "ref_words": 50948
   },
   "test-clean": {
-   "wer": 0.06807,
+   "wer": 0.04207,
    "utterances": 2620,
    "ref_words": 52576
   },
   "test-other": {
-   "wer": 0.15102,
+   "wer": 0.10422,
    "utterances": 2939,
    "ref_words": 52343
   },
   "fixtures": {
-   "wer": 0.04806,
+   "wer": 0.03697,
    "utterances": 20,
    "ref_words": 541
   },