Instructions to use junafinity/VeriSafe-Tanglish-1-STT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use junafinity/VeriSafe-Tanglish-1-STT with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download junafinity/VeriSafe-Tanglish-1-STT --local-dir VeriSafe-Tanglish-1-STT
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
- Built by VeriSafe
- VeriSafe-Tanglish-1-STT
Built by VeriSafe
Open-weights models, built, fine-tuned, quantized and served.
VeriSafe.Live company services
These are services of VeriSafe.Live, the company. They are not features of this model. In particular, this model's GGUF files do not run in llama.cpp and its MLX build is untested on Apple silicon: see Files and formats for what this model's files do.
| Offering | What you get |
|---|---|
| 🎯 Custom fine-tuning | Any open-weights model adapted to your data, domain and tone. |
| 🧪 Training from scratch | Purpose-built models designed and trained for your use case. |
| ⚡ Quantization (MLX / GGUF) | Lean builds that run fast on Apple Silicon and llama.cpp. |
| 🌐 Hosted API access | Instant access to served models, with no wait time. |
| 🔓 Uncensored models | Refusal-removed builds for research, red teaming and evaluation. |
VeriSafe-Tanglish-1-STT
🎙️ One T, one ழ, one model that hears both.
The logo is the whole idea. The navy T is English. The red ழ (zha) is the letter Tamil is proudest of, the sound English has no letter for. They are locked together because that is how millions of people actually talk: "Meeting-ku late aayiduchu, traffic semma." Two languages in one breath, switching mid-sentence, sometimes mid-word.
VeriSafe-Tanglish-1-STT is a 398M-parameter streaming speech-to-text model for Tamil, English and the space between them. It is a post-train of Google's EmbeddingGemma 2 audio tower: we took Gemma's ears and taught them Chennai. Its internal build name is STT-2: this card calls it STT-2 for short, and its files are named stt2_model.*. This repository holds this model only; where the card compares it with its predecessor STT-1 (v1.2), it gives the numbers only.
The best open streaming Tanglish (Tamil-English) model we tested. Among models we tested, STT-2 beats SraVaani-0.5-live, the only open streaming Tamil model we tested, on 8 of 8 sets at its 480 ms setting, every 95% interval below zero. The two Tanglish sets are scored with STT-2's own streaming rows (SPRING-INX test −5.13 [−5.53, −4.71], CS-FLEURS ta-en −3.49 [−4.61, −2.39]); the six Tamil-mode sets use STT-1.2's rows, because STT-2's Tamil mode is the same model, byte for byte (below). Details, caveats and the sets are in Streaming vs open streaming models.
Smaller than the open models it matches. Among models we tested: about 398M parameters, against about 430M for SraVaani-1.0 and 600M for IndicConformer-600M, and within about 1 point of the better of those two on each of the five Tamil sets (at most 1.12 behind, on Kathbath valid; ahead on IISc-MILE test). See the scoreboard.
Writes Tanglish the way people type it. Among models we tested: English words in Latin script, Tamil words in Tamil script (--mode codemix, the default). On CS-FLEURS ta-en the scorer's English-word error is 79.25 for STT-2's offline default, against 96.36 for IndicConformer-600M, which writes English words in Tamil script; STT-2's figure is 1.35 points worse than its predecessor STT-1.2's 77.90.
Each of these three lines holds among the models we tested, on our clips; we tested few models. We do not claim "best streaming overall", and we do not claim to beat Sarvam: Sarvam stays on this card as a reference bar (hosted, closed, offline), and on Tanglish STT-2 is Equal on one test set and Behind on the other.
New in STT-2 (2026-11-15): a code-mix reader fork. Code-mix mode now reads the tower through its own copy of everything between the tower and the heads (layer mix, input norm and projection, position conv, the 6-layer reader, output norm): 81 tensors and 43,330,317 added parameters, so the model grows from about 355M to about 398M parameters, still one file. The tower, the Tamil stack and the Tamil heads are STT-1's, byte for byte, so Tamil mode is identical to STT-1.2: 845 of 845 shared tensors are byte-equal, and the Tamil-mode text was identical on 336 of 336 CS-FLEURS ta-en clips, offline and streaming. Code-mix results from one test read, STT-2's finals against STT-1.2's finals (paired, 95% interval):
- Beats STT-1.2 on vlog: unseen Tamil-English YouTube vlog clips (96 clips), 52.94 vs 54.92, −1.98 [−3.42, −0.57].
- Level or better (point) on SPRING-INX test (32.24 vs 32.38, −0.13 [−0.29, +0.01]) and CS-FLEURS ta-en (51.06 vs 51.50, −0.44 [−1.02, +0.13]).
Decoding and flags are unchanged from STT-1 v1.2: partials streaming (480 ms, no look-ahead); two-pass finals at end of utterance. See the changelog.
It has two output modes from one tower:
codemix(default): English words in Latin script, Tamil words in Tamil script, the way Tanglish is typed. Uses the code-mix reader fork and the code-mix heads.tamil: everything in Tamil script, as the earlier builds did. Uses the Tamil reader and the Tamil heads, unchanged from STT-1.
Offline code-mix decoding uses CTC-assist rescoring J(0.3, 0) by default. Code-mix streaming uses 480 ms chunks with no look-ahead: partials are RNN-T greedy with the start cue; at the end of each utterance the reader re-reads the whole utterance with full attention and the final text is decoded from that pass with J(0.3, 0), as offline ("partials streaming (480 ms, no look-ahead); two-pass finals at end of utterance"). In tamil mode, offline decoding is greedy and J is opt-in with --ctc-assist; Tamil-mode streaming partials are the v1 Tamil stream, and its two-pass final is the Tamil offline greedy text.
Demo clip; the on-screen transcript was not produced by this model.
Contents: Highlights · The family · Files · Usage · Validation · Scoreboard · Streaming vs open streaming models · How it was made · Leakage · Known limits · Licence · Responsible use · Changelog · What's next · Credits
✨ Model highlights
| 🏆 The best open streaming Tanglish (Tamil-English) model we tested | Among models we tested: 8 of 8 paired wins against SraVaani-0.5-live, the only open streaming Tamil model we tested, at its 480 ms setting (WER differences from −2.30 to −9.28 points, every 95% interval below zero; SPRING-INX and CS-FLEURS with STT-2's own rows, the six Tamil-mode sets with STT-1.2's identical rows). Caveats: the CS-FLEURS win is Equal in the folded view (−0.69 [−1.70, +0.34]), SPRING may be in SraVaani-0.5-live's training data, and two sets depend on how its input is padded. See Streaming vs open streaming models. |
| 🪶 Smaller than the open models it matches | Among models we tested: about 398M parameters, against about 430M (SraVaani-1.0) and 600M (IndicConformer-600M); on the five Tamil sets within about 1 point of the better of the two (at most 1.12 behind, Kathbath valid; ahead on IISc-MILE test). |
| ✍️ Writes Tanglish the way people type it | Among models we tested: English in Latin script, Tamil in Tamil script. CS-FLEURS ta-en English-word error 79.25 (STT-2 offline default) against 96.36 for IndicConformer-600M, which writes English words in Tamil script (STT-2 is 1.35 worse than STT-1.2's 77.90). |
| 🆕 Code-mix reader fork (new in STT-2) | Code-mix mode reads the tower through its own layer mix and 6-layer reader (81 tensors, 43,330,317 added parameters; about 355M to about 398M). Tamil mode is STT-1.2's, byte for byte (845 of 845 shared tensors; text identical on 336 of 336 clips, offline and streaming). Beats STT-1.2 on unseen YouTube vlog clips (finals −1.98 [−3.42, −0.57], 96 clips); level or better (point) on SPRING-INX test (−0.13) and CS-FLEURS (−0.44). |
| 🧠 Base | EmbeddingGemma 2 audio tower (USM-style conformer, 303M), post-trained end to end. We say it plainly: this is Gemma's encoder, fine-tuned, not a model trained from zero. |
| 🔀 Two output modes | codemix (default) writes English in Latin script and Tamil in Tamil script; tamil writes everything in Tamil script. One tower, two reader stacks, two head sets, chosen with --mode. |
| ⚡ Streaming-native | One set of weights runs offline and streaming (480 ms chunks, no look-ahead). Partials streaming (480 ms, no look-ahead); two-pass finals at end of utterance: partials stay RNN-T greedy, and at the end of each utterance the reader re-reads it with full attention (the tower is not re-run), so the final text is the offline decode. It arrives after the utterance ends (STT-2: median 1.34 s, p90 3.03 s, at most 5.41 s on an idle 4-thread laptop CPU, 48 clips; the wait on a busy CPU was not measured for STT-2, see Known limits). |
| 🎯 Four heads | RNN-T and CTC for each mode. Offline code-mix, and the final text of code-mix streaming, re-rank the RNN-T beam with the CTC head (CTC-assist J(0.3, 0)). No n-gram LM ships with this release. |
| 🗣️ Tamil + English | About 2,100 hours of Tamil (read, extempore, conversational, broadcast; estimate, from the earlier build's data) plus Indian and US English. STT-2's code-mix stack was also trained on Tamil-English YouTube clips labelled with their own captions. |
| 📉 Code-mix development rows | Against STT-1.2, same decode: held-out YouTube vlog adopt set (342 clips) −2.61 [−3.61, −1.71]; vlog tune set (295) −2.791 [−3.559, −2.076]; SPRING-INX valid subset (2,000) −0.100 [−0.297, +0.091]; Vaani Tamil gloss (4,448) +0.027 [−0.115, +0.170]. Development sets, not test: see Validation results. |
| 🟪 Tanglish test vs Sarvam | SPRING-INX test (4,038 clips, number-fair): 32.24 vs Sarvam codemix 32.83, Equal (gap −0.59 [−1.05, −0.13]; the digit-free gap flips sign, +0.31 [−0.17, +0.80]). CS-FLEURS ta-en (336 clips, unseen): 50.99 vs 44.71, Behind (+6.28 [+4.92, +7.69]). Two-pass streaming finals: SPRING 32.24 (the offline text), CS-FLEURS 51.06. See the scoreboard. |
| 📦 Three formats, four files | Safetensors at full precision (float32), GGUF f16 and q8_0, and MLX 8-bit. GGUF f16 gave the same text as the safetensors on 30 of 30 checks and q8_0 on 26 of 30; MLX 8-bit: on 48 clips, WER minus safetensors +0.00 / +0.12 / −0.35 (the 0.2 keep rule passes). How each one runs is spelled out in Files. |
🧬 The family
| Build | What it is | Formats | Status |
|---|---|---|---|
| VeriSafe-Tanglish-1-STT (internal build STT-2; this card) | STT-1's tower, Tamil stack and Tamil heads, plus a code-mix reader fork and code-mix heads from a 12,000-step tower-frozen run, blended 0.5 / 0.5 with STT-1's code-mix weights. About 398M parameters | Safetensors (full precision), GGUF (f16, q8_0), MLX 8-bit (stt2_model.*) |
This repository. Research preview |
| STT-1 v1.2 (predecessor) | Code-mix main run, EMA, step 84,000. About 355M parameters. v1.1 and v1.2 changed decoding only | — | Superseded by STT-2; not in this repository |
| Tanglish stage 7 build | An earlier build (EMA, 72k steps). Tamil script only | — | Superseded; not in this repository |
| Tanglish stage 6 build | The earlier sibling (step 69k), Tamil-first | — | Superseded; not in this repository |
📁 Files and formats
Read this first.
- GGUF needs our own
code/transcribe.py(PyTorch). It does not run in llama.cpp or whisper.cpp: llama.cpp stops withunknown model architecture: 'indic-stt-eg2'(checked on release b12175 for an earlier build's files; the architecture string is unchanged and was not re-run on these files). A native ggml runner would be new work. - MLX is not tested on Apple silicon or Metal. It was tested on Windows x86-64 with the CPU backend only (
mlx-cpu); no Mac was available. - Safetensors is the reference format and runs through
code/transcribe.py(PyTorch, CPU or NVIDIA GPU), as does GGUF. MLX (mlx/run_mlx.py) has the same flags, including offline--ctc-assist, but no streaming finals and no--finals: its--streamingstays RNN-T greedy, and--streaming --ctc-assistis refused. No n-gram LM is included in this repository.
Files (sizes measured on the release folder, GB = 10^9 bytes):
| Path | Size | What |
|---|---|---|
safetensors/ |
1.63 GB | stt2_model.safetensors (full precision: 939 float32 tensors, 1,633,292,432 bytes: one tower, two reader stacks, two head sets), stt_config.json ("codemix_fork": true), eg2/ (EmbeddingGemma 2 configs), requirements, README |
gguf/ |
1.25 GB | stt2_model.f16.gguf 0.82 GB, stt2_model.q8_0.gguf 0.44 GB, requirements, README |
mlx/ |
0.46 GB | stt2_model.mlx8.safetensors (8-bit), layout.json, MLX port and runner, requirements, README |
tokenizer/ |
0.29 MB | spm.model (SentencePiece, 2,058 pieces) |
code/ |
0.16 MB | transcribe.py (the runner for safetensors and GGUF), ctc_assist.py, indic_stt/, export/, training config |
samples/ |
1.03 MB | 2 FLEURS-ta test clips and 1 LibriSpeech test-clean clip; expected outputs in refs_stt2.jsonl |
assets/ |
2.82 MB | Logos and the demo clip |
Whole repository: about 3.35 GB (49 files, with MANIFEST.sha256, which lists the sha256 and size of every other file). You only need one format folder plus code/ and tokenizer/ (GGUF files carry their own tokenizer).
| Format | How it runs | Decoding rows | Checked |
|---|---|---|---|
Safetensors, full precision (float32): safetensors/stt2_model.safetensors, 1.63 GB |
code/transcribe.py, PyTorch |
Both modes, RNN-T and CTC, offline and streaming, CTC-assist, streaming finals (two-pass, and --finals f) |
Every tensor byte-equal to the training candidate; the release reader output was bit-identical to the project's own code (both modes, offline and streaming, 5 of 5 clips, on a smoke build of the same code); two-pass finals equal the offline text on 48 of 48 clips |
GGUF f16: gguf/stt2_model.f16.gguf, 0.82 GB |
code/transcribe.py --weights FILE.gguf (reads with the gguf package, dequantizes to float32, runs in PyTorch). Not llama.cpp. |
Same as safetensors | Text identical to the safetensors on 30 of 30 (6 CS-FLEURS ta-en clips × 5 rows, both modes), and on 48 clips 47, 46 and 46 of 48 identical (code-mix offline default, code-mix greedy stream, Tamil offline; 7 word edits in 2,159 words; plain WER minus safetensors 0.00 / +0.12 / 0.00) |
GGUF q8_0: gguf/stt2_model.q8_0.gguf, 0.44 GB |
Same | Same | 26 of 30 on the 6 clips (Tamil rows 12 of 12; code-mix rows 14 of 18), and on 48 clips 39, 37 and 40 of 48 identical (45 word edits in 2,159 words, about 2%; plain WER minus safetensors −0.12 / −0.12 / 0.00) |
MLX 8-bit: mlx/stt2_model.mlx8.safetensors, 0.46 GB |
mlx/run_mlx.py --weights stt2_model.mlx8.safetensors, no PyTorch needed. Not tested on Apple silicon or Metal |
Same flags as transcribe.py, except the streaming finals and --finals: its --streaming is RNN-T greedy |
On 48 CS-FLEURS ta-en clips (code-mix offline default, code-mix greedy stream, Tamil offline): 42, 38 and 38 of 48 identical to the safetensors (44 word edits in 2,159 words, about 2%); plain WER minus safetensors +0.00 / +0.12 / −0.35, so the keep rule (at most +0.2) passes on each row. On 15 clips × 11 checks: 137 of 165 texts identical (the 3 samples 33 of 33). The port itself, run with float32 weights, gave 165 of 165 |
The GGUF files save download size only: the runner dequantizes them to float32, so RAM and speed are those of the full-precision model. The MLX runner dequantizes the 8-bit weights to float32 once, on load, so it also holds about 1.63 GB of weights in RAM (on the MLX CPU backend a quantized matmul was about 270 times slower than a float32 one). A 591-clip FLEURS-ta fidelity table of the GGUF and MLX files was not run for this release.
🚀 Usage
Python 3.12, from the repository root (the folder that holds safetensors/, code/ and tokenizer/). Each format folder has a README with the full detail; this is the short version. On macOS or Linux use .venv/bin/python and /.
Safetensors (PyTorch):
python -m venv .venv
.venv\Scripts\python -m pip install -r safetensors\requirements.txt
.venv\Scripts\python code\transcribe.py samples\fleurs_ta_test_000.wav
GGUF (our runner, not llama.cpp):
python -m venv .venv
.venv\Scripts\python -m pip install -r gguf\requirements.txt
.venv\Scripts\python code\transcribe.py samples\fleurs_ta_test_000.wav --weights gguf\stt2_model.q8_0.gguf
MLX 8-bit (Windows CPU tested; Apple silicon and Metal untested):
python -m venv .venv
.venv\Scripts\python -m pip install -r mlx\requirements.txt
.venv\Scripts\python mlx\run_mlx.py samples\fleurs_ta_test_000.wav --weights stt2_model.mlx8.safetensors
Two output styles, one model. --mode codemix (default) writes Tamil words in Tamil script and English words in Latin script, as Tanglish speakers type. --mode tamil writes everything in Tamil script, as the earlier builds did.
Decoding rows (code/transcribe.py for safetensors and GGUF; mlx/run_mlx.py has the same flags, except the streaming finals and --finals: its --streaming is the greedy stream and it refuses --streaming --ctc-assist):
| Row | Flags |
|---|---|
| Code-mix, offline: RNN-T + CTC-assisted rescoring J(0.3, 0) (default) | none |
| Code-mix, partials streaming (480 ms, no look-ahead); two-pass finals at end of utterance (default): partials RNN-T greedy with start cue; at the end of the utterance a full-attention reader pass, decoded with J(0.3, 0) (the offline text) | --streaming |
| Code-mix, 480 ms streaming, STT-1 v1.1 finals: the final text re-ranked with J(0.3, 0) on the streaming pass itself | --streaming --finals f |
| Code-mix, 480 ms streaming, greedy only (the v1 stream; also the partials' text) | --streaming --finals f --no-ctc-assist |
| Code-mix, offline RNN-T greedy with start cue | --no-ctc-assist |
| Tamil script, offline RNN-T greedy with start cue | --mode tamil |
| Tamil script, 480 ms streaming: partials the v1 Tamil stream; two-pass final, greedy (the Tamil offline text) | --mode tamil --streaming |
| Tamil script, 480 ms streaming, greedy only (the v1 Tamil stream) | --mode tamil --streaming --finals f |
| Tamil script, offline with the CTC-assisted rescoring (opt-in) | --mode tamil --ctc-assist |
| CTC greedy of the mode's CTC head | --head ctc (add --mode tamil for the Tamil head) |
Output. transcribe.py prints one line per file: the path, a tab, the text. It treats each file as one utterance and has no live-microphone loop, so with --streaming it prints the final text only; the partials' text is the --streaming --finals f --no-ctc-assist row. A live app would show the greedy text while the speaker talks, keep the tower's output, and at the end of the utterance run the reader once more with full attention and swap in the final. A live app must reset its state at each utterance: start the tower and the reader's position-conv state fresh when an utterance starts, and run the full pass over that utterance's frames only; otherwise the previous utterance leaks into this one and the final is no longer the offline text. With --json OUT.jsonl, each row carries "streaming", "ctc_assist" and "finals" ("two-pass", "f", or null offline). The row in use is printed to stderr (for the default on STT-2: codemix (code-mix reader fork), RNN-T, streaming, two-pass finals (full-attention reader pass at the end of the utterance, then ctc-assist J(0.3, 0)), partials greedy).
Other options: --weights FILE, --device cuda, --no-bf16, --threads N, --tokenizer FILE, --json OUT.jsonl. STT-1 v1.2 added one flag, --finals two-pass|f (with --streaming only; default two-pass): two-pass gives the final text the offline decode of the same flags, on a second, full-attention reader pass at the end of the utterance; f gives the STT-1 v1.1 finals, the same decode on the streaming pass. Utterances over 30 s get f, with a note on stderr: training clips were at most 30 s. --ctc-assist is RNN-T only (refused with --head ctc). With --streaming it decodes only the final text of each utterance, once, at its end: on by default in code-mix mode, off by default in tamil mode. Since STT-1 v1.2, --streaming --no-ctc-assist gives the two-pass greedy final (the offline greedy text); the v1 greedy stream is --streaming --finals f --no-ctc-assist. --mode tamil --streaming --finals f --ctc-assist runs, but it has no evaluated numbers. Output is normalized: lower case, no punctuation. Clips up to 60 s were evaluated.
Start cue: RNN-T greedy lowers blank by 1 until the first word, from the first frame where the mode's CTC head hears speech; it stops clips from staying silent at the start. It is part of the default decoding rows above (the earlier card's start-cue rows were a research decode; here they are what transcribe.py does).
CTC-assist J(0.3, 0): the offline code-mix default, and the final text of code-mix streaming, re-rank the RNN-T beam-4 hypotheses together with the CTC hypothesis (0.7 RNN-T + 0.3 CTC, exact scores). It needs the whole utterance, so in streaming the partials stay greedy and the re-rank runs once, at the end of the utterance. It is slower than greedy: offline, on a shared CPU, about 4 s per 13 s clip on top of greedy (measured on STT-1; an estimate, roughly 0.3 s per second of audio). It always runs on the CPU in float32, also with --device cuda. No n-gram LM ships.
Two-pass streaming finals: the final text arrives after the utterance ends. When the utterance ends, the reader (6 layers; on STT-2 the code-mix fork in code-mix mode) runs once more over the whole utterance with full attention, on the tower output already computed for the partials, and J(0.3, 0) re-ranks that pass on the CPU. On 48 CS-FLEURS ta-en clips (CPU, 4 threads, the laptop CPU idle: total load under 15% for about 3 minutes before each run, no other job), STT-2's wait for the two-pass final was median 1.34 s, p90 3.03 s, and at most 5.41 s, on a 26.5 s clip (the middle of three runs by p90; the three p90s were 3.04, 2.98 and 3.03 s); the --finals f finals in the same run waited median 1.26 s, p90 2.81 s and at most 4.96 s, so the second reader pass adds about 0.1 s at the median and 0.3 s at p90 (paired per clip; estimates). STT-2's wait on a busy CPU was not measured: STT-1.2's rose from p90 3.0 s idle to 4.3 to 4.7 s with a training job on the same laptop, and STT-2's is likely to rise in the same way (an estimate). STT-1.2's idle reading was taken on another day, so the two are not a paired comparison. With --device cuda the reader pass runs on the GPU and the re-rank on the CPU; the GPU wait was not measured for STT-2. Up to 30 s the wait grows with the length of the utterance; over 30 s the --finals f finals take over, whose wait has no cap. Partials are not delayed. Use --streaming --finals f --no-ctc-assist for greedy-only finals: no second pass and no wait. Streaming-latency figures for the partials are in Known limits.
--finals f (the STT-1 v1.1 finals) re-rank the streaming pass itself, with no cap on the wait; on the idle CPU of the STT-2 measurement they waited median 1.26 s, p90 2.81 s and at most 4.96 s (48 clips, 4 threads).
Python API (STT-2 files). Release.mix, read, encode, greedy and ctc_assist in code/transcribe.py need mode="codemix" or mode="tamil" on a file with the code-mix fork; they refuse mode=None there, so that a Tamil-stack output is never decoded with the code-mix heads or the reverse. The live loop is greedy(*read(x, lens, True, mode=m), mode=m) on mix(feats, mode=m); transcribe() and the command line pass the mode for you. The same holds for MlxModel.encode, ctc_text, rnnt_text and assist_text in mlx/run_mlx.py.
🧪 Validation results
These are development numbers. They are not the test tables below. STT-2 was picked on development sets with rules fixed before the run, and the test sets were read once, after the pick.
Code-mix mode, offline default decode (RNN-T + J(0.3, 0)) unless marked, number-fair WER %, STT-2 against STT-1.2 (the code-mix main run, the same decode), paired difference with a 95% clip-bootstrap interval (2,000 draws):
| Set (clips) | STT-2 | STT-1.2 | Difference | Note |
|---|---|---|---|---|
| YouTube vlog adopt set (342) | 55.45 | 58.06 | −2.61 [−3.61, −1.71] | Held-out vlog videos; the adopt check, run once on the pick |
| YouTube vlog adopt set, 480 ms partials (342) | 56.91 | 59.44 | −2.53 [−3.35, −1.71] | Greedy stream with the start cue |
| YouTube vlog tune set (295) | 51.377 | 54.168 | −2.791 [−3.559, −2.076] | Used in checkpoint selection |
| SPRING-INX valid subset (2,000) | 43.192 | 43.292 | −0.100 [−0.297, +0.091] | Used in checkpoint selection |
| Vaani Tamil gloss (4,448) | 28.731 | 28.705 | +0.027 [−0.115, +0.170] | Used in checkpoint selection |
These rows use this build's selection and adopt scoring (three decimals as logged), so they are not comparable with the validation rows of earlier builds' cards.
Tamil mode, RNN-T greedy, offline: STT-2's Tamil mode is STT-1's (EMA step 84,000), so these are STT-1's training-log rows; the Tamil-head rows of STT-2's selection run were equal to STT-1's. They are the checkpoint-selection subsets, not the same clips as the scoreboard's IndicVoices valid, Kathbath valid and FLEURS-ta test, so do not compare them with those rows:
| Set (clips) | WER % |
|---|---|
| IndicVoices valid, selection subset (4,051) | 31.54 |
| Kathbath valid, selection subset (1,602) | 23.53 |
| FLEURS-ta dev (374) | 25.39 |
📊 The scoreboard (no spin)
Word error rate, %, lower is better. Same clips and the same text normalisation for every system. Greedy decoding, offline, unless the column says otherwise. Bold = best in row. Numbers for the alternatives are our own measurements on our clips. Clip counts are in brackets.
Tamil and English rows are STT-1.2's. STT-2's Tamil mode is the same model: the tower, the Tamil stack and the Tamil heads are byte-equal to STT-1's (845 of 845 shared tensors), and the Tamil-mode text was identical on 336 of 336 CS-FLEURS ta-en clips offline and streaming. These rows were not decoded again for STT-2.
🟥 Tamil (Tamil mode, --mode tamil)
| Set (clips) | STT-2 = STT-1.2, greedy | 480 ms greedy stream | Stage 7, greedy | IndicConformer-600M | SraVaani-430M | Sarvam saaras:v4 |
|---|---|---|---|---|---|---|
| IndicVoices valid (600) ⚑ | 30.48 | 31.77 | 30.49 | 30.14 | 29.98 | — |
| FLEURS-ta test (591) | 28.15 | 29.83 | 28.53 | 27.10 | 27.92 | 23.78 |
| Kathbath valid (1000) ⚑ | 19.60 | 21.39 | 19.78 | 18.88 | 18.48 | — |
| IISc-MILE test (1000) ⚑ | 20.29 | 22.38 | 20.78 | 21.09 | 20.36 | 23.57 |
| DiarBench clean (2555) | 46.10 | 47.63 | 47.14 | 45.40 | 46.77 | — |
🟦 English
| Set (clips) | STT-2 = STT-1.2 | 480 ms greedy stream | Stage 7 | Whisper small | Whisper base | Sarvam saaras:v4 |
|---|---|---|---|---|---|---|
| LibriSpeech test-clean (2620) ⚑ | 5.43 | 6.27 | 5.66 | 3.96 | 5.56 | 2.62 |
| Svarah, Indian English (6227) | 25.22 | 29.27 | 27.87 | 10.37 | 14.12 | — |
| Monsoon en-IN (2102) | 20.83 | 29.62 | 28.54 | 8.45 | 10.65 | 4.82 |
Whisper models were told "en"; Sarvam was told en-IN; ours is told nothing. The English rows use Tamil mode (--mode tamil, Tamil head, greedy RNN-T with the start cue on), told nothing about the language; the 480 ms column is the same decode streamed (the greedy stream, --mode tamil --streaming --finals f; the default --mode tamil --streaming gives the two-pass final, which is the offline text). "Stage 7" is its plain greedy row. Whisper and Sarvam cells are carried from the stage 7 card.
🟪 Tanglish (code-mixed calls): the one that matters
All rows in this part are STT-2's own, from one test read after the pick. SPRING-INX Tamil test, 4,038 clips, number-fair scoring (numerals spelled out the same way for every system), against Sarvam's dedicated codemix mode. The reference is the raw-script text; Sarvam is scored as returned, every other system through the frozen loanword view. Gaps are row minus Sarvam on the same clips (2,000 paired resamples; negative = fewer errors).
| System | WER | Gap to Sarvam [95% paired] |
|---|---|---|
Sarvam saaras:v4, mode codemix |
32.83 | — |
| STT-2, code-mix, offline default (RNN-T + J(0.3, 0)) | 32.24 | −0.59 [−1.05, −0.13]; digit-free +0.31 [−0.17, +0.80] (3,625 clips) |
STT-2 default streaming: partials streaming (480 ms, no look-ahead); two-pass finals at end of utterance (--streaming) |
32.24 | −0.59 [−1.05, −0.13] (the same text as the offline default; no SPRING test clip is over 30 s) |
STT-2, code-mix, offline greedy + start cue (--no-ctc-assist) † |
33.02 | +0.18 [−0.24, +0.60] |
STT-2, code-mix, 480 ms streaming greedy + start cue (the partials; --streaming --finals f --no-ctc-assist) |
34.82 | +1.99 [+1.57, +2.40] |
| STT-1.2, offline default and two-pass finals | 32.38 | −0.45 [−0.91, −0.01] |
| Stage 7 greedy (Tamil script only) | 35.96 | +3.13 [+2.61, +3.68] |
| SraVaani-430M | 36.25 | +3.42 |
CS-FLEURS ta-en (code-switched Tamil-English read speech; 336 clips, one synthetic voice, not seen in training), against Sarvam saaras:v4, mode codemix (ta-IN, as returned). Number-fair scoring against the reference text; Sarvam's Latin words are scored as returned, every other system goes through the frozen loanword view. Gaps are row minus Sarvam on the same clips (2,000 paired resamples).
| System | Clips | WER | Gap to Sarvam [95% paired] |
|---|---|---|---|
Sarvam saaras:v4, mode codemix |
336 | 44.71 | — |
| STT-2, code-mix, offline default | 336 | 50.99 | +6.28 [+4.92, +7.69] |
STT-2 default streaming (--streaming; 4 of the 336 clips are over 30 s and get the --finals f finals) |
336 | 51.06 | +6.34 [+4.96, +7.74] |
STT-2, code-mix, offline greedy (--no-ctc-assist) † |
336 | 52.38 | +7.67 [+6.27, +9.07] |
| STT-2, code-mix, 480 ms streaming greedy (the partials) | 336 | 55.10 | +10.38 [+8.91, +11.87] |
| STT-1.2, offline default | 336 | 51.39 | +6.67 [+5.23, +8.14] |
| Stage 7 greedy | 336 | 57.12 | +12.40 [+10.51, +14.28] |
| IndicConformer-600M RNNT, offline greedy (open, no streaming checkpoint tested) ‡ | 336 | 52.82 | +8.11 (point) |
| SraVaani-1.0, offline greedy TDT (open) ‡ | 336 | 56.07 | +11.36 (point) |
† Scored from the greedy row of the same CTC-assist decode (the model's bf16 reader output saved by the evaluation, start penalty 1 and start cue, as --no-ctc-assist decodes), not from a separate run of transcribe.py --no-ctc-assist.
‡ Folded-view caveat. The rows above use the number-fair scoring in which Latin words count as written. IndicConformer writes English words in Tamil script, so it is penalised on English words in this view (the scorer's English-word error measure is 96.36 for it, against 79.25 for STT-2's offline default). Row minus STT-2's offline default (50.99): IndicConformer +1.83 [+0.70, +2.95] in this view, but −2.07 [−3.01, −1.15] in the folded view (hypothesis and reference both mapped to Tamil script), where IndicConformer is ahead; SraVaani-1.0 is +5.08 [+3.42, +6.86] here and +2.38 [+0.83, +4.04] folded. So STT-2's offline default is ahead of both open models in the number-fair view, ahead of SraVaani-1.0 in the folded view too, and not ahead of IndicConformer in the folded view. STT-2's offline greedy (52.38) and IndicConformer (52.82) are level (point). Our own measurements, with each alternative's published checkpoint run offline; this is not a streaming comparison.
Tanglish YouTube vlogs, unseen (tanglish_unseen_v1: clips drawn from 57 held-out YouTube videos; held-out videos, not held-out channels; references are the videos' caption drafts, checked by hand). Two views: vlog-96, the 96 clips the checker marked as heard (4 unclear clips dropped), and heard-35, the 35 clips whose listening time was logged in full. Number-fair WER %, row minus STT-1.2 (paired, 2,000 draws). STT-1.2's partials are STT-1.1's.
| Row | View (clips) | STT-2 | STT-1.2 | Difference [95%] |
|---|---|---|---|---|
Two-pass finals (--streaming, the default) |
vlog-96 (96) | 52.94 | 54.92 | −1.98 [−3.42, −0.57] |
| Offline default | vlog-96 (96) | 52.94 | 54.92 | −1.98 [−3.42, −0.57] |
| 480 ms partials (greedy stream) | vlog-96 (96) | 54.74 | 57.88 | −3.14 [−4.78, −1.54] |
| Two-pass finals | heard-35 (35) | 53.71 | 56.25 | −2.54 [−6.24, +0.53] |
| 480 ms partials | heard-35 (35) | 55.27 | 57.62 | −2.34 [−4.44, −0.21] |
On vlog-96 the finals interval is below zero, so STT-2 beats STT-1.2 on vlog; on heard-35 (n = 35) the finals interval crosses zero. No Sarvam or other alternative was run on these clips. Against STT-1.1's finals (--finals f), STT-2's finals are −3.43 [−5.49, −1.40] on vlog-96.
Honest read: against Sarvam's code-mix mode STT-2 is Equal on SPRING-INX test and Behind on CS-FLEURS. On SPRING-INX test the offline default is −0.59 [−1.05, −0.13] points from Sarvam on all clips, but on the 3,625 clips with no digit the sign flips (+0.31 [−0.17, +0.80]), so we call it Equal, not a win. SPRING-INX test is in-domain for us. On CS-FLEURS, which we have not trained on, we are Behind: +6.28 [+4.92, +7.69] points. The 480 ms greedy partials are Behind on both sets (SPRING +1.99 [+1.57, +2.40]; CS-FLEURS +10.38 [+8.91, +11.87]). Against STT-1.2, STT-2's finals are −0.13 [−0.29, +0.01] on SPRING-INX test and −0.44 [−1.02, +0.13] on CS-FLEURS: level or better (point). On CS-FLEURS the two open offline alternatives sit at 52.82 (IndicConformer-600M) and 56.07 (SraVaani-1.0), with the folded-view caveat above. Sarvam's rows are offline and hosted. We do not claim to equal or beat Sarvam on Tanglish.
⚑ See Leakage and overlap for the sets marked ⚑ and for SPRING-INX test, which is in-domain for us. Sarvam was told ta-IN; our model is told nothing.
🟨 How close are we to Sarvam?
The sets where both models were scored, STT-2 against Sarvam saaras:v4. "Matched" = our word accuracy ÷ Sarvam's, where word accuracy = 100 − WER. "Error ratio" = our errors ÷ Sarvam's errors.
| Set | STT-2 WER | Sarvam WER | Matched | Error ratio | STT-1.2 matched |
|---|---|---|---|---|---|
| IISc-MILE test (Tamil) ⚑ | 20.29 | 23.57 | 104% | 0.86× | 104% |
| LibriSpeech test-clean (English) ⚑ | 5.43 | 2.62 | 97% | 2.07× | 97% |
SPRING-INX test (Tanglish, codemix mode, number-fair) ⚑ |
32.24 | 32.83 | 101% | 0.98× | 101% |
| FLEURS-ta test (Tamil) | 28.15 | 23.78 | 94% | 1.18× | 94% |
| Monsoon en-IN (Indian English) | 20.83 | 4.82 | 83% | 4.32× | 83% |
| CS-FLEURS ta-en (Tanglish, read) | 50.99 | 44.71 | 89% | 1.14× | 88% |
Overall about 95% (plain mean of the six rows; STT-1.2: about 95%); about 97% on Tamil and Tanglish (MILE, FLEURS-ta, SPRING-INX, CS-FLEURS; STT-1.2: about 97%). Tamil and English rows are STT-1.2's (Tamil mode is unchanged); SPRING-INX test and CS-FLEURS use STT-2's code-mix offline default (RNN-T + J(0.3, 0), number-fair). The "matched" figure flatters low-WER sets: on LibriSpeech we match 97% yet make 2.1× Sarvam's errors, and on Monsoon en-IN 4.3×.
📡 Streaming vs open streaming models
The alternative is SraVaani-0.5-live, the only open streaming Tamil model we tested. We ran it ourselves, in a harness we wrote around the repository's own streaming simulation (the checkpoint's published files, loaded without running any repository code), at its 480 ms setting: 0.56 s chunks with 480 ms of look-ahead. Our side is the 480 ms greedy stream with the start cue (480 ms chunks, no look-ahead), which is also the partials of the default --streaming: --mode tamil --streaming --finals f on the five Tamil sets and LibriSpeech, the same stream as the "480 ms greedy stream" columns above, and --streaming --finals f --no-ctc-assist on SPRING-INX and CS-FLEURS. It is not a finals decode. Both are greedy, no language model. SraVaani-0.5-live takes no language hint and is given its better of two protocols (whole-clip feature normalisation, "file" mode). No per-set live numbers are published for it; the harness's ratio of live to offline WER on our five Tamil sets (x1.279) matches the published six-set ratio (x1.252) within 2%, so we count it as reproduced in shape, not number by number.
Which rows are STT-2's own. SPRING-INX and CS-FLEURS use STT-2's code-mix partials from its one test read, paired clip by clip with SraVaani-0.5-live's saved outputs (no model was run again for this table). The six Tamil-mode rows are STT-1.2's: STT-2's Tamil mode is the same model (845 of 845 shared tensors byte-equal; Tamil-mode streaming text identical on 336 of 336 clips).
WER %, lower is better. Difference = ours minus SraVaani-0.5-live (negative = we are better), 95% paired clip bootstrap, 2,000 draws. SPRING and CS-FLEURS are number-fair mixed-script scores; the other six are plain Tamil or English normalised scores.
| Set (clips) | Ours, 480 ms greedy stream | SraVaani-0.5-live, 480 ms | Difference [95%] |
|---|---|---|---|
| IndicVoices valid (600) ⚑ | 31.77 | 38.49 | −6.71 [−7.86, −5.57] |
| FLEURS-ta test (591) | 29.83 | 32.12 | −2.30 [−3.32, −1.37] |
| Kathbath valid (1000) ⚑ | 21.39 | 25.57 | −4.18 [−5.06, −3.32] |
| IISc-MILE test (1000) ⚑ | 22.38 | 30.49 | −8.11 [−9.06, −7.15] |
| DiarBench clean (2555) | 47.63 | 56.91 | −9.28 [−9.97, −8.56] |
| LibriSpeech test-clean (2620) ⚑ | 6.27 | 10.32 | −4.05 [−4.33, −3.77] |
| SPRING-INX test (4038) ⚑, STT-2 | 34.82 | 39.95 | −5.13 [−5.53, −4.71] |
| CS-FLEURS ta-en (336), STT-2 | 55.10 | 58.58 | −3.49 [−4.61, −2.39] |
8 of 8 sets are wins and every interval is below zero. The digit-free subsets agree in sign (SPRING −5.22 [−5.66, −4.77] on 3,625 clips; CS-FLEURS −4.45 [−5.86, −3.06] on 256 clips). STT-1.2's code-mix rows, for comparison: SPRING 34.92 (−5.03 [−5.45, −4.62]), CS-FLEURS 55.73 (−2.86 [−4.07, −1.72]).
- Causal bracket. If SraVaani-0.5-live is run strictly causally, as a live deployment would be (no whole-clip normalisation), it gets worse on every set and our lead grows: differences from −4.22 (FLEURS-ta) to −10.24 (DiarBench), 8 of 8 sets (point estimates, no intervals computed; STT-2's SPRING −6.53 and CS-FLEURS −5.30).
- Padded bracket. SraVaani-0.5-live is sensitive to tight trimming: with 0.5 s of silence added at both ends it improves, most on MILE (30.49 to 25.23) and Kathbath (25.57 to 23.70). We are still ahead on all 6 sets where this was run (IndicVoices −7.46, FLEURS-ta −2.52, Kathbath −2.31 [−3.12, −1.53], MILE −2.85 [−3.71, −1.97], DiarBench −6.73; CS-FLEURS, STT-2, −1.27 with no interval computed, which we would not call a win). The five Tamil-set intervals are below zero. This is not the repository's protocol, so it is not the main row, but for MILE and Kathbath the size of the win depends on it. Not run on LibriSpeech or SPRING-INX.
- CS-FLEURS is Equal in the folded view (hypotheses and reference mapped to Tamil script): −0.69 [−1.70, +0.34] for STT-2. The win above depends on scoring Latin words as written, and n is only 336. On SPRING-INX the folded view is still a win (−4.59 [−4.98, −4.18]).
- SPRING-INX may be in SraVaani-0.5-live's training data. SraVaani-1.0 was trained on SPRING-INX and the live model is from the same family, so we assume the same. SPRING-INX is also in-domain for us. If it saw SPRING-INX, the real gap is likely larger than shown.
- Scoring notes. Our decode uses the start cue and SraVaani-0.5-live takes no hint. It writes some Latin script on SPRING-INX and CS-FLEURS (about 12% and 11% of words), which is why the mixed-script scorer is used there. On very short DiarBench clips it often emits nothing (138 of 2,555 empty, against 63 for us). MILE has a known 2.1% overlap with our training data (leakage notes). All differences are our own measurements on our clips.
- What this does not show. Only this one open streaming Tamil model was tested; no open streaming Tamil checkpoint of IndicConformer was among the models we tested, so we say nothing about one. The comparison above is against the greedy stream, so it says nothing about the finals versus SraVaani-0.5-live. Sarvam's hosted rows are offline and are not part of this table.
🛠️ How it was made
16 kHz audio ─► log-mel (128)
─► EmbeddingGemma 2 audio tower (12 conformer layers, 25 frames/s, post-trained; shared by both modes)
├► Tamil stack: softmax mix of all 13 layer outputs ─► 6-layer reader (d = 768)
│ ├► Tamil heads: RNN-T ─► greedy / beam CTC ─► greedy / start cue
└► Code-mix fork: its own softmax layer mix ─► its own 6-layer reader (d = 768) (new in STT-2)
└► Code-mix heads: RNN-T ─► greedy / beam CTC ─► greedy / start cue
offline and streaming finals: beam n-best + CTC hypothesis ─► J(0.3, 0) rescoring
readers: causal conv positions; streaming: 480 ms blocks, no look-ahead; offline and two-pass finals: full attention
over the kept tower output
- STT-1 first. The code-mix main run trained the tower, the Tamil stack and both head sets on a single RTX 5090 Laptop GPU (24 GB) at about 175 to 220× real time, to step 84,000 (EMA weights). That is STT-1, and STT-2 keeps its tower, Tamil stack and Tamil heads byte for byte.
- Then the fork. A 12,000-step run started from STT-1 with the tower frozen (learning rate 0) and trained the post-tower stack and the code-mix heads on the code-mix recipe of an earlier experiment, adding Tamil-English YouTube clips labelled with the videos' own captions. Its post-tower stack (layer mix and reader) became
fork_cm.*and its code-mix heads replaced STT-1's; its Tamil heads were discarded. The released weights blend the two code-mix parts 0.5 / 0.5 with STT-1's (stt_config.jsonrecords the provenance;stepthere is the fork run's 12,000, andfork.base_stepis STT-1's 84,000). - Data: IndicVoices, Shrutilipi, Kathbath, IISc-MILE, SPRING-INX train, Common Voice, FLEURS train, IndicTTS, Vaani (Tamil and English), NISP, LibriSpeech 960 and Indian-English lectures (NPTEL); for the fork, also YouTube Tamil-English clips (YODAS3) with their captions. Reverb, noise and overlap augmentation. The code-mix heads are trained on text prepared with English words in Latin script; the Tamil heads use the usual all-Tamil-script text.
- Common Voice came from a re-host. The Common Voice Tamil training audio came from a Hugging Face re-host (
Prajwal-143/ASR-Tamil-cleaned, Common Voice 16.0 rows), not from Mozilla's official release. Like the earlier versions, this release was trained with a re-hosted copy of Common Voice Tamil; a future release will use the official Common Voice source only. - Discipline we kept: every eval and test set is on a never-train list, every training source is leakage-checked against them, and every number above comes from per-clip files with paired confidence intervals. The fork's recipe and every selection rule were fixed before its first evaluation row; candidates were screened and picked on development sets only, and the test sets were read once, after the pick. The fork's recipe was chosen from earlier development rows, and no test output was read for its design. STT-2 ships only because it passed ship lines fixed before the run: on SPRING-INX test and CS-FLEURS its finals beat STT-1.1's by at least 0.5 points with the interval below zero, no Tanglish test row is worse with confidence, Tamil mode is identical, the finals wait p90 is at most 4.0 s on an idle CPU, and its finals are not worse than STT-1.2's on the vlog set and by at most 0.3 points (point) on SPRING-INX test and CS-FLEURS.
- Built by an AI crew: planned, trained, reviewed and evaluated by an agent team (scouts, engineers, a code reviewer, an evaluator, a PM) with a human holding the gates.
🔍 Leakage and overlap
Leakage. No clip from any evaluation set in these tables was used for training. FLEURS dev, SPRING-INX valid, LibriSpeech dev, TIE, Vaani validation and test, and our held-out NISP and NPTEL dev sets were not used either. Every training source was checked against all of them by whole sentence and by 12-word run, and exact audio copies were removed wherever they could be found. Known overlaps, all inherited from how the corpora were split:
- IndicVoices valid and Kathbath valid share speakers with their train splits (97% and 100% of our scored clips).
- IISc-MILE test shares speakers with MILE train. 21 of our 1,000 scored MILE clips (2.1%) have their sentence (4+ words) in MILE train (1.05% on the full test split). 2 full-split clips, none of ours, are exact audio copies of training clips.
- SPRING-INX test likely comes from the same recordings as SPRING-INX train: after exact copies were removed, 38 of 4,038 clips still share a 12-word run with training text. This release also trains the code-mix heads on SPRING-INX train, so the same caveat applies to it.
- LibriSpeech test-clean: 47 clips share a 12-word run of book text with LibriSpeech-960 (speakers are disjoint).
FLEURS test, DiarBench, Svarah and Monsoon show no overlap. CS-FLEURS ta-en is on the never-train list and was never trained on; its sentences were not checked for overlap with FLEURS train text. The checkpoint was chosen on FLEURS dev, IndicVoices and Kathbath selection subsets, SPRING-INX valid and Vaani Tamil gloss clips that are not in the test tables. The audio-copy check covered the scored IndicVoices and MILE clips only. Sarvam, IndicConformer and SraVaani (including SraVaani-0.5-live) have unknown or different training data, so these flags do not say which way a gap would move on clean speakers.
STT-2's fork. Its YouTube training clips were drawn with every video and channel of the unseen vlog test set, and every development vlog video, filtered out, together with any training video whose voice matched a test or development speaker. The vlog test clips are held-out videos, not held-out channels (the source gives no channel ids). The development sets used to pick STT-2 (the YouTube vlog tune and adopt sets, a SPRING-INX valid subset and Vaani Tamil gloss clips) are not in the test tables.
⚠️ Known limits
- 🔤
tamilmode writes English inside Tamil in Tamil script ("meeting" comes out as மீட்டிங்). Usecodemixmode (the default) to get English in Latin script. - 🔢 Numbers come out as words, not digits (checked on the earlier build's Tamil mode; the number-fair scoring above spells out numerals the same way for every system).
- 🔇 It can drop words on hard conversational audio. On SPRING-INX test STT-2's offline default deletes 5.68 reference words per 100 and its offline greedy 6.19, against Sarvam's 5.36 and stage 7's 8.74 (number-fair, same clips).
- 🐢 The offline code-mix default is slower than greedy (CTC-assist runs a beam search on the CPU). Use
--no-ctc-assistfor greedy offline speed. - ⏱️ Streaming finals arrive after the utterance ends. In code-mix streaming (
--streaming, the default) the second, full-attention reader pass and the J(0.3, 0) re-rank start when the utterance ends. On 48 CS-FLEURS ta-en clips (CPU, 4 threads, an idle laptop CPU, no other job) STT-2's wait was median 1.34 s, p90 3.03 s and at most 5.41 s, on a 26.5 s clip (three runs gave p90 2.98 to 3.04 s), about 0.1 s more than the--finals ffinals at the median and 0.3 s at p90 (estimates). STT-2's wait on a busy CPU was not measured. STT-1.2's wait rose from p90 3.0 s on an idle CPU to 4.3 to 4.7 s with a training job running on the same laptop; expect STT-2's to rise in the same way (an estimate). STT-1.2's idle reading and STT-2's were taken on different days, so they are not a paired comparison. A longer utterance waits longer, and a slower or busier CPU waits longer; over 30 s the--finals ffinals take over, whose wait also has no cap. Partials are not delayed. The final can differ from the last partial, because it comes from a different reader pass and a re-ranked hypothesis. Use--streaming --finals f --no-ctc-assistfor greedy-only finals (no second pass, no wait). Onlycode/transcribe.py(safetensors and GGUF) has finals; MLX streaming stays greedy. - 📏 Two-pass finals stop at 30 s. Over 30 s an utterance gets the
--finals ffinals (re-rank of the streaming pass), with a note on stderr; this tool has no endpointer, so a long file is one utterance. Split long audio at pauses first. - 🕐 Streaming has a first-text floor. Measured on STT-1 at 480 ms on 60 LibriSpeech dev-clean clips and 60 FLEURS-ta dev clips (CPU, greedy partials; the finals are not included): the first token is not emitted before 0.96 s on LibriSpeech with the default Tamil-head recipe (120 of 120 clips across both sets), and with the code-mix recipe 2 of 60 LibriSpeech clips emitted at 0.48 s. Median delay of a token behind its audio is 0.28 s (p90 0.44 to 0.48 s). The Tamil-head numbers hold for STT-2 (same model); STT-2's code-mix partials were not re-measured. We did not test why the floor sits one chunk later than the first chunk close.
- 🗣️ Indian English still lags dedicated English models (Whisper small, Sarvam): Monsoon en-IN 20.83 vs Whisper small 8.45 and Sarvam 4.82; Svarah 25.22 vs Whisper small 10.37; LibriSpeech test-clean 5.43 vs Sarvam 2.62 (Tamil-mode rows, the same as STT-1.2).
- 🧪 Research preview. No speaker diarisation, no punctuation, no timestamps. GPU bf16 was smoke-tested once on STT-1 (67 of 75 texts equal to CPU float32), not on STT-2; MLX on Apple silicon and llama.cpp on these files are untested.
📜 Licence
Research only.
- Weights, tokenizer, code and samples: CC BY-NC-SA 4.0 (
LICENSE). Non-commercial use only, share alike. The reason: part of the training audio, NPTEL lectures, is itself CC BY-NC-SA. - Base model: EmbeddingGemma 2, Apache License 2.0 (
LICENSE-APACHE-2.0). Its card asks that deployments follow the Gemma Prohibited Use Policy (https://ai.google.dev/gemma/prohibited_use_policy); we pass that on as a notice. - Modified work: the audio tower was fine-tuned end to end, and a layer mix, a 6-layer reader and RNN-T and CTC heads (Tamil and code-mix) were added; STT-2 adds a second layer mix and 6-layer reader for code-mix mode (the code-mix reader fork). Our modifications are under CC BY-NC-SA 4.0.
- n-gram LMs: no LM is released or included with this release.
- Dataset credits, the IndicTTS copyright line and the full notices are in
NOTICE. The training datasets are not redistributed.
🛡️ Responsible use
- Built for research, accessibility and Tamil-language tooling. Do not use it to transcribe people without their consent, or for surveillance.
- Expect errors on noisy, overlapped or accented speech. Never treat a transcript as a verbatim legal or medical record without human review.
- Several training sets are gated datasets released under their own terms (e.g. IndicVoices, Kathbath, Vaani). Check that your use fits those terms and the Gemma Prohibited Use Policy.
📝 Changelog
VeriSafe-Tanglish-1-STT (internal build STT-2), 2026-11-16. The first release in this repository, which holds this model only: safetensors at full precision, GGUF f16 and q8_0, and MLX 8-bit. No earlier build's files are included.
- Weights: STT-1's tower, Tamil stack and Tamil heads, byte for byte, plus a code-mix reader fork (
fork_cm.*: 81 tensors, 43,330,317 parameters) and new code-mix heads: a 12,000-step tower-frozen run from STT-1, blended 0.5 / 0.5 with STT-1's code-mix weights. About 355M to about 398M parameters, one file per format (stt2_model.*). - Tamil mode unchanged from STT-1.2: 845 of 845 shared tensors byte-equal to STT-1's; Tamil-mode text identical on 336 of 336 CS-FLEURS ta-en clips, offline and
--streaming(and--streaming --finals f). - Code-mix, one test read, finals vs STT-1.2's finals: unseen YouTube vlog clips (96) −1.98 [−3.42, −0.57] (beats STT-1.2 on vlog); SPRING-INX test −0.13 [−0.29, +0.01] and CS-FLEURS ta-en −0.44 [−1.02, +0.13] (level or better, point). Against Sarvam: SPRING-INX Equal, CS-FLEURS Behind, as for STT-1.2.
- Code and flags: the command-line flags of STT-1 v1.2.
code/transcribe.pyandcode/indic_stt/model.pyload the fork whenstt_config.json(or a GGUF file's settings) says"codemix_fork": true, and refuse a file whose fork tensors do not match that flag. In the Python API,Release.mix,read,encode,greedyandctc_assistneedmode="codemix"ormode="tamil"on a fork file (MLX:MlxModel.encode,ctc_text,rnnt_text,assist_text). MLX gets the fork's 16th relaid conv kernel. The MLX weights are 8-bit (stt2_model.mlx8.safetensors, 0.46 GB:mx.quantize, affine, group size 64, on the 198 2-D tensors), dequantized to float32 on load;mlx/run_mlx.pyreads them with--weights stt2_model.mlx8.safetensors. - Streaming finals wait measured for STT-2 on an idle CPU (median 1.34 s, p90 3.03 s).
- Card wording: "alternatives" for the other models we measured; the headline is three claims, each among the models we tested.
Earlier builds (not in this repository). STT-1 v1.2 (2026-11-14) added the two-pass streaming finals and --finals two-pass|f; STT-1 v1.1 (2026-11-13) added the CTC-assist streaming finals (now --finals f) and the comparison with SraVaani-0.5-live; STT-1 v1 was the code-mix main run (EMA step 84,000), after the stage 7 build. Their numbers appear on this card as comparison rows only.
🔭 What's next
STT-2 adds the code-mix reader fork. Next: an endpointer, so that long audio is cut into utterances (two-pass finals stop at 30 s), a decoding fix for dropped words, an earlier first token in streaming (the 0.96 s floor above), further gains on conversational Tanglish (including the CS-FLEURS gap to Sarvam, which stays on this card as a reference bar), and a release trained on the official Common Voice source only.
💬 Need a custom model?
This model is for research and non-commercial use. For commercial use, VeriSafe builds licensed custom models; contact us.
Drop us a line at info@verisafe.live
🙏 Credits
- Base model: Google's EmbeddingGemma 2 (audio tower, Google DeepMind), under the Apache License 2.0.
- Data: AI4Bharat (IndicVoices, Kathbath, Shrutilipi), IISc MILE lab, SPRING Lab IIT Madras (SPRING-INX), IISc and ARTPARK with Google (Vaani), Google (FLEURS), Mozilla (Common Voice), IIT Madras (IndicTTS), LibriSpeech, NISP (IISc LEAP), NPTEL, YODAS (YouTube clips with captions, for the STT-2 fork), and the Indian-English corpora named in the training report. See
NOTICE. - Alternatives we measured against: AI4Bharat IndicConformer, SraVaani (1.0 and 0.5-live), OpenAI Whisper, Sarvam
saaras:v4.
Built and published by VeriSafe. Post-trained from EmbeddingGemma 2 (Apache License 2.0); weights under CC BY-NC-SA 4.0, see the Licence section.
- Downloads last month
- -
Model tree for junafinity/VeriSafe-Tanglish-1-STT
Base model
google/embeddinggemma-2