"""Transcribe audio files with VeriSafe-Tanglish-STT-2 (STT-1.1 added streaming finals, STT-1.2 made them two-pass, below; STT-2 adds the code-mix reader fork: STT-1's audio tower, Tamil stack and Tamil heads, cm-main ema.pt step 84,000, byte for byte, plus code-mix heads and a code-mix post-tower stack of their own, fork_cm; stt_config.json names the checkpoint). Runs in PyTorch from the safetensors or a GGUF file. python transcribe.py CLIP.wav [CLIP2.mp3 ...] [--weights FILE] [--mode codemix|tamil] [--streaming [--finals two-pass|f]] [--head rnnt|ctc] [--ctc-assist | --no-ctc-assist] [--device cpu|cuda] [--threads N] [--json OUT.jsonl] --weights: ../safetensors/stt2_model.safetensors (default), or a GGUF from ../gguf/ (f32, f16 or q8_0). A GGUF holds the tokenizer and both config files, so ../gguf/ and this code folder are enough for it. The GGUF runtime is this script: it reads the file with the `gguf` package, dequantizes to float32 and runs PyTorch. llama.cpp and whisper.cpp cannot run this model. Defaults (the code-mix recipe): --mode codemix; offline = RNN-T with --ctc-assist (J(0.3, 0) rescoring); --streaming = RNN-T greedy partials with the start penalty and cue, and two-pass finals (--finals below): each utterance's final text is the offline decode of the same flags, J(0.3, 0) in code-mix ("partials streaming (480 ms, no look-ahead); two-pass finals at end of utterance"). --no-ctc-assist gives greedy, offline and in the two-pass final; --streaming --finals f --no-ctc-assist is STT-1's streaming row, which is also the partials' text. --mode tamil gives the Tamil heads, greedy by default (the Tamil rows reported are greedy); --mode tamil --ctc-assist opts in to J. One shared audio tower, two sets of heads (each an RNN-T head and a CTC head): --mode codemix (default) the code-mix heads: Tamil words in Tamil script, English words in Latin script. --mode tamil the Tamil heads: Tamil script throughout, as stages 6 and 7 wrote. STT-2: each mode reads its own post-tower stack (layer mix, in_proj, position conv, 6 reader layers, out_norm) on the one tower: --mode tamil STT-1's (model.mix/read, unchanged), --mode codemix the fork (model.mix_cm/read_cm), for the offline text, the partials and the two-pass final alike. A weights file with fork_cm.* tensors needs "codemix_fork": true in its stt_config, and then all of them (strict load); fork_cm.* without the flag, or the flag without them, is refused. A file without the fork (STT-1, -1.1, -1.2) runs as before: both modes on the one stack. RNN-T greedy (--head rnnt, default) takes START_PENALTY (1) off blank's logit while a clip has emitted nothing, from the first frame where the mode's CTC head leaves blank (the start cue). This is the project's evaluated decode, exactly: codemix = scripts/eval_dump.py --mode codemix --start-penalty 1 --start-cue --cue-head cm tamil = scripts/eval_dump.py --start-penalty 1 --start-cue --head ctc: the mode's CTC head, greedy (eval_dump.py's CTC rows of the same run). --streaming: the 480 ms block-causal mask over the whole clip, as every streaming number was measured; there is no live-microphone loop here. Default: offline (the reader sees the whole clip). --finals (with --streaming; default FINALS_DEFAULT below): how each utterance's final text is made. The partials, the live text while the speaker talks, are the same in both: RNN-T greedy with the start cue on the block-causal reader output (a live caller shows greedy(*read(x, lens, True, mode=m), mode=m) on the mix(feats, mode=m) output so far; this script prints the finals only). On an STT-2 file Release.mix/read/encode and greedy/ctc_assist need mode= (None raises): the stack and the heads depend on it. two-pass (STT-1.2, default) At the end of the utterance the reader runs once more, with full attention, over the kept mix() output (the tower and layer mix, which do not depend on the mask, so they are not recomputed), and the final is the offline decode of the same flags on that pass: J(0.3, 0) in code-mix, greedy with --no-ctc-assist, --mode tamil or --head ctc. It is the offline text of the same flags on the same device. Utterances over TWO_PASS_MAX_SECS (30 s: the full-attention pass costs O(T^2), and training clips were at most 30 s) get f instead, with a note on stderr. f (STT-1.1) The same decode on the block-causal reader output itself: J(0.3, 0) re-ranks the streaming hypotheses (STT-1.1's default --streaming); with --no-ctc-assist the final is the greedy stream (STT-1's --streaming row, no wait). --ctc-assist (RNN-T only; on by default in --mode codemix): the J(0.3, 0) CTC-assisted rescoring of scripts/decode_ctc_assist.py (--joint 0.3:0): the mode's RNN-T beam-4 n-best (no LM; the start penalty does not apply to it) plus the mode's CTC greedy hypothesis, ranked by 0.7 log P_RNNT + 0.3 log P_CTC (exact forced scores, ctc_assist.py). It runs on the CPU in float32 on the reader output rounded to bf16, as the evaluated decode read it from eval_dump.py --save-enc dumps. Slower than greedy (a beam plus forced scores). Its default is CTC_ASSIST_DEFAULT below (True) for the modes in CTC_ASSIST_DEFAULT_MODES (codemix); --ctc-assist / --no-ctc-assist override it per run. With --head ctc it does not apply (it rescores the RNN-T head): that row is greedy, as is --mode tamil without --ctc-assist. --streaming --ctc-assist (the code-mix default since STT-1.1; STREAMING_FINALS_DEFAULT below): J(0.3, 0) runs once per utterance, at its end, and gives the final text; the partials stay RNN-T greedy with the start cue. With --finals two-pass it reads the full-attention reader pass, so the final equals the offline code-mix default. With --finals f it reads the streaming reader output, the project's evaluated decode scripts/decode_ctc_assist.py --head cm --mode streaming --start-penalty 1 --start-cue --joint 0.3:0 (its J0.3_b0 rows; its greedy rows are --streaming --finals f --no-ctc-assist). In --mode tamil the two-pass final is the Tamil offline greedy text; --mode tamil --streaming --finals f --ctc-assist runs but has no reported numbers. Output text is normalize()d (lower case, no punctuation), as the model was scored. The evals ran on GPU with bf16 autocast; --device cuda does the same, CPU runs float32, so a few words can differ between the two. """ from __future__ import annotations import argparse import copy import json import shutil import sys import tempfile from pathlib import Path import numpy as np import soundfile as sf import torch import indic_stt.model as stt_model from indic_stt.model import FORK, EG2SpeechModel, codemix_fork_in, codemix_heads_in from indic_stt.text import load_tokenizer, normalize RELEASE = Path(__file__).resolve().parents[1] DEFAULT_WEIGHTS = RELEASE / "safetensors" / "stt2_model.safetensors" TOKENIZER = RELEASE / "tokenizer" / "spm.model" SAMPLE_RATE = 16000 EVALUATED_SECS = 60.0 # eval_dump.py --max-secs for the reported numbers MODES = {"codemix": ("transducer_cm", "cm"), "tamil": ("transducer", "ta")} # mode -> (RNN-T head, CTC head of ctc_log_probs) START_PENALTY = 1.0 # eval_dump.py --start-penalty 1 --start-cue, both modes LAMBDA, BETA, BEAM = 0.3, 0.0, 4 # --ctc-assist: decode_ctc_assist.py --joint 0.3:0, beam 4 CTC_ASSIST_DEFAULT = True # THE switch for the offline RNN-T default: True = --ctc-assist unless --no-ctc-assist (also mlx/run_mlx.py) CTC_ASSIST_DEFAULT_MODES = ("codemix",) # modes the default applies to; --mode tamil is greedy unless --ctc-assist (PM, 2026-11-12) STREAMING_FINALS_DEFAULT = True # STT-1.1 (PM, 2026-11-13): the default above also applies with --streaming, to the final text only # (partials greedy); False = no J by default with --streaming (with FINALS_DEFAULT "f" too: STT-1's greedy --streaming). # mlx/run_mlx.py has no streaming finals: its --streaming stays greedy. FINALS = ("two-pass", "f") # --finals choices (module docstring) FINALS_DEFAULT = "two-pass" # STT-1.2 (PM, 2026-11-13): "f" = STT-1.1's finals; "f" with STREAMING_FINALS_DEFAULT False = STT-1 TWO_PASS_MAX_SECS = 30.0 # --finals two-pass up to this utterance length; longer ones get f, with a note (O(T^2) reader pass; # training clips were at most 30 s) def finals_for(finals: str | None, streaming: bool, secs: float) -> str | None: """The finals a clip of `secs` seconds gets: None offline; f for two-pass over TWO_PASS_MAX_SECS; else `finals`.""" if not streaming: return None return "f" if finals == "two-pass" and secs > TWO_PASS_MAX_SECS else finals def load_audio(path: str | Path) -> np.ndarray: """Mono float32 at 16 kHz. Other rates are resampled with librosa, as the project's data loader does.""" wav, sr = sf.read(str(path), dtype="float32", always_2d=True) wav = wav.mean(axis=1) if sr != SAMPLE_RATE: import librosa wav = librosa.resample(wav, orig_sr=sr, target_sr=SAMPLE_RATE) return wav def features(fe, wav: np.ndarray) -> np.ndarray: """Log-mel frames (T, 128) with no padding: data.py _prepare, eval branch (no 30 s truncation).""" out = fe([wav], sampling_rate=SAMPLE_RATE, return_tensors="np", truncation=False) return out["input_features"][0][: int(out["input_features_mask"][0].sum())].astype(np.float32) class Release: """The model, tokenizer and feature extractor from one weights file. model, tok, fe, cfg, source.""" def __init__(self, weights: str | Path = DEFAULT_WEIGHTS, device: str = "cpu", tokenizer: str | Path | None = None): from transformers import AutoFeatureExtractor weights = Path(weights) tmp = None try: if weights.suffix == ".gguf": from indic_stt.gguf_io import read_gguf arrays, meta = read_gguf(weights) state = {k: torch.from_numpy(v) for k, v in arrays.items()} del arrays tmp = Path(tempfile.mkdtemp(prefix="indic_stt_gguf_")) (tmp / "config.json").write_text(meta["eg2_config"], encoding="utf-8") (tmp / "preprocessor_config.json").write_text(meta["preprocessor_config"], encoding="utf-8") (tmp / "spm.model").write_bytes(meta["spm_model"]) config_dir, cfg = tmp, meta["stt_config"] tok_path = Path(tokenizer) if tokenizer else tmp / "spm.model" self.source = f"{weights.name} ({meta['source']}; tensor types {meta['tensor_types']})" else: from safetensors.torch import load_file state = load_file(str(weights)) config_dir = weights.parent / "eg2" cfg = json.loads((weights.parent / "stt_config.json").read_text(encoding="utf-8")) tok_path = Path(tokenizer) if tokenizer else TOKENIZER self.source = weights.name self.tok = load_tokenizer(tok_path) if self.tok.vocab_size != cfg["vocab_size"]: raise ValueError(f"tokenizer {tok_path} has {self.tok.vocab_size} ids, the model {cfg['vocab_size']}") heads = codemix_heads_in(state) if heads != cfg.get("codemix_heads", heads): raise ValueError(f"{weights.name}: code-mix heads {heads}, but its config says {cfg['codemix_heads']}") fork = codemix_fork_in(state) # STT-2: the code-mix reader fork, fork_cm.* if fork != (cfg.get("codemix_fork", False) is True): raise ValueError(f"{weights.name}: {'holds' if fork else 'has no'} {FORK}.* tensors, but its config says " f"codemix_fork = {cfg.get('codemix_fork', 'absent')}: refused (a fork must be declared and complete, " "or the code-mix heads would read the wrong stack)") if fork and not all(heads.values()): raise ValueError(f"{weights.name}: the code-mix fork without both code-mix heads {heads}") self.fe = AutoFeatureExtractor.from_pretrained(str(config_dir)) stt_model.BASE_MODEL = str(config_dir) # the tower is built from this folder's config.json model = EG2SpeechModel(cfg["vocab_size"], train_tower_layers=cfg["train_tower_layers"], reader_layers=cfg["reader_layers"], stream_chunk_frames=cfg["stream_chunk_frames"], transducer=cfg["transducer"]) if model.offline_right: # one mix() feeds both reader passes: a tower with right context would leak future audio raise ValueError(f"offline_right = {model.offline_right}: this runner shares mix() between the streaming and " "two-pass reader passes, so the tower must have no right context (0)") if heads["transducer_cm"]: model.add_codemix_head(ctc=heads["ctc_head_cm"]) # before the load, as eval_dump.py does if fork: model.add_codemix_fork() # before the load, as eval_dump.py does; strict below: every fork_cm.* tensor or none want = {f"{FORK}.{k}" for k in model.fork_cm.state_dict()} have = {k for k in state if k.startswith(FORK + ".")} if have != want: raise ValueError(f"{weights.name}: codemix_fork needs all {len(want)} {FORK}.* tensors: missing " f"{len(want - have)} {sorted(want - have)[:3]}, unexpected {len(have - want)} {sorted(have - want)[:3]}") model.load_state_dict(state, strict=True) # strict: no tensor may silently keep its random start del state self.model, self.cfg, self.device = model.to(device).eval(), cfg, device self.modes = [m for m, (rnnt, _) in MODES.items() if getattr(model, rnnt) is not None and (m == "tamil" or model.ctc_head_cm is not None)] self._cpu_heads = {} finally: if tmp is not None: shutil.rmtree(tmp, ignore_errors=True) def forked(self, mode: str | None) -> bool: """Whether `mode` reads the code-mix fork (STT-2): --mode codemix on a file with fork_cm.*; else the one shared stack. mode None (STT-1.2's mix/read/encode took no mode): the one stack on a file without the fork, as before; refused on a fork file, where the stack depends on the mode and the reader output does not record which stack made it (R3 B1).""" if mode is None: if self.model.fork_cm is not None: raise ValueError(f"{self.source} has the code-mix reader fork ({FORK}): pass mode='codemix' or 'tamil' to " "mix/read/encode, and the same mode to greedy/ctc_assist") return False return mode == "codemix" and self.model.fork_cm is not None @torch.no_grad() def mix(self, feats: np.ndarray, bf16: bool = False, mode: str | None = None): """The reader's input (1, T, D) and lengths (1,) for `mode`'s stack: model.mix (tower, layer mix, position conv), or model.mix_cm (the same tower, the fork's layer mix and position conv) in code-mix mode on a fork file. It does not depend on the mask (offline_right = 0, checked in __init__), so a streaming caller computes it once and keeps it for both reader passes. Pass the same mode to read() and to the decode (greedy, ctc_assist). mode is required on a fork file (None raises, see forked()); None on a file without the fork reads the one stack, as STT-1.2's mix(feats) did.""" f = self.model.mix_cm if self.forked(mode) else self.model.mix x = torch.from_numpy(feats)[None].to(self.device) mask = torch.ones(x.shape[:2], dtype=torch.bool, device=self.device) with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16): return f(x, mask) @torch.no_grad() def read(self, x, lens, streaming: bool, bf16: bool = False, mode: str | None = None): """Reader output (1, T, D) and lengths (1,) from mix()'s output of the same mode: 480 ms block-causal with streaming, else full attention. model.read, or model.read_cm (the fork's reader) in code-mix mode on a fork file. mode: as mix().""" f = self.model.read_cm if self.forked(mode) else self.model.read with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16): return f(x, lens, streaming=streaming) def encode(self, feats: np.ndarray, streaming: bool, bf16: bool = False, mode: str | None = None): """Reader output (1, T, D) and lengths (1,): read(mix()), the same operations as STT-1.1's one-step encode. mode: as mix().""" return self.read(*self.mix(feats, bf16, mode=mode), streaming, bf16, mode=mode) def _check(self, mode: str) -> tuple[str, str]: if mode not in self.modes: raise ValueError(f"--mode {mode}: this weights file has the heads for {self.modes} only") return MODES[mode] def _decode_mode(self, mode: str | None) -> str: """greedy/ctc_assist's mode: required on a fork file, as for mix/read/encode (None raises, see forked()); None on a file without the fork is "codemix", STT-1.2's default for greedy/ctc_assist.""" self.forked(mode) return "codemix" if mode is None else mode @torch.no_grad() def greedy(self, enc, lens, mode: str | None = None, head: str = "rnnt", bf16: bool = False) -> str: """eval_dump.py's rows for one clip: CTC greedy (head="ctc"), or RNN-T greedy with the start penalty and the start cue. mode: the mode given to mix/read (required on a fork file, see _decode_mode()).""" rnnt_name, ctc_name = self._check(self._decode_mode(mode)) with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16): logp = self.model.ctc_log_probs(enc, ctc_name) if head == "ctc": return normalize(self.tok.greedy(logp, lens)[0]) cue = (logp.argmax(-1) != 0).cumsum(1) > 0 hyp = getattr(self.model, rnnt_name).greedy(enc, lens, allowed=None, swap=True, start_penalty=START_PENALTY, cue=cue, style=None, all_frame_penalty=0.0)[0] return normalize(self.tok.decode(hyp)) @torch.no_grad() def ctc_assist(self, enc, lens, mode: str | None = None) -> str: """--ctc-assist for one clip: J(LAMBDA, BETA) over the RNN-T beam n-best and the CTC greedy hypothesis (ctc_assist.py), CPU float32 on the bf16-rounded reader output. Offline or streaming reader output alike: with --streaming, the final text. mode: as greedy().""" from ctc_assist import joint_rescore mode = self._decode_mode(mode) rnnt_name, ctc_name = self._check(mode) if mode not in self._cpu_heads: # the heads as decode_ctc_assist.py holds them: float32 copies on the CPU ctc = self.model.ctc_head_cm if ctc_name == "cm" else self.model.ctc_head rnnt = getattr(self.model, rnnt_name) self._cpu_heads[mode] = ((rnnt, ctc) if self.device == "cpu" else (copy.deepcopy(rnnt).float().cpu().eval(), copy.deepcopy(ctc).float().cpu().eval())) rnnt, ctc = self._cpu_heads[mode] e = enc[0, : int(lens[0])].bfloat16().float().cpu() # eval_dump.py --save-enc stores bf16; decode_ctc_assist reads .float() best, _ = joint_rescore(rnnt, ctc, e, LAMBDA, BETA, BEAM) return normalize(self.tok.decode(best)) def transcribe(self, feats: np.ndarray, mode: str = "codemix", head: str = "rnnt", streaming: bool = False, ctc_assist: bool = False, bf16: bool = False, finals: str = FINALS_DEFAULT) -> str: """One clip's text: offline, or with streaming the utterance's final. finals "two-pass": the offline decode on a full-attention read() of the kept mix() output, so the offline text; "f": the decode on the block-causal read() (with ctc_assist=False the greedy stream, which is also the partials' text). finals_for() gives f over TWO_PASS_MAX_SECS, as main() applies it; this method runs what it is given.""" if ctc_assist and head != "rnnt": raise ValueError("--ctc-assist rescores the RNN-T head: drop --head ctc") if finals not in FINALS: raise ValueError(f"finals must be one of {FINALS}, not {finals!r}") x, lens = self.mix(feats, bf16, mode=mode) enc, lens = self.read(x, lens, streaming and finals == "f", bf16, mode=mode) # two-pass: the end-of-utterance pass, full attention return self.ctc_assist(enc, lens, mode) if ctc_assist else self.greedy(enc, lens, mode, head, bf16) def main() -> None: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("audio", nargs="+") ap.add_argument("--weights", default=str(DEFAULT_WEIGHTS), help="a .safetensors (default) or .gguf file of this release") ap.add_argument("--mode", choices=list(MODES), default="codemix", help="codemix (default): code-mix heads; tamil: Tamil heads") ap.add_argument("--streaming", action="store_true", help="the 480 ms block-causal mask (default: offline)") ap.add_argument("--finals", choices=FINALS, help=f"with --streaming, how each utterance's final text is made (default " f"{FINALS_DEFAULT}): two-pass = the offline decode on a full-attention reader pass at the end of the utterance " f"(over {TWO_PASS_MAX_SECS:.0f} s: f, with a note); f = STT-1.1's, the decode on the streaming reader output") ap.add_argument("--head", choices=["rnnt", "ctc"], default="rnnt") ap.add_argument("--ctc-assist", action=argparse.BooleanOptionalAction, default=None, help=f"RNN-T: J(0.3, 0) CTC-assisted rescoring; with --streaming, of the final text only (default " f"{'on' if CTC_ASSIST_DEFAULT else 'off'} for --mode {', '.join(CTC_ASSIST_DEFAULT_MODES)}" f"{'' if STREAMING_FINALS_DEFAULT else ' offline'}, off otherwise)") ap.add_argument("--device", default="cpu") ap.add_argument("--no-bf16", action="store_true", help="on GPU, run float32 instead of the evals' bf16 autocast") ap.add_argument("--threads", type=int, help="CPU threads for PyTorch (default: PyTorch's choice)") ap.add_argument("--tokenizer", help="spm.model; default ../tokenizer/spm.model, or the copy inside a GGUF") ap.add_argument("--json", help="also write one JSON line per clip to this file") args = ap.parse_args() sys.stdout.reconfigure(encoding="utf-8") # Tamil text: Windows writes a redirected stdout in the ANSI code page otherwise if args.ctc_assist and args.head == "ctc": ap.error("--ctc-assist rescores the RNN-T head: drop --head ctc") if args.finals and not args.streaming: ap.error("--finals sets how --streaming makes its final text: add --streaming, or drop --finals") assist = (args.ctc_assist if args.ctc_assist is not None else CTC_ASSIST_DEFAULT and args.mode in CTC_ASSIST_DEFAULT_MODES and args.head == "rnnt" and (STREAMING_FINALS_DEFAULT or not args.streaming)) finals = (args.finals or FINALS_DEFAULT) if args.streaming else None if args.threads: torch.set_num_threads(args.threads) bf16 = args.device.startswith("cuda") and not args.no_bf16 rel = Release(args.weights, args.device, args.tokenizer) rel._check(args.mode) decode = "ctc-assist J(0.3, 0)" if assist else f"{'CTC' if args.head == 'ctc' else 'RNN-T'} greedy" how = ("" if finals == "f" and not assist else ", finals re-ranked with ctc-assist J(0.3, 0), partials greedy" if finals == "f" else f", two-pass finals (full-attention reader pass at the end of the utterance, then {decode}), partials greedy" if finals else ", ctc-assist J(0.3, 0)" if assist else "") row_name = (f"{args.mode}{' (code-mix reader fork)' if rel.forked(args.mode) else ''}, {'CTC' if args.head == 'ctc' else 'RNN-T'}, " f"{'streaming' if args.streaming else 'offline'}{how}") print(f"# weights: {rel.source}; {row_name}", file=sys.stderr, flush=True) out = open(args.json, "w", encoding="utf-8") if args.json else None for clip in args.audio: wav = load_audio(clip) secs = len(wav) / SAMPLE_RATE if secs > EVALUATED_SECS: print(f"# {clip}: {secs:.1f} s, longer than the {EVALUATED_SECS:.0f} s the reported numbers cover", file=sys.stderr) use = finals_for(finals, args.streaming, secs) if use != finals: print(f"# {clip}: {secs:.1f} s, over the {TWO_PASS_MAX_SECS:.0f} s of two-pass finals: STT-1.1 finals (--finals f) " f"for this file", file=sys.stderr) text = rel.transcribe(features(rel.fe, wav), args.mode, args.head, args.streaming, assist, bf16, use or FINALS_DEFAULT) print(f"{clip}\t{text}", flush=True) if out: out.write(json.dumps({"clip": str(clip), "secs": round(secs, 2), "mode": args.mode, "head": args.head, "streaming": args.streaming, "ctc_assist": assist, "finals": use, "hyp": text}, ensure_ascii=False) + "\n") if out: out.close() if __name__ == "__main__": main()