Automatic Speech Recognition
PyTorch
Safetensors
GGUF
MLX
Tamil
English
speech
asr
streaming
rnnt
ctc
tamil
tanglish
code-switching
indian-english
verisafe
Instructions to use junafinity/VeriSafe-Tanglish-1-STT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use junafinity/VeriSafe-Tanglish-1-STT with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download junafinity/VeriSafe-Tanglish-1-STT --local-dir VeriSafe-Tanglish-1-STT
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Download code/transcribe.py from junafinity/VeriSafe-Tanglish-1-STT: direct link, hf CLI and curl.
- Browser
- Download file 25.6 kB
-
https://huggingface.co/junafinity/VeriSafe-Tanglish-1-STT/resolve/main/code/transcribe.py
- Command line
-
hf download hf://junafinity/VeriSafe-Tanglish-1-STT/code/transcribe.py
-
curl -L -o transcribe.py https://huggingface.co/junafinity/VeriSafe-Tanglish-1-STT/resolve/main/code/transcribe.py
25.6 kB
| """Transcribe audio files with VeriSafe-Tanglish-STT-2 (STT-1.1 added streaming finals, STT-1.2 made them two-pass, below; STT-2 | |
| adds the code-mix reader fork: STT-1's audio tower, Tamil stack and Tamil heads, cm-main ema.pt step 84,000, byte for byte, plus | |
| code-mix heads and a code-mix post-tower stack of their own, fork_cm; stt_config.json names the checkpoint). Runs in PyTorch from | |
| the safetensors or a GGUF file. | |
| python transcribe.py CLIP.wav [CLIP2.mp3 ...] [--weights FILE] [--mode codemix|tamil] [--streaming [--finals two-pass|f]] | |
| [--head rnnt|ctc] [--ctc-assist | --no-ctc-assist] [--device cpu|cuda] [--threads N] [--json OUT.jsonl] | |
| --weights: ../safetensors/stt2_model.safetensors (default), or a GGUF from ../gguf/ (f32, f16 or q8_0). A GGUF holds the | |
| tokenizer and both config files, so ../gguf/ and this code folder are enough for it. The GGUF runtime is this script: it reads | |
| the file with the `gguf` package, dequantizes to float32 and runs PyTorch. llama.cpp and whisper.cpp cannot run this model. | |
| Defaults (the code-mix recipe): --mode codemix; offline = RNN-T with --ctc-assist (J(0.3, 0) rescoring); --streaming = RNN-T | |
| greedy partials with the start penalty and cue, and two-pass finals (--finals below): each utterance's final text is the | |
| offline decode of the same flags, J(0.3, 0) in code-mix ("partials streaming (480 ms, no look-ahead); two-pass finals at end of | |
| utterance"). --no-ctc-assist gives greedy, offline and in the two-pass final; --streaming --finals f --no-ctc-assist is STT-1's | |
| streaming row, which is also the partials' text. --mode tamil gives the Tamil heads, greedy by default (the Tamil rows reported | |
| are greedy); --mode tamil --ctc-assist opts in to J. | |
| One shared audio tower, two sets of heads (each an RNN-T head and a CTC head): | |
| --mode codemix (default) the code-mix heads: Tamil words in Tamil script, English words in Latin script. | |
| --mode tamil the Tamil heads: Tamil script throughout, as stages 6 and 7 wrote. | |
| STT-2: each mode reads its own post-tower stack (layer mix, in_proj, position conv, 6 reader layers, out_norm) on the one tower: | |
| --mode tamil STT-1's (model.mix/read, unchanged), --mode codemix the fork (model.mix_cm/read_cm), for the offline text, the | |
| partials and the two-pass final alike. A weights file with fork_cm.* tensors needs "codemix_fork": true in its stt_config, and | |
| then all of them (strict load); fork_cm.* without the flag, or the flag without them, is refused. A file without the fork (STT-1, | |
| -1.1, -1.2) runs as before: both modes on the one stack. | |
| RNN-T greedy (--head rnnt, default) takes START_PENALTY (1) off blank's logit while a clip has emitted nothing, from the first | |
| frame where the mode's CTC head leaves blank (the start cue). This is the project's evaluated decode, exactly: | |
| codemix = scripts/eval_dump.py --mode codemix --start-penalty 1 --start-cue --cue-head cm | |
| tamil = scripts/eval_dump.py --start-penalty 1 --start-cue | |
| --head ctc: the mode's CTC head, greedy (eval_dump.py's CTC rows of the same run). | |
| --streaming: the 480 ms block-causal mask over the whole clip, as every streaming number was measured; there is no | |
| live-microphone loop here. Default: offline (the reader sees the whole clip). | |
| --finals (with --streaming; default FINALS_DEFAULT below): how each utterance's final text is made. The partials, the live text | |
| while the speaker talks, are the same in both: RNN-T greedy with the start cue on the block-causal reader output (a live caller | |
| shows greedy(*read(x, lens, True, mode=m), mode=m) on the mix(feats, mode=m) output so far; this script prints the finals only). | |
| On an STT-2 file Release.mix/read/encode and greedy/ctc_assist need mode= (None raises): the stack and the heads depend on it. | |
| two-pass (STT-1.2, default) At the end of the utterance the reader runs once more, with full attention, over the kept mix() | |
| output (the tower and layer mix, which do not depend on the mask, so they are not recomputed), and the final is the | |
| offline decode of the same flags on that pass: J(0.3, 0) in code-mix, greedy with --no-ctc-assist, --mode tamil or | |
| --head ctc. It is the offline text of the same flags on the same device. Utterances over TWO_PASS_MAX_SECS (30 s: the | |
| full-attention pass costs O(T^2), and training clips were at most 30 s) get f instead, with a note on stderr. | |
| f (STT-1.1) The same decode on the block-causal reader output itself: J(0.3, 0) re-ranks the streaming hypotheses (STT-1.1's | |
| default --streaming); with --no-ctc-assist the final is the greedy stream (STT-1's --streaming row, no wait). | |
| --ctc-assist (RNN-T only; on by default in --mode codemix): the J(0.3, 0) CTC-assisted rescoring of scripts/decode_ctc_assist.py (--joint 0.3:0): the | |
| mode's RNN-T beam-4 n-best (no LM; the start penalty does not apply to it) plus the mode's CTC greedy hypothesis, ranked by | |
| 0.7 log P_RNNT + 0.3 log P_CTC (exact forced scores, ctc_assist.py). It runs on the CPU in float32 on the reader output rounded | |
| to bf16, as the evaluated decode read it from eval_dump.py --save-enc dumps. Slower than greedy (a beam plus forced scores). | |
| Its default is CTC_ASSIST_DEFAULT below (True) for the modes in CTC_ASSIST_DEFAULT_MODES (codemix); --ctc-assist / | |
| --no-ctc-assist override it per run. With --head ctc it does not apply (it rescores the RNN-T head): that row is greedy, as is | |
| --mode tamil without --ctc-assist. | |
| --streaming --ctc-assist (the code-mix default since STT-1.1; STREAMING_FINALS_DEFAULT below): J(0.3, 0) runs once per | |
| utterance, at its end, and gives the final text; the partials stay RNN-T greedy with the start cue. With --finals two-pass it | |
| reads the full-attention reader pass, so the final equals the offline code-mix default. With --finals f it reads the streaming | |
| reader output, the project's evaluated decode | |
| scripts/decode_ctc_assist.py --head cm --mode streaming --start-penalty 1 --start-cue --joint 0.3:0 (its J0.3_b0 rows; | |
| its greedy rows are --streaming --finals f --no-ctc-assist). | |
| In --mode tamil the two-pass final is the Tamil offline greedy text; --mode tamil --streaming --finals f --ctc-assist runs but has | |
| no reported numbers. | |
| Output text is normalize()d (lower case, no punctuation), as the model was scored. The evals ran on GPU with bf16 autocast; | |
| --device cuda does the same, CPU runs float32, so a few words can differ between the two. | |
| """ | |
| from __future__ import annotations | |
| import argparse | |
| import copy | |
| import json | |
| import shutil | |
| import sys | |
| import tempfile | |
| from pathlib import Path | |
| import numpy as np | |
| import soundfile as sf | |
| import torch | |
| import indic_stt.model as stt_model | |
| from indic_stt.model import FORK, EG2SpeechModel, codemix_fork_in, codemix_heads_in | |
| from indic_stt.text import load_tokenizer, normalize | |
| RELEASE = Path(__file__).resolve().parents[1] | |
| DEFAULT_WEIGHTS = RELEASE / "safetensors" / "stt2_model.safetensors" | |
| TOKENIZER = RELEASE / "tokenizer" / "spm.model" | |
| SAMPLE_RATE = 16000 | |
| EVALUATED_SECS = 60.0 # eval_dump.py --max-secs for the reported numbers | |
| MODES = {"codemix": ("transducer_cm", "cm"), "tamil": ("transducer", "ta")} # mode -> (RNN-T head, CTC head of ctc_log_probs) | |
| START_PENALTY = 1.0 # eval_dump.py --start-penalty 1 --start-cue, both modes | |
| LAMBDA, BETA, BEAM = 0.3, 0.0, 4 # --ctc-assist: decode_ctc_assist.py --joint 0.3:0, beam 4 | |
| CTC_ASSIST_DEFAULT = True # THE switch for the offline RNN-T default: True = --ctc-assist unless --no-ctc-assist (also mlx/run_mlx.py) | |
| CTC_ASSIST_DEFAULT_MODES = ("codemix",) # modes the default applies to; --mode tamil is greedy unless --ctc-assist (PM, 2026-11-12) | |
| STREAMING_FINALS_DEFAULT = True # STT-1.1 (PM, 2026-11-13): the default above also applies with --streaming, to the final text only | |
| # (partials greedy); False = no J by default with --streaming (with FINALS_DEFAULT "f" too: STT-1's greedy --streaming). | |
| # mlx/run_mlx.py has no streaming finals: its --streaming stays greedy. | |
| FINALS = ("two-pass", "f") # --finals choices (module docstring) | |
| FINALS_DEFAULT = "two-pass" # STT-1.2 (PM, 2026-11-13): "f" = STT-1.1's finals; "f" with STREAMING_FINALS_DEFAULT False = STT-1 | |
| TWO_PASS_MAX_SECS = 30.0 # --finals two-pass up to this utterance length; longer ones get f, with a note (O(T^2) reader pass; | |
| # training clips were at most 30 s) | |
| def finals_for(finals: str | None, streaming: bool, secs: float) -> str | None: | |
| """The finals a clip of `secs` seconds gets: None offline; f for two-pass over TWO_PASS_MAX_SECS; else `finals`.""" | |
| if not streaming: | |
| return None | |
| return "f" if finals == "two-pass" and secs > TWO_PASS_MAX_SECS else finals | |
| def load_audio(path: str | Path) -> np.ndarray: | |
| """Mono float32 at 16 kHz. Other rates are resampled with librosa, as the project's data loader does.""" | |
| wav, sr = sf.read(str(path), dtype="float32", always_2d=True) | |
| wav = wav.mean(axis=1) | |
| if sr != SAMPLE_RATE: | |
| import librosa | |
| wav = librosa.resample(wav, orig_sr=sr, target_sr=SAMPLE_RATE) | |
| return wav | |
| def features(fe, wav: np.ndarray) -> np.ndarray: | |
| """Log-mel frames (T, 128) with no padding: data.py _prepare, eval branch (no 30 s truncation).""" | |
| out = fe([wav], sampling_rate=SAMPLE_RATE, return_tensors="np", truncation=False) | |
| return out["input_features"][0][: int(out["input_features_mask"][0].sum())].astype(np.float32) | |
| class Release: | |
| """The model, tokenizer and feature extractor from one weights file. model, tok, fe, cfg, source.""" | |
| def __init__(self, weights: str | Path = DEFAULT_WEIGHTS, device: str = "cpu", tokenizer: str | Path | None = None): | |
| from transformers import AutoFeatureExtractor | |
| weights = Path(weights) | |
| tmp = None | |
| try: | |
| if weights.suffix == ".gguf": | |
| from indic_stt.gguf_io import read_gguf | |
| arrays, meta = read_gguf(weights) | |
| state = {k: torch.from_numpy(v) for k, v in arrays.items()} | |
| del arrays | |
| tmp = Path(tempfile.mkdtemp(prefix="indic_stt_gguf_")) | |
| (tmp / "config.json").write_text(meta["eg2_config"], encoding="utf-8") | |
| (tmp / "preprocessor_config.json").write_text(meta["preprocessor_config"], encoding="utf-8") | |
| (tmp / "spm.model").write_bytes(meta["spm_model"]) | |
| config_dir, cfg = tmp, meta["stt_config"] | |
| tok_path = Path(tokenizer) if tokenizer else tmp / "spm.model" | |
| self.source = f"{weights.name} ({meta['source']}; tensor types {meta['tensor_types']})" | |
| else: | |
| from safetensors.torch import load_file | |
| state = load_file(str(weights)) | |
| config_dir = weights.parent / "eg2" | |
| cfg = json.loads((weights.parent / "stt_config.json").read_text(encoding="utf-8")) | |
| tok_path = Path(tokenizer) if tokenizer else TOKENIZER | |
| self.source = weights.name | |
| self.tok = load_tokenizer(tok_path) | |
| if self.tok.vocab_size != cfg["vocab_size"]: | |
| raise ValueError(f"tokenizer {tok_path} has {self.tok.vocab_size} ids, the model {cfg['vocab_size']}") | |
| heads = codemix_heads_in(state) | |
| if heads != cfg.get("codemix_heads", heads): | |
| raise ValueError(f"{weights.name}: code-mix heads {heads}, but its config says {cfg['codemix_heads']}") | |
| fork = codemix_fork_in(state) # STT-2: the code-mix reader fork, fork_cm.* | |
| if fork != (cfg.get("codemix_fork", False) is True): | |
| raise ValueError(f"{weights.name}: {'holds' if fork else 'has no'} {FORK}.* tensors, but its config says " | |
| f"codemix_fork = {cfg.get('codemix_fork', 'absent')}: refused (a fork must be declared and complete, " | |
| "or the code-mix heads would read the wrong stack)") | |
| if fork and not all(heads.values()): | |
| raise ValueError(f"{weights.name}: the code-mix fork without both code-mix heads {heads}") | |
| self.fe = AutoFeatureExtractor.from_pretrained(str(config_dir)) | |
| stt_model.BASE_MODEL = str(config_dir) # the tower is built from this folder's config.json | |
| model = EG2SpeechModel(cfg["vocab_size"], train_tower_layers=cfg["train_tower_layers"], reader_layers=cfg["reader_layers"], | |
| stream_chunk_frames=cfg["stream_chunk_frames"], transducer=cfg["transducer"]) | |
| if model.offline_right: # one mix() feeds both reader passes: a tower with right context would leak future audio | |
| raise ValueError(f"offline_right = {model.offline_right}: this runner shares mix() between the streaming and " | |
| "two-pass reader passes, so the tower must have no right context (0)") | |
| if heads["transducer_cm"]: | |
| model.add_codemix_head(ctc=heads["ctc_head_cm"]) # before the load, as eval_dump.py does | |
| if fork: | |
| model.add_codemix_fork() # before the load, as eval_dump.py does; strict below: every fork_cm.* tensor or none | |
| want = {f"{FORK}.{k}" for k in model.fork_cm.state_dict()} | |
| have = {k for k in state if k.startswith(FORK + ".")} | |
| if have != want: | |
| raise ValueError(f"{weights.name}: codemix_fork needs all {len(want)} {FORK}.* tensors: missing " | |
| f"{len(want - have)} {sorted(want - have)[:3]}, unexpected {len(have - want)} {sorted(have - want)[:3]}") | |
| model.load_state_dict(state, strict=True) # strict: no tensor may silently keep its random start | |
| del state | |
| self.model, self.cfg, self.device = model.to(device).eval(), cfg, device | |
| self.modes = [m for m, (rnnt, _) in MODES.items() if getattr(model, rnnt) is not None | |
| and (m == "tamil" or model.ctc_head_cm is not None)] | |
| self._cpu_heads = {} | |
| finally: | |
| if tmp is not None: | |
| shutil.rmtree(tmp, ignore_errors=True) | |
| def forked(self, mode: str | None) -> bool: | |
| """Whether `mode` reads the code-mix fork (STT-2): --mode codemix on a file with fork_cm.*; else the one shared stack. | |
| mode None (STT-1.2's mix/read/encode took no mode): the one stack on a file without the fork, as before; refused on a fork | |
| file, where the stack depends on the mode and the reader output does not record which stack made it (R3 B1).""" | |
| if mode is None: | |
| if self.model.fork_cm is not None: | |
| raise ValueError(f"{self.source} has the code-mix reader fork ({FORK}): pass mode='codemix' or 'tamil' to " | |
| "mix/read/encode, and the same mode to greedy/ctc_assist") | |
| return False | |
| return mode == "codemix" and self.model.fork_cm is not None | |
| def mix(self, feats: np.ndarray, bf16: bool = False, mode: str | None = None): | |
| """The reader's input (1, T, D) and lengths (1,) for `mode`'s stack: model.mix (tower, layer mix, position conv), or | |
| model.mix_cm (the same tower, the fork's layer mix and position conv) in code-mix mode on a fork file. It does not depend | |
| on the mask (offline_right = 0, checked in __init__), so a streaming caller computes it once and keeps it for both reader | |
| passes. Pass the same mode to read() and to the decode (greedy, ctc_assist). mode is required on a fork file (None | |
| raises, see forked()); None on a file without the fork reads the one stack, as STT-1.2's mix(feats) did.""" | |
| f = self.model.mix_cm if self.forked(mode) else self.model.mix | |
| x = torch.from_numpy(feats)[None].to(self.device) | |
| mask = torch.ones(x.shape[:2], dtype=torch.bool, device=self.device) | |
| with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16): | |
| return f(x, mask) | |
| def read(self, x, lens, streaming: bool, bf16: bool = False, mode: str | None = None): | |
| """Reader output (1, T, D) and lengths (1,) from mix()'s output of the same mode: 480 ms block-causal with streaming, else | |
| full attention. model.read, or model.read_cm (the fork's reader) in code-mix mode on a fork file. mode: as mix().""" | |
| f = self.model.read_cm if self.forked(mode) else self.model.read | |
| with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16): | |
| return f(x, lens, streaming=streaming) | |
| def encode(self, feats: np.ndarray, streaming: bool, bf16: bool = False, mode: str | None = None): | |
| """Reader output (1, T, D) and lengths (1,): read(mix()), the same operations as STT-1.1's one-step encode. mode: as mix().""" | |
| return self.read(*self.mix(feats, bf16, mode=mode), streaming, bf16, mode=mode) | |
| def _check(self, mode: str) -> tuple[str, str]: | |
| if mode not in self.modes: | |
| raise ValueError(f"--mode {mode}: this weights file has the heads for {self.modes} only") | |
| return MODES[mode] | |
| def _decode_mode(self, mode: str | None) -> str: | |
| """greedy/ctc_assist's mode: required on a fork file, as for mix/read/encode (None raises, see forked()); None on a file | |
| without the fork is "codemix", STT-1.2's default for greedy/ctc_assist.""" | |
| self.forked(mode) | |
| return "codemix" if mode is None else mode | |
| def greedy(self, enc, lens, mode: str | None = None, head: str = "rnnt", bf16: bool = False) -> str: | |
| """eval_dump.py's rows for one clip: CTC greedy (head="ctc"), or RNN-T greedy with the start penalty and the start cue. | |
| mode: the mode given to mix/read (required on a fork file, see _decode_mode()).""" | |
| rnnt_name, ctc_name = self._check(self._decode_mode(mode)) | |
| with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16): | |
| logp = self.model.ctc_log_probs(enc, ctc_name) | |
| if head == "ctc": | |
| return normalize(self.tok.greedy(logp, lens)[0]) | |
| cue = (logp.argmax(-1) != 0).cumsum(1) > 0 | |
| hyp = getattr(self.model, rnnt_name).greedy(enc, lens, allowed=None, swap=True, start_penalty=START_PENALTY, cue=cue, | |
| style=None, all_frame_penalty=0.0)[0] | |
| return normalize(self.tok.decode(hyp)) | |
| def ctc_assist(self, enc, lens, mode: str | None = None) -> str: | |
| """--ctc-assist for one clip: J(LAMBDA, BETA) over the RNN-T beam n-best and the CTC greedy hypothesis (ctc_assist.py), | |
| CPU float32 on the bf16-rounded reader output. Offline or streaming reader output alike: with --streaming, the final text. | |
| mode: as greedy().""" | |
| from ctc_assist import joint_rescore | |
| mode = self._decode_mode(mode) | |
| rnnt_name, ctc_name = self._check(mode) | |
| if mode not in self._cpu_heads: # the heads as decode_ctc_assist.py holds them: float32 copies on the CPU | |
| ctc = self.model.ctc_head_cm if ctc_name == "cm" else self.model.ctc_head | |
| rnnt = getattr(self.model, rnnt_name) | |
| self._cpu_heads[mode] = ((rnnt, ctc) if self.device == "cpu" | |
| else (copy.deepcopy(rnnt).float().cpu().eval(), copy.deepcopy(ctc).float().cpu().eval())) | |
| rnnt, ctc = self._cpu_heads[mode] | |
| e = enc[0, : int(lens[0])].bfloat16().float().cpu() # eval_dump.py --save-enc stores bf16; decode_ctc_assist reads .float() | |
| best, _ = joint_rescore(rnnt, ctc, e, LAMBDA, BETA, BEAM) | |
| return normalize(self.tok.decode(best)) | |
| def transcribe(self, feats: np.ndarray, mode: str = "codemix", head: str = "rnnt", streaming: bool = False, | |
| ctc_assist: bool = False, bf16: bool = False, finals: str = FINALS_DEFAULT) -> str: | |
| """One clip's text: offline, or with streaming the utterance's final. finals "two-pass": the offline decode on a | |
| full-attention read() of the kept mix() output, so the offline text; "f": the decode on the block-causal read() (with | |
| ctc_assist=False the greedy stream, which is also the partials' text). finals_for() gives f over TWO_PASS_MAX_SECS, as | |
| main() applies it; this method runs what it is given.""" | |
| if ctc_assist and head != "rnnt": | |
| raise ValueError("--ctc-assist rescores the RNN-T head: drop --head ctc") | |
| if finals not in FINALS: | |
| raise ValueError(f"finals must be one of {FINALS}, not {finals!r}") | |
| x, lens = self.mix(feats, bf16, mode=mode) | |
| enc, lens = self.read(x, lens, streaming and finals == "f", bf16, mode=mode) # two-pass: the end-of-utterance pass, full attention | |
| return self.ctc_assist(enc, lens, mode) if ctc_assist else self.greedy(enc, lens, mode, head, bf16) | |
| def main() -> None: | |
| ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) | |
| ap.add_argument("audio", nargs="+") | |
| ap.add_argument("--weights", default=str(DEFAULT_WEIGHTS), help="a .safetensors (default) or .gguf file of this release") | |
| ap.add_argument("--mode", choices=list(MODES), default="codemix", help="codemix (default): code-mix heads; tamil: Tamil heads") | |
| ap.add_argument("--streaming", action="store_true", help="the 480 ms block-causal mask (default: offline)") | |
| ap.add_argument("--finals", choices=FINALS, help=f"with --streaming, how each utterance's final text is made (default " | |
| f"{FINALS_DEFAULT}): two-pass = the offline decode on a full-attention reader pass at the end of the utterance " | |
| f"(over {TWO_PASS_MAX_SECS:.0f} s: f, with a note); f = STT-1.1's, the decode on the streaming reader output") | |
| ap.add_argument("--head", choices=["rnnt", "ctc"], default="rnnt") | |
| ap.add_argument("--ctc-assist", action=argparse.BooleanOptionalAction, default=None, | |
| help=f"RNN-T: J(0.3, 0) CTC-assisted rescoring; with --streaming, of the final text only (default " | |
| f"{'on' if CTC_ASSIST_DEFAULT else 'off'} for --mode {', '.join(CTC_ASSIST_DEFAULT_MODES)}" | |
| f"{'' if STREAMING_FINALS_DEFAULT else ' offline'}, off otherwise)") | |
| ap.add_argument("--device", default="cpu") | |
| ap.add_argument("--no-bf16", action="store_true", help="on GPU, run float32 instead of the evals' bf16 autocast") | |
| ap.add_argument("--threads", type=int, help="CPU threads for PyTorch (default: PyTorch's choice)") | |
| ap.add_argument("--tokenizer", help="spm.model; default ../tokenizer/spm.model, or the copy inside a GGUF") | |
| ap.add_argument("--json", help="also write one JSON line per clip to this file") | |
| args = ap.parse_args() | |
| sys.stdout.reconfigure(encoding="utf-8") # Tamil text: Windows writes a redirected stdout in the ANSI code page otherwise | |
| if args.ctc_assist and args.head == "ctc": | |
| ap.error("--ctc-assist rescores the RNN-T head: drop --head ctc") | |
| if args.finals and not args.streaming: | |
| ap.error("--finals sets how --streaming makes its final text: add --streaming, or drop --finals") | |
| assist = (args.ctc_assist if args.ctc_assist is not None | |
| else CTC_ASSIST_DEFAULT and args.mode in CTC_ASSIST_DEFAULT_MODES and args.head == "rnnt" | |
| and (STREAMING_FINALS_DEFAULT or not args.streaming)) | |
| finals = (args.finals or FINALS_DEFAULT) if args.streaming else None | |
| if args.threads: | |
| torch.set_num_threads(args.threads) | |
| bf16 = args.device.startswith("cuda") and not args.no_bf16 | |
| rel = Release(args.weights, args.device, args.tokenizer) | |
| rel._check(args.mode) | |
| decode = "ctc-assist J(0.3, 0)" if assist else f"{'CTC' if args.head == 'ctc' else 'RNN-T'} greedy" | |
| how = ("" if finals == "f" and not assist | |
| else ", finals re-ranked with ctc-assist J(0.3, 0), partials greedy" if finals == "f" | |
| else f", two-pass finals (full-attention reader pass at the end of the utterance, then {decode}), partials greedy" if finals | |
| else ", ctc-assist J(0.3, 0)" if assist else "") | |
| row_name = (f"{args.mode}{' (code-mix reader fork)' if rel.forked(args.mode) else ''}, {'CTC' if args.head == 'ctc' else 'RNN-T'}, " | |
| f"{'streaming' if args.streaming else 'offline'}{how}") | |
| print(f"# weights: {rel.source}; {row_name}", file=sys.stderr, flush=True) | |
| out = open(args.json, "w", encoding="utf-8") if args.json else None | |
| for clip in args.audio: | |
| wav = load_audio(clip) | |
| secs = len(wav) / SAMPLE_RATE | |
| if secs > EVALUATED_SECS: | |
| print(f"# {clip}: {secs:.1f} s, longer than the {EVALUATED_SECS:.0f} s the reported numbers cover", file=sys.stderr) | |
| use = finals_for(finals, args.streaming, secs) | |
| if use != finals: | |
| print(f"# {clip}: {secs:.1f} s, over the {TWO_PASS_MAX_SECS:.0f} s of two-pass finals: STT-1.1 finals (--finals f) " | |
| f"for this file", file=sys.stderr) | |
| text = rel.transcribe(features(rel.fe, wav), args.mode, args.head, args.streaming, assist, bf16, use or FINALS_DEFAULT) | |
| print(f"{clip}\t{text}", flush=True) | |
| if out: | |
| out.write(json.dumps({"clip": str(clip), "secs": round(secs, 2), "mode": args.mode, "head": args.head, | |
| "streaming": args.streaming, "ctc_assist": assist, "finals": use, "hyp": text}, | |
| ensure_ascii=False) + "\n") | |
| if out: | |
| out.close() | |
| if __name__ == "__main__": | |
| main() | |