Automatic Speech Recognition
PyTorch
Safetensors
GGUF
MLX
Tamil
English
speech
asr
streaming
rnnt
ctc
tamil
tanglish
code-switching
indian-english
verisafe
Instructions to use junafinity/VeriSafe-Tanglish-1-STT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use junafinity/VeriSafe-Tanglish-1-STT with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download junafinity/VeriSafe-Tanglish-1-STT --local-dir VeriSafe-Tanglish-1-STT
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
File size: 25,558 Bytes
ec01e2d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 | """Transcribe audio files with VeriSafe-Tanglish-STT-2 (STT-1.1 added streaming finals, STT-1.2 made them two-pass, below; STT-2
adds the code-mix reader fork: STT-1's audio tower, Tamil stack and Tamil heads, cm-main ema.pt step 84,000, byte for byte, plus
code-mix heads and a code-mix post-tower stack of their own, fork_cm; stt_config.json names the checkpoint). Runs in PyTorch from
the safetensors or a GGUF file.
python transcribe.py CLIP.wav [CLIP2.mp3 ...] [--weights FILE] [--mode codemix|tamil] [--streaming [--finals two-pass|f]]
[--head rnnt|ctc] [--ctc-assist | --no-ctc-assist] [--device cpu|cuda] [--threads N] [--json OUT.jsonl]
--weights: ../safetensors/stt2_model.safetensors (default), or a GGUF from ../gguf/ (f32, f16 or q8_0). A GGUF holds the
tokenizer and both config files, so ../gguf/ and this code folder are enough for it. The GGUF runtime is this script: it reads
the file with the `gguf` package, dequantizes to float32 and runs PyTorch. llama.cpp and whisper.cpp cannot run this model.
Defaults (the code-mix recipe): --mode codemix; offline = RNN-T with --ctc-assist (J(0.3, 0) rescoring); --streaming = RNN-T
greedy partials with the start penalty and cue, and two-pass finals (--finals below): each utterance's final text is the
offline decode of the same flags, J(0.3, 0) in code-mix ("partials streaming (480 ms, no look-ahead); two-pass finals at end of
utterance"). --no-ctc-assist gives greedy, offline and in the two-pass final; --streaming --finals f --no-ctc-assist is STT-1's
streaming row, which is also the partials' text. --mode tamil gives the Tamil heads, greedy by default (the Tamil rows reported
are greedy); --mode tamil --ctc-assist opts in to J.
One shared audio tower, two sets of heads (each an RNN-T head and a CTC head):
--mode codemix (default) the code-mix heads: Tamil words in Tamil script, English words in Latin script.
--mode tamil the Tamil heads: Tamil script throughout, as stages 6 and 7 wrote.
STT-2: each mode reads its own post-tower stack (layer mix, in_proj, position conv, 6 reader layers, out_norm) on the one tower:
--mode tamil STT-1's (model.mix/read, unchanged), --mode codemix the fork (model.mix_cm/read_cm), for the offline text, the
partials and the two-pass final alike. A weights file with fork_cm.* tensors needs "codemix_fork": true in its stt_config, and
then all of them (strict load); fork_cm.* without the flag, or the flag without them, is refused. A file without the fork (STT-1,
-1.1, -1.2) runs as before: both modes on the one stack.
RNN-T greedy (--head rnnt, default) takes START_PENALTY (1) off blank's logit while a clip has emitted nothing, from the first
frame where the mode's CTC head leaves blank (the start cue). This is the project's evaluated decode, exactly:
codemix = scripts/eval_dump.py --mode codemix --start-penalty 1 --start-cue --cue-head cm
tamil = scripts/eval_dump.py --start-penalty 1 --start-cue
--head ctc: the mode's CTC head, greedy (eval_dump.py's CTC rows of the same run).
--streaming: the 480 ms block-causal mask over the whole clip, as every streaming number was measured; there is no
live-microphone loop here. Default: offline (the reader sees the whole clip).
--finals (with --streaming; default FINALS_DEFAULT below): how each utterance's final text is made. The partials, the live text
while the speaker talks, are the same in both: RNN-T greedy with the start cue on the block-causal reader output (a live caller
shows greedy(*read(x, lens, True, mode=m), mode=m) on the mix(feats, mode=m) output so far; this script prints the finals only).
On an STT-2 file Release.mix/read/encode and greedy/ctc_assist need mode= (None raises): the stack and the heads depend on it.
two-pass (STT-1.2, default) At the end of the utterance the reader runs once more, with full attention, over the kept mix()
output (the tower and layer mix, which do not depend on the mask, so they are not recomputed), and the final is the
offline decode of the same flags on that pass: J(0.3, 0) in code-mix, greedy with --no-ctc-assist, --mode tamil or
--head ctc. It is the offline text of the same flags on the same device. Utterances over TWO_PASS_MAX_SECS (30 s: the
full-attention pass costs O(T^2), and training clips were at most 30 s) get f instead, with a note on stderr.
f (STT-1.1) The same decode on the block-causal reader output itself: J(0.3, 0) re-ranks the streaming hypotheses (STT-1.1's
default --streaming); with --no-ctc-assist the final is the greedy stream (STT-1's --streaming row, no wait).
--ctc-assist (RNN-T only; on by default in --mode codemix): the J(0.3, 0) CTC-assisted rescoring of scripts/decode_ctc_assist.py (--joint 0.3:0): the
mode's RNN-T beam-4 n-best (no LM; the start penalty does not apply to it) plus the mode's CTC greedy hypothesis, ranked by
0.7 log P_RNNT + 0.3 log P_CTC (exact forced scores, ctc_assist.py). It runs on the CPU in float32 on the reader output rounded
to bf16, as the evaluated decode read it from eval_dump.py --save-enc dumps. Slower than greedy (a beam plus forced scores).
Its default is CTC_ASSIST_DEFAULT below (True) for the modes in CTC_ASSIST_DEFAULT_MODES (codemix); --ctc-assist /
--no-ctc-assist override it per run. With --head ctc it does not apply (it rescores the RNN-T head): that row is greedy, as is
--mode tamil without --ctc-assist.
--streaming --ctc-assist (the code-mix default since STT-1.1; STREAMING_FINALS_DEFAULT below): J(0.3, 0) runs once per
utterance, at its end, and gives the final text; the partials stay RNN-T greedy with the start cue. With --finals two-pass it
reads the full-attention reader pass, so the final equals the offline code-mix default. With --finals f it reads the streaming
reader output, the project's evaluated decode
scripts/decode_ctc_assist.py --head cm --mode streaming --start-penalty 1 --start-cue --joint 0.3:0 (its J0.3_b0 rows;
its greedy rows are --streaming --finals f --no-ctc-assist).
In --mode tamil the two-pass final is the Tamil offline greedy text; --mode tamil --streaming --finals f --ctc-assist runs but has
no reported numbers.
Output text is normalize()d (lower case, no punctuation), as the model was scored. The evals ran on GPU with bf16 autocast;
--device cuda does the same, CPU runs float32, so a few words can differ between the two.
"""
from __future__ import annotations
import argparse
import copy
import json
import shutil
import sys
import tempfile
from pathlib import Path
import numpy as np
import soundfile as sf
import torch
import indic_stt.model as stt_model
from indic_stt.model import FORK, EG2SpeechModel, codemix_fork_in, codemix_heads_in
from indic_stt.text import load_tokenizer, normalize
RELEASE = Path(__file__).resolve().parents[1]
DEFAULT_WEIGHTS = RELEASE / "safetensors" / "stt2_model.safetensors"
TOKENIZER = RELEASE / "tokenizer" / "spm.model"
SAMPLE_RATE = 16000
EVALUATED_SECS = 60.0 # eval_dump.py --max-secs for the reported numbers
MODES = {"codemix": ("transducer_cm", "cm"), "tamil": ("transducer", "ta")} # mode -> (RNN-T head, CTC head of ctc_log_probs)
START_PENALTY = 1.0 # eval_dump.py --start-penalty 1 --start-cue, both modes
LAMBDA, BETA, BEAM = 0.3, 0.0, 4 # --ctc-assist: decode_ctc_assist.py --joint 0.3:0, beam 4
CTC_ASSIST_DEFAULT = True # THE switch for the offline RNN-T default: True = --ctc-assist unless --no-ctc-assist (also mlx/run_mlx.py)
CTC_ASSIST_DEFAULT_MODES = ("codemix",) # modes the default applies to; --mode tamil is greedy unless --ctc-assist (PM, 2026-11-12)
STREAMING_FINALS_DEFAULT = True # STT-1.1 (PM, 2026-11-13): the default above also applies with --streaming, to the final text only
# (partials greedy); False = no J by default with --streaming (with FINALS_DEFAULT "f" too: STT-1's greedy --streaming).
# mlx/run_mlx.py has no streaming finals: its --streaming stays greedy.
FINALS = ("two-pass", "f") # --finals choices (module docstring)
FINALS_DEFAULT = "two-pass" # STT-1.2 (PM, 2026-11-13): "f" = STT-1.1's finals; "f" with STREAMING_FINALS_DEFAULT False = STT-1
TWO_PASS_MAX_SECS = 30.0 # --finals two-pass up to this utterance length; longer ones get f, with a note (O(T^2) reader pass;
# training clips were at most 30 s)
def finals_for(finals: str | None, streaming: bool, secs: float) -> str | None:
"""The finals a clip of `secs` seconds gets: None offline; f for two-pass over TWO_PASS_MAX_SECS; else `finals`."""
if not streaming:
return None
return "f" if finals == "two-pass" and secs > TWO_PASS_MAX_SECS else finals
def load_audio(path: str | Path) -> np.ndarray:
"""Mono float32 at 16 kHz. Other rates are resampled with librosa, as the project's data loader does."""
wav, sr = sf.read(str(path), dtype="float32", always_2d=True)
wav = wav.mean(axis=1)
if sr != SAMPLE_RATE:
import librosa
wav = librosa.resample(wav, orig_sr=sr, target_sr=SAMPLE_RATE)
return wav
def features(fe, wav: np.ndarray) -> np.ndarray:
"""Log-mel frames (T, 128) with no padding: data.py _prepare, eval branch (no 30 s truncation)."""
out = fe([wav], sampling_rate=SAMPLE_RATE, return_tensors="np", truncation=False)
return out["input_features"][0][: int(out["input_features_mask"][0].sum())].astype(np.float32)
class Release:
"""The model, tokenizer and feature extractor from one weights file. model, tok, fe, cfg, source."""
def __init__(self, weights: str | Path = DEFAULT_WEIGHTS, device: str = "cpu", tokenizer: str | Path | None = None):
from transformers import AutoFeatureExtractor
weights = Path(weights)
tmp = None
try:
if weights.suffix == ".gguf":
from indic_stt.gguf_io import read_gguf
arrays, meta = read_gguf(weights)
state = {k: torch.from_numpy(v) for k, v in arrays.items()}
del arrays
tmp = Path(tempfile.mkdtemp(prefix="indic_stt_gguf_"))
(tmp / "config.json").write_text(meta["eg2_config"], encoding="utf-8")
(tmp / "preprocessor_config.json").write_text(meta["preprocessor_config"], encoding="utf-8")
(tmp / "spm.model").write_bytes(meta["spm_model"])
config_dir, cfg = tmp, meta["stt_config"]
tok_path = Path(tokenizer) if tokenizer else tmp / "spm.model"
self.source = f"{weights.name} ({meta['source']}; tensor types {meta['tensor_types']})"
else:
from safetensors.torch import load_file
state = load_file(str(weights))
config_dir = weights.parent / "eg2"
cfg = json.loads((weights.parent / "stt_config.json").read_text(encoding="utf-8"))
tok_path = Path(tokenizer) if tokenizer else TOKENIZER
self.source = weights.name
self.tok = load_tokenizer(tok_path)
if self.tok.vocab_size != cfg["vocab_size"]:
raise ValueError(f"tokenizer {tok_path} has {self.tok.vocab_size} ids, the model {cfg['vocab_size']}")
heads = codemix_heads_in(state)
if heads != cfg.get("codemix_heads", heads):
raise ValueError(f"{weights.name}: code-mix heads {heads}, but its config says {cfg['codemix_heads']}")
fork = codemix_fork_in(state) # STT-2: the code-mix reader fork, fork_cm.*
if fork != (cfg.get("codemix_fork", False) is True):
raise ValueError(f"{weights.name}: {'holds' if fork else 'has no'} {FORK}.* tensors, but its config says "
f"codemix_fork = {cfg.get('codemix_fork', 'absent')}: refused (a fork must be declared and complete, "
"or the code-mix heads would read the wrong stack)")
if fork and not all(heads.values()):
raise ValueError(f"{weights.name}: the code-mix fork without both code-mix heads {heads}")
self.fe = AutoFeatureExtractor.from_pretrained(str(config_dir))
stt_model.BASE_MODEL = str(config_dir) # the tower is built from this folder's config.json
model = EG2SpeechModel(cfg["vocab_size"], train_tower_layers=cfg["train_tower_layers"], reader_layers=cfg["reader_layers"],
stream_chunk_frames=cfg["stream_chunk_frames"], transducer=cfg["transducer"])
if model.offline_right: # one mix() feeds both reader passes: a tower with right context would leak future audio
raise ValueError(f"offline_right = {model.offline_right}: this runner shares mix() between the streaming and "
"two-pass reader passes, so the tower must have no right context (0)")
if heads["transducer_cm"]:
model.add_codemix_head(ctc=heads["ctc_head_cm"]) # before the load, as eval_dump.py does
if fork:
model.add_codemix_fork() # before the load, as eval_dump.py does; strict below: every fork_cm.* tensor or none
want = {f"{FORK}.{k}" for k in model.fork_cm.state_dict()}
have = {k for k in state if k.startswith(FORK + ".")}
if have != want:
raise ValueError(f"{weights.name}: codemix_fork needs all {len(want)} {FORK}.* tensors: missing "
f"{len(want - have)} {sorted(want - have)[:3]}, unexpected {len(have - want)} {sorted(have - want)[:3]}")
model.load_state_dict(state, strict=True) # strict: no tensor may silently keep its random start
del state
self.model, self.cfg, self.device = model.to(device).eval(), cfg, device
self.modes = [m for m, (rnnt, _) in MODES.items() if getattr(model, rnnt) is not None
and (m == "tamil" or model.ctc_head_cm is not None)]
self._cpu_heads = {}
finally:
if tmp is not None:
shutil.rmtree(tmp, ignore_errors=True)
def forked(self, mode: str | None) -> bool:
"""Whether `mode` reads the code-mix fork (STT-2): --mode codemix on a file with fork_cm.*; else the one shared stack.
mode None (STT-1.2's mix/read/encode took no mode): the one stack on a file without the fork, as before; refused on a fork
file, where the stack depends on the mode and the reader output does not record which stack made it (R3 B1)."""
if mode is None:
if self.model.fork_cm is not None:
raise ValueError(f"{self.source} has the code-mix reader fork ({FORK}): pass mode='codemix' or 'tamil' to "
"mix/read/encode, and the same mode to greedy/ctc_assist")
return False
return mode == "codemix" and self.model.fork_cm is not None
@torch.no_grad()
def mix(self, feats: np.ndarray, bf16: bool = False, mode: str | None = None):
"""The reader's input (1, T, D) and lengths (1,) for `mode`'s stack: model.mix (tower, layer mix, position conv), or
model.mix_cm (the same tower, the fork's layer mix and position conv) in code-mix mode on a fork file. It does not depend
on the mask (offline_right = 0, checked in __init__), so a streaming caller computes it once and keeps it for both reader
passes. Pass the same mode to read() and to the decode (greedy, ctc_assist). mode is required on a fork file (None
raises, see forked()); None on a file without the fork reads the one stack, as STT-1.2's mix(feats) did."""
f = self.model.mix_cm if self.forked(mode) else self.model.mix
x = torch.from_numpy(feats)[None].to(self.device)
mask = torch.ones(x.shape[:2], dtype=torch.bool, device=self.device)
with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16):
return f(x, mask)
@torch.no_grad()
def read(self, x, lens, streaming: bool, bf16: bool = False, mode: str | None = None):
"""Reader output (1, T, D) and lengths (1,) from mix()'s output of the same mode: 480 ms block-causal with streaming, else
full attention. model.read, or model.read_cm (the fork's reader) in code-mix mode on a fork file. mode: as mix()."""
f = self.model.read_cm if self.forked(mode) else self.model.read
with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16):
return f(x, lens, streaming=streaming)
def encode(self, feats: np.ndarray, streaming: bool, bf16: bool = False, mode: str | None = None):
"""Reader output (1, T, D) and lengths (1,): read(mix()), the same operations as STT-1.1's one-step encode. mode: as mix()."""
return self.read(*self.mix(feats, bf16, mode=mode), streaming, bf16, mode=mode)
def _check(self, mode: str) -> tuple[str, str]:
if mode not in self.modes:
raise ValueError(f"--mode {mode}: this weights file has the heads for {self.modes} only")
return MODES[mode]
def _decode_mode(self, mode: str | None) -> str:
"""greedy/ctc_assist's mode: required on a fork file, as for mix/read/encode (None raises, see forked()); None on a file
without the fork is "codemix", STT-1.2's default for greedy/ctc_assist."""
self.forked(mode)
return "codemix" if mode is None else mode
@torch.no_grad()
def greedy(self, enc, lens, mode: str | None = None, head: str = "rnnt", bf16: bool = False) -> str:
"""eval_dump.py's rows for one clip: CTC greedy (head="ctc"), or RNN-T greedy with the start penalty and the start cue.
mode: the mode given to mix/read (required on a fork file, see _decode_mode())."""
rnnt_name, ctc_name = self._check(self._decode_mode(mode))
with torch.autocast(self.device.split(":")[0], dtype=torch.bfloat16, enabled=bf16):
logp = self.model.ctc_log_probs(enc, ctc_name)
if head == "ctc":
return normalize(self.tok.greedy(logp, lens)[0])
cue = (logp.argmax(-1) != 0).cumsum(1) > 0
hyp = getattr(self.model, rnnt_name).greedy(enc, lens, allowed=None, swap=True, start_penalty=START_PENALTY, cue=cue,
style=None, all_frame_penalty=0.0)[0]
return normalize(self.tok.decode(hyp))
@torch.no_grad()
def ctc_assist(self, enc, lens, mode: str | None = None) -> str:
"""--ctc-assist for one clip: J(LAMBDA, BETA) over the RNN-T beam n-best and the CTC greedy hypothesis (ctc_assist.py),
CPU float32 on the bf16-rounded reader output. Offline or streaming reader output alike: with --streaming, the final text.
mode: as greedy()."""
from ctc_assist import joint_rescore
mode = self._decode_mode(mode)
rnnt_name, ctc_name = self._check(mode)
if mode not in self._cpu_heads: # the heads as decode_ctc_assist.py holds them: float32 copies on the CPU
ctc = self.model.ctc_head_cm if ctc_name == "cm" else self.model.ctc_head
rnnt = getattr(self.model, rnnt_name)
self._cpu_heads[mode] = ((rnnt, ctc) if self.device == "cpu"
else (copy.deepcopy(rnnt).float().cpu().eval(), copy.deepcopy(ctc).float().cpu().eval()))
rnnt, ctc = self._cpu_heads[mode]
e = enc[0, : int(lens[0])].bfloat16().float().cpu() # eval_dump.py --save-enc stores bf16; decode_ctc_assist reads .float()
best, _ = joint_rescore(rnnt, ctc, e, LAMBDA, BETA, BEAM)
return normalize(self.tok.decode(best))
def transcribe(self, feats: np.ndarray, mode: str = "codemix", head: str = "rnnt", streaming: bool = False,
ctc_assist: bool = False, bf16: bool = False, finals: str = FINALS_DEFAULT) -> str:
"""One clip's text: offline, or with streaming the utterance's final. finals "two-pass": the offline decode on a
full-attention read() of the kept mix() output, so the offline text; "f": the decode on the block-causal read() (with
ctc_assist=False the greedy stream, which is also the partials' text). finals_for() gives f over TWO_PASS_MAX_SECS, as
main() applies it; this method runs what it is given."""
if ctc_assist and head != "rnnt":
raise ValueError("--ctc-assist rescores the RNN-T head: drop --head ctc")
if finals not in FINALS:
raise ValueError(f"finals must be one of {FINALS}, not {finals!r}")
x, lens = self.mix(feats, bf16, mode=mode)
enc, lens = self.read(x, lens, streaming and finals == "f", bf16, mode=mode) # two-pass: the end-of-utterance pass, full attention
return self.ctc_assist(enc, lens, mode) if ctc_assist else self.greedy(enc, lens, mode, head, bf16)
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("audio", nargs="+")
ap.add_argument("--weights", default=str(DEFAULT_WEIGHTS), help="a .safetensors (default) or .gguf file of this release")
ap.add_argument("--mode", choices=list(MODES), default="codemix", help="codemix (default): code-mix heads; tamil: Tamil heads")
ap.add_argument("--streaming", action="store_true", help="the 480 ms block-causal mask (default: offline)")
ap.add_argument("--finals", choices=FINALS, help=f"with --streaming, how each utterance's final text is made (default "
f"{FINALS_DEFAULT}): two-pass = the offline decode on a full-attention reader pass at the end of the utterance "
f"(over {TWO_PASS_MAX_SECS:.0f} s: f, with a note); f = STT-1.1's, the decode on the streaming reader output")
ap.add_argument("--head", choices=["rnnt", "ctc"], default="rnnt")
ap.add_argument("--ctc-assist", action=argparse.BooleanOptionalAction, default=None,
help=f"RNN-T: J(0.3, 0) CTC-assisted rescoring; with --streaming, of the final text only (default "
f"{'on' if CTC_ASSIST_DEFAULT else 'off'} for --mode {', '.join(CTC_ASSIST_DEFAULT_MODES)}"
f"{'' if STREAMING_FINALS_DEFAULT else ' offline'}, off otherwise)")
ap.add_argument("--device", default="cpu")
ap.add_argument("--no-bf16", action="store_true", help="on GPU, run float32 instead of the evals' bf16 autocast")
ap.add_argument("--threads", type=int, help="CPU threads for PyTorch (default: PyTorch's choice)")
ap.add_argument("--tokenizer", help="spm.model; default ../tokenizer/spm.model, or the copy inside a GGUF")
ap.add_argument("--json", help="also write one JSON line per clip to this file")
args = ap.parse_args()
sys.stdout.reconfigure(encoding="utf-8") # Tamil text: Windows writes a redirected stdout in the ANSI code page otherwise
if args.ctc_assist and args.head == "ctc":
ap.error("--ctc-assist rescores the RNN-T head: drop --head ctc")
if args.finals and not args.streaming:
ap.error("--finals sets how --streaming makes its final text: add --streaming, or drop --finals")
assist = (args.ctc_assist if args.ctc_assist is not None
else CTC_ASSIST_DEFAULT and args.mode in CTC_ASSIST_DEFAULT_MODES and args.head == "rnnt"
and (STREAMING_FINALS_DEFAULT or not args.streaming))
finals = (args.finals or FINALS_DEFAULT) if args.streaming else None
if args.threads:
torch.set_num_threads(args.threads)
bf16 = args.device.startswith("cuda") and not args.no_bf16
rel = Release(args.weights, args.device, args.tokenizer)
rel._check(args.mode)
decode = "ctc-assist J(0.3, 0)" if assist else f"{'CTC' if args.head == 'ctc' else 'RNN-T'} greedy"
how = ("" if finals == "f" and not assist
else ", finals re-ranked with ctc-assist J(0.3, 0), partials greedy" if finals == "f"
else f", two-pass finals (full-attention reader pass at the end of the utterance, then {decode}), partials greedy" if finals
else ", ctc-assist J(0.3, 0)" if assist else "")
row_name = (f"{args.mode}{' (code-mix reader fork)' if rel.forked(args.mode) else ''}, {'CTC' if args.head == 'ctc' else 'RNN-T'}, "
f"{'streaming' if args.streaming else 'offline'}{how}")
print(f"# weights: {rel.source}; {row_name}", file=sys.stderr, flush=True)
out = open(args.json, "w", encoding="utf-8") if args.json else None
for clip in args.audio:
wav = load_audio(clip)
secs = len(wav) / SAMPLE_RATE
if secs > EVALUATED_SECS:
print(f"# {clip}: {secs:.1f} s, longer than the {EVALUATED_SECS:.0f} s the reported numbers cover", file=sys.stderr)
use = finals_for(finals, args.streaming, secs)
if use != finals:
print(f"# {clip}: {secs:.1f} s, over the {TWO_PASS_MAX_SECS:.0f} s of two-pass finals: STT-1.1 finals (--finals f) "
f"for this file", file=sys.stderr)
text = rel.transcribe(features(rel.fe, wav), args.mode, args.head, args.streaming, assist, bf16, use or FINALS_DEFAULT)
print(f"{clip}\t{text}", flush=True)
if out:
out.write(json.dumps({"clip": str(clip), "secs": round(secs, 2), "mode": args.mode, "head": args.head,
"streaming": args.streaming, "ctc_assist": assist, "finals": use, "hyp": text},
ensure_ascii=False) + "\n")
if out:
out.close()
if __name__ == "__main__":
main()
|