Automatic Speech Recognition
PyTorch
Safetensors
GGUF
MLX
Tamil
English
speech
asr
streaming
rnnt
ctc
tamil
tanglish
code-switching
indian-english
verisafe
Instructions to use junafinity/VeriSafe-Tanglish-1-STT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use junafinity/VeriSafe-Tanglish-1-STT with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download junafinity/VeriSafe-Tanglish-1-STT --local-dir VeriSafe-Tanglish-1-STT
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
File size: 19,020 Bytes
ec01e2d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 | """Build the VeriSafe-Tanglish-STT-2 release formats from a push-8 fork candidate, and verify each against the one before it.
python -X utf8 export_release.py safetensors CANDIDATE.pt TOWER_REST.safetensors RELEASE_DIR STT1.safetensors
python -X utf8 export_release.py gguf RELEASE_DIR f32|f16|q8_0
python -X utf8 export_release.py mlx RELEASE_DIR
python -X utf8 export_release.py mlx8 RELEASE_DIR
python -X utf8 export_release.py check-stt1 RELEASE_DIR STT1.safetensors
safetensors: the candidate's 457 tensors (the project's scripts/build_fork.py output: cm-main ema.pt's 363 non-code-mix tensors,
step 84,000, unchanged = the tower and the Tamil stack and heads; the 13 code-mix head tensors transducer_cm + ctc_head_cm and
the code-mix reader fork, 81 fork_cm.* tensors, from the push-8 run) + the 482 tower tensors training never saved
(EmbeddingGemma 2's 480 clip bounds and output_proj, from extract_tower_rest.py) -> safetensors/stt2_model.safetensors,
939 float32 tensors, plus safetensors/stt_config.json (the settings transcribe.py builds the model with; "codemix_fork": true
makes transcribe.py require every fork_cm.* tensor). STT1.safetensors (STT-1's file) is required: check-stt1 runs after the
write (without it the command exits non-zero before writing anything).
gguf: the safetensors file -> gguf/stt2_model.FTYPE.gguf through indic_stt/gguf_io.py (tokenizer and configs inside).
f32 is checked byte-identical to the safetensors; f16 and q8_0 are checked for names, shapes and largest error.
mlx: the safetensors file -> mlx/stt2_model.mlx.safetensors with the 16 conv kernels (15 + the fork's position conv) in MLX's
channels-last layout, plus the port (mlx_model.py), its runner (run_mlx.py), text_norm.py and preprocessor_config.json;
layout.json records "codemix_fork". Checked byte-identical after the documented layout change. Writing it needs numpy and
safetensors, not MLX.
mlx8: mlx/stt2_model.mlx.safetensors -> mlx/stt2_model.mlx8.safetensors (needs MLX): the 198 2-D tensors (every matmul weight,
both embeddings; gguf_io's q8_0 rule with MLX's group of 64: 2-D, last dim a multiple of 64) quantized with mx.quantize,
affine, 8 bits, group 64: the packed uint32 under the tensor's name, float32 scales and biases as <stem>.scales and
<stem>.biases (mlx_model.quant_stem); the 16 conv kernels (MLX layout, as in the full file) and the 725 1-D tensors and
scalars float32, unchanged. Metadata "quantization" (mlx_model.QUANT) says so; layout.json applies unchanged. Checked: every
unquantized tensor byte-identical to the full MLX file, every quantized one equal to a fresh mx.quantize, and
mlx_model.load_weights reads the file back to the 939 names and shapes, float32.
check-stt1: every tensor of RELEASE_DIR's safetensors outside the code-mix heads and fork_cm.* is byte-equal (name, dtype, shape,
bytes) to STT-1's, and the file holds exactly STT-1's names plus the 81 fork_cm.* (push-8 rule T-EQ, second half).
"""
import hashlib
import json
import re
import shutil
import sys
from pathlib import Path
import numpy as np
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE.parent))
NAME = "stt2_model"
ST_NAME = f"{NAME}.safetensors"
MLX_NAME = f"{NAME}.mlx.safetensors"
MLX8_NAME = f"{NAME}.mlx8.safetensors"
MLX8_BITS, MLX8_GROUP = 8, 64
BASE_STEP = 84000 # cm-main ema.pt: the tower, the Tamil stack and the Tamil heads
BASE_SHA256 = "2e1dec78d078a2e04b622b1e545fadadbeb4a18e9e945c4c0931327b0ebf4b49" # cm-main ema.pt (build_fork.py refuses any other)
FORK_TENSORS = 81 # fork_cm.*: layer_logits, in_norm, in_proj, position, 6 reader layers, out_norm
TENSORS = 858 + FORK_TENSORS
RUN = "cm-main"
CODEMIX = ("transducer_cm.", "ctc_head_cm.")
CONV2D = re.compile(r"tower\.subsample_conv_projection\.layer\d\.conv\.weight")
CONV1D = re.compile(r"tower\.layers\.\d+\.lconv1d\.depthwise_conv1d\.weight|(fork_cm\.)?position\.conv\.weight")
def sha256(p: Path) -> str:
h = hashlib.sha256()
with open(p, "rb") as f:
while b := f.read(1 << 24):
h.update(b)
return h.hexdigest()
def cmd_safetensors(cand: str, rest: str, rel: str, stt1: str | None = None) -> None:
if not stt1 or not Path(stt1).is_file(): # R3 F1: T-EQ's byte half is not optional; refuse before writing anything
raise SystemExit(f"safetensors needs STT1.safetensors (STT-1's file) as its 5th argument, for check-stt1 (T-EQ byte half); "
f"got {stt1!r}: T-EQ byte half NOT run, nothing written")
import torch
from safetensors import safe_open
from safetensors.torch import load_file, save_file
cand, rel = Path(cand), Path(rel)
ck = torch.load(cand, map_location="cpu", weights_only=False, mmap=True)
state = dict(ck["model"])
cfg = ck["cfg"]
prov = cfg.get("codemix_fork")
assert isinstance(prov, dict), f"{cand.name}: no cfg['codemix_fork'] provenance: not a scripts/build_fork.py candidate"
assert prov["base_sha256"] == BASE_SHA256 and prov["base_step"] == BASE_STEP, (prov["base_sha256"], prov["base_step"])
from indic_stt.model import FORK, FORK_MODULES, codemix_fork_in, codemix_heads_in
assert codemix_fork_in(state), f"{cand.name} holds no {FORK}.* tensor"
fork = {k for k in state if k.startswith(FORK + ".")}
want = {f"{FORK}.{k}" for k in state if k.split(".")[0] in FORK_MODULES} # the fork copies the stack: the same names
assert fork == want and len(fork) == FORK_TENSORS, (len(fork), sorted(fork ^ want)[:5])
assert all(state[k].shape == state[k[len(FORK) + 1:]].shape for k in fork), "a fork tensor's shape differs from its twin's"
more = load_file(rest)
assert not set(state) & set(more), "the two sources overlap"
state.update(more)
assert len(state) == TENSORS and all(v.dtype == torch.float32 for v in state.values()), len(state)
heads = codemix_heads_in(state)
assert all(heads.values()), heads
digest = sha256(cand)
src_file = re.split(r"[\\/]", prov["src_ckpt"])[-1]
src = f"{prov['src_run']}/{src_file}" # run/file only: no local paths in the release
fork_info = {"kind": prov["kind"], "blend_alpha": prov["blend_alpha"], "base": f"{RUN}/ema.pt", "base_step": prov["base_step"],
"base_sha256": prov["base_sha256"], "src": src, "src_step": prov["src_step"], "src_sha256": prov["src_sha256"]}
out = rel / "safetensors" / ST_NAME
out.parent.mkdir(parents=True, exist_ok=True)
save_file({k: v.contiguous() for k, v in state.items()}, str(out), metadata={
"format": "pt", "source": f"{cand.name} (push-8 fork candidate, step {ck['step']}) sha256 {digest}",
"fork": json.dumps(fork_info, sort_keys=True),
"tower_rest": "google/embeddinggemma-2 snapshot f9972ac22e696a105c482202646979e33e4b5a1e: 480 clip bounds + output_proj"})
stt = {"vocab_size": int(state["ctc_head.weight"].shape[0]), "train_tower_layers": cfg["train_tower_layers"],
"reader_layers": cfg["reader_layers"], "stream_chunk_frames": cfg["stream_chunk_frames"], "transducer": cfg["transducer"],
"codemix_heads": heads, "codemix_fork": True, "blank_id": 0, "sample_rate": 16000, "step": ck["step"],
"checkpoint": cand.name, "checkpoint_sha256": digest, "fork": fork_info}
(out.parent / "stt_config.json").write_text(json.dumps(stt, indent=1) + "\n", encoding="utf-8", newline="\n")
with safe_open(str(out), "pt") as f:
names = set(f.keys())
assert names == set(state), "names differ"
bad = [k for k in names if not torch.equal(f.get_tensor(k), state[k])]
assert not bad, f"{len(bad)} tensors differ: {bad[:5]}"
print(f"{cand.name} sha256 {digest}; {len(state)} tensors ({len(fork)} {FORK}.*), {sum(v.numel() for v in state.values())} "
f"values -> {out} ({out.stat().st_size} bytes); read back equal; stt_config {stt}")
cmd_check_stt1(str(rel), stt1) # always (checked above): an assert here exits non-zero
def cmd_check_stt1(rel: str, stt1: str) -> None:
"""T-EQ, second half: every tensor outside the code-mix heads and fork_cm.* byte-equal to STT-1's safetensors."""
from safetensors import safe_open
out = Path(rel) / "safetensors" / ST_NAME
with safe_open(str(out), "np") as f, safe_open(stt1, "np") as g:
mine, ref = set(f.keys()), set(g.keys())
fork = {k for k in mine if k.startswith("fork_cm.")}
assert mine - fork == ref, f"names: {sorted((mine - fork) ^ ref)[:5]}"
assert len(fork) == FORK_TENSORS, len(fork)
same = [k for k in sorted(ref) if not k.startswith(CODEMIX)]
bad = []
for k in same:
a, b = f.get_tensor(k), g.get_tensor(k)
if a.dtype != b.dtype or a.shape != b.shape or a.tobytes() != b.tobytes():
bad.append(k)
cm_equal = sum(f.get_tensor(k).tobytes() == g.get_tensor(k).tobytes() for k in ref if k.startswith(CODEMIX))
assert not bad, f"{len(bad)} of {len(same)} tensors differ from STT-1's: {bad[:5]}"
print(f"check-stt1: {len(same)} of {len(same)} tensors outside the code-mix heads and fork_cm.* byte-equal to {Path(stt1).name}; "
f"names = STT-1's {len(ref)} + {len(fork)} fork_cm.*; code-mix head tensors equal to STT-1's: {cm_equal} of "
f"{len(ref) - len(same)} (information only)")
def cmd_gguf(rel: str, ftype: str) -> None:
from safetensors import safe_open
from safetensors.numpy import load_file
from indic_stt.gguf_io import read_gguf, write_gguf
rel = Path(rel)
src = rel / "safetensors" / ST_NAME
stt = json.loads((rel / "safetensors" / "stt_config.json").read_text(encoding="utf-8"))
with safe_open(str(src), "np") as f:
source = f.metadata()["source"]
state = load_file(str(src))
out = rel / "gguf" / f"{NAME}.{ftype}.gguf"
out.parent.mkdir(exist_ok=True)
meta = {"name": f"VeriSafe-Tanglish-STT-2 ({stt['checkpoint']}, step {stt['step']}), {ftype}", "step": stt["step"], "source": source,
"description": "Tamil + English streaming STT: EmbeddingGemma 2 audio tower + two post-tower stacks (layer mix + 6-layer "
"reader each: Tamil, and the code-mix fork fork_cm) + two head sets, each CTC + RNN-T (Tamil: ctc_head, "
"transducer; code-mix: ctc_head_cm, transducer_cm, which read the fork). Custom architecture: runs only with this release's own runner (code/transcribe.py, PyTorch); "
"no llama.cpp or whisper.cpp graph exists for it.",
"stt_config": {k: stt[k] for k in ("vocab_size", "train_tower_layers", "reader_layers", "stream_chunk_frames", "transducer",
"codemix_heads", "codemix_fork")},
"eg2_config": (rel / "safetensors" / "eg2" / "config.json").read_text(encoding="utf-8"),
"preprocessor_config": (rel / "safetensors" / "eg2" / "preprocessor_config.json").read_text(encoding="utf-8"),
"spm_model": (rel / "tokenizer" / "spm.model").read_bytes()}
counts = write_gguf(state, out, ftype, meta)
back, m = read_gguf(out)
assert set(back) == set(state), f"names differ: {sorted(set(back) ^ set(state))[:5]}"
assert all(back[k].shape == state[k].shape for k in state), "shapes differ"
assert m["spm_model"] == meta["spm_model"] and m["eg2_config"] == meta["eg2_config"] and m["preprocessor_config"] == meta["preprocessor_config"]
assert m["stt_config"] == meta["stt_config"] and m["stt_config"]["codemix_fork"] is True, m["stt_config"] # transcribe.py's fork rule
err = {k: float(np.abs(back[k] - state[k]).max()) for k in state}
worst = max(err, key=err.get)
if ftype == "f32":
bad = [k for k in state if back[k].tobytes() != state[k].tobytes()]
assert not bad, f"{len(bad)} tensors differ: {bad[:5]}"
rel_err = float(np.abs(back[worst] - state[worst]).max() / max(np.abs(state[worst]).max(), 1e-12))
print(f"{out} ({out.stat().st_size} bytes): {len(state)} tensors, types {counts}; configs and tokenizer bytes equal; "
f"largest abs error {err[worst]:.3g} in {worst} (relative to its max |w|: {rel_err:.3g})"
+ ("; byte-identical to the safetensors" if ftype == "f32" else ""))
def to_mlx(name: str, t: np.ndarray) -> np.ndarray:
if CONV2D.fullmatch(name):
return np.ascontiguousarray(t.transpose(0, 2, 3, 1)) # safetensors writes a view's raw memory, not its transposed values
if CONV1D.fullmatch(name):
return np.ascontiguousarray(t.transpose(0, 2, 1))
return t
def cmd_mlx(rel: str) -> None:
from safetensors.numpy import load_file, save_file
rel = Path(rel)
out = rel / "mlx"
out.mkdir(exist_ok=True)
state = load_file(str(rel / "safetensors" / ST_NAME))
stt = json.loads((rel / "safetensors" / "stt_config.json").read_text(encoding="utf-8"))
fork = sum(k.startswith("fork_cm.") for k in state)
assert fork == (FORK_TENSORS if stt.get("codemix_fork") else 0), (fork, stt.get("codemix_fork"))
conv = sorted(k for k in state if CONV2D.fullmatch(k) or CONV1D.fullmatch(k))
assert len(conv) == 15 + bool(fork), conv # the fork's position conv is the 16th
save_file({k: to_mlx(k, v) for k, v in state.items()}, str(out / MLX_NAME), metadata={"format": "mlx"})
(out / "layout.json").write_text(json.dumps({"changed_layout": conv, "conv2d": "(out,in,kT,kF)->(out,kT,kF,in)",
"conv1d": "(out,in,k)->(out,k,in)", "tensors": len(state),
"codemix_fork": bool(stt.get("codemix_fork"))}, indent=1),
encoding="utf-8", newline="\n")
shutil.copy2(rel / "safetensors" / "eg2" / "preprocessor_config.json", out / "preprocessor_config.json")
for f in ["mlx_model.py", "run_mlx.py", "text_norm.py"]:
shutil.copy2(HERE / f, out / f)
del state
ref = load_file(str(rel / "safetensors" / ST_NAME))
got = load_file(str(out / MLX_NAME))
assert set(ref) == set(got), f"names differ: {sorted(set(ref) ^ set(got))[:5]}"
bad = [k for k in ref if to_mlx(k, ref[k]).tobytes() != got[k].tobytes() or to_mlx(k, ref[k]).shape != got[k].shape or got[k].dtype != np.float32]
assert not bad, f"{len(bad)} tensors differ: {bad[:5]}"
# text_norm.py must stay equal to indic_stt/text.py's normalize
sys.path.insert(0, str(out))
import text_norm
from indic_stt.text import normalize
probe = ["Hello, World! it's “fine” — ok", "வணக்கம்। நான் don't-know…", " क़ुतुब मीनार ", "A.B.C; 12:30"]
assert all(text_norm.normalize(s) == normalize(s) for s in probe), "text_norm.normalize differs from indic_stt.text.normalize"
print(f"{len(ref)} tensors, {len(conv)} with changed layout -> {out / MLX_NAME} ({(out / MLX_NAME).stat().st_size} bytes); "
"every tensor byte-identical after the layout change, all float32; text_norm equal on the probes")
def cmd_mlx8(rel: str) -> None:
import mlx.core as mx
sys.path.insert(0, str(HERE))
from mlx_model import QUANT, load_weights, quant_stem
out = Path(rel) / "mlx"
src, dst = out / MLX_NAME, out / MLX8_NAME
layout = json.loads((out / "layout.json").read_text(encoding="utf-8"))
state = mx.load(str(src))
assert len(state) == layout["tensors"] == TENSORS and all(v.dtype == mx.float32 for v in state.values()), len(state)
assert not [k for k in state if k.endswith((".scales", ".biases"))], "a source name ends in .scales or .biases"
quant = sorted(k for k, v in state.items() if v.ndim == 2 and v.shape[-1] % MLX8_GROUP == 0)
assert len(quant) == sum(v.ndim == 2 for v in state.values()) == 198, len(quant) # every 2-D tensor, as q8_0's Q8_0 set
assert not set(quant) & set(layout["changed_layout"]), "a relaid conv kernel would be quantized"
stems = {quant_stem(k) for k in quant}
assert len(stems) == len(quant) and not {f"{s}.{x}" for s in stems for x in ("scales", "biases")} & set(state)
arrays = {}
for k, v in state.items():
if k in quant:
wq, s, b = mx.quantize(v, group_size=MLX8_GROUP, bits=MLX8_BITS, mode="affine")
arrays[k], arrays[f"{quant_stem(k)}.scales"], arrays[f"{quant_stem(k)}.biases"] = wq, s, b
else:
arrays[k] = v
mx.eval(list(arrays.values()))
q = {"mode": "affine", "bits": MLX8_BITS, "group_size": MLX8_GROUP, "tensors": len(quant)}
meta = {"format": "mlx", QUANT: json.dumps(q, sort_keys=True), "source": f"{MLX_NAME} sha256 {sha256(src)}"}
mx.save_safetensors(str(dst), arrays, metadata=meta)
# read back: the stored tensors, then load_weights' float32 view against the full file
got, got_meta = mx.load(str(dst), return_metadata=True)
assert set(got) == set(arrays) and len(got) == TENSORS + 2 * len(quant) and got_meta == meta, (len(got), got_meta)
bad = [k for k in arrays if got[k].dtype != arrays[k].dtype or got[k].shape != arrays[k].shape
or np.array(got[k]).tobytes() != np.array(arrays[k]).tobytes()]
assert not bad, f"{len(bad)} stored tensors differ from what was written: {bad[:5]}"
again = [k for k in quant if any(np.array(a).tobytes() != np.array(got[n]).tobytes() for a, n in zip(
mx.quantize(state[k], group_size=MLX8_GROUP, bits=MLX8_BITS, mode="affine"),
(k, f"{quant_stem(k)}.scales", f"{quant_stem(k)}.biases")))]
assert not again, f"{len(again)} tensors differ from a fresh mx.quantize: {again[:5]}"
deq, info = load_weights(dst)
assert info == q and set(deq) == set(state), "load_weights: names or quantization differ"
assert all(deq[k].dtype == mx.float32 and deq[k].shape == state[k].shape for k in state), "load_weights: dtype or shape"
same = [k for k in state if k not in quant and np.array(deq[k]).tobytes() == np.array(state[k]).tobytes()]
assert len(same) == len(state) - len(quant), f"{len(state) - len(quant) - len(same)} unquantized tensors differ"
err = {k: float(mx.abs(deq[k] - state[k]).max()) for k in quant}
worst = max(err, key=err.get)
rel_err = err[worst] / max(float(mx.abs(state[worst]).max()), 1e-12)
print(f"{dst} ({dst.stat().st_size} bytes): {len(got)} stored tensors = {len(state) - len(quant)} float32 unchanged "
f"(byte-identical to {MLX_NAME}: 16 conv kernels + {len(state) - len(quant) - 16} 1-D and scalar) + {len(quant)} "
f"quantized x (packed uint32, scales, biases); metadata {meta}; load_weights -> {len(deq)} float32 tensors, names and "
f"shapes equal; largest abs error {err[worst]:.3g} in {worst} (relative to its max |w|: {rel_err:.3g}); "
f"equal to a fresh mx.quantize")
if __name__ == "__main__":
{"safetensors": cmd_safetensors, "gguf": cmd_gguf, "mlx": cmd_mlx, "mlx8": cmd_mlx8, "check-stt1": cmd_check_stt1}[
sys.argv[1]](*sys.argv[2:])
|