junafinity's picture
claude opus 5.5
VeriSafe-Tanglish-1-STT (STT-2)
ec01e2d
|
Raw History Blame Contribute Delete
5.99 kB

Code

Shared by the format folders. Python 3.12.

Path What
transcribe.py The PyTorch runner for ../safetensors/ and ../gguf/ (--weights FILE; default ../safetensors/stt2_model.safetensors). A file with fork_cm.* tensors needs "codemix_fork": true in its settings and the reverse (strict check on load). --mode codemix (default) or --mode tamil, --head rnnt (default) or ctc, --streaming with --finals two-pass (default) or --finals f, --ctc-assist (default in code-mix mode: offline, and with --streaming for the final text, partials greedy) / --no-ctc-assist
ctc_assist.py --ctc-assist: J(0.3, 0) CTC-assisted rescoring. Its decoding functions are verbatim copies of the project's scripts/rnnt_beam.py and scripts/decode_ctc_assist.py (sha256 of both in its docstring)
indic_stt/model.py The model. Same as the project's src/indic_stt/model.py except at the three places marked PACKAGE CHANGE (plus two imports): the audio tower is built from a local config.json instead of downloading EmbeddingGemma 2 (two places), (STT-1.2) encode() is split into mix() (tower and layer mix) and read() (the reader), the same operations in the same order, so that a streaming caller can run the reader twice on one tower output, and (STT-2) a second post-tower stack, the code-mix reader fork fork_cm (layer mix, input norm and projection, position conv, 6 reader layers, output norm: 81 tensors), which the code-mix heads read when the file holds it; mix, read and encode then take mode="codemix" or mode="tamil"
indic_stt/text.py, indic_stt/tower_ctx.py Tokenizer wrapper and text normalization; tower context helper. Unchanged from the project
indic_stt/gguf_io.py GGUF writer and reader for this release
export/ How the formats were built and checked: export_release.py (safetensors, GGUF, MLX), extract_tower_rest.py, make_manifest.py, and the MLX sources copied into ../mlx/
cm_main_training_config.yaml The training config of STT-1 (run cm-main), whose tower, Tamil stack and Tamil heads STT-2 keeps, for the record. The fork run's own config is not shipped; ../safetensors/stt_config.json records the fork's provenance. Its local paths are not read

STT-2 has one tower, two post-tower stacks and two head sets, each an RNN-T head and a CTC head: the Tamil stack and heads (transducer, ctc_head; --mode tamil, Tamil script throughout; unchanged from STT-1) and the code-mix reader fork with the code-mix heads (fork_cm.*, transducer_cm, ctc_head_cm; --mode codemix, English words in Latin script). In the Python API, Release.mix, read, encode, greedy and ctc_assist need mode= on a fork file. In each mode the CTC head also gives the start cue of RNN-T greedy decoding: a start penalty of 1 on blank while nothing has been emitted, from the first frame where that CTC head leaves blank.

The tokenizer, ../tokenizer/spm.model, is a SentencePiece model with 2,058 pieces; model id = piece id + 1, and id 0 is the blank.

transcribe.py has one switch for the default: CTC_ASSIST_DEFAULT (shipped True, for the modes in CTC_ASSIST_DEFAULT_MODES = code-mix only: code-mix RNN-T rows use the J(0.3, 0) rescoring unless --no-ctc-assist; --mode tamil is greedy unless --ctc-assist; --head ctc rows are greedy either way). STT-1.1 adds STREAMING_FINALS_DEFAULT (shipped True): the default also applies with --streaming, where J re-ranks each utterance's final text once, at its end, on the whole utterance's streaming reader output; the partials (the live text) stay RNN-T greedy with the start cue, which is --streaming --no-ctc-assist and STT-1's streaming row. STT-1.2 adds --finals and its default FINALS_DEFAULT (shipped "two-pass"): with --streaming, at the end of each utterance the reader runs once more, with full attention, over the kept mix() output, and the final text is the offline decode of the same flags (J(0.3, 0) in code-mix; greedy with --no-ctc-assist, --mode tamil or --head ctc). So since STT-1.2 the partials' text, STT-1's streaming row, is --streaming --finals f --no-ctc-assist (Tamil: --mode tamil --streaming --finals f); --finals f is STT-1.1's finals, J on the streaming reader output. Utterances over TWO_PASS_MAX_SECS (30 s) get f, with a note on stderr. FINALS_DEFAULT = "f" restores STT-1.1; with STREAMING_FINALS_DEFAULT = False as well, STT-1's greedy --streaming. Release refuses a model with tower right context (offline_right > 0), since one mix() feeds both reader passes. A live caller runs mix() and read(..., streaming=True) + greedy() for the partials, starts the tower and position-conv state fresh at each utterance, and at its end runs read(..., streaming=False) over that utterance's frames only. The MLX runner, ../mlx/run_mlx.py, has the same CTC_ASSIST_DEFAULT with the same value, but no streaming finals and no --finals: its --streaming stays greedy.

Rebuild the formats from the fork candidate (f_B0.5.pt, written by the project's scripts/build_fork.py from STT-1's ema.pt and the fork run's EMA; neither checkpoint is published; needs torch, safetensors, gguf, sentencepiece, numpy). The safetensors step needs STT-1's safetensors file (STT1_FILE below; it is not in this repository) and checks the Tamil half against it byte for byte:

python -X utf8 export/extract_tower_rest.py EG2/model.safetensors ema.pt tower_rest.safetensors
python -X utf8 export/export_release.py safetensors f_B0.5.pt tower_rest.safetensors RELEASE_DIR STT1_FILE
python -X utf8 export/export_release.py gguf RELEASE_DIR f16      (also q8_0)
python -X utf8 export/export_release.py mlx RELEASE_DIR       (the float32 MLX file, not published; the input of mlx8)
python -X utf8 export/export_release.py mlx8 RELEASE_DIR      (the 8-bit file; needs MLX)
python export/make_manifest.py verify RELEASE_DIR