dictabert-tiny language check (int8 word table) + vocab

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+dictabert_vocab.txt filter=lfs diff=lfs merge=lfs -text

README.md

@@ -18,5 +18,9 @@ Files downloaded by the Android app (github.com/cyttic/android-hwrg), which runs
 | `line_large_int8.onnx` | CTC line model `cyttic/exp35-trocr-large-en-ctc-seqhead-allreal-synpunct` (TrOCR-large ViT encoder, 24 layers + 2-layer transformer CTC head), same inputs/outputs; dynamic int8 except the ViT layer-0 MLP fc2 (fp32). Decoded with the punctuation-aware LM fusion (alpha 0.3, beta 0.6, LM on Hebrew letters + space). |
 | `charset.json` | `chars[k]` = character of class k; `chars[0]` = CTC blank. |
 | `heb_char6_kn.bin.gz` | Character 6-gram LM, interpolated Kneser-Ney (Ben-Yehuda + DictaGen text), binary layout in `tools/export_lm.py`. |
+| `mlm_tiny_int8.onnx` | Language check (optional second pass): **`dicta-il/dictabert-tiny`** by Dicta (CC BY 4.0, not MIT), converted to ONNX; the 128k x 312 word table in int8 with one scale per row, everything else fp32. Inputs `input_ids`, `attention_mask` (R, L) int64 with `[MASK]` at `pos` (R,), `tgt` (R,) = the original token; output `logp` (R,) = ln P(tgt at pos). The app keeps the line decoder's 16 best lines and picks the one with the best  beam score + 0.25 x PLL + 2.0 x word pieces. Made by `tools/export_mlm.py`. |
+| `dictabert_vocab.txt` | dictabert-tiny's WordPiece vocabulary (line number = token id; CC BY 4.0, Dicta). |
 
 The line finder is `PaddlePaddle/PP-OCRv5_server_det_onnx` (Apache-2.0), downloaded from its own repo.
+
+License: MIT for our files; `mlm_tiny_int8.onnx` and `dictabert_vocab.txt` are derived from dicta-il/dictabert-tiny and stay under its CC BY 4.0 license (Shmidman et al., DictaBERT, 2023).

dictabert_vocab.txt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0fb90bfa35244d26f0065d1fcd0b5becc3da3d44d616a7e2aacaf6320b9fa2d0
+size 1500244

mlm_tiny_int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0440fa06317327f2a9ba046c7be253958627196d090b21c4b5797ed9960711d3
+size 60489001