|
Download README.md from harshitv804/Indic-Transliterate: direct link, hf CLI and curl.
- Browser
- Download file 11.8 kB
-
https://huggingface.co/harshitv804/Indic-Transliterate/resolve/main/README.md
- Command line
-
hf download hf://harshitv804/Indic-Transliterate/README.md
-
curl -L -o README.md https://huggingface.co/harshitv804/Indic-Transliterate/resolve/main/README.md
11.8 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| - ta | |
| - hi | |
| - te | |
| - ml | |
| - kn | |
| datasets: | |
| - ai4bharat/Aksharantar | |
| library_name: onnx | |
| pipeline_tag: translation | |
| tags: | |
| - transliteration | |
| - indic | |
| - tamil | |
| - hindi | |
| - telugu | |
| - malayalam | |
| - kannada | |
| - onnx | |
| - encoder-decoder | |
| - beam-search | |
| # Indic Transliterate | |
| A compact **5.37M-parameter encoder-decoder transformer** for word-level | |
| transliteration between English and five Indic scripts — Tamil, Hindi, Telugu, | |
| Malayalam, and Kannada. | |
| The whole model ships as **~12 MB of ONNX** and runs on CPU, which makes it | |
| cheap to self-host and fast enough for live, word-by-word typing. | |
| **Live demo:** [harshitv804/Indic-Transliterate](https://huggingface.co/spaces/harshitv804/Indic-Transliterate) | |
| · **Weights:** [harshitv804/Indic-Transliterate](https://huggingface.co/harshitv804/Indic-Transliterate) | |
|  | |
| ## Highlights | |
| | | | | |
| |---|---| | |
| | Parameters | 5.37 M | | |
| | Vocabulary | 278 (byte-level BPE, character granularity) | | |
| | Embedding dim | 128 (4 heads × 32) | | |
| | Encoder / decoder | 12 / 12 layers | | |
| | FFN hidden size | 256 (SwiGLU) | | |
| | ONNX size | **12.8 MB** (fp16 weights, fp32 compute) | | |
| | Training loss | **0.063** | | |
| | Validation loss | **0.155** | | |
| | Decoding | Beam search (beam size 4 by default) | | |
| | Languages | English, Tamil, Hindi, Telugu, Malayalam, Kannada | | |
| | Dataset | [AI4Bharat Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar) | | |
| ## Requirements | |
| - **Python 3.10+** | |
| - **ONNX inference (CPU only):** `huggingface-hub`, `numpy`, `onnxruntime`, | |
| `tokenizers` | |
| - **Web app:** `gradio` | |
| - **Training / PyTorch inference:** `torch` (CUDA recommended) and | |
| `liger-kernel`; training additionally needs `datasets` | |
| Versions used by the Space (`requirements.txt`): | |
| ```text | |
| gradio==6.32.0 | |
| huggingface-hub>=1.16.0,<3.0.0 | |
| numpy>=1.24,<3.0 | |
| onnxruntime>=1.17.0 | |
| tokenizers>=0.15.0 | |
| ``` | |
| Minimal install for ONNX-only inference: | |
| ```bash | |
| pip install huggingface-hub numpy onnxruntime tokenizers | |
| ``` | |
| ## Quickstart | |
| ### 1. Install | |
| This repository ships the ONNX weights **and** the CPU inference code, so you | |
| can run it without PyTorch: | |
| ```bash | |
| git clone https://huggingface.co/harshitv804/Indic-Transliterate | |
| cd Indic-Transliterate | |
| pip install huggingface-hub numpy onnxruntime tokenizers | |
| ``` | |
| For training, the training loop, or the PyTorch checkpoint, clone the code | |
| repository instead: | |
| ```bash | |
| git clone https://github.com/harshitv804/Indic-Transliterate | |
| cd Indic-Transliterate | |
| pip install -r requirements.txt | |
| ``` | |
| ### 2. Import and run | |
| ```python | |
| # Run from the repo root so ``transliterate`` is importable. | |
| from transliterate import IndicTransliterator | |
| tr = IndicTransliterator("onnx_model") | |
| print(tr.translate("How are you?", "en-ta")) # ஹோவ் அரே யூ? | |
| print(tr.translate("hello", "en-ta")) # ஹெல்லோ | |
| print(tr.translate("vanakkam", "en-ta")) # வணக்கம் | |
| print(tr.translate("வணக்கம்", "ta-en")) # vanakkam | |
| ``` | |
| Or point at a Hub snapshot directly: | |
| ```python | |
| from huggingface_hub import snapshot_download | |
| from transliterate import IndicTransliterator | |
| model_dir = snapshot_download( | |
| "harshitv804/Indic-Transliterate", | |
| allow_patterns=["onnx_model/*"], | |
| ) | |
| tr = IndicTransliterator(f"{model_dir}/onnx_model") | |
| ``` | |
| Decoding options: | |
| ```python | |
| tr.translate("How are you?", "en-ta", beam_size=4, length_penalty=1.0) | |
| tr.translate_batch( | |
| ["How are you?", "Good morning"], | |
| ["en-ta", "en-ta"], | |
| ) | |
| ``` | |
| ### 3. Live demo | |
| The interactive app (input left, output right, direction dropdown, live | |
| word-by-word transliteration) runs here: | |
| **[harshitv804/Indic-Transliterate on Spaces](https://huggingface.co/spaces/harshitv804/Indic-Transliterate)** | |
| To run it locally, use the code repository and `python app.py`. | |
| ### 4. Run the PyTorch checkpoint | |
| The training script, `inference.py`, and `step_105000.pt` live in the code | |
| repository, not here: | |
| ```python | |
| from inference import Transliterator | |
| tr = Transliterator.from_checkpoint("step_105000.pt", device="cuda") | |
| print(tr.translate("How are you?", "en-ta")) | |
| ``` | |
| ```bash | |
| python inference.py --checkpoint step_105000.pt --direction en-ta --text "How are you?" | |
| ``` | |
| Omit `--text` for an interactive prompt. The PyTorch path needs a CUDA GPU; the | |
| ONNX path above is CPU-only and is what the Space uses. | |
| ## Architecture | |
| A pre-norm encoder-decoder transformer. A source sentence is encoded once, then | |
| the decoder emits one token at a time while cross-attending to the encoder | |
| memory. | |
| ### Encoder (12 layers, bidirectional) | |
| 1. **Source token embedding** — `278 × 128`. | |
| 2. **ShortConv stem** — a depthwise convolution (`kernel 7`, `groups = dim`) | |
| followed by SiLU, a pointwise linear projection, and a residual add. This | |
| gives every position a local window of neighbouring characters, which helps | |
| with script-level sound changes. | |
| 3. **12 gated self-attention + SwiGLU blocks**, each with two RMSNorms: | |
| - Gated multi-head self-attention, **4 heads**, **bidirectional** (no causal | |
| mask), with **QK-norm** and **RoPE** (`theta = 50 000`). | |
| - SwiGLU feed-forward network with hidden size **256**. | |
| 4. **Final RMSNorm**, producing the encoder memory `[B, S, 128]`. | |
| ### Decoder (12 layers, causal + cross-attention) | |
| 1. **Target token embedding** — `278 × 128`, tied to the source embedding. | |
| 2. Each of the **12 blocks** contains three sub-layers, each pre-normed: | |
| - **Gated causal self-attention** — RoPE, 4 heads, QK-norm. | |
| - **Gated cross-attention** — Q/K from the decoder, K/V from the encoder | |
| memory. No RoPE here, since query and key come from different sequences. | |
| QK-norm still applies. | |
| - **SwiGLU feed-forward**, hidden size 256. | |
| 3. **Final RMSNorm** and a **linear output layer** to `[B, T, 278] vocabulary` | |
| logits, tied to the token embeddings. | |
| So the `278 × 128` embedding matrix is **tied three ways**: source embedding, | |
| target embedding, and output projection. That keeps the parameter count — and | |
| the ONNX file — small. | |
| ### SwiGLU feed-forward | |
| ``` | |
| gate = SiLU(gate_proj(x)) | |
| up = up_proj(x) | |
| out = down_proj(gate * up) | |
| ``` | |
| ### ShortConv stem (encoder only) | |
| ``` | |
| y = x + Linear(SiLU(DepthwiseConv1d(x, kernel=7, groups=dim))) | |
| ``` | |
| ## Tokenizer | |
| A **byte-level BPE tokenizer with a 278-token vocabulary** and no normalizer. | |
| With so few merges it behaves at **character level**, and because it is | |
| byte-level it can never hit an unknown token for any script or punctuation. | |
| The 22 non-byte tokens are: | |
| - `<|bos|>`, `<|eos|>`, `<|pad|>` | |
| - 10 **direction tags** — `<|ta-en|>`, `<|en-ta|>`, `<|hi-en|>`, `<|en-hi|>`, | |
| `<|te-en|>`, `<|en-te|>`, `<|ml-en|>`, `<|en-ml|>`, `<|kn-en|>`, | |
| `<|en-kn|>` | |
| - 8 reserved slots | |
| Zero-shot or new language pairs only need a new direction tag and a fine-tune — | |
| the character inventory is already covered. | |
| ## Supported directions | |
| | Source | Targets | | |
| |---|---| | |
| | English | Tamil, Hindi, Telugu, Malayalam, Kannada | | |
| | Tamil, Hindi, Telugu, Malayalam, Kannada | English | | |
| Ten directions in total: `ta-en`, `en-ta`, `hi-en`, `en-hi`, `te-en`, `en-te`, | |
| `ml-en`, `en-ml`, `kn-en`, `en-kn`. | |
| ## Training | |
| - **Dataset:** [AI4Bharat Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar), | |
| the large parallel transliteration corpus covering Indic scripts. All five | |
| languages are mixed into one model, with a direction tag prepended to every | |
| source sequence. | |
| - **Objective:** token-level cross-entropy, teacher forcing. | |
| - **Final losses:** training **0.063**, validation **0.155**. | |
| - **Optimizer:** AdamW, peak LR `1e-5`, `betas (0.9, 0.95)`, weight decay `0.1` | |
| (skipped on embeddings, norms and biases), gradient clipping `1.0`. | |
| - **Schedule:** 10k linear warmup, then constant LR. | |
| - **Precision:** bf16; batch size 256; dropout 0.2. | |
| - **Checkpoint used for this release:** `step_105000.pt` (105k optimizer steps). | |
| ## Inference | |
| ### Word-level decoding, sentence-level API | |
| **The model transliterates one word at a time** — it is trained on single-word | |
| pairs, so feeding it a whole sentence degrades quality. `transliterate.py` | |
| therefore applies a custom word-boundary splitter (regex-style, Unicode-aware: | |
| runs of letters and combining marks versus everything else), translates each | |
| word run on its own, and reassembles the sentence with the original spaces, | |
| punctuation, digits, and newlines untouched. | |
| `en-*` inputs are lowercased first; Indic source text is passed through | |
| unchanged. | |
| ### Beam search | |
| Decoding is **beam search only** — no sampling. Beam size defaults to **4** | |
| (`beam_size`, plus `length_penalty`), and the encoder runs once per word while | |
| the decoder batches all beams together. | |
| ### ONNX on CPU | |
| `export_onnx.py` exports the trained checkpoint to ONNX: | |
| ```bash | |
| python export_onnx.py --checkpoint step_105000.pt --precision fp16 | |
| ``` | |
| The fp16 build stores weights in half precision and **widens them to fp32 | |
| in-graph**, so compute stays numerically identical to the PyTorch model while | |
| the bundle shrinks to **12.8 MB** (`encoder.onnx` 5.3 MB + `decoder.onnx` | |
| 7.5 MB). `onnx_inference.py` runs it with ONNX Runtime on CPU and needs no | |
| PyTorch at all. | |
| A practical note on speed: these layers are small, so ONNX Runtime thread | |
| oversubscription hurts. The app caps intra-op threads at `min(4, cpu_count)`, | |
| which took per-word latency on a shared container from ~2–7 s down to | |
| ~0.1–0.3 s. | |
| ## Gradio Space | |
| `app.py` is the live demo app: input on the left, output on the right, a | |
| direction dropdown, and **live word-by-word transliteration** — a word is | |
| translated as soon as it is followed by a space or punctuation, the word | |
| currently being typed stays untouched, and backspacing also removes its | |
| translation. Completed words are cached per `(word, direction)` so retyping a | |
| sentence costs nothing. | |
| Environment variables: | |
| - `MODEL_REPO` — Hugging Face repo to download the ONNX bundle from. | |
| - `BEAM_SIZE` — beam width, default `4`. | |
| - `WORD_CACHE_SIZE` — cached word translations, default `8192`. | |
| - `ONNX_NUM_THREADS` — intra-op threads, default `min(4, cpu_count)`. | |
| - `DIAG_TIMING` — set to `1` to log per-request timings. | |
| ## Files in this repository | |
| ``` | |
| onnx_model/encoder.onnx 5.26 MB encoder graph (fp16 weights) | |
| onnx_model/decoder.onnx 7.52 MB decoder graph (fp16 weights) | |
| onnx_model/tokenizer.json byte-level BPE tokenizer, 278 tokens | |
| onnx_model/config.json architecture + direction ids + special tokens | |
| ``` | |
| Load the bundle with `IndicTransliterator("onnx_model")`; no PyTorch needed. | |
| ## Limitations | |
| - Transliteration, not translation: it maps sounds and script, so the output is | |
| phonetic, not semantic. | |
| - Trained on five Indic languages; other scripts are not supported. | |
| - Accuracy is highest on single words and short phrases. Long sentences are | |
| handled word by word, so cross-word context is not modelled. | |
| - Sanskrit-derived or very rare words may be spelled inconsistently. | |
| - Inputs are lowercased for `en-*` directions, so casing is not preserved. | |
| ## License | |
| Apache 2.0 — see [LICENSE](LICENSE). | |
| ## Acknowledgements | |
| Trained on the [AI4Bharat Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar) | |
| dataset by AI4Bharat, IIT Madras. Training used | |
| [Liger Kernel](https://github.com/linkedin/Liger-Kernel) RMSNorm/SwiGLU kernels; | |
| inference uses [ONNX Runtime](https://onnxruntime.ai/) and | |
| [Gradio](https://gradio.app/). | |