Indic-Transliterate / README.md
harshitv804's picture
Add Requirements section to model card
68485c0 verified
|
Raw History Blame Contribute Delete
11.8 kB
---
license: apache-2.0
language:
- en
- ta
- hi
- te
- ml
- kn
datasets:
- ai4bharat/Aksharantar
library_name: onnx
pipeline_tag: translation
tags:
- transliteration
- indic
- tamil
- hindi
- telugu
- malayalam
- kannada
- onnx
- encoder-decoder
- beam-search
---
# Indic Transliterate
A compact **5.37M-parameter encoder-decoder transformer** for word-level
transliteration between English and five Indic scripts — Tamil, Hindi, Telugu,
Malayalam, and Kannada.
The whole model ships as **~12 MB of ONNX** and runs on CPU, which makes it
cheap to self-host and fast enough for live, word-by-word typing.
**Live demo:** [harshitv804/Indic-Transliterate](https://huggingface.co/spaces/harshitv804/Indic-Transliterate)
· **Weights:** [harshitv804/Indic-Transliterate](https://huggingface.co/harshitv804/Indic-Transliterate)
![Model architecture](assets/architecture.png)
## Highlights
| | |
|---|---|
| Parameters | 5.37 M |
| Vocabulary | 278 (byte-level BPE, character granularity) |
| Embedding dim | 128 (4 heads × 32) |
| Encoder / decoder | 12 / 12 layers |
| FFN hidden size | 256 (SwiGLU) |
| ONNX size | **12.8 MB** (fp16 weights, fp32 compute) |
| Training loss | **0.063** |
| Validation loss | **0.155** |
| Decoding | Beam search (beam size 4 by default) |
| Languages | English, Tamil, Hindi, Telugu, Malayalam, Kannada |
| Dataset | [AI4Bharat Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar) |
## Requirements
- **Python 3.10+**
- **ONNX inference (CPU only):** `huggingface-hub`, `numpy`, `onnxruntime`,
`tokenizers`
- **Web app:** `gradio`
- **Training / PyTorch inference:** `torch` (CUDA recommended) and
`liger-kernel`; training additionally needs `datasets`
Versions used by the Space (`requirements.txt`):
```text
gradio==6.32.0
huggingface-hub>=1.16.0,<3.0.0
numpy>=1.24,<3.0
onnxruntime>=1.17.0
tokenizers>=0.15.0
```
Minimal install for ONNX-only inference:
```bash
pip install huggingface-hub numpy onnxruntime tokenizers
```
## Quickstart
### 1. Install
This repository ships the ONNX weights **and** the CPU inference code, so you
can run it without PyTorch:
```bash
git clone https://huggingface.co/harshitv804/Indic-Transliterate
cd Indic-Transliterate
pip install huggingface-hub numpy onnxruntime tokenizers
```
For training, the training loop, or the PyTorch checkpoint, clone the code
repository instead:
```bash
git clone https://github.com/harshitv804/Indic-Transliterate
cd Indic-Transliterate
pip install -r requirements.txt
```
### 2. Import and run
```python
# Run from the repo root so ``transliterate`` is importable.
from transliterate import IndicTransliterator
tr = IndicTransliterator("onnx_model")
print(tr.translate("How are you?", "en-ta")) # ஹோவ் அரே யூ?
print(tr.translate("hello", "en-ta")) # ஹெல்லோ
print(tr.translate("vanakkam", "en-ta")) # வணக்கம்
print(tr.translate("வணக்கம்", "ta-en")) # vanakkam
```
Or point at a Hub snapshot directly:
```python
from huggingface_hub import snapshot_download
from transliterate import IndicTransliterator
model_dir = snapshot_download(
"harshitv804/Indic-Transliterate",
allow_patterns=["onnx_model/*"],
)
tr = IndicTransliterator(f"{model_dir}/onnx_model")
```
Decoding options:
```python
tr.translate("How are you?", "en-ta", beam_size=4, length_penalty=1.0)
tr.translate_batch(
["How are you?", "Good morning"],
["en-ta", "en-ta"],
)
```
### 3. Live demo
The interactive app (input left, output right, direction dropdown, live
word-by-word transliteration) runs here:
**[harshitv804/Indic-Transliterate on Spaces](https://huggingface.co/spaces/harshitv804/Indic-Transliterate)**
To run it locally, use the code repository and `python app.py`.
### 4. Run the PyTorch checkpoint
The training script, `inference.py`, and `step_105000.pt` live in the code
repository, not here:
```python
from inference import Transliterator
tr = Transliterator.from_checkpoint("step_105000.pt", device="cuda")
print(tr.translate("How are you?", "en-ta"))
```
```bash
python inference.py --checkpoint step_105000.pt --direction en-ta --text "How are you?"
```
Omit `--text` for an interactive prompt. The PyTorch path needs a CUDA GPU; the
ONNX path above is CPU-only and is what the Space uses.
## Architecture
A pre-norm encoder-decoder transformer. A source sentence is encoded once, then
the decoder emits one token at a time while cross-attending to the encoder
memory.
### Encoder (12 layers, bidirectional)
1. **Source token embedding** — `278 × 128`.
2. **ShortConv stem** — a depthwise convolution (`kernel 7`, `groups = dim`)
followed by SiLU, a pointwise linear projection, and a residual add. This
gives every position a local window of neighbouring characters, which helps
with script-level sound changes.
3. **12 gated self-attention + SwiGLU blocks**, each with two RMSNorms:
- Gated multi-head self-attention, **4 heads**, **bidirectional** (no causal
mask), with **QK-norm** and **RoPE** (`theta = 50 000`).
- SwiGLU feed-forward network with hidden size **256**.
4. **Final RMSNorm**, producing the encoder memory `[B, S, 128]`.
### Decoder (12 layers, causal + cross-attention)
1. **Target token embedding** — `278 × 128`, tied to the source embedding.
2. Each of the **12 blocks** contains three sub-layers, each pre-normed:
- **Gated causal self-attention** — RoPE, 4 heads, QK-norm.
- **Gated cross-attention** — Q/K from the decoder, K/V from the encoder
memory. No RoPE here, since query and key come from different sequences.
QK-norm still applies.
- **SwiGLU feed-forward**, hidden size 256.
3. **Final RMSNorm** and a **linear output layer** to `[B, T, 278] vocabulary`
logits, tied to the token embeddings.
So the `278 × 128` embedding matrix is **tied three ways**: source embedding,
target embedding, and output projection. That keeps the parameter count — and
the ONNX file — small.
### SwiGLU feed-forward
```
gate = SiLU(gate_proj(x))
up = up_proj(x)
out = down_proj(gate * up)
```
### ShortConv stem (encoder only)
```
y = x + Linear(SiLU(DepthwiseConv1d(x, kernel=7, groups=dim)))
```
## Tokenizer
A **byte-level BPE tokenizer with a 278-token vocabulary** and no normalizer.
With so few merges it behaves at **character level**, and because it is
byte-level it can never hit an unknown token for any script or punctuation.
The 22 non-byte tokens are:
- `<|bos|>`, `<|eos|>`, `<|pad|>`
- 10 **direction tags** — `<|ta-en|>`, `<|en-ta|>`, `<|hi-en|>`, `<|en-hi|>`,
`<|te-en|>`, `<|en-te|>`, `<|ml-en|>`, `<|en-ml|>`, `<|kn-en|>`,
`<|en-kn|>`
- 8 reserved slots
Zero-shot or new language pairs only need a new direction tag and a fine-tune —
the character inventory is already covered.
## Supported directions
| Source | Targets |
|---|---|
| English | Tamil, Hindi, Telugu, Malayalam, Kannada |
| Tamil, Hindi, Telugu, Malayalam, Kannada | English |
Ten directions in total: `ta-en`, `en-ta`, `hi-en`, `en-hi`, `te-en`, `en-te`,
`ml-en`, `en-ml`, `kn-en`, `en-kn`.
## Training
- **Dataset:** [AI4Bharat Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar),
the large parallel transliteration corpus covering Indic scripts. All five
languages are mixed into one model, with a direction tag prepended to every
source sequence.
- **Objective:** token-level cross-entropy, teacher forcing.
- **Final losses:** training **0.063**, validation **0.155**.
- **Optimizer:** AdamW, peak LR `1e-5`, `betas (0.9, 0.95)`, weight decay `0.1`
(skipped on embeddings, norms and biases), gradient clipping `1.0`.
- **Schedule:** 10k linear warmup, then constant LR.
- **Precision:** bf16; batch size 256; dropout 0.2.
- **Checkpoint used for this release:** `step_105000.pt` (105k optimizer steps).
## Inference
### Word-level decoding, sentence-level API
**The model transliterates one word at a time** — it is trained on single-word
pairs, so feeding it a whole sentence degrades quality. `transliterate.py`
therefore applies a custom word-boundary splitter (regex-style, Unicode-aware:
runs of letters and combining marks versus everything else), translates each
word run on its own, and reassembles the sentence with the original spaces,
punctuation, digits, and newlines untouched.
`en-*` inputs are lowercased first; Indic source text is passed through
unchanged.
### Beam search
Decoding is **beam search only** — no sampling. Beam size defaults to **4**
(`beam_size`, plus `length_penalty`), and the encoder runs once per word while
the decoder batches all beams together.
### ONNX on CPU
`export_onnx.py` exports the trained checkpoint to ONNX:
```bash
python export_onnx.py --checkpoint step_105000.pt --precision fp16
```
The fp16 build stores weights in half precision and **widens them to fp32
in-graph**, so compute stays numerically identical to the PyTorch model while
the bundle shrinks to **12.8 MB** (`encoder.onnx` 5.3 MB + `decoder.onnx`
7.5 MB). `onnx_inference.py` runs it with ONNX Runtime on CPU and needs no
PyTorch at all.
A practical note on speed: these layers are small, so ONNX Runtime thread
oversubscription hurts. The app caps intra-op threads at `min(4, cpu_count)`,
which took per-word latency on a shared container from ~2–7 s down to
~0.1–0.3 s.
## Gradio Space
`app.py` is the live demo app: input on the left, output on the right, a
direction dropdown, and **live word-by-word transliteration** — a word is
translated as soon as it is followed by a space or punctuation, the word
currently being typed stays untouched, and backspacing also removes its
translation. Completed words are cached per `(word, direction)` so retyping a
sentence costs nothing.
Environment variables:
- `MODEL_REPO` — Hugging Face repo to download the ONNX bundle from.
- `BEAM_SIZE` — beam width, default `4`.
- `WORD_CACHE_SIZE` — cached word translations, default `8192`.
- `ONNX_NUM_THREADS` — intra-op threads, default `min(4, cpu_count)`.
- `DIAG_TIMING` — set to `1` to log per-request timings.
## Files in this repository
```
onnx_model/encoder.onnx 5.26 MB encoder graph (fp16 weights)
onnx_model/decoder.onnx 7.52 MB decoder graph (fp16 weights)
onnx_model/tokenizer.json byte-level BPE tokenizer, 278 tokens
onnx_model/config.json architecture + direction ids + special tokens
```
Load the bundle with `IndicTransliterator("onnx_model")`; no PyTorch needed.
## Limitations
- Transliteration, not translation: it maps sounds and script, so the output is
phonetic, not semantic.
- Trained on five Indic languages; other scripts are not supported.
- Accuracy is highest on single words and short phrases. Long sentences are
handled word by word, so cross-word context is not modelled.
- Sanskrit-derived or very rare words may be spelled inconsistently.
- Inputs are lowercased for `en-*` directions, so casing is not preserved.
## License
Apache 2.0 — see [LICENSE](LICENSE).
## Acknowledgements
Trained on the [AI4Bharat Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar)
dataset by AI4Bharat, IIT Madras. Training used
[Liger Kernel](https://github.com/linkedin/Liger-Kernel) RMSNorm/SwiGLU kernels;
inference uses [ONNX Runtime](https://onnxruntime.ai/) and
[Gradio](https://gradio.app/).