Instructions to use altslate/indic-chat-xlit-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use altslate/indic-chat-xlit-small with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="altslate/indic-chat-xlit-small")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("altslate/indic-chat-xlit-small") model = AutoModelForSeq2SeqLM.from_pretrained("altslate/indic-chat-xlit-small", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Indic Chat Transliteration (small)
A 300M-parameter model that turns romanized chat ("kal meeting cancel ho gayi") back into native script ("कल मीटिंग कैंसल हो गई") for 12 South Asian languages: Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Sindhi and Sinhala.
It works on whole sentences, so context helps with ambiguous spellings, and it is trained on casual chat as well as on Wikipedia-style text. An optional keep-English mode (hi+en:) leaves English words in Latin script ("कल meeting cancel हो गई"). It is a fine-tuned ByT5-small, which works on bytes, so there is no vocabulary to run out of.
Try it in your browser: altslate/indic-chat-xlit Space (runs locally; optional auto-detect of the language).
If you don't know the language, detect it first with altslate/indic-chat-lid-small (it returns labels such as hi-Latn; use the part before the dash here).
Usage
The input is <language code>: <romanized text>.
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/indic-chat-xlit-small")
model = AutoModelForSeq2SeqLM.from_pretrained("altslate/indic-chat-xlit-small")
texts = ["hi: kal meeting cancel ho gayi, ghar pe hi hoon", "ta: naan naalaikku varen, nee ready-a iru",
"ur: aap kaise hain? sab khairiyat?", "bn: ami tomake pore phone korbo"]
enc = tok(texts, return_tensors="pt", padding=True)
out = model.generate(**enc, max_new_tokens=256)
print(tok.batch_decode(out, skip_special_tokens=True))
# ['कल मीटिंग कैंसल हो गई, घर पर ही हूँ', 'நான் நாளைக்கு வரேன், நீ ரெடியா இரு',
# 'آپ کیسے ہیں؟ سب خیریت؟', 'আমি তোমাকে পড়ে ফোন করব']
Keep English words in Latin script: use <code>+en: instead of <code>:, for example "hi+en: kal meeting cancel ho gayi" → कल meeting cancel हो गई. This mode is new in this version and has no test set of its own; see Known limitations.
The outputs show typical errors. In Bengali, pore here means "later" (পরে), not "read" (পড়ে). And pe became पर: the model sometimes writes the standard form rather than the exact chat spelling.
Language codes: hi bn ta te mr gu kn ml pa ur sd si (each also with +en). Set max_new_tokens to about 4× the input length in characters; native scripts take 2–3 bytes per character.
ONNX (onnxruntime, transformers.js)
onnx/ has the encoder and decoder (with and without past, and merged) in fp32 and per-channel dynamic int8 (*_quantized.onnx, about 300 MB in total). fp32 matches PyTorch (on 60 chat messages, CER 0.053 vs 0.052 for PyTorch in bf16; 97% of outputs identical). int8 is worse: on the same 60 messages CER rose to 0.070, and 62% of outputs were identical to PyTorch. Use fp32 if quality matters more than download size.
In transformers.js, ByT5 has no tokenizer.json, so encode bytes yourself (each UTF-8 byte + 3, then </s> = 1):
import { AutoModelForSeq2SeqLM, Tensor } from "@huggingface/transformers";
const model = await AutoModelForSeq2SeqLM.from_pretrained("altslate/indic-chat-xlit-small", { dtype: "q8" });
const ids = [...new TextEncoder().encode("hi: kal meeting cancel ho gayi")].map((b) => b + 3).concat([1]);
const t = (a) => new Tensor("int64", BigInt64Array.from(a.map(BigInt)), [1, a.length]);
const out = await model.generate({ input_ids: t(ids), attention_mask: t(ids.map(() => 1)), max_new_tokens: 4 * ids.length + 32 });
console.log(new TextDecoder().decode(new Uint8Array(out.tolist()[0].map(Number).filter((i) => i >= 3).map((i) => i - 3))));
Results
Metric: CER (character error rate against the reference native text, lower is better), macro-averaged over languages. Neither test set was used for training or for choosing settings.
- Test A: Dakshina test split, 100 sentences per language (1,200). Human romanizations of Wikipedia sentences.
- Test B: 696 casual chat messages. Qwen3.8-27B (which wrote none of our training data) wrote each message in native script, then romanized it; the pair was kept only if GlotLID agreed on the language. Its romanizations are not always faithful, so B has some reference noise and is best read as a comparison.
| A (Dakshina) | B (chat) | |
|---|---|---|
| copy input (no transliteration) | 1.004 | 0.964 |
| AI4Bharat IndicXlit (word by word, beam 4, no rescoring) | 0.097 | 0.160 |
previous version (v0, Nov 20; revision="v0") |
0.070 | 0.123 |
v1 (Nov 21, revision="v1") |
0.063 | 0.117 |
| this model (v2, greedy) | 0.057 | 0.116 |
Per language, CER (A / B); bold is the better of IndicXlit and this model:
| bn | gu | hi | kn | ml | mr | pa | sd | si | ta | te | ur | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| IndicXlit | .105 / .147 | .089 / .166 | .103 / .089 | .048 / .144 | .064 / .163 | .078 / .138 | .158 / .212 | .198 / .243 | .090 / .195 | .060 / .176 | .049 / .116 | .117 / .128 |
| previous version (v0) | .091 / .106 | .041 / .107 | .062 / .039 | .074 / .124 | .087 / .171 | .079 / .123 | .065 / .120 | .092 / .173 | .079 / .166 | .058 / .162 | .057 / .119 | .054 / .065 |
| v1 | .070 / .097 | .039 / .106 | .054 / .033 | .097 / .119 | .069 / .153 | .068 / .111 | .058 / .113 | .082 / .172 | .071 / .158 | .053 / .170 | .051 / .111 | .045 / .060 |
| this model (v2) | .063 / .102 | .036 / .109 | .054 / .026 | .061 / .118 | .063 / .145 | .066 / .111 | .056 / .112 | .078 / .172 | .066 / .160 | .052 / .163 | .043 / .115 | .043 / .062 |
Hindi only: UnimeType Transliteration-2.0-0.8B, whose only Indic language is Hindi, scores 0.292 (A) and 0.227 (B); this model scores 0.054 and 0.026 (default mode). UnimeType keeps English words in Latin script by design, and our references write them in Devanagari, which accounts for part of that gap.
Where other models are better
IndicXlit is better on Kannada Dakshina text (.048 vs .061). This version overtook it on Malayalam and Telugu. It is also much smaller (its checkpoint is 138 MB vs our 1.2 GB) and covers more languages. We ran it without its optional word-frequency rescoring, which would likely improve it further.
IndicXlit is a word-level model; it can be faster for single words and dictionary-style input.
Compared with v1, this version is slightly worse on chat in five languages (Bengali .097→.102, Gujarati .106→.109, Sinhala .158→.160, Telugu .111→.115, Urdu .060→.062), while the chat average is about the same (.117→.116) and Dakshina text improves in every language but Hindi (unchanged).
Known limitations
- Repetition on long inputs: on rare long sentences (2 of 1,896 test inputs in this version, down from 4 in v1) greedy decoding repeats a phrase until
max_new_tokens. Split long text into sentences; if an output is much longer than the input, re-run it in smaller pieces. Ano_repeat_ngram_sizesetting did not fix it on our dev set. - Default mode transliterates English words ("meeting" → मीटिंग), because that is how the references write them. The
+enmode keeps them in Latin script instead. It was trained on about 11k examples where English words were marked by a word-list rule, and has no test set of its own. On our chat test it kept mostly real English words (recharge, fridge, traffic, phone, appointment), but it also keeps misspelled English (tren, frij) and occasionally a romanized Indic word in Latin script. - Ambiguous romanizations (Bengali pore = পরে or পড়ে) are resolved from context and are sometimes wrong. Long vowels and nasalization (ं, ँ) are a common source of errors.
- The model normalizes some chat spellings to standard forms (pe → पर).
- You must give the language; mixing two Indic languages in one message is not supported.
- Long inputs: trained on up to about 380 bytes of input. Split long text into sentences.
Training
Training and evaluation code: github.com/AltSlate-Labs/indic-chat-toolkit.
Base: google/byt5-small (Apache-2.0). v0: 2 epochs on 178k pairs, batch 32, AdamW, lr 5e-4, linear schedule, bf16, about 90 minutes on one NVIDIA GB10 (DGX Spark). v1: v0 trained for 1 more epoch on 361k pairs, lr 2e-4, about 75 minutes. v2 (this version): v1 trained for 1 more epoch on 788k pairs (v1 data plus the Aksharantar additions below), lr 2e-4, about 2.3 hours.
Data (v0, also used in v1):
- Dakshina (Google, CC BY-SA 4.0): human-romanized sentences from the dev split (the test split is test A and was not used), and 8k word pairs per language from the training lexicons.
- About 25k romanized chat messages with their native-script originals, written by Ornith-1.5-35B-A3B (MIT) and kept only if GlotLID agreed on the language. No message matches a test-B message.
- Half of the romanized inputs are lowercased.
Added in v1:
- 20k word pairs per language from the Dakshina training lexicons (up from 8k).
- Native-script Dakshina Wikipedia sentences (train split) whose words are all in the lexicon, romanized word by word with the lexicon's human romanizations (6k each for Bengali, Hindi and Urdu, much fewer for others: Tamil 44, Gujarati 391).
- About 11k keep-English examples (
<code>+en:) from the chat pairs: English loanwords (a dictionary word frequent in English text and not a romanization of a common native word, plus a manual list of homographs such as main, men, the) are kept in Latin in the target.
Added in v2: word pairs from the train split of Aksharantar (AI4Bharat; mined pairs CC0, manually collected pairs CC-BY): 60k per language for Kannada, Malayalam, Telugu and Tamil, 25k for the others (Sindhi 17k; Sinhala is not in Aksharantar). Most are mined pairs from IndicCorp, of which we used the better-scoring half, plus names from Samanantar and Wikidata. These romanizations were mostly produced by AI4Bharat's own models, so part of what this version learned comes from them. We also added 2k pseudo-sentences per language, made of 10–25 random Aksharantar words, so that the model sees long outputs during training.
License
Apache-2.0. The base model is Apache-2.0. Training used the data above under its own licenses; Dakshina is CC BY-SA 4.0, Aksharantar CC0 / CC-BY.
Citation
Dakshina: Roark et al., Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset, LREC 2020. IndicXlit and Aksharantar: Madhani et al., Aksharantar: Open Indic-language Transliteration datasets and models for the Next Billion Users, 2023.
- Downloads last month
- 86
Model tree for altslate/indic-chat-xlit-small
Base model
google/byt5-small