Indic Chat LID (small)

A 140M-parameter language-identification model for short chat messages in 12 South Asian languages, in both native script and the Latin alphabet ("romanized", as people type on phones: "kal mere interview tha", "tuition-e jabe ki bhabcho?").

General LID models handle native script well, but they mostly fail on romanized text: fastText LID-218 has no romanized Indic labels at all, and on our romanized chat test GlotLID gets 42% and AI4Bharat IndicLID 39%. This model gets 77%, and 93% at 5 words on Dakshina human romanizations.

Try it in your browser: altslate/indic-chat-lid Space (runs locally with transformers.js; your text never leaves your device).

Convert romanized text to native script with altslate/indic-chat-xlit-small once you know the language.

Labels (26): hi bn ta te mr gu kn ml pa ur sd si (native script), the same codes with -Latn (romanized), en, and other.

Usage

from transformers import pipeline
lid = pipeline("text-classification", model="altslate/indic-chat-lid-small")
lid(["kal mere interview tha", "ami tomake bhalobashi", "เฎจเฎพเฎณเฏˆเฎ•เฏเฎ•เฏ เฎตเฎฐเฏ‡เฎฉเฏ", "see you tomorrow"])
# [{'label': 'hi-Latn', 'score': 0.60}, {'label': 'bn-Latn', 'score': 0.98},
#  {'label': 'ta', 'score': 1.00}, {'label': 'en', 'score': 1.00}]
# (short romanized Hindi is often close to Urdu; see Known limitations)

Inputs are truncated to 64 tokens; the model is meant for messages and short passages. For long documents, classify a few sentences and take the majority.

ONNX (onnxruntime, transformers.js)

onnx/model.onnx (fp32, 563 MB) and onnx/model_quantized.onnx (per-channel dynamic int8, 142 MB). The fp32 export matches PyTorch (max logit difference 5e-5). The int8 file gives the same top label as PyTorch on 97.9% of 4,392 test messages, so expect slightly lower accuracy than the tables below.

import { pipeline } from "@huggingface/transformers";
const lid = await pipeline("text-classification", "altslate/indic-chat-lid-small", { dtype: "q8" });
console.log(await lid("kal mere interview tha"));

Results

All numbers are accuracy, averaged over languages (macro). None of these test sets was used for training or for choosing hyperparameters. IndicLID is the full official pipeline (FTN / FTR, with the BERT model for low-confidence romanized input). Its outputs for languages outside our 12 (for example Maithili or Nepali) count as wrong, which costs it most on romanized Hindi.

Test B: romanized chat messages (720, 60 per language)

Messages written by Qwen3.8-27B, a model that wrote none of the training text. The model wrote each message in native script first; we kept it only if GlotLID agreed on its language. It then romanized its own message. Topics are disjoint from the training-data topics.

romanized native
GlotLID .423 1.000
IndicLID .393 .849
this model .772 .997

Per language (romanized): bn .84, gu .81, hi .84, kn .82, ml .82, mr .77, pa .72, sd .81, si .87, ta .71, te .85, ur .42. With Hindi and Urdu merged: .825.

Test A: Dakshina test split (human romanizations, 500 sentences per language)

words GlotLID IndicLID this model
first 3 .623 .473 .846
first 5 .768 .540 .931
first 8 .841 .604 .965
full .880 .676 .975

Native script, first 5 words: GlotLID .984, IndicLID .878, this model .993. Caveat: our training data includes the Dakshina dev split's human romanizations, from the same annotators as this test split. Test A therefore flatters us; test B is the fairer measure.

Test C: support-style messages with IDs and phone numbers (800)

English, Hindi, Hinglish and six native-script languages, written by Qwen3.8-27B for a different project (PII detection) and never used here.

Hinglish (hi-Latn) English native (7 langs)
GlotLID .309 .994 .987โ€“1.000
this model .809 .974 .987โ€“1.000

Where other models are better

  • English: GlotLID is better on English (.994 vs .974 on test C). We sometimes call English other, or call an English message with a few Hindi words hi-Latn.
  • Languages we don't cover: GlotLID and IndicLID cover many more languages. This model has only other for everything else. On romanized sentences in 9 Indic languages we don't cover (from the Bhasha-Abhijnaanam test set), it still gives one of its 12 Indic labels to a share of full sentences that depends on the language: Manipuri 6%, Bodo 13%, Nepali 26%, Sanskrit 27%, Odia 34%, Kashmiri 36%, Maithili 42%, Assamese 54%, Konkani 59%. On the first 5 words only it is worse, 34โ€“81%. The worst cases are close neighbours: Assamese โ†’ bn-Latn, Konkani โ†’ mr-Latn. If your text may include those languages, use GlotLID or IndicLID, or treat low-confidence predictions as unknown.
  • Native script: GlotLID is equally good; IndicLID is lower on our tests.

Known limitations

  • One label per message. A message that mixes two Indian languages gets one label, often with high confidence: "repu chinemaki velli akkada minverva mein coffee peene jaate hain" (Telugu, then Hindi) is hi-Latn at 99%. For code-mixed text use altslate/indic-chat-wordlid-small, which labels each word.
  • Hindi vs Urdu when romanized is often undecidable ("kya qeemat hai?" is both), and romanized Urdu is mostly predicted as hi-Latn (test B: .42). Merge the two labels if you can.
  • Punjabi, Tamil and Marathi romanized chat are the next weakest (.71โ€“.77). Very short inputs (1โ€“3 words) are much less reliable.
  • Code-mixed messages get a single label; there is no per-word tagging.
  • Sinhala is not an Indian language but is included because Dakshina covers it.

Training

Training and evaluation code: github.com/AltSlate-Labs/indic-chat-toolkit.

  • Base: jhu-clsp/mmBERT-small (MIT), fine-tuned for sequence classification (26 labels), 1 epoch, batch 128, lr 5e-5, max length 64, on one NVIDIA GB10 (DGX Spark), about 30 minutes. 982k training rows, mostly random 2โ€“15-word windows.
  • Data:
    • Dakshina (Google, CC BY-SA 4.0): native-script Wikipedia sentences (train split), romanization lexicons (used to romanize sentences word by word, with chat-style noise), and the human romanizations of the dev split. The Dakshina test split was never used for training.
    • Romanized chat pairs written by Ornith-1.5-35B-A3B (MIT): about 25k messages in our 12 languages, plus about 5.6k in 9 other Indic languages (Assamese, Odia, Nepali, Maithili, Sanskrit, Konkani, Kashmiri, Manipuri, Bodo) labelled other. Pairs were kept only if GlotLID agreed on the native-script language.
    • Romanized Hindi: Abhishekcr448/Hinglish-Everyday-Conversations-1M (MIT) and findnitai/english-to-hinglish (Apache-2.0); English from the latter.
    • other: Wikipedia in 27 Latin-script languages via wikimedia/wikipedia (20231101), CC BY-SA 4.0.
  • No test message (B or Bhasha-Abhijnaanam) appears in the training data (exact-match check).

License

Apache-2.0. The base model is MIT. Training used the datasets above under their own licenses; Dakshina and Wikipedia text are CC BY-SA 4.0.

Citation

If you use the Dakshina-derived data or test, please cite: Roark et al., Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset, LREC 2020.

Downloads last month
52
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for altslate/indic-chat-lid-small

Quantized
(318)
this model

Spaces using altslate/indic-chat-lid-small 2

Collection including altslate/indic-chat-lid-small