How to use from the
Use from the
Transformers library
# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("token-classification", model="altslate/indic-punctuation-base")
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForTokenClassification

tokenizer = AutoTokenizer.from_pretrained("altslate/indic-punctuation-base")
model = AutoModelForTokenClassification.from_pretrained("altslate/indic-punctuation-base", device_map="auto")
Quick Links

Indic Punctuation Restoration (base)

Adds punctuation to unpunctuated text in 12 Indian languages and English: Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Sindhi, Sinhala. Use it after a speech-recognition model that writes no punctuation (IndicConformer, many Whisper fine-tunes), before altslate/indic-itn-small turns spoken numbers into digits.

Try it in your browser: altslate/indic-tts-normalizer, mode "Spoken → text", tick "Add punctuation first" (punctuation, then ITN; int8 in the browser, so commas are weaker than in the tables below).

For each word it predicts what follows it: nothing, a comma, a sentence end, or a question mark. It never changes the words themselves, and the marks are written in each language's own convention: । for Hindi, Bengali and Punjabi; ۔ ، ؟ for Urdu; ، ؟ with . for Sindhi; . , ? for the others.

मुझे कल ऑफिस जाना है मीटिंग कैंसल हो गई क्या तुम आओगे  →  मुझे कल ऑफिस जाना है। मीटिंग कैंसल हो गई, क्या तुम आओगे?
আমি কাল আসব তুমি কোথায় থাকবে                        →  আমি কাল আসব, তুমি কোথায় থাকবে?
میں کل آؤں گا آپ کہاں ہیں                              →  میں کل آؤں گا، آپ کہاں ہیں؟
ఈ రోజు వర్షం పడుతుంది మీరు గొడుగు తీసుకెళ్ళండి        →  ఈ రోజు వర్షం పడుతుంది, మీరు గొడుగు తీసుకెళ్ళండి.

(real model output, from the usage code below)

Results

Score: F1 per class (comma, sentence end, question) over words, macro-averaged over the classes present in the test. No test sentence was used for training or for choosing settings.

  • Wikipedia: 3,600 rows of 1–4 sentences from Dakshina's test split (all 12 languages).
  • Messages: 600 short messages written by Qwen3.8-27B, which wrote none of the training text (10 languages; they contain sentence ends and 47 commas, no questions).
  • Chat (in-domain): 1,779 held-out chat messages from the same generator as part of the training data (with 993 questions); this flatters our model.

Comparison: 1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase (Apache-2.0, XLM-R, 47 languages including 10 of ours, run with its punctuators package). It sometimes changes the text (12% of the Wikipedia rows came back with a different number of words); the table scores both models only on the rows where it did not, so the comparison is fair to it.

Wikipedia (3,151 rows) messages (578) chat, in-domain (1,597)
1-800-BAD-CODE xlm-roberta 0.610 0.809 0.562
this model 0.667 0.783 0.750

By language on Wikipedia (this model / 1-800-BAD-CODE): bn .75/.72, gu .72/.71, hi .70/.72, kn .47/.50, ml .77/.77, mr .74/.67, pa .73/.69, ta .73/.80, te .73/.77, ur .52/.45, sd .53/.43, si .66/.63. On the 10 languages both models cover, the averages are equal (0.704 vs 0.705): we are better on Bengali, Gujarati, Marathi and Punjabi, worse on Tamil, Telugu, Hindi and Kannada. Urdu, Sindhi and Sinhala are not among its listed languages, and in Urdu it writes Latin , ? instead of ، ؟.

By class on Wikipedia (all 3,600 rows): sentence end 0.94, comma 0.58, question 0.44 (few questions in Wikipedia).

Usage

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/indic-punctuation-base")
model = AutoModelForTokenClassification.from_pretrained("altslate/indic-punctuation-base").eval()
MARKS = {"hi": "।", "bn": "।", "pa": "।", "ur": "۔"}            # sentence end; others use "."
COMMA = {"ur": "،", "sd": "،"}; QUESTION = {"ur": "؟", "sd": "؟"}

def punctuate(text, lang):
    words = text.split()
    enc = tok(words, is_split_into_words=True, truncation=True, max_length=512, return_tensors="pt")
    with torch.no_grad():
        pred = model(**enc).logits[0].argmax(-1).tolist()
    labels, prev = {}, None
    for pos, wid in enumerate(enc.word_ids()):   # one label per word, from its first sub-token
        if wid is not None and wid != prev:
            labels[wid] = model.config.id2label[pred[pos]]
        prev = wid
    mark = {"O": "", "COMMA": COMMA.get(lang, ","), "PERIOD": MARKS.get(lang, "."), "QUESTION": QUESTION.get(lang, "?")}
    return " ".join(w + mark[labels[i]] for i, w in enumerate(words))

print(punctuate("मुझे कल ऑफिस जाना है मीटिंग कैंसल हो गई क्या तुम आओगे", "hi"))

Feed up to about 100 words at a time (split longer text into windows).

ONNX (onnxruntime, transformers.js)

file size messages F1 notes
onnx/model.onnx (fp32) 1.2 GB 0.775 same labels as PyTorch on every word we checked
onnx/model_fp16.onnx 615 MB 0.765 close to PyTorch in Python onnxruntime (Wikipedia first 300 rows equal); but wrong output in transformers.js 4.3.2 (Node, dtype: "fp16"): commas and question marks everywhere. Use it from Python only.
onnx/model_quantized.onnx (int8) 309 MB 0.623 works in transformers.js (dtype: "q8"); commas suffer (comma F1 0.29 vs 0.59 on messages; sentence ends unchanged)

In the browser, use q8 (smaller, weaker commas) or fp32 (1.2 GB). transformers.js usage is the same as for altslate/indic-chat-wordlid-small: run AutoModelForTokenClassification, then take each word's label from its first sub-token (tokens that start with ▁).

Training

Training and evaluation code: github.com/AltSlate-Labs/indic-chat-toolkit (src/punct/, jobs punct-*).

  • Base: jhu-clsp/mmBERT-base (307M), token classification (label on each word's first sub-token), 1.5 epochs on 358k rows, batch 32, lr 4e-5, about 1 h 45 min on one NVIDIA GB10 (DGX Spark).
  • Data: punctuated text with the punctuation stripped as input (; : and quotes are dropped and not predicted; ! counts as a sentence end):
    • Dakshina (Google, CC BY-SA 4.0) native-script Wikipedia sentences, train split, 1–4 sentences per row;
    • chat messages written by Ornith-1.5-35B-A3B (MIT), the main source of questions;
    • English Wikipedia (CC BY-SA 4.0), lowercased.

Known limitations

  • Commas are the hard part (F1 around 0.6): where a comma or a full stop goes is often a matter of style, and in short chat the model tends to end the sentence where people write a comma ("my number is 98765 43210, please call" → "…43210. please call").
  • Weaker than 1-800-BAD-CODE's model in Tamil, Telugu, Hindi and Kannada on Wikipedia text (see the table). If you only need those languages, compare both.
  • Questions are rare in Wikipedia; question marks rely mostly on LLM-written chat, and the in-domain chat test (same generator) flatters them.
  • Trained on written text with its punctuation removed, not on real speech-recognition output; disfluencies, fillers and recognition errors are not modelled.
  • Sentence boundaries only: no capitalization (Indian scripts have none; English output stays lowercase), no other marks (;, :, quotes, !).
  • Romanized text ("naan naalaikku varuven…") was not in training; expect a sentence end at most.
  • Not checked by native speakers.

License

Apache-2.0. Training data licenses as listed above.

Downloads last month
-
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for altslate/indic-punctuation-base

Quantized
(302)
this model

Space using altslate/indic-punctuation-base 1

Collection including altslate/indic-punctuation-base