Instructions to use altslate/indic-punctuation-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use altslate/indic-punctuation-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="altslate/indic-punctuation-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("altslate/indic-punctuation-base") model = AutoModelForTokenClassification.from_pretrained("altslate/indic-punctuation-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Indic Punctuation Restoration (base)
Adds punctuation to unpunctuated text in 12 Indian languages and English: Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Sindhi, Sinhala. Use it after a speech-recognition model that writes no punctuation (IndicConformer, many Whisper fine-tunes), before altslate/indic-itn-small turns spoken numbers into digits.
Try it in your browser: altslate/indic-tts-normalizer, mode "Spoken → text", tick "Add punctuation first" (punctuation, then ITN; int8 in the browser, so commas are weaker than in the tables below).
For each word it predicts what follows it: nothing, a comma, a sentence end, or a question mark. It never changes the words themselves, and the marks are written in each language's own convention: । for Hindi, Bengali and Punjabi; ۔ ، ؟ for Urdu; ، ؟ with . for Sindhi; . , ? for the others.
मुझे कल ऑफिस जाना है मीटिंग कैंसल हो गई क्या तुम आओगे → मुझे कल ऑफिस जाना है। मीटिंग कैंसल हो गई, क्या तुम आओगे?
আমি কাল আসব তুমি কোথায় থাকবে → আমি কাল আসব, তুমি কোথায় থাকবে?
میں کل آؤں گا آپ کہاں ہیں → میں کل آؤں گا، آپ کہاں ہیں؟
ఈ రోజు వర్షం పడుతుంది మీరు గొడుగు తీసుకెళ్ళండి → ఈ రోజు వర్షం పడుతుంది, మీరు గొడుగు తీసుకెళ్ళండి.
(real model output, from the usage code below)
Results
Score: F1 per class (comma, sentence end, question) over words, macro-averaged over the classes present in the test. No test sentence was used for training or for choosing settings.
- Wikipedia: 3,600 rows of 1–4 sentences from Dakshina's test split (all 12 languages).
- Messages: 600 short messages written by Qwen3.8-27B, which wrote none of the training text (10 languages; they contain sentence ends and 47 commas, no questions).
- Chat (in-domain): 1,779 held-out chat messages from the same generator as part of the training data (with 993 questions); this flatters our model.
Comparison: 1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase (Apache-2.0, XLM-R, 47 languages including 10 of ours, run with its punctuators package). It sometimes changes the text (12% of the Wikipedia rows came back with a different number of words); the table scores both models only on the rows where it did not, so the comparison is fair to it.
| Wikipedia (3,151 rows) | messages (578) | chat, in-domain (1,597) | |
|---|---|---|---|
| 1-800-BAD-CODE xlm-roberta | 0.610 | 0.809 | 0.562 |
| this model | 0.667 | 0.783 | 0.750 |
By language on Wikipedia (this model / 1-800-BAD-CODE): bn .75/.72, gu .72/.71, hi .70/.72, kn .47/.50, ml .77/.77, mr .74/.67, pa .73/.69, ta .73/.80, te .73/.77, ur .52/.45, sd .53/.43, si .66/.63. On the 10 languages both models cover, the averages are equal (0.704 vs 0.705): we are better on Bengali, Gujarati, Marathi and Punjabi, worse on Tamil, Telugu, Hindi and Kannada. Urdu, Sindhi and Sinhala are not among its listed languages, and in Urdu it writes Latin , ? instead of ، ؟.
By class on Wikipedia (all 3,600 rows): sentence end 0.94, comma 0.58, question 0.44 (few questions in Wikipedia).
Usage
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
tok = AutoTokenizer.from_pretrained("altslate/indic-punctuation-base")
model = AutoModelForTokenClassification.from_pretrained("altslate/indic-punctuation-base").eval()
MARKS = {"hi": "।", "bn": "।", "pa": "।", "ur": "۔"} # sentence end; others use "."
COMMA = {"ur": "،", "sd": "،"}; QUESTION = {"ur": "؟", "sd": "؟"}
def punctuate(text, lang):
words = text.split()
enc = tok(words, is_split_into_words=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
pred = model(**enc).logits[0].argmax(-1).tolist()
labels, prev = {}, None
for pos, wid in enumerate(enc.word_ids()): # one label per word, from its first sub-token
if wid is not None and wid != prev:
labels[wid] = model.config.id2label[pred[pos]]
prev = wid
mark = {"O": "", "COMMA": COMMA.get(lang, ","), "PERIOD": MARKS.get(lang, "."), "QUESTION": QUESTION.get(lang, "?")}
return " ".join(w + mark[labels[i]] for i, w in enumerate(words))
print(punctuate("मुझे कल ऑफिस जाना है मीटिंग कैंसल हो गई क्या तुम आओगे", "hi"))
Feed up to about 100 words at a time (split longer text into windows).
ONNX (onnxruntime, transformers.js)
| file | size | messages F1 | notes |
|---|---|---|---|
onnx/model.onnx (fp32) |
1.2 GB | 0.775 | same labels as PyTorch on every word we checked |
onnx/model_fp16.onnx |
615 MB | 0.765 | close to PyTorch in Python onnxruntime (Wikipedia first 300 rows equal); but wrong output in transformers.js 4.3.2 (Node, dtype: "fp16"): commas and question marks everywhere. Use it from Python only. |
onnx/model_quantized.onnx (int8) |
309 MB | 0.623 | works in transformers.js (dtype: "q8"); commas suffer (comma F1 0.29 vs 0.59 on messages; sentence ends unchanged) |
In the browser, use q8 (smaller, weaker commas) or fp32 (1.2 GB). transformers.js usage is the same as for altslate/indic-chat-wordlid-small: run AutoModelForTokenClassification, then take each word's label from its first sub-token (tokens that start with ▁).
Training
Training and evaluation code: github.com/AltSlate-Labs/indic-chat-toolkit (src/punct/, jobs punct-*).
- Base: jhu-clsp/mmBERT-base (307M), token classification (label on each word's first sub-token), 1.5 epochs on 358k rows, batch 32, lr 4e-5, about 1 h 45 min on one NVIDIA GB10 (DGX Spark).
- Data: punctuated text with the punctuation stripped as input (
; :and quotes are dropped and not predicted;!counts as a sentence end):- Dakshina (Google, CC BY-SA 4.0) native-script Wikipedia sentences, train split, 1–4 sentences per row;
- chat messages written by Ornith-1.5-35B-A3B (MIT), the main source of questions;
- English Wikipedia (CC BY-SA 4.0), lowercased.
Known limitations
- Commas are the hard part (F1 around 0.6): where a comma or a full stop goes is often a matter of style, and in short chat the model tends to end the sentence where people write a comma ("my number is 98765 43210, please call" → "…43210. please call").
- Weaker than 1-800-BAD-CODE's model in Tamil, Telugu, Hindi and Kannada on Wikipedia text (see the table). If you only need those languages, compare both.
- Questions are rare in Wikipedia; question marks rely mostly on LLM-written chat, and the in-domain chat test (same generator) flatters them.
- Trained on written text with its punctuation removed, not on real speech-recognition output; disfluencies, fillers and recognition errors are not modelled.
- Sentence boundaries only: no capitalization (Indian scripts have none; English output stays lowercase), no other marks (
;,:, quotes,!). - Romanized text ("naan naalaikku varuven…") was not in training; expect a sentence end at most.
- Not checked by native speakers.
License
Apache-2.0. Training data licenses as listed above.
- Downloads last month
- -
Model tree for altslate/indic-punctuation-base
Base model
jhu-clsp/mmBERT-base