---
license: apache-2.0
language:
- hi
- en
pipeline_tag: translation
tags:
- transliteration
- hindi
- devanagari
- romanized-hindi
- on-device
- browser
- javascript
- sentence-level
- language-model
- tiny
datasets:
- ai4bharat/IndicCorpV2
- ai4bharat/Aksharantar
metrics:
- accuracy
- cer
- wer
model-index:
- name: AksharaLP-Sentence-2.8M
results:
- task:
type: translation
name: Romanized Hindi → Devanagari transliteration (sentence level)
dataset:
type: google/dakshina
name: Dakshina (Hindi romanized sentences)
split: test
metrics:
- type: accuracy
name: Word accuracy
value: 89.44
- type: exact_match
name: Sentence exact match
value: 26.73
- type: cer
name: CER
value: 3.93
- type: wer
name: WER
value: 10.78
---
# AksharaLP-Sentence-2.8M
It doesn’t guess one word at a time. It fixes the whole sentence.
Romanized Hindi → Devanagari, decided sentence by sentence, in 10.85 MB, on the device.
A word model proposes the spellings; a 2.77M-parameter neural Hindi language model picks between them jointly.
---
**The one-line pitch.** Hindi typed in English letters usually goes wrong at the sentence level, not the word level: *hu* should be हूं here and हुआ there, *hai* should be है or हैं, *pata* should be पता or पाता. AksharaLP Sentence keeps the word model's proposals and adds a tiny neural language model that reads the whole sentence before choosing — and it is still small enough to ship inside a web page.
Recorded from the live demo: Boht ache h Bhai, mai ghr pe hu is typed, decoded by the word decoder, then re-decoded by the sentence decoder in a Web Worker in the browser tab. Both decoders are in this repository.
## The numbers, up front
| | | |
|:---:|:---:|:---:|
| **26.73%**
whole sentences exactly right
Dakshina Hindi sentence test, 4,976 sentences | **+1.02 pts**
sentences right vs the word model
95% CI 0.40–1.65 · significant | **+0.42 pts**
words right vs the word model
95% CI 0.33–0.51 · significant |
| **4.07 → 3.93**
CER on the same 4,976 sentences
WER 11.20 → 10.78 | **132,299**
output words audited
0 outside their own candidate list | **2.77M**
parameters in the sentence LM
+ 989K in the word model underneath |
Every number above was measured by us, in a single one-shot run of the shipped configuration. v2 and v3 were scored with the same harness on the same sentences; intervals are paired bootstrap over sentences (2,000 resamples, seed 0). Settings were chosen on validation and internal dev sets only.
## Same sentence, two decoders
**Real outputs**, produced by running both decoders in this repository on the inputs below (the 21-sentence casual set in `cases/`, plus a few more of the same kind). Nothing here is illustrative: these are the exact strings the runtime returns.
| You type | Word mode (v2) | Sentence mode (v3) | |
|---|---|---|---|
| `mai ghr pe hu` | मैं घर पे हुआ | **मैं घर पे हूं** | fixed |
| `mai school me hu` | मैं स्कूल में हुआ | **मैं स्कूल में हूं** | fixed |
| `mujhe nahi pata` | मुझे नहीं पाता | **मुझे नहीं पता** | fixed |
| `kal milte hai` | कल मिलते है | **कल मिलते हैं** | fixed |
| `hum ghr ja rhe hai` | हम घर जा रहे है | **हम घर जा रहे हैं** | fixed |
| `yaar mai thak gya hu` | यार मैं तक गया हूं | **यार मैं थक गया हूं** | fixed |
| `Boht ache h Bhai` | बोहत अच्छे है भाई | बोहत अच्छे है भाई | **not fixed** (see below) |
| `boht accha laga` | बोहत अच्छा लगा | बोहत अच्छा लगा | **not fixed** (see below) |
| `mein bazaar ja rahi hu` | में बाजार जा रही हूं | में बाजार जा रही हूं | **not fixed** (see below) |
| `nahi pata` | नहीं पाता | नहीं पाता | **not fixed** (see below) |
| `main ghar ja raha hoon` | मैं घर जा रहा हूं | मैं घर जा रहा हूं | already right |
| `wo ladki school gayi thi` | वो लड़की स्कूल गई थी | वो लड़की स्कूल गई थी | already right |
| `tum kya kar rahe ho` | तुम क्या कर रहे हो | तुम क्या कर रहे हो | already right |
| `ye kitab bahut acchi hai` | ये किताब बहुत अच्छी है | ये किताब बहुत अच्छी है | already right |
| `wo ghar me hai` | वो घर में है | वो घर में है | already right |
Word mode = the v2 decoder (word model + dictionary pack + bigram Viterbi). Sentence mode = the same candidates re-scored by the neural LM (`search3`, beam 8). Both ran on the same machine, same files, same harness. `Boht` → बोहत is a channel/candidate problem, not a sentence problem: standard spelling is बहुत.
### The four cases it does not fix, and why
| Input | What comes out | Why the sentence LM cannot help |
|---|---|---|
| `Boht ache h Bhai` | बोहत अच्छे है भाई | **बहुत is candidate #11** for *boht* — channel score −5.83, below the top-6 the LM is allowed to see (`nlm_k 6`, `nlm_margin 2`). The LM can only reorder what it is given, so the channel, not the LM, has to be fixed. |
| `mein bazaar ja rahi hu` | में बाजार जा रही हूं | **मैं is candidate #2 at −2.14**, just outside `nlm_margin 2` (the LM is skipped for a slot when the top candidate leads by more than 2). हूं is right; the pronoun stays में. |
| `boht accha laga` | बोहत अच्छा लगा | Same बहुत/बोहत channel problem as the first row. अच्छा is already correct here, so there is nothing else for the LM to change. |
| `nahi pata` | नहीं पाता | **No context.** Two words, no auxiliary, no agreement: पाता is a perfectly good sentence ending in a bigger corpus than the two-word one. `mujhe nahi pata` — three words — *is* fixed to मुझे नहीं पता, which is exactly the point: the LM needs a sentence to have a sentence to read. |
**And one it does not get credit for:** `kal` → कल. The older bigram LM already prefers कल (−1.03) to काल (−0.70) in every sentence we tried, so the neural LM has nothing left to do there.
## Measured results
| Set | Sentences | Word % v2 → v3 | Sentence % v2 → v3 | Δ sentence (95% CI) |
|---|---|---|---|---|
| **Dakshina Hindi sentence test** (held-out) | 4,976 | 89.02 → **89.44** | 25.70 → **26.73** | **+1.02 [0.40, 1.65]** ✓ |
| Context set (ambiguous words in context) | 66 | 94.89 → **97.02** | 86.36 → **90.91** | +4.55 [0.00, 10.61] (words +2.13 ✓) |
| val600 (selection) | 600 | 88.80 → **89.35** | 23.83 → **26.33** | **+2.50 [0.67, 4.33]** ✓ |
| val2 (selection-side) | 1,000 | 89.33 → **89.91** | 25.60 → **26.60** | +1.00 [−0.50, 2.50] |
| devA (casual chat, tuning) | 488 | 92.37 → **93.82** | 84.22 → **87.30** | **+3.07 [1.23, 4.92]** ✓ |
| devB (casual chat, report-only) | 480 | 91.54 → **92.22** | 82.50 → **83.96** | +1.46 [−0.42, 3.54] **not significant** |
Word-side intervals on the test set: **+0.42 [0.33, 0.51]**; sentence-exact: **+1.02 [0.40, 1.65]**. CER 4.07 → 3.93, WER 11.20 → 10.78. Word accuracy on the casual sets is token accuracy and sentence accuracy is whole-sentence exact match, both after Unicode NFC normalisation.
The test set was run once, after all settings were fixed. val600, val2 and devA are selection/tuning sets and are shown for completeness — their gains are real but they are not held-out. devB is the honest one: the sentence model gains 1.46 points of sentence accuracy there and that gain is not significant, which is why the card leads with the test-set number and not with devB.
### Word mode is unchanged, by construction
| Benchmark | v2 | v3 |
|---|---|---|
| Aksharantar Hindi test (9,991 words), top-1 | 70.04 | **70.04** |
| top-3 | 88.31 | **88.31** |
| top-8 | 93.63 | **93.63** |
The sentence model overrides only the search over an existing candidate lattice; candidate generation is the v2 code path, unmodified. These are the v2 numbers, not a re-run — that is the point. (Not re-run for v3 because there is nothing different to run.)
## What it changes
Gains and regressions on the 480-sentence casual-chat set (word accuracy per spelling pattern):
| Improves | v2 → v3 | | Regresses | v2 → v3 |
|---|---|---|---|---|
| Word-final y / h (`y_end`) | 82.98 → **89.36** (+6.4) | | Extra h (`asp_add`) | 85.45 → 81.82 (−3.6) |
| Missing nasal (`nasal`) | 86.00 → **90.00** (+4.0) | | Stretched letters (`repeat`) | 82.14 → 78.57 (−3.6) |
| Missing h in kh/th/bh (`asp_drop`) | 71.19 → **74.58** (+3.4) | | Dropped or added a (`schwa`) | 97.44 → 94.87 (−2.6) |
| Chat abbreviations (`abbr`) | 91.67 → **93.75** (+2.1) | | Long / short vowel spelling (`vlen`) | 96.43 → 94.64 (−1.8) |
| Extra doubled letters (`gem_add`) | 98.39 → **100.00** (+1.6) | | English words in chat (`english`) | 98.39 → 96.77 (−1.6) |
| Ambiguous short words (`homograph`) | 93.59 → **94.87** (+1.3) | | | |
The LM is good exactly where a bigram cannot be: verb endings, auxiliaries and agreement across the whole sentence. It is slightly too eager to normalise chat spellings toward dictionary forms — an extra *h*, stretched letters, a dropped or added *a* — and those rows are the cost. They are shown because they are real.
## The guardrail
> ### ✓ It reorders candidates. It never invents words.
> Every Devanagari word in the output is a candidate the word model already produced **for that typed word**. The sentence search can only choose *between* those candidates. Punctuation, digits and kept-as-Latin tokens pass through untouched. Each result carries `guard: {ok, words, outside}`, and the demo re-checks it on every keystroke.
>
> Across every evaluation set — **132,299 output words — 0 fell outside their slot's candidate list.**
That property is what makes this safe to ship: the sentence LM cannot hallucinate a spelling, drop a word, or reorder the sentence. Worst case, it picks a different candidate from the same short list the word model already offered.
## Size
| Part | On disk | gzip -9 |
|---|---|---|
| Word model package (v2: `xlit.q4` + `params.json` + dictionary pack + runtime) | 16,259,506 B | **7,919,816 B (7.92 MB)** |
| `nlm.q8` — sentence LM, 2,770,752 parameters, tokenizer embedded | 3,239,640 B | 2,921,848 B |
| `nlm.mjs` — LM loader + inference | 8,430 B | 3,132 B |
| `v3.mjs` — lattice beam search | 4,942 B | 2,115 B |
| **Add-on total** | **3,253,012 B** | **2,927,095 B (2.93 MB)** |
| **Complete sentence package** | **19,512,518 B** | **10,846,911 B (10.85 MB)** |
| *Same system with the 4-bit LM* (`nlm.q4`) | 1,862,360 B | 1,537,849 B → **9.46 MB complete** |
So: **10.85 MB complete with the 8-bit LM**, of which the word-only model is **7.92 MB**; **9.46 MB** if you take the 4-bit LM. (`nlm.q4e8` sits between the two at 2,184,860 B gz.) The 4-bit LM is within tuning noise and saves 1.39 MB: on the same tuning row (`w_nlm 0.3`, `margin 4`) it scores 89.45 val / 93.73 devA / 90.01 val2 against 89.44 / 93.82 / 89.95 for the 8-bit one, i.e. everything inside ±0.1 pt. The 8-bit LM ships because it is the file the final one-shot evaluation was run with. Every byte count above was measured with `du -b` and `gzip -9`.
## Why the LM is this big (and not bigger)
The shipped LM is the Kaggle **S** model: 2,770,752 parameters, 21k steps, valid cross-entropy **4.188 nats/token** (4.758 nats/word). A larger **M** model (3,766,080 parameters, 4.063 nats/token, 4.648 nats/word) was trained and evaluated too, and **it was no better** at the sentence task while being 1.3× bigger — so S ships. The LM is deliberately only asked to *rank about six spellings per word*, not to generate text, which is why 2.77M parameters are enough.
- **Data:** IndicCorp v2 Hindi, **CC0**. No scraped chat logs, no non-commercial corpora.
- **Tokenizer:** our own 8,000-piece unigram set, Viterbi segmentation reimplemented so Python and JavaScript tokenize identically.
- **Model:** 2-layer LSTMP (512 hidden, 160-dim embedding, tied input/output embedding, dropout 0.05), 8-bit quantised to 3.1 MB.
- **Scoring:** trained with a self-normalisation penalty, so the runtime can skip the softmax normaliser (`norm: "self"`) — that is the fast setting used here (95 ms/sentence in JS vs 193 ms for exact normalisation).
- **Dependency-free inference:** `nlm.mjs` is plain JavaScript on `Float32Array`, no ONNX, no WASM, no GPU.
## Speed
| Path | Per sentence | Notes |
|---|---|---|
| Python reference, search only | **13.6 ms** | lattice given; on a shared box |
| JavaScript (Node), margin 2, norm self — **the shipped setting** | **95 ms** | 200 sentences, box shared with other jobs |
| JavaScript (Node), margin 2, exact normalisation | 193 ms | 2× slower, ~+0.1 pt on tuning |
| JavaScript (Node), margin 4, norm self | 406 ms | 4.3× slower for −0.02 val / −0.10 devA tokens |
These JavaScript numbers are deliberately pessimistic: they were measured on a shared machine while other research jobs were running, one sentence at a time, cold caches. A quiet machine, a warm candidate cache and a shorter sentence are all faster. Browser latency in the demo is shown live per sentence, in your tab, on your hardware — trust that number over this table.
## Usage
### Browser (ES modules, no build step)
Download the files and serve them yourself — **the Hub returns `.mjs` as `text/plain`**, so a cross-origin
`import` straight from `huggingface.co/.../resolve/main/` is rejected by the browser's MIME check. Any static
server that sends `text/javascript` for `.mjs` works (`python3 -m http.server` in the folder below does).
```bash
huggingface-cli download sourojitdotcom/AksharaLP-Sentence-2.8M --local-dir aklp-sent
cd aklp-sent && python3 -m http.server 8000 # then open http://localhost:8000/
```
```html
```
`sent.mjs` is six dozen lines: it builds the v3 lattice from the word model's candidates and calls
`search3(lat, p, coreLm, nlm)` from `v3.mjs`. Read it before you change it — the guardrail lives there.
Run it in a Web Worker, as the demo does, so the page stays at 60 fps. If you only want to try it in a
browser, use the [demo Space](https://huggingface.co/spaces/sourojitdotcom/AksharaLP-Sentence-demo) — it is
this exact bundle, already served with the right MIME types.
### Node.js (≥ 18)
```bash
huggingface-cli download sourojitdotcom/AksharaLP-Sentence-2.8M --local-dir aksharalp-sent
```
```js
import fs from "node:fs";
import { loadFromDir } from "./aksharalp-sent/aksharalp.mjs";
import { attachSentence } from "./aksharalp-sent/sent.mjs";
const aklp = await loadFromDir("./aksharalp-sent/");
attachSentence(aklp, fs.readFileSync("./aksharalp-sent/nlm.q8"));
console.log(aklp.transliterate("kal milte hai").output); // कल मिलते हैं
```
### Python (reference implementation)
```python
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("sourojitdotcom/AksharaLP-Sentence-2.8M")
sys.path.insert(0, f"{path}/python") # needs numpy + torch (CPU)
from bhc.translit3 import BhcTranslit3
from bhc.nlm import NLM
nlm = NLM.load(f"{path}/nlm.q8", norm="self")
xl = BhcTranslit3(path, nlm=nlm, w_nlm=0.4, w_lm=0.25, w_ch_s=1.5, w_agn=2, w_agr=0,
nlm_margin=2, nlm_k=6, beam_s=8, nlm_norm="self")
print(xl.convert("mai ghr pe hu")) # मैं घर पे हूं
```
The Python package is the reference used for training and evaluation; the JavaScript runtime reproduces its decisions exactly (verified sentence by sentence on the casual set and on the evaluation sets: identical v2 and v3 outputs, identical accuracies).
### API at a glance
| Call | Returns |
|---|---|
| `attachSentence(engine, nlmBytes)` | attaches sentence mode; returns `{nlm, info}` where `info` has the decoding params, the parameter count (2,770,752) and the LM config |
| `engine.sentenceMode` | `true` (sentence) or `false` (word mode) — toggles the decoder without reloading anything |
| `engine.transliterate(text, {mode, context, k, overrides})` | same result object as the word model, plus `guard: {ok, words, outside}` |
| `search3(lattice, params, coreLm, nlm)` | the beam search itself, if you build your own lattice (format documented at the top of `v3.mjs`) |
## Files
| File | What it is | Size (gz) |
|---|---|---|
| `nlm.q8` | **the sentence LM** — 2,770,752 params, 8-bit, tokenizer embedded | 2.92 MB |
| `v3.mjs` | lattice beam search (`search3`) | 2.1 KB |
| `nlm.mjs` | LM file reader + inference | 3.1 KB |
| `sent.mjs` | glue: builds the lattice from the word model, toggles word/sentence mode, audits the guardrail | 1.8 KB |
| `xlit.q4`, `params.json`, `dict/` | the **v2 word model** this builds on (unchanged) | 7.92 MB total |
| `aksharalp.mjs`, `aksharalp.min.mjs` | the word-model runtime | 20 KB |
| `manifest.json` | file list, sizes, checksums | — |
| `python/` | reference Python implementation (numpy + PyTorch) | — |
| `cases/` | the 21-sentence casual set used for the tables above | — |
| `assets/` | banner, charts, screenshots (`assets/screenshots/`) | — |
| `LICENSE`, `NOTICE` | Apache-2.0, third-party attributions | — |
Decode settings (all fixed before the test run): `w_nlm 0.4`, `w_lm 0.25`, `w_ch_s 1.5`, `w_agn 2`, `w_agr 0`, `nlm_margin 2`, `nlm_k 6`, `beam_s 8`, `lat_k 12`, `rec_n 3`, `nlm_norm self`.
## Limitations
- **Hindi only.** One script pair: romanized Hindi → Devanagari. No other Indic language, no reverse direction.
- **It cannot add a candidate it was not given.** If the word model's top-6 misses the right spelling (see `Boht` → बोहत), no amount of sentence context finds it. That is a channel/recall problem and it is the single biggest remaining error source.
- **Two-word inputs gain nothing.** Short fragments with no verb or auxiliary have no sentence signal (`nahi pata` → नहीं पाता). Add the subject and it works (`mujhe nahi pata` → मुझे नहीं पता).
- **Chat normalisation cuts both ways.** The LM sometimes rewrites deliberate chat spellings into dictionary forms (extra *h*, doubled letters, stretched vowels): see the regressions table.
- **Casual-chat gains are not significant.** On our 480-sentence casual set the sentence-accuracy gain is +1.46 points with a 95% CI of [−0.42, 3.54]. The significant wins are on the held-out Dakshina sentence test (sentence accuracy and word accuracy) and on the context set in word accuracy (+2.13 points, CI [0.42, 4.64]); the context set has only 66 sentences, so its sentence-level +4.55 has a CI of [0.00, 10.61] and is not flagged.
- **Names, rare words and code-mixing** remain the weak spot, the same as the word model: proper nouns, brand-new slang, heavy English mixing.
- **Speed is CPU JavaScript.** 95 ms per sentence at the shipped setting on a shared box; a long sentence, a cold cache or an old phone is slower. It is not a keystroke-latency budget at full sentence length — the demo debounces.
- **One decoding setting.** These weights were tuned for `nlm_norm: "self"`. Changing to `exact` normalisation needs a re-tune to recover the 0.1–0.3 points it is worth on dev data.
- **Not a translator.** It changes the script, not the language.
## How it fits the family
| Model | What it is | Size | Best at |
|---|---|---|---|
| **[AksharaLP-1M](https://huggingface.co/sourojitdotcom/AksharaLP-1M)** | the first tiny word model, core only | 4.53 MB | smallest footprint, standalone word conversion |
| **[AksharaLP-1M-v2](https://huggingface.co/sourojitdotcom/AksharaLP-1M-v2)** | the word model + dictionary pack this repo builds on | 7.92 MB | **word-mode accuracy** — 70.04 top-1 / 88.31 top-3 / 93.63 top-8 on Aksharantar Hindi test; best top-3 and top-8 in the family and best on chat spellings (v1 edges it on top-1 by 0.33 pts, CIs overlap) |
| **[AksharaLP-Kosh-1.2M](https://huggingface.co/sourojitdotcom/AksharaLP-Kosh-1.2M)** | the sub-megabyte word model (its own card is CC-BY-SA-4.0) | 0.78 MB | the smallest possible download at 68.09% top-1 |
| **AksharaLP-Sentence-2.8M** (this) | v2 word model + a 2.77M neural sentence LM | 10.85 MB | **sentence accuracy** — 26.73% sentences exact and 89.44% words on the Dakshina sentence test, the best in the family |
Kosh is the word model for when one megabyte is the whole budget — 0.78 MB, 68.09% top-1, with its own licence. This is the sentence-accurate one: same word-mode candidates, plus a language model that reads the sentence. **v2 remains the best word-mode accuracy in the family on top-3/top-8 and on casual chat** (AksharaLP-1M is 0.33 points ahead on top-1 alone, inside overlapping intervals) — this repo does not claim otherwise, and does not change word mode at all.
## Licence & attribution
**Commercial use is allowed.** Code and model weights ship under the **Apache License 2.0** (see `LICENSE`), Copyright 2026 Sourojit Dhua.
**The sentence LM is commercially clean, and that is the important part.** It was trained only on **IndicCorp v2 Hindi (CC0)** with our own tokenizer, our own training code and our own runtime. No non-commercial corpus, no scraped chat data, no third-party model weights. The full system still inherits the v2 word package's terms (Apache-2.0, with the data attributions below listed in `NOTICE`).
**Attribution is required when you redistribute it.** Apache-2.0 §4(d): include `NOTICE` in every redistribution or derivative, and keep the runtime's `/*! … */` header (including in `aksharalp.min.mjs`).
```text
Transliteration by AksharaLP (Sourojit Dhua) — https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M
```
| Source | Licence | Used for |
|---|---|---|
| [IndicCorp v2](https://github.com/AI4Bharat/IndicBERT/tree/main#indiccorp-v2), AI4Bharat | **CC0** | sentence LM training data; word-model lexicon and bigram LM |
| [Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar), AI4Bharat | CC0 (mined) / CC-BY (manually collected) | word-model training pairs |
| [IndicXlit](https://github.com/AI4Bharat/IndicXlit), AI4Bharat | MIT | teacher model for word-model distillation |
| [CMUdict](https://github.com/cmusphinx/cmudict), Carnegie Mellon University | BSD-2-Clause-style | English word cues in the word model |
Fonts in the demo Space (Inter, Noto Sans Devanagari) are under the SIL Open Font License 1.1.
## Citation
```bibtex
@software{dhua_aksharalp_sentence_2026,
author = {Dhua, Sourojit},
title = {{AksharaLP-Sentence}: Tiny, Sentence-Level Hindi Transliteration for the Browser},
year = {2026},
version = {3.0.0},
url = {https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M},
note = {Word model: AksharaLP-1M-v2 (989K parameters). Sentence LM: 2.77M parameters, trained on IndicCorp v2 Hindi (CC0).},
license = {Apache-2.0}
}
```
AksharaLP · sentence model v3.0.0 · built 2026-11-04 · Questions and feedback: open a discussion on this repository.