README.md

metadata

license: cc-by-nc-4.0
language:
  - hi
  - bn
  - te
  - kn
  - ta
tags:
  - transliteration
  - indic
  - multilingual
  - tiny
  - browser
  - javascript
  - offline
  - input-method
  - gguf
  - custom-architecture
datasets:
  - ai4bharat/Aksharantar
  - ai4bharat/IndicCorpV2
metrics:
  - accuracy

Try the browser demo ↗ · Measured benchmarks · 13 GGUF files · Attribution required · Licence & data

AksharaLP-MultiIndic-8M

Five word transliterators. One optional Hindi sentence model. Your device.

Hindi · Bengali · Telugu · Kannada · Tamil — familiar Roman letters in, native script out. Five independent ~989K-parameter character Transformers, a lazily loaded dictionary per language, and the already-shipped 2,770,752-parameter Hindi sentence LM. No inference server and no GPU required for inference. This is transliteration, not translation or a general-purpose language model.

Attribution required. Non-commercial (CC-BY-NC-4.0). Any use — including research, demos, derivatives, and redistribution — must credit Sourojit Dhua (sourojitdotcom) and link to this model page. CC-BY-NC-4.0 requires attribution and forbids commercial use. The new Bengali/Telugu/Kannada/Tamil weights include Samanantar-derived Aksharantar rows; the upstream Samanantar card states CC-BY-NC-4.0, so this release does not certify those weights as commercially clean. The Hindi word and sentence components remain independently available under Apache-2.0 in their original repositories. See Attribution required and Licence & data.

4.94M five word models combined

7.72M with the optional Hindi LM

20.55 MB all word payloads, gzip-9

23.48 MB including Hindi sentence mode

The 8M name rounds 7,715,520 neural parameters, not a billion-parameter model. Dictionaries and small candidate-reranker coefficients are separate from that count. Loading one language does not load the other four. Browser inference uses float32 arithmetic after loading low-bit weights: small files do not mean equally small RAM use.

What this release does — and does not prove

• Every word Transformer is 988,928 parameters, except Bengali (989,056, with one additional output character). The five sum to 4,944,768; optional Hindi LM adds 2,770,752.
• New word weights are about 0.54 MB gzip-9 per language; model-plus-dictionary payloads are about 3.05–3.23 MB each. Hindi keeps its larger, established 3M-word dictionary pack.
• IndicXlit has 11,487,744 neural parameters in the comparison port. Its per-language original dictionaries are much larger; the exact byte comparison below uses their released JSON formats, not an optimized repacking.
• We trail IndicXlit's full package in every new language on top-1, top-3 and top-8. This release is a storage/accuracy trade-off, not a state-of-the-art or “best in class” claim.
• Sentence-level neural rescoring is Hindi-only. Bengali, Telugu, Kannada and Tamil convert word by word. The separate Bengali real-chat research front-end is not part of this bundle.

Benchmarks

Aksharantar test, exact match, any accepted reference. Top-1 %, 95% bootstrap intervals in brackets; paired differences are percentage points, ours minus IndicXlit.

Language	Roman words	Ours + dictionary	IndicXlit full package	Paired gap [95% CI]
Hindi	9,991	70.04 [69.16, 70.90]	69.33 [68.44, 70.23]	+0.71 [-0.04, +1.40]
Bengali	13,367	64.64 [63.83, 65.44]	70.52 [69.76, 71.34]	-5.88 [-6.52, -5.22]
Telugu	9,936	78.74 [77.93, 79.53]	82.42 [81.65, 83.14]	-3.67 [-4.27, -3.07]
Kannada	11,165	77.37 [76.61, 78.13]	82.61 [81.92, 83.29]	-5.24 [-5.83, -4.67]
Tamil	11,405	71.78 [71.01, 72.61]	75.95 [75.20, 76.73]	-4.16 [-4.75, -3.59]

Language	Ours top-1 / top-3 / top-8 %	IndicXlit full top-1 / top-3 / top-8 %
Hindi	70.04 / 88.31 / 93.63	69.33 / 85.40 / 88.29
Bengali	64.64 / 75.66 / 77.97	70.52 / 82.25 / 85.44
Telugu	78.74 / 86.87 / 88.63	82.42 / 90.63 / 92.75
Kannada	77.37 / 85.80 / 87.68	82.61 / 90.68 / 92.41
Tamil	71.78 / 80.05 / 82.61	75.95 / 84.21 / 87.26

Protocol and caveats

1. For Bengali/Telugu/Kannada/Tamil, the shipped q4 file, top-500k IndicCorp frequency list and eight-feature linear reranker were frozen before test. The reranker was fitted on all VALID, not on test, using the same fixed feature design and L2 regularization selected during validation work. Quantized VALID candidates were regenerated before that fit. There was one inference pass on the full test per frozen artifact, with cached predictions scored once; no post-test tuning.
2. NFC exact match, ZWJ/ZWNJ removed, any reference accepted. Top-k means at least one reference among the first k candidates, not “k times more accurate.” Intervals use 2,000 bootstrap resamples, seed 0; paired intervals resample the same words together. Full top-k intervals, hashes and frozen configurations are in evaluation/.
3. IndicXlit scores reuse existing beam-8 PyTorch-port predictions with the full official frequency dictionary formula, rescored on exactly the same rows. This is not a fresh run of the official fairseq package. The port masks output to each language's script block. Bengali has one empty Roman input skipped by that runner; its missing prediction is counted incorrect and the row is not dropped.
4. Hindi numbers are copied from the measured AksharaLP-1M-v2 release, not re-tested. Its candidate generation/reranker and 3M-word dictionary differ from the four new models. The old Hindi full-package comparison interval includes zero; it does not establish a significant top-1 win.
5. Training reports record zero valid/test native-word and Roman-form overlap for all four new training files. The general IndicCorp frequency dictionaries are not test-filtered and can naturally contain test reference words. Distillation teachers may have their own historical overlap. This is not an unseen-vocabulary-only benchmark, and Aksharantar words are not a WhatsApp sentence benchmark.

Hindi sentence mode — the shipped add-on, unchanged

The AksharaLP-Sentence-2.8M LM chooses jointly among candidates already proposed for each typed Hindi word. It cannot translate, invent a missing candidate, reorder words, or answer questions. The runtime exposes the existing candidate-membership guard.

Dakshina Hindi sentences, 4,976 test items	Existing Hindi v2 word package + bigram context	With the neural sentence LM
Word accuracy %	89.02	89.44
Entire sentences exactly right %	25.70	26.73
Character error rate %	4.07	3.93
Word error rate %	11.20	10.78

These are the existing one-time published results, not new evaluations. The 89.02/25.70 baseline already uses the older Hindi bigram context decoder; it is not the context-off demo toggle. Paired reported gain: word accuracy +0.42 [0.33, 0.51] points, sentence exact match +1.02 [0.40, 1.65] points (the difference of rounded sentence percentages is 1.03). This release retains the default q8 LM and original settings. No sentence claims are made for the other four languages.

Footprint & latency

Language payload	Canonical raw bytes	Sum gzip-9 bytes	JS CPU p50 / p95 ms per word
Hindi	16,221,771	7,905,041 (7.91 MB)	207.02 / 319.99
Bengali	14,658,574	3,049,313 (3.05 MB)	51.61 / 74.00
Telugu	16,579,274	3,222,365 (3.22 MB)	51.77 / 87.87
Kannada	17,376,245	3,224,222 (3.22 MB)	55.35 / 100.77
Tamil	18,695,389	3,130,045 (3.13 MB)	60.98 / 103.39
Shared word JS runtime + root configuration	68,781	23,186	—
All five word packages + shared runtime	83,600,034	20,554,172 (20.55 MB)	—
Optional Hindi q8 sentence LM + JS add-on	3,256,675	2,928,839 (2.93 MB)	Not re-timed
All languages + Hindi sentence add-on	86,856,709	23,483,011 (23.48 MB)	—

MB = 1,000,000 bytes. Canonical raw dictionary size means decompressed frequency TSV for new languages, and native lexicon/bigram binaries for Hindi. Browser downloads use gzip companions where provided. gzip-9 values are sums of per-file compression, not actual RAM, not total repository storage, and not a promise about HTTP Content-Encoding. Charts/docs/fonts/Python/GGUF alternatives and duplicated compressed companions are excluded. All exact component sizes are in sizes.json.

Latency: 100 evenly spaced VALID words per language, five warmups, cleared candidate memo per word, single JavaScript thread in Node v24.22.1 on Intel(R) Xeon(R) Processor, Linux; initialization excluded. This is Node CPU latency, not a browser or phone speed claim. Loading the large Hindi dictionary is substantially more expensive. Browser demo inference happens in a Web Worker and reports live timings. The 15 initial example word candidate lists matched Python exactly; a broader independent GGUF validation sample is below.

IndicXlit size comparison: 45,950,976 bytes of FP32 neural parameters, plus original dictionary JSON raw bytes: Hindi 358,118,856; Bengali 403,487,845; Telugu 761,940,466; Kannada 1,045,504,407; Tamil 1,044,620,827. These compare different precisions, vocabularies and data encodings; IndicXlit was not requantized or repacked. They do not prove that ours is the smallest possible system.

Real outputs

Generated by the bundled JavaScript runtime, not hand-written target strings. These are fixed illustrative examples, not an accuracy suite or proof of sentence understanding. Multiword inputs outside Hindi are independent word conversions. Some outputs are wrong: in particular, Tamil naan currently produces ணன், not the intended நான். We show it rather than hiding the failure.

Hindi

Roman input	Actual browser-runtime output
namaste	नमस्ते
kaise	कैसे
dhanyavaad	धन्यवाद
mera naam rahul hai	मेरा नाम राहुल है

Bengali

Roman input	Actual browser-runtime output
nomoskar	নমস্কার
ami	আমি
bhalo	ভালো
tumi kemon acho	তুমি কেমন আছো

Telugu

Roman input	Actual browser-runtime output
namaskaram	నమస్కారం
nenu	నేను
manchi	మంచి
meeru ela unnaru	మీరు ఎలా ఉన్నారు

Kannada

Roman input	Actual browser-runtime output
namaskara	ನಮಸ್ಕಾರ
nanu	ನನು
chennagide	ಚೆನ್ನಾಗಿದೆ
neevu hegiddira	ನೀವು ಹೇಗಿದ್ದೀರಾ

Tamil

Roman input	Actual browser-runtime output
vanakkam	வணக்கம்
naan	ணன்
nandri	நன்றி
neenga eppadi irukeenga	நீங்க எப்படி இருகீங்க

Hindi sentence examples

Input	Actual output with the optional sentence LM
main theek hoon	मैं ठीक हूं
ghar main kaun hai	घर में कौन है

JavaScript — browser, worker, or Node

Copy multiindic.mjs, aksharalp.mjs, the models/ tree and (for Hindi sentence mode) nlm.mjs, v3.mjs, sent.mjs. Serve over HTTPS or localhost; ES modules do not reliably work from file://.

import { MultiIndic } from './multiindic.mjs';

// Loads Bengali's weights + dictionary, not every language.
const engine = await MultiIndic.load('./', { language: 'bn' });
console.log(engine.word('nomoskar', 8));       // ranked {text, score} objects
console.log(engine.transliterate('tumi kemon acho').output);

await engine.setLanguage('hi');
await engine.setSentence(true);               // lazily loads the q8 Hindi LM
console.log(engine.transliterate('main theek hoon').output);
await engine.setSentence(false);              // word-by-word, context off

// Node >= 18: no dependencies, uses local files.
import {loadFromDir} from './multiindic.mjs';
const engine = await loadFromDir('./', {language: 'ta'});
console.log(engine.word('nandri', 3));

Browser gzip decoding requires DecompressionStream support; Node uses its built-in zlib. Switching languages retains already loaded models in memory for reuse. Recreate the engine to release that cache. The demo preserves URLs, email addresses, @mentions, #tags, digits and already-native text. Hindi retains the plain reference convention of changing a final full stop to danda. Roman English words are otherwise treated as transliteration input; there is no multilingual code-switch detector.

Python reference

Requires numpy and torch. The repository is public; no Hugging Face token is required to download it. Never put a write token in browser code.

from pathlib import Path
import sys
from huggingface_hub import snapshot_download

root = Path(snapshot_download('sourojitdotcom/AksharaLP-MultiIndic-8M'))
sys.path.insert(0, str(root / 'python'))
from multiindic import MultiIndic

engine = MultiIndic(root, language='te', threads=1)
print(engine.word('namaskaram', k=8))
engine.set_language('hi').set_sentence(True)
print(engine.transliterate('main theek hoon'))

The Python reference returns candidate strings from word(); JavaScript additionally exposes uncalibrated ranking scores. Those scores are not probabilities. Hindi uses the plain reference token conversion in both runtimes, not the older demo's extra brand/acronym heuristics. The model/decoder parity samples are measured separately.

GGUF containers

These are custom-architecture GGUF v3 tensor containers. They do NOT run in llama.cpp, Ollama, or LM Studio. Use the bundled gguf_read.py plus our decoder. A GGUF is not a standalone dictionary or language-model runtime.

The writer/reader follows the existing v2 export. Word q4 GGUF carries the byte-exact shipped mixed q4/q8/f16 blob in an I8 tensor; it is not standard GGML Q4_0. F16 materializes those already-quantized weights as half floats; it does not restore an original full-precision checkpoint. Exact reranker configuration is retained as JSON metadata, and the external dictionary/runtime remain required.

For Hindi sentence mode, q8 GGUF preserves the test-measured default byte-exactly. An existing q4 LM and F16 materialization are included as alternatives with validation parity only. Do not attach the default q8 test accuracies to q4 or F16 alternatives. The q4 LM is not the demo default.

GGUF file	Raw bytes	gzip-9 bytes
aksharalp-hi-word-q4.gguf	632,416	562,579
aksharalp-hi-word-f16.gguf	2,026,048	1,348,288
aksharalp-bn-word-q4.gguf	591,136	541,220
aksharalp-bn-word-f16.gguf	1,984,864	1,321,177
aksharalp-te-word-q4.gguf	591,008	542,744
aksharalp-te-word-f16.gguf	1,984,608	1,329,379
aksharalp-kn-word-q4.gguf	591,008	540,642
aksharalp-kn-word-f16.gguf	1,984,608	1,319,622
aksharalp-ta-word-q4.gguf	591,008	542,094
aksharalp-ta-word-f16.gguf	1,984,608	1,328,348
aksharalp-hi-sentence-q8.gguf	3,488,192	2,981,426
aksharalp-hi-sentence-q4.gguf	2,110,912	1,596,808
aksharalp-hi-sentence-f16.gguf	5,790,336	5,159,347

Validation-only parity — not a second test evaluation

First 40 ASCII-alphabetic Roman words per language from full VALID (200 total). Same dictionary and reranker; no fitting or tuning during parity. “Top-8 order” compares the complete ordered candidate list.

Language	q4 top-1 identical	q4 top-8 order identical	F16 top-1 identical	F16 top-8 order identical
Hindi	40/40	40/40	40/40	40/40
Bengali	40/40	40/40	40/40	40/40
Telugu	40/40	40/40	40/40	40/40
Kannada	40/40	40/40	40/40	40/40
Tamil	40/40	40/40	40/40	40/40

Hindi LM: first 12 Dakshina VALID sentences with at most 12 Roman words, fixed word-model lattices. q8 reproduces 12/12 source decoder outputs; F16 12/12; q4 11/12. Maximum absolute sequence-score deltas vs q8 are 0.000000, 0.008702, and 6.343046, respectively. This is conversion agreement, not accuracy. Across the same validation sample, JavaScript and Python matched 200/200 top-1 and complete top-8 ordered lists, and 12/12 Hindi sentence outputs, after using the same plain-reference token policy. Both our structural reader and independent gguf-py read all 13 files; complete data is in gguf_parity.json.

# After the Python setup above:
engine.set_language('bn')
engine.load_word_gguf(root / 'gguf/aksharalp-bn-word-q4.gguf')
print(engine.word('nomoskar'))
engine.set_language('hi')
engine.load_sentence_gguf(root / 'gguf/aksharalp-hi-sentence-q8.gguf')
print(engine.transliterate('main theek hoon'))

Attribution required

Any use of this model or demo — including research, demos, derivatives, and redistribution — must credit Sourojit Dhua (sourojitdotcom) and link to this model page: https://huggingface.co/sourojitdotcom/AksharaLP-MultiIndic-8M.

This release is licensed under CC-BY-NC-4.0, which requires attribution and forbids commercial use.

Ready-to-copy credit line:

AksharaLP-MultiIndic-8M by Sourojit Dhua (sourojitdotcom) — https://huggingface.co/sourojitdotcom/AksharaLP-MultiIndic-8M

BibTeX:

@misc{dhua_aksharalp_multiindic_8m_2026,
  author = {Sourojit Dhua},
  title  = {AksharaLP-MultiIndic-8M},
  year   = {2026},
  url    = {https://huggingface.co/sourojitdotcom/AksharaLP-MultiIndic-8M}
}

The same text is in NOTICE and ATTRIBUTION.

Licence & data

Model-card licence: CC-BY-NC-4.0. This release is non-commercial, and attribution is required. The Bengali, Telugu, Kannada and Tamil weights contain aks-Samanantar Aksharantar training rows. The current upstream Samanantar card states CC-BY-NC-4.0. Aksharantar and the publisher's Samanantar packaging page describe CC0 packaging, so provenance terms conflict; this release does not certify those weights as commercially clean.

Resource	Terms / role
Aksharantar, AI4Bharat	CC0 mined/existing packaging; CC-BY manual data. New training includes Samanantar-derived rows: Bengali 114,558, Telugu 42,413, Kannada 36,482, Tamil 34,447; per-language training reports are bundled.
Samanantar, AI4Bharat	CC-BY-NC-4.0 upstream card; reason for this release's non-commercial restriction.
Dakshina, Google Research	CC-BY-SA-4.0 upstream. Source-attributed pairs appear within Aksharantar; original terms remain, and derivative-rights questions are not resolved by selecting a more restrictive model-card licence. Cite Roark et al., LREC 2020.
IndicCorp v2, AI4Bharat	CC0; frequency dictionaries and Hindi LM training.
IndicXlit, AI4Bharat	MIT teacher for distillation, and comparison baseline; original checkpoint not redistributed.
Runtime code / original Hindi components	Apache-2.0, retained with their existing notices. Inter and Noto Indic demo fonts: SIL OFL 1.1.

The Hindi components are also separately available under Apache-2.0: AksharaLP-1M-v2 and AksharaLP-Sentence-2.8M. This combined release does not revoke or replace those existing grants. Code, dictionaries and individual components retain their own upstream terms; see NOTICE, LICENSE and LICENSE-APACHE-2.0. This documentation is engineering provenance, not legal advice.

FAQ

Is this a multilingual LLM? No. It cannot answer questions, reason generally, summarize, or translate between languages. It changes Romanized spellings into native script.

Why 8M when a word model has 989K parameters? Five independent word models total 4,944,768; optional Hindi LM adds 2,770,752; total 7,715,520, rounded to 8M. That count excludes small reranker coefficients and dictionary data. Only selected components load.

Is it the best model for these languages? Not established. It trails IndicXlit's full package on all four new test languages. We claim measured footprint and measured results, not smallest-ever or best-in-class status.

Does every language understand a sentence? No. Only Hindi has neural sentence rescoring. Others concatenate independent word predictions, preserving text layout. Hindi rescoring cannot recover a spelling outside its candidate list.

Can I use it commercially? No. This combined release is CC-BY-NC-4.0: commercial use is not allowed, and any use must credit Sourojit Dhua (sourojitdotcom) with a link to this model page. Independently Apache-licensed Hindi releases remain available for Hindi-only use where their terms apply. Retain all source attributions and review applicable component terms.

Does text leave the device? The demo fetches local model assets from the Space when loading a language. Typed text is processed in the browser and is not sent to an inference API or telemetry service. Once the selected assets are loaded, typing needs no network. This is not a promise to avoid the initial asset download.

Can I run the GGUFs with llama.cpp? No. GGUF is a container here, not a supported llama.cpp architecture. Use our reader, dictionaries and decoding code.

Limitations & responsible use

Names, ambiguous short words, unseen vocabulary, English code-switching, dialects and chat abbreviations can fail. The output must be user-reviewable; do not silently replace important legal, medical or financial text. Candidate suggestions are uncalibrated. Input encoding truncates a word after 31 Roman characters. Type-as-you-go inference is debounced; older phones and long sentences may be slow. All measured compute here is CPU, not an on-device mobile benchmark. Tiny low-bit downloads still expand to float32 arrays and large dictionary structures in RAM.

This release records the current artifacts. No “best in class,” general-LLM capability, sub-5-MB complete bundle, commercial-clean, or multilingual-sentence claim is made.

---

AksharaLP · Sourojit Dhua (sourojitdotcom) · 2026-11-10 (IST) · CC-BY-NC-4.0. Attribution required. Non-commercial use only.