Try the private browser demo ↗ · Measured benchmarks · 13 GGUF files · Licence & data
+ +# AksharaLP-MultiIndic-8M + +### Five word transliterators. One optional Hindi sentence model. Your device. + +**Hindi · Bengali · Telugu · Kannada · Tamil** — familiar Roman letters in, native script out. Five independent ~989K-parameter character Transformers, a lazily loaded dictionary per language, and the already-shipped **2,770,752-parameter Hindi sentence LM**. No inference server and no GPU required for inference. This is **transliteration, not translation or a general-purpose language model**. + +> **Private review · non-commercial release.** This bundle's model-card licence is **CC-BY-NC-4.0**, not Apache-2.0. The new Bengali/Telugu/Kannada/Tamil weights include Samanantar-derived Aksharantar rows with non-commercial upstream terms. A commercially clean Apache-2.0 retrain is planned. The Hindi word and sentence components remain independently available under Apache-2.0 in their original repositories. See [Licence & data](#licence--data). + +4.94Mfive word models combined | 7.72Mwith the optional Hindi LM | 20.55 MBall word payloads, gzip-9 | 23.48 MBincluding Hindi sentence mode |
+
+**Aksharantar test, exact match, any accepted reference.** Top-1 %, 95% bootstrap intervals in brackets; paired differences are percentage points, ours minus IndicXlit.
+
+| Language | Roman words | Ours + dictionary | IndicXlit full package | Paired gap [95% CI] |
+|---|---:|---:|---:|---:|
+| Hindi | 9,991 | 70.04 [69.16, 70.90] | 69.33 [68.44, 70.23] | +0.71 [-0.04, +1.40] |
+| Bengali | 13,367 | 64.64 [63.83, 65.44] | 70.52 [69.76, 71.34] | -5.88 [-6.52, -5.22] |
+| Telugu | 9,936 | 78.74 [77.93, 79.53] | 82.42 [81.65, 83.14] | -3.67 [-4.27, -3.07] |
+| Kannada | 11,165 | 77.37 [76.61, 78.13] | 82.61 [81.92, 83.29] | -5.24 [-5.83, -4.67] |
+| Tamil | 11,405 | 71.78 [71.01, 72.61] | 75.95 [75.20, 76.73] | -4.16 [-4.75, -3.59] |
+
+| Language | Ours top-1 / top-3 / top-8 % | IndicXlit full top-1 / top-3 / top-8 % |
+|---|---:|---:|
+| Hindi | 70.04 / 88.31 / 93.63 | 69.33 / 85.40 / 88.29 |
+| Bengali | 64.64 / 75.66 / 77.97 | 70.52 / 82.25 / 85.44 |
+| Telugu | 78.74 / 86.87 / 88.63 | 82.42 / 90.63 / 92.75 |
+| Kannada | 77.37 / 85.80 / 87.68 | 82.61 / 90.68 / 92.41 |
+| Tamil | 71.78 / 80.05 / 82.61 | 75.95 / 84.21 / 87.26 |
+
+**Protocol and caveats**
+
+1. For Bengali/Telugu/Kannada/Tamil, the shipped q4 file, top-500k IndicCorp frequency list and eight-feature linear reranker were frozen before test. The reranker was fitted on **all VALID**, not on test, using the same fixed feature design and L2 regularization selected during validation work. Quantized VALID candidates were regenerated before that fit. There was **one inference pass on the full test per frozen artifact**, with cached predictions scored once; no post-test tuning.
+2. NFC exact match, ZWJ/ZWNJ removed, any reference accepted. Top-k means at least one reference among the first k candidates, not “k times more accurate.” Intervals use **2,000 bootstrap resamples, seed 0**; paired intervals resample the same words together. Full top-k intervals, hashes and frozen configurations are in [evaluation/](evaluation/).
+3. IndicXlit scores reuse existing **beam-8 PyTorch-port predictions with the full official frequency dictionary formula**, rescored on exactly the same rows. This is not a fresh run of the official fairseq package. The port masks output to each language's script block. Bengali has one empty Roman input skipped by that runner; its missing prediction is counted incorrect and the row is not dropped.
+4. Hindi numbers are copied from the measured [AksharaLP-1M-v2 release](https://huggingface.co/sourojitdotcom/AksharaLP-1M-v2), **not re-tested**. Its candidate generation/reranker and 3M-word dictionary differ from the four new models. The old Hindi full-package comparison interval includes zero; it does not establish a significant top-1 win.
+5. Training reports record **zero valid/test native-word and Roman-form overlap** for all four new training files. The general IndicCorp frequency dictionaries are not test-filtered and can naturally contain test reference words. Distillation teachers may have their own historical overlap. This is not an unseen-vocabulary-only benchmark, and Aksharantar words are not a WhatsApp sentence benchmark.
+
+## Hindi sentence mode — the shipped add-on, unchanged
+
+The [AksharaLP-Sentence-2.8M](https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M) LM chooses jointly among candidates already proposed for each typed Hindi word. It cannot translate, invent a missing candidate, reorder words, or answer questions. The runtime exposes the existing candidate-membership guard.
+
+| Dakshina Hindi sentences, 4,976 test items | Existing Hindi v2 word package + bigram context | With the neural sentence LM |
+|---|---:|---:|
+| Word accuracy % | 89.02 | 89.44 |
+| Entire sentences exactly right % | 25.70 | 26.73 |
+| Character error rate % | 4.07 | 3.93 |
+| Word error rate % | 11.20 | 10.78 |
+
+These are the **existing one-time published results, not new evaluations**. The 89.02/25.70 baseline already uses the older Hindi bigram context decoder; it is not the context-off demo toggle. Paired reported gain: word accuracy **+0.42 [0.33, 0.51]** points, sentence exact match **+1.02 [0.40, 1.65]** points (the difference of rounded sentence percentages is 1.03). This release retains the default **q8** LM and original settings. No sentence claims are made for the other four languages.
+
+## Footprint & latency
+
+
+
+| Language payload | Canonical raw bytes | Sum gzip-9 bytes | JS CPU p50 / p95 ms per word |
+|---|---:|---:|---:|
+| Hindi | 16,221,771 | 7,905,041 (7.91 MB) | 207.02 / 319.99 |
+| Bengali | 14,658,574 | 3,049,313 (3.05 MB) | 51.61 / 74.00 |
+| Telugu | 16,579,274 | 3,222,365 (3.22 MB) | 51.77 / 87.87 |
+| Kannada | 17,376,245 | 3,224,222 (3.22 MB) | 55.35 / 100.77 |
+| Tamil | 18,695,389 | 3,130,045 (3.13 MB) | 60.98 / 103.39 |
+| Shared word JS runtime + root configuration | 68,781 | 23,186 | — |
+| **All five word packages + shared runtime** | **83,600,034** | **20,554,172 (20.55 MB)** | — |
+| Optional Hindi q8 sentence LM + JS add-on | 3,256,675 | 2,928,839 (2.93 MB) | Not re-timed |
+| **All languages + Hindi sentence add-on** | **86,856,709** | **23,483,011 (23.48 MB)** | — |
+
+MB = 1,000,000 bytes. Canonical raw dictionary size means decompressed frequency TSV for new languages, and native lexicon/bigram binaries for Hindi. Browser downloads use gzip companions where provided. gzip-9 values are sums of per-file compression, not actual RAM, not total repository storage, and not a promise about HTTP Content-Encoding. Charts/docs/fonts/Python/GGUF alternatives and duplicated compressed companions are excluded. All exact component sizes are in [sizes.json](evaluation/sizes.json).
+
+Latency: **100 evenly spaced VALID words per language**, five warmups, cleared candidate memo per word, single JavaScript thread in **Node v24.22.1 on Intel(R) Xeon(R) Processor, Linux**; initialization excluded. This is **Node CPU latency, not a browser or phone speed claim**. Loading the large Hindi dictionary is substantially more expensive. Browser demo inference happens in a Web Worker and reports live timings. The 15 initial example word candidate lists matched Python exactly; a broader independent GGUF validation sample is below.
+
+IndicXlit size comparison: **45,950,976 bytes of FP32 neural parameters**, plus original dictionary JSON raw bytes: Hindi 358,118,856; Bengali 403,487,845; Telugu 761,940,466; Kannada 1,045,504,407; Tamil 1,044,620,827. These compare different precisions, vocabularies and data encodings; IndicXlit was not requantized or repacked. They do not prove that ours is the smallest possible system.
+
+## Real outputs
+
+Generated by the bundled JavaScript runtime, **not hand-written target strings**. These are fixed illustrative examples, not an accuracy suite or proof of sentence understanding. Multiword inputs outside Hindi are independent word conversions. Some outputs are wrong: in particular, Tamil `naan` currently produces **ணன்**, not the intended நான். We show it rather than hiding the failure.
+
+### Hindi
+
+| Roman input | Actual browser-runtime output |
+|---|---|
+| `namaste` | नमस्ते |
+| `kaise` | कैसे |
+| `dhanyavaad` | धन्यवाद |
+| `mera naam rahul hai` | मेरा नाम राहुल है |
+
+### Bengali
+
+| Roman input | Actual browser-runtime output |
+|---|---|
+| `nomoskar` | নমস্কার |
+| `ami` | আমি |
+| `bhalo` | ভালো |
+| `tumi kemon acho` | তুমি কেমন আছো |
+
+### Telugu
+
+| Roman input | Actual browser-runtime output |
+|---|---|
+| `namaskaram` | నమస్కారం |
+| `nenu` | నేను |
+| `manchi` | మంచి |
+| `meeru ela unnaru` | మీరు ఎలా ఉన్నారు |
+
+### Kannada
+
+| Roman input | Actual browser-runtime output |
+|---|---|
+| `namaskara` | ನಮಸ್ಕಾರ |
+| `nanu` | ನನು |
+| `chennagide` | ಚೆನ್ನಾಗಿದೆ |
+| `neevu hegiddira` | ನೀವು ಹೇಗಿದ್ದೀರಾ |
+
+### Tamil
+
+| Roman input | Actual browser-runtime output |
+|---|---|
+| `vanakkam` | வணக்கம் |
+| `naan` | ணன் |
+| `nandri` | நன்றி |
+| `neenga eppadi irukeenga` | நீங்க எப்படி இருகீங்க |
+
+### Hindi sentence examples
+
+| Input | Actual output with the optional sentence LM |
+|---|---|
+| `main theek hoon` | मैं ठीक हूं |
+| `ghar main kaun hai` | घर में कौन है |
+
+## JavaScript — browser, worker, or Node
+
+Copy `multiindic.mjs`, `aksharalp.mjs`, the `models/` tree and (for Hindi sentence mode) `nlm.mjs`, `v3.mjs`, `sent.mjs`. Serve over HTTPS or localhost; ES modules do not reliably work from `file://`.
+
+```js
+import { MultiIndic } from './multiindic.mjs';
+
+// Loads Bengali's weights + dictionary, not every language.
+const engine = await MultiIndic.load('./', { language: 'bn' });
+console.log(engine.word('nomoskar', 8)); // ranked {text, score} objects
+console.log(engine.transliterate('tumi kemon acho').output);
+
+await engine.setLanguage('hi');
+await engine.setSentence(true); // lazily loads the q8 Hindi LM
+console.log(engine.transliterate('main theek hoon').output);
+await engine.setSentence(false); // word-by-word, context off
+```
+
+```js
+// Node >= 18: no dependencies, uses local files.
+import {loadFromDir} from './multiindic.mjs';
+const engine = await loadFromDir('./', {language: 'ta'});
+console.log(engine.word('nandri', 3));
+```
+
+Browser gzip decoding requires `DecompressionStream` support; Node uses its built-in zlib. Switching languages retains already loaded models in memory for reuse. Recreate the engine to release that cache. The demo preserves URLs, email addresses, @mentions, #tags, digits and already-native text. Hindi retains the plain reference convention of changing a final full stop to danda. Roman English words are otherwise treated as transliteration input; there is no multilingual code-switch detector.
+
+## Python reference
+
+Requires `numpy` and `torch`. For a private review repository, authenticate with Hugging Face normally; never put a write token in browser code.
+
+```python
+from pathlib import Path
+import sys
+from huggingface_hub import snapshot_download
+
+root = Path(snapshot_download('sourojitdotcom/AksharaLP-MultiIndic-8M', token=True))
+sys.path.insert(0, str(root / 'python'))
+from multiindic import MultiIndic
+
+engine = MultiIndic(root, language='te', threads=1)
+print(engine.word('namaskaram', k=8))
+engine.set_language('hi').set_sentence(True)
+print(engine.transliterate('main theek hoon'))
+```
+
+The Python reference returns candidate strings from `word()`; JavaScript additionally exposes uncalibrated ranking scores. Those scores are **not probabilities**. Hindi uses the plain reference token conversion in both runtimes, not the older demo's extra brand/acronym heuristics. The model/decoder parity samples are measured separately.
+
+## GGUF containers
+
+**These are custom-architecture GGUF v3 tensor containers. They do NOT run in llama.cpp, Ollama, or LM Studio.** Use the bundled [gguf_read.py](gguf_read.py) plus our decoder. A GGUF is not a standalone dictionary or language-model runtime.
+
+The writer/reader follows the [existing v2 export](https://huggingface.co/sourojitdotcom/AksharaLP-1M-v2). Word **q4 GGUF carries the byte-exact shipped mixed q4/q8/f16 blob in an I8 tensor**; it is not standard GGML Q4_0. F16 materializes those already-quantized weights as half floats; it does not restore an original full-precision checkpoint. Exact reranker configuration is retained as JSON metadata, and the external dictionary/runtime remain required.
+
+For Hindi sentence mode, **q8 GGUF preserves the test-measured default byte-exactly**. An existing **q4 LM** and F16 materialization are included as alternatives with validation parity only. **Do not attach the default q8 test accuracies to q4 or F16 alternatives.** The q4 LM is not the demo default.
+
+| GGUF file | Raw bytes | gzip-9 bytes |
+|---|---:|---:|
+| [aksharalp-hi-word-q4.gguf](gguf/aksharalp-hi-word-q4.gguf) | 632,416 | 562,579 |
+| [aksharalp-hi-word-f16.gguf](gguf/aksharalp-hi-word-f16.gguf) | 2,026,048 | 1,348,288 |
+| [aksharalp-bn-word-q4.gguf](gguf/aksharalp-bn-word-q4.gguf) | 591,136 | 541,220 |
+| [aksharalp-bn-word-f16.gguf](gguf/aksharalp-bn-word-f16.gguf) | 1,984,864 | 1,321,177 |
+| [aksharalp-te-word-q4.gguf](gguf/aksharalp-te-word-q4.gguf) | 591,008 | 542,744 |
+| [aksharalp-te-word-f16.gguf](gguf/aksharalp-te-word-f16.gguf) | 1,984,608 | 1,329,379 |
+| [aksharalp-kn-word-q4.gguf](gguf/aksharalp-kn-word-q4.gguf) | 591,008 | 540,642 |
+| [aksharalp-kn-word-f16.gguf](gguf/aksharalp-kn-word-f16.gguf) | 1,984,608 | 1,319,622 |
+| [aksharalp-ta-word-q4.gguf](gguf/aksharalp-ta-word-q4.gguf) | 591,008 | 542,094 |
+| [aksharalp-ta-word-f16.gguf](gguf/aksharalp-ta-word-f16.gguf) | 1,984,608 | 1,328,348 |
+| [aksharalp-hi-sentence-q8.gguf](gguf/aksharalp-hi-sentence-q8.gguf) | 3,488,192 | 2,981,426 |
+| [aksharalp-hi-sentence-q4.gguf](gguf/aksharalp-hi-sentence-q4.gguf) | 2,110,912 | 1,596,808 |
+| [aksharalp-hi-sentence-f16.gguf](gguf/aksharalp-hi-sentence-f16.gguf) | 5,790,336 | 5,159,347 |
+
+### Validation-only parity — not a second test evaluation
+
+First **40 ASCII-alphabetic Roman words per language from full VALID** (200 total). Same dictionary and reranker; no fitting or tuning during parity. “Top-8 order” compares the complete ordered candidate list.
+
+| Language | q4 top-1 identical | q4 top-8 order identical | F16 top-1 identical | F16 top-8 order identical |
+|---|---:|---:|---:|---:|
+| Hindi | 40/40 | 40/40 | 40/40 | 40/40 |
+| Bengali | 40/40 | 40/40 | 40/40 | 40/40 |
+| Telugu | 40/40 | 40/40 | 40/40 | 40/40 |
+| Kannada | 40/40 | 40/40 | 40/40 | 40/40 |
+| Tamil | 40/40 | 40/40 | 40/40 | 40/40 |
+
+Hindi LM: first **12 Dakshina VALID sentences with at most 12 Roman words**, fixed word-model lattices. q8 reproduces **12/12** source decoder outputs; F16 **12/12**; q4 **11/12**. Maximum absolute sequence-score deltas vs q8 are 0.000000, 0.008702, and 6.343046, respectively. This is conversion agreement, **not accuracy**. Across the same validation sample, JavaScript and Python matched **200/200 top-1 and complete top-8 ordered lists**, and **12/12 Hindi sentence outputs**, after using the same plain-reference token policy. Both our structural reader and independent `gguf-py` read all 13 files; complete data is in [gguf_parity.json](evaluation/gguf_parity.json).
+
+```python
+# After the Python setup above:
+engine.set_language('bn')
+engine.load_word_gguf(root / 'gguf/aksharalp-bn-word-q4.gguf')
+print(engine.word('nomoskar'))
+engine.set_language('hi')
+engine.load_sentence_gguf(root / 'gguf/aksharalp-hi-sentence-q8.gguf')
+print(engine.transliterate('main theek hoon'))
+```
+
+## Licence & data
+
+**Model-card licence: CC-BY-NC-4.0. This release is non-commercial.** The Bengali, Telugu, Kannada and Tamil weights contain `aks-Samanantar` Aksharantar training rows. The [current upstream Samanantar card](https://huggingface.co/datasets/ai4bharat/samanantar) states **CC-BY-NC-4.0**. Aksharantar and the publisher's Samanantar packaging page describe CC0 packaging, so provenance terms conflict; this release does **not** certify those weights as commercially clean. A commercially clean **Apache-2.0 retrain is planned**, and would be separately evaluated before replacement.
+
+| Resource | Terms / role |
+|---|---|
+| [Aksharantar, AI4Bharat](https://huggingface.co/datasets/ai4bharat/Aksharantar) | CC0 mined/existing packaging; CC-BY manual data. New training includes Samanantar-derived rows: Bengali **114,558**, Telugu **42,413**, Kannada **36,482**, Tamil **34,447**; per-language training reports are bundled. |
+| [Samanantar, AI4Bharat](https://huggingface.co/datasets/ai4bharat/samanantar) | CC-BY-NC-4.0 upstream card; reason for this release's non-commercial restriction. |
+| [Dakshina, Google Research](https://github.com/google-research-datasets/dakshina) | CC-BY-SA-4.0 upstream. Source-attributed pairs appear within Aksharantar; original terms remain, and derivative-rights questions are not resolved by selecting a more restrictive model-card licence. Cite Roark et al., LREC 2020. |
+| [IndicCorp v2, AI4Bharat](https://huggingface.co/datasets/ai4bharat/IndicCorpV2) | CC0; frequency dictionaries and Hindi LM training. |
+| [IndicXlit, AI4Bharat](https://github.com/AI4Bharat/IndicXlit) | MIT teacher for distillation, and comparison baseline; original checkpoint not redistributed. |
+| Runtime code / original Hindi components | Apache-2.0, retained with their existing notices. Inter and Noto Indic demo fonts: SIL OFL 1.1. |
+
+The **Hindi components are also separately available under Apache-2.0**: [AksharaLP-1M-v2](https://huggingface.co/sourojitdotcom/AksharaLP-1M-v2) and [AksharaLP-Sentence-2.8M](https://huggingface.co/sourojitdotcom/AksharaLP-Sentence-2.8M). This combined release does not revoke or replace those existing grants. Code, dictionaries and individual components retain their own upstream terms; see [NOTICE](NOTICE), [LICENSE](LICENSE) and [LICENSE-APACHE-2.0](LICENSE-APACHE-2.0). This documentation is engineering provenance, not legal advice.
+
+## FAQ
+
+**Is this a multilingual LLM?** No. It cannot answer questions, reason generally, summarize, or translate between languages. It changes Romanized spellings into native script.
+
+**Why 8M when a word model has 989K parameters?** Five independent word models total 4,944,768; optional Hindi LM adds 2,770,752; total 7,715,520, rounded to 8M. That count excludes small reranker coefficients and dictionary data. Only selected components load.
+
+**Is it the best model for these languages?** Not established. It trails IndicXlit's full package on all four new test languages. We claim measured footprint and measured results, not smallest-ever or best-in-class status.
+
+**Does every language understand a sentence?** No. Only Hindi has neural sentence rescoring. Others concatenate independent word predictions, preserving text layout. Hindi rescoring cannot recover a spelling outside its candidate list.
+
+**Can I use it commercially?** Not this combined release. Its model-card licence is CC-BY-NC-4.0. Use independently Apache-licensed Hindi releases where appropriate, or wait for a cleared retrain. Retain all source attributions and review applicable component terms.
+
+**Does text leave the device?** The demo fetches local model assets from the Space when loading a language. Typed text is processed in the browser and is not sent to an inference API or telemetry service. Once the selected assets are loaded, typing needs no network. This is not a promise to avoid the initial asset download.
+
+**Can I run the GGUFs with llama.cpp?** No. GGUF is a container here, not a supported llama.cpp architecture. Use our reader, dictionaries and decoding code.
+
+## Limitations & responsible use
+
+Names, ambiguous short words, unseen vocabulary, English code-switching, dialects and chat abbreviations can fail. The output must be user-reviewable; do not silently replace important legal, medical or financial text. Candidate suggestions are uncalibrated. Input encoding truncates a word after 31 Roman characters. Type-as-you-go inference is debounced; older phones and long sentences may be slow. All measured compute here is CPU, not an on-device mobile benchmark. Tiny low-bit downloads still expand to float32 arrays and large dictionary structures in RAM.
+
+This private release records the current artifacts. No “best in class,” general-LLM capability, sub-5-MB complete bundle, commercial-clean, or multilingual-sentence claim is made.
+
+---
+**AksharaLP · Sourojit Dhua · 2026-11-10 (IST)** · Private review copy. Do not make public without the owner's approval.
diff --git a/SHA256SUMS b/SHA256SUMS
new file mode 100644
index 0000000000000000000000000000000000000000..3d69acc958333615d362c89f25d8c02509aac2ba
--- /dev/null
+++ b/SHA256SUMS
@@ -0,0 +1,101 @@
+d39ccdda1d2005a1b51113aec99f0c29a55d70dc0e35399656866ce0d9438e94 .gitattributes
+cc97a2c4ff7723a1f25179dc7c18f676fa84191bbb36ea7d0061acaf3c102559 LICENSE
+e61722325c93e58388069e84e208e43ca5b6294050676ee27194921cf453a071 LICENSE-APACHE-2.0
+8dd4007ec7ce19774061f88c9173ef176951e329083c079728c901ce15b15f13 NOTICE
+edc975f244807c134d306d1376fdfb8b29073d5db540e30db0bc90371b3c1e45 README.md
+f6484f5caaebaf2acb68540cc2cb29402030aee22a7e45b25d37424872f769a8 aksharalp.mjs
+e29d4e04be4a7235d4f2b94b6578cd580486960a7068b048e1bf6c1b1799679a assets/accuracy.png
+c5378130af32052f6020b7cdc5a491aa5f1d11de763c37ee5491ba1eaef0bf49 assets/accuracy.svg
+27da3ac213f72636663b84026093cf2a1c80f825d8d67f110466ae9df0d3d6fc assets/banner.png
+a9f8a61342401feb1e4bb6c1f82d0831d3fcbf35e3a6788eb7027d193b8de129 assets/banner.svg
+70d7aee1fe5e1e61dc7154c0aa4e33c818ff30ebbbe5d78fb5f52ba09fad1e04 assets/sizes.png
+e6a92bb1337fb4997c5d657e12d0b34208d0240e006aa3eb58bce6d3a9b53bec assets/sizes.svg
+f5689263059364546fe23824634183d8781defb4691f36d951fff63558f16afc config.json
+13350ec5c2ffa334de1dd6fb11e318fb1eeb5b80791506768546c3e43607539d evaluation/bn_frozen.json
+79b140c2feac29957461d9ee879db331ea30885dee334675694169315f0d7535 evaluation/examples.json
+8d7b75ab880b89c749765cba386fcf0c3a290cdb6c1d2c9491be4a341a5453bc evaluation/gguf_independent_validation.json
+4f898f14ad3912b8ecfa2025476f17ddad7aaa307bcac381620ffa17f2652a27 evaluation/gguf_parity.json
+bf7eba31593b49e8223d56726022d6e68627ad437661003771144995b5cba07f evaluation/gguf_sizes.json
+392f25d66278a10a0e4f0093539d8f117727737bfd1b3b2d0e4dc31cfb5c792d evaluation/hindi_existing_results.json
+8e29063e276abfa4320d1fc299b3a5560ec2bbada77bb7147a8fba97563b3acd evaluation/indicxlit_storage.json
+157f7694918ba21b4e5268edbcf54f3fa5f1416506234ef5b78f1b9a502755da evaluation/kn_frozen.json
+8a9b660df627f538d67decc52b16b5349dc416d0cb0b25fb0b793f7752f54763 evaluation/latency.json
+50f1594f15182c2ade9a10d4b971418932e98d7413a40590d2fc96f30ff6b80b evaluation/multiindic_test.json
+25b2cc588a557f77ee852427fe06c77b108cad74fb6082d299b72cc1e1d58fe0 evaluation/release_summary.json
+a8912129bb1143d06e7f8761b515516b6e503d2f36951e684ee2f602b7911285 evaluation/sizes.json
+4798b6de7e820251a93924186934287d70407705df242573910c74b60d85e431 evaluation/ta_frozen.json
+eb327b20a375b534bad337dbf90b4e054b58a788fc17a34b142668ccbdae0cca evaluation/te_frozen.json
+7a0b78e133bb3ed773f6a8bae5e3f1af4b75f5158de4e814efadeaa0002ed580 gguf/aksharalp-bn-word-f16.gguf
+8311c77889381ce559d483b965027c2d409787ac090b146ae1af08060e4a4552 gguf/aksharalp-bn-word-q4.gguf
+446ff8f003c3680e9098df94727f8eb67a6a05a7c418888fb0bfeeb2742e098e gguf/aksharalp-hi-sentence-f16.gguf
+8ecc83bc618b9c67173328a34f0f40e5c83ce77a993a6279acdeae6d724690db gguf/aksharalp-hi-sentence-q4.gguf
+5ff23bc8d131b560ee29c79ee16d15294600149200948433d76f29f4b8c3e496 gguf/aksharalp-hi-sentence-q8.gguf
+3b6670429a226a39d357160691c634160667728bae8cfc7a711067424a2d224e gguf/aksharalp-hi-word-f16.gguf
+737d14197514a2625eb9505eb0a22ebd62cd2a4b764c65761f4084599bb6eb36 gguf/aksharalp-hi-word-q4.gguf
+c9d273c8339864bd37c2b051cab60d2741c1d71e56e8c27d00a981ff28ef3864 gguf/aksharalp-kn-word-f16.gguf
+0e000f7fc0e52bbcb1f05753a46e6de50d1a1bdd6bb0ef46303361b9c39138ed gguf/aksharalp-kn-word-q4.gguf
+3baf8764ecc77250666eb3f8eb83f2d7a33763e13227e2d6db8272868dd5a864 gguf/aksharalp-ta-word-f16.gguf
+f270e77e6355d53dc140f7cc7282551f5126302bcfbdbb17d802861510896c04 gguf/aksharalp-ta-word-q4.gguf
+3c556cec413db8f1485a1ee9ea237406f967937019e0d9d6076c3ae41b7a6231 gguf/aksharalp-te-word-f16.gguf
+56b9f956b69e4fcf9e8d3f23c9391a091208651e90b06e2bde6d7f46d7cb6a60 gguf/aksharalp-te-word-q4.gguf
+9348b0d8b522e9e3e45843a95c47087a3331f811a26c52813fa177a506f5cdcf gguf_read.py
+edb2ad0bd7de4e87bd4d49ac406a9ef01011451bf23ac82d4c063bb9cce1dc2b models/bn/dict.tsv.gz
+7007c674102f7d242b71832c877ea1d58b6b83393ac1ea2d2814f93dec499cda models/bn/inventory.json
+e0bd564ad61f6dcef8eca6bf3d64f8b231146c9885aa4c5b282a2b2420606c4f models/bn/params.json
+7d85a15252ea42676289a216bb775c2e48fea96634b32d2f2f0dce214951ada3 models/bn/training_report.json
+9f099f0b9524f446debf1d3f0ae1dc7ed1e88625c33f6dbb7b45e0348bf76376 models/bn/xlit.q4
+1c87e62a4249a63c333942c1c8736eb2c89f9ee3148ef8fc422784ba6ce21e43 models/bn/xlit.q4.gz
+73bd5436b77db4ee433c564521f19a2b78cb6a45c429f274d43bef909708347f models/hi/dict/lex.bin
+3b4dbff703f395e97b4503646fde7782bab0f3d79b9731ef703df6b961e38f55 models/hi/dict/lex.bin.gz
+254e34abe7c1e6ed82f74d5cbf175aba1aedbee316363fe2a67e06291ba589ee models/hi/dict/lm.bin
+ab3f6b4135862b5bbf42815cc1219274eed711b5a7dfc66f09fc7b0fe32ceed6 models/hi/dict/lm.bin.gz
+6d60eaf2b4e89a847a750a226680021460e454c6d1a06070a56e8f8664500565 models/hi/dict/params.json
+bc7dd23ba83ec6bc5841b03fa7b8e3cb934d0322dda652863537ce03958ea79d models/hi/dict/params.json.gz
+620ab5ebf213fc4b8469b88338368b34616390154f72f64f3dcd49a550f01472 models/hi/manifest.json
+acb2bd2c0c12813d18c9696058260d57ea504db76db6d2d0810054a9e65a875e models/hi/manifest.json.gz
+1dc58564cd649278c819641a65141e9e96a42f3fc7fba555b9072628674c2935 models/hi/params.json
+8e42f8aec0204bca50891244478d94ac3ba1801f8cc774c475b84c5581377c73 models/hi/params.json.gz
+2c90f300c0114354fcf0c5a12300b9ba98ce71b4580b54f0b870a9395e869227 models/hi/sentence/nlm.q4
+b8f3850506d93f3b0531beb37362cf1ad7c02a9e1d8c81ebc128efee9e44c17e models/hi/sentence/nlm.q8
+7face18513d342c59b005f3ea57a0e86310dc863778a828e7743cea0d5c68526 models/hi/sentence/nlm.q8.gz
+652eafe0c1e2b46fec85663825621284f0a5911f2d7cc6e13b1182ce47cd9803 models/hi/xlit.q4
+191eb28f75f29c45ec29f044031a6ae5aedd6a80edc3f3a9d1ae78a53c04fb9d models/hi/xlit.q4.gz
+97035264d30eddb5810467dc5bbf088614221ea6ba2c3fff349c6bbd196bb644 models/kn/dict.tsv.gz
+cc2085a1f34c929678658159bc22eaa7641ce4aecada94fa3336dc82e1e553a3 models/kn/inventory.json
+6076d9126ffce66630b634e47ac5b38d03cda3bce7ab4d07162aeb0cbc925356 models/kn/params.json
+bfae38dd1fb1c799e9005750b3e7ed93860a2be8051489a3cd0ba9ead17ae30f models/kn/training_report.json
+3788fd545c297febb7dbbdbdd780e831042395b4ec99260b95c963607afc8664 models/kn/xlit.q4
+bedc5a698dc5c83a964f03586bc0ded67e88fa0b644cdd82fafc21f1e50925d9 models/kn/xlit.q4.gz
+4d14f514020e7737144c22991e9324a29734b5ab203bef3b68ec6d9f51e7cddb models/ta/dict.tsv.gz
+e1ea48ffbd41b725a435c611d0ebd1da454904bc90e2368eb2fbaecee9f1870b models/ta/inventory.json
+f4520a8320f02573e6b8b6530a372c980903fbf757999540698957e8eec58385 models/ta/params.json
+0c93e563d879dfef97ca0d36ae6c6ce4ca21dd417741be90478b7191214b0165 models/ta/training_report.json
+5dd185a033db79835fede66ab41a25f5897f24ad5d7c8ecc9864ca6ec167fdc0 models/ta/xlit.q4
+c15be889468520e6d5ee8dfbc4c77cad781a79f454d4681026fc082ae217afc2 models/ta/xlit.q4.gz
+20612a52ccf39c816d3068e78e9e9a3aa9aed9156f10de45ebbd543363943640 models/te/dict.tsv.gz
+ff20b9635ecef52f692a53807be1c6a5acf3922212d2bc14f7dd9283002bc690 models/te/inventory.json
+653f897efe1af78bb52149f2a18dc804b49ecac9c93add867871e64f56e07470 models/te/params.json
+da8bc7af7e481e995e3ada79f21585bdbcf7df8fc751fdf151d8a774f8f0f343 models/te/training_report.json
+aea094e46a3c61eb9487d1aaba3824d3d9dc60f3f1cf6b6f99cadf04fd4318f9 models/te/xlit.q4
+3513df30b5d3bfa07085baa19f0c9c1076e85268a5df88fcc4b492630b1b4eb2 models/te/xlit.q4.gz
+037d1f4e40f61457e4e33a9b8a034d2509316b723f27907849d55f0fb9cf23cf multiindic.mjs
+a0bc14ac9629a0bb6b69fe94609eb922a9b71bbd327b17553581b91648d970c2 nlm.mjs
+e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 python/bhc/__init__.py
+af77db2406b19466f538ebc35be31b6ffcf23de2d8447e48247a8ba58c1c2697 python/bhc/agree.py
+aebfdd405693b5096744cbab03a30662e440308583302e9b5b54575ce0498231 python/bhc/casual.py
+b07229b0d484048f6f75fd8304dc62e2a05d2dbc450289bfb57f6efd669c9ffc python/bhc/lexlm.py
+5b7c49b45309ac3f324906bd2546c35cfc4d5a199a9311a9d845233337a41b39 python/bhc/lextrie.py
+ba9c1c4b326328cbbc65085b379835fac522d116e1ea163625bfcadf5c0de723 python/bhc/nlm.py
+39e674d163ebcb7e7adcf02dd267fd6c9b188be1346486f2695736240c4f42ff python/bhc/skel.py
+09617bd0f9b90a4179628dd40d786a30ba6a3a2e043d3993ecf36cf223524483 python/bhc/text.py
+67de6da4dc780c3128ca5c2da4461d1e0616d3f1c65619cc62ef59699a28c2d2 python/bhc/translit2.py
+e93f9cc572853158f7414a46ac24815aea40c76e0939a2b79a6034542aeb2615 python/bhc/translit3.py
+8eb7235292a0b42a4cdfc07c548809fe312e2eec2c622317ff40e0c203ff5766 python/bhc/xlit_model.py
+1a577ff4b9075b6ee73debed2955840d30a5796a846ba74af58561866681307f python/bhc/xlit_q.py
+9348b0d8b522e9e3e45843a95c47087a3331f811a26c52813fa177a506f5cdcf python/gguf_read.py
+e22e0dfcd2cee31d13d5eee303e07dea973032bfdfbd09b2c4060d69b0f12367 python/multiindic.py
+72cabbf383d3b033df0eb37be00bb7079f51b0d08d793f0ac9d7eec4709dffbb python/requirements.txt
+de4599eedf54731c4d61f2bee9afb0d3776aaa5a49516fa4d2bf720bc2fdbb7a sent.mjs
+f62025ba5b1c6b7a0e3559a0b6a6dc6f53773f188f8be118f9d0e6083b4e5211 tools/build_gguf.py
+9fcbb1848e3a46daaf9d9b317f240a1d04d7442a7242af99df7a23d24fd48940 tools/gguf_format.py
+dcd348d0f8f26b3316123228ae534d37155bfe552520141a38aa55b14707f0d9 v3.mjs
diff --git a/aksharalp.mjs b/aksharalp.mjs
new file mode 100644
index 0000000000000000000000000000000000000000..f6664a2477d0c235c853bf09608ba2393186ad09
--- /dev/null
+++ b/aksharalp.mjs
@@ -0,0 +1,822 @@
+/*!
+ * BhashaCore JS runtime — romanized Hindi / Hinglish -> Devanagari, fully offline (browser, Web Worker, Node >= 18).
+ * Zero dependencies. Pure-JS port of the reference Python runtime (bhc/translit2.py, bhc/xlit_q.py, bhc/lexlm.py,
+ * bhc/lextrie.py, bhc/lid.py, bhc/hinglish.py) over the exported asset formats:
+ * xlit.q4 | xlit.q8 | xlit.int8 char Transformer (pre-LN enc/dec), 4/8-bit grouped or per-row INT8 weights
+ * lex.bin Devanagari lexicon (LX4 alphabetical front-coded + 4-bit bands, or v2 bucketed)
+ * lm.bin pruned backoff word-bigram LM (varint CSR + 4-bit codebooks)
+ * params.json decoding params + reranker (linear w or MLP w1/b1/w2/b2; mu / sd)
+ * lid.int8 char-CNN + BiGRU token / sentence language ID (optional)
+ * loan.tsv, natural_hi.tsv Hinglish loanword lexicon, Natural-Hindi glossary (optional)
+ * Bundle layout: a small CORE (model + params.json) that works standalone, plus an optional dictionary pack in
+ * dict/ (lex.bin + lm.bin + params.json overriding the core params), fetched on first use, cached with the Cache API
+ * when available, and attached with engine.loadDict(); if it cannot be fetched the engine stays in core-only mode.
+ *
+ * AksharaLP by Sourojit Dhua — https://huggingface.co/sourojitdotcom/AksharaLP-1M-v2
+ * Copyright 2026 Sourojit Dhua. Licensed under the Apache License, Version 2.0; commercial use permitted.
+ * Attribution required (Apache-2.0 §4(d)): keep this notice and the NOTICE file in redistributions and derivatives:
+ * "Transliteration by AksharaLP (Sourojit Dhua) — https://huggingface.co/sourojitdotcom/AksharaLP-1M-v2"
+ */
+
+// ------------------------------------------------------------------ vocab
+const PAD = 0, BOS = 1, EOS = 2, UNK = 3, MAXLEN = 32, V = 132;
+const NEG = -Infinity;
+const encSrc = (w) => {
+ const s = w.toLowerCase().slice(0, MAXLEN - 1), ids = [];
+ for (const c of s) { const k = c.charCodeAt(0); ids.push(c.length === 1 && k >= 97 && k <= 122 ? k - 97 + 4 : UNK); }
+ ids.push(EOS); return ids;
+};
+const t2i = (ch) => { const k = ch.codePointAt(0); return k >= 0x900 && k < 0x980 ? k - 0x900 + 4 : -1; };
+const encTgt = (w) => { const ids = []; for (const c of [...w].slice(0, MAXLEN - 2)) { const i = t2i(c); ids.push(i < 0 ? UNK : i); } ids.push(EOS); return ids; };
+const decTgt = (ids) => { let s = ""; for (const i of ids) if (i > UNK) s += String.fromCharCode(0x900 + i - 4); return s; };
+
+// ------------------------------------------------------------------ binary helpers
+function f16(h) {
+ const s = (h & 0x8000) ? -1 : 1, e = (h >> 10) & 0x1f, f = h & 0x3ff;
+ if (e === 0) return s * f * 5.960464477539063e-8;
+ if (e === 31) return f ? NaN : s * Infinity;
+ return s * (1 + f / 1024) * Math.pow(2, e - 15);
+}
+function readHeader(buf) {
+ const u8 = new Uint8Array(buf), dv = new DataView(u8.buffer, u8.byteOffset, u8.byteLength);
+ const n = dv.getUint32(0, true);
+ const meta = JSON.parse(new TextDecoder().decode(u8.subarray(4, 4 + n)));
+ const data = u8.subarray(4 + n);
+ return { meta, data, dv: new DataView(data.buffer, data.byteOffset, data.byteLength) };
+}
+const prod = (a) => a.reduce((x, y) => x * y, 1);
+const f16arr = (dv, off, n) => { const o = new Float32Array(n); for (let i = 0; i < n; i++) o[i] = f16(dv.getUint16(off + 2 * i, true)); return o; };
+
+/** Parse xlit.int8 / xlit.q4 / xlit.q8 / lid.int8 into {cfg, extra, T: {name: {shape, data: Float32Array}}}. */
+export function parseWeights(buf) {
+ const { meta, data, dv } = readHeader(buf); const T = {};
+ for (const t of meta.tensors) {
+ const [k, shape, kind, off] = t; const cnt = prod(shape); let w;
+ if (kind === "f16") w = f16arr(dv, off, cnt);
+ else if (kind === "i8") { // per-row symmetric INT8 (+ fp16 row scale)
+ const r = shape[0], c = cnt / r, sc = f16arr(dv, off + cnt, r); w = new Float32Array(cnt);
+ for (let i = 0; i < r; i++) for (let j = 0; j < c; j++) { const q = (data[off + i * c + j] << 24) >> 24; w[i * c + j] = q * sc[i]; }
+ } else { // q4 / q8: group-wise symmetric (G inputs per fp16 scale)
+ const bits = kind.length > 1 ? parseInt(kind.slice(1)) : meta.bits, G = meta.G, ln = t[4];
+ const [r, c] = shape, ng = Math.ceil(c / G), sc = f16arr(dv, off + ln, r * ng); w = new Float32Array(cnt);
+ for (let i = 0; i < cnt; i++) {
+ let q;
+ if (bits === 4) { const b = data[off + (i >> 1)]; q = ((i & 1) ? (b & 15) : (b >> 4)) - 8; }
+ else q = (data[off + i] << 24) >> 24;
+ const row = (i / c) | 0, col = i - row * c; w[i] = q * sc[row * ng + ((col / G) | 0)];
+ }
+ }
+ T[k] = { shape, data: w };
+ }
+ return { cfg: meta.cfg, extra: meta.extra || {}, T };
+}
+
+// ------------------------------------------------------------------ math kernels
+function linear(x, xo, W, b, out, oo, nin, nout) { // out[oo..] = W x + b ; W row-major [nout, nin]
+ for (let o = 0; o < nout; o++) {
+ let s = b ? b[o] : 0; const r = o * nin;
+ for (let i = 0; i < nin; i++) s += W[r + i] * x[xo + i];
+ out[oo + o] = s;
+ }
+}
+function layerNorm(x, xo, g, b, out, oo, d) {
+ let m = 0; for (let i = 0; i < d; i++) m += x[xo + i]; m /= d;
+ let v = 0; for (let i = 0; i < d; i++) { const t = x[xo + i] - m; v += t * t; } v /= d;
+ const r = 1 / Math.sqrt(v + 1e-5);
+ for (let i = 0; i < d; i++) out[oo + i] = (x[xo + i] - m) * r * g[i] + b[i];
+}
+function erf(x) { // |err| < 1.2e-7 (Numerical Recipes erfc)
+ const z = Math.abs(x), t = 1 / (1 + 0.5 * z);
+ const r = t * Math.exp(-z * z - 1.26551223 + t * (1.00002368 + t * (0.37409196 + t * (0.09678418 + t * (-0.18628806 +
+ t * (0.27886807 + t * (-1.13520398 + t * (1.48851587 + t * (-0.82215223 + t * 0.17087277)))))))));
+ return x >= 0 ? 1 - r : r - 1;
+}
+const gelu = (x) => 0.5 * x * (1 + erf(x / Math.SQRT2));
+const sigmoid = (x) => 1 / (1 + Math.exp(-x));
+
+// ------------------------------------------------------------------ char Transformer
+export class XlitModel {
+ constructor(w) {
+ const c = w.cfg; this.V = w.T["temb.weight"].shape[0];
+ const start = {hi:0x900,bn:0x980,te:0xc00,kn:0xc80,ta:0xb80}[c.vocab || "hi"];
+ this.target = Array.from({length:128}, (_,i)=>String.fromCharCode(start+i));
+ if (c.vocab === "bn") this.target.push("\u0964");
+ this.decTgt = ids => ids.filter(i=>i>3).map(i=>this.target[i-4]).join("");
+ this.d = c.d; this.h = c.h; this.dk = c.d / c.h; this.ff = c.ff; this.ne = c.enc; this.nd = c.dec;
+ const g = (k) => { const t = w.T[k]; if (!t) throw new Error("missing tensor " + k); return t.data; };
+ this.semb = g("semb.weight"); this.temb = g("temb.weight"); this.spos = g("spos.weight"); this.tpos = g("tpos.weight");
+ const att = (p) => ({ q: g(p + ".q.weight"), qb: g(p + ".q.bias"), kv: g(p + ".kv.weight"), kvb: g(p + ".kv.bias"), o: g(p + ".o.weight"), ob: g(p + ".o.bias") });
+ const ln = (p) => ({ g: g(p + ".weight"), b: g(p + ".bias") });
+ const ffn = (p) => ({ w1: g(p + ".ff.0.weight"), b1: g(p + ".ff.0.bias"), w2: g(p + ".ff.2.weight"), b2: g(p + ".ff.2.bias") });
+ this.enc = []; for (let i = 0; i < c.enc; i++) { const p = "enc." + i; this.enc.push({ n1: ln(p + ".n1"), att: att(p + ".att"), n2: ln(p + ".n2"), ff: ffn(p) }); }
+ this.dec = []; for (let i = 0; i < c.dec; i++) { const p = "dec." + i; this.dec.push({ n1: ln(p + ".n1"), sa: att(p + ".sa"), n2: ln(p + ".n2"), ca: att(p + ".ca"), n3: ln(p + ".n3"), ff: ffn(p) }); }
+ this.en = ln("en"); this.dn = ln("dn"); this.scale = Math.sqrt(c.d);
+ this.nParams = Object.values(w.T).reduce((a, t) => a + t.data.length, 0);
+ const d = this.d; this._y = new Float32Array(d); this._a = new Float32Array(d); this._f = new Float32Array(this.ff); this._q = new Float32Array(d); this._kv = new Float32Array(2 * d);
+ this._sc = new Float64Array(MAXLEN);
+ }
+ _ffn(L, x, xo, n) { // x[xo] += FF(LN(x))
+ const d = this.d, y = this._y, f = this._f, a = this._a;
+ layerNorm(x, xo, n.g, n.b, y, 0, d); linear(y, 0, L.w1, L.b1, f, 0, d, this.ff);
+ for (let i = 0; i < this.ff; i++) f[i] = gelu(f[i]);
+ linear(f, 0, L.w2, L.b2, a, 0, this.ff, d); for (let i = 0; i < d; i++) x[xo + i] += a[i];
+ }
+ /** attention of one query q (len d) over K/V rows (stride 2d in kvbuf: K at +0, V at +d) -> out (len d) */
+ _attend(q, kvbuf, kvo, n, out) {
+ const { h, dk, d } = this, sc = n <= MAXLEN ? this._sc : new Float64Array(n), inv = 1 / Math.sqrt(dk);
+ for (let hh = 0; hh < h; hh++) {
+ let mx = -Infinity;
+ for (let j = 0; j < n; j++) { let s = 0; const ko = kvo + j * 2 * d + hh * dk; for (let i = 0; i < dk; i++) s += q[hh * dk + i] * kvbuf[ko + i]; s *= inv; sc[j] = s; if (s > mx) mx = s; }
+ let z = 0; for (let j = 0; j < n; j++) { sc[j] = Math.exp(sc[j] - mx); z += sc[j]; }
+ for (let i = 0; i < dk; i++) { let s = 0; for (let j = 0; j < n; j++) s += sc[j] * kvbuf[kvo + j * 2 * d + d + hh * dk + i]; out[hh * dk + i] = s / z; }
+ }
+ }
+ /** encode one word -> {S, cross: per decoder layer K/V of memory (Float32Array S*2d)} */
+ encode(ids) {
+ const d = this.d, S = ids.length, x = new Float32Array(S * d), y = new Float32Array(S * d), kv = new Float32Array(S * 2 * d), att = new Float32Array(d), o = this._a;
+ for (let t = 0; t < S; t++) for (let i = 0; i < d; i++) x[t * d + i] = this.semb[ids[t] * d + i] * this.scale + this.spos[t * d + i];
+ for (const L of this.enc) {
+ for (let t = 0; t < S; t++) { layerNorm(x, t * d, L.n1.g, L.n1.b, y, t * d, d); linear(y, t * d, L.att.kv, L.att.kvb, kv, t * 2 * d, d, 2 * d); }
+ const nx = new Float32Array(x);
+ for (let t = 0; t < S; t++) {
+ linear(y, t * d, L.att.q, L.att.qb, this._q, 0, d, d); this._attend(this._q, kv, 0, S, att);
+ linear(att, 0, L.att.o, L.att.ob, o, 0, d, d); for (let i = 0; i < d; i++) nx[t * d + i] += o[i];
+ }
+ for (let t = 0; t < S; t++) this._ffn(L.ff, nx, t * d, L.n2);
+ x.set(nx);
+ }
+ const mem = new Float32Array(S * d); for (let t = 0; t < S; t++) layerNorm(x, t * d, this.en.g, this.en.b, mem, t * d, d);
+ const cross = this.dec.map((L) => { const c = new Float32Array(S * 2 * d); for (let t = 0; t < S; t++) linear(mem, t * d, L.ca.kv, L.ca.kvb, c, t * 2 * d, d, 2 * d); return c; });
+ return { S, cross };
+ }
+ newCache(K) { const d = this.d; return this.dec.map(() => new Float32Array(K * MAXLEN * 2 * d)); }
+ /** one decoder step for beam rows `rows` (indices into cache) with tokens tok[row] at position t -> logprobs[row] (Float64Array V) */
+ step(enc, cache, rows, tok, t, out) {
+ const V = this.V, d = this.d, x = new Float32Array(d), y = this._y, att = new Float32Array(d), o = this._a;
+ for (const r of rows) {
+ for (let i = 0; i < d; i++) x[i] = this.temb[tok[r] * d + i] * this.scale + this.tpos[t * d + i];
+ for (let l = 0; l < this.nd; l++) {
+ const L = this.dec[l], C = cache[l], base = r * MAXLEN * 2 * d;
+ layerNorm(x, 0, L.n1.g, L.n1.b, y, 0, d);
+ linear(y, 0, L.sa.kv, L.sa.kvb, C, base + t * 2 * d, d, 2 * d);
+ linear(y, 0, L.sa.q, L.sa.qb, this._q, 0, d, d); this._attend(this._q, C, base, t + 1, att);
+ linear(att, 0, L.sa.o, L.sa.ob, o, 0, d, d); for (let i = 0; i < d; i++) x[i] += o[i];
+ layerNorm(x, 0, L.n2.g, L.n2.b, y, 0, d);
+ linear(y, 0, L.ca.q, L.ca.qb, this._q, 0, d, d); this._attend(this._q, enc.cross[l], 0, enc.S, att);
+ linear(att, 0, L.ca.o, L.ca.ob, o, 0, d, d); for (let i = 0; i < d; i++) x[i] += o[i];
+ this._ffn(L.ff, x, 0, L.n3);
+ }
+ layerNorm(x, 0, this.dn.g, this.dn.b, y, 0, d);
+ const lp = out[r]; let mx = -Infinity;
+ for (let v = 0; v < V; v++) { let s = 0; for (let i = 0; i < d; i++) s += y[i] * this.temb[v * d + i]; lp[v] = s; if (s > mx) mx = s; }
+ let z = 0; for (let v = 0; v < V; v++) z += Math.exp(lp[v] - mx); const lz = mx + Math.log(z);
+ for (let v = 0; v < V; v++) lp[v] -= lz;
+ }
+ }
+ /** beam search for one word (identical procedure to bhc.xlit_model.beam_search_fast for one batch row) */
+ beam(enc, S, { beam: K = 8, nbest = 8, trie = null } = {}) {
+ const V = this.V, d = this.d, cache = this.newCache(K), out = Array.from({ length: K }, () => new Float64Array(V));
+ let score = new Array(K).fill(NEG); score[0] = 0; let tok = new Array(K).fill(BOS); let hist = Array.from({ length: K }, () => []);
+ let nodes = trie ? new Array(K).fill(0) : null; const done = []; const L = Math.min(MAXLEN - 1, S + 8);
+ const tmp = new Float32Array(MAXLEN * 2 * d);
+ for (let t = 0; t < L; t++) {
+ const rows = []; for (let k = 0; k < K; k++) if (score[k] > NEG) rows.push(k);
+ this.step(enc, cache, rows, tok, t, out);
+ const cand = []; // (total, beam, v)
+ for (const k of rows) {
+ const lp = out[k]; let allow = null;
+ if (trie) { const n = nodes[k]; if (n === null) continue; allow = trie.allowed(n); }
+ if (allow) { for (const v of allow) if (v !== PAD && v !== BOS && v !== UNK) cand.push([score[k] + lp[v], k, v]); }
+ else { cand.push([score[k] + lp[EOS], k, EOS]); for (let v = 4; v < V; v++) cand.push([score[k] + lp[v], k, v]); }
+ }
+ cand.sort((a, b) => b[0] - a[0] || (a[1] * V + a[2]) - (b[1] * V + b[2]));
+ const top = cand.slice(0, 2 * K);
+ const ns = new Array(K).fill(NEG), nt = new Array(K).fill(EOS), par = new Array(K).fill(0), nn = trie ? new Array(K).fill(null) : null;
+ let j = 0;
+ for (const [sc, b, v] of top) {
+ if (!(sc > NEG)) continue;
+ if (v === EOS) { if (done.length < nbest) done.push([this.decTgt(hist[b]), sc]); continue; }
+ if (j < K) { ns[j] = sc; nt[j] = v; par[j] = b; if (trie) nn[j] = trie.child(nodes[b], v); j++; }
+ }
+ let alive = true;
+ if (done.length >= nbest && (j === 0 || ns[0] < Math.min(...done.map((x) => x[1])))) alive = false;
+ else if (!j) alive = false;
+ if (!alive) break;
+ for (const C of cache) { // reorder self-attention caches by parent beam
+ const n = (t + 1) * 2 * d, olds = [];
+ for (let k = 0; k < j; k++) { const src = par[k] * MAXLEN * 2 * d; olds.push(C.slice(src, src + n)); }
+ for (let k = 0; k < j; k++) C.set(olds[k], k * MAXLEN * 2 * d);
+ }
+ hist = Array.from({ length: K }, (_, k) => (k < j ? hist[par[k]].concat([nt[k]]) : []));
+ score = ns; tok = nt; nodes = nn;
+ }
+ const best = new Map();
+ for (const [s, sc] of done) if (!best.has(s) || sc > best.get(s)) best.set(s, sc);
+ return [...best.entries()].sort((a, b) => b[1] - a[1]).slice(0, nbest);
+ }
+ /** teacher-forced ln P(tgt | src) (as bhc.xlit_model.score_pairs) */
+ scorePair(enc, tgt) {
+ const V = this.V, ids = [...tgt].slice(0,MAXLEN-2).map(c=>{const i=this.target.indexOf(c);return i<0?UNK:i+4;}).concat([EOS]), cache = this.newCache(1), out = [new Float64Array(V)]; let tok = [BOS], s = 0;
+ for (let t = 0; t < ids.length; t++) { this.step(enc, cache, [0], tok, t, out); s += out[0][ids[t]]; tok = [ids[t]]; }
+ return s;
+ }
+}
+
+// ------------------------------------------------------------------ lexicon + bigram LM
+function readVarints(b, n, pos) {
+ const out = new Uint32Array(n);
+ for (let i = 0; i < n; i++) { let v = 0, sh = 0, c; do { c = b[pos++]; v += (c & 0x7f) * 2 ** sh; sh += 7; } while (c >= 0x80); out[i] = v; }
+ return [out, pos];
+}
+const unpack4 = (b, n) => { const o = new Uint8Array(n); for (let i = 0; i < n; i++) o[i] = (i & 1) ? b[i >> 1] & 15 : b[i >> 1] >> 4; return o; };
+
+export class LexLM {
+ constructor(lexBuf, lmBuf) {
+ if (!lexBuf) { // core-only: empty lexicon, no LM
+ this.words = []; this.sortedWords = []; this.uni = new Float32Array(0); this.N = 0; this.unk = 0; this.alpha = true; this.hasLM = false; return;
+ }
+ const b = new Uint8Array(lexBuf), dv = new DataView(b.buffer, b.byteOffset, b.byteLength); const words = [];
+ const decodeFC = (P, S, nWords) => { let prev = [], j = 0, pi = 0;
+ while (words.length < nWords) { const k = P ? P[pi++] : S[j++]; const cur = prev.slice(0, k);
+ for (;;) { const c = S[j++], ch = c & 0x7f; if (ch !== 0x7f) cur.push(ch); if (c & 0x80) break; }
+ words.push(String.fromCharCode(...cur.map((x) => 0x900 + x))); prev = cur; } };
+ if (b[0] === 0x4c && b[1] === 0x58 && b[2] === 0x34 && b[3] === 0) { // "LX4\0"
+ const N = dv.getUint32(4, true), nfc = dv.getUint32(8, true), NB = dv.getUint32(16, true); this.unk = dv.getFloat32(12, true);
+ let pos = 20; const ucb = new Float32Array(NB); for (let i = 0; i < NB; i++) ucb[i] = dv.getFloat32(pos + 4 * i, true); pos += 4 * NB;
+ const band = unpack4(b.subarray(pos, pos + ((N + 1) >> 1)), N); pos += (N + 1) >> 1;
+ this.uni = new Float32Array(N); for (let i = 0; i < N; i++) this.uni[i] = ucb[band[i]];
+ decodeFC(b.subarray(pos, pos + N), b.subarray(pos + N, pos + N + nfc), N); this.N = N; this.alpha = true;
+ } else { // v2 (frequency-bucketed)
+ const N = dv.getUint32(0, true), nb = dv.getUint32(4, true), nfc = dv.getUint32(8, true); this.unk = dv.getFloat32(12, true);
+ let [bn, pos] = readVarints(b, nb, 16); const blp = []; for (let i = 0; i < nb; i++) blp.push(f16(dv.getUint16(pos + 2 * i, true))); pos += 2 * nb;
+ this.uni = new Float32Array(N); let w = 0; for (let i = 0; i < nb; i++) for (let k = 0; k < bn[i]; k++) this.uni[w++] = blp[i];
+ decodeFC(null, b.subarray(pos, pos + nfc), N); this.N = N; this.alpha = false;
+ }
+ this.words = words;
+ let sorted = this.alpha; for (let i = 1; sorted && i < words.length; i++) if (!(words[i - 1] < words[i])) sorted = false;
+ if (!sorted) { this.map = new Map(); words.forEach((w, i) => this.map.set(w, i)); }
+ this.sortedWords = sorted ? words : [...new Set(words)].sort();
+ this.hasLM = false;
+ if (lmBuf) this._loadLM(new Uint8Array(lmBuf));
+ }
+ id(w) {
+ if (this.map) { const i = this.map.get(w); return i === undefined ? null : i; }
+ let lo = 0, hi = this.words.length; while (lo < hi) { const m = (lo + hi) >> 1; if (this.words[m] < w) lo = m + 1; else hi = m; }
+ return lo < this.words.length && this.words[lo] === w ? lo : null;
+ }
+ _loadLM(b) {
+ const dv = new DataView(b.buffer, b.byteOffset, b.byteLength), N = this.N;
+ const nbg = dv.getUint32(4, true), nh = dv.getUint32(16, true); let pos = 20;
+ const cb = [], cbb = []; for (let i = 0; i < 16; i++) cb.push(dv.getFloat32(pos + 4 * i, true)); pos += 64;
+ for (let i = 0; i < 16; i++) cbb.push(dv.getFloat32(pos + 4 * i, true)); pos += 64;
+ let cnt, delta; [cnt, pos] = readVarints(b, N + 1, pos); [delta, pos] = readVarints(b, nbg, pos);
+ const q = unpack4(b.subarray(pos, pos + ((nbg + 1) >> 1)), nbg); pos += (nbg + 1) >> 1;
+ const qb = unpack4(b.subarray(pos, pos + ((nh + 1) >> 1)), nh);
+ this.off = new Uint32Array(N + 2); this.succ = new Uint32Array(nbg); this.bq = q; this.cb = Float32Array.from(cb);
+ this.bow = new Float32Array(N + 1); let i = 0, hi = 0;
+ for (let h = 0; h <= N; h++) {
+ this.off[h] = i; const n = cnt[h]; if (!n) continue;
+ this.bow[h] = cbb[qb[hi++]]; let acc = -1;
+ for (let k = 0; k < n; k++) { acc = acc < 0 ? delta[i] : acc + delta[i] + 1; this.succ[i++] = acc; }
+ }
+ this.off[N + 1] = i; this.hasLM = nbg > 0;
+ }
+ unigram(wid) { return wid === null ? this.unk : wid === this.N ? Math.log(0.04) : this.uni[wid]; }
+ /** ln P(wid | prev); prev null = sentence start; wid null = OOV; wid N = */
+ logp(prev, wid) {
+ const h = prev === null ? this.N : prev;
+ if (wid !== null && this.hasLM) {
+ let lo = this.off[h], hi = this.off[h + 1];
+ while (lo < hi) { const m = (lo + hi) >> 1; if (this.succ[m] < wid) lo = m + 1; else hi = m; }
+ if (lo < this.off[h + 1] && this.succ[lo] === wid) return this.cb[this.bq[lo]];
+ }
+ return (this.hasLM ? this.bow[h] : 0) + this.unigram(wid);
+ }
+}
+
+/** lazy lexicon trie over the sorted word list (same node semantics as bhc/lextrie.py) */
+class LazyTrie {
+ constructor(sortedWords) { this.w = sortedWords; this.nodes = [[0, sortedWords.length, 0]]; this.ids = new Map([["0,0," + sortedWords.length, 0]]); this.cache = new Map(); this.allow = new Map(); }
+ _bisect(x, lo, hi) { const w = this.w; while (lo < hi) { const m = (lo + hi) >> 1; if (w[m] < x) lo = m + 1; else hi = m; } return lo; }
+ children(n) {
+ let c = this.cache.get(n); if (c) return c;
+ const [lo, hi, d] = this.nodes[n], w = this.w; c = new Map(); let pos = lo;
+ if (pos < hi && w[pos].length === d) pos++;
+ while (pos < hi) {
+ const ch = w[pos].charCodeAt(d), pre = w[pos].slice(0, d);
+ const end = this._bisect(pre + String.fromCharCode(ch + 1), pos, hi);
+ const key = pos + "," + (d + 1) + "," + end; let nid = this.ids.get(key);
+ if (nid === undefined) { nid = this.nodes.length; this.nodes.push([pos, end, d + 1]); this.ids.set(key, nid); }
+ c.set(ch - 0x900 + 4, nid); pos = end;
+ }
+ if (this.cache.size > 400000) { this.cache.clear(); this.allow.clear(); }
+ this.cache.set(n, c); return c;
+ }
+ final(n) { const [lo, , d] = this.nodes[n]; return lo < this.w.length && this.w[lo].length === d; }
+ allowed(n) { let a = this.allow.get(n); if (!a) { a = [...this.children(n).keys()]; if (this.final(n)) a.push(EOS); this.allow.set(n, a); } return a; }
+ child(n, v) { return this.children(n).get(v); }
+}
+
+// ------------------------------------------------------------------ language ID (char-CNN + BiGRU)
+const LID_CH = {}; [..."abcdefghijklmnopqrstuvwxyz'"].forEach((c, i) => (LID_CH[c] = i + 3));
+const WMAX = 16, MODES3 = ["english", "romanized_hindi", "hinglish"];
+const RE_LATIN_APOS = /^[A-Za-z]+(?:'[A-Za-z]+)?$/, RE_DEVA = /^[\u0900-\u0963\u0971-\u097F]+$/;
+function whash(w, n) { let h = 2166136261; for (const ch of new TextEncoder().encode(w.toLowerCase())) h = Math.imul(h ^ ch, 16777619) >>> 0; return h % n; }
+
+export class LID {
+ constructor(w) {
+ this.cfg = w.cfg; this.x = w.extra || {}; const g = (k) => w.T[k] && w.T[k].data;
+ Object.assign(this, { ce: g("ce.weight"), c1: g("c1.weight"), c1b: g("c1.bias"), c2: g("c2.weight"), c2b: g("c2.bias"), wemb: g("wemb.weight"),
+ wih: g("gru.weight_ih_l0"), whh: g("gru.weight_hh_l0"), bih: g("gru.bias_ih_l0"), bhh: g("gru.bias_hh_l0"),
+ wihR: g("gru.weight_ih_l0_reverse"), whhR: g("gru.weight_hh_l0_reverse"), bihR: g("gru.bias_ih_l0_reverse"), bhhR: g("gru.bias_hh_l0_reverse"),
+ tw: g("tok.weight"), tb: g("tok.bias"), s0: g("sent.0.weight"), s0b: g("sent.0.bias"), s2: g("sent.2.weight"), s2b: g("sent.2.bias") });
+ this.nParams = Object.values(w.T).reduce((a, t) => a + t.data.length, 0);
+ }
+ _wvec(word) {
+ const { ce: E, cf } = this.cfg; const ids = new Array(WMAX).fill(0); const w = [...word.toLowerCase()].slice(0, WMAX - 2);
+ ids[0] = 1; w.forEach((c, i) => (ids[i + 1] = LID_CH[c] ?? 2)); ids[w.length + 1] = 2;
+ const x = new Float32Array(E * WMAX); for (let t = 0; t < WMAX; t++) for (let i = 0; i < E; i++) x[i * WMAX + t] = this.ce[ids[t] * E + i];
+ const conv = (inp, cin, W, b) => { const o = new Float32Array(cf * WMAX);
+ for (let oc = 0; oc < cf; oc++) for (let t = 0; t < WMAX; t++) { let s = b[oc];
+ for (let ic = 0; ic < cin; ic++) for (let k = 0; k < 3; k++) { const tt = t + k - 1; if (tt >= 0 && tt < WMAX) s += W[(oc * cin + ic) * 3 + k] * inp[ic * WMAX + tt]; }
+ o[oc * WMAX + t] = s > 0 ? s : 0; } return o; };
+ const h1 = conv(x, E, this.c1, this.c1b), h2 = conv(h1, cf, this.c2, this.c2b);
+ const wh = this.cfg.wh, wd = this.cfg.wd, v = new Float32Array(cf + (wh ? wd : 0));
+ for (let oc = 0; oc < cf; oc++) { let m = -Infinity; for (let t = 0; t < WMAX; t++) { const val = ids[t] === 0 ? -1e4 : h2[oc * WMAX + t] + h1[oc * WMAX + t]; if (val > m) m = val; } v[oc] = m; }
+ if (wh) { const r = whash(word, wh); for (let i = 0; i < wd; i++) v[cf + i] = this.wemb[r * wd + i]; }
+ return v;
+ }
+ _gru(xs, Wih, Whh, bih, bhh, rev) {
+ const H = this.cfg.hid, nin = xs[0].length, T = xs.length, out = new Array(T); let h = new Float32Array(H);
+ const gi = new Float32Array(3 * H), gh = new Float32Array(3 * H);
+ for (let s = 0; s < T; s++) {
+ const t = rev ? T - 1 - s : s; linear(xs[t], 0, Wih, bih, gi, 0, nin, 3 * H); linear(h, 0, Whh, bhh, gh, 0, H, 3 * H);
+ const nh = new Float32Array(H);
+ for (let i = 0; i < H; i++) { const r = sigmoid(gi[i] + gh[i]), z = sigmoid(gi[H + i] + gh[H + i]), n = Math.tanh(gi[2 * H + i] + r * gh[2 * H + i]); nh[i] = (1 - z) * n + z * h[i]; }
+ h = nh; out[t] = h;
+ }
+ return out;
+ }
+ /** Latin words -> {pen: P(EN) per word, ps: [english, romanized_hindi, hinglish]} */
+ run(words) {
+ if (!words.length) return { pen: [], ps: null };
+ const H = this.cfg.hid, xs = words.map((w) => this._wvec(w)), T = xs.length, nin = xs[0].length;
+ const f = this._gru(xs, this.wih, this.whh, this.bih, this.bhh, false), r = this._gru(xs, this.wihR, this.whhR, this.bihR, this.bhhR, true);
+ const pen = [], hm = new Float32Array(2 * H), hx = new Float32Array(2 * H).fill(-1e4), z = new Float32Array(2 * H + nin), lo = new Float32Array(2);
+ for (let t = 0; t < T; t++) {
+ for (let i = 0; i < H; i++) { z[i] = f[t][i]; z[H + i] = r[t][i]; } z.set(xs[t], 2 * H);
+ linear(z, 0, this.tw, this.tb, lo, 0, 2 * H + nin, 2); const m = Math.max(lo[0], lo[1]), e0 = Math.exp(lo[0] - m), e1 = Math.exp(lo[1] - m); pen.push(e0 / (e0 + e1));
+ for (let i = 0; i < 2 * H; i++) { hm[i] += z[i] / T; if (z[i] > hx[i]) hx[i] = z[i]; }
+ }
+ const sin = new Float32Array(4 * H + 1); sin.set(hm, 0); sin.set(hx, 2 * H); sin[4 * H] = pen.reduce((a, b) => a + b, 0) / T;
+ const h1 = new Float32Array(32); linear(sin, 0, this.s0, this.s0b, h1, 0, 4 * H + 1, 32); for (let i = 0; i < 32; i++) h1[i] = Math.max(0, h1[i]);
+ const s = new Float32Array(3); linear(h1, 0, this.s2, this.s2b, s, 0, 32, 3); const m = Math.max(...s); const e = [...s].map((v) => Math.exp(v - m)); const Z = e.reduce((a, b) => a + b, 0);
+ return { pen, ps: e.map((v) => v / Z) };
+ }
+ tag(tokens) {
+ const lat = tokens.filter((t) => RE_LATIN_APOS.test(t)); const { pen } = this.run(lat); let k = 0; const th = this.x.tok_th ?? 0.5;
+ return tokens.map((t) => (RE_DEVA.test(t) ? "DEV" : RE_LATIN_APOS.test(t) ? (pen[k++] >= th ? "EN" : "HI") : "OTHER"));
+ }
+ mode(tokens) {
+ const nd = tokens.filter((t) => RE_DEVA.test(t)).length, nl = tokens.filter((t) => RE_LATIN_APOS.test(t)).length;
+ if (nd && !nl) return "devanagari";
+ if (nd && nl) return nd / (nd + nl) >= (this.x.dev_frac ?? 0.5) ? "devanagari" : "mixed_script";
+ if (!nl) return "english";
+ const { ps } = this.run(tokens.filter((t) => RE_LATIN_APOS.test(t))); const bias = this.x.mode_bias || [0, 0, 0];
+ let best = 0, bv = -Infinity; ps.forEach((p, i) => { const v = Math.log(p + 1e-9) + bias[i]; if (v > bv) { bv = v; best = i; } });
+ return MODES3[best];
+ }
+}
+
+// ------------------------------------------------------------------ tokenizer + policy tables
+export const TOKEN_RE = /(https?:\/\/\S+|www\.\S+|[\p{L}\p{N}_.+-]+@[\p{L}\p{N}_-]+\.[\p{L}\p{N}_.]+|[@#][\p{L}\p{N}_]+|[A-Za-z]+(?:'[A-Za-z]+)?|[\u0900-\u0963\u0971-\u097F]+|[0-9\u0966-\u096F]+(?:[.,:][0-9\u0966-\u096F]+)*|[\u0964\u0965]|[^\sA-Za-z0-9\u0900-\u097F])/gu;
+const RE_LATIN = /^[A-Za-z]+$/;
+const RE_ACRO = /^[b-df-hj-np-tv-z]{2,7}$/;
+const LETTER = Object.fromEntries([..."abcdefghijklmnopqrstuvwxyz"].map((c, i) => [c, ("\u090f \u092c\u0940 \u0938\u0940 \u0921\u0940 \u0908 \u090f\u092b \u091c\u0940 \u090f\u091a \u0906\u0908 \u091c\u0947 \u0915\u0947 \u090f\u0932 \u090f\u092e \u090f\u0928 " +
+ "\u0913 \u092a\u0940 \u0915\u094d\u092f\u0942 \u0906\u0930 \u090f\u0938 \u091f\u0940 \u092f\u0942 \u0935\u0940 \u0921\u092c\u094d\u0932\u094d\u092f\u0942 \u090f\u0915\u094d\u0938 \u0935\u093e\u0908 \u091c\u0947\u0921").split(" ")[i]]));
+// Hinglish policy letter names (bhc/hinglish baseline): f -> एफ़, z -> ज़ेड
+const LETTERS_H = { ...LETTER, f: "\u090f\u092b\u093c", z: "\u091c\u093c\u0947\u0921" };
+const VAR_SWAPS = [["\u0949", "\u094b"], ["\u094b", "\u0949"], ["\u093e", "\u0949"], ["\u0949", "\u093e"], ["\u0903", ""], ["\u092b\u093c", "\u092b"], ["\u092b", "\u092b\u093c"],
+ ["\u091c\u093c", "\u091c"], ["\u091c", "\u091c\u093c"], ["\u0901", "\u0902"], ["\u0902", "\u0901"]];
+export const FEATS = ["sc", "scn", "inlex", "uni", "free0", "infree", "incon", "aw", "visarga", "nukta", "chandra", "acro", "variant", "frank",
+ "skel", "skmatch", "lev", "sc_rel", "scn_rel", "uni_rel", "lenr", "visr", "hfin", "acro_nv",
+ "ck1", "ck2", "ckr", "abbr", "abbr_r", "novow"]; // v2 casual features (append-only; older params use the first mu.length)
+const BRANDS = new Set(`whatsapp google meet paytm amazon flipkart youtube instagram facebook gmail excel uber ola swiggy zomato
+netflix linkedin github python java jira slack zoom iphone android windows microsoft apple samsung twitter telegram
+chrome firefox safari office365 outlook teams spotify hotstar jio airtel phonepe gpay`.split(/\s+/));
+const KEEP_RE = /^(https?:\/\/|www\.|[@#])|@.+\.|_|\d.*[A-Za-z]|[A-Za-z].*\d|^[a-z]+[A-Z]/;
+const isUpper = (t) => /[A-Z]/.test(t) && t === t.toUpperCase();
+const DEFAULTS = { beam: 8, nbest: 8, cbeam: 8, variants: true, acro: true, w: null, mu: null, sd: null, w_lm: 0.8, oov_pen_s: 4.0, w_ch_s: 1.0, use_lm: true,
+ skel: false, skel_k: 200, skel_m: 16, skel_wu: 0.03, visr: false, acro_v: false,
+ casual: false, cas_top: 300000, cas_n1: 6, cas_n2: 4, abbr: false, squeeze: 0, k1gate: 99, k2gate: 99 };
+const parseTSV = (txt) => { const m = new Map(); for (const l of txt.split("\n")) { const i = l.indexOf("\t"); if (i > 0) m.set(l.slice(0, i).trim(), l.slice(i + 1).trim()); } return m; };
+const now = () => (typeof performance !== "undefined" ? performance.now() : Date.now());
+
+// Sizes are checked against the manifest, never against Content-Length (a CDN redirect or an auth error page has its
+// own, unrelated length); a wrong-size body (e.g. an HTML/plain-text error page served with 200) is rejected.
+const checkSize = (b, expect, url) => { if (expect && b.length !== expect) throw new Error(`fetch ${url}: got ${b.length} bytes, manifest says ${expect}`); return b; };
+async function fetchBytes(base, name, fetchImpl, expect = 0) {
+ const url = base.endsWith("/") ? base + name : base + "/" + name;
+ const r = await fetchImpl(url); if (!r.ok) throw new Error(`fetch ${url}: ${r.status}`);
+ if (name.endsWith(".gz") && typeof DecompressionStream !== "undefined") return new Uint8Array(await new Response(r.body.pipeThrough(new DecompressionStream("gzip"))).arrayBuffer());
+ return checkSize(new Uint8Array(await r.arrayBuffer()), expect, url);
+}
+
+// ------------------------------------------------------------------ consonant-skeleton index (bhc/skel.py; tables generated from it)
+const SK_DEV = {"\u0915": "K", "\u0916": "K", "\u0917": "G", "\u0918": "G", "\u091a": "C", "\u091b": "C", "\u091c": "J", "\u091d": "J", "\u091f": "T", "\u0920": "T", "\u0924": "T", "\u0925": "T", "\u0921": "D", "\u0922": "D", "\u0926": "D", "\u0927": "D", "\u0919": "N", "\u091e": "N", "\u0923": "N", "\u0928": "N", "\u0929": "N", "\u092e": "N", "\u0902": "N", "\u0901": "N", "\u092a": "P", "\u092b": "P", "\u092c": "B", "\u092d": "B", "\u0930": "R", "\u0931": "R", "\u090b": "R", "\u0943": "R", "\u0932": "L", "\u0933": "L", "\u0935": "V", "\u0936": "S", "\u0937": "S", "\u0938": "S", "\u0939": "H", "\u0903": "H"};
+const SK_ROM1 = {"k": "K", "q": "K", "g": "G", "j": "J", "z": "J", "t": "T", "d": "D", "n": "N", "m": "N", "p": "P", "f": "P", "b": "B", "r": "R", "l": "L", "v": "V", "w": "V", "s": "S", "h": "H"};
+const SK_ASP = "bcdgjkpstz";
+const DR_C = {"\u0915": "k", "\u0916": "kh", "\u0917": "g", "\u0918": "gh", "\u0919": "n", "\u091a": "ch", "\u091b": "chh", "\u091c": "j", "\u091d": "jh", "\u091e": "n", "\u091f": "t", "\u0920": "th", "\u0921": "d", "\u0922": "dh", "\u0923": "n", "\u0924": "t", "\u0925": "th", "\u0926": "d", "\u0927": "dh", "\u0928": "n", "\u092a": "p", "\u092b": "ph", "\u092c": "b", "\u092d": "bh", "\u092e": "m", "\u092f": "y", "\u0930": "r", "\u0932": "l", "\u0935": "v", "\u0936": "sh", "\u0937": "sh", "\u0938": "s", "\u0939": "h", "\u0933": "l"};
+const DR_V = {"\u0905": "a", "\u0906": "aa", "\u0907": "i", "\u0908": "ee", "\u0909": "u", "\u090a": "oo", "\u090b": "ri", "\u090f": "e", "\u0910": "ai", "\u0913": "o", "\u0914": "au", "\u0911": "o"};
+const DR_M = {"\u093e": "a", "\u093f": "i", "\u0940": "ee", "\u0941": "u", "\u0942": "oo", "\u0943": "ri", "\u0947": "e", "\u0948": "ai", "\u094b": "o", "\u094c": "au", "\u0949": "o", "\u0945": "e"};
+const NUKTA_ROM = { k: "q", kh: "kh", g: "g", j: "z", ph: "f", d: "r", dh: "rh" };
+export function devSkel(w) { let o = ""; for (const ch of w) { const c = SK_DEV[ch]; if (c && o[o.length - 1] !== c) o += c; } return o; }
+export function romSkels(w, maxk = 4) {
+ w = w.toLowerCase(); let outs = [""]; let i = 0; const n = w.length;
+ while (i < n) {
+ const ch = w[i]; let step = 1, opts;
+ if ((ch === "t" || ch === "s") && w.startsWith("ion", i + 1)) { opts = ch === "t" ? ["SN", "TN"] : ["SN"]; step = 4; }
+ else if (ch === "c") {
+ const nx = i + 1 < n ? w[i + 1] : "";
+ if (nx === "h") { opts = ["C", "K"]; step = 2; while (i + step < n && (w[i + step] === "c" || w[i + step] === "h")) step++; }
+ else if (nx === "e" || nx === "i" || nx === "y") opts = ["S", "K"];
+ else if (nx === "k") { opts = ["K"]; step = 2; }
+ else opts = ["K", "S"];
+ } else if (ch === "x") opts = ["KS"];
+ else if (SK_ASP.includes(ch) && i + 1 < n && w[i + 1] === "h") { opts = [SK_ROM1[ch] || ""]; step = 2; while (i + step < n && w[i + step] === "h") step++; }
+ else opts = [SK_ROM1[ch] || ""];
+ const nw = [];
+ for (const o of outs) for (const q of opts) { let s = o; for (const c of q) if (!s || s[s.length - 1] !== c) s += c; nw.push(s); }
+ outs = [...new Set(nw)].slice(0, maxk); i += step;
+ }
+ return outs;
+}
+export function devRom(w) {
+ const out = [], a = [...w], L = a.length;
+ for (let j = 0; j < L; j++) {
+ const ch = a[j];
+ if (DR_C[ch] !== undefined) {
+ out.push(DR_C[ch]); let nx = j + 1 < L ? a[j + 1] : "";
+ if (nx === "\u093c") nx = j + 2 < L ? a[j + 2] : "";
+ if (nx && DR_M[nx] === undefined && nx !== "\u094d" && j + 1 < L - 1) out.push("a");
+ } else if (DR_V[ch] !== undefined) out.push(DR_V[ch]);
+ else if (DR_M[ch] !== undefined) out.push(DR_M[ch]);
+ else if (ch === "\u0902" || ch === "\u0901") out.push("n");
+ else if (ch === "\u0903") out.push("h");
+ else if (ch === "\u093c" && out.length) { const l = out[out.length - 1]; out[out.length - 1] = NUKTA_ROM[l] ?? l; }
+ }
+ return out.join("");
+}
+export function lev(a, b) {
+ if (a.length < b.length) [a, b] = [b, a];
+ let prev = Array.from({ length: b.length + 1 }, (_, j) => j);
+ for (let i = 1; i <= a.length; i++) {
+ const cur = [i];
+ for (let j = 1; j <= b.length; j++) cur.push(Math.min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (a[i - 1] !== b[j - 1] ? 1 : 0)));
+ prev = cur;
+ }
+ return prev[b.length];
+}
+class SkelIndex {
+ constructor(lm) {
+ const b = new Map(), N = lm.words.length;
+ for (let i = 0; i < N; i++) { const k = devSkel(lm.words[i]); if (k) { let v = b.get(k); if (!v) b.set(k, (v = [])); v.push(i); } }
+ for (const v of b.values()) v.sort((x, y) => (lm.uni[y] - lm.uni[x]) || (x - y));
+ this.b = b;
+ }
+ lookup(roman, topk = 24) {
+ const seen = [], S = new Set();
+ for (const k of romSkels(roman)) { const v = this.b.get(k); if (!v) continue; for (const i of v.slice(0, topk)) if (!S.has(i)) { S.add(i); seen.push(i); } }
+ return seen;
+ }
+}
+// ------------------------------------------------------------------ casual / chat-romanization keys (bhc/casual.py; keep 1:1)
+const CAS_DC = {"\u0915": "k", "\u0916": "k", "\u0917": "g", "\u0918": "g", "\u091a": "c", "\u091b": "c", "\u091c": "j", "\u091d": "j", "\u091f": "t", "\u0920": "t", "\u0924": "t", "\u0925": "t", "\u0921": "d", "\u0922": "d", "\u0926": "d", "\u0927": "d", "\u0923": "n", "\u0928": "n", "\u0929": "n", "\u092a": "p", "\u092b": "p", "\u092c": "b", "\u092d": "b", "\u092e": "m", "\u092f": "y", "\u0930": "r", "\u0931": "r", "\u0932": "l", "\u0933": "l", "\u0935": "v", "\u0936": "s", "\u0937": "s", "\u0938": "s", "\u0939": "h", "\u0958": "k", "\u0959": "k", "\u095a": "g", "\u095b": "j", "\u095c": "R", "\u095d": "R", "\u095e": "p", "\u095f": "y"};
+const CAS_DV = {"\u0905": "a", "\u0906": "a", "\u093e": "a", "\u0907": "i", "\u0908": "i", "\u093f": "i", "\u0940": "i", "\u0909": "u", "\u090a": "u", "\u0941": "u", "\u0942": "u", "\u090f": "e", "\u0910": "e", "\u0947": "e", "\u0948": "e", "\u0945": "e", "\u090d": "e", "\u0913": "o", "\u0914": "o", "\u0911": "o", "\u094b": "o", "\u094c": "o", "\u0949": "o", "\u090b": "r", "\u0943": "r"};
+export const ABBR = {"h": ["\u0939\u0948", "\u0939\u0948\u0902"], "hy": ["\u0939\u0948"], "hn": ["\u0939\u093e\u0901", "\u0939\u0948\u0902"], "hm": ["\u0939\u092e", "\u0939\u092e\u094d\u092e"], "k": ["\u0915\u0947", "\u0915\u0940", "\u0913\u0915\u0947"], "ki": ["\u0915\u0940", "\u0915\u093f"], "n": ["\u0928\u093e", "\u0928"], "ni": ["\u0928\u0939\u0940\u0902", "\u0928\u0940"], "nai": ["\u0928\u0939\u0940\u0902", "\u0928\u0908"], "nhi": ["\u0928\u0939\u0940\u0902"], "nhe": ["\u0928\u0939\u0940\u0902"], "nahi": ["\u0928\u0939\u0940\u0902"], "kr": ["\u0915\u0930"], "krna": ["\u0915\u0930\u0928\u093e"], "kro": ["\u0915\u0930\u094b"], "tm": ["\u0924\u0941\u092e"], "tu": ["\u0924\u0942"], "mai": ["\u092e\u0948\u0902", "\u092e\u0947\u0902"], "me": ["\u092e\u0947\u0902", "\u092e\u0948\u0902"], "m": ["\u092e\u0948\u0902", "\u092e\u0947\u0902"], "thk": ["\u0920\u0940\u0915"], "tk": ["\u0920\u0940\u0915", "\u0924\u0915"], "yr": ["\u092f\u093e\u0930"], "yrr": ["\u092f\u093e\u0930"], "plz": ["\u092a\u094d\u0932\u0940\u091c\u093c"], "pls": ["\u092a\u094d\u0932\u0940\u091c\u093c"], "plzz": ["\u092a\u094d\u0932\u0940\u091c\u093c"], "tq": ["\u0925\u0948\u0902\u0915 \u092f\u0942"], "ty": ["\u0925\u0948\u0902\u0915 \u092f\u0942"], "thx": ["\u0925\u0948\u0902\u0915\u094d\u0938"], "thnx": ["\u0925\u0948\u0902\u0915\u094d\u0938"], "thnks": ["\u0925\u0948\u0902\u0915\u094d\u0938"], "gn": ["\u0917\u0941\u0921 \u0928\u093e\u0907\u091f"], "gm": ["\u0917\u0941\u0921 \u092e\u0949\u0930\u094d\u0928\u093f\u0902\u0917"], "ok": ["\u0913\u0915\u0947"], "okk": ["\u0913\u0915\u0947"], "kk": ["\u0913\u0915\u0947"], "sry": ["\u0938\u0949\u0930\u0940"], "bro": ["\u092c\u094d\u0930\u094b"], "msg": ["\u092e\u0948\u0938\u0947\u091c"], "bday": ["\u092c\u0930\u094d\u0925\u0921\u0947"], "clg": ["\u0915\u0949\u0932\u0947\u091c"], "ofc": ["\u0911\u092b\u093c\u093f\u0938"], "pic": ["\u092a\u093f\u0915"], "bhai": ["\u092d\u093e\u0908"], "bht": ["\u092c\u0939\u0941\u0924"], "bhot": ["\u092c\u0939\u0941\u0924"], "bohot": ["\u092c\u0939\u0941\u0924"], "pta": ["\u092a\u0924\u093e"], "kb": ["\u0915\u092c"], "jb": ["\u091c\u092c"], "sb": ["\u0938\u092c"], "mtlb": ["\u092e\u0924\u0932\u092c"], "kch": ["\u0915\u0941\u091b"], "kuch": ["\u0915\u0941\u091b"]};
+const CAS_VOW = new Set(["a", "e", "i", "o", "u"]), CAS_ASP = new Set([..."kgjtdpbc"]);
+function casFin(s) { let o = ""; for (let i = 0; i < s.length; i++) { const ch = s[i]; if (ch === "a" && i > 0) continue; if (o && o[o.length - 1] === ch) continue; o += ch; } return o; }
+export function k2Of(k1) {
+ if (!k1) return "";
+ let s = k1[0]; for (const c of k1.slice(1)) if (!CAS_VOW.has(c)) s += c;
+ if (s && CAS_VOW.has(s[0])) s = s.slice(1) || s;
+ let o = ""; for (const ch of s) if (!(o && o[o.length - 1] === ch)) o += ch; return o;
+}
+export function devKeys(w) {
+ const a = [...w], raw = [];
+ for (let i = 0; i < a.length; i++) {
+ const ch = a[i];
+ if (ch === "\u093c") { if (raw.length && raw[raw.length - 1] === "d" && i > 0 && (a[i - 1] === "\u0921" || a[i - 1] === "\u0922")) raw[raw.length - 1] = "R"; continue; }
+ if (CAS_DC[ch] !== undefined) raw.push(CAS_DC[ch]); else if (CAS_DV[ch] !== undefined) raw.push(CAS_DV[ch]);
+ }
+ const s = raw.join(""); if (!s) return [];
+ const ks = [...new Set([casFin(s.split("R").join("d")), casFin(s.split("R").join("r"))])].filter((k) => k);
+ return ks.sort();
+}
+export function romKeys(t, cap = 12) {
+ t = t.toLowerCase().replace(/[^a-z]/g, ""); if (!t) return [[], []];
+ t = t.replace(/(.)\1{2,}/g, "$1");
+ t = t.split("chh").join("c").split("cch").join("c").split("ch").join("c").split("sh").join("s");
+ t = t.split("x").join("ks").split("q").join("k").split("z").join("j").split("f").join("p").split("w").join("v");
+ const alts = [], L = t.length, V = (c) => CAS_VOW.has(c); let i = 0;
+ while (i < L) {
+ const c = t[i], nx = i + 1 < L ? t[i + 1] : "", prev = i > 0 ? t[i - 1] : "";
+ if (c === "h" && CAS_ASP.has(prev) && i > 0) { alts.push(["", "h"]); i += 1; continue; }
+ if (nx && (c === "a" || c === "e") && (nx === "i" || nx === "y") && (i + 2 >= L || !V(t[i + 2]))) {
+ if (c === "a" && nx === "i") { alts.push(["e", "ai"]); i += 2; continue; }
+ if (c === "a" && nx === "y" && i + 2 >= L) { alts.push(["e"]); i += 2; continue; }
+ if (c === "e" && nx === "y" && i + 2 >= L) { alts.push(["e"]); i += 2; continue; }
+ }
+ if (nx && (c === "a" || c === "o") && "uvw".includes(nx) && (i + 2 >= L || !V(t[i + 2])) && c + nx !== "av") { alts.push(["o", c + "u"]); i += 2; continue; }
+ if (c === "o" && nx === "u") { alts.push(["o", "u"]); i += 2; continue; }
+ if (c === "e" && nx === "e") { alts.push(["i"]); i += 2; continue; }
+ if (c === "o" && nx === "o") { alts.push(["u", "o"]); i += 2; continue; }
+ if (c === "e" && nx === "i" && (i + 2 >= L || !V(t[i + 2]))) { alts.push(["e"]); i += 2; continue; }
+ if (c === "e" && nx === "h" && i + 2 < L && !V(t[i + 2]) && i > 0) { alts.push(["e", ""]); i += 1; continue; }
+ if (nx && c === "i" && "aeou".includes(nx) && i > 0) { alts.push(["y", "i"]); i += 1; continue; }
+ if (c === "y" && i === L - 1 && prev && !V(prev)) { alts.push(["e", "i"]); i += 1; continue; }
+ if (c === "n" && V(prev) && i > 0 && (i === L - 1 || (nx && !V(nx) && nx !== "y"))) { alts.push(["n", ""]); i += 1; continue; }
+ if ((c === "i" || c === "u") && prev === "r" && i >= 2 && !V(t[i - 2])) { alts.push([c, ""]); i += 1; continue; }
+ alts.push([c]); i += 1;
+ }
+ let outs = [""];
+ for (const a of alts) { const nxt = []; for (const o of outs) for (const x of a) nxt.push(o + x); outs = [...new Set(nxt)].slice(0, 64); }
+ const k1 = [], seen = new Set();
+ for (const o of outs) { const k = casFin(o); if (k && !seen.has(k)) { seen.add(k); k1.push(k); } }
+ const k1c = k1.slice(0, cap); const k2 = [...new Set(k1c.map(k2Of).filter((k) => k))];
+ return [k1c, k2];
+}
+export function squeezeIn(t, mode = 0) {
+ if (!mode) return t;
+ if (mode === 1) return t.replace(/([a-z])\1{2,}/g, "$1");
+ return t.replace(/([a-z])\1{2,}/g, (m, c) => (/[aeo]/.test(c) ? c + c : c));
+}
+class CasualIndex {
+ constructor(lm, topN = 300000, perKey = 8) {
+ const N = lm.words.length, uni = lm.uni, order = Array.from({ length: N }, (_, i) => i);
+ order.sort((x, y) => (uni[y] - uni[x]) || (x - y));
+ this.k1 = new Map(); this.k2 = new Map();
+ const put = (m, k, i) => { let b = m.get(k); if (!b) m.set(k, (b = [])); if (b.length < perKey && !b.includes(i)) b.push(i); };
+ for (let r = 0; r < Math.min(N, topN); r++) { const i = order[r]; for (const k of devKeys(lm.words[i])) { put(this.k1, k, i); const k2 = k2Of(k); if (k2) put(this.k2, k2, i); } }
+ }
+ lookup(roman, n1 = 6, n2 = 4) {
+ const [k1, k2] = romKeys(roman), r1 = [], r2 = [];
+ for (const k of k1) for (const i of this.k1.get(k) || []) if (!r1.includes(i)) r1.push(i);
+ for (const k of k2) for (const i of this.k2.get(k) || []) if (!r2.includes(i)) r2.push(i);
+ return [r1.slice(0, n1 * 2), r2.slice(0, n2 * 2)];
+ }
+}
+const VIS_RE = /[aeiou]h$/;
+
+// ------------------------------------------------------------------ the engine
+export class BhashaCore {
+ /**
+ * Load a bundle (directory with manifest.json) from a URL (browser / worker) or via a custom fetch (Node: file loader).
+ * dict: true (default) = also load the dictionary pack and wait for it; false = core only (call loadDict() later).
+ * cache: Cache API cache name for the dictionary pack (browser / worker; default "aksharalp-dict"), or false.
+ */
+ static async load(base, { fetch: fetchImpl = globalThis.fetch, onProgress, dict = true, cache = "aksharalp-dict" } = {}) {
+ const t0 = now(); const txt = (b) => new TextDecoder().decode(b);
+ let man; try { man = JSON.parse(txt(await fetchBytes(base, "manifest.json", fetchImpl))); } catch { man = { files: { model: "xlit.q4", lex: "lex.bin", lm: "lm.bin", params: "params.json", lid: "lid.int8", loan: "loan.tsv", natural: "natural_hi.tsv" } }; }
+ const F = man.files, SZ = man.sizes || {}, get = async (n, opt) => { if (!n) return null; try { const b = await fetchBytes(base, n, fetchImpl, SZ[n] && SZ[n].bytes); onProgress && onProgress(n); return b; } catch (e) { if (opt) return null; throw e; } };
+ const [mb, lexb, lmb, pb, lidb, loanb, natb] = await Promise.all([get(F.model), get(F.lex, true), get(F.lm, true), get(F.params, true), get(F.lid, true), get(F.loan, true), get(F.natural, true)]);
+ const e = new BhashaCore({ model: mb, lex: lexb, lm: lmb, params: pb ? JSON.parse(txt(pb)) : {}, lid: lidb, loan: loanb ? txt(loanb) : null, natural: natb ? txt(natb) : null, manifest: man });
+ e.base = base; e.fetchImpl = fetchImpl;
+ if (dict && man.packs && man.packs.dict) await e.loadDict({ onProgress, cache });
+ e.loadMs = now() - t0; return e;
+ }
+ /** Fetch (or read from cache) the dictionary pack and attach it. Returns true on success; on any failure the engine
+ * keeps working in core-only mode and the error is kept in this.dictError. */
+ async loadDict({ onProgress, cache = "aksharalp-dict" } = {}) {
+ const pk = this.manifest && this.manifest.packs && this.manifest.packs.dict; if (!pk) return false;
+ const dir = pk.dir.endsWith("/") ? pk.dir : pk.dir + "/", fetchImpl = this.fetchImpl || globalThis.fetch;
+ const useCache = cache && typeof caches !== "undefined";
+ const get = async (n) => {
+ if (!n) return null;
+ const ver = (pk.sizes && pk.sizes[n] && pk.sizes[n].sha256) || "", expect = (pk.sizes && pk.sizes[n] && pk.sizes[n].bytes) || 0;
+ if (useCache) {
+ try {
+ const url = new URL(dir + n, new URL(this.base.endsWith("/") ? this.base : this.base + "/", globalThis.location && globalThis.location.href)).href;
+ const c = await caches.open(cache), key = url + "?v=" + ver.slice(0, 16), hit = await c.match(key);
+ if (hit) { const hb = new Uint8Array(await hit.arrayBuffer()); if (!expect || hb.length === expect) { onProgress && onProgress(n, "cache"); return hb; } await c.delete(key); }
+ const r = await fetchImpl(url); if (!r.ok) throw new Error(`fetch ${url}: ${r.status}`);
+ const b = checkSize(new Uint8Array(await r.arrayBuffer()), expect, url); try { await c.put(key, new Response(b)); } catch (_) {} // stored as a fresh, non-redirected response
+ onProgress && onProgress(n, "network"); return b;
+ } catch (e) { if (String(e).includes("fetch ")) throw e; }
+ }
+ const b = await fetchBytes(this.base, dir + n, fetchImpl, expect); onProgress && onProgress(n, "network"); return b;
+ };
+ try {
+ const [lexb, lmb, pb] = await Promise.all([get(pk.files.lex), get(pk.files.lm), get(pk.files.params)]);
+ this.attachDict({ lex: lexb, lm: lmb, params: pb ? JSON.parse(new TextDecoder().decode(pb)) : {} });
+ this.dictInfo = { name: pk.name || "dict", words: this.lm.N, bytes: pk.bytes, gz: pk.gz }; return true;
+ } catch (e) { this.dictError = String(e && e.message || e); return false; }
+ }
+ /** attach an already-fetched dictionary pack: {lex, lm, params} (params override the core params) */
+ attachDict({ lex, lm = null, params = {} }) {
+ this.p = { ...DEFAULTS, ...this.coreParams, ...params };
+ this._setLex(lex, lm); this.cache = new Map(); this.hasDict = this.lm.N > 0;
+ }
+ _setLex(lex, lm) {
+ this.lm = new LexLM(lex, lm && this.p.use_lm ? lm : null);
+ if (!this.lm.N) { this.p.cbeam = 0; this.p.skel = false; }
+ this.cix = this.p.casual && this.lm.N ? new CasualIndex(this.lm, this.p.cas_top) : null;
+ // the trie never contains the empty string (as in Python's LazyTrie), otherwise "" would take a constrained-beam slot
+ const sw = this.lm.sortedWords, tw = sw.length && sw[0] === "" ? sw.slice(1) : sw;
+ this.trie = this.p.cbeam ? new LazyTrie(tw) : null;
+ this.skix = this.p.skel ? new SkelIndex(this.lm) : null;
+ }
+ constructor({ model, lex = null, lm = null, params = {}, lid = null, loan = null, natural = null, manifest = null }) {
+ this.coreParams = params; this.p = { ...DEFAULTS, ...params }; this.manifest = manifest;
+ this.model = new XlitModel(parseWeights(model));
+ this._setLex(lex, lm); this.hasDict = this.lm.N > 0; this.dictInfo = null; this.dictError = null;
+ this.lid = lid ? new LID(parseWeights(lid)) : null;
+ this.loan = loan ? parseTSV(loan) : new Map();
+ this.natural = natural ? parseTSV(natural) : new Map();
+ this.cache = new Map();
+ }
+ info() {
+ const s = this.manifest && this.manifest.sizes ? this.manifest.sizes : {};
+ const used = Object.values(this.manifest ? this.manifest.files : {}).filter((f) => f && s[f]);
+ return { params: this.model.nParams + (this.lid ? this.lid.nParams : 0), xlitParams: this.model.nParams, lidParams: this.lid ? this.lid.nParams : 0,
+ lexWords: this.lm.N, bytes: used.reduce((a, f) => a + s[f].bytes, 0), gz: used.reduce((a, f) => a + s[f].gz, 0), hasLID: !!this.lid, hasLM: this.lm.hasLM,
+ hasDict: this.hasDict, dict: this.dictInfo, dictError: this.dictError, hasLoan: this.loan.size > 0, hasNatural: this.natural.size > 0 };
+ }
+
+ /** candidate rows with reranker features for one (lower-cased) Latin word: [[devanagari, fv, wid], ...] */
+ candFeats(word) {
+ const p = this.p, w = squeezeIn(word.toLowerCase(), p.squeeze); const hit = this.cache.get(w); if (hit) return hit;
+ const m = this.model, ids = encSrc(w), enc = m.encode(ids), S = ids.length;
+ const free = m.beam(enc, S, { beam: p.beam, nbest: p.nbest });
+ const con = p.cbeam && this.trie ? m.beam(enc, S, { beam: p.beam, nbest: p.nbest, trie: this.trie }) : [];
+ const d = new Map();
+ free.forEach(([s, sc], i) => { if (s) d.set(s, { sc, infree: 1, frank: i, free0: +(i === 0), beamEntry: 1 }); });
+ for (const [s, sc] of con) if (s) { if (!d.has(s)) d.set(s, { sc, infree: 0, frank: p.nbest, free0: 0, beamEntry: 1 }); d.get(s).incon = 1; }
+ if (p.variants) {
+ const top = [...d.keys()].sort((a, b) => d.get(b).sc - d.get(a).sc).slice(0, 4);
+ for (const s of top) for (const [a, b] of VAR_SWAPS) if (s.includes(a)) {
+ const v = s.split(a).join(b); if (!d.has(v) && this.lm.id(v) !== null) d.set(v, { sc: m.scorePair(enc, v), infree: 0, frank: p.nbest, free0: 0, variant: 1 });
+ }
+ }
+ if (p.acro && (RE_ACRO.test(w) || (p.acro_v && w.length >= 2 && w.length <= 4))) {
+ const v = [...w].map((c) => LETTER[c]).join("");
+ if (!d.has(v)) d.set(v, { sc: m.scorePair(enc, v), infree: 0, frank: p.nbest, free0: 0, acro: 1 }); else d.get(v).acro = 1;
+ }
+ if (p.visr && VIS_RE.test(w)) { // Aksharantar convention: literal word-final h -> visarga
+ const top = [...d.keys()].filter((x) => d.get(x).beamEntry).sort((x, y) => d.get(y).sc - d.get(x).sc).slice(0, 2);
+ for (const s of top) {
+ const v = s.endsWith("\u0939") ? s.slice(0, -1) + "\u0903" : (!s.endsWith("\u0903") ? s + "\u0903" : null);
+ if (v && !d.has(v)) d.set(v, { sc: m.scorePair(enc, v), infree: 0, frank: p.nbest, free0: 0, visr: 1 });
+ }
+ }
+ if (this.skix) {
+ const L = w.length + 1, words = this.lm.words, uni = this.lm.uni;
+ const ids = this.skix.lookup(w, p.skel_k).map((i) => [i, lev(w, devRom(words[i])) / L - p.skel_wu * uni[i]])
+ .sort((x, y) => x[1] - y[1]).slice(0, p.skel_m);
+ for (const [i] of ids) { const v = words[i]; if (!d.has(v)) d.set(v, { sc: m.scorePair(enc, v), infree: 0, frank: p.nbest, free0: 0, skel: 1 }); else d.get(v).skel = 1; }
+ }
+ if (this.cix) {
+ const [r1, r2] = this.cix.lookup(w, p.cas_n1, p.cas_n2), words = this.lm.words, nv = (w.match(/[aeiou]/g) || []).length;
+ const ab = new Set(p.abbr && Object.prototype.hasOwnProperty.call(ABBR, w) ? ABBR[w] : []), s1 = new Set(r1.slice(0, p.cas_n1).map((i) => words[i]));
+ for (const [key, ids, n] of [["ck1", r1, p.cas_n1], ["ck2", r2, p.cas_n2]]) ids.slice(0, n).forEach((i, rk) => {
+ const v = words[i];
+ if (!d.has(v)) {
+ if (!ab.has(v) && ((s1.has(v) && nv > p.k1gate) || (!s1.has(v) && nv > p.k2gate))) return; // vowel gate
+ d.set(v, { sc: m.scorePair(enc, v), infree: 0, frank: p.nbest, free0: 0 });
+ }
+ const e = d.get(v); e[key] = 1; e.ckr = Math.min(e.ckr ?? 1, rk / n);
+ });
+ }
+ if (p.abbr && Object.prototype.hasOwnProperty.call(ABBR, w)) ABBR[w].forEach((v, rk) => {
+ if (!d.has(v)) d.set(v, { sc: m.scorePair(enc, v), infree: 0, frank: p.nbest, free0: 0 });
+ const e = d.get(v); e.abbr = 1; e.abbr_r = rk;
+ });
+ const rows = [], sks = new Set(romSkels(w)), Lw = w.length + 1, vis = VIS_RE.test(w), novow = !/[aeiou]/.test(w);
+ for (const [s, e] of d) {
+ const wid = this.lm.id(s), L = [...s].length;
+ rows.push([s, [e.sc, e.sc / (L + 1), +(wid !== null), wid !== null ? this.lm.unigram(wid) : 0, e.free0 || 0, e.infree || 0, e.incon || 0,
+ +(s.includes("\u0949") || s.includes("\u0911")), +s.includes("\u0903"), +s.includes("\u093c"), +s.includes("\u0901"), e.acro || 0, e.variant || 0, (e.frank ?? p.nbest) / p.nbest,
+ e.skel || 0, +sks.has(devSkel(s)), lev(w, devRom(s)) / Lw, 0, 0, 0, L / Lw, e.visr || 0, +(vis && s.endsWith("\u0903")), +(!!e.acro && novow),
+ e.ck1 || 0, e.ck2 || 0, e.ckr ?? 1, e.abbr || 0, e.abbr_r ?? 3, +!/[aeiouy]/.test(w)], wid]);
+ }
+ if (rows.length) {
+ let msc = -Infinity, mscn = -Infinity, mu = null;
+ for (const [, fv] of rows) { msc = Math.max(msc, fv[0]); mscn = Math.max(mscn, fv[1]); if (fv[2]) mu = mu === null ? fv[3] : Math.max(mu, fv[3]); }
+ for (const [, fv] of rows) { fv[17] = fv[0] - msc; fv[18] = fv[1] - mscn; fv[19] = fv[2] ? fv[3] - (mu ?? 0) : 0; }
+ }
+ if (this.cache.size > 20000) this.cache.clear();
+ this.cache.set(w, rows); return rows;
+ }
+ score(fv) {
+ const p = this.p; if (!p.w && !p.w1) return fv[0] + 0.2 * (fv[2] ? fv[3] : -25.0);
+ const n = p.mu.length, z = new Array(n); for (let i = 0; i < n; i++) z[i] = (fv[i] - p.mu[i]) / p.sd[i];
+ if (p.w1) { // 1-hidden-layer tanh MLP reranker (fit on VALID only)
+ let s = p.b2 || 0;
+ for (let h = 0; h < p.w1.length; h++) { const r = p.w1[h]; let a = p.b1[h]; for (let i = 0; i < n; i++) a += r[i] * z[i]; s += p.w2[h] * Math.tanh(a); }
+ return s;
+ }
+ let s = 0; for (let i = 0; i < Math.min(n, p.w.length); i++) s += p.w[i] * z[i]; return s;
+ }
+ /** word mode: top-k Devanagari candidates for a romanized word */
+ word(w, k = 8) {
+ if (!RE_LATIN.test(w)) return [{ text: w, score: 0 }];
+ const rows = this.candFeats(w).map(([s, fv]) => ({ text: s, score: this.score(fv) }));
+ return rows.sort((a, b) => b.score - a.score).slice(0, k);
+ }
+ /** sentence Viterbi over candidate lattices with the bigram LM (bhc/translit2.py decode_slots) */
+ decodeSlots(slots, context = true) {
+ const p = this.p, lm = this.lm, lat = [];
+ for (const [kind, t] of slots) {
+ if (kind === "x") {
+ let sc = this.candFeats(t).map(([s, fv, wid]) => [s, this.score(fv), wid]);
+ if (sc.length) {
+ const mx = Math.max(...sc.map((x) => x[1])), lse = mx + Math.log(sc.reduce((a, x) => a + Math.exp(x[1] - mx), 0));
+ sc = sc.map(([s, v, wid]) => [s, p.w_ch_s * (v - lse), wid]).sort((a, b) => b[1] - a[1]).slice(0, 12);
+ }
+ lat.push(sc.length ? sc : [[t, 0, null]]);
+ } else {
+ const wid = RE_DEVA.test(t) ? lm.id(t) ?? -1 : -2; lat.push([[t, 0, wid]]);
+ }
+ }
+ const post = lat.map((o) => o.map(([s, v]) => [s, v])); // per-slot candidate list (for UI chips)
+ if (!context || !p.use_lm || !lm.hasLM) return { out: lat.map((o) => o.reduce((a, b) => (b[1] > a[1] ? b : a))[0]), lat: post };
+ let beams = [[0, null, []]];
+ for (const opts of lat) {
+ if (opts[0][2] === -2 && opts.length === 1) { beams = beams.map(([s, h, ch]) => [s, h, ch.concat([opts[0][0]])]); continue; }
+ const best = new Map();
+ for (const [s, h, ch] of beams) for (const [cand, csc, wid] of opts) {
+ const w = wid === null || wid === -1 ? null : wid;
+ const tot = s + csc + p.w_lm * (lm.logp(h, w) - (w === null ? p.oov_pen_s : 0));
+ if (!best.has(cand) || tot > best.get(cand)[0]) best.set(cand, [tot, w, ch.concat([cand])]);
+ }
+ beams = [...best.values()].sort((a, b) => b[0] - a[0]).slice(0, 16);
+ }
+ let bb = null, bv = -Infinity;
+ for (const b of beams) { const v = b[0] + p.w_lm * lm.logp(b[1], lm.N); if (v > bv) { bv = v; bb = b; } }
+ return { out: bb[2], lat: post };
+ }
+ /** plain Track-B sentence conversion (bhc/translit2.py convert) */
+ convertPlain(text) {
+ const spans = [...text.matchAll(TOKEN_RE)].map((m) => [m.index, m.index + m[0].length, m[0]]);
+ const { out } = this.decodeSlots(spans.map(([, , t]) => (RE_LATIN.test(t) ? ["x", t] : ["fix", t])));
+ if (out.length && out[out.length - 1] === ".") out[out.length - 1] = "\u0964";
+ let s = "", pos = 0; spans.forEach(([a, b], i) => { s += text.slice(pos, a) + out[i]; pos = b; }); return s + text.slice(pos);
+ }
+
+ /**
+ * Full pipeline: tokenise -> language ID -> policy (keep / acronym / loanword / glossary) -> contextual decoding.
+ * @param {string} text
+ * @param {{mode?: 'conversational'|'natural'|'strict', k?: number, context?: boolean, overrides?: Object