codemix_xlit assets 1.4.11+72ed46e29ae6

LICENSE

@@ -1,4 +1,4 @@
-indictrans (codemix_xlit assets) version 1.5.4: licences of the files in this bundle
+indictrans (codemix_xlit assets) version 1.4.11: licences of the files in this bundle
 
 Sources and licences
 `license: other` because the files carry different licences; each is listed here with its notice.
@@ -68,16 +68,3 @@ Sources and licences
   (https://creativecommons.org/publicdomain/zero/1.0/), checked against the IndicXlit model above.
 - Lexicons, corrections, ranker and the other romanized tables: derived from the above as described
   below.
-- `intl/ja/`: UniDic-cwj 2023.02 (NINJAL, BSD-3-Clause of its GPL/LGPL/BSD choice) and SudachiDict
-  (Works Applications, Apache-2.0) loanword entries, with forms derived through CMUdict
-- `intl/arb/`, `intl/arz/`, `intl/afb/`, ...: Wikidata labels (CC0) and spellings from Gemma 4 E2B
-  (Google, Apache-2.0) where two prompts agree, checked against the CMUdict renderer and reviewed
-- `intl/arz/arabizi_table.json`, `intl/arz/arabizi_freq.tsv`: the Egyptian Arabizi everyday-word table, written
-  for this project (`tools/codemix_xlit/intl/arabizi/conv_arz.json`), and word counts from Tatoeba sentences
-  in Egyptian and Standard Arabic (https://tatoeba.org, by the Tatoeba contributors, CC BY 2.0 FR,
-  https://creativecommons.org/licenses/by/2.0/fr/), the Common Voice Arabic sentence collector (CC0 1.0) and
-  Wikidata lexemes (CC0 1.0); only the counts are included
-- `intl/es/`: words measured on the Finesse Spanish voice, respelled from CMUdict
-- `intl/en/`: Wikidata labels (CC0), SudachiDict kanji readings (Apache-2.0), this bundle's own lists
-  reversed; rows marked `roman` come from the romanized tables (`roman/`), which use Dakshina counts and
-  are shared under CC BY-SA 4.0 like them

NOTICE

@@ -1,5 +1,5 @@
 indictrans: runtime assets of the Finesse TTS code-mix transliteration layer (utils/codemix_xlit)
-Version 1.5.4
+Version 1.4.11
 
 This bundle redistributes and adapts third-party material. Each part keeps its own licence (see
 LICENSE):
@@ -20,17 +20,7 @@ LICENSE):
   https://huggingface.co/datasets/ai4bharat/sangraha. Changes: neighbour-word counts (roman_ctx/).
 - Wikidata, CC0 1.0, https://www.wikidata.org. Changes: Indian given names and surnames, and India's
   states, union territories and larger places, with their native-script labels (names/), checked
-  against the IndicXlit model; labels of names, places and brands in Arabic and other scripts (intl/).
-- UniDic-cwj 2023.02, Copyright (c) 2023 National Institute for Japanese Language and Linguistics,
-  BSD-3-Clause (its full text: intl/ja/UNIDIC_LICENSE.txt), https://clrd.ninjal.ac.jp/unidic/. Changes: the
-  katakana spellings of its English loanword entries (intl/ja/lexicon.tsv).
-- SudachiDict, (c) Works Applications Co., Ltd., Apache License 2.0,
-  https://github.com/WorksApplications/SudachiDict. Changes: the katakana readings of its Latin-spelled
-  entries (intl/ja/lexicon.tsv, intl/en/ja.tsv).
-- Tatoeba, sentences by the Tatoeba contributors, CC BY 2.0 FR, https://tatoeba.org,
-  https://creativecommons.org/licenses/by/2.0/fr/. Changes: only word counts from its Egyptian and Standard
-  Arabic sentences are included (intl/arz/arabizi_freq.tsv).
-- Mozilla Common Voice, Arabic sentence collector, CC0 1.0. Changes: word counts (intl/arz/arabizi_freq.tsv).
+  against the IndicXlit model.
 
 The reviewed lexicons, pronunciation corrections, Indian-word list, everyday romanized tables and the
 ranker were written or labelled by a large language model run as writer and grader agents, not by

README.md

@@ -1,135 +0,0 @@
----
-license: other
-language: [hi, mr, ne, mai, sa, brx, kok, doi, gu, pa, or, ta, te, kn, ml, mni, bn, as, ur, sd, ks, sat]
-tags: [transliteration, code-mixing, text-to-speech, indic]
----
-# indictrans: English words in Indian-language text, written in the request's script
-
-Runtime assets of the code-mix transliteration layer of the Finesse TTS (`utils/codemix_xlit`):
-English words inside Indic text are rewritten in the language's script, spelled the way an
-Indian-English speaker says them, so an Indic voice reads them correctly; text typed entirely in
-Latin is read word by word, the language's own words back into its script. 22 languages (Dogri and
-Santali through the Hindi model). From 1.5, word lists for the non-Indic languages too (`intl/`).
-
-Version 1.5.4+38626e6b36b7. Built by `tools/codemix_xlit/build_assets.py`; `manifest.json` lists every file
-with its size and SHA-256.
-
-## Contents
-- `LICENSE`, `NOTICE`: the licences and attributions below, inside the bundle, so a copy of the folder
-  keeps them
-- `model/`: the IndicXlit en->indic v1.0 transformer converted to CTranslate2
-- `lexicon/<code>.reviewed.json`, `lexicon/<code>.generated.json`: reviewed and generated spellings
-- `rescore/`: pruned IndicXlit word-frequency dictionaries
-- `cmudict.json`, `english_*`: English pronunciations, vocabulary and reviewed corrections
-- `indic_words.json`: Indian words read natively; `ranker.json`: candidate ranker
-- `roman/<code>.json`: how people type each language's words in Latin, how often, and the other words
-  a typing can be (`alts`)
-- `native_freq/<token>.json`: word frequencies of the eleven languages without a `rescore/` list, for
-  the language's own words typed in Latin
-- `roman_ctx/<code>.json`: for typings that are two different words, how often each is seen next to
-  the words that tell them apart (sentence context)
-- `names/<code>.json`: Indian given names, surnames and place names typed in Latin and how the language
-  writes them (`tools/codemix_xlit/build_names.py`), read for a capitalised word (Sneha स्नेहा, not स्नेह)
-- `intl/<code>/lexicon.tsv`: for the non-Indic languages (`utils/codemix_xlit/intl`), English words and
-  the spelling each language writes them in (Japanese katakana; Standard, Egyptian, Gulf, Najdi and
-  Hijazi Arabic; Spanish), with the source of every row; built by `tools/codemix_xlit/intl/build_<lang>.py`
-- `intl/en/<script>.tsv`: for English requests, words in another script and their Latin spelling (names,
-  places, loanwords back to English, everyday words as people type them, kanji readings)
-
-## Sources and licences
-`license: other` because the files carry different licences; each is listed here with its notice.
-
-- `model/`, `rescore/`, `native_freq/`: the IndicXlit model and word-frequency dictionaries, AI4Bharat,
-  MIT licence (https://github.com/AI4Bharat/IndicXlit). Notice:
-
-  > MIT License
-  >
-  > Copyright (c) 2022 AI4Bhārat
-  >
-  > Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
-  > associated documentation files (the "Software"), to deal in the Software without restriction,
-  > including without limitation the rights to use, copy, modify, merge, publish, distribute,
-  > sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
-  > furnished to do so, subject to the following conditions:
-  >
-  > The above copyright notice and this permission notice shall be included in all copies or
-  > substantial portions of the Software.
-  >
-  > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
-  > NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  > NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-  > DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
-  > OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-- `cmudict.json`: the CMU Pronouncing Dictionary (https://github.com/cmusphinx/cmudict), BSD-style
-  licence. Notice:
-
-  > Copyright (C) 1993-2015 Carnegie Mellon University. All rights reserved.
-  >
-  > Redistribution and use in source and binary forms, with or without modification, are permitted
-  > provided that the following conditions are met:
-  >
-  > 1. Redistributions of source code must retain the above copyright notice, this list of conditions
-  >    and the following disclaimer. The contents of this file are deemed to be source code.
-  > 2. Redistributions in binary form must reproduce the above copyright notice, this list of
-  >    conditions and the following disclaimer in the documentation and/or other materials provided
-  >    with the distribution.
-  >
-  > This work was supported in part by funding from the Defense Advanced Research Projects Agency, the
-  > Office of Naval Research and the National Science Foundation of the United States of America, and
-  > by member companies of the Carnegie Mellon Sphinx Speech Consortium. We acknowledge the
-  > contributions of many volunteers to the expansion and improvement of this dictionary.
-  >
-  > THIS SOFTWARE IS PROVIDED BY CARNEGIE MELLON UNIVERSITY ``AS IS'' AND ANY EXPRESSED OR IMPLIED
-  > WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
-  > FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY NOR ITS
-  > EMPLOYEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-  > DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-  > DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
-  > IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
-  > OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-- `english_vocab.txt`, `english_pron_vocab.txt`: English word lists derived from wordfreq
-  (https://github.com/rspeer/wordfreq), whose data is CC BY-SA 4.0
-  (https://creativecommons.org/licenses/by-sa/4.0/); these lists are shared under the same licence.
-- `roman/` of bn, gu, hi, kn, ml, mr, pa, sd, ta, te, ur: counts derived from the romanized sentences
-  of the Dakshina dataset (Roark et al. 2020, https://github.com/google-research-datasets/dakshina),
-  CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); these tables are shared under the
-  same licence.
-- `roman_ctx/`: neighbour counts from AI4Bharat Sangraha, verified subset (Khan et al. 2024,
-  https://huggingface.co/datasets/ai4bharat/sangraha), CC BY 4.0
-  (https://creativecommons.org/licenses/by/4.0/).
-- `names/`: the English and native-script labels of people with Indian citizenship, of the name items
-  they carry and of India's states, union territories and places of 100,000 people, from Wikidata (https://www.wikidata.org), CC0 1.0
-  (https://creativecommons.org/publicdomain/zero/1.0/), checked against the IndicXlit model above.
-- Lexicons, corrections, ranker and the other romanized tables: derived from the above as described
-  below.
-- `intl/ja/`: UniDic-cwj 2023.02 (NINJAL, BSD-3-Clause of its GPL/LGPL/BSD choice) and SudachiDict
-  (Works Applications, Apache-2.0) loanword entries, with forms derived through CMUdict
-- `intl/arb/`, `intl/arz/`, `intl/afb/`, ...: Wikidata labels (CC0) and spellings from Gemma 4 E2B
-  (Google, Apache-2.0) where two prompts agree, checked against the CMUdict renderer and reviewed
-- `intl/arz/arabizi_table.json`, `intl/arz/arabizi_freq.tsv`: the Egyptian Arabizi everyday-word table, written
-  for this project (`tools/codemix_xlit/intl/arabizi/conv_arz.json`), and word counts from Tatoeba sentences
-  in Egyptian and Standard Arabic (https://tatoeba.org, by the Tatoeba contributors, CC BY 2.0 FR,
-  https://creativecommons.org/licenses/by/2.0/fr/), the Common Voice Arabic sentence collector (CC0 1.0) and
-  Wikidata lexemes (CC0 1.0); only the counts are included
-- `intl/es/`: words measured on the Finesse Spanish voice, respelled from CMUdict
-- `intl/en/`: Wikidata labels (CC0), SudachiDict kanji readings (Apache-2.0), this bundle's own lists
-  reversed; rows marked `roman` come from the romanized tables (`roman/`), which use Dakshina counts and
-  are shared under CC BY-SA 4.0 like them
-
-## Who wrote the generated parts
-- `lexicon/<code>.generated.json`: written offline by the layer's own pipeline (`build_assets.py
-  lexicon`: IndicXlit's candidates, spellings rendered from the pronunciation, the ranker). Its
-  entries that did not read back as the English word, among each language's 5,000 most frequent,
-  were reviewed by the LLM graders below and the errors corrected in the reviewed lexicon.
-- `english_extra_prons.json` (Indian-English readings of words the CMU dictionary lacks: names,
-  brands, foreign words), `english_pron_overrides.json`, `indic_words.json`, the gold spellings in
-  `lexicon/<code>.reviewed.json`, and the everyday words in `roman/`: written by a large language
-  model, run as per-language writer and grader agents under fixed spelling conventions. They were
-  reviewed by further agents of the same model (a reading is read
-  back against the English pronunciation; a spelling is graded C/A/W blind against plain
-  IndicXlit's). Two independent graders agreed on 96.2-97.7% of labels.
-- `ranker.json`: fitted on those graders' labels.
-- The accuracy figures published for this bundle come from the same model graders. **No native
-  speaker has reviewed any of these files yet**; treat them as unverified for external claims.

intl/acw/lexicon.tsv

@@ -1,2 +0,0 @@
-# Hijazi: entries that differ from afb (none so far)
-# built by tools/codemix_xlit/intl/build_ar.py: word, spelling, source

intl/afb/lexicon.tsv

@@ -1,2 +0,0 @@
-# Gulf: entries that differ from arb (g as غ)
-# built by tools/codemix_xlit/intl/build_ar.py: word, spelling, source

intl/arb/lexicon.tsv

@@ -1,221 +0,0 @@
-# Standard Arabic: every entry (English g as غ)
-# built by tools/codemix_xlit/intl/build_ar.py: word, spelling, source
-absher	أبشر	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-academy	أكاديمي	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-adcb	إيه دي سي بي	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-adidas	أديداس	wikidata:Q3895 (CC0)
-agoda	أغودا	wikidata:Q395182 (CC0)
-airbnb	إير بي إن بي	wikidata:Q63327 (CC0)
-airbus	إيرباص	wikidata:Q67 (CC0)
-airpods	آيربودز	wikidata:Q26936286 (CC0)
-alcohol	ألكحول	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-alexa	ألكسا	wikidata:Q28022144 (CC0)
-alexandria	الإسكندرية	wikidata:Q87 (CC0)
-aliexpress	علي إكسبريس	wikidata:Q29904317 (CC0)
-alternative	ألتيرناتيف	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-amazon	أمازون	wikidata:Q3884 (CC0)
-amman	عمان	wikidata:Q3805 (CC0)
-android	أندرويد	wikidata:Q94 (CC0)
-apple	أبل	wikidata:Q312 (CC0)
-aramex	أرامكس	wikidata:Q4120390 (CC0)
-asap	إيه إس إيه بي	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-asus	آسوس	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-baghdad	بغداد	wikidata:Q1530 (CC0)
-bahrain	البحرين	wikidata:Q398 (CC0)
-bank	بنك	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-baskin	باسكن	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-battery	بطارية	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-beirut	بيروت	wikidata:Q3820 (CC0)
-boeing	بوينغ	wikidata:Q66 (CC0)
-boston	بوسطن	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-breadfast	بريدفاست	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-burger	برغر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-bus	باص	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-cable	كابل	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-cairo	القاهرة	wikidata:Q85 (CC0)
-camera	كاميرا	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-canada	كندا	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-cappuccino	كابتشينو	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-carbon	كربون	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-careem	كريم	wikidata:Q25112257 (CC0)
-carrefour	كارفور	wikidata:Q217599 (CC0)
-cashback	كاش باك	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-casino	كازينو	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-category	كاتيغوري	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-catholic	كاثوليك	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-center	سنتر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-central	سنترال	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-centre	سنتر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-character	كاراكتر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-characters	كاراكترز	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-chatgpt	شات جي بي تي	wikidata:Q115564437 (CC0)
-cheque	شيك	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-chevrolet	شيفروليه	wikidata:Q29570 (CC0)
-chicago	شيكاغو	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-chocolate	تشوكليت	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-chrome	كروم	wikidata:Q777 (CC0)
-club	كلوب	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-color	كلر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-computer	كمبيوتر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-corporate	كوربوريت	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-cover	كفر	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-covid	كوفيد	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-custom	كاستم	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-customer	كاستمر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-customers	كاستمرز	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-damascus	دمشق	wikidata:Q3766 (CC0)
-deliveroo	ديليفرو	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-dell	ديل	wikidata:Q30873 (CC0)
-diesel	ديزل	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-doctor	دكتور	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-doctors	دكتورز	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-doha	الدوحة	wikidata:Q3861 (CC0)
-done	دن	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-dubai	دبي	wikidata:Q612 (CC0)
-dunkin	دانكن	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-ebay	إيباي	wikidata:Q58024 (CC0)
-egypt	مصر	wikidata:Q79 (CC0)
-electric	إلكتريك	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-electronic	إلكترونيك	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-emax	إيماكس	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-esim	إي سيم	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-etisalat	اتصالات	wikidata:Q543449 (CC0)
-excel	إكسل	wikidata:Q11272 (CC0)
-facebook	فيسبوك	wikidata:Q355 (CC0)
-fawry	فوري	wikidata:Q109298750 (CC0)
-fedex	فيديكس	wikidata:Q459477 (CC0)
-filter	فلتر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-finance	فاينانس	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-financial	فاينانشال	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-firefox	فايرفوكس	wikidata:Q698 (CC0)
-ford	فورد	wikidata:Q44294 (CC0)
-gmail	جيميل	wikidata:Q9334 (CC0)
-google	غوغل	wikidata:Q95 (CC0)
-guarantee	غارانتي	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-hdmi	إتش دي إم آي	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-honda	هوندا	wikidata:Q9584 (CC0)
-huawei	هواوي	wikidata:Q160120 (CC0)
-hypermarket	هايبرماركت	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-hyundai	هيونداي	wikidata:Q55931 (CC0)
-iban	آيبان	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-icloud	آي كلاود	wikidata:Q283469 (CC0)
-ikea	إيكيا	wikidata:Q54078 (CC0)
-instagram	إنستغرام	wikidata:Q209330 (CC0)
-instapay	إنستاباي	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-instashop	إنستاشوب	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-insurance	إنشورنس	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-intel	إنتل	wikidata:Q248 (CC0)
-ipad	آي باد	wikidata:Q2796 (CC0)
-iphone	آيفون	wikidata:Q2766 (CC0)
-iraq	العراق	wikidata:Q796 (CC0)
-italy	إيطاليا	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-jahez	جاهز	wikidata:Q133285017 (CC0)
-jarir	جرير	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-jeddah	جدة	wikidata:Q374365 (CC0)
-keeta	كيتا	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-kia	كيا	wikidata:Q35349 (CC0)
-kuwait	الكويت	wikidata:Q817 (CC0)
-lebanon	لبنان	wikidata:Q822 (CC0)
-lenovo	لينوفو	wikidata:Q14799 (CC0)
-lexus	لكزس	wikidata:Q35919 (CC0)
-liberal	ليبرال	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-linkedin	لينكد إن	wikidata:Q213660 (CC0)
-london	لندن	wikidata:Q84 (CC0)
-love	لاف	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-macbook	ماك بوك	wikidata:Q486053 (CC0)
-maintenance	مينتيننس	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-makkah	مكة	wikidata:Q5806 (CC0)
-massage	مساج	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-mastercard	ماستركارد	wikidata:Q489921 (CC0)
-mcdonalds	ماكدونالدز	wikidata:Q38076 (CC0)
-mecca	مكة	wikidata:Q5806 (CC0)
-medina	المدينة	wikidata:Q35484 (CC0)
-menu	منيو	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-messenger	ماسنجر	wikidata:Q116947 (CC0)
-microsoft	مايكروسوفت	wikidata:Q2283 (CC0)
-million	مليون	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-millions	مليونز	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-mobily	موبايلي	wikidata:Q6887331 (CC0)
-model	موديل	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-mrsool	مرسول	wikidata:Q131560293 (CC0)
-muscat	مسقط	wikidata:Q3826 (CC0)
-namshi	نمشي	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-net	نت	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-netflix	نتفليكس	wikidata:Q907311 (CC0)
-network	نتورك	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-nike	نايكي	wikidata:Q483915 (CC0)
-nikon	نيكون	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-nintendo	نينتندو	wikidata:Q8093 (CC0)
-nissan	نيسان	wikidata:Q20165 (CC0)
-nokia	نوكيا	wikidata:Q1418 (CC0)
-noon	نون	wikidata:Q60914457 (CC0)
-nvidia	إنفيديا	wikidata:Q182477 (CC0)
-oman	عمان	wikidata:Q842 (CC0)
-openai	أوبن أيه آي	wikidata:Q21708200 (CC0)
-oppo	أوبو	wikidata:Q2084900 (CC0)
-ounass	أوناس	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-outlook	آوتلوك	wikidata:Q80911 (CC0)
-pakistan	باكستان	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-paymob	باي موب	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-paypal	باي بال	wikidata:Q483959 (CC0)
-personal	بيرسونال	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-pinterest	بنترست	wikidata:Q255381 (CC0)
-pizza	بيتزا	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-playstation	بلاي ستيشن	wikidata:Q10677 (CC0)
-plus	بلس	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-powerpoint	باوربوينت	wikidata:Q11266 (CC0)
-privacy	برايفسي	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-process	بروسيس	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-production	برودكشن	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-puma	بوما	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-qatar	قطر	wikidata:Q846 (CC0)
-radio	راديو	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-realme	ريلمي	wikidata:Q56275466 (CC0)
-reddit	ريديت	wikidata:Q1136 (CC0)
-release	ريليز	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-restaurant	ريستورانت	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-riyadh	الرياض	wikidata:Q3692 (CC0)
-robbins	روبينز	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-router	راوتر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-salon	صالون	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-samsung	سامسونغ	wikidata:Q20716 (CC0)
-shein	شي إن	wikidata:Q107377209 (CC0)
-sivvi	سيفي	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-skype	سكايب	wikidata:Q40984 (CC0)
-smartwatch	سمارت ووتش	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-snapchat	سناب شات	wikidata:Q333618 (CC0)
-social	سوشيال	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-sony	سوني	wikidata:Q41187 (CC0)
-spotify	سبوتيفاي	wikidata:Q689141 (CC0)
-stadium	ستاديوم	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-starbucks	ستاربكس	wikidata:Q37158 (CC0)
-swvl	سويفل	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-tabby	تابي	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-talabat	طلبات	wikidata:Q39058477 (CC0)
-tamara	تمارا	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-tawakkalna	توكلنا	wikidata:Q94096241 (CC0)
-technology	تكنولوجي	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-telegram	تيليغرام	wikidata:Q15616276 (CC0)
-temu	تيمو	wikidata:Q114258161 (CC0)
-tennis	تنس	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-tesla	تسلا	wikidata:Q478214 (CC0)
-test	تست	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-tiktok	تيك توك	wikidata:Q48938223 (CC0)
-tiktokshop	تيك توك شوب	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-toters	توترز	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-toyota	تويوتا	wikidata:Q53268 (CC0)
-trendyol	ترينديول	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules
-twitter	تويتر	wikidata:Q15718485 (CC0)
-uber	أوبر	wikidata:Q780442 (CC0)
-visa	فيزا	wikidata:Q328840 (CC0)
-vodafone	فودافون	wikidata:Q122141 (CC0)
-whatsapp	واتساب	wikidata:Q1049511 (CC0)
-wifi	واي فاي	wikidata:Q29643 (CC0)
-wikipedia	ويكيبيديا	wikidata:Q52 (CC0)
-windows	ويندوز	wikidata:Q1406 (CC0)
-xiaomi	شاومي	wikidata:Q1636958 (CC0)
-yahoo	ياهو	wikidata:Q37093 (CC0)
-yemen	اليمن	wikidata:Q805 (CC0)
-youtube	يوتيوب	wikidata:Q866 (CC0)
-zain	زين	wikidata:Q144364 (CC0)
-zomato	زوماتو	reviewed (Arabic probe round, 2026-10-29); consonants checked against the CMUdict rules

intl/ars/lexicon.tsv

@@ -1,2 +0,0 @@
-# Najdi: entries that differ from afb (none so far)
-# built by tools/codemix_xlit/intl/build_ar.py: word, spelling, source

intl/arz/lexicon.tsv

@@ -1,18 +0,0 @@
-# Egyptian: entries that differ from arb (g as ج, Egyptian Wikidata labels)
-# built by tools/codemix_xlit/intl/build_ar.py: word, spelling, source
-agoda	أجودا	wikidata:Q395182 (CC0)
-boeing	بوينج	wikidata:Q66 (CC0)
-burger	برجر	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-category	كاتيجوري	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-chicago	شيكاجو	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-facebook	فيس بوك	wikidata:Q355 (CC0)
-gmail	جي ميل	wikidata:Q9334 (CC0)
-google	جوجل	wikidata:Q95 (CC0)
-guarantee	جارانتي	gemma-4-e2b-it (Apache-2.0), reviewed; consonants checked against the CMUdict rules
-instagram	إنستجرام	wikidata:Q209330 (CC0)
-iphone	آي فون	wikidata:Q2766 (CC0)
-netflix	نيتفليكس	wikidata:Q907311 (CC0)
-nike	نايك	wikidata:Q483915 (CC0)
-samsung	سامسونج	wikidata:Q20716 (CC0)
-telegram	تيليجرام	wikidata:Q15616276 (CC0)
-tesla	تيسلا	wikidata:Q478214 (CC0)

intl/en/ar.tsv

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:10f76c626dbb1af0185d64441c2ace0782fb7240275a5239c5c2d7134611d699
-size 22504110

intl/en/beng.tsv

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:56b1e8b621e538ead26b261097af6bab2dab6a36b1d525266e96bd1c8f95ff9e
-size 11124194

intl/en/ja.tsv

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:8ec5f794b20ae72bbe48bcd58c9100e2e92dae357a8eec5c46ce0d9fffe1295a
-size 36522339

intl/en/latin.tsv

@@ -1 +0,0 @@
-# English layer: romanized words an English voice misreads (lowercase key TAB respelling TAB source). Empty: no misreading confirmed by a voice test yet.

intl/en/ur.tsv

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ef2254d9a2bc13078ca9c4894bfc2bd5ad6c4001e431504a74dfe723d8dbdcdf
-size 17620733

intl/es/lexicon.tsv

@@ -1,10 +0,0 @@
-# Spanish: English words the Spanish voice misread as typed and said right respelled
-# built by tools/codemix_xlit/intl/build_es.py: word, spelling, source
-approved	apruvd	measured on the Finesse Spanish voice 2026-10-28; respell.py (CMUdict BSD-2)
-consultant	consáltant	measured on the Finesse Spanish voice 2026-10-28; respell.py (CMUdict BSD-2)
-crush	crash	measured on the Finesse Spanish voice 2026-10-28; respell.py (CMUdict BSD-2)
-earphones	irfouns	measured on the Finesse Spanish voice 2026-10-28; respell.py (CMUdict BSD-2)
-luxury	lágyuri	measured on the Finesse Spanish voice 2026-10-28; respell.py (CMUdict BSD-2)
-return	retérn	measured on the Finesse Spanish voice 2026-10-28; respell.py (CMUdict BSD-2)
-sample	sámpal	measured on the Finesse Spanish voice 2026-10-28; respell.py (CMUdict BSD-2)
-subscription	subscripshon	measured on the Finesse Spanish voice 2026-10-28; respell.py (CMUdict BSD-2)

intl/ja/UNIDIC_LICENSE.txt

@@ -1,29 +0,0 @@
-Copyright (c) 2023 National Institute for Japanese Language and Linguistics
-
-Redistribution and use in source and binary forms, with or without
-modification, are permitted provided that the following conditions are
-met:
-
- * Redistributions of source code must retain the above copyright
-   notice, this list of conditions and the following disclaimer.
-
- * Redistributions in binary form must reproduce the above copyright
-   notice, this list of conditions and the following disclaimer in the
-   documentation and/or other materials provided with the
-   distribution.
-
- * Neither the name of the UniDic Consortium nor the names of its
-   contributors may be used to endorse or promote products derived
-   from this software without specific prior written permission.
-
-THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

manifest.json

@@ -1,12 +1,12 @@
 {
  "files": {
   "LICENSE": {
-   "bytes": 5900,
-   "sha256": "dccc127f27d1f02c17e49a5ea7701e176b11ff4f65b88beb0af8293507b48429"
+   "bytes": 4698,
+   "sha256": "9a91afdd358df0cb3df81bf0f02a394cad8dee3925b530af192e56ad1187c16b"
   },
   "NOTICE": {
-   "bytes": 2863,
-   "sha256": "7a4564bc3ec5fc511284f0046b17e5183527cc9fea9c0e623fe7b5756a6ad86b"
+   "bytes": 1923,
+   "sha256": "223c185853aa8c17ae5bcf19a6b7d1a8c055d1b8d4374ceee4d2bb4f9bb682f6"
   },
   "cmudict.json": {
    "bytes": 4464854,
@@ -32,98 +32,6 @@
    "bytes": 95417,
    "sha256": "a1e8db5de7c3e1374d67792919af8b025ee6412137b9e02f56b807805df75193"
   },
-  "intl/acw/lexicon.tsv": {
-   "bytes": 124,
-   "sha256": "e6aaafd121a307d86a1ff6b6241a2d0647b444cb1865b13498fc3e1f2b1edb76"
-  },
-  "intl/afb/lexicon.tsv": {
-   "bytes": 118,
-   "sha256": "84749b3c0a34583e8bad807143eea94e62b05986022e27e6784b94033d3e7ff7"
-  },
-  "intl/arb/lexicon.tsv": {
-   "bytes": 15993,
-   "sha256": "acd95dba2de6852c364ee4bb6cc9f1481cb36d167b417c89bc44ae40afee1edc"
-  },
-  "intl/ars/lexicon.tsv": {
-   "bytes": 123,
-   "sha256": "d17f435766d51ec5c27d64b6511766fe124fde443291bc4d6b59ad43047cd75c"
-  },
-  "intl/arz/arabizi_freq.tsv": {
-   "bytes": 519713,
-   "sha256": "d1808faf747913284bc22db44cf30b4fe60a0c67a8a4832d7a9ec75ef4fdd50f"
-  },
-  "intl/arz/arabizi_table.json": {
-   "bytes": 140231,
-   "sha256": "04a6176ebe4c1ac8f8d832ae7d102ce8c67ae579708c4fb032c9e0a1801ed93b"
-  },
-  "intl/arz/lexicon.tsv": {
-   "bytes": 1077,
-   "sha256": "01fb4d680c39b434ef3cb2d4ddb802cac2f0926da45595fe6ef6a82d3237ff09"
-  },
-  "intl/en/ar.tsv": {
-   "bytes": 22504110,
-   "sha256": "10f76c626dbb1af0185d64441c2ace0782fb7240275a5239c5c2d7134611d699"
-  },
-  "intl/en/beng.tsv": {
-   "bytes": 11124194,
-   "sha256": "56b1e8b621e538ead26b261097af6bab2dab6a36b1d525266e96bd1c8f95ff9e"
-  },
-  "intl/en/deva.tsv": {
-   "bytes": 9158314,
-   "sha256": "88135a91bfd248cc2c4bc7255c6e8ace7c087aa7964dae5c91398ccfd09d5c67"
-  },
-  "intl/en/gujr.tsv": {
-   "bytes": 1584864,
-   "sha256": "09b1e8110f14924fa1960e99dc55099c8d97c9c0344bef3f2e1ec11d0242a805"
-  },
-  "intl/en/guru.tsv": {
-   "bytes": 3162158,
-   "sha256": "8427abdcc295aac9c321ef49aca311b78ff3ae9c2a98829fe8c161915166eeac"
-  },
-  "intl/en/ja.tsv": {
-   "bytes": 36522339,
-   "sha256": "8ec5f794b20ae72bbe48bcd58c9100e2e92dae357a8eec5c46ce0d9fffe1295a"
-  },
-  "intl/en/knda.tsv": {
-   "bytes": 2424099,
-   "sha256": "920313a7714b0da209c642752fecf20c7140ea13ef20fcb182431693ff30ed19"
-  },
-  "intl/en/latin.tsv": {
-   "bytes": 154,
-   "sha256": "85c995c72329829af0def817120f5c1d76a391fe0b00df39fc7389304d8f75e5"
-  },
-  "intl/en/mlym.tsv": {
-   "bytes": 4867086,
-   "sha256": "d53570414c7e7fb5b48908739f6fbee5e412330df06007aec8b229094f5ce317"
-  },
-  "intl/en/orya.tsv": {
-   "bytes": 2578550,
-   "sha256": "3d6b5ae68ddaa2e3fcbcb7f42a221fcc0a78e152f1317fb7d77e9384da919beb"
-  },
-  "intl/en/taml.tsv": {
-   "bytes": 5786790,
-   "sha256": "5e93c84a378188b961be3545d376c65e7cb6b3246f3477b0ecb7543bb17047ba"
-  },
-  "intl/en/telu.tsv": {
-   "bytes": 5448410,
-   "sha256": "3f36637fc8af7bfe385dd8ac7b39fbed5d76eb4bd63fedaf65cf66b326b39bac"
-  },
-  "intl/en/ur.tsv": {
-   "bytes": 17620733,
-   "sha256": "ef2254d9a2bc13078ca9c4894bfc2bd5ad6c4001e431504a74dfe723d8dbdcdf"
-  },
-  "intl/es/lexicon.tsv": {
-   "bytes": 913,
-   "sha256": "a394671c85134e63db0de0d5ace6ddc925eba2c8d1ccec24dee92bbcdfaae24a"
-  },
-  "intl/ja/UNIDIC_LICENSE.txt": {
-   "bytes": 1523,
-   "sha256": "9980b1824f0d1dac41ea93dc96780c222e5cca3c4d1b5677ac8ab923a8533c3f"
-  },
-  "intl/ja/lexicon.tsv": {
-   "bytes": 2549597,
-   "sha256": "686625798082d4e26151d15485ece84f5f057a26f5a21fb8050b89d9d68b4516"
-  },
   "lexicon/as.generated.json": {
    "bytes": 3526946,
    "sha256": "d8983505e0fabaab9658d94a6916d33b6bd4311eabaf52d410c686131f221b7b"
@@ -209,8 +117,8 @@
    "sha256": "7c73f11e2361cd704055cfc4168516207fc9f1882309555b0627f7150a75f7c6"
   },
   "lexicon/ml.reviewed.json": {
-   "bytes": 273249,
-   "sha256": "cfd50bad6c4894216264d667de39d121b3c4eb6edec45cbb9af530812d58f70f"
+   "bytes": 273301,
+   "sha256": "bff2b8cf2f7cf8b4748c6d56bbbe12faeb55e92d052fb88db0034f793efc2399"
   },
   "lexicon/mni.generated.json": {
    "bytes": 3220999,
@@ -273,8 +181,8 @@
    "sha256": "31a1cc4d556f3ca3bbe7d2836850cbfdffbeda952e839588e659740b4ba70af5"
   },
   "lexicon/ta.reviewed.json": {
-   "bytes": 238738,
-   "sha256": "58d644af4956b8d4e856f99f9ccfd9751c0aabf39ce0fe36ea9a852e8088ff39"
+   "bytes": 238754,
+   "sha256": "06c4bfa311d4e370e11ae41f125a3c567cf3af5f176786fe3ad6827bb0e27b43"
   },
   "lexicon/te.generated.json": {
    "bytes": 3728261,
@@ -648,6 +556,6 @@
  "model_source": "https://github.com/AI4Bharat/IndicXlit/releases/download/v1.0/indicxlit-en-indic-v1.0.zip",
  "pronunciations": "CMU Pronouncing Dictionary (cmudict)",
  "rescore_source": "https://github.com/AI4Bharat/IndicXlit/releases/download/v1.0/word_prob_dicts.zip",
- "version": "1.5.4+38626e6b36b7",
+ "version": "1.4.11+72ed46e29ae6",
  "vocab": "wordfreq top English words"
 }