codemix_xlit assets 1.4.16+12d0f820ea16

LICENSE

@@ -1,4 +1,4 @@
-indictrans (codemix_xlit assets) version 1.4.15: licences of the files in this bundle
+indictrans (codemix_xlit assets) version 1.4.16: licences of the files in this bundle
 
 Sources and licences
 `license: other` because the files carry different licences; each is listed here with its notice.

NOTICE

@@ -1,5 +1,5 @@
 indictrans: runtime assets of the Finesse TTS code-mix transliteration layer (utils/codemix_xlit)
-Version 1.4.15
+Version 1.4.16
 
 This bundle redistributes and adapts third-party material. Each part keeps its own licence (see
 LICENSE):

README.md

@@ -1,117 +0,0 @@
----
-license: other
-language: [hi, mr, ne, mai, sa, brx, kok, doi, gu, pa, or, ta, te, kn, ml, mni, bn, as, ur, sd, ks, sat]
-tags: [transliteration, code-mixing, text-to-speech, indic]
----
-# indictrans: English words in Indian-language text, written in the request's script
-
-Runtime assets of the code-mix transliteration layer of the Finesse TTS (`utils/codemix_xlit`):
-English words inside Indic text are rewritten in the language's script, spelled the way an
-Indian-English speaker says them, so an Indic voice reads them correctly; text typed entirely in
-Latin is read word by word, the language's own words back into its script. 22 languages (Dogri and
-Santali through the Hindi model).
-
-Version 1.4.15+631ae6081786. Built by `tools/codemix_xlit/build_assets.py`; `manifest.json` lists every file
-with its size and SHA-256.
-
-## Contents
-- `LICENSE`, `NOTICE`: the licences and attributions below, inside the bundle, so a copy of the folder
-  keeps them
-- `model/`: the IndicXlit en->indic v1.0 transformer converted to CTranslate2
-- `lexicon/<code>.reviewed.json`, `lexicon/<code>.generated.json`: reviewed and generated spellings
-- `rescore/`: pruned IndicXlit word-frequency dictionaries
-- `cmudict.json`, `english_*`: English pronunciations, vocabulary and reviewed corrections
-- `indic_words.json`: Indian words read natively; `ranker.json`: candidate ranker
-- `roman/<code>.json`: how people type each language's words in Latin, how often, and the other words
-  a typing can be (`alts`)
-- `native_freq/<token>.json`: word frequencies of the eleven languages without a `rescore/` list, for
-  the language's own words typed in Latin
-- `roman_ctx/<code>.json`: for typings that are two different words, how often each is seen next to
-  the words that tell them apart (sentence context)
-- `names/<code>.json`: Indian given names, surnames and place names typed in Latin and how the language
-  writes them (`tools/codemix_xlit/build_names.py`), read for a capitalised word (Sneha स्नेहा, not स्नेह)
-
-## Sources and licences
-`license: other` because the files carry different licences; each is listed here with its notice.
-
-- `model/`, `rescore/`, `native_freq/`: the IndicXlit model and word-frequency dictionaries, AI4Bharat,
-  MIT licence (https://github.com/AI4Bharat/IndicXlit). Notice:
-
-  > MIT License
-  >
-  > Copyright (c) 2022 AI4Bhārat
-  >
-  > Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
-  > associated documentation files (the "Software"), to deal in the Software without restriction,
-  > including without limitation the rights to use, copy, modify, merge, publish, distribute,
-  > sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
-  > furnished to do so, subject to the following conditions:
-  >
-  > The above copyright notice and this permission notice shall be included in all copies or
-  > substantial portions of the Software.
-  >
-  > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
-  > NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  > NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-  > DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
-  > OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-- `cmudict.json`: the CMU Pronouncing Dictionary (https://github.com/cmusphinx/cmudict), BSD-style
-  licence. Notice:
-
-  > Copyright (C) 1993-2015 Carnegie Mellon University. All rights reserved.
-  >
-  > Redistribution and use in source and binary forms, with or without modification, are permitted
-  > provided that the following conditions are met:
-  >
-  > 1. Redistributions of source code must retain the above copyright notice, this list of conditions
-  >    and the following disclaimer. The contents of this file are deemed to be source code.
-  > 2. Redistributions in binary form must reproduce the above copyright notice, this list of
-  >    conditions and the following disclaimer in the documentation and/or other materials provided
-  >    with the distribution.
-  >
-  > This work was supported in part by funding from the Defense Advanced Research Projects Agency, the
-  > Office of Naval Research and the National Science Foundation of the United States of America, and
-  > by member companies of the Carnegie Mellon Sphinx Speech Consortium. We acknowledge the
-  > contributions of many volunteers to the expansion and improvement of this dictionary.
-  >
-  > THIS SOFTWARE IS PROVIDED BY CARNEGIE MELLON UNIVERSITY ``AS IS'' AND ANY EXPRESSED OR IMPLIED
-  > WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
-  > FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY NOR ITS
-  > EMPLOYEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-  > DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-  > DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
-  > IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
-  > OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-- `english_vocab.txt`, `english_pron_vocab.txt`: English word lists derived from wordfreq
-  (https://github.com/rspeer/wordfreq), whose data is CC BY-SA 4.0
-  (https://creativecommons.org/licenses/by-sa/4.0/); these lists are shared under the same licence.
-- `roman/` of bn, gu, hi, kn, ml, mr, pa, sd, ta, te, ur: counts derived from the romanized sentences
-  of the Dakshina dataset (Roark et al. 2020, https://github.com/google-research-datasets/dakshina),
-  CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); these tables are shared under the
-  same licence.
-- `roman_ctx/`: neighbour counts from AI4Bharat Sangraha, verified subset (Khan et al. 2024,
-  https://huggingface.co/datasets/ai4bharat/sangraha), CC BY 4.0
-  (https://creativecommons.org/licenses/by/4.0/).
-- `names/`: the English and native-script labels of people with Indian citizenship, of the name items
-  they carry and of India's states, union territories and places of 100,000 people, from Wikidata (https://www.wikidata.org), CC0 1.0
-  (https://creativecommons.org/publicdomain/zero/1.0/), checked against the IndicXlit model above.
-- Lexicons, corrections, ranker and the other romanized tables: derived from the above as described
-  below.
-
-## Who wrote the generated parts
-- `lexicon/<code>.generated.json`: written offline by the layer's own pipeline (`build_assets.py
-  lexicon`: IndicXlit's candidates, spellings rendered from the pronunciation, the ranker). Its
-  entries that did not read back as the English word, among each language's 5,000 most frequent,
-  were reviewed by the LLM graders below and the errors corrected in the reviewed lexicon.
-- `english_extra_prons.json` (Indian-English readings of words the CMU dictionary lacks: names,
-  brands, foreign words), `english_pron_overrides.json`, `indic_words.json`, the gold spellings in
-  `lexicon/<code>.reviewed.json`, and the everyday words in `roman/`: written by a large language
-  model, run as per-language writer and grader agents under fixed spelling conventions. They were
-  reviewed by further agents of the same model (a reading is read
-  back against the English pronunciation; a spelling is graded C/A/W blind against plain
-  IndicXlit's). Two independent graders agreed on 96.2-97.7% of labels.
-- `ranker.json`: fitted on those graders' labels.
-- The accuracy figures published for this bundle come from the same model graders. **No native
-  speaker has reviewed any of these files yet**; treat them as unverified for external claims.

manifest.json

@@ -2,11 +2,11 @@
  "files": {
   "LICENSE": {
    "bytes": 4698,
-   "sha256": "15948212d7c8fef32faddca9221b4893a618cbf0bbfce8ae727bf0595e56fd7a"
+   "sha256": "cf33ed18850028c1e6756d8f4fc4aba3d73c470e4d7102bf60e4f07e91aa23a1"
   },
   "NOTICE": {
    "bytes": 1923,
-   "sha256": "4c8863ba6dbce31481e7bf67cc3541237e66568a627dae3cdacd59c0c02e7f54"
+   "sha256": "58caf9d6534b71bf493495ca5858ae48fb47e593ce84b61407718fd8adc5b630"
   },
   "cmudict.json": {
    "bytes": 4464854,
@@ -118,7 +118,7 @@
   },
   "lexicon/ml.reviewed.json": {
    "bytes": 273368,
-   "sha256": "dad1bde5a1d271b6237df612825b3d3e48e8e2104a49817098c79f21ddbf4d66"
+   "sha256": "bb4ca639ee6b73eaba752f36541ded0c9299d7daba85ebc2a828bfea60aa7ac1"
   },
   "lexicon/mni.generated.json": {
    "bytes": 3220999,
@@ -229,8 +229,8 @@
    "sha256": "3a69fd1b28ca150108df3649eadc7954d051d398d25c699a85039e667e3ca9bb"
   },
   "names/gu.json": {
-   "bytes": 191667,
-   "sha256": "6208a7de388c9f189dd13f123455f67b9173c5896cd3b432efbc882e3e075e4d"
+   "bytes": 191692,
+   "sha256": "ad6756adc5d90b8cc841459827cbff5470c61bebc78af562befc02accf53f821"
   },
   "names/hi.json": {
    "bytes": 193154,
@@ -253,8 +253,8 @@
    "sha256": "ec94c3dd375db1c2ee8b1327dfe90f47d28a2dde92324b3b6a97cbda72d627f5"
   },
   "names/ml.json": {
-   "bytes": 199122,
-   "sha256": "5886917b584d6838de060f92b144c8128d7e9bb046cc767868239a7ed87c6d33"
+   "bytes": 199144,
+   "sha256": "3103324b41c340b6d218655cb320a9c7873fbf041e9b52e64d3c7e7f52f6ec83"
   },
   "names/mni.json": {
    "bytes": 179994,
@@ -385,12 +385,12 @@
    "sha256": "a41e07ecdfabf400aecfde068fdaa06c76e777bbf6c846fabd91a59755cf8892"
   },
   "roman/as.json": {
-   "bytes": 455206,
-   "sha256": "a66a5fbfaa09c17adad05aa7825c1cfb816cc1ec2ee1df049c5a1da909424d5d"
+   "bytes": 455298,
+   "sha256": "3517745b1b9ac90b42b545a178a3d66e157157d7b8eb4e3016ad2b41c22c99cc"
   },
   "roman/bn.json": {
-   "bytes": 524325,
-   "sha256": "e1c6342da31ee8fd2555228304d0dff618f0e48918482761f156cb9db6c9f58d"
+   "bytes": 524568,
+   "sha256": "c3a3dead063f8e4017af833383942305b02a9fcc2f7d8aafb867d234c01aff58"
   },
   "roman/brx.json": {
    "bytes": 373560,
@@ -401,16 +401,16 @@
    "sha256": "db6f5446711a770538e5a03527cb20af2467dd40e9e20db7e1e013557790a952"
   },
   "roman/gu.json": {
-   "bytes": 437007,
-   "sha256": "87f82a7420f95a9b2a760705a9733d02b01859c007aef9e67fb02d83ebb5bce2"
+   "bytes": 437062,
+   "sha256": "7bb02ef1c17bdbf97b58e4a1ce6ea03f5eb7c204e2119bdb20903f0e89555c80"
   },
   "roman/hi.json": {
-   "bytes": 431851,
-   "sha256": "e625676d5b59f8bc8825473180323055e554a689c1038b3c65bd48cd5b478920"
+   "bytes": 432372,
+   "sha256": "9ba2121b06d13daade5adca0dfd79bb085378212a86fe313a17df2d414b8253d"
   },
   "roman/kn.json": {
-   "bytes": 605271,
-   "sha256": "263e50039ac99179371c0e59d5f45ea19cf153ae469f81cb575819c27d07d81a"
+   "bytes": 605568,
+   "sha256": "119ec11bb9e3d457c02a654d62ca656df4c01f4ee90f627c04a0ec1a49758fce"
   },
   "roman/knn.json": {
    "bytes": 391716,
@@ -425,28 +425,28 @@
    "sha256": "45ef988a0d720483dee6122aa7221b479f203e90e329e8f2f6e057cf728239fd"
   },
   "roman/ml.json": {
-   "bytes": 792120,
-   "sha256": "73beb2452c0f1d396ac65a509facd7b22db622fbdcba56a668fda471ea7d778e"
+   "bytes": 792370,
+   "sha256": "001627ca958267ba7b157b3b8f41e28e1f912b4c245319edd39fa84fb945bd68"
   },
   "roman/mni.json": {
    "bytes": 189975,
    "sha256": "97222729dd156238bf850424697c9e2c862d31bc0122981b932602ce45fb8899"
   },
   "roman/mr.json": {
-   "bytes": 492617,
-   "sha256": "88692298fc1af165516be36832af5c5854708b1c746b0c5363f47f1716057045"
+   "bytes": 492731,
+   "sha256": "5d24a53a92b8cfb4f2182704f12430c1e9bee3ccca3cc867a04e8f4c16a4361b"
   },
   "roman/npi.json": {
    "bytes": 409495,
    "sha256": "942bbc70fa6fd47648ceec3693c0800baeef87da010dadf3702a042d57f3caed"
   },
   "roman/ory.json": {
-   "bytes": 451809,
-   "sha256": "109b180fd84e8748c1802b650d38f12acbae435c976f9c8e60fa9bc2d3990294"
+   "bytes": 451975,
+   "sha256": "86ba5981bcbd1ee00e85c59eb183e5f20a8002d2391658e1a3f68dd30e697c46"
   },
   "roman/pa.json": {
-   "bytes": 433081,
-   "sha256": "ec138c259bb20972f07c55cd07e5666419c4f2da8711f63223d69ed2805e0bbd"
+   "bytes": 433374,
+   "sha256": "d6365ddd7cd2425b7978aab75931e20236049e386497af259f45df28242c9798"
   },
   "roman/sa.json": {
    "bytes": 480676,
@@ -457,12 +457,12 @@
    "sha256": "e5591262d21a5a29fd3c70810a921239fc57a46cc04c1e532b8710c00dd6cbfa"
   },
   "roman/ta.json": {
-   "bytes": 679033,
-   "sha256": "734246d7c52921b9139440f1a6fba60804fbcc32d39538bbf91b283265f67fe4"
+   "bytes": 679837,
+   "sha256": "4fb325ad488b381cdbb316821f9a3045dbe7f5bb50dd01c8cfa984ec7eabf9ac"
   },
   "roman/te.json": {
-   "bytes": 696052,
-   "sha256": "62581297669ed8ed4479aa0e0c111238705c2f13271deaaaacfd15434b76186b"
+   "bytes": 696182,
+   "sha256": "09e98c8fe7dd459174552ffc26e558f1daacb4f87a26743daffbb2112a5ae298"
   },
   "roman/ur.json": {
    "bytes": 321608,
@@ -556,6 +556,6 @@
  "model_source": "https://github.com/AI4Bharat/IndicXlit/releases/download/v1.0/indicxlit-en-indic-v1.0.zip",
  "pronunciations": "CMU Pronouncing Dictionary (cmudict)",
  "rescore_source": "https://github.com/AI4Bharat/IndicXlit/releases/download/v1.0/word_prob_dicts.zip",
- "version": "1.4.15+631ae6081786",
+ "version": "1.4.16+12d0f820ea16",
  "vocab": "wordfreq top English words"
 }

names/gu.json

@@ -1337,6 +1337,7 @@
 "dhanraj": "ધનરાજ",
 "dhanya": "ધન્યા",
 "dhar": "ધર",
+"dhara": "ધારા",
 "dharam": "ધરમ",
 "dharamvir": "ધર્મવીર",
 "dharashiv": "ધારાશિવ",

names/ml.json

@@ -5298,6 +5298,7 @@
 "thankam": "തങ്കം",
 "thapa": "ഥാപ്പ",
 "thapar": "ഥാപ്പർ",
+"thara": "താര",
 "thavasi": "തവസി",
 "theni": "തേനി",
 "thennavan": "തെന്നവൻ",