codemix_xlit assets 1.4.6+7d2bae823016

LICENSE

@@ -1,4 +1,4 @@
-indictrans (codemix_xlit assets) version 1.4.5: licences of the files in this bundle
+indictrans (codemix_xlit assets) version 1.4.6: licences of the files in this bundle
 
 Sources and licences
 `license: other` because the files carry different licences; each is listed here with its notice.

NOTICE

@@ -1,5 +1,5 @@
 indictrans: runtime assets of the Finesse TTS code-mix transliteration layer (utils/codemix_xlit)
-Version 1.4.5
+Version 1.4.6
 
 This bundle redistributes and adapts third-party material. Each part keeps its own licence (see
 LICENSE):

README.md

@@ -1,117 +0,0 @@
----
-license: other
-language: [hi, mr, ne, mai, sa, brx, kok, doi, gu, pa, or, ta, te, kn, ml, mni, bn, as, ur, sd, ks, sat]
-tags: [transliteration, code-mixing, text-to-speech, indic]
----
-# indictrans: English words in Indian-language text, written in the request's script
-
-Runtime assets of the code-mix transliteration layer of the Finesse TTS (`utils/codemix_xlit`):
-English words inside Indic text are rewritten in the language's script, spelled the way an
-Indian-English speaker says them, so an Indic voice reads them correctly; text typed entirely in
-Latin is read word by word, the language's own words back into its script. 22 languages (Dogri and
-Santali through the Hindi model).
-
-Version 1.4.5+d240dd9acaaa. Built by `tools/codemix_xlit/build_assets.py`; `manifest.json` lists every file
-with its size and SHA-256.
-
-## Contents
-- `LICENSE`, `NOTICE`: the licences and attributions below, inside the bundle, so a copy of the folder
-  keeps them
-- `model/`: the IndicXlit en->indic v1.0 transformer converted to CTranslate2
-- `lexicon/<code>.reviewed.json`, `lexicon/<code>.generated.json`: reviewed and generated spellings
-- `rescore/`: pruned IndicXlit word-frequency dictionaries
-- `cmudict.json`, `english_*`: English pronunciations, vocabulary and reviewed corrections
-- `indic_words.json`: Indian words read natively; `ranker.json`: candidate ranker
-- `roman/<code>.json`: how people type each language's words in Latin, how often, and the other words
-  a typing can be (`alts`)
-- `native_freq/<token>.json`: word frequencies of the eleven languages without a `rescore/` list, for
-  the language's own words typed in Latin
-- `roman_ctx/<code>.json`: for typings that are two different words, how often each is seen next to
-  the words that tell them apart (sentence context)
-- `names/<code>.json`: Indian given names, surnames and place names typed in Latin and how the language
-  writes them (`tools/codemix_xlit/build_names.py`), read for a capitalised word (Sneha स्नेहा, not स्नेह)
-
-## Sources and licences
-`license: other` because the files carry different licences; each is listed here with its notice.
-
-- `model/`, `rescore/`, `native_freq/`: the IndicXlit model and word-frequency dictionaries, AI4Bharat,
-  MIT licence (https://github.com/AI4Bharat/IndicXlit). Notice:
-
-  > MIT License
-  >
-  > Copyright (c) 2022 AI4Bhārat
-  >
-  > Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
-  > associated documentation files (the "Software"), to deal in the Software without restriction,
-  > including without limitation the rights to use, copy, modify, merge, publish, distribute,
-  > sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
-  > furnished to do so, subject to the following conditions:
-  >
-  > The above copyright notice and this permission notice shall be included in all copies or
-  > substantial portions of the Software.
-  >
-  > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
-  > NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  > NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-  > DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
-  > OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-- `cmudict.json`: the CMU Pronouncing Dictionary (https://github.com/cmusphinx/cmudict), BSD-style
-  licence. Notice:
-
-  > Copyright (C) 1993-2015 Carnegie Mellon University. All rights reserved.
-  >
-  > Redistribution and use in source and binary forms, with or without modification, are permitted
-  > provided that the following conditions are met:
-  >
-  > 1. Redistributions of source code must retain the above copyright notice, this list of conditions
-  >    and the following disclaimer. The contents of this file are deemed to be source code.
-  > 2. Redistributions in binary form must reproduce the above copyright notice, this list of
-  >    conditions and the following disclaimer in the documentation and/or other materials provided
-  >    with the distribution.
-  >
-  > This work was supported in part by funding from the Defense Advanced Research Projects Agency, the
-  > Office of Naval Research and the National Science Foundation of the United States of America, and
-  > by member companies of the Carnegie Mellon Sphinx Speech Consortium. We acknowledge the
-  > contributions of many volunteers to the expansion and improvement of this dictionary.
-  >
-  > THIS SOFTWARE IS PROVIDED BY CARNEGIE MELLON UNIVERSITY ``AS IS'' AND ANY EXPRESSED OR IMPLIED
-  > WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
-  > FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY NOR ITS
-  > EMPLOYEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-  > DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-  > DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
-  > IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
-  > OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-- `english_vocab.txt`, `english_pron_vocab.txt`: English word lists derived from wordfreq
-  (https://github.com/rspeer/wordfreq), whose data is CC BY-SA 4.0
-  (https://creativecommons.org/licenses/by-sa/4.0/); these lists are shared under the same licence.
-- `roman/` of bn, gu, hi, kn, ml, mr, pa, sd, ta, te, ur: counts derived from the romanized sentences
-  of the Dakshina dataset (Roark et al. 2020, https://github.com/google-research-datasets/dakshina),
-  CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); these tables are shared under the
-  same licence.
-- `roman_ctx/`: neighbour counts from AI4Bharat Sangraha, verified subset (Khan et al. 2024,
-  https://huggingface.co/datasets/ai4bharat/sangraha), CC BY 4.0
-  (https://creativecommons.org/licenses/by/4.0/).
-- `names/`: the English and native-script labels of people with Indian citizenship, of the name items
-  they carry and of India's states, union territories and places of 100,000 people, from Wikidata (https://www.wikidata.org), CC0 1.0
-  (https://creativecommons.org/publicdomain/zero/1.0/), checked against the IndicXlit model above.
-- Lexicons, corrections, ranker and the other romanized tables: derived from the above as described
-  below.
-
-## Who wrote the generated parts
-- `lexicon/<code>.generated.json`: written offline by the layer's own pipeline (`build_assets.py
-  lexicon`: IndicXlit's candidates, spellings rendered from the pronunciation, the ranker). Its
-  entries that did not read back as the English word, among each language's 5,000 most frequent,
-  were reviewed by the LLM graders below and the errors corrected in the reviewed lexicon.
-- `english_extra_prons.json` (Indian-English readings of words the CMU dictionary lacks: names,
-  brands, foreign words), `english_pron_overrides.json`, `indic_words.json`, the gold spellings in
-  `lexicon/<code>.reviewed.json`, and the everyday words in `roman/`: written by a large language
-  model, run as per-language writer and grader agents under fixed spelling conventions. They were
-  reviewed by further agents of the same model (a reading is read
-  back against the English pronunciation; a spelling is graded C/A/W blind against plain
-  IndicXlit's). Two independent graders agreed on 96.2-97.7% of labels.
-- `ranker.json`: fitted on those graders' labels.
-- The accuracy figures published for this bundle come from the same model graders. **No native
-  speaker has reviewed any of these files yet**; treat them as unverified for external claims.

manifest.json

@@ -2,11 +2,11 @@
  "files": {
   "LICENSE": {
    "bytes": 4697,
-   "sha256": "72e013267004e79d2632ef2fb154e61dbe25a6cf7c129484d3fd4b0c71f9fd6e"
+   "sha256": "8979fa054ddbcb5162b9377cf53786b035944b1ba7ca8cb6a972267ceccc8f36"
   },
   "NOTICE": {
    "bytes": 1922,
-   "sha256": "ebf42bfa6987f387768953c5607d09267df9facf51215da752d46fa8f90bfc7c"
+   "sha256": "fae31ab60617fa0b0b46e3f20a18a6c8139120ced7b93c81694786a387789edb"
   },
   "cmudict.json": {
    "bytes": 4464854,
@@ -154,7 +154,7 @@
   },
   "lexicon/pa.generated.json": {
    "bytes": 3276407,
-   "sha256": "3c5d81854975b500dfdd894f9ddaf705e0a7449efd7218b1cac4f4a2a48fec65"
+   "sha256": "51dca8b64d4ee57186a6ffa6aa06b77ddccfab9554a140f24214eb24f56f321a"
   },
   "lexicon/pa.reviewed.json": {
    "bytes": 194204,
@@ -233,8 +233,8 @@
    "sha256": "380cf3f4ef4849f6d579ff0e0d83e26a68edde87d057f95f7880d5530cf4365a"
   },
   "names/hi.json": {
-   "bytes": 193108,
-   "sha256": "78aed34c69b41546f3afc25d2f29039646a7152d891f02fefe10dfec9390816b"
+   "bytes": 193133,
+   "sha256": "1ab686620c5cea198743e128f8f8a774ed2e39edaee19ed0767d7ad2c9b3f6a2"
   },
   "names/kn.json": {
    "bytes": 200657,
@@ -293,8 +293,8 @@
    "sha256": "6fbda748720808373885f25e339e76bd2983b2e6f822020c93cf74d98dc5cdde"
   },
   "names/ur.json": {
-   "bytes": 115264,
-   "sha256": "5faffd278d5e9c8e24478d17d527c717dcc5b0b331a5bdac9c126dc7a68bd766"
+   "bytes": 115285,
+   "sha256": "4c9ff541cdd8c7b0c9f3dab3bf81ba35ac9a88434bdea17d68d8ad0ec264413c"
   },
   "native_freq/as.json": {
    "bytes": 1423749,
@@ -457,8 +457,8 @@
    "sha256": "e5591262d21a5a29fd3c70810a921239fc57a46cc04c1e532b8710c00dd6cbfa"
   },
   "roman/ta.json": {
-   "bytes": 678723,
-   "sha256": "26ec1c6d4a6cc3497e58f347940239e532e3ddda7273dc5935efc5b425cea7ba"
+   "bytes": 678724,
+   "sha256": "383bf624402f735482972500db3d3e6865d82e9f46001ca0f1992088c984b204"
   },
   "roman/te.json": {
    "bytes": 696049,
@@ -556,6 +556,6 @@
  "model_source": "https://github.com/AI4Bharat/IndicXlit/releases/download/v1.0/indicxlit-en-indic-v1.0.zip",
  "pronunciations": "CMU Pronouncing Dictionary (cmudict)",
  "rescore_source": "https://github.com/AI4Bharat/IndicXlit/releases/download/v1.0/word_prob_dicts.zip",
- "version": "1.4.5+d240dd9acaaa",
+ "version": "1.4.6+7d2bae823016",
  "vocab": "wordfreq top English words"
 }

names/hi.json

@@ -2357,6 +2357,7 @@
 "karauli": "करौली",
 "karayalar": "करायालर",
 "karbi": "कार्बी",
+"karen": "केरन",
 "kargil": "कारगिल",
 "karim": "करीम",
 "karimganj": "करीमगंज",

names/ur.json

@@ -1866,6 +1866,7 @@
 "karanth": "کرنت",
 "karayalar": "کرایالر",
 "karbi": "کاربی",
+"karen": "کیرن",
 "kargil": "کارگل",
 "karim": "کریم",
 "karimganj": "کریمگنج",