codemix_xlit assets 1.4.14+4f6061cdc9f8

LICENSE

@@ -1,4 +1,4 @@
-indictrans (codemix_xlit assets) version 1.4.13: licences of the files in this bundle
+indictrans (codemix_xlit assets) version 1.4.14: licences of the files in this bundle
 
 Sources and licences
 `license: other` because the files carry different licences; each is listed here with its notice.

NOTICE

@@ -1,5 +1,5 @@
 indictrans: runtime assets of the Finesse TTS code-mix transliteration layer (utils/codemix_xlit)
-Version 1.4.13
+Version 1.4.14
 
 This bundle redistributes and adapts third-party material. Each part keeps its own licence (see
 LICENSE):

README.md

@@ -1,117 +0,0 @@
----
-license: other
-language: [hi, mr, ne, mai, sa, brx, kok, doi, gu, pa, or, ta, te, kn, ml, mni, bn, as, ur, sd, ks, sat]
-tags: [transliteration, code-mixing, text-to-speech, indic]
----
-# indictrans: English words in Indian-language text, written in the request's script
-
-Runtime assets of the code-mix transliteration layer of the Finesse TTS (`utils/codemix_xlit`):
-English words inside Indic text are rewritten in the language's script, spelled the way an
-Indian-English speaker says them, so an Indic voice reads them correctly; text typed entirely in
-Latin is read word by word, the language's own words back into its script. 22 languages (Dogri and
-Santali through the Hindi model).
-
-Version 1.4.13+a80ebb8ecb94. Built by `tools/codemix_xlit/build_assets.py`; `manifest.json` lists every file
-with its size and SHA-256.
-
-## Contents
-- `LICENSE`, `NOTICE`: the licences and attributions below, inside the bundle, so a copy of the folder
-  keeps them
-- `model/`: the IndicXlit en->indic v1.0 transformer converted to CTranslate2
-- `lexicon/<code>.reviewed.json`, `lexicon/<code>.generated.json`: reviewed and generated spellings
-- `rescore/`: pruned IndicXlit word-frequency dictionaries
-- `cmudict.json`, `english_*`: English pronunciations, vocabulary and reviewed corrections
-- `indic_words.json`: Indian words read natively; `ranker.json`: candidate ranker
-- `roman/<code>.json`: how people type each language's words in Latin, how often, and the other words
-  a typing can be (`alts`)
-- `native_freq/<token>.json`: word frequencies of the eleven languages without a `rescore/` list, for
-  the language's own words typed in Latin
-- `roman_ctx/<code>.json`: for typings that are two different words, how often each is seen next to
-  the words that tell them apart (sentence context)
-- `names/<code>.json`: Indian given names, surnames and place names typed in Latin and how the language
-  writes them (`tools/codemix_xlit/build_names.py`), read for a capitalised word (Sneha स्नेहा, not स्नेह)
-
-## Sources and licences
-`license: other` because the files carry different licences; each is listed here with its notice.
-
-- `model/`, `rescore/`, `native_freq/`: the IndicXlit model and word-frequency dictionaries, AI4Bharat,
-  MIT licence (https://github.com/AI4Bharat/IndicXlit). Notice:
-
-  > MIT License
-  >
-  > Copyright (c) 2022 AI4Bhārat
-  >
-  > Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
-  > associated documentation files (the "Software"), to deal in the Software without restriction,
-  > including without limitation the rights to use, copy, modify, merge, publish, distribute,
-  > sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
-  > furnished to do so, subject to the following conditions:
-  >
-  > The above copyright notice and this permission notice shall be included in all copies or
-  > substantial portions of the Software.
-  >
-  > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
-  > NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-  > NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
-  > DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
-  > OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-- `cmudict.json`: the CMU Pronouncing Dictionary (https://github.com/cmusphinx/cmudict), BSD-style
-  licence. Notice:
-
-  > Copyright (C) 1993-2015 Carnegie Mellon University. All rights reserved.
-  >
-  > Redistribution and use in source and binary forms, with or without modification, are permitted
-  > provided that the following conditions are met:
-  >
-  > 1. Redistributions of source code must retain the above copyright notice, this list of conditions
-  >    and the following disclaimer. The contents of this file are deemed to be source code.
-  > 2. Redistributions in binary form must reproduce the above copyright notice, this list of
-  >    conditions and the following disclaimer in the documentation and/or other materials provided
-  >    with the distribution.
-  >
-  > This work was supported in part by funding from the Defense Advanced Research Projects Agency, the
-  > Office of Naval Research and the National Science Foundation of the United States of America, and
-  > by member companies of the Carnegie Mellon Sphinx Speech Consortium. We acknowledge the
-  > contributions of many volunteers to the expansion and improvement of this dictionary.
-  >
-  > THIS SOFTWARE IS PROVIDED BY CARNEGIE MELLON UNIVERSITY ``AS IS'' AND ANY EXPRESSED OR IMPLIED
-  > WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
-  > FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY NOR ITS
-  > EMPLOYEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-  > DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-  > DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
-  > IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
-  > OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-- `english_vocab.txt`, `english_pron_vocab.txt`: English word lists derived from wordfreq
-  (https://github.com/rspeer/wordfreq), whose data is CC BY-SA 4.0
-  (https://creativecommons.org/licenses/by-sa/4.0/); these lists are shared under the same licence.
-- `roman/` of bn, gu, hi, kn, ml, mr, pa, sd, ta, te, ur: counts derived from the romanized sentences
-  of the Dakshina dataset (Roark et al. 2020, https://github.com/google-research-datasets/dakshina),
-  CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); these tables are shared under the
-  same licence.
-- `roman_ctx/`: neighbour counts from AI4Bharat Sangraha, verified subset (Khan et al. 2024,
-  https://huggingface.co/datasets/ai4bharat/sangraha), CC BY 4.0
-  (https://creativecommons.org/licenses/by/4.0/).
-- `names/`: the English and native-script labels of people with Indian citizenship, of the name items
-  they carry and of India's states, union territories and places of 100,000 people, from Wikidata (https://www.wikidata.org), CC0 1.0
-  (https://creativecommons.org/publicdomain/zero/1.0/), checked against the IndicXlit model above.
-- Lexicons, corrections, ranker and the other romanized tables: derived from the above as described
-  below.
-
-## Who wrote the generated parts
-- `lexicon/<code>.generated.json`: written offline by the layer's own pipeline (`build_assets.py
-  lexicon`: IndicXlit's candidates, spellings rendered from the pronunciation, the ranker). Its
-  entries that did not read back as the English word, among each language's 5,000 most frequent,
-  were reviewed by the LLM graders below and the errors corrected in the reviewed lexicon.
-- `english_extra_prons.json` (Indian-English readings of words the CMU dictionary lacks: names,
-  brands, foreign words), `english_pron_overrides.json`, `indic_words.json`, the gold spellings in
-  `lexicon/<code>.reviewed.json`, and the everyday words in `roman/`: written by a large language
-  model, run as per-language writer and grader agents under fixed spelling conventions. They were
-  reviewed by further agents of the same model (a reading is read
-  back against the English pronunciation; a spelling is graded C/A/W blind against plain
-  IndicXlit's). Two independent graders agreed on 96.2-97.7% of labels.
-- `ranker.json`: fitted on those graders' labels.
-- The accuracy figures published for this bundle come from the same model graders. **No native
-  speaker has reviewed any of these files yet**; treat them as unverified for external claims.

manifest.json

@@ -2,11 +2,11 @@
  "files": {
   "LICENSE": {
    "bytes": 4698,
-   "sha256": "93e90b95dff02a3d6907a43b8acb2e72a89bd29d7a0efeb563f75a24ac93708b"
+   "sha256": "e193f82aba7b871d138a9cd851e0944ecf4a53b0cd25167f5f5d9bf14d5aa629"
   },
   "NOTICE": {
    "bytes": 1923,
-   "sha256": "4bbd55c489c0d5e4c81327f7aa09443ba5e784ef1bb2d9547fbd61075cc5c9ad"
+   "sha256": "43bc5dba28bcce828c20436d8aabd743216ce08459a70bd585a9d751abe6bb97"
   },
   "cmudict.json": {
    "bytes": 4464854,
@@ -69,8 +69,8 @@
    "sha256": "8a143ebb13d218b6789def81499e0938de6a025bb0127175b4bf2b10c1afc5fe"
   },
   "lexicon/gu.reviewed.json": {
-   "bytes": 189282,
-   "sha256": "5d0e4866387066450a8ecebd5fe59fb200009bf230c58b979a82d99d7a903324"
+   "bytes": 189302,
+   "sha256": "3a27b95527b83897ab3ef11aa28a04a48d67108cc3b10afcb835db21a6776f11"
   },
   "lexicon/hi.generated.json": {
    "bytes": 3504173,
@@ -197,8 +197,8 @@
    "sha256": "acbff17132d5d027bdd63d2a0fbe3eb330180e44f116c594f77171608ec920e2"
   },
   "lexicon/ur.reviewed.json": {
-   "bytes": 136840,
-   "sha256": "cf885f5661b7e935b880f78927bce2f8c6d55f28329bd507258ba2280cff8cdc"
+   "bytes": 136857,
+   "sha256": "4e7e78f52f36e2abbd5e5a64265c9e1dc232138da381ac25132a1965238d51be"
   },
   "model/config.json": {
    "bytes": 222,
@@ -389,8 +389,8 @@
    "sha256": "a66a5fbfaa09c17adad05aa7825c1cfb816cc1ec2ee1df049c5a1da909424d5d"
   },
   "roman/bn.json": {
-   "bytes": 524128,
-   "sha256": "0b2dd7b2436ba1d53b3a735382d6bd412793d84f9ec858319f06ca525c44122b"
+   "bytes": 524183,
+   "sha256": "1bb45294fd804925e248cfb9f0ac3f6eee460889671391901de1b88654b4b83f"
   },
   "roman/brx.json": {
    "bytes": 373560,
@@ -433,8 +433,8 @@
    "sha256": "97222729dd156238bf850424697c9e2c862d31bc0122981b932602ce45fb8899"
   },
   "roman/mr.json": {
-   "bytes": 492500,
-   "sha256": "bcbdd9d779d62077e48034603d3a43edb73aa49cf1e11cf08609aaa049c39860"
+   "bytes": 492524,
+   "sha256": "42e9ad27b9d7704ffa2b53647e7611a2a6e7dedafc5c726761e91c6c4b17c559"
   },
   "roman/npi.json": {
    "bytes": 409495,
@@ -556,6 +556,6 @@
  "model_source": "https://github.com/AI4Bharat/IndicXlit/releases/download/v1.0/indicxlit-en-indic-v1.0.zip",
  "pronunciations": "CMU Pronouncing Dictionary (cmudict)",
  "rescore_source": "https://github.com/AI4Bharat/IndicXlit/releases/download/v1.0/word_prob_dicts.zip",
- "version": "1.4.13+a80ebb8ecb94",
+ "version": "1.4.14+4f6061cdc9f8",
  "vocab": "wordfreq top English words"
 }