Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,110 +1,111 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: mpl-2.0
|
| 3 |
-
language: [en, de, fr, es, it, nl, pt, pl, sv, da, nb, fi, ja, ko, zh]
|
| 4 |
-
tags: [translation, bergamot, marian, int8, language-identification]
|
| 5 |
-
pipeline_tag: translation
|
| 6 |
-
---
|
| 7 |
-
|
| 8 |
-
# Ordicio on-device translator
|
| 9 |
-
|
| 10 |
-
Small translation models for the 16 languages of the Ordicio apps, which translate attribute values and data
|
| 11 |
-
sheets of collectors' items on the device. There is one model per direction between English and each language;
|
| 12 |
-
other pairs go through English. A language detector of 2.2 MB (0.75 MB to download) tells which model to use.
|
| 13 |
-
|
| 14 |
-
| | |
|
| 15 |
-
| --- | --- |
|
| 16 |
-
| Languages | en, de, fr, es, it, nl, pt (Brazil), pl, sv, da, nb, fi, ja, ko, zh-Hans, zh-Hant |
|
| 17 |
-
| Models | 30 directions (en↔x), transformer encoder + SSRU decoder, int8 (bergamot-translator `intgemm8` with alphas) |
|
| 18 |
-
| Download | about 23 MB per direction, 13 MB for sv, da, nb and 34 MB for ja, ko, zh (gzip compressed); the detector 0.75 MB |
|
| 19 |
-
| Speed | about 18 ms per value in the WebAssembly engine on one core |
|
| 20 |
-
| Base | the [Firefox Translations](https://github.com/mozilla/translations) students of each direction |
|
| 21 |
-
| Training | fine-tuned on collecting texts translated by Gemma 4 31B, plus a quantization-aware epoch |
|
| 22 |
-
|
| 23 |
-
## Files
|
| 24 |
-
|
| 25 |
-
`manifest.json` lists every file with its sha256 and size (of the stored, gzip-compressed file) and the
|
| 26 |
-
unpacked size:
|
| 27 |
-
|
| 28 |
-
```json
|
| 29 |
-
{
|
| 30 |
-
"id": "ordicio-translator-v1",
|
| 31 |
-
"languages": ["en", "de", "..."],
|
| 32 |
-
"pivot": "en",
|
| 33 |
-
"engine": {"beam-size": "1", "gemm-precision": "int8shiftAlphaAll", "...": "..."},
|
| 34 |
-
"pairs": {
|
| 35 |
-
"en-de": {"from": "en", "to": "de",
|
| 36 |
-
"model": {"path": "v1/en-de/model.bin.gz", "sha256": "…", "size": 0, "gzip": true, "unpackedSize": 0},
|
| 37 |
-
"srcVocab": {"path": "v1/en-de/vocab.spm.gz", "...": "..."}, "trgVocab": {"...": "..."}}
|
| 38 |
-
},
|
| 39 |
-
"lid": {"model": {"path": "v1/lid.bin.gz", "...": "..."}, "labels": ["en", "...", "zxx"], "buckets": 131072,
|
| 40 |
-
"ngrams": [1, 4], "scale": 0.0, "bias": [], "maxChars": 1000, "hash": "fnv1a32-utf8"}
|
| 41 |
-
}
|
| 42 |
-
```
|
| 43 |
-
|
| 44 |
-
- Paths are relative to the manifest. `srcVocab` and `trgVocab` name the same file when the direction has one
|
| 45 |
-
vocabulary.
|
| 46 |
-
- `engine` holds the bergamot-translator options the models were evaluated with. They run without a lexical
|
| 47 |
-
shortlist (it cost quality on short values).
|
| 48 |
-
- `source/v1/<src>-<tgt>.npz` are the float Marian weights of each model (the form to change them in); the
|
| 49 |
-
apps do not download them.
|
| 50 |
-
|
| 51 |
-
## Language detector
|
| 52 |
-
|
| 53 |
-
A linear model over hashed character n-grams with 17 labels: the 16 languages and `zxx` (names, codes, model
|
| 54 |
-
numbers: nothing to translate). `lid.bin` is int8, `buckets` × `labels` (the weights of one bucket together).
|
| 55 |
-
|
| 56 |
-
1. Text → lower case → every digit `0`-`9` to `0` → runs of whitespace (space, tab, line feed, carriage
|
| 57 |
-
return, form feed, vertical tab, U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F, U+3000)
|
| 58 |
-
to one space → spaces trimmed → the first `maxChars` code points. No Unicode normalization.
|
| 59 |
-
2. Each word `w` (split on spaces) is padded to `" " + w + " "`; every n-gram of 1 to 4 code points of the
|
| 60 |
-
padded word except a single space counts once.
|
| 61 |
-
3. The bucket of an n-gram is the 32-bit FNV-1a hash of its UTF-8 bytes modulo `buckets`. The feature vector
|
| 62 |
-
holds the count per bucket, L2 normalized.
|
| 63 |
-
4. Score of label `i` = `bias[i] + scale × Σ feature × weight[bucket][i]`; softmax gives probabilities.
|
| 64 |
-
|
| 65 |
-
| Accuracy (16 languages) | this detector | Lingua | fastText lid.176 |
|
| 66 |
-
| --- | --- | --- | --- |
|
| 67 |
-
| short values (held out, 6,877) | **0.89** | 0.83 | 0.74 |
|
| 68 |
-
| sentences (WMT / NTREX held out, 2,649) | **1.00** | 0.99 | 0.98 |
|
| 69 |
-
|
| 70 |
-
Short values are often ambiguous (Danish and Norwegian, Spanish and Portuguese, Chinese characters in Japanese).
|
| 71 |
-
On them, 78 % of the predictions reach a probability of 0.9, and 98.7 % of those are right.
|
| 72 |
-
|
| 73 |
-
## Quality
|
| 74 |
-
|
| 75 |
-
chrF++ in the bergamot WebAssembly engine without shortlist, against the shipped Firefox model of the same
|
| 76 |
-
direction. Test: 150 English values of a collecting catalog (examples, list values, member notes) with
|
| 77 |
-
reference translations.
|
| 78 |
-
|
| 79 |
-
| en → | de | fr | es | pl | fi | ja | ko | zh-Hans |
|
| 80 |
-
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 81 |
-
| Firefox | 69.1 | 67.5 | 67.8 | 60.0 | 60.7 | 32.5 | 42.5 | 25.1 |
|
| 82 |
-
| this | **70.7** | **70.9** | **69.2** | **64.4** | **65.1** | **40.6** | **
|
| 83 |
-
|
| 84 |
-
Values that every reference keeps unchanged (names, titles, codes) stay unchanged twice as often (9 to 16 of
|
| 85 |
-
19 instead of 1 to 11).
|
| 86 |
-
|
| 87 |
-
| → en (the references as source) | de | fr | es | pl | fi | ja | ko | zh-Hans |
|
| 88 |
-
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 89 |
-
| Firefox | 75.0 | 74.4 | 73.6 | 69.3 | 66.0 | 58.2 | 57.9 | 55.0 |
|
| 90 |
-
| this | **76.8** | **74.7** | **75.4** | **69.4** | **67.4** | **59.2** | **61.9** | **59.0** |
|
| 91 |
-
|
| 92 |
-
it, nl, pt, sv, da, nb and zh-Hant have no reference set. On 300 held-out teacher translations they gain 4 to
|
| 93 |
-
32 chrF++ over the Firefox models from English and 1 to 9 into English.
|
| 94 |
-
|
| 95 |
-
## Training
|
| 96 |
-
|
| 97 |
-
1. 10,900 English collecting texts (attribute values of a classification catalog, member notes) translated by
|
| 98 |
-
Gemma 4 31B (4-bit) into the 15 languages.
|
| 99 |
-
2. Each Firefox student (float weights from Mozilla's published training runs) fine-tuned on these pairs (both
|
| 100 |
-
directions; for x → en the teacher's translation is the source) mixed with WMT and NTREX-128 sentences:
|
| 101 |
-
4 epochs, learning rate 2e-5, then one quantization-aware epoch (8 bit), so the int8 model scores like the
|
| 102 |
-
float one.
|
| 103 |
-
3. For Korean, teacher
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mpl-2.0
|
| 3 |
+
language: [en, de, fr, es, it, nl, pt, pl, sv, da, nb, fi, ja, ko, zh]
|
| 4 |
+
tags: [translation, bergamot, marian, int8, language-identification]
|
| 5 |
+
pipeline_tag: translation
|
| 6 |
+
---
|
| 7 |
+
|
| 8 |
+
# Ordicio on-device translator
|
| 9 |
+
|
| 10 |
+
Small translation models for the 16 languages of the Ordicio apps, which translate attribute values and data
|
| 11 |
+
sheets of collectors' items on the device. There is one model per direction between English and each language;
|
| 12 |
+
other pairs go through English. A language detector of 2.2 MB (0.75 MB to download) tells which model to use.
|
| 13 |
+
|
| 14 |
+
| | |
|
| 15 |
+
| --- | --- |
|
| 16 |
+
| Languages | en, de, fr, es, it, nl, pt (Brazil), pl, sv, da, nb, fi, ja, ko, zh-Hans, zh-Hant |
|
| 17 |
+
| Models | 30 directions (en↔x), transformer encoder + SSRU decoder, int8 (bergamot-translator `intgemm8` with alphas) |
|
| 18 |
+
| Download | about 23 MB per direction, 13 MB for sv, da, nb and 34 MB for ja, ko, zh (gzip compressed); the detector 0.75 MB |
|
| 19 |
+
| Speed | about 18 ms per value in the WebAssembly engine on one core |
|
| 20 |
+
| Base | the [Firefox Translations](https://github.com/mozilla/translations) students of each direction |
|
| 21 |
+
| Training | fine-tuned on collecting texts translated by Gemma 4 31B, plus a quantization-aware epoch |
|
| 22 |
+
|
| 23 |
+
## Files
|
| 24 |
+
|
| 25 |
+
`manifest.json` lists every file with its sha256 and size (of the stored, gzip-compressed file) and the
|
| 26 |
+
unpacked size:
|
| 27 |
+
|
| 28 |
+
```json
|
| 29 |
+
{
|
| 30 |
+
"id": "ordicio-translator-v1",
|
| 31 |
+
"languages": ["en", "de", "..."],
|
| 32 |
+
"pivot": "en",
|
| 33 |
+
"engine": {"beam-size": "1", "gemm-precision": "int8shiftAlphaAll", "...": "..."},
|
| 34 |
+
"pairs": {
|
| 35 |
+
"en-de": {"from": "en", "to": "de",
|
| 36 |
+
"model": {"path": "v1/en-de/model.bin.gz", "sha256": "…", "size": 0, "gzip": true, "unpackedSize": 0},
|
| 37 |
+
"srcVocab": {"path": "v1/en-de/vocab.spm.gz", "...": "..."}, "trgVocab": {"...": "..."}}
|
| 38 |
+
},
|
| 39 |
+
"lid": {"model": {"path": "v1/lid.bin.gz", "...": "..."}, "labels": ["en", "...", "zxx"], "buckets": 131072,
|
| 40 |
+
"ngrams": [1, 4], "scale": 0.0, "bias": [], "maxChars": 1000, "hash": "fnv1a32-utf8"}
|
| 41 |
+
}
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
- Paths are relative to the manifest. `srcVocab` and `trgVocab` name the same file when the direction has one
|
| 45 |
+
vocabulary.
|
| 46 |
+
- `engine` holds the bergamot-translator options the models were evaluated with. They run without a lexical
|
| 47 |
+
shortlist (it cost quality on short values).
|
| 48 |
+
- `source/v1/<src>-<tgt>.npz` are the float Marian weights of each model (the form to change them in); the
|
| 49 |
+
apps do not download them.
|
| 50 |
+
|
| 51 |
+
## Language detector
|
| 52 |
+
|
| 53 |
+
A linear model over hashed character n-grams with 17 labels: the 16 languages and `zxx` (names, codes, model
|
| 54 |
+
numbers: nothing to translate). `lid.bin` is int8, `buckets` × `labels` (the weights of one bucket together).
|
| 55 |
+
|
| 56 |
+
1. Text → lower case → every digit `0`-`9` to `0` → runs of whitespace (space, tab, line feed, carriage
|
| 57 |
+
return, form feed, vertical tab, U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F, U+3000)
|
| 58 |
+
to one space → spaces trimmed → the first `maxChars` code points. No Unicode normalization.
|
| 59 |
+
2. Each word `w` (split on spaces) is padded to `" " + w + " "`; every n-gram of 1 to 4 code points of the
|
| 60 |
+
padded word except a single space counts once.
|
| 61 |
+
3. The bucket of an n-gram is the 32-bit FNV-1a hash of its UTF-8 bytes modulo `buckets`. The feature vector
|
| 62 |
+
holds the count per bucket, L2 normalized.
|
| 63 |
+
4. Score of label `i` = `bias[i] + scale × Σ feature × weight[bucket][i]`; softmax gives probabilities.
|
| 64 |
+
|
| 65 |
+
| Accuracy (16 languages) | this detector | Lingua | fastText lid.176 |
|
| 66 |
+
| --- | --- | --- | --- |
|
| 67 |
+
| short values (held out, 6,877) | **0.89** | 0.83 | 0.74 |
|
| 68 |
+
| sentences (WMT / NTREX held out, 2,649) | **1.00** | 0.99 | 0.98 |
|
| 69 |
+
|
| 70 |
+
Short values are often ambiguous (Danish and Norwegian, Spanish and Portuguese, Chinese characters in Japanese).
|
| 71 |
+
On them, 78 % of the predictions reach a probability of 0.9, and 98.7 % of those are right.
|
| 72 |
+
|
| 73 |
+
## Quality
|
| 74 |
+
|
| 75 |
+
chrF++ in the bergamot WebAssembly engine without shortlist, against the shipped Firefox model of the same
|
| 76 |
+
direction. Test: 150 English values of a collecting catalog (examples, list values, member notes) with
|
| 77 |
+
reference translations.
|
| 78 |
+
|
| 79 |
+
| en → | de | fr | es | pl | fi | ja | ko | zh-Hans |
|
| 80 |
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 81 |
+
| Firefox | 69.1 | 67.5 | 67.8 | 60.0 | 60.7 | 32.5 | 42.5 | 25.1 |
|
| 82 |
+
| this | **70.7** | **70.9** | **69.2** | **64.4** | **65.1** | **40.6** | **51.0** | **36.3** |
|
| 83 |
+
|
| 84 |
+
Values that every reference keeps unchanged (names, titles, codes) stay unchanged twice as often (9 to 16 of
|
| 85 |
+
19 instead of 1 to 11). Korean writes most of them in Hangul, as Korean catalogs do (4 of 19 unchanged).
|
| 86 |
+
|
| 87 |
+
| → en (the references as source) | de | fr | es | pl | fi | ja | ko | zh-Hans |
|
| 88 |
+
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
| 89 |
+
| Firefox | 75.0 | 74.4 | 73.6 | 69.3 | 66.0 | 58.2 | 57.9 | 55.0 |
|
| 90 |
+
| this | **76.8** | **74.7** | **75.4** | **69.4** | **67.4** | **59.2** | **61.9** | **59.0** |
|
| 91 |
+
|
| 92 |
+
it, nl, pt, sv, da, nb and zh-Hant have no reference set. On 300 held-out teacher translations they gain 4 to
|
| 93 |
+
32 chrF++ over the Firefox models from English and 1 to 9 into English.
|
| 94 |
+
|
| 95 |
+
## Training
|
| 96 |
+
|
| 97 |
+
1. 10,900 English collecting texts (attribute values of a classification catalog, member notes) translated by
|
| 98 |
+
Gemma 4 31B (4-bit) into the 15 languages.
|
| 99 |
+
2. Each Firefox student (float weights from Mozilla's published training runs) fine-tuned on these pairs (both
|
| 100 |
+
directions; for x → en the teacher's translation is the source) mixed with WMT and NTREX-128 sentences:
|
| 101 |
+
4 epochs, learning rate 2e-5, then one quantization-aware epoch (8 bit), so the int8 model scores like the
|
| 102 |
+
float one.
|
| 103 |
+
3. For Korean, the teacher prompt asks for brands, people, places and characters in their usual Hangul form (as
|
| 104 |
+
Korean catalogs write them), keeps inscriptions, codes and scientific species names unchanged, and leaves noun
|
| 105 |
+
phrases without a sentence ending.
|
| 106 |
+
|
| 107 |
+
## License
|
| 108 |
+
|
| 109 |
+
The models are derived from the Firefox Translations models and are released under the
|
| 110 |
+
[Mozilla Public License 2.0](LICENSE) like them. The teacher, Gemma 4, is licensed under Apache 2.0.
|
| 111 |
+
Fine-tuning data also includes NTREX-128 (CC BY-SA 4.0) and WMT test sets.
|