Translation
Transformers
ONNX
Safetensors
PyTorch
Khmer
blenderbot
text2text-generation
transliteration
romanization
romanized-khmer
khmer
character-level
int8
Eval Results (legacy)
Instructions to use samdei/samdei-type-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use samdei/samdei-type-base with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "translation" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("translation", model="samdei/samdei-type-base")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("samdei/samdei-type-base") model = AutoModelForSeq2SeqLM.from_pretrained("samdei/samdei-type-base", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Model folders in the transformers format (Blenderbot classes, tokenizer, generation config)
Browse files- README.md +37 -4
- channel/config.json +43 -14
- channel/generation_config.json +10 -0
- channel/model.safetensors +2 -2
- channel/tokenizer.json +607 -0
- channel/tokenizer_config.json +15 -0
- config.json +43 -14
- generation_config.json +10 -0
- model.safetensors +2 -2
- tokenizer.json +607 -0
- tokenizer_config.json +15 -0
README.md
CHANGED
|
@@ -2,6 +2,7 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
- km
|
|
|
|
| 5 |
pipeline_tag: text2text-generation
|
| 6 |
tags:
|
| 7 |
- transliteration
|
|
@@ -68,13 +69,15 @@ Paper: *Samdei Type: Separating What Is Written from How It Is Spelled for Roman
|
|
| 68 |
## Files
|
| 69 |
|
| 70 |
```
|
| 71 |
-
config.json configuration and vocabularies (
|
| 72 |
-
|
|
|
|
|
|
|
|
|
|
| 73 |
onnx/encoder_kv.int8.onnx 11.3 MiB input characters -> cross-attention K/V of every decoder layer
|
| 74 |
onnx/decoder_step.int8.onnx 6.3 MiB one decoder step with a self-attention KV cache
|
| 75 |
onnx/vocab.json vocabularies and graph shapes
|
| 76 |
-
channel/
|
| 77 |
-
channel/model.safetensors 27.3 MiB its weights (fp32)
|
| 78 |
lm/char5_nowiki.npz 8.9 MiB hashed character 5-gram LM
|
| 79 |
lm/wordfull_nowiki.json.gz 12.2 MiB word bigram LM over khmercut words (counts, gzipped JSON)
|
| 80 |
rerank.json decoding and rerank settings used for every number below
|
|
@@ -130,6 +133,35 @@ conv = OnnxConverter(f"{d}/onnx")
|
|
| 130 |
hyps = conv.beam("knhom jong tov phsar", k=5) # rerank with rr.rerank(...) as above
|
| 131 |
```
|
| 132 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
**Decoding, as in the paper.**
|
| 134 |
- Beam search with k = 5.
|
| 135 |
- The input is cut into pieces of at most eight words, with a break after any punctuation mark. Each piece is decoded and reranked on its own, and the outputs are joined with a space.
|
|
@@ -153,6 +185,7 @@ hyps = conv.beam("knhom jong tov phsar", k=5) # rerank with rr.rerank(...) as
|
|
| 153 |
|
| 154 |
Int8 quantization does not change CER on Dev-Sentences with beam 5 and no LMs (0.007 in fp32 and in int8). The cached ONNX decoder reproduces full re-decoding exactly.
|
| 155 |
|
|
|
|
| 156 |
## Training
|
| 157 |
|
| 158 |
Base was trained in three stages. Each stage started from the best checkpoint of the stage before and replayed its data.
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
language:
|
| 4 |
- km
|
| 5 |
+
library_name: transformers
|
| 6 |
pipeline_tag: text2text-generation
|
| 7 |
tags:
|
| 8 |
- transliteration
|
|
|
|
| 69 |
## Files
|
| 70 |
|
| 71 |
```
|
| 72 |
+
config.json configuration and vocabularies (transformers format)
|
| 73 |
+
generation_config.json beam search, k = 5
|
| 74 |
+
model.safetensors 71.1 MiB weights (fp32)
|
| 75 |
+
tokenizer.json character tokenizer
|
| 76 |
+
tokenizer_config.json its settings
|
| 77 |
onnx/encoder_kv.int8.onnx 11.3 MiB input characters -> cross-attention K/V of every decoder layer
|
| 78 |
onnx/decoder_step.int8.onnx 6.3 MiB one decoder step with a self-attention KV cache
|
| 79 |
onnx/vocab.json vocabularies and graph shapes
|
| 80 |
+
channel/ 28.4 MiB reverse channel model (Khmer -> romanized), 7.1M parameters, used in reranking; the same five files
|
|
|
|
| 81 |
lm/char5_nowiki.npz 8.9 MiB hashed character 5-gram LM
|
| 82 |
lm/wordfull_nowiki.json.gz 12.2 MiB word bigram LM over khmercut words (counts, gzipped JSON)
|
| 83 |
rerank.json decoding and rerank settings used for every number below
|
|
|
|
| 133 |
hyps = conv.beam("knhom jong tov phsar", k=5) # rerank with rr.rerank(...) as above
|
| 134 |
```
|
| 135 |
|
| 136 |
+
**transformers, without rkhmer** (`pip install transformers torch`; see Model). This is beam search with k = 5 and no reranking: 0.062 CER / 24% exact on Samdei Type Bench, against 0.052 / 30% with the reranker above.
|
| 137 |
+
|
| 138 |
+
```python
|
| 139 |
+
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
| 140 |
+
|
| 141 |
+
repo = "samdei/samdei-type-base"
|
| 142 |
+
tok = AutoTokenizer.from_pretrained(repo)
|
| 143 |
+
model = AutoModelForSeq2SeqLM.from_pretrained(repo)
|
| 144 |
+
|
| 145 |
+
def pieces(text: str, max_words: int = 8) -> list[str]:
|
| 146 |
+
"""At most max_words words per piece, with a break after punctuation, as in the paper."""
|
| 147 |
+
out, cur = [], []
|
| 148 |
+
for w in text.split():
|
| 149 |
+
cur.append(w)
|
| 150 |
+
if len(cur) == max_words or w[-1] in ",.;:?!។":
|
| 151 |
+
out.append(" ".join(cur))
|
| 152 |
+
cur = []
|
| 153 |
+
if cur:
|
| 154 |
+
out.append(" ".join(cur))
|
| 155 |
+
return out or [text]
|
| 156 |
+
|
| 157 |
+
def convert(text: str) -> str:
|
| 158 |
+
batch = tok(pieces(text), padding=True, truncation=True, return_tensors="pt")
|
| 159 |
+
out = model.generate(**batch) # beam search, k = 5 (generation_config.json)
|
| 160 |
+
return " ".join(tok.batch_decode(out, skip_special_tokens=True))
|
| 161 |
+
|
| 162 |
+
print(convert("knhom jong tov phsar")) # ខ្ញុំចង់ទៅផ្សារ
|
| 163 |
+
```
|
| 164 |
+
|
| 165 |
**Decoding, as in the paper.**
|
| 166 |
- Beam search with k = 5.
|
| 167 |
- The input is cut into pieces of at most eight words, with a break after any punctuation mark. Each piece is decoded and reranked on its own, and the outputs are joined with a space.
|
|
|
|
| 185 |
|
| 186 |
Int8 quantization does not change CER on Dev-Sentences with beam 5 and no LMs (0.007 in fp32 and in int8). The cached ONNX decoder reproduces full re-decoding exactly.
|
| 187 |
|
| 188 |
+
**transformers format.** The model folders load with `AutoModelForSeq2SeqLM` and `AutoTokenizer` and need no custom code. They use transformers' Blenderbot classes only because the architecture is the same (a pre-LN encoder-decoder with scaled embeddings, learned positions from 0 and no embedding LayerNorm). The model has nothing else in common with BlenderBot. Blenderbot has one vocabulary for both sides, so the tokenizer stacks the two: ids 0–227 are the output symbols and the input symbols follow. The output layer gives input ids a fixed logit of −10⁴, so they are never generated. For the same reason transformers reports 18.63M (7.43M for the channel model) parameters: the stacked embedding tables and the output layer carry rows that are never used. `rkhmer` reads the same files and maps them back to its own layout.
|
| 189 |
## Training
|
| 190 |
|
| 191 |
Base was trained in three stages. Each stage started from the best checkpoint of the stage before and replayed its data.
|
channel/config.json
CHANGED
|
@@ -1,22 +1,50 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
"
|
|
|
|
| 6 |
"d_model": 256,
|
| 7 |
-
"
|
| 8 |
-
"
|
| 9 |
-
"
|
| 10 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
"dropout": 0.1,
|
| 12 |
-
"
|
| 13 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
"special_tokens": [
|
| 15 |
"<pad>",
|
| 16 |
"<s>",
|
| 17 |
"</s>",
|
| 18 |
"<unk>"
|
| 19 |
],
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
"source": [
|
| 21 |
" ",
|
| 22 |
"!",
|
|
@@ -456,9 +484,10 @@
|
|
| 456 |
"😀",
|
| 457 |
"😛",
|
| 458 |
"🙂"
|
| 459 |
-
]
|
| 460 |
-
|
| 461 |
-
|
| 462 |
-
|
|
|
|
| 463 |
}
|
| 464 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"BlenderbotForConditionalGeneration"
|
| 4 |
+
],
|
| 5 |
+
"model_type": "blenderbot",
|
| 6 |
+
"vocab_size": 444,
|
| 7 |
"d_model": 256,
|
| 8 |
+
"encoder_layers": 6,
|
| 9 |
+
"decoder_layers": 2,
|
| 10 |
+
"encoder_attention_heads": 4,
|
| 11 |
+
"decoder_attention_heads": 4,
|
| 12 |
+
"encoder_ffn_dim": 1024,
|
| 13 |
+
"decoder_ffn_dim": 1024,
|
| 14 |
+
"activation_function": "gelu",
|
| 15 |
+
"max_position_embeddings": 256,
|
| 16 |
+
"scale_embedding": true,
|
| 17 |
"dropout": 0.1,
|
| 18 |
+
"attention_dropout": 0.1,
|
| 19 |
+
"activation_dropout": 0.1,
|
| 20 |
+
"encoder_layerdrop": 0.0,
|
| 21 |
+
"decoder_layerdrop": 0.0,
|
| 22 |
+
"init_std": 0.02,
|
| 23 |
+
"pad_token_id": 0,
|
| 24 |
+
"bos_token_id": 1,
|
| 25 |
+
"eos_token_id": 2,
|
| 26 |
+
"decoder_start_token_id": 1,
|
| 27 |
+
"forced_eos_token_id": null,
|
| 28 |
+
"encoder_no_repeat_ngram_size": 0,
|
| 29 |
+
"is_encoder_decoder": true,
|
| 30 |
+
"tie_word_embeddings": false,
|
| 31 |
+
"use_cache": true,
|
| 32 |
+
"torch_dtype": "float32",
|
| 33 |
+
"samdei_type": {
|
| 34 |
+
"format_version": 2,
|
| 35 |
"special_tokens": [
|
| 36 |
"<pad>",
|
| 37 |
"<s>",
|
| 38 |
"</s>",
|
| 39 |
"<unk>"
|
| 40 |
],
|
| 41 |
+
"source_special_tokens": [
|
| 42 |
+
"<src_pad>",
|
| 43 |
+
"<src_s>",
|
| 44 |
+
"</src>",
|
| 45 |
+
"<src_unk>"
|
| 46 |
+
],
|
| 47 |
+
"source_offset": 216,
|
| 48 |
"source": [
|
| 49 |
" ",
|
| 50 |
"!",
|
|
|
|
| 484 |
"😀",
|
| 485 |
"😛",
|
| 486 |
"🙂"
|
| 487 |
+
],
|
| 488 |
+
"max_input_chars": 249,
|
| 489 |
+
"training": {
|
| 490 |
+
"step": 20000
|
| 491 |
+
}
|
| 492 |
}
|
| 493 |
}
|
channel/generation_config.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"decoder_start_token_id": 1,
|
| 3 |
+
"bos_token_id": 1,
|
| 4 |
+
"eos_token_id": 2,
|
| 5 |
+
"pad_token_id": 0,
|
| 6 |
+
"forced_eos_token_id": null,
|
| 7 |
+
"num_beams": 5,
|
| 8 |
+
"max_length": 201,
|
| 9 |
+
"length_penalty": 1.0
|
| 10 |
+
}
|
channel/model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b222590113a348452d4652b33ed83f08e933d7c3a25cecce8fcb6346e26ca797
|
| 3 |
+
size 29748272
|
channel/tokenizer.json
ADDED
|
@@ -0,0 +1,607 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": "1.0",
|
| 3 |
+
"truncation": null,
|
| 4 |
+
"padding": null,
|
| 5 |
+
"added_tokens": [
|
| 6 |
+
{
|
| 7 |
+
"id": 0,
|
| 8 |
+
"content": "<pad>",
|
| 9 |
+
"single_word": false,
|
| 10 |
+
"lstrip": false,
|
| 11 |
+
"rstrip": false,
|
| 12 |
+
"normalized": false,
|
| 13 |
+
"special": true
|
| 14 |
+
},
|
| 15 |
+
{
|
| 16 |
+
"id": 1,
|
| 17 |
+
"content": "<s>",
|
| 18 |
+
"single_word": false,
|
| 19 |
+
"lstrip": false,
|
| 20 |
+
"rstrip": false,
|
| 21 |
+
"normalized": false,
|
| 22 |
+
"special": true
|
| 23 |
+
},
|
| 24 |
+
{
|
| 25 |
+
"id": 2,
|
| 26 |
+
"content": "</s>",
|
| 27 |
+
"single_word": false,
|
| 28 |
+
"lstrip": false,
|
| 29 |
+
"rstrip": false,
|
| 30 |
+
"normalized": false,
|
| 31 |
+
"special": true
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"id": 3,
|
| 35 |
+
"content": "<unk>",
|
| 36 |
+
"single_word": false,
|
| 37 |
+
"lstrip": false,
|
| 38 |
+
"rstrip": false,
|
| 39 |
+
"normalized": false,
|
| 40 |
+
"special": true
|
| 41 |
+
},
|
| 42 |
+
{
|
| 43 |
+
"id": 216,
|
| 44 |
+
"content": "<src_pad>",
|
| 45 |
+
"single_word": false,
|
| 46 |
+
"lstrip": false,
|
| 47 |
+
"rstrip": false,
|
| 48 |
+
"normalized": false,
|
| 49 |
+
"special": true
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"id": 217,
|
| 53 |
+
"content": "<src_s>",
|
| 54 |
+
"single_word": false,
|
| 55 |
+
"lstrip": false,
|
| 56 |
+
"rstrip": false,
|
| 57 |
+
"normalized": false,
|
| 58 |
+
"special": true
|
| 59 |
+
},
|
| 60 |
+
{
|
| 61 |
+
"id": 218,
|
| 62 |
+
"content": "</src>",
|
| 63 |
+
"single_word": false,
|
| 64 |
+
"lstrip": false,
|
| 65 |
+
"rstrip": false,
|
| 66 |
+
"normalized": false,
|
| 67 |
+
"special": true
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"id": 219,
|
| 71 |
+
"content": "<src_unk>",
|
| 72 |
+
"single_word": false,
|
| 73 |
+
"lstrip": false,
|
| 74 |
+
"rstrip": false,
|
| 75 |
+
"normalized": false,
|
| 76 |
+
"special": true
|
| 77 |
+
}
|
| 78 |
+
],
|
| 79 |
+
"normalizer": null,
|
| 80 |
+
"pre_tokenizer": {
|
| 81 |
+
"type": "Split",
|
| 82 |
+
"pattern": {
|
| 83 |
+
"Regex": "[\\s\\S]"
|
| 84 |
+
},
|
| 85 |
+
"behavior": "Isolated",
|
| 86 |
+
"invert": false
|
| 87 |
+
},
|
| 88 |
+
"post_processor": {
|
| 89 |
+
"type": "TemplateProcessing",
|
| 90 |
+
"single": [
|
| 91 |
+
{
|
| 92 |
+
"Sequence": {
|
| 93 |
+
"id": "A",
|
| 94 |
+
"type_id": 0
|
| 95 |
+
}
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"SpecialToken": {
|
| 99 |
+
"id": "</src>",
|
| 100 |
+
"type_id": 0
|
| 101 |
+
}
|
| 102 |
+
}
|
| 103 |
+
],
|
| 104 |
+
"pair": [
|
| 105 |
+
{
|
| 106 |
+
"Sequence": {
|
| 107 |
+
"id": "A",
|
| 108 |
+
"type_id": 0
|
| 109 |
+
}
|
| 110 |
+
},
|
| 111 |
+
{
|
| 112 |
+
"SpecialToken": {
|
| 113 |
+
"id": "</src>",
|
| 114 |
+
"type_id": 0
|
| 115 |
+
}
|
| 116 |
+
},
|
| 117 |
+
{
|
| 118 |
+
"Sequence": {
|
| 119 |
+
"id": "B",
|
| 120 |
+
"type_id": 1
|
| 121 |
+
}
|
| 122 |
+
},
|
| 123 |
+
{
|
| 124 |
+
"SpecialToken": {
|
| 125 |
+
"id": "</src>",
|
| 126 |
+
"type_id": 1
|
| 127 |
+
}
|
| 128 |
+
}
|
| 129 |
+
],
|
| 130 |
+
"special_tokens": {
|
| 131 |
+
"</src>": {
|
| 132 |
+
"id": "</src>",
|
| 133 |
+
"ids": [
|
| 134 |
+
218
|
| 135 |
+
],
|
| 136 |
+
"tokens": [
|
| 137 |
+
"</src>"
|
| 138 |
+
]
|
| 139 |
+
}
|
| 140 |
+
}
|
| 141 |
+
},
|
| 142 |
+
"decoder": {
|
| 143 |
+
"type": "Sequence",
|
| 144 |
+
"decoders": [
|
| 145 |
+
{
|
| 146 |
+
"type": "Replace",
|
| 147 |
+
"pattern": {
|
| 148 |
+
"String": ""
|
| 149 |
+
},
|
| 150 |
+
"content": ""
|
| 151 |
+
},
|
| 152 |
+
{
|
| 153 |
+
"type": "Fuse"
|
| 154 |
+
}
|
| 155 |
+
]
|
| 156 |
+
},
|
| 157 |
+
"model": {
|
| 158 |
+
"type": "WordLevel",
|
| 159 |
+
"vocab": {
|
| 160 |
+
"<pad>": 0,
|
| 161 |
+
"<s>": 1,
|
| 162 |
+
"</s>": 2,
|
| 163 |
+
"<unk>": 3,
|
| 164 |
+
" ": 4,
|
| 165 |
+
"!": 5,
|
| 166 |
+
"\"": 6,
|
| 167 |
+
"#": 7,
|
| 168 |
+
"$": 8,
|
| 169 |
+
"%": 9,
|
| 170 |
+
"&": 10,
|
| 171 |
+
"'": 11,
|
| 172 |
+
"(": 12,
|
| 173 |
+
")": 13,
|
| 174 |
+
"*": 14,
|
| 175 |
+
"+": 15,
|
| 176 |
+
",": 16,
|
| 177 |
+
"-": 17,
|
| 178 |
+
".": 18,
|
| 179 |
+
"/": 19,
|
| 180 |
+
"0": 20,
|
| 181 |
+
"1": 21,
|
| 182 |
+
"2": 22,
|
| 183 |
+
"3": 23,
|
| 184 |
+
"4": 24,
|
| 185 |
+
"5": 25,
|
| 186 |
+
"6": 26,
|
| 187 |
+
"7": 27,
|
| 188 |
+
"8": 28,
|
| 189 |
+
"9": 29,
|
| 190 |
+
":": 30,
|
| 191 |
+
";": 31,
|
| 192 |
+
"<": 32,
|
| 193 |
+
"=": 33,
|
| 194 |
+
">": 34,
|
| 195 |
+
"?": 35,
|
| 196 |
+
"@": 36,
|
| 197 |
+
"A": 37,
|
| 198 |
+
"B": 38,
|
| 199 |
+
"C": 39,
|
| 200 |
+
"D": 40,
|
| 201 |
+
"E": 41,
|
| 202 |
+
"F": 42,
|
| 203 |
+
"G": 43,
|
| 204 |
+
"H": 44,
|
| 205 |
+
"I": 45,
|
| 206 |
+
"J": 46,
|
| 207 |
+
"K": 47,
|
| 208 |
+
"L": 48,
|
| 209 |
+
"M": 49,
|
| 210 |
+
"N": 50,
|
| 211 |
+
"O": 51,
|
| 212 |
+
"P": 52,
|
| 213 |
+
"Q": 53,
|
| 214 |
+
"R": 54,
|
| 215 |
+
"S": 55,
|
| 216 |
+
"T": 56,
|
| 217 |
+
"U": 57,
|
| 218 |
+
"V": 58,
|
| 219 |
+
"W": 59,
|
| 220 |
+
"X": 60,
|
| 221 |
+
"Y": 61,
|
| 222 |
+
"Z": 62,
|
| 223 |
+
"[": 63,
|
| 224 |
+
"]": 64,
|
| 225 |
+
"^": 65,
|
| 226 |
+
"_": 66,
|
| 227 |
+
"a": 67,
|
| 228 |
+
"b": 68,
|
| 229 |
+
"c": 69,
|
| 230 |
+
"d": 70,
|
| 231 |
+
"e": 71,
|
| 232 |
+
"f": 72,
|
| 233 |
+
"g": 73,
|
| 234 |
+
"h": 74,
|
| 235 |
+
"i": 75,
|
| 236 |
+
"j": 76,
|
| 237 |
+
"k": 77,
|
| 238 |
+
"l": 78,
|
| 239 |
+
"m": 79,
|
| 240 |
+
"n": 80,
|
| 241 |
+
"o": 81,
|
| 242 |
+
"p": 82,
|
| 243 |
+
"q": 83,
|
| 244 |
+
"r": 84,
|
| 245 |
+
"s": 85,
|
| 246 |
+
"t": 86,
|
| 247 |
+
"u": 87,
|
| 248 |
+
"v": 88,
|
| 249 |
+
"w": 89,
|
| 250 |
+
"x": 90,
|
| 251 |
+
"y": 91,
|
| 252 |
+
"z": 92,
|
| 253 |
+
"|": 93,
|
| 254 |
+
"~": 94,
|
| 255 |
+
"¡": 95,
|
| 256 |
+
"«": 96,
|
| 257 |
+
"®": 97,
|
| 258 |
+
"°": 98,
|
| 259 |
+
"·": 99,
|
| 260 |
+
"»": 100,
|
| 261 |
+
"×": 101,
|
| 262 |
+
"à": 102,
|
| 263 |
+
"á": 103,
|
| 264 |
+
"ç": 104,
|
| 265 |
+
"è": 105,
|
| 266 |
+
"é": 106,
|
| 267 |
+
"í": 107,
|
| 268 |
+
"ǔ": 108,
|
| 269 |
+
"ɓ": 109,
|
| 270 |
+
"ə": 110,
|
| 271 |
+
"ก": 111,
|
| 272 |
+
"ท": 112,
|
| 273 |
+
"น": 113,
|
| 274 |
+
"ภ": 114,
|
| 275 |
+
"ย": 115,
|
| 276 |
+
"ร": 116,
|
| 277 |
+
"ษ": 117,
|
| 278 |
+
"า": 118,
|
| 279 |
+
"ิ": 119,
|
| 280 |
+
"เ": 120,
|
| 281 |
+
"ไ": 121,
|
| 282 |
+
"๒": 122,
|
| 283 |
+
"๕": 123,
|
| 284 |
+
"ລ": 124,
|
| 285 |
+
"ວ": 125,
|
| 286 |
+
"ສ": 126,
|
| 287 |
+
"າ": 127,
|
| 288 |
+
"ក": 128,
|
| 289 |
+
"ខ": 129,
|
| 290 |
+
"គ": 130,
|
| 291 |
+
"ឃ": 131,
|
| 292 |
+
"ង": 132,
|
| 293 |
+
"ច": 133,
|
| 294 |
+
"ឆ": 134,
|
| 295 |
+
"ជ": 135,
|
| 296 |
+
"ឈ": 136,
|
| 297 |
+
"ញ": 137,
|
| 298 |
+
"ដ": 138,
|
| 299 |
+
"ឋ": 139,
|
| 300 |
+
"ឌ": 140,
|
| 301 |
+
"ណ": 141,
|
| 302 |
+
"ត": 142,
|
| 303 |
+
"ថ": 143,
|
| 304 |
+
"ទ": 144,
|
| 305 |
+
"ធ": 145,
|
| 306 |
+
"ន": 146,
|
| 307 |
+
"ប": 147,
|
| 308 |
+
"ផ": 148,
|
| 309 |
+
"ព": 149,
|
| 310 |
+
"ភ": 150,
|
| 311 |
+
"ម": 151,
|
| 312 |
+
"��យ": 152,
|
| 313 |
+
"រ": 153,
|
| 314 |
+
"ល": 154,
|
| 315 |
+
"វ": 155,
|
| 316 |
+
"ស": 156,
|
| 317 |
+
"ហ": 157,
|
| 318 |
+
"ឡ": 158,
|
| 319 |
+
"អ": 159,
|
| 320 |
+
"ឧ": 160,
|
| 321 |
+
"ឫ": 161,
|
| 322 |
+
"ឬ": 162,
|
| 323 |
+
"ឯ": 163,
|
| 324 |
+
"ឲ": 164,
|
| 325 |
+
"ា": 165,
|
| 326 |
+
"ិ": 166,
|
| 327 |
+
"ី": 167,
|
| 328 |
+
"ឹ": 168,
|
| 329 |
+
"ឺ": 169,
|
| 330 |
+
"ុ": 170,
|
| 331 |
+
"ូ": 171,
|
| 332 |
+
"ួ": 172,
|
| 333 |
+
"ើ": 173,
|
| 334 |
+
"ឿ": 174,
|
| 335 |
+
"ៀ": 175,
|
| 336 |
+
"េ": 176,
|
| 337 |
+
"ែ": 177,
|
| 338 |
+
"ៃ": 178,
|
| 339 |
+
"ោ": 179,
|
| 340 |
+
"ៅ": 180,
|
| 341 |
+
"ំ": 181,
|
| 342 |
+
"ះ": 182,
|
| 343 |
+
"ៈ": 183,
|
| 344 |
+
"៉": 184,
|
| 345 |
+
"៊": 185,
|
| 346 |
+
"់": 186,
|
| 347 |
+
"៌": 187,
|
| 348 |
+
"៍": 188,
|
| 349 |
+
"៏": 189,
|
| 350 |
+
"័": 190,
|
| 351 |
+
"្": 191,
|
| 352 |
+
"។": 192,
|
| 353 |
+
"៕": 193,
|
| 354 |
+
"៖": 194,
|
| 355 |
+
"ៗ": 195,
|
| 356 |
+
"៘": 196,
|
| 357 |
+
"៛": 197,
|
| 358 |
+
"‑": 198,
|
| 359 |
+
"–": 199,
|
| 360 |
+
"—": 200,
|
| 361 |
+
"‘": 201,
|
| 362 |
+
"’": 202,
|
| 363 |
+
"“": 203,
|
| 364 |
+
"”": 204,
|
| 365 |
+
"•": 205,
|
| 366 |
+
"…": 206,
|
| 367 |
+
"‹": 207,
|
| 368 |
+
"›": 208,
|
| 369 |
+
"→": 209,
|
| 370 |
+
"−": 210,
|
| 371 |
+
"●": 211,
|
| 372 |
+
"�": 212,
|
| 373 |
+
"😀": 213,
|
| 374 |
+
"😛": 214,
|
| 375 |
+
"🙂": 215,
|
| 376 |
+
"<src_pad>": 216,
|
| 377 |
+
"<src_s>": 217,
|
| 378 |
+
"</src>": 218,
|
| 379 |
+
"<src_unk>": 219,
|
| 380 |
+
" ": 220,
|
| 381 |
+
"!": 221,
|
| 382 |
+
"\"": 222,
|
| 383 |
+
"#": 223,
|
| 384 |
+
"$": 224,
|
| 385 |
+
"%": 225,
|
| 386 |
+
"&": 226,
|
| 387 |
+
"'": 227,
|
| 388 |
+
"(": 228,
|
| 389 |
+
")": 229,
|
| 390 |
+
"*": 230,
|
| 391 |
+
"+": 231,
|
| 392 |
+
",": 232,
|
| 393 |
+
"-": 233,
|
| 394 |
+
".": 234,
|
| 395 |
+
"/": 235,
|
| 396 |
+
"0": 236,
|
| 397 |
+
"1": 237,
|
| 398 |
+
"2": 238,
|
| 399 |
+
"3": 239,
|
| 400 |
+
"4": 240,
|
| 401 |
+
"5": 241,
|
| 402 |
+
"6": 242,
|
| 403 |
+
"7": 243,
|
| 404 |
+
"8": 244,
|
| 405 |
+
"9": 245,
|
| 406 |
+
":": 246,
|
| 407 |
+
";": 247,
|
| 408 |
+
"<": 248,
|
| 409 |
+
"=": 249,
|
| 410 |
+
">": 250,
|
| 411 |
+
"?": 251,
|
| 412 |
+
"@": 252,
|
| 413 |
+
"A": 253,
|
| 414 |
+
"B": 254,
|
| 415 |
+
"C": 255,
|
| 416 |
+
"D": 256,
|
| 417 |
+
"E": 257,
|
| 418 |
+
"F": 258,
|
| 419 |
+
"G": 259,
|
| 420 |
+
"H": 260,
|
| 421 |
+
"I": 261,
|
| 422 |
+
"J": 262,
|
| 423 |
+
"K": 263,
|
| 424 |
+
"L": 264,
|
| 425 |
+
"M": 265,
|
| 426 |
+
"N": 266,
|
| 427 |
+
"O": 267,
|
| 428 |
+
"P": 268,
|
| 429 |
+
"Q": 269,
|
| 430 |
+
"R": 270,
|
| 431 |
+
"S": 271,
|
| 432 |
+
"T": 272,
|
| 433 |
+
"U": 273,
|
| 434 |
+
"V": 274,
|
| 435 |
+
"W": 275,
|
| 436 |
+
"X": 276,
|
| 437 |
+
"Y": 277,
|
| 438 |
+
"Z": 278,
|
| 439 |
+
"[": 279,
|
| 440 |
+
"]": 280,
|
| 441 |
+
"^": 281,
|
| 442 |
+
"_": 282,
|
| 443 |
+
"a": 283,
|
| 444 |
+
"b": 284,
|
| 445 |
+
"c": 285,
|
| 446 |
+
"d": 286,
|
| 447 |
+
"e": 287,
|
| 448 |
+
"f": 288,
|
| 449 |
+
"g": 289,
|
| 450 |
+
"h": 290,
|
| 451 |
+
"i": 291,
|
| 452 |
+
"j": 292,
|
| 453 |
+
"k": 293,
|
| 454 |
+
"l": 294,
|
| 455 |
+
"m": 295,
|
| 456 |
+
"n": 296,
|
| 457 |
+
"o": 297,
|
| 458 |
+
"p": 298,
|
| 459 |
+
"q": 299,
|
| 460 |
+
"r": 300,
|
| 461 |
+
"s": 301,
|
| 462 |
+
"t": 302,
|
| 463 |
+
"u": 303,
|
| 464 |
+
"v": 304,
|
| 465 |
+
"w": 305,
|
| 466 |
+
"x": 306,
|
| 467 |
+
"y": 307,
|
| 468 |
+
"z": 308,
|
| 469 |
+
"|": 309,
|
| 470 |
+
"~": 310,
|
| 471 |
+
"¡": 311,
|
| 472 |
+
"«": 312,
|
| 473 |
+
"®": 313,
|
| 474 |
+
"°": 314,
|
| 475 |
+
"·": 315,
|
| 476 |
+
"»": 316,
|
| 477 |
+
"×": 317,
|
| 478 |
+
"à": 318,
|
| 479 |
+
"á": 319,
|
| 480 |
+
"ç": 320,
|
| 481 |
+
"è": 321,
|
| 482 |
+
"é": 322,
|
| 483 |
+
"í": 323,
|
| 484 |
+
"ǔ": 324,
|
| 485 |
+
"ก": 325,
|
| 486 |
+
"ท": 326,
|
| 487 |
+
"น": 327,
|
| 488 |
+
"ภ": 328,
|
| 489 |
+
"ย": 329,
|
| 490 |
+
"ร": 330,
|
| 491 |
+
"ษ": 331,
|
| 492 |
+
"า": 332,
|
| 493 |
+
"ิ": 333,
|
| 494 |
+
"เ": 334,
|
| 495 |
+
"ไ": 335,
|
| 496 |
+
"๒": 336,
|
| 497 |
+
"๕": 337,
|
| 498 |
+
"ລ": 338,
|
| 499 |
+
"ວ": 339,
|
| 500 |
+
"ສ": 340,
|
| 501 |
+
"າ": 341,
|
| 502 |
+
"ក": 342,
|
| 503 |
+
"ខ": 343,
|
| 504 |
+
"គ": 344,
|
| 505 |
+
"ឃ": 345,
|
| 506 |
+
"ង": 346,
|
| 507 |
+
"ច": 347,
|
| 508 |
+
"ឆ": 348,
|
| 509 |
+
"ជ": 349,
|
| 510 |
+
"ឈ": 350,
|
| 511 |
+
"ញ": 351,
|
| 512 |
+
"ដ": 352,
|
| 513 |
+
"ឋ": 353,
|
| 514 |
+
"ឌ": 354,
|
| 515 |
+
"ឍ": 355,
|
| 516 |
+
"ណ": 356,
|
| 517 |
+
"ត": 357,
|
| 518 |
+
"ថ": 358,
|
| 519 |
+
"ទ": 359,
|
| 520 |
+
"ធ": 360,
|
| 521 |
+
"ន": 361,
|
| 522 |
+
"ប": 362,
|
| 523 |
+
"ផ": 363,
|
| 524 |
+
"ព": 364,
|
| 525 |
+
"ភ": 365,
|
| 526 |
+
"ម": 366,
|
| 527 |
+
"យ": 367,
|
| 528 |
+
"រ": 368,
|
| 529 |
+
"ល": 369,
|
| 530 |
+
"វ": 370,
|
| 531 |
+
"ឝ": 371,
|
| 532 |
+
"ឞ": 372,
|
| 533 |
+
"ស": 373,
|
| 534 |
+
"ហ": 374,
|
| 535 |
+
"ឡ": 375,
|
| 536 |
+
"អ": 376,
|
| 537 |
+
"ឥ": 377,
|
| 538 |
+
"ឦ": 378,
|
| 539 |
+
"ឧ": 379,
|
| 540 |
+
"ឩ": 380,
|
| 541 |
+
"ឪ": 381,
|
| 542 |
+
"ឫ": 382,
|
| 543 |
+
"ឬ": 383,
|
| 544 |
+
"ឭ": 384,
|
| 545 |
+
"ឮ": 385,
|
| 546 |
+
"ឯ": 386,
|
| 547 |
+
"ឰ": 387,
|
| 548 |
+
"ឱ": 388,
|
| 549 |
+
"ឲ": 389,
|
| 550 |
+
"ឳ": 390,
|
| 551 |
+
"ា": 391,
|
| 552 |
+
"ិ": 392,
|
| 553 |
+
"ី": 393,
|
| 554 |
+
"ឹ": 394,
|
| 555 |
+
"ឺ": 395,
|
| 556 |
+
"ុ": 396,
|
| 557 |
+
"ូ": 397,
|
| 558 |
+
"ួ": 398,
|
| 559 |
+
"ើ": 399,
|
| 560 |
+
"ឿ": 400,
|
| 561 |
+
"ៀ": 401,
|
| 562 |
+
"េ": 402,
|
| 563 |
+
"ែ": 403,
|
| 564 |
+
"ៃ": 404,
|
| 565 |
+
"ោ": 405,
|
| 566 |
+
"ៅ": 406,
|
| 567 |
+
"ំ": 407,
|
| 568 |
+
"ះ": 408,
|
| 569 |
+
"ៈ": 409,
|
| 570 |
+
"៉": 410,
|
| 571 |
+
"៊": 411,
|
| 572 |
+
"់": 412,
|
| 573 |
+
"៌": 413,
|
| 574 |
+
"៍": 414,
|
| 575 |
+
"៎": 415,
|
| 576 |
+
"៏": 416,
|
| 577 |
+
"័": 417,
|
| 578 |
+
"៑": 418,
|
| 579 |
+
"្": 419,
|
| 580 |
+
"។": 420,
|
| 581 |
+
"៕": 421,
|
| 582 |
+
"៖": 422,
|
| 583 |
+
"ៗ": 423,
|
| 584 |
+
"៘": 424,
|
| 585 |
+
"៛": 425,
|
| 586 |
+
"‑": 426,
|
| 587 |
+
"–": 427,
|
| 588 |
+
"—": 428,
|
| 589 |
+
"‘": 429,
|
| 590 |
+
"’": 430,
|
| 591 |
+
"“": 431,
|
| 592 |
+
"”": 432,
|
| 593 |
+
"•": 433,
|
| 594 |
+
"…": 434,
|
| 595 |
+
"‹": 435,
|
| 596 |
+
"›": 436,
|
| 597 |
+
"→": 437,
|
| 598 |
+
"−": 438,
|
| 599 |
+
"●": 439,
|
| 600 |
+
"�": 440,
|
| 601 |
+
"😀": 441,
|
| 602 |
+
"😛": 442,
|
| 603 |
+
"🙂": 443
|
| 604 |
+
},
|
| 605 |
+
"unk_token": "<src_unk>"
|
| 606 |
+
}
|
| 607 |
+
}
|
channel/tokenizer_config.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 3 |
+
"model_max_length": 250,
|
| 4 |
+
"padding_side": "right",
|
| 5 |
+
"truncation_side": "right",
|
| 6 |
+
"clean_up_tokenization_spaces": false,
|
| 7 |
+
"bos_token": "<s>",
|
| 8 |
+
"eos_token": "</s>",
|
| 9 |
+
"pad_token": "<pad>",
|
| 10 |
+
"unk_token": "<src_unk>",
|
| 11 |
+
"model_input_names": [
|
| 12 |
+
"input_ids",
|
| 13 |
+
"attention_mask"
|
| 14 |
+
]
|
| 15 |
+
}
|
config.json
CHANGED
|
@@ -1,22 +1,50 @@
|
|
| 1 |
{
|
| 2 |
-
"
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
"
|
|
|
|
| 6 |
"d_model": 384,
|
| 7 |
-
"
|
| 8 |
-
"
|
| 9 |
-
"
|
| 10 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
"dropout": 0.1,
|
| 12 |
-
"
|
| 13 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
"special_tokens": [
|
| 15 |
"<pad>",
|
| 16 |
"<s>",
|
| 17 |
"</s>",
|
| 18 |
"<unk>"
|
| 19 |
],
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
"source": [
|
| 21 |
" ",
|
| 22 |
"!",
|
|
@@ -456,9 +484,10 @@
|
|
| 456 |
"😀",
|
| 457 |
"😛",
|
| 458 |
"🙂"
|
| 459 |
-
]
|
| 460 |
-
|
| 461 |
-
|
| 462 |
-
|
|
|
|
| 463 |
}
|
| 464 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"BlenderbotForConditionalGeneration"
|
| 4 |
+
],
|
| 5 |
+
"model_type": "blenderbot",
|
| 6 |
+
"vocab_size": 444,
|
| 7 |
"d_model": 384,
|
| 8 |
+
"encoder_layers": 6,
|
| 9 |
+
"decoder_layers": 3,
|
| 10 |
+
"encoder_attention_heads": 6,
|
| 11 |
+
"decoder_attention_heads": 6,
|
| 12 |
+
"encoder_ffn_dim": 1536,
|
| 13 |
+
"decoder_ffn_dim": 1536,
|
| 14 |
+
"activation_function": "gelu",
|
| 15 |
+
"max_position_embeddings": 256,
|
| 16 |
+
"scale_embedding": true,
|
| 17 |
"dropout": 0.1,
|
| 18 |
+
"attention_dropout": 0.1,
|
| 19 |
+
"activation_dropout": 0.1,
|
| 20 |
+
"encoder_layerdrop": 0.0,
|
| 21 |
+
"decoder_layerdrop": 0.0,
|
| 22 |
+
"init_std": 0.02,
|
| 23 |
+
"pad_token_id": 0,
|
| 24 |
+
"bos_token_id": 1,
|
| 25 |
+
"eos_token_id": 2,
|
| 26 |
+
"decoder_start_token_id": 1,
|
| 27 |
+
"forced_eos_token_id": null,
|
| 28 |
+
"encoder_no_repeat_ngram_size": 0,
|
| 29 |
+
"is_encoder_decoder": true,
|
| 30 |
+
"tie_word_embeddings": false,
|
| 31 |
+
"use_cache": true,
|
| 32 |
+
"torch_dtype": "float32",
|
| 33 |
+
"samdei_type": {
|
| 34 |
+
"format_version": 2,
|
| 35 |
"special_tokens": [
|
| 36 |
"<pad>",
|
| 37 |
"<s>",
|
| 38 |
"</s>",
|
| 39 |
"<unk>"
|
| 40 |
],
|
| 41 |
+
"source_special_tokens": [
|
| 42 |
+
"<src_pad>",
|
| 43 |
+
"<src_s>",
|
| 44 |
+
"</src>",
|
| 45 |
+
"<src_unk>"
|
| 46 |
+
],
|
| 47 |
+
"source_offset": 228,
|
| 48 |
"source": [
|
| 49 |
" ",
|
| 50 |
"!",
|
|
|
|
| 484 |
"😀",
|
| 485 |
"😛",
|
| 486 |
"🙂"
|
| 487 |
+
],
|
| 488 |
+
"max_input_chars": 249,
|
| 489 |
+
"training": {
|
| 490 |
+
"step": 3000
|
| 491 |
+
}
|
| 492 |
}
|
| 493 |
}
|
generation_config.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"decoder_start_token_id": 1,
|
| 3 |
+
"bos_token_id": 1,
|
| 4 |
+
"eos_token_id": 2,
|
| 5 |
+
"pad_token_id": 0,
|
| 6 |
+
"forced_eos_token_id": null,
|
| 7 |
+
"num_beams": 5,
|
| 8 |
+
"max_length": 201,
|
| 9 |
+
"length_penalty": 1.0
|
| 10 |
+
}
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:00f73802fe247d91a2337cffc8fdf5e8aae33416ab37bb775eaa7a6e8a6bf55a
|
| 3 |
+
size 74529304
|
tokenizer.json
ADDED
|
@@ -0,0 +1,607 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"version": "1.0",
|
| 3 |
+
"truncation": null,
|
| 4 |
+
"padding": null,
|
| 5 |
+
"added_tokens": [
|
| 6 |
+
{
|
| 7 |
+
"id": 0,
|
| 8 |
+
"content": "<pad>",
|
| 9 |
+
"single_word": false,
|
| 10 |
+
"lstrip": false,
|
| 11 |
+
"rstrip": false,
|
| 12 |
+
"normalized": false,
|
| 13 |
+
"special": true
|
| 14 |
+
},
|
| 15 |
+
{
|
| 16 |
+
"id": 1,
|
| 17 |
+
"content": "<s>",
|
| 18 |
+
"single_word": false,
|
| 19 |
+
"lstrip": false,
|
| 20 |
+
"rstrip": false,
|
| 21 |
+
"normalized": false,
|
| 22 |
+
"special": true
|
| 23 |
+
},
|
| 24 |
+
{
|
| 25 |
+
"id": 2,
|
| 26 |
+
"content": "</s>",
|
| 27 |
+
"single_word": false,
|
| 28 |
+
"lstrip": false,
|
| 29 |
+
"rstrip": false,
|
| 30 |
+
"normalized": false,
|
| 31 |
+
"special": true
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"id": 3,
|
| 35 |
+
"content": "<unk>",
|
| 36 |
+
"single_word": false,
|
| 37 |
+
"lstrip": false,
|
| 38 |
+
"rstrip": false,
|
| 39 |
+
"normalized": false,
|
| 40 |
+
"special": true
|
| 41 |
+
},
|
| 42 |
+
{
|
| 43 |
+
"id": 228,
|
| 44 |
+
"content": "<src_pad>",
|
| 45 |
+
"single_word": false,
|
| 46 |
+
"lstrip": false,
|
| 47 |
+
"rstrip": false,
|
| 48 |
+
"normalized": false,
|
| 49 |
+
"special": true
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"id": 229,
|
| 53 |
+
"content": "<src_s>",
|
| 54 |
+
"single_word": false,
|
| 55 |
+
"lstrip": false,
|
| 56 |
+
"rstrip": false,
|
| 57 |
+
"normalized": false,
|
| 58 |
+
"special": true
|
| 59 |
+
},
|
| 60 |
+
{
|
| 61 |
+
"id": 230,
|
| 62 |
+
"content": "</src>",
|
| 63 |
+
"single_word": false,
|
| 64 |
+
"lstrip": false,
|
| 65 |
+
"rstrip": false,
|
| 66 |
+
"normalized": false,
|
| 67 |
+
"special": true
|
| 68 |
+
},
|
| 69 |
+
{
|
| 70 |
+
"id": 231,
|
| 71 |
+
"content": "<src_unk>",
|
| 72 |
+
"single_word": false,
|
| 73 |
+
"lstrip": false,
|
| 74 |
+
"rstrip": false,
|
| 75 |
+
"normalized": false,
|
| 76 |
+
"special": true
|
| 77 |
+
}
|
| 78 |
+
],
|
| 79 |
+
"normalizer": null,
|
| 80 |
+
"pre_tokenizer": {
|
| 81 |
+
"type": "Split",
|
| 82 |
+
"pattern": {
|
| 83 |
+
"Regex": "[\\s\\S]"
|
| 84 |
+
},
|
| 85 |
+
"behavior": "Isolated",
|
| 86 |
+
"invert": false
|
| 87 |
+
},
|
| 88 |
+
"post_processor": {
|
| 89 |
+
"type": "TemplateProcessing",
|
| 90 |
+
"single": [
|
| 91 |
+
{
|
| 92 |
+
"Sequence": {
|
| 93 |
+
"id": "A",
|
| 94 |
+
"type_id": 0
|
| 95 |
+
}
|
| 96 |
+
},
|
| 97 |
+
{
|
| 98 |
+
"SpecialToken": {
|
| 99 |
+
"id": "</src>",
|
| 100 |
+
"type_id": 0
|
| 101 |
+
}
|
| 102 |
+
}
|
| 103 |
+
],
|
| 104 |
+
"pair": [
|
| 105 |
+
{
|
| 106 |
+
"Sequence": {
|
| 107 |
+
"id": "A",
|
| 108 |
+
"type_id": 0
|
| 109 |
+
}
|
| 110 |
+
},
|
| 111 |
+
{
|
| 112 |
+
"SpecialToken": {
|
| 113 |
+
"id": "</src>",
|
| 114 |
+
"type_id": 0
|
| 115 |
+
}
|
| 116 |
+
},
|
| 117 |
+
{
|
| 118 |
+
"Sequence": {
|
| 119 |
+
"id": "B",
|
| 120 |
+
"type_id": 1
|
| 121 |
+
}
|
| 122 |
+
},
|
| 123 |
+
{
|
| 124 |
+
"SpecialToken": {
|
| 125 |
+
"id": "</src>",
|
| 126 |
+
"type_id": 1
|
| 127 |
+
}
|
| 128 |
+
}
|
| 129 |
+
],
|
| 130 |
+
"special_tokens": {
|
| 131 |
+
"</src>": {
|
| 132 |
+
"id": "</src>",
|
| 133 |
+
"ids": [
|
| 134 |
+
230
|
| 135 |
+
],
|
| 136 |
+
"tokens": [
|
| 137 |
+
"</src>"
|
| 138 |
+
]
|
| 139 |
+
}
|
| 140 |
+
}
|
| 141 |
+
},
|
| 142 |
+
"decoder": {
|
| 143 |
+
"type": "Sequence",
|
| 144 |
+
"decoders": [
|
| 145 |
+
{
|
| 146 |
+
"type": "Replace",
|
| 147 |
+
"pattern": {
|
| 148 |
+
"String": ""
|
| 149 |
+
},
|
| 150 |
+
"content": ""
|
| 151 |
+
},
|
| 152 |
+
{
|
| 153 |
+
"type": "Fuse"
|
| 154 |
+
}
|
| 155 |
+
]
|
| 156 |
+
},
|
| 157 |
+
"model": {
|
| 158 |
+
"type": "WordLevel",
|
| 159 |
+
"vocab": {
|
| 160 |
+
"<pad>": 0,
|
| 161 |
+
"<s>": 1,
|
| 162 |
+
"</s>": 2,
|
| 163 |
+
"<unk>": 3,
|
| 164 |
+
" ": 4,
|
| 165 |
+
"!": 5,
|
| 166 |
+
"\"": 6,
|
| 167 |
+
"#": 7,
|
| 168 |
+
"$": 8,
|
| 169 |
+
"%": 9,
|
| 170 |
+
"&": 10,
|
| 171 |
+
"'": 11,
|
| 172 |
+
"(": 12,
|
| 173 |
+
")": 13,
|
| 174 |
+
"*": 14,
|
| 175 |
+
"+": 15,
|
| 176 |
+
",": 16,
|
| 177 |
+
"-": 17,
|
| 178 |
+
".": 18,
|
| 179 |
+
"/": 19,
|
| 180 |
+
"0": 20,
|
| 181 |
+
"1": 21,
|
| 182 |
+
"2": 22,
|
| 183 |
+
"3": 23,
|
| 184 |
+
"4": 24,
|
| 185 |
+
"5": 25,
|
| 186 |
+
"6": 26,
|
| 187 |
+
"7": 27,
|
| 188 |
+
"8": 28,
|
| 189 |
+
"9": 29,
|
| 190 |
+
":": 30,
|
| 191 |
+
";": 31,
|
| 192 |
+
"<": 32,
|
| 193 |
+
"=": 33,
|
| 194 |
+
">": 34,
|
| 195 |
+
"?": 35,
|
| 196 |
+
"@": 36,
|
| 197 |
+
"A": 37,
|
| 198 |
+
"B": 38,
|
| 199 |
+
"C": 39,
|
| 200 |
+
"D": 40,
|
| 201 |
+
"E": 41,
|
| 202 |
+
"F": 42,
|
| 203 |
+
"G": 43,
|
| 204 |
+
"H": 44,
|
| 205 |
+
"I": 45,
|
| 206 |
+
"J": 46,
|
| 207 |
+
"K": 47,
|
| 208 |
+
"L": 48,
|
| 209 |
+
"M": 49,
|
| 210 |
+
"N": 50,
|
| 211 |
+
"O": 51,
|
| 212 |
+
"P": 52,
|
| 213 |
+
"Q": 53,
|
| 214 |
+
"R": 54,
|
| 215 |
+
"S": 55,
|
| 216 |
+
"T": 56,
|
| 217 |
+
"U": 57,
|
| 218 |
+
"V": 58,
|
| 219 |
+
"W": 59,
|
| 220 |
+
"X": 60,
|
| 221 |
+
"Y": 61,
|
| 222 |
+
"Z": 62,
|
| 223 |
+
"[": 63,
|
| 224 |
+
"]": 64,
|
| 225 |
+
"^": 65,
|
| 226 |
+
"_": 66,
|
| 227 |
+
"a": 67,
|
| 228 |
+
"b": 68,
|
| 229 |
+
"c": 69,
|
| 230 |
+
"d": 70,
|
| 231 |
+
"e": 71,
|
| 232 |
+
"f": 72,
|
| 233 |
+
"g": 73,
|
| 234 |
+
"h": 74,
|
| 235 |
+
"i": 75,
|
| 236 |
+
"j": 76,
|
| 237 |
+
"k": 77,
|
| 238 |
+
"l": 78,
|
| 239 |
+
"m": 79,
|
| 240 |
+
"n": 80,
|
| 241 |
+
"o": 81,
|
| 242 |
+
"p": 82,
|
| 243 |
+
"q": 83,
|
| 244 |
+
"r": 84,
|
| 245 |
+
"s": 85,
|
| 246 |
+
"t": 86,
|
| 247 |
+
"u": 87,
|
| 248 |
+
"v": 88,
|
| 249 |
+
"w": 89,
|
| 250 |
+
"x": 90,
|
| 251 |
+
"y": 91,
|
| 252 |
+
"z": 92,
|
| 253 |
+
"|": 93,
|
| 254 |
+
"~": 94,
|
| 255 |
+
"¡": 95,
|
| 256 |
+
"«": 96,
|
| 257 |
+
"®": 97,
|
| 258 |
+
"°": 98,
|
| 259 |
+
"·": 99,
|
| 260 |
+
"»": 100,
|
| 261 |
+
"×": 101,
|
| 262 |
+
"à": 102,
|
| 263 |
+
"á": 103,
|
| 264 |
+
"ç": 104,
|
| 265 |
+
"è": 105,
|
| 266 |
+
"é": 106,
|
| 267 |
+
"í": 107,
|
| 268 |
+
"ǔ": 108,
|
| 269 |
+
"ก": 109,
|
| 270 |
+
"ท": 110,
|
| 271 |
+
"น": 111,
|
| 272 |
+
"ภ": 112,
|
| 273 |
+
"ย": 113,
|
| 274 |
+
"ร": 114,
|
| 275 |
+
"ษ": 115,
|
| 276 |
+
"า": 116,
|
| 277 |
+
"ิ": 117,
|
| 278 |
+
"เ": 118,
|
| 279 |
+
"ไ": 119,
|
| 280 |
+
"๒": 120,
|
| 281 |
+
"๕": 121,
|
| 282 |
+
"ລ": 122,
|
| 283 |
+
"ວ": 123,
|
| 284 |
+
"ສ": 124,
|
| 285 |
+
"າ": 125,
|
| 286 |
+
"ក": 126,
|
| 287 |
+
"ខ": 127,
|
| 288 |
+
"គ": 128,
|
| 289 |
+
"ឃ": 129,
|
| 290 |
+
"ង": 130,
|
| 291 |
+
"ច": 131,
|
| 292 |
+
"ឆ": 132,
|
| 293 |
+
"ជ": 133,
|
| 294 |
+
"ឈ": 134,
|
| 295 |
+
"ញ": 135,
|
| 296 |
+
"ដ": 136,
|
| 297 |
+
"ឋ": 137,
|
| 298 |
+
"ឌ": 138,
|
| 299 |
+
"ឍ": 139,
|
| 300 |
+
"ណ": 140,
|
| 301 |
+
"ត": 141,
|
| 302 |
+
"ថ": 142,
|
| 303 |
+
"ទ": 143,
|
| 304 |
+
"ធ": 144,
|
| 305 |
+
"ន": 145,
|
| 306 |
+
"ប": 146,
|
| 307 |
+
"ផ": 147,
|
| 308 |
+
"ព": 148,
|
| 309 |
+
"ភ": 149,
|
| 310 |
+
"ម": 150,
|
| 311 |
+
"យ": 151,
|
| 312 |
+
"រ": 152,
|
| 313 |
+
"ល": 153,
|
| 314 |
+
"វ": 154,
|
| 315 |
+
"ឝ": 155,
|
| 316 |
+
"ឞ": 156,
|
| 317 |
+
"ស": 157,
|
| 318 |
+
"ហ": 158,
|
| 319 |
+
"ឡ": 159,
|
| 320 |
+
"អ": 160,
|
| 321 |
+
"ឥ": 161,
|
| 322 |
+
"ឦ": 162,
|
| 323 |
+
"ឧ": 163,
|
| 324 |
+
"ឩ": 164,
|
| 325 |
+
"ឪ": 165,
|
| 326 |
+
"ឫ": 166,
|
| 327 |
+
"ឬ": 167,
|
| 328 |
+
"ឭ": 168,
|
| 329 |
+
"ឮ": 169,
|
| 330 |
+
"ឯ": 170,
|
| 331 |
+
"ឰ": 171,
|
| 332 |
+
"ឱ": 172,
|
| 333 |
+
"ឲ": 173,
|
| 334 |
+
"ឳ": 174,
|
| 335 |
+
"ា": 175,
|
| 336 |
+
"ិ": 176,
|
| 337 |
+
"ី": 177,
|
| 338 |
+
"ឹ": 178,
|
| 339 |
+
"ឺ": 179,
|
| 340 |
+
"ុ": 180,
|
| 341 |
+
"ូ": 181,
|
| 342 |
+
"ួ": 182,
|
| 343 |
+
"ើ": 183,
|
| 344 |
+
"ឿ": 184,
|
| 345 |
+
"ៀ": 185,
|
| 346 |
+
"េ": 186,
|
| 347 |
+
"ែ": 187,
|
| 348 |
+
"ៃ": 188,
|
| 349 |
+
"ោ": 189,
|
| 350 |
+
"ៅ": 190,
|
| 351 |
+
"ំ": 191,
|
| 352 |
+
"ះ": 192,
|
| 353 |
+
"ៈ": 193,
|
| 354 |
+
"៉": 194,
|
| 355 |
+
"៊": 195,
|
| 356 |
+
"់": 196,
|
| 357 |
+
"៌": 197,
|
| 358 |
+
"៍": 198,
|
| 359 |
+
"៎": 199,
|
| 360 |
+
"៏": 200,
|
| 361 |
+
"័": 201,
|
| 362 |
+
"៑": 202,
|
| 363 |
+
"្": 203,
|
| 364 |
+
"។": 204,
|
| 365 |
+
"៕": 205,
|
| 366 |
+
"៖": 206,
|
| 367 |
+
"ៗ": 207,
|
| 368 |
+
"៘": 208,
|
| 369 |
+
"៛": 209,
|
| 370 |
+
"‑": 210,
|
| 371 |
+
"–": 211,
|
| 372 |
+
"—": 212,
|
| 373 |
+
"‘": 213,
|
| 374 |
+
"’": 214,
|
| 375 |
+
"“": 215,
|
| 376 |
+
"”": 216,
|
| 377 |
+
"•": 217,
|
| 378 |
+
"…": 218,
|
| 379 |
+
"‹": 219,
|
| 380 |
+
"›": 220,
|
| 381 |
+
"→": 221,
|
| 382 |
+
"−": 222,
|
| 383 |
+
"●": 223,
|
| 384 |
+
"�": 224,
|
| 385 |
+
"😀": 225,
|
| 386 |
+
"😛": 226,
|
| 387 |
+
"🙂": 227,
|
| 388 |
+
"<src_pad>": 228,
|
| 389 |
+
"<src_s>": 229,
|
| 390 |
+
"</src>": 230,
|
| 391 |
+
"<src_unk>": 231,
|
| 392 |
+
" ": 232,
|
| 393 |
+
"!": 233,
|
| 394 |
+
"\"": 234,
|
| 395 |
+
"#": 235,
|
| 396 |
+
"$": 236,
|
| 397 |
+
"%": 237,
|
| 398 |
+
"&": 238,
|
| 399 |
+
"'": 239,
|
| 400 |
+
"(": 240,
|
| 401 |
+
")": 241,
|
| 402 |
+
"*": 242,
|
| 403 |
+
"+": 243,
|
| 404 |
+
",": 244,
|
| 405 |
+
"-": 245,
|
| 406 |
+
".": 246,
|
| 407 |
+
"/": 247,
|
| 408 |
+
"0": 248,
|
| 409 |
+
"1": 249,
|
| 410 |
+
"2": 250,
|
| 411 |
+
"3": 251,
|
| 412 |
+
"4": 252,
|
| 413 |
+
"5": 253,
|
| 414 |
+
"6": 254,
|
| 415 |
+
"7": 255,
|
| 416 |
+
"8": 256,
|
| 417 |
+
"9": 257,
|
| 418 |
+
":": 258,
|
| 419 |
+
";": 259,
|
| 420 |
+
"<": 260,
|
| 421 |
+
"=": 261,
|
| 422 |
+
">": 262,
|
| 423 |
+
"?": 263,
|
| 424 |
+
"@": 264,
|
| 425 |
+
"A": 265,
|
| 426 |
+
"B": 266,
|
| 427 |
+
"C": 267,
|
| 428 |
+
"D": 268,
|
| 429 |
+
"E": 269,
|
| 430 |
+
"F": 270,
|
| 431 |
+
"G": 271,
|
| 432 |
+
"H": 272,
|
| 433 |
+
"I": 273,
|
| 434 |
+
"J": 274,
|
| 435 |
+
"K": 275,
|
| 436 |
+
"L": 276,
|
| 437 |
+
"M": 277,
|
| 438 |
+
"N": 278,
|
| 439 |
+
"O": 279,
|
| 440 |
+
"P": 280,
|
| 441 |
+
"Q": 281,
|
| 442 |
+
"R": 282,
|
| 443 |
+
"S": 283,
|
| 444 |
+
"T": 284,
|
| 445 |
+
"U": 285,
|
| 446 |
+
"V": 286,
|
| 447 |
+
"W": 287,
|
| 448 |
+
"X": 288,
|
| 449 |
+
"Y": 289,
|
| 450 |
+
"Z": 290,
|
| 451 |
+
"[": 291,
|
| 452 |
+
"]": 292,
|
| 453 |
+
"^": 293,
|
| 454 |
+
"_": 294,
|
| 455 |
+
"a": 295,
|
| 456 |
+
"b": 296,
|
| 457 |
+
"c": 297,
|
| 458 |
+
"d": 298,
|
| 459 |
+
"e": 299,
|
| 460 |
+
"f": 300,
|
| 461 |
+
"g": 301,
|
| 462 |
+
"h": 302,
|
| 463 |
+
"i": 303,
|
| 464 |
+
"j": 304,
|
| 465 |
+
"k": 305,
|
| 466 |
+
"l": 306,
|
| 467 |
+
"m": 307,
|
| 468 |
+
"n": 308,
|
| 469 |
+
"o": 309,
|
| 470 |
+
"p": 310,
|
| 471 |
+
"q": 311,
|
| 472 |
+
"r": 312,
|
| 473 |
+
"s": 313,
|
| 474 |
+
"t": 314,
|
| 475 |
+
"u": 315,
|
| 476 |
+
"v": 316,
|
| 477 |
+
"w": 317,
|
| 478 |
+
"x": 318,
|
| 479 |
+
"y": 319,
|
| 480 |
+
"z": 320,
|
| 481 |
+
"|": 321,
|
| 482 |
+
"~": 322,
|
| 483 |
+
"¡": 323,
|
| 484 |
+
"«": 324,
|
| 485 |
+
"®": 325,
|
| 486 |
+
"°": 326,
|
| 487 |
+
"·": 327,
|
| 488 |
+
"»": 328,
|
| 489 |
+
"×": 329,
|
| 490 |
+
"à": 330,
|
| 491 |
+
"á": 331,
|
| 492 |
+
"ç": 332,
|
| 493 |
+
"è": 333,
|
| 494 |
+
"é": 334,
|
| 495 |
+
"í": 335,
|
| 496 |
+
"ǔ": 336,
|
| 497 |
+
"ɓ": 337,
|
| 498 |
+
"ə": 338,
|
| 499 |
+
"ก": 339,
|
| 500 |
+
"ท": 340,
|
| 501 |
+
"น": 341,
|
| 502 |
+
"ภ": 342,
|
| 503 |
+
"ย": 343,
|
| 504 |
+
"ร": 344,
|
| 505 |
+
"ษ": 345,
|
| 506 |
+
"า": 346,
|
| 507 |
+
"ิ": 347,
|
| 508 |
+
"เ": 348,
|
| 509 |
+
"ไ": 349,
|
| 510 |
+
"๒": 350,
|
| 511 |
+
"๕": 351,
|
| 512 |
+
"ລ": 352,
|
| 513 |
+
"ວ": 353,
|
| 514 |
+
"ສ": 354,
|
| 515 |
+
"າ": 355,
|
| 516 |
+
"ក": 356,
|
| 517 |
+
"ខ": 357,
|
| 518 |
+
"គ": 358,
|
| 519 |
+
"ឃ": 359,
|
| 520 |
+
"ង": 360,
|
| 521 |
+
"ច": 361,
|
| 522 |
+
"ឆ": 362,
|
| 523 |
+
"ជ": 363,
|
| 524 |
+
"ឈ": 364,
|
| 525 |
+
"ញ": 365,
|
| 526 |
+
"ដ": 366,
|
| 527 |
+
"ឋ": 367,
|
| 528 |
+
"ឌ": 368,
|
| 529 |
+
"ណ": 369,
|
| 530 |
+
"ត": 370,
|
| 531 |
+
"ថ": 371,
|
| 532 |
+
"ទ": 372,
|
| 533 |
+
"ធ": 373,
|
| 534 |
+
"ន": 374,
|
| 535 |
+
"ប": 375,
|
| 536 |
+
"ផ": 376,
|
| 537 |
+
"ព": 377,
|
| 538 |
+
"ភ": 378,
|
| 539 |
+
"ម": 379,
|
| 540 |
+
"យ": 380,
|
| 541 |
+
"រ": 381,
|
| 542 |
+
"ល": 382,
|
| 543 |
+
"វ": 383,
|
| 544 |
+
"ស": 384,
|
| 545 |
+
"ហ": 385,
|
| 546 |
+
"ឡ": 386,
|
| 547 |
+
"អ": 387,
|
| 548 |
+
"ឧ": 388,
|
| 549 |
+
"ឫ": 389,
|
| 550 |
+
"ឬ": 390,
|
| 551 |
+
"ឯ": 391,
|
| 552 |
+
"ឲ": 392,
|
| 553 |
+
"ា": 393,
|
| 554 |
+
"ិ": 394,
|
| 555 |
+
"ី": 395,
|
| 556 |
+
"ឹ": 396,
|
| 557 |
+
"ឺ": 397,
|
| 558 |
+
"ុ": 398,
|
| 559 |
+
"ូ": 399,
|
| 560 |
+
"ួ": 400,
|
| 561 |
+
"ើ": 401,
|
| 562 |
+
"ឿ": 402,
|
| 563 |
+
"ៀ": 403,
|
| 564 |
+
"េ": 404,
|
| 565 |
+
"ែ": 405,
|
| 566 |
+
"ៃ": 406,
|
| 567 |
+
"ោ": 407,
|
| 568 |
+
"ៅ": 408,
|
| 569 |
+
"ំ": 409,
|
| 570 |
+
"ះ": 410,
|
| 571 |
+
"ៈ": 411,
|
| 572 |
+
"៉": 412,
|
| 573 |
+
"៊": 413,
|
| 574 |
+
"់": 414,
|
| 575 |
+
"៌": 415,
|
| 576 |
+
"៍": 416,
|
| 577 |
+
"៏": 417,
|
| 578 |
+
"័": 418,
|
| 579 |
+
"្": 419,
|
| 580 |
+
"។": 420,
|
| 581 |
+
"៕": 421,
|
| 582 |
+
"៖": 422,
|
| 583 |
+
"ៗ": 423,
|
| 584 |
+
"៘": 424,
|
| 585 |
+
"៛": 425,
|
| 586 |
+
"‑": 426,
|
| 587 |
+
"–": 427,
|
| 588 |
+
"—": 428,
|
| 589 |
+
"‘": 429,
|
| 590 |
+
"’": 430,
|
| 591 |
+
"“": 431,
|
| 592 |
+
"”": 432,
|
| 593 |
+
"•": 433,
|
| 594 |
+
"…": 434,
|
| 595 |
+
"‹": 435,
|
| 596 |
+
"›": 436,
|
| 597 |
+
"→": 437,
|
| 598 |
+
"−": 438,
|
| 599 |
+
"●": 439,
|
| 600 |
+
"�": 440,
|
| 601 |
+
"😀": 441,
|
| 602 |
+
"😛": 442,
|
| 603 |
+
"🙂": 443
|
| 604 |
+
},
|
| 605 |
+
"unk_token": "<src_unk>"
|
| 606 |
+
}
|
| 607 |
+
}
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"tokenizer_class": "PreTrainedTokenizerFast",
|
| 3 |
+
"model_max_length": 250,
|
| 4 |
+
"padding_side": "right",
|
| 5 |
+
"truncation_side": "right",
|
| 6 |
+
"clean_up_tokenization_spaces": false,
|
| 7 |
+
"bos_token": "<s>",
|
| 8 |
+
"eos_token": "</s>",
|
| 9 |
+
"pad_token": "<pad>",
|
| 10 |
+
"unk_token": "<src_unk>",
|
| 11 |
+
"model_input_names": [
|
| 12 |
+
"input_ids",
|
| 13 |
+
"attention_mask"
|
| 14 |
+
]
|
| 15 |
+
}
|