mengsay commited on
Commit
0420834
·
verified ·
1 Parent(s): 7a7ad8f

Model folders in the transformers format (Blenderbot classes, tokenizer, generation config)

Browse files
README.md CHANGED
@@ -2,6 +2,7 @@
2
  license: apache-2.0
3
  language:
4
  - km
 
5
  pipeline_tag: text2text-generation
6
  tags:
7
  - transliteration
@@ -68,13 +69,15 @@ Paper: *Samdei Type: Separating What Is Written from How It Is Spelled for Roman
68
  ## Files
69
 
70
  ```
71
- config.json configuration and vocabularies (JSON)
72
- model.safetensors 69.5 MiB weights (fp32)
 
 
 
73
  onnx/encoder_kv.int8.onnx 11.3 MiB input characters -> cross-attention K/V of every decoder layer
74
  onnx/decoder_step.int8.onnx 6.3 MiB one decoder step with a self-attention KV cache
75
  onnx/vocab.json vocabularies and graph shapes
76
- channel/config.json reverse channel model (Khmer -> romanized), 7.1M parameters, used in reranking
77
- channel/model.safetensors 27.3 MiB its weights (fp32)
78
  lm/char5_nowiki.npz 8.9 MiB hashed character 5-gram LM
79
  lm/wordfull_nowiki.json.gz 12.2 MiB word bigram LM over khmercut words (counts, gzipped JSON)
80
  rerank.json decoding and rerank settings used for every number below
@@ -130,6 +133,35 @@ conv = OnnxConverter(f"{d}/onnx")
130
  hyps = conv.beam("knhom jong tov phsar", k=5) # rerank with rr.rerank(...) as above
131
  ```
132
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
133
  **Decoding, as in the paper.**
134
  - Beam search with k = 5.
135
  - The input is cut into pieces of at most eight words, with a break after any punctuation mark. Each piece is decoded and reranked on its own, and the outputs are joined with a space.
@@ -153,6 +185,7 @@ hyps = conv.beam("knhom jong tov phsar", k=5) # rerank with rr.rerank(...) as
153
 
154
  Int8 quantization does not change CER on Dev-Sentences with beam 5 and no LMs (0.007 in fp32 and in int8). The cached ONNX decoder reproduces full re-decoding exactly.
155
 
 
156
  ## Training
157
 
158
  Base was trained in three stages. Each stage started from the best checkpoint of the stage before and replayed its data.
 
2
  license: apache-2.0
3
  language:
4
  - km
5
+ library_name: transformers
6
  pipeline_tag: text2text-generation
7
  tags:
8
  - transliteration
 
69
  ## Files
70
 
71
  ```
72
+ config.json configuration and vocabularies (transformers format)
73
+ generation_config.json beam search, k = 5
74
+ model.safetensors 71.1 MiB weights (fp32)
75
+ tokenizer.json character tokenizer
76
+ tokenizer_config.json its settings
77
  onnx/encoder_kv.int8.onnx 11.3 MiB input characters -> cross-attention K/V of every decoder layer
78
  onnx/decoder_step.int8.onnx 6.3 MiB one decoder step with a self-attention KV cache
79
  onnx/vocab.json vocabularies and graph shapes
80
+ channel/ 28.4 MiB reverse channel model (Khmer -> romanized), 7.1M parameters, used in reranking; the same five files
 
81
  lm/char5_nowiki.npz 8.9 MiB hashed character 5-gram LM
82
  lm/wordfull_nowiki.json.gz 12.2 MiB word bigram LM over khmercut words (counts, gzipped JSON)
83
  rerank.json decoding and rerank settings used for every number below
 
133
  hyps = conv.beam("knhom jong tov phsar", k=5) # rerank with rr.rerank(...) as above
134
  ```
135
 
136
+ **transformers, without rkhmer** (`pip install transformers torch`; see Model). This is beam search with k = 5 and no reranking: 0.062 CER / 24% exact on Samdei Type Bench, against 0.052 / 30% with the reranker above.
137
+
138
+ ```python
139
+ from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
140
+
141
+ repo = "samdei/samdei-type-base"
142
+ tok = AutoTokenizer.from_pretrained(repo)
143
+ model = AutoModelForSeq2SeqLM.from_pretrained(repo)
144
+
145
+ def pieces(text: str, max_words: int = 8) -> list[str]:
146
+ """At most max_words words per piece, with a break after punctuation, as in the paper."""
147
+ out, cur = [], []
148
+ for w in text.split():
149
+ cur.append(w)
150
+ if len(cur) == max_words or w[-1] in ",.;:?!។":
151
+ out.append(" ".join(cur))
152
+ cur = []
153
+ if cur:
154
+ out.append(" ".join(cur))
155
+ return out or [text]
156
+
157
+ def convert(text: str) -> str:
158
+ batch = tok(pieces(text), padding=True, truncation=True, return_tensors="pt")
159
+ out = model.generate(**batch) # beam search, k = 5 (generation_config.json)
160
+ return " ".join(tok.batch_decode(out, skip_special_tokens=True))
161
+
162
+ print(convert("knhom jong tov phsar")) # ខ្ញុំចង់ទៅផ្សារ
163
+ ```
164
+
165
  **Decoding, as in the paper.**
166
  - Beam search with k = 5.
167
  - The input is cut into pieces of at most eight words, with a break after any punctuation mark. Each piece is decoded and reranked on its own, and the outputs are joined with a space.
 
185
 
186
  Int8 quantization does not change CER on Dev-Sentences with beam 5 and no LMs (0.007 in fp32 and in int8). The cached ONNX decoder reproduces full re-decoding exactly.
187
 
188
+ **transformers format.** The model folders load with `AutoModelForSeq2SeqLM` and `AutoTokenizer` and need no custom code. They use transformers' Blenderbot classes only because the architecture is the same (a pre-LN encoder-decoder with scaled embeddings, learned positions from 0 and no embedding LayerNorm). The model has nothing else in common with BlenderBot. Blenderbot has one vocabulary for both sides, so the tokenizer stacks the two: ids 0–227 are the output symbols and the input symbols follow. The output layer gives input ids a fixed logit of −10⁴, so they are never generated. For the same reason transformers reports 18.63M (7.43M for the channel model) parameters: the stacked embedding tables and the output layer carry rows that are never used. `rkhmer` reads the same files and maps them back to its own layout.
189
  ## Training
190
 
191
  Base was trained in three stages. Each stage started from the best checkpoint of the stage before and replayed its data.
channel/config.json CHANGED
@@ -1,22 +1,50 @@
1
  {
2
- "model_type": "samdei-type",
3
- "format_version": 1,
4
- "src_vocab": 228,
5
- "tgt_vocab": 216,
 
6
  "d_model": 256,
7
- "nhead": 4,
8
- "enc_layers": 6,
9
- "dec_layers": 2,
10
- "ff": 1024,
 
 
 
 
 
11
  "dropout": 0.1,
12
- "max_len": 256,
13
- "vocab": {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
  "special_tokens": [
15
  "<pad>",
16
  "<s>",
17
  "</s>",
18
  "<unk>"
19
  ],
 
 
 
 
 
 
 
20
  "source": [
21
  " ",
22
  "!",
@@ -456,9 +484,10 @@
456
  "😀",
457
  "😛",
458
  "🙂"
459
- ]
460
- },
461
- "training": {
462
- "step": 20000
 
463
  }
464
  }
 
1
  {
2
+ "architectures": [
3
+ "BlenderbotForConditionalGeneration"
4
+ ],
5
+ "model_type": "blenderbot",
6
+ "vocab_size": 444,
7
  "d_model": 256,
8
+ "encoder_layers": 6,
9
+ "decoder_layers": 2,
10
+ "encoder_attention_heads": 4,
11
+ "decoder_attention_heads": 4,
12
+ "encoder_ffn_dim": 1024,
13
+ "decoder_ffn_dim": 1024,
14
+ "activation_function": "gelu",
15
+ "max_position_embeddings": 256,
16
+ "scale_embedding": true,
17
  "dropout": 0.1,
18
+ "attention_dropout": 0.1,
19
+ "activation_dropout": 0.1,
20
+ "encoder_layerdrop": 0.0,
21
+ "decoder_layerdrop": 0.0,
22
+ "init_std": 0.02,
23
+ "pad_token_id": 0,
24
+ "bos_token_id": 1,
25
+ "eos_token_id": 2,
26
+ "decoder_start_token_id": 1,
27
+ "forced_eos_token_id": null,
28
+ "encoder_no_repeat_ngram_size": 0,
29
+ "is_encoder_decoder": true,
30
+ "tie_word_embeddings": false,
31
+ "use_cache": true,
32
+ "torch_dtype": "float32",
33
+ "samdei_type": {
34
+ "format_version": 2,
35
  "special_tokens": [
36
  "<pad>",
37
  "<s>",
38
  "</s>",
39
  "<unk>"
40
  ],
41
+ "source_special_tokens": [
42
+ "<src_pad>",
43
+ "<src_s>",
44
+ "</src>",
45
+ "<src_unk>"
46
+ ],
47
+ "source_offset": 216,
48
  "source": [
49
  " ",
50
  "!",
 
484
  "😀",
485
  "😛",
486
  "🙂"
487
+ ],
488
+ "max_input_chars": 249,
489
+ "training": {
490
+ "step": 20000
491
+ }
492
  }
493
  }
channel/generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "decoder_start_token_id": 1,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 0,
6
+ "forced_eos_token_id": null,
7
+ "num_beams": 5,
8
+ "max_length": 201,
9
+ "length_penalty": 1.0
10
+ }
channel/model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:33ff4dc755cf97f9fbb7e07c46a23d63eb3fff3a10ab082e9a678cac22dfaf7c
3
- size 28598808
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b222590113a348452d4652b33ed83f08e933d7c3a25cecce8fcb6346e26ca797
3
+ size 29748272
channel/tokenizer.json ADDED
@@ -0,0 +1,607 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "version": "1.0",
3
+ "truncation": null,
4
+ "padding": null,
5
+ "added_tokens": [
6
+ {
7
+ "id": 0,
8
+ "content": "<pad>",
9
+ "single_word": false,
10
+ "lstrip": false,
11
+ "rstrip": false,
12
+ "normalized": false,
13
+ "special": true
14
+ },
15
+ {
16
+ "id": 1,
17
+ "content": "<s>",
18
+ "single_word": false,
19
+ "lstrip": false,
20
+ "rstrip": false,
21
+ "normalized": false,
22
+ "special": true
23
+ },
24
+ {
25
+ "id": 2,
26
+ "content": "</s>",
27
+ "single_word": false,
28
+ "lstrip": false,
29
+ "rstrip": false,
30
+ "normalized": false,
31
+ "special": true
32
+ },
33
+ {
34
+ "id": 3,
35
+ "content": "<unk>",
36
+ "single_word": false,
37
+ "lstrip": false,
38
+ "rstrip": false,
39
+ "normalized": false,
40
+ "special": true
41
+ },
42
+ {
43
+ "id": 216,
44
+ "content": "<src_pad>",
45
+ "single_word": false,
46
+ "lstrip": false,
47
+ "rstrip": false,
48
+ "normalized": false,
49
+ "special": true
50
+ },
51
+ {
52
+ "id": 217,
53
+ "content": "<src_s>",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ {
61
+ "id": 218,
62
+ "content": "</src>",
63
+ "single_word": false,
64
+ "lstrip": false,
65
+ "rstrip": false,
66
+ "normalized": false,
67
+ "special": true
68
+ },
69
+ {
70
+ "id": 219,
71
+ "content": "<src_unk>",
72
+ "single_word": false,
73
+ "lstrip": false,
74
+ "rstrip": false,
75
+ "normalized": false,
76
+ "special": true
77
+ }
78
+ ],
79
+ "normalizer": null,
80
+ "pre_tokenizer": {
81
+ "type": "Split",
82
+ "pattern": {
83
+ "Regex": "[\\s\\S]"
84
+ },
85
+ "behavior": "Isolated",
86
+ "invert": false
87
+ },
88
+ "post_processor": {
89
+ "type": "TemplateProcessing",
90
+ "single": [
91
+ {
92
+ "Sequence": {
93
+ "id": "A",
94
+ "type_id": 0
95
+ }
96
+ },
97
+ {
98
+ "SpecialToken": {
99
+ "id": "</src>",
100
+ "type_id": 0
101
+ }
102
+ }
103
+ ],
104
+ "pair": [
105
+ {
106
+ "Sequence": {
107
+ "id": "A",
108
+ "type_id": 0
109
+ }
110
+ },
111
+ {
112
+ "SpecialToken": {
113
+ "id": "</src>",
114
+ "type_id": 0
115
+ }
116
+ },
117
+ {
118
+ "Sequence": {
119
+ "id": "B",
120
+ "type_id": 1
121
+ }
122
+ },
123
+ {
124
+ "SpecialToken": {
125
+ "id": "</src>",
126
+ "type_id": 1
127
+ }
128
+ }
129
+ ],
130
+ "special_tokens": {
131
+ "</src>": {
132
+ "id": "</src>",
133
+ "ids": [
134
+ 218
135
+ ],
136
+ "tokens": [
137
+ "</src>"
138
+ ]
139
+ }
140
+ }
141
+ },
142
+ "decoder": {
143
+ "type": "Sequence",
144
+ "decoders": [
145
+ {
146
+ "type": "Replace",
147
+ "pattern": {
148
+ "String": ""
149
+ },
150
+ "content": ""
151
+ },
152
+ {
153
+ "type": "Fuse"
154
+ }
155
+ ]
156
+ },
157
+ "model": {
158
+ "type": "WordLevel",
159
+ "vocab": {
160
+ "<pad>": 0,
161
+ "<s>": 1,
162
+ "</s>": 2,
163
+ "<unk>": 3,
164
+ " ": 4,
165
+ "!": 5,
166
+ "\"": 6,
167
+ "#": 7,
168
+ "$": 8,
169
+ "%": 9,
170
+ "&": 10,
171
+ "'": 11,
172
+ "(": 12,
173
+ ")": 13,
174
+ "*": 14,
175
+ "+": 15,
176
+ ",": 16,
177
+ "-": 17,
178
+ ".": 18,
179
+ "/": 19,
180
+ "0": 20,
181
+ "1": 21,
182
+ "2": 22,
183
+ "3": 23,
184
+ "4": 24,
185
+ "5": 25,
186
+ "6": 26,
187
+ "7": 27,
188
+ "8": 28,
189
+ "9": 29,
190
+ ":": 30,
191
+ ";": 31,
192
+ "<": 32,
193
+ "=": 33,
194
+ ">": 34,
195
+ "?": 35,
196
+ "@": 36,
197
+ "A": 37,
198
+ "B": 38,
199
+ "C": 39,
200
+ "D": 40,
201
+ "E": 41,
202
+ "F": 42,
203
+ "G": 43,
204
+ "H": 44,
205
+ "I": 45,
206
+ "J": 46,
207
+ "K": 47,
208
+ "L": 48,
209
+ "M": 49,
210
+ "N": 50,
211
+ "O": 51,
212
+ "P": 52,
213
+ "Q": 53,
214
+ "R": 54,
215
+ "S": 55,
216
+ "T": 56,
217
+ "U": 57,
218
+ "V": 58,
219
+ "W": 59,
220
+ "X": 60,
221
+ "Y": 61,
222
+ "Z": 62,
223
+ "[": 63,
224
+ "]": 64,
225
+ "^": 65,
226
+ "_": 66,
227
+ "a": 67,
228
+ "b": 68,
229
+ "c": 69,
230
+ "d": 70,
231
+ "e": 71,
232
+ "f": 72,
233
+ "g": 73,
234
+ "h": 74,
235
+ "i": 75,
236
+ "j": 76,
237
+ "k": 77,
238
+ "l": 78,
239
+ "m": 79,
240
+ "n": 80,
241
+ "o": 81,
242
+ "p": 82,
243
+ "q": 83,
244
+ "r": 84,
245
+ "s": 85,
246
+ "t": 86,
247
+ "u": 87,
248
+ "v": 88,
249
+ "w": 89,
250
+ "x": 90,
251
+ "y": 91,
252
+ "z": 92,
253
+ "|": 93,
254
+ "~": 94,
255
+ "¡": 95,
256
+ "«": 96,
257
+ "®": 97,
258
+ "°": 98,
259
+ "·": 99,
260
+ "»": 100,
261
+ "×": 101,
262
+ "à": 102,
263
+ "á": 103,
264
+ "ç": 104,
265
+ "è": 105,
266
+ "é": 106,
267
+ "í": 107,
268
+ "ǔ": 108,
269
+ "ɓ": 109,
270
+ "ə": 110,
271
+ "ก": 111,
272
+ "ท": 112,
273
+ "น": 113,
274
+ "ภ": 114,
275
+ "ย": 115,
276
+ "ร": 116,
277
+ "ษ": 117,
278
+ "า": 118,
279
+ "ิ": 119,
280
+ "เ": 120,
281
+ "ไ": 121,
282
+ "๒": 122,
283
+ "๕": 123,
284
+ "ລ": 124,
285
+ "ວ": 125,
286
+ "ສ": 126,
287
+ "າ": 127,
288
+ "ក": 128,
289
+ "ខ": 129,
290
+ "គ": 130,
291
+ "ឃ": 131,
292
+ "ង": 132,
293
+ "ច": 133,
294
+ "ឆ": 134,
295
+ "ជ": 135,
296
+ "ឈ": 136,
297
+ "ញ": 137,
298
+ "ដ": 138,
299
+ "ឋ": 139,
300
+ "ឌ": 140,
301
+ "ណ": 141,
302
+ "ត": 142,
303
+ "ថ": 143,
304
+ "ទ": 144,
305
+ "ធ": 145,
306
+ "ន": 146,
307
+ "ប": 147,
308
+ "ផ": 148,
309
+ "ព": 149,
310
+ "ភ": 150,
311
+ "ម": 151,
312
+ "��យ": 152,
313
+ "រ": 153,
314
+ "ល": 154,
315
+ "វ": 155,
316
+ "ស": 156,
317
+ "ហ": 157,
318
+ "ឡ": 158,
319
+ "អ": 159,
320
+ "ឧ": 160,
321
+ "ឫ": 161,
322
+ "ឬ": 162,
323
+ "ឯ": 163,
324
+ "ឲ": 164,
325
+ "ា": 165,
326
+ "ិ": 166,
327
+ "ី": 167,
328
+ "ឹ": 168,
329
+ "ឺ": 169,
330
+ "ុ": 170,
331
+ "ូ": 171,
332
+ "ួ": 172,
333
+ "ើ": 173,
334
+ "ឿ": 174,
335
+ "ៀ": 175,
336
+ "េ": 176,
337
+ "ែ": 177,
338
+ "ៃ": 178,
339
+ "ោ": 179,
340
+ "ៅ": 180,
341
+ "ំ": 181,
342
+ "ះ": 182,
343
+ "ៈ": 183,
344
+ "៉": 184,
345
+ "៊": 185,
346
+ "់": 186,
347
+ "៌": 187,
348
+ "៍": 188,
349
+ "៏": 189,
350
+ "័": 190,
351
+ "្": 191,
352
+ "។": 192,
353
+ "៕": 193,
354
+ "៖": 194,
355
+ "ៗ": 195,
356
+ "៘": 196,
357
+ "៛": 197,
358
+ "‑": 198,
359
+ "–": 199,
360
+ "—": 200,
361
+ "‘": 201,
362
+ "’": 202,
363
+ "“": 203,
364
+ "”": 204,
365
+ "•": 205,
366
+ "…": 206,
367
+ "‹": 207,
368
+ "›": 208,
369
+ "→": 209,
370
+ "−": 210,
371
+ "●": 211,
372
+ "�": 212,
373
+ "😀": 213,
374
+ "😛": 214,
375
+ "🙂": 215,
376
+ "<src_pad>": 216,
377
+ "<src_s>": 217,
378
+ "</src>": 218,
379
+ "<src_unk>": 219,
380
+ " ": 220,
381
+ "!": 221,
382
+ "\"": 222,
383
+ "#": 223,
384
+ "$": 224,
385
+ "%": 225,
386
+ "&": 226,
387
+ "'": 227,
388
+ "(": 228,
389
+ ")": 229,
390
+ "*": 230,
391
+ "+": 231,
392
+ ",": 232,
393
+ "-": 233,
394
+ ".": 234,
395
+ "/": 235,
396
+ "0": 236,
397
+ "1": 237,
398
+ "2": 238,
399
+ "3": 239,
400
+ "4": 240,
401
+ "5": 241,
402
+ "6": 242,
403
+ "7": 243,
404
+ "8": 244,
405
+ "9": 245,
406
+ ":": 246,
407
+ ";": 247,
408
+ "<": 248,
409
+ "=": 249,
410
+ ">": 250,
411
+ "?": 251,
412
+ "@": 252,
413
+ "A": 253,
414
+ "B": 254,
415
+ "C": 255,
416
+ "D": 256,
417
+ "E": 257,
418
+ "F": 258,
419
+ "G": 259,
420
+ "H": 260,
421
+ "I": 261,
422
+ "J": 262,
423
+ "K": 263,
424
+ "L": 264,
425
+ "M": 265,
426
+ "N": 266,
427
+ "O": 267,
428
+ "P": 268,
429
+ "Q": 269,
430
+ "R": 270,
431
+ "S": 271,
432
+ "T": 272,
433
+ "U": 273,
434
+ "V": 274,
435
+ "W": 275,
436
+ "X": 276,
437
+ "Y": 277,
438
+ "Z": 278,
439
+ "[": 279,
440
+ "]": 280,
441
+ "^": 281,
442
+ "_": 282,
443
+ "a": 283,
444
+ "b": 284,
445
+ "c": 285,
446
+ "d": 286,
447
+ "e": 287,
448
+ "f": 288,
449
+ "g": 289,
450
+ "h": 290,
451
+ "i": 291,
452
+ "j": 292,
453
+ "k": 293,
454
+ "l": 294,
455
+ "m": 295,
456
+ "n": 296,
457
+ "o": 297,
458
+ "p": 298,
459
+ "q": 299,
460
+ "r": 300,
461
+ "s": 301,
462
+ "t": 302,
463
+ "u": 303,
464
+ "v": 304,
465
+ "w": 305,
466
+ "x": 306,
467
+ "y": 307,
468
+ "z": 308,
469
+ "|": 309,
470
+ "~": 310,
471
+ "¡": 311,
472
+ "«": 312,
473
+ "®": 313,
474
+ "°": 314,
475
+ "·": 315,
476
+ "»": 316,
477
+ "×": 317,
478
+ "à": 318,
479
+ "á": 319,
480
+ "ç": 320,
481
+ "è": 321,
482
+ "é": 322,
483
+ "í": 323,
484
+ "ǔ": 324,
485
+ "ก": 325,
486
+ "ท": 326,
487
+ "น": 327,
488
+ "ภ": 328,
489
+ "ย": 329,
490
+ "ร": 330,
491
+ "ษ": 331,
492
+ "า": 332,
493
+ "ิ": 333,
494
+ "เ": 334,
495
+ "ไ": 335,
496
+ "๒": 336,
497
+ "๕": 337,
498
+ "ລ": 338,
499
+ "ວ": 339,
500
+ "ສ": 340,
501
+ "າ": 341,
502
+ "ក": 342,
503
+ "ខ": 343,
504
+ "គ": 344,
505
+ "ឃ": 345,
506
+ "ង": 346,
507
+ "ច": 347,
508
+ "ឆ": 348,
509
+ "ជ": 349,
510
+ "ឈ": 350,
511
+ "ញ": 351,
512
+ "ដ": 352,
513
+ "ឋ": 353,
514
+ "ឌ": 354,
515
+ "ឍ": 355,
516
+ "ណ": 356,
517
+ "ត": 357,
518
+ "ថ": 358,
519
+ "ទ": 359,
520
+ "ធ": 360,
521
+ "ន": 361,
522
+ "ប": 362,
523
+ "ផ": 363,
524
+ "ព": 364,
525
+ "ភ": 365,
526
+ "ម": 366,
527
+ "យ": 367,
528
+ "រ": 368,
529
+ "ល": 369,
530
+ "វ": 370,
531
+ "ឝ": 371,
532
+ "ឞ": 372,
533
+ "ស": 373,
534
+ "ហ": 374,
535
+ "ឡ": 375,
536
+ "អ": 376,
537
+ "ឥ": 377,
538
+ "ឦ": 378,
539
+ "ឧ": 379,
540
+ "ឩ": 380,
541
+ "ឪ": 381,
542
+ "ឫ": 382,
543
+ "ឬ": 383,
544
+ "ឭ": 384,
545
+ "ឮ": 385,
546
+ "ឯ": 386,
547
+ "ឰ": 387,
548
+ "ឱ": 388,
549
+ "ឲ": 389,
550
+ "ឳ": 390,
551
+ "ា": 391,
552
+ "ិ": 392,
553
+ "ី": 393,
554
+ "ឹ": 394,
555
+ "ឺ": 395,
556
+ "ុ": 396,
557
+ "ូ": 397,
558
+ "ួ": 398,
559
+ "ើ": 399,
560
+ "ឿ": 400,
561
+ "ៀ": 401,
562
+ "េ": 402,
563
+ "ែ": 403,
564
+ "ៃ": 404,
565
+ "ោ": 405,
566
+ "ៅ": 406,
567
+ "ំ": 407,
568
+ "ះ": 408,
569
+ "ៈ": 409,
570
+ "៉": 410,
571
+ "៊": 411,
572
+ "់": 412,
573
+ "៌": 413,
574
+ "៍": 414,
575
+ "៎": 415,
576
+ "៏": 416,
577
+ "័": 417,
578
+ "៑": 418,
579
+ "្": 419,
580
+ "។": 420,
581
+ "៕": 421,
582
+ "៖": 422,
583
+ "ៗ": 423,
584
+ "៘": 424,
585
+ "៛": 425,
586
+ "‑": 426,
587
+ "–": 427,
588
+ "—": 428,
589
+ "‘": 429,
590
+ "’": 430,
591
+ "“": 431,
592
+ "”": 432,
593
+ "•": 433,
594
+ "…": 434,
595
+ "‹": 435,
596
+ "›": 436,
597
+ "→": 437,
598
+ "−": 438,
599
+ "●": 439,
600
+ "�": 440,
601
+ "😀": 441,
602
+ "😛": 442,
603
+ "🙂": 443
604
+ },
605
+ "unk_token": "<src_unk>"
606
+ }
607
+ }
channel/tokenizer_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "model_max_length": 250,
4
+ "padding_side": "right",
5
+ "truncation_side": "right",
6
+ "clean_up_tokenization_spaces": false,
7
+ "bos_token": "<s>",
8
+ "eos_token": "</s>",
9
+ "pad_token": "<pad>",
10
+ "unk_token": "<src_unk>",
11
+ "model_input_names": [
12
+ "input_ids",
13
+ "attention_mask"
14
+ ]
15
+ }
config.json CHANGED
@@ -1,22 +1,50 @@
1
  {
2
- "model_type": "samdei-type",
3
- "format_version": 1,
4
- "src_vocab": 216,
5
- "tgt_vocab": 228,
 
6
  "d_model": 384,
7
- "nhead": 6,
8
- "enc_layers": 6,
9
- "dec_layers": 3,
10
- "ff": 1536,
 
 
 
 
 
11
  "dropout": 0.1,
12
- "max_len": 256,
13
- "vocab": {
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
  "special_tokens": [
15
  "<pad>",
16
  "<s>",
17
  "</s>",
18
  "<unk>"
19
  ],
 
 
 
 
 
 
 
20
  "source": [
21
  " ",
22
  "!",
@@ -456,9 +484,10 @@
456
  "😀",
457
  "😛",
458
  "🙂"
459
- ]
460
- },
461
- "training": {
462
- "step": 3000
 
463
  }
464
  }
 
1
  {
2
+ "architectures": [
3
+ "BlenderbotForConditionalGeneration"
4
+ ],
5
+ "model_type": "blenderbot",
6
+ "vocab_size": 444,
7
  "d_model": 384,
8
+ "encoder_layers": 6,
9
+ "decoder_layers": 3,
10
+ "encoder_attention_heads": 6,
11
+ "decoder_attention_heads": 6,
12
+ "encoder_ffn_dim": 1536,
13
+ "decoder_ffn_dim": 1536,
14
+ "activation_function": "gelu",
15
+ "max_position_embeddings": 256,
16
+ "scale_embedding": true,
17
  "dropout": 0.1,
18
+ "attention_dropout": 0.1,
19
+ "activation_dropout": 0.1,
20
+ "encoder_layerdrop": 0.0,
21
+ "decoder_layerdrop": 0.0,
22
+ "init_std": 0.02,
23
+ "pad_token_id": 0,
24
+ "bos_token_id": 1,
25
+ "eos_token_id": 2,
26
+ "decoder_start_token_id": 1,
27
+ "forced_eos_token_id": null,
28
+ "encoder_no_repeat_ngram_size": 0,
29
+ "is_encoder_decoder": true,
30
+ "tie_word_embeddings": false,
31
+ "use_cache": true,
32
+ "torch_dtype": "float32",
33
+ "samdei_type": {
34
+ "format_version": 2,
35
  "special_tokens": [
36
  "<pad>",
37
  "<s>",
38
  "</s>",
39
  "<unk>"
40
  ],
41
+ "source_special_tokens": [
42
+ "<src_pad>",
43
+ "<src_s>",
44
+ "</src>",
45
+ "<src_unk>"
46
+ ],
47
+ "source_offset": 228,
48
  "source": [
49
  " ",
50
  "!",
 
484
  "😀",
485
  "😛",
486
  "🙂"
487
+ ],
488
+ "max_input_chars": 249,
489
+ "training": {
490
+ "step": 3000
491
+ }
492
  }
493
  }
generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "decoder_start_token_id": 1,
3
+ "bos_token_id": 1,
4
+ "eos_token_id": 2,
5
+ "pad_token_id": 0,
6
+ "forced_eos_token_id": null,
7
+ "num_beams": 5,
8
+ "max_length": 201,
9
+ "length_penalty": 1.0
10
+ }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2c1d2e0d9cf12aabb304d8a725fd7ac5761e84a3f00dd7f4a3bf8847595740cc
3
- size 72825832
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:00f73802fe247d91a2337cffc8fdf5e8aae33416ab37bb775eaa7a6e8a6bf55a
3
+ size 74529304
tokenizer.json ADDED
@@ -0,0 +1,607 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "version": "1.0",
3
+ "truncation": null,
4
+ "padding": null,
5
+ "added_tokens": [
6
+ {
7
+ "id": 0,
8
+ "content": "<pad>",
9
+ "single_word": false,
10
+ "lstrip": false,
11
+ "rstrip": false,
12
+ "normalized": false,
13
+ "special": true
14
+ },
15
+ {
16
+ "id": 1,
17
+ "content": "<s>",
18
+ "single_word": false,
19
+ "lstrip": false,
20
+ "rstrip": false,
21
+ "normalized": false,
22
+ "special": true
23
+ },
24
+ {
25
+ "id": 2,
26
+ "content": "</s>",
27
+ "single_word": false,
28
+ "lstrip": false,
29
+ "rstrip": false,
30
+ "normalized": false,
31
+ "special": true
32
+ },
33
+ {
34
+ "id": 3,
35
+ "content": "<unk>",
36
+ "single_word": false,
37
+ "lstrip": false,
38
+ "rstrip": false,
39
+ "normalized": false,
40
+ "special": true
41
+ },
42
+ {
43
+ "id": 228,
44
+ "content": "<src_pad>",
45
+ "single_word": false,
46
+ "lstrip": false,
47
+ "rstrip": false,
48
+ "normalized": false,
49
+ "special": true
50
+ },
51
+ {
52
+ "id": 229,
53
+ "content": "<src_s>",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ {
61
+ "id": 230,
62
+ "content": "</src>",
63
+ "single_word": false,
64
+ "lstrip": false,
65
+ "rstrip": false,
66
+ "normalized": false,
67
+ "special": true
68
+ },
69
+ {
70
+ "id": 231,
71
+ "content": "<src_unk>",
72
+ "single_word": false,
73
+ "lstrip": false,
74
+ "rstrip": false,
75
+ "normalized": false,
76
+ "special": true
77
+ }
78
+ ],
79
+ "normalizer": null,
80
+ "pre_tokenizer": {
81
+ "type": "Split",
82
+ "pattern": {
83
+ "Regex": "[\\s\\S]"
84
+ },
85
+ "behavior": "Isolated",
86
+ "invert": false
87
+ },
88
+ "post_processor": {
89
+ "type": "TemplateProcessing",
90
+ "single": [
91
+ {
92
+ "Sequence": {
93
+ "id": "A",
94
+ "type_id": 0
95
+ }
96
+ },
97
+ {
98
+ "SpecialToken": {
99
+ "id": "</src>",
100
+ "type_id": 0
101
+ }
102
+ }
103
+ ],
104
+ "pair": [
105
+ {
106
+ "Sequence": {
107
+ "id": "A",
108
+ "type_id": 0
109
+ }
110
+ },
111
+ {
112
+ "SpecialToken": {
113
+ "id": "</src>",
114
+ "type_id": 0
115
+ }
116
+ },
117
+ {
118
+ "Sequence": {
119
+ "id": "B",
120
+ "type_id": 1
121
+ }
122
+ },
123
+ {
124
+ "SpecialToken": {
125
+ "id": "</src>",
126
+ "type_id": 1
127
+ }
128
+ }
129
+ ],
130
+ "special_tokens": {
131
+ "</src>": {
132
+ "id": "</src>",
133
+ "ids": [
134
+ 230
135
+ ],
136
+ "tokens": [
137
+ "</src>"
138
+ ]
139
+ }
140
+ }
141
+ },
142
+ "decoder": {
143
+ "type": "Sequence",
144
+ "decoders": [
145
+ {
146
+ "type": "Replace",
147
+ "pattern": {
148
+ "String": ""
149
+ },
150
+ "content": ""
151
+ },
152
+ {
153
+ "type": "Fuse"
154
+ }
155
+ ]
156
+ },
157
+ "model": {
158
+ "type": "WordLevel",
159
+ "vocab": {
160
+ "<pad>": 0,
161
+ "<s>": 1,
162
+ "</s>": 2,
163
+ "<unk>": 3,
164
+ " ": 4,
165
+ "!": 5,
166
+ "\"": 6,
167
+ "#": 7,
168
+ "$": 8,
169
+ "%": 9,
170
+ "&": 10,
171
+ "'": 11,
172
+ "(": 12,
173
+ ")": 13,
174
+ "*": 14,
175
+ "+": 15,
176
+ ",": 16,
177
+ "-": 17,
178
+ ".": 18,
179
+ "/": 19,
180
+ "0": 20,
181
+ "1": 21,
182
+ "2": 22,
183
+ "3": 23,
184
+ "4": 24,
185
+ "5": 25,
186
+ "6": 26,
187
+ "7": 27,
188
+ "8": 28,
189
+ "9": 29,
190
+ ":": 30,
191
+ ";": 31,
192
+ "<": 32,
193
+ "=": 33,
194
+ ">": 34,
195
+ "?": 35,
196
+ "@": 36,
197
+ "A": 37,
198
+ "B": 38,
199
+ "C": 39,
200
+ "D": 40,
201
+ "E": 41,
202
+ "F": 42,
203
+ "G": 43,
204
+ "H": 44,
205
+ "I": 45,
206
+ "J": 46,
207
+ "K": 47,
208
+ "L": 48,
209
+ "M": 49,
210
+ "N": 50,
211
+ "O": 51,
212
+ "P": 52,
213
+ "Q": 53,
214
+ "R": 54,
215
+ "S": 55,
216
+ "T": 56,
217
+ "U": 57,
218
+ "V": 58,
219
+ "W": 59,
220
+ "X": 60,
221
+ "Y": 61,
222
+ "Z": 62,
223
+ "[": 63,
224
+ "]": 64,
225
+ "^": 65,
226
+ "_": 66,
227
+ "a": 67,
228
+ "b": 68,
229
+ "c": 69,
230
+ "d": 70,
231
+ "e": 71,
232
+ "f": 72,
233
+ "g": 73,
234
+ "h": 74,
235
+ "i": 75,
236
+ "j": 76,
237
+ "k": 77,
238
+ "l": 78,
239
+ "m": 79,
240
+ "n": 80,
241
+ "o": 81,
242
+ "p": 82,
243
+ "q": 83,
244
+ "r": 84,
245
+ "s": 85,
246
+ "t": 86,
247
+ "u": 87,
248
+ "v": 88,
249
+ "w": 89,
250
+ "x": 90,
251
+ "y": 91,
252
+ "z": 92,
253
+ "|": 93,
254
+ "~": 94,
255
+ "¡": 95,
256
+ "«": 96,
257
+ "®": 97,
258
+ "°": 98,
259
+ "·": 99,
260
+ "»": 100,
261
+ "×": 101,
262
+ "à": 102,
263
+ "á": 103,
264
+ "ç": 104,
265
+ "è": 105,
266
+ "é": 106,
267
+ "í": 107,
268
+ "ǔ": 108,
269
+ "ก": 109,
270
+ "ท": 110,
271
+ "น": 111,
272
+ "ภ": 112,
273
+ "ย": 113,
274
+ "ร": 114,
275
+ "ษ": 115,
276
+ "า": 116,
277
+ "ิ": 117,
278
+ "เ": 118,
279
+ "ไ": 119,
280
+ "๒": 120,
281
+ "๕": 121,
282
+ "ລ": 122,
283
+ "ວ": 123,
284
+ "ສ": 124,
285
+ "າ": 125,
286
+ "ក": 126,
287
+ "ខ": 127,
288
+ "គ": 128,
289
+ "ឃ": 129,
290
+ "ង": 130,
291
+ "ច": 131,
292
+ "ឆ": 132,
293
+ "ជ": 133,
294
+ "ឈ": 134,
295
+ "ញ": 135,
296
+ "ដ": 136,
297
+ "ឋ": 137,
298
+ "ឌ": 138,
299
+ "ឍ": 139,
300
+ "ណ": 140,
301
+ "ត": 141,
302
+ "ថ": 142,
303
+ "ទ": 143,
304
+ "ធ": 144,
305
+ "ន": 145,
306
+ "ប": 146,
307
+ "ផ": 147,
308
+ "ព": 148,
309
+ "ភ": 149,
310
+ "ម": 150,
311
+ "យ": 151,
312
+ "រ": 152,
313
+ "ល": 153,
314
+ "វ": 154,
315
+ "ឝ": 155,
316
+ "ឞ": 156,
317
+ "ស": 157,
318
+ "ហ": 158,
319
+ "ឡ": 159,
320
+ "អ": 160,
321
+ "ឥ": 161,
322
+ "ឦ": 162,
323
+ "ឧ": 163,
324
+ "ឩ": 164,
325
+ "ឪ": 165,
326
+ "ឫ": 166,
327
+ "ឬ": 167,
328
+ "ឭ": 168,
329
+ "ឮ": 169,
330
+ "ឯ": 170,
331
+ "ឰ": 171,
332
+ "ឱ": 172,
333
+ "ឲ": 173,
334
+ "ឳ": 174,
335
+ "ា": 175,
336
+ "ិ": 176,
337
+ "ី": 177,
338
+ "ឹ": 178,
339
+ "ឺ": 179,
340
+ "ុ": 180,
341
+ "ូ": 181,
342
+ "ួ": 182,
343
+ "ើ": 183,
344
+ "ឿ": 184,
345
+ "ៀ": 185,
346
+ "េ": 186,
347
+ "ែ": 187,
348
+ "ៃ": 188,
349
+ "ោ": 189,
350
+ "ៅ": 190,
351
+ "ំ": 191,
352
+ "ះ": 192,
353
+ "ៈ": 193,
354
+ "៉": 194,
355
+ "៊": 195,
356
+ "់": 196,
357
+ "៌": 197,
358
+ "៍": 198,
359
+ "៎": 199,
360
+ "៏": 200,
361
+ "័": 201,
362
+ "៑": 202,
363
+ "្": 203,
364
+ "។": 204,
365
+ "៕": 205,
366
+ "៖": 206,
367
+ "ៗ": 207,
368
+ "៘": 208,
369
+ "៛": 209,
370
+ "‑": 210,
371
+ "–": 211,
372
+ "—": 212,
373
+ "‘": 213,
374
+ "’": 214,
375
+ "“": 215,
376
+ "”": 216,
377
+ "•": 217,
378
+ "…": 218,
379
+ "‹": 219,
380
+ "›": 220,
381
+ "→": 221,
382
+ "−": 222,
383
+ "●": 223,
384
+ "�": 224,
385
+ "😀": 225,
386
+ "😛": 226,
387
+ "🙂": 227,
388
+ "<src_pad>": 228,
389
+ "<src_s>": 229,
390
+ "</src>": 230,
391
+ "<src_unk>": 231,
392
+ " ": 232,
393
+ "!": 233,
394
+ "\"": 234,
395
+ "#": 235,
396
+ "$": 236,
397
+ "%": 237,
398
+ "&": 238,
399
+ "'": 239,
400
+ "(": 240,
401
+ ")": 241,
402
+ "*": 242,
403
+ "+": 243,
404
+ ",": 244,
405
+ "-": 245,
406
+ ".": 246,
407
+ "/": 247,
408
+ "0": 248,
409
+ "1": 249,
410
+ "2": 250,
411
+ "3": 251,
412
+ "4": 252,
413
+ "5": 253,
414
+ "6": 254,
415
+ "7": 255,
416
+ "8": 256,
417
+ "9": 257,
418
+ ":": 258,
419
+ ";": 259,
420
+ "<": 260,
421
+ "=": 261,
422
+ ">": 262,
423
+ "?": 263,
424
+ "@": 264,
425
+ "A": 265,
426
+ "B": 266,
427
+ "C": 267,
428
+ "D": 268,
429
+ "E": 269,
430
+ "F": 270,
431
+ "G": 271,
432
+ "H": 272,
433
+ "I": 273,
434
+ "J": 274,
435
+ "K": 275,
436
+ "L": 276,
437
+ "M": 277,
438
+ "N": 278,
439
+ "O": 279,
440
+ "P": 280,
441
+ "Q": 281,
442
+ "R": 282,
443
+ "S": 283,
444
+ "T": 284,
445
+ "U": 285,
446
+ "V": 286,
447
+ "W": 287,
448
+ "X": 288,
449
+ "Y": 289,
450
+ "Z": 290,
451
+ "[": 291,
452
+ "]": 292,
453
+ "^": 293,
454
+ "_": 294,
455
+ "a": 295,
456
+ "b": 296,
457
+ "c": 297,
458
+ "d": 298,
459
+ "e": 299,
460
+ "f": 300,
461
+ "g": 301,
462
+ "h": 302,
463
+ "i": 303,
464
+ "j": 304,
465
+ "k": 305,
466
+ "l": 306,
467
+ "m": 307,
468
+ "n": 308,
469
+ "o": 309,
470
+ "p": 310,
471
+ "q": 311,
472
+ "r": 312,
473
+ "s": 313,
474
+ "t": 314,
475
+ "u": 315,
476
+ "v": 316,
477
+ "w": 317,
478
+ "x": 318,
479
+ "y": 319,
480
+ "z": 320,
481
+ "|": 321,
482
+ "~": 322,
483
+ "¡": 323,
484
+ "«": 324,
485
+ "®": 325,
486
+ "°": 326,
487
+ "·": 327,
488
+ "»": 328,
489
+ "×": 329,
490
+ "à": 330,
491
+ "á": 331,
492
+ "ç": 332,
493
+ "è": 333,
494
+ "é": 334,
495
+ "í": 335,
496
+ "ǔ": 336,
497
+ "ɓ": 337,
498
+ "ə": 338,
499
+ "ก": 339,
500
+ "ท": 340,
501
+ "น": 341,
502
+ "ภ": 342,
503
+ "ย": 343,
504
+ "ร": 344,
505
+ "ษ": 345,
506
+ "า": 346,
507
+ "ิ": 347,
508
+ "เ": 348,
509
+ "ไ": 349,
510
+ "๒": 350,
511
+ "๕": 351,
512
+ "ລ": 352,
513
+ "ວ": 353,
514
+ "ສ": 354,
515
+ "າ": 355,
516
+ "ក": 356,
517
+ "ខ": 357,
518
+ "គ": 358,
519
+ "ឃ": 359,
520
+ "ង": 360,
521
+ "ច": 361,
522
+ "ឆ": 362,
523
+ "ជ": 363,
524
+ "ឈ": 364,
525
+ "ញ": 365,
526
+ "ដ": 366,
527
+ "ឋ": 367,
528
+ "ឌ": 368,
529
+ "ណ": 369,
530
+ "ត": 370,
531
+ "ថ": 371,
532
+ "ទ": 372,
533
+ "ធ": 373,
534
+ "ន": 374,
535
+ "ប": 375,
536
+ "ផ": 376,
537
+ "ព": 377,
538
+ "ភ": 378,
539
+ "ម": 379,
540
+ "យ": 380,
541
+ "រ": 381,
542
+ "ល": 382,
543
+ "វ": 383,
544
+ "ស": 384,
545
+ "ហ": 385,
546
+ "ឡ": 386,
547
+ "អ": 387,
548
+ "ឧ": 388,
549
+ "ឫ": 389,
550
+ "ឬ": 390,
551
+ "ឯ": 391,
552
+ "ឲ": 392,
553
+ "ា": 393,
554
+ "ិ": 394,
555
+ "ី": 395,
556
+ "ឹ": 396,
557
+ "ឺ": 397,
558
+ "ុ": 398,
559
+ "ូ": 399,
560
+ "ួ": 400,
561
+ "ើ": 401,
562
+ "ឿ": 402,
563
+ "ៀ": 403,
564
+ "េ": 404,
565
+ "ែ": 405,
566
+ "ៃ": 406,
567
+ "ោ": 407,
568
+ "ៅ": 408,
569
+ "ំ": 409,
570
+ "ះ": 410,
571
+ "ៈ": 411,
572
+ "៉": 412,
573
+ "៊": 413,
574
+ "់": 414,
575
+ "៌": 415,
576
+ "៍": 416,
577
+ "៏": 417,
578
+ "័": 418,
579
+ "្": 419,
580
+ "។": 420,
581
+ "៕": 421,
582
+ "៖": 422,
583
+ "ៗ": 423,
584
+ "៘": 424,
585
+ "៛": 425,
586
+ "‑": 426,
587
+ "–": 427,
588
+ "—": 428,
589
+ "‘": 429,
590
+ "’": 430,
591
+ "“": 431,
592
+ "”": 432,
593
+ "•": 433,
594
+ "…": 434,
595
+ "‹": 435,
596
+ "›": 436,
597
+ "→": 437,
598
+ "−": 438,
599
+ "●": 439,
600
+ "�": 440,
601
+ "😀": 441,
602
+ "😛": 442,
603
+ "🙂": 443
604
+ },
605
+ "unk_token": "<src_unk>"
606
+ }
607
+ }
tokenizer_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "model_max_length": 250,
4
+ "padding_side": "right",
5
+ "truncation_side": "right",
6
+ "clean_up_tokenization_spaces": false,
7
+ "bos_token": "<s>",
8
+ "eos_token": "</s>",
9
+ "pad_token": "<pad>",
10
+ "unk_token": "<src_unk>",
11
+ "model_input_names": [
12
+ "input_ids",
13
+ "attention_mask"
14
+ ]
15
+ }