bctnry commited on
Commit
2309fb1
·
verified ·
1 Parent(s): 6070dfa

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -1
README.md CHANGED
@@ -78,7 +78,14 @@ breadth rather than on English alone.
78
  | Type | byte-level BPE, NFC normalisation |
79
  | Scripts | Latin, Cyrillic, Greek, Hebrew, Arabic, Thai, Devanagari, Bengali, Tamil, Telugu, Malayalam, Gujarati, Kannada, Odia, Gurmukhi, Sinhala, Han, Kana, Hangul |
80
  | Languages | 58 |
81
- | Special tokens | `<|pad|>` `<|bos|>` `<|eos|>` `<|unk|>` `<|system|>` `<|user|>` `<|assistant|>` `<|end|>` `<|fim_prefix|>` `<|fim_middle|>` `<|fim_suffix|>` `<|tool_call|>` `<|tool_result|>` `<|image|>` |
 
 
 
 
 
 
 
82
 
83
  ## Usage
84
 
 
78
  | Type | byte-level BPE, NFC normalisation |
79
  | Scripts | Latin, Cyrillic, Greek, Hebrew, Arabic, Thai, Devanagari, Bengali, Tamil, Telugu, Malayalam, Gujarati, Kannada, Odia, Gurmukhi, Sinhala, Han, Kana, Hangul |
80
  | Languages | 58 |
81
+
82
+ Special tokens:
83
+
84
+ ```
85
+ <|pad|> <|bos|> <|eos|> <|unk|> <|system|> <|user|> <|assistant|>
86
+ <|end|> <|fim_prefix|> <|fim_middle|> <|fim_suffix|> <|tool_call|>
87
+ <|tool_result|> <|image|>
88
+ ```
89
 
90
  ## Usage
91