Update README.md
Browse files
README.md
CHANGED
|
@@ -64,6 +64,8 @@ JetonCount is a 7k-parameter MLP regression model trained to predict the number
|
|
| 64 |
- Tokenizers: [`tokenizers_used.txt`](https://huggingface.co/fromziro/JetonCount/blob/main/tokenizers_used.txt)
|
| 65 |
- Datasets: [`datasets_used.txt`](https://huggingface.co/fromziro/JetonCount/edit/main/datasets_used.txt)
|
| 66 |
|
|
|
|
|
|
|
| 67 |
|
| 68 |
### Training Details
|
| 69 |
|
|
|
|
| 64 |
- Tokenizers: [`tokenizers_used.txt`](https://huggingface.co/fromziro/JetonCount/blob/main/tokenizers_used.txt)
|
| 65 |
- Datasets: [`datasets_used.txt`](https://huggingface.co/fromziro/JetonCount/edit/main/datasets_used.txt)
|
| 66 |
|
| 67 |
+
The sources include math, code, educational text, general web, social media posts (reddit), and instruct data.
|
| 68 |
+
The tokenizers range from GPT2 to DeepSeek-v4 and have varying vocabulary sizes (250 to 256,000).
|
| 69 |
|
| 70 |
### Training Details
|
| 71 |
|