File size: 832 Bytes
28e7ae1 3136c33 28e7ae1 3136c33 28e7ae1 3136c33 28e7ae1 3136c33 28e7ae1 3136c33 28e7ae1 3136c33 28e7ae1 3136c33 28e7ae1 3136c33 28e7ae1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | ---
license: mit
tags:
- small-language-model
- tiny-language-model
- word-generation
---
# DistilWord-23k
DistilWord-23k is an ultra-compact language model trained on 3.8k synthetic words from [Harley-ml/TinyWord2-128k](https://huggingface.co/Harley-ml/TinyWord2-128k).
It solves the single-head attention collapse and character-stutter issues present in `MicroWord-23k`.
## Benchmark Comparison
| Metric | Original MicroWord-23k | DistilWord-23k (Ours) |
| :--- | :--- | :--- |
| **Validation Loss** | 6.4105 | **3.0633** |
| **Validation PPL** | 608.18 | **21.40** *(Lower = Better)* |
## Generations from the model
| Prompt | Original MicroWord-23k | DistilWord-23k |
| :--- | :--- | :--- |
| `w` | `wwwhwwwwgandwlwss` | `wxing` |
| `z` | `zzzzx's` | `zred` |
| `el` | `elel` | `ely` |
| `app` | `appco` | `applval` |
|