| license: mit | |
| tags: | |
| - small-language-model | |
| - tiny-language-model | |
| - word-generation | |
| # DistilWord-23k | |
| DistilWord-23k is an ultra-compact language model trained on 3.8k synthetic words from [Harley-ml/TinyWord2-128k](https://huggingface.co/Harley-ml/TinyWord2-128k). | |
| It solves the single-head attention collapse and character-stutter issues present in `MicroWord-23k`. | |
| ## Benchmark Comparison | |
| | Metric | Original MicroWord-23k | DistilWord-23k (Ours) | | |
| | :--- | :--- | :--- | | |
| | **Validation Loss** | 6.4105 | **3.0633** | | |
| | **Validation PPL** | 608.18 | **21.40** *(Lower = Better)* | | |
| ## Generations from the model | |
| | Prompt | Original MicroWord-23k | DistilWord-23k | | |
| | :--- | :--- | :--- | | |
| | `w` | `wwwhwwwwgandwlwss` | `wxing` | | |
| | `z` | `zzzzx's` | `zred` | | |
| | `el` | `elel` | `ely` | | |
| | `app` | `appco` | `applval` | | |