metadata
license: mit
tags:
- small-language-model
- tiny-language-model
- word-generation
DistilWord-23k
DistilWord-23k is an ultra-compact language model trained on 3.8k synthetic words from Harley-ml/TinyWord2-128k.
It solves the single-head attention collapse and character-stutter issues present in MicroWord-23k.
Benchmark Comparison
| Metric | Original MicroWord-23k | DistilWord-23k (Ours) |
|---|---|---|
| Validation Loss | 6.4105 | 3.0633 |
| Validation PPL | 608.18 | 21.40 (Lower = Better) |
Generations from the model
| Prompt | Original MicroWord-23k | DistilWord-23k |
|---|---|---|
w |
wwwhwwwwgandwlwss |
wxing |
z |
zzzzx's |
zred |
el |
elel |
ely |
app |
appco |
applval |