DistilWord-23k

DistilWord-23k is an ultra-compact language model trained on 3.8k synthetic words from Harley-ml/TinyWord2-128k.

It solves the single-head attention collapse and character-stutter issues present in MicroWord-23k.

Benchmark Comparison

Metric	Original MicroWord-23k	DistilWord-23k (Ours)
Validation Loss	6.4105	3.0633
Validation PPL	608.18	21.40 (Lower = Better)

Generations from the model

Prompt	Original MicroWord-23k	DistilWord-23k
w	wwwhwwwwgandwlwss	wxing
z	zzzzx's	zred
el	elel	ely
app	appco	applval