DistilWord2-23k

DistilWord2-23k is the second-generation ultra-compact language model (~23k parameters) in the DistilWord series.

Trained on 12,000 clean, low-entropy synthetic words generated by Harley-ml/LargeWord-1.5M, it achieves state-of-the-art sample efficiency and morphology for its size class.

Benchmark Progression

Model	Parameters	Training Distribution	Val Loss	Val PPL	Signal PSNR	Behavior
MicroWord-23k (Original)	~23k	750k Natural Words	6.4105	608.18	2.57 dB	Attention collapsed (wwww..., zzzz...)
DistilWord-23k (v1)	~23k	3.8k Synthetic (TinyWord2)	3.0633	21.40	4.37 dB	Stutter eliminated; basic morphology
DistilWord2-23k (Ours)	~23k	12k Synthetic (LargeWord-1.5M)	2.5415	12.70	4.50 dB	Real morphemes (afields, appers, zrings)

---

Qualitative Generations

Prompt	Original MicroWord-23k	DistilWord2-23k (Ours)	Learned Morphology
a	a (Stalled)	afields	Complete dictionary compound word (a + field + -s)
app	appco	appers	Agent noun pluralization (app + -er + -s)
z	zzzzx's (Stutter loop)	zrings	Complex inflectional cluster (-ing + -s)
dis	dis	disers	Morpheme suffix attachment
el	elel (Repeat loop)	elys	Adverbial suffix (-ly + -s)
sub	sub	subs	Standard plural inflection

---

Quick Inference

import torch
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

repo_id = "Useruser2statsaltalt/DistilWord2-23k"

tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
model.eval()

prompt = "app"
bos = tokenizer.bos_token or ""
inputs = tokenizer(bos + prompt, return_tensors="pt", add_special_tokens=False)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=12,
        do_sample=True,
        temperature=0.80,
        top_p=0.90,
        top_k=35,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

prompt_len = inputs["input_ids"].shape[-1]
completion = tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True)
print(f"Generated: {prompt + completion}")
## Architecture Specifications
Architecture: Qwen Causal LM (Micro-scale)
Hidden Layers: 1
Hidden Size: 16
Attention Heads: 1
Intermediate (SwiGLU): 56
Tied Embeddings: True
Unique Parameters: ~23,024