File size: 2,877 Bytes
bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a bfd7833 3c1510a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 | ---
license: mit
tags:
- small-language-model
- tiny-language-model
- word-generation
- distillation
- synthetic-data
datasets:
- Harley-ml/es-en-words
pipeline_tag: text-generation
---
# DistilWord2-23k
**DistilWord2-23k** is the second-generation ultra-compact language model (~23k parameters) in the DistilWord series.
Trained on **12,000 clean, low-entropy synthetic words** generated by [Harley-ml/LargeWord-1.5M](https://huggingface.co/Harley-ml/LargeWord-1.5M), it achieves state-of-the-art sample efficiency and morphology for its size class.
## Benchmark Progression
| Model | Parameters | Training Distribution | Val Loss | Val PPL | Signal PSNR | Behavior |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **MicroWord-23k (Original)** | ~23k | 750k Natural Words | 6.4105 | 608.18 | 2.57 dB | Attention collapsed (`wwww...`, `zzzz...`) |
| **DistilWord-23k (v1)** | ~23k | 3.8k Synthetic (`TinyWord2`) | 3.0633 | 21.40 | 4.37 dB | Stutter eliminated; basic morphology |
| **DistilWord2-23k (Ours)** | **~23k** | **12k Synthetic (`LargeWord-1.5M`)** | **2.5415** | **12.70** | **4.50 dB** | **Real morphemes (`afields`, `appers`, `zrings`)** |
---
## Qualitative Generations
| Prompt | Original MicroWord-23k | DistilWord2-23k (Ours) | Learned Morphology |
| :--- | :--- | :--- | :--- |
| `a` | `a` (Stalled) | **`afields`** | Complete dictionary compound word (*a* + *field* + *-s*) |
| `app` | `appco` | **`appers`** | Agent noun pluralization (*app* + *-er* + *-s*) |
| `z` | `zzzzx's` (Stutter loop) | **`zrings`** | Complex inflectional cluster (*-ing* + *-s*) |
| `dis` | `dis` | **`disers`** | Morpheme suffix attachment |
| `el` | `elel` (Repeat loop) | **`elys`** | Adverbial suffix (*-ly* + *-s*) |
| `sub` | `sub` | **`subs`** | Standard plural inflection |
---
## Quick Inference
```python
import torch
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast
repo_id = "Useruser2statsaltalt/DistilWord2-23k"
tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)
model.eval()
prompt = "app"
bos = tokenizer.bos_token or ""
inputs = tokenizer(bos + prompt, return_tensors="pt", add_special_tokens=False)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=12,
do_sample=True,
temperature=0.80,
top_p=0.90,
top_k=35,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
prompt_len = inputs["input_ids"].shape[-1]
completion = tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True)
print(f"Generated: {prompt + completion}")
## Architecture Specifications
Architecture: Qwen Causal LM (Micro-scale)
Hidden Layers: 1
Hidden Size: 16
Attention Heads: 1
Intermediate (SwiGLU): 56
Tied Embeddings: True
Unique Parameters: ~23,024
|