| --- |
| license: mit |
| tags: |
| - small-language-model |
| - tiny-language-model |
| - word-generation |
| - distillation |
| - synthetic-data |
| datasets: |
| - Harley-ml/es-en-words |
| pipeline_tag: text-generation |
| --- |
| |
| # DistilWord2-23k |
|
|
| **DistilWord2-23k** is the second-generation ultra-compact language model (~23k parameters) in the DistilWord series. |
|
|
| Trained on **12,000 clean, low-entropy synthetic words** generated by [Harley-ml/LargeWord-1.5M](https://huggingface.co/Harley-ml/LargeWord-1.5M), it achieves state-of-the-art sample efficiency and morphology for its size class. |
|
|
| ## Benchmark Progression |
|
|
| | Model | Parameters | Training Distribution | Val Loss | Val PPL | Signal PSNR | Behavior | |
| | :--- | :--- | :--- | :--- | :--- | :--- | :--- | |
| | **MicroWord-23k (Original)** | ~23k | 750k Natural Words | 6.4105 | 608.18 | 2.57 dB | Attention collapsed (`wwww...`, `zzzz...`) | |
| | **DistilWord-23k (v1)** | ~23k | 3.8k Synthetic (`TinyWord2`) | 3.0633 | 21.40 | 4.37 dB | Stutter eliminated; basic morphology | |
| | **DistilWord2-23k (Ours)** | **~23k** | **12k Synthetic (`LargeWord-1.5M`)** | **2.5415** | **12.70** | **4.50 dB** | **Real morphemes (`afields`, `appers`, `zrings`)** | |
|
|
| --- |
|
|
| ## Qualitative Generations |
|
|
| | Prompt | Original MicroWord-23k | DistilWord2-23k (Ours) | Learned Morphology | |
| | :--- | :--- | :--- | :--- | |
| | `a` | `a` (Stalled) | **`afields`** | Complete dictionary compound word (*a* + *field* + *-s*) | |
| | `app` | `appco` | **`appers`** | Agent noun pluralization (*app* + *-er* + *-s*) | |
| | `z` | `zzzzx's` (Stutter loop) | **`zrings`** | Complex inflectional cluster (*-ing* + *-s*) | |
| | `dis` | `dis` | **`disers`** | Morpheme suffix attachment | |
| | `el` | `elel` (Repeat loop) | **`elys`** | Adverbial suffix (*-ly* + *-s*) | |
| | `sub` | `sub` | **`subs`** | Standard plural inflection | |
|
|
| --- |
|
|
| ## Quick Inference |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast |
| |
| repo_id = "Useruser2statsaltalt/DistilWord2-23k" |
| |
| tokenizer = PreTrainedTokenizerFast.from_pretrained(repo_id) |
| model = AutoModelForCausalLM.from_pretrained(repo_id) |
| model.eval() |
| |
| prompt = "app" |
| bos = tokenizer.bos_token or "" |
| inputs = tokenizer(bos + prompt, return_tensors="pt", add_special_tokens=False) |
| |
| with torch.inference_mode(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=12, |
| do_sample=True, |
| temperature=0.80, |
| top_p=0.90, |
| top_k=35, |
| eos_token_id=tokenizer.eos_token_id, |
| pad_token_id=tokenizer.pad_token_id, |
| ) |
| |
| prompt_len = inputs["input_ids"].shape[-1] |
| completion = tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True) |
| print(f"Generated: {prompt + completion}") |
| ## Architecture Specifications |
| Architecture: Qwen Causal LM (Micro-scale) |
| Hidden Layers: 1 |
| Hidden Size: 16 |
| Attention Heads: 1 |
| Intermediate (SwiGLU): 56 |
| Tied Embeddings: True |
| Unique Parameters: ~23,024 |
| |
