File size: 3,290 Bytes
775cee0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
- tiny
- tiny-lm
- tiny-model
- slm
- small-language-model
- sub-1m
- from-scratch
- character-level
- char-gpt
metrics:
- perplexity
---
# Testr-100K
A **106,568-parameter** character-level GPT trained from scratch. Fulfils model request [#17](https://huggingface.co/spaces/Compactbot/model-requests/discussions/17) from @GGUFGuy.
## What it is
A minimal nanoGPT-style causal transformer operating at the **character level** (128-char vocabulary). This is a demonstration of training a working language model from absolute scratch with a very small parameter budget — not a tool for generating coherent text.
## Architecture
| Parameter | Value |
|-----------|-------|
| Layers | 4 |
| Embedding dim | 44 |
| Attention heads | 2 |
| Context length | 512 chars |
| FFN | GELU, mult 2.667 |
| Positional encoding | RoPE (θ=100000) |
| Normalization | LayerNorm (pre-norm) |
| Vocab | 128 chars + 1 UNK |
| Tied embeddings | Yes (head = embedding) |
| **Total params** | **106,568** |
## Training
- **Data:** 191 MB web text (character-level, uint8)
- **Steps:** 8,000
- **Batch size:** 32 × seq 512
- **Optimizer:** Muon (0.02) + AdamW (1e-4) for biases/norms
- **Schedule:** Cosine decay with warmup
- **Hardware:** RTX 5090 (32 GB)
- **dtype:** float32
## Quality (honest)
This is a **character-level** model at 106K params. It learns English character statistics and produces text that is *grammatical in shape* but **degenerates into repetition loops** within 2–3 sentences. It is not a coherent text generator.
**Val perplexity:** 39.77 (1M held-out chars from the same corpus)
### Sample outputs (greedy, temp=0)
> `The` → ` mean to the box and said, "I don't know what the boy was so happy to the box and said, "I want to t`
> `Once upon a time` → `, there was a little girl named Lily. He was so happy to the box and said, "I want to the box and sa`
> `I think that` → ` the boy was so happy to the box and said, "I want to the box and said. "I want to the`
### Sample outputs (temp=0.7)
> `The` → ` little girl named Lily was hands, "It was not want to be and field. It could go him to play. She pu`
> `Once upon a time` → ` there was a mommy was playing and the grandma happy.\n\nLily was very happy thast and pretty was so d`
The first sentence or two is often grammatical; after that the model locks into a phrase and repeats it. This is expected at this scale and vocabulary size.
## What it is NOT
- Not a subword/token-level model (it cannot "read" word-level benchmarks like BLiMP or ARC)
- Not a coherent paragraph generator
- Not comparable to sub-1M subword models on any word-level metric
## Reproducing
The model is a standard nanoGPT-style architecture. The training script is a standard causal transformer with Muon optimizer. The checkpoint is saved as safetensors with tied embeddings.
To generate: load `model.safetensors` into a GPT class matching the config, map characters to token IDs via `vocab.json`, and sample.
## Files
| File | Description |
|------|-------------|
| `model.safetensors` | Model weights (408 KB) |
| `config.json` | Architecture config |
| `vocab.json` | Character → token ID mapping (128 chars + UNK) | |