30k — Single-Head Language Model
A small next-token language model built in PyTorch, with one causal self-attention head and a tokenizer I trained separately. This is a text-continuation experiment, not a chat model.
Model ID: vishal24p/30k. The checkpoint was trained for 30,000 steps.
Model
| Setting | Value |
|---|---|
| Parameters | 2,134,016 |
| Vocabulary | 16,384 |
| Context length | 128 tokens |
| Embedding / head size | 64 / 64 |
| Attention heads | 1 |
Token embeddings + learned position embeddings → Q/K/V → masked attention → LM head. No MLP, residual connections, LayerNorm, or additional attention layers.
This model uses my custom tokenizer. See the repository for tokenizer details and code.
Language-model training
Model dataset:
| Split | Tokens |
|---|---|
| Training | Approximately 700 million |
| Validation | Approximately 5 million |
Total: approximately 705 million tokens.
Optimizer: AdamW. Learning rate: 3e-4. Objective: next-token cross-entropy.
Run
From the release folder (locally, hf_release/):
uv run --with torch --with regex python train.py
Despite its filename, the release's train.py runs interactive generation. It loads model/30k.pt and the included custom tokenizer. Enter a prompt to continue it; type exit to quit.
The model has a short context window and is not instruction-tuned. Its text may be incoherent.