File size: 6,188 Bytes
46144df | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 | ---
license: mit
language:
- en
pipeline_tag: text-generation
library_name: pytorch
datasets:
- karpathy/tiny_shakespeare
tags:
- character-level
- educational
- custom-architecture
- cpu
---
# OxMini
OxMini is a 2.17M-parameter character language model trained as a transparent, CPU-scale architecture experiment.
> **Architecture honesty notice:** OxMini is not “Ox Alpha shrunk down,” is not GLM-5.3-Flash, and is not a reproduction of either system. Public reporting identifies Ox Alpha as GLM-5.3-Flash, a 320B-total/18B-active sparse MoE with KDA, MLA+DSA, four-stream mHC, and vision. OxMini only studies toy approximations of the 3:1 hybrid pattern and multi-stream residual routing. It has no MoE, DSA indexer, vision encoder, long-context kernel, or comparable capability.
## Architecture
| Source component | OxMini approximation | Status |
|---|---|---|
| 3:1 KDA : MLA/DSA attention | Six recurrent KDA-lite + two global MLA-lite blocks | Simplified |
| KDA | Explicit channel-gated delta-rule recurrence, NoPE | Simplified; no chunk kernel |
| MLA | Full queries plus low-rank joint K/V bottleneck and RoPE | Simplified |
| DSA | None | Dropped |
| Sparse MoE | Dense SwiGLU | Dropped |
| Four-stream mHC | Static four-stream softmax routing; optional Sinkhorn projection | Substantially simplified |
| Vision | None | Dropped |
- Parameters: **2,169,388**
- Context: 256 characters
- Layers: 8 (`KDA-lite ×3, MLA-lite ×1`, repeated)
- Width / heads: 128 / 4
- Residual streams: 4
- Tokenizer: corpus-built character vocabulary with `<unk>`
- Weight format: safetensors
## Training data
The checkpoint uses Tiny Shakespeare downloaded from the Karpathy `char-rnn` corpus URL. It is deterministically split 90%/5%/5% by character into train/validation/test, and the vocabulary is built only from the training split. Shakespeare's underlying plays are public-domain text; this model repository does not redistribute prepared corpus files and does not claim ownership over source text.
- Corpus: 1,115,394 characters; SHA-256 `86c4e6aa9db7c042ec79f339dcb96d42b0075e16b8fc2e86bf0ca57e2dc565ed`
- Splits: 1,003,854 train / 55,770 validation / 55,770 test characters
- Vocabulary: 66 tokens including `<unk>`; zero unknown tokens in all splits
- Optimizer: AdamW with 5% warmup, cosine decay, and gradient clipping
- Run: 1,000 updates, batch size 8, seed 1337, CPU only
- Hardware/time: Apple M4 Pro, 10 threads, PyTorch 2.9.1, 1,820.9 seconds
## Measured evaluation and ablation
### Release checkpoint
| Metric | Value |
|---|---:|
| Initial validation loss | 4.2799 |
| Final/best validation loss | 1.7944 |
| Test cross-entropy | 1.80425 |
| Test perplexity | 6.075 |
| Test next-character accuracy | 47.79% |
The seen/unseen copy probe was inconclusive: seen accuracy was higher (50.0% vs 37.5%), but seen cross-entropy was worse (2.213 vs 2.075). The arithmetic probe was skipped because the vocabulary does not contain all digits.
### Matched 120-step ablation
| Variant | Parameters | Validation loss | Test PPL | Next-char accuracy | Time |
|---|---:|---:|---:|---:|---:|
| Plain GPT | 2,107,776 | 2.8228 | 16.54 | 26.17% | 17.1s |
| Hybrid, one stream | 2,168,984 | **2.4979** | **12.02** | **37.35%** | 150.2s |
| Hybrid + mHC-lite | 2,169,388 | 2.5595 | 12.78 | 35.21% | 153.5s |
All values are copied from local JSON/CSV artifacts included in this repository. The ablation arms used the same data, seed, update count, context, width/depth, and training recipe; parameter spread was 2.84%. Hybrid-only did best in this short run; mHC-lite did not help. The recurrent arms were about nine times slower than baseline because this implementation prioritizes a legible Python recurrence over a chunk-parallel kernel. These toy, single-seed results are descriptive and do not establish that KDA or mHC is generally better.
## Sample generations
Temperature 0.7, 120 new characters, unedited:
```text
ROMEO:
He prove sword in Recadars, he with that my grow say the broble joy these dead
The words.
WARWICK:
King to have done's
```
The sample learns speaker labels, line breaks, and some Shakespeare-like surface form, but remains incoherent. See `samples.md` for every fixed prompt at temperatures 0.7 and 1.0; none were filtered or cherry-picked.
## Load
This is a plain PyTorch custom architecture, not a Transformers `AutoModel` package. Download the repository and import its bundled source:
```python
from pathlib import Path
import sys
from huggingface_hub import snapshot_download
snapshot = Path(snapshot_download("Shivam3002/OxMini"))
sys.path.insert(0, str(snapshot / "src"))
from oxmini import CharTokenizer, OxMiniForCausalLM
from oxmini.generate import generate_text
model = OxMiniForCausalLM.from_pretrained(snapshot).eval()
tokenizer = CharTokenizer.load(snapshot / "tokenizer.json")
print(generate_text(model, tokenizer, "ROMEO:\n", max_new_tokens=120, temperature=0.8))
```
## Intended use
Educational inspection, CPU smoke tests, and toy ablations only.
## Limitations
This model is not suitable for factual, safety-critical, production, or user-facing tasks. Its corpus is tiny and narrow, its 256-character context is short, and character-level samples can be incoherent. The probes are diagnostic rather than standard language-model benchmarks. Architecture labels ending in “-lite” are approximations, not faithful kernels.
## References
- Sebastian Raschka, [GLM-5.3-Flash Architecture Notes](https://sebastianraschka.com/blog/2026/glm-5-3-flash-architecture-notes.html), Aug. 26, 2026.
- Kimi Team et al., [Kimi Linear](https://arxiv.org/abs/2510.26692) and [official code](https://github.com/MoonshotAI/Kimi-Linear).
- DeepSeek-AI, [DeepSeek-V3.2](https://arxiv.org/abs/2512.02556) and [V3.2-Exp code](https://github.com/deepseek-ai/DeepSeek-V3.2-Exp).
- Xie et al., [mHC](https://arxiv.org/abs/2512.24880).
- Hugging Face, [DeepSeek-V4 documentation](https://huggingface.co/docs/transformers/model_doc/deepseek_v4).
- Zhu et al., [Hyper-Connections](https://arxiv.org/abs/2409.19606).
## License
MIT for OxMini code and original model artifacts. Source-text rights remain with their respective sources.
|