| --- |
| license: mit |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: pytorch |
| datasets: |
| - karpathy/tiny_shakespeare |
| tags: |
| - character-level |
| - educational |
| - custom-architecture |
| - cpu |
| --- |
| |
| # OxMini |
|
|
| OxMini is a 2.17M-parameter character language model trained as a transparent, CPU-scale architecture experiment. |
|
|
| > **Architecture honesty notice:** OxMini is not “Ox Alpha shrunk down,” is not GLM-5.3-Flash, and is not a reproduction of either system. Public reporting identifies Ox Alpha as GLM-5.3-Flash, a 320B-total/18B-active sparse MoE with KDA, MLA+DSA, four-stream mHC, and vision. OxMini only studies toy approximations of the 3:1 hybrid pattern and multi-stream residual routing. It has no MoE, DSA indexer, vision encoder, long-context kernel, or comparable capability. |
|
|
| ## Architecture |
|
|
| | Source component | OxMini approximation | Status | |
| |---|---|---| |
| | 3:1 KDA : MLA/DSA attention | Six recurrent KDA-lite + two global MLA-lite blocks | Simplified | |
| | KDA | Explicit channel-gated delta-rule recurrence, NoPE | Simplified; no chunk kernel | |
| | MLA | Full queries plus low-rank joint K/V bottleneck and RoPE | Simplified | |
| | DSA | None | Dropped | |
| | Sparse MoE | Dense SwiGLU | Dropped | |
| | Four-stream mHC | Static four-stream softmax routing; optional Sinkhorn projection | Substantially simplified | |
| | Vision | None | Dropped | |
|
|
| - Parameters: **2,169,388** |
| - Context: 256 characters |
| - Layers: 8 (`KDA-lite ×3, MLA-lite ×1`, repeated) |
| - Width / heads: 128 / 4 |
| - Residual streams: 4 |
| - Tokenizer: corpus-built character vocabulary with `<unk>` |
| - Weight format: safetensors |
|
|
| ## Training data |
|
|
| The checkpoint uses Tiny Shakespeare downloaded from the Karpathy `char-rnn` corpus URL. It is deterministically split 90%/5%/5% by character into train/validation/test, and the vocabulary is built only from the training split. Shakespeare's underlying plays are public-domain text; this model repository does not redistribute prepared corpus files and does not claim ownership over source text. |
|
|
| - Corpus: 1,115,394 characters; SHA-256 `86c4e6aa9db7c042ec79f339dcb96d42b0075e16b8fc2e86bf0ca57e2dc565ed` |
| - Splits: 1,003,854 train / 55,770 validation / 55,770 test characters |
| - Vocabulary: 66 tokens including `<unk>`; zero unknown tokens in all splits |
| - Optimizer: AdamW with 5% warmup, cosine decay, and gradient clipping |
| - Run: 1,000 updates, batch size 8, seed 1337, CPU only |
| - Hardware/time: Apple M4 Pro, 10 threads, PyTorch 2.9.1, 1,820.9 seconds |
|
|
| ## Measured evaluation and ablation |
|
|
| ### Release checkpoint |
|
|
| | Metric | Value | |
| |---|---:| |
| | Initial validation loss | 4.2799 | |
| | Final/best validation loss | 1.7944 | |
| | Test cross-entropy | 1.80425 | |
| | Test perplexity | 6.075 | |
| | Test next-character accuracy | 47.79% | |
|
|
| The seen/unseen copy probe was inconclusive: seen accuracy was higher (50.0% vs 37.5%), but seen cross-entropy was worse (2.213 vs 2.075). The arithmetic probe was skipped because the vocabulary does not contain all digits. |
|
|
| ### Matched 120-step ablation |
|
|
| | Variant | Parameters | Validation loss | Test PPL | Next-char accuracy | Time | |
| |---|---:|---:|---:|---:|---:| |
| | Plain GPT | 2,107,776 | 2.8228 | 16.54 | 26.17% | 17.1s | |
| | Hybrid, one stream | 2,168,984 | **2.4979** | **12.02** | **37.35%** | 150.2s | |
| | Hybrid + mHC-lite | 2,169,388 | 2.5595 | 12.78 | 35.21% | 153.5s | |
|
|
| All values are copied from local JSON/CSV artifacts included in this repository. The ablation arms used the same data, seed, update count, context, width/depth, and training recipe; parameter spread was 2.84%. Hybrid-only did best in this short run; mHC-lite did not help. The recurrent arms were about nine times slower than baseline because this implementation prioritizes a legible Python recurrence over a chunk-parallel kernel. These toy, single-seed results are descriptive and do not establish that KDA or mHC is generally better. |
|
|
| ## Sample generations |
|
|
| Temperature 0.7, 120 new characters, unedited: |
|
|
| ```text |
| ROMEO: |
| He prove sword in Recadars, he with that my grow say the broble joy these dead |
| The words. |
| |
| WARWICK: |
| King to have done's |
| ``` |
|
|
| The sample learns speaker labels, line breaks, and some Shakespeare-like surface form, but remains incoherent. See `samples.md` for every fixed prompt at temperatures 0.7 and 1.0; none were filtered or cherry-picked. |
|
|
| ## Load |
|
|
| This is a plain PyTorch custom architecture, not a Transformers `AutoModel` package. Download the repository and import its bundled source: |
|
|
| ```python |
| from pathlib import Path |
| import sys |
| from huggingface_hub import snapshot_download |
| |
| snapshot = Path(snapshot_download("Shivam3002/OxMini")) |
| sys.path.insert(0, str(snapshot / "src")) |
| |
| from oxmini import CharTokenizer, OxMiniForCausalLM |
| from oxmini.generate import generate_text |
| |
| model = OxMiniForCausalLM.from_pretrained(snapshot).eval() |
| tokenizer = CharTokenizer.load(snapshot / "tokenizer.json") |
| print(generate_text(model, tokenizer, "ROMEO:\n", max_new_tokens=120, temperature=0.8)) |
| ``` |
|
|
| ## Intended use |
|
|
| Educational inspection, CPU smoke tests, and toy ablations only. |
|
|
| ## Limitations |
|
|
| This model is not suitable for factual, safety-critical, production, or user-facing tasks. Its corpus is tiny and narrow, its 256-character context is short, and character-level samples can be incoherent. The probes are diagnostic rather than standard language-model benchmarks. Architecture labels ending in “-lite” are approximations, not faithful kernels. |
|
|
| ## References |
|
|
| - Sebastian Raschka, [GLM-5.3-Flash Architecture Notes](https://sebastianraschka.com/blog/2026/glm-5-3-flash-architecture-notes.html), Aug. 26, 2026. |
| - Kimi Team et al., [Kimi Linear](https://arxiv.org/abs/2510.26692) and [official code](https://github.com/MoonshotAI/Kimi-Linear). |
| - DeepSeek-AI, [DeepSeek-V3.2](https://arxiv.org/abs/2512.02556) and [V3.2-Exp code](https://github.com/deepseek-ai/DeepSeek-V3.2-Exp). |
| - Xie et al., [mHC](https://arxiv.org/abs/2512.24880). |
| - Hugging Face, [DeepSeek-V4 documentation](https://huggingface.co/docs/transformers/model_doc/deepseek_v4). |
| - Zhu et al., [Hyper-Connections](https://arxiv.org/abs/2409.19606). |
|
|
| ## License |
|
|
| MIT for OxMini code and original model artifacts. Source-text rights remain with their respective sources. |
|
|
