| ---
|
| language:
|
| - en
|
| - zh
|
| license: apache-2.0
|
| tags:
|
| - decision-model
|
| - calibration
|
| - system-one
|
| - from-scratch
|
| - probabilistic-decisions
|
| - pytorch
|
| pretty_name: MiniSystemOne
|
| ---
|
|
|
| # MiniSystemOne
|
|
|
| **Train a probabilistic decision model from scratch — no LLM, no decoding, no JSON generation.**
|
|
|
| A 26.89M-parameter model that takes a **state** plus **typed questions** and returns
|
| **typed decisions with calibrated probabilities** — in a single parallel forward pass.
|
| No autoregressive loop, no text output, no constrained-decoding tricks.
|
|
|
| Built from **random initialization**, MiniMind-style: one shared bidirectional encoder,
|
| one decision head, one loss. Trained in **4.7 hours** on a single RTX 4070 Laptop (8 GB).
|
|
|
| > ### ⚠️ This is not a Jev reproduction
|
| >
|
| > It is an **independent educational implementation**, inspired by the "System One Model"
|
| > idea popularized by TypeSafe AI's Jev (2026-09-15) — the way MiniMind is an educational
|
| > reimplementation of the LLaMA/GPT recipe rather than a reproduction of any model.
|
| > No Jev output was used in training, and no attempt was made to reverse-engineer it.
|
| > It does **not** claim to beat Jev on any axis.
|
|
|
| ---
|
|
|
| ## Files
|
|
|
| | File | Stage | Params | What it is |
|
| |---|---|---|---|
|
| | `decision.pth` | 2 | 26.89M | **The decision model.** The artifact this repo is about. |
|
| | `mlm.pth` | 1 | 25.31M | The MLM-pretrained encoder. `decision.pth` was warm-started from it. |
|
| | `tokenizer.json` / `tokenizer_config.json` | — | vocab 6400 | A **new** BPE, 11 special tokens. Not MiniMind's. |
|
|
|
| Both checkpoints are **self-describing**: the file carries the config it was built from
|
| and the provenance of its training data.
|
|
|
| ```
|
| decision.pth format=1 stage=decision step=33795 n_params=26889729
|
| tokenizer_sha1=bf7a131a109ea445 gen_version=1.0.0
|
| encoder_init=mlm.pth trained_on=synth max_len=1024 epochs=3
|
|
|
| mlm.pth format=1 stage=mlm step=66000 n_params=25309184
|
| tokenizer_sha1=bf7a131a109ea445 synth_gen_version=1.0.0
|
| max_len=512 epochs=8
|
| ```
|
|
|
| ---
|
|
|
| ## How to use these weights
|
|
|
| **There is no `from_pretrained` path, on purpose.** This is a decision-native architecture
|
| with a custom input packing format, not a causal LM. The packing is canonical and lives in
|
| exactly one place in the source repository; re-implementing it here would create a second
|
| implementation that drifts from the first.
|
|
|
| The code is at **`github.com/Colvin0315/MiniSystemOne`** (Apache-2.0).
|
|
|
| ```bash
|
| git clone https://github.com/Colvin0315/MiniSystemOne && cd MiniSystemOne
|
| pip install -r requirements.txt # torch 2.6.0+cu124, transformers, datasets
|
|
|
| # put decision.pth + tokenizer.json + tokenizer_config.json into model/ and out/decision/
|
| python eval/eval_harness.py --ckpt out/decision/decision.pth \
|
| --data dataset/synth --sets test_known
|
| ```
|
|
|
| Loading a checkpoint is one call, and it reads the embedded config and refuses a mismatch:
|
|
|
| ```python
|
| from trainer.trainer_utils import ckpt_info, init_model, verify_tokenizer
|
| from model.model_system_one import DecisionConfig, MiniSystemOneForDecision
|
|
|
| verify_tokenizer("decision.pth", "model") # rejects a mismatched vocabulary
|
| cfg = DecisionConfig(**{k: v for k, v in ckpt_info("decision.pth")["config"].items()
|
| if k in ("hidden_size", "num_hidden_layers", "vocab_size",
|
| "num_attention_heads", "num_key_value_heads",
|
| "intermediate_size")})
|
| model = init_model(MiniSystemOneForDecision, cfg, "decision.pth", "cuda")
|
| ```
|
|
|
| ---
|
|
|
| ## Results
|
|
|
| All numbers are from `decision.pth`, reproducible from the GitHub repo.
|
|
|
| ### Calibration — the point of the whole project
|
|
|
| One-hot labels **cannot** teach calibration: a Brier loss on one-hot targets is just a
|
| confidence-pushing regularizer. The training data therefore contains samples whose **true
|
| conditional distribution is known** — a known randomized rule (`explicit_rng`),
|
| marginalization over a hidden variable (`marginalized`), genuinely tied answer sets
|
| (`tie_set`), and real human disagreement from ChaosNLI (`human_annotators`).
|
|
|
| | Synthetic `test_known` (n=18,000, K≤255) | acc | ECE | Brier |
|
| |---|---|---|---|
|
| | uncalibrated | 0.647 | 0.0047 | 0.0249 |
|
| | after global temperature | 0.647 | 0.0038 | 0.0249 |
|
| | **calibration subset** (one-hot `hard` excluded, n=16,361) | **0.612** | **0.0061** | 0.0273 |
|
|
|
| **Temperature calibration is a no-op** — fitted `T = 0.965`, NLL moves 1.0629 → 1.0628.
|
| That is the intended result: the model is *natively* calibrated because its targets were
|
| distributions to begin with.
|
|
|
| ### Read this table, not the accuracy column
|
|
|
| `accuracy` pools two kinds of sample with structurally different ceilings, so it measures
|
| the provenance mix more than the model. On a `tie_set` record the target is uniform over
|
| the valid answers, so soft-accuracy is capped at `1/k` no matter how good the model is.
|
| The honest denominator is the **oracle ceiling** `mean(max_k t_k)` — computable from the
|
| data with no model at all.
|
|
|
| | generator | model | oracle ceiling | achieved |
|
| |---|---|---|---|
|
| | `tool_router` | 0.7137 | 0.7140 | **100.0%** |
|
| | `security_gate` | 0.6403 | 0.6426 | 99.6% |
|
| | `refund_policy` | 0.7561 | 0.7601 | 99.5% |
|
| | `agent_trace_score` | 0.6366 | 0.6450 | 98.7% |
|
| | `banking_balance` | 0.4544 | 0.6091 | 74.6% |
|
| | `calendar_slot` | 0.0920 | 0.1416 | 65.0% |
|
|
|
| Four of six are essentially solved. `banking_balance` is the arithmetic one and is also
|
| where calibration is worst (ECE 0.1637 vs 0.0035–0.0495 elsewhere); `calendar_slot` is the
|
| large-K one, ambiguous by construction.
|
|
|
| ### Where it fails — real text
|
|
|
| | ChaosNLI (474 items, N≈100 annotators) | |
|
| |---|---|
|
| | ECE | **0.0613** |
|
| | binomial noise floor | 0.0068 |
|
| | noise-corrected | 0.0545 |
|
|
|
| The floor explains only 11% of it. **On real human disagreement the model is genuinely
|
| miscalibrated**, and no noise correction rescues that claim.
|
|
|
| **Synthetic → real gap: 0.612 → 0.424 soft-accuracy — about 19 points lost** crossing from
|
| program-generated rules to real natural language. This is the most informative single
|
| number this project produces, reported as a limitation rather than a footnote.
|
|
|
| ### Efficiency
|
|
|
| | | |
|
| |---|---|
|
| | latency (B=1, per request) | 20.24 ms median, 27.09 ms p95 |
|
| | peak VRAM | **0.13 GB**, flat across K ∈ {2, 32, 128, 255} |
|
| | K=255 | 49.97 ms median (chunked path) |
|
|
|
| Latency is **overhead-bound, not compute-bound**: K=2 takes 19.88 ms and K=32 takes
|
| 18.37 ms — 30 extra candidates cost nothing.
|
|
|
| ---
|
|
|
| ## What it cannot do
|
|
|
| - **Match an LLM's open-domain NLU.** It is a **schema-bound decision model**, not a
|
| general assistant.
|
| - **Win on accuracy against a language model.** The claims are latency, native calibrated
|
| distributions, and **zero schema errors by construction** — not accuracy.
|
| - **Give reliable probabilities on genuinely OOD inputs.** No model does. The ChaosNLI
|
| figure above shows the degradation rather than claiming otherwise.
|
| - **Replace an LLM in any sense.** It is a component.
|
|
|
| ## License
|
|
|
| Apache-2.0 — see the [source repository](https://github.com/Colvin0315/MiniSystemOne).
|
|
|
