README.md

---

language:
- en
- zh
license: apache-2.0
tags:
- decision-model
- calibration
- system-one
- from-scratch
- probabilistic-decisions
- pytorch
pretty_name: MiniSystemOne
---


# MiniSystemOne

**Train a probabilistic decision model from scratch — no LLM, no decoding, no JSON generation.**

A 26.89M-parameter model that takes a **state** plus **typed questions** and returns
**typed decisions with calibrated probabilities** — in a single parallel forward pass.
No autoregressive loop, no text output, no constrained-decoding tricks.

Built from **random initialization**, MiniMind-style: one shared bidirectional encoder,
one decision head, one loss. Trained in **4.7 hours** on a single RTX 4070 Laptop (8 GB).

> ### ⚠️ This is not a Jev reproduction
>

> It is an **independent educational implementation**, inspired by the "System One Model"
> idea popularized by TypeSafe AI's Jev (2026-09-15) — the way MiniMind is an educational
> reimplementation of the LLaMA/GPT recipe rather than a reproduction of any model.
> No Jev output was used in training, and no attempt was made to reverse-engineer it.
> It does **not** claim to beat Jev on any axis.

---

## Files

| File | Stage | Params | What it is |
|---|---|---|---|
| `decision.pth` | 2 | 26.89M | **The decision model.** The artifact this repo is about. |
| `mlm.pth` | 1 | 25.31M | The MLM-pretrained encoder. `decision.pth` was warm-started from it. |
| `tokenizer.json` / `tokenizer_config.json` | — | vocab 6400 | A **new** BPE, 11 special tokens. Not MiniMind's. |

Both checkpoints are **self-describing**: the file carries the config it was built from
and the provenance of its training data.

```

decision.pth  format=1  stage=decision  step=33795  n_params=26889729

              tokenizer_sha1=bf7a131a109ea445  gen_version=1.0.0

              encoder_init=mlm.pth  trained_on=synth  max_len=1024  epochs=3



mlm.pth       format=1  stage=mlm  step=66000  n_params=25309184

              tokenizer_sha1=bf7a131a109ea445  synth_gen_version=1.0.0

              max_len=512  epochs=8

```

---

## How to use these weights

**There is no `from_pretrained` path, on purpose.** This is a decision-native architecture

with a custom input packing format, not a causal LM. The packing is canonical and lives in

exactly one place in the source repository; re-implementing it here would create a second

implementation that drifts from the first.



The code is at **`github.com/Colvin0315/MiniSystemOne`** (Apache-2.0).



```bash

git clone https://github.com/Colvin0315/MiniSystemOne && cd MiniSystemOne

pip install -r requirements.txt          # torch 2.6.0+cu124, transformers, datasets



# put decision.pth + tokenizer.json + tokenizer_config.json into model/ and out/decision/

python eval/eval_harness.py --ckpt out/decision/decision.pth \

    --data dataset/synth --sets test_known

```



Loading a checkpoint is one call, and it reads the embedded config and refuses a mismatch:



```python

from trainer.trainer_utils import ckpt_info, init_model, verify_tokenizer

from model.model_system_one import DecisionConfig, MiniSystemOneForDecision



verify_tokenizer("decision.pth", "model")        # rejects a mismatched vocabulary

cfg = DecisionConfig(**{k: v for k, v in ckpt_info("decision.pth")["config"].items()

                        if k in ("hidden_size", "num_hidden_layers", "vocab_size",

                                 "num_attention_heads", "num_key_value_heads",

                                 "intermediate_size")})

model = init_model(MiniSystemOneForDecision, cfg, "decision.pth", "cuda")

```



---



## Results



All numbers are from `decision.pth`, reproducible from the GitHub repo.



### Calibration — the point of the whole project



One-hot labels **cannot** teach calibration: a Brier loss on one-hot targets is just a

confidence-pushing regularizer. The training data therefore contains samples whose **true

conditional distribution is known** — a known randomized rule (`explicit_rng`),

marginalization over a hidden variable (`marginalized`), genuinely tied answer sets

(`tie_set`), and real human disagreement from ChaosNLI (`human_annotators`).



| Synthetic `test_known` (n=18,000, K≤255) | acc | ECE | Brier |

|---|---|---|---|

| uncalibrated | 0.647 | 0.0047 | 0.0249 |

| after global temperature | 0.647 | 0.0038 | 0.0249 |

| **calibration subset** (one-hot `hard` excluded, n=16,361) | **0.612** | **0.0061** | 0.0273 |



**Temperature calibration is a no-op** — fitted `T = 0.965`, NLL moves 1.0629 → 1.0628.

That is the intended result: the model is *natively* calibrated because its targets were

distributions to begin with.



### Read this table, not the accuracy column



`accuracy` pools two kinds of sample with structurally different ceilings, so it measures

the provenance mix more than the model. On a `tie_set` record the target is uniform over

the valid answers, so soft-accuracy is capped at `1/k` no matter how good the model is.

The honest denominator is the **oracle ceiling** `mean(max_k t_k)` — computable from the
data with no model at all.

| generator | model | oracle ceiling | achieved |
|---|---|---|---|
| `tool_router` | 0.7137 | 0.7140 | **100.0%** |
| `security_gate` | 0.6403 | 0.6426 | 99.6% |
| `refund_policy` | 0.7561 | 0.7601 | 99.5% |
| `agent_trace_score` | 0.6366 | 0.6450 | 98.7% |
| `banking_balance` | 0.4544 | 0.6091 | 74.6% |
| `calendar_slot` | 0.0920 | 0.1416 | 65.0% |

Four of six are essentially solved. `banking_balance` is the arithmetic one and is also
where calibration is worst (ECE 0.1637 vs 0.0035–0.0495 elsewhere); `calendar_slot` is the
large-K one, ambiguous by construction.

### Where it fails — real text

| ChaosNLI (474 items, N≈100 annotators) | |
|---|---|
| ECE | **0.0613** |
| binomial noise floor | 0.0068 |
| noise-corrected | 0.0545 |

The floor explains only 11% of it. **On real human disagreement the model is genuinely

miscalibrated**, and no noise correction rescues that claim.

**Synthetic → real gap: 0.612 → 0.424 soft-accuracy — about 19 points lost** crossing from
program-generated rules to real natural language. This is the most informative single
number this project produces, reported as a limitation rather than a footnote.

### Efficiency

| | |
|---|---|
| latency (B=1, per request) | 20.24 ms median, 27.09 ms p95 |
| peak VRAM | **0.13 GB**, flat across K ∈ {2, 32, 128, 255} |
| K=255 | 49.97 ms median (chunked path) |

Latency is **overhead-bound, not compute-bound**: K=2 takes 19.88 ms and K=32 takes
18.37 ms — 30 extra candidates cost nothing.

---

## What it cannot do

- **Match an LLM's open-domain NLU.** It is a **schema-bound decision model**, not a
  general assistant.
- **Win on accuracy against a language model.** The claims are latency, native calibrated
  distributions, and **zero schema errors by construction** — not accuracy.
- **Give reliable probabilities on genuinely OOD inputs.** No model does. The ChaosNLI
  figure above shows the degradation rather than claiming otherwise.
- **Replace an LLM in any sense.** It is a component.

## License

Apache-2.0 — see the [source repository](https://github.com/Colvin0315/MiniSystemOne).