File size: 11,387 Bytes
05f5056
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
# Architecture

Five layers, strictly one-directional. Nothing below a layer imports from above it.

```
                       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                       β”‚        scripts/             β”‚  smoke.py, e2e.py
                       β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                      β”‚
  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚                                   β”‚                                   β”‚
β”Œβ”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”            β”Œβ”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”                β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  rsi.py        β”‚            β”‚    train.py     β”‚                β”‚     eval.py       β”‚
β”‚  accept/revert │◄────────────  loop, ckpt     │───────────────►│ score, verify,    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  scores    β”‚  resume, CLI    β”‚  model+tokenizerβ”‚ gate, repair      β”‚
                                   β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜                β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                           β”‚                                 β”‚
                              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                   β”‚
                              β”‚        model.py          β”‚β—„β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              β”‚  GQA, RoPE, RMSNorm,    β”‚
                              β”‚  SwiGLU, MiniTransformerβ”‚
                              β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                           β”‚
  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚                          data.py        β”‚        config.py       β”‚
  β”‚        tokenizer, manifest, streams     β”‚   frozen config objects β”‚
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

  tools.py  β€”  orthogonal: called by eval (tool scoring) and by any agent runtime
```

`tools.py` is deliberately a leaf. The model never imports it; only the evaluation loop and external callers do. That keeps the sandbox auditable in isolation.

---

## Module contracts

### `config.py` β€” validation at the boundary

Four frozen dataclasses: `ModelConfig`, `TrainingConfig`, `EvalConfig`, `SafetyConfig`. Every one validates itself in `__post_init__`, so an invalid configuration cannot be constructed β€” it raises at construction, not deep inside a training step.

Cross-field invariants that are enforced (and would otherwise be silent bugs):

- `num_query_heads % num_kv_heads == 0` β€” GQA cannot broadcast otherwise.
- `d_model % num_query_heads == 0` β€” head_dim must be integral.
- `head_dim % 2 == 0` β€” RoPE rotates pairs of dimensions.
- `max_steps > warmup_steps` β€” the LR schedule would divide by zero.
- `precision == "fp32"` and `gradient_checkpointing is True` β€” the CPU contract is not negotiable.
- `schedule == "cosine"` β€” the only implemented schedule.

`ModelConfig` also owns the **parameter accounting** (`parameter_count_tied` / `parameter_count_untied`) as a closed-form formula, so the number in the README is derived, not measured. `scripts/smoke.py` asserts it against the constructed model.

### `model.py` β€” the network

`RMSNorm` β†’ `GroupedQueryAttention` β†’ `SwiGLU` β†’ `DecoderBlock` β†’ `MiniTransformer`.

`MiniTransformer.forward` returns logits directly: with tied embeddings the output projection is `hidden @ E.T` and `self.output_projection is None`. There is no bias anywhere in the projections.

Attention is computed in **query blocks of 128** so the score matrix is never materialised for the full sequence. This is the single most important memory decision in the model and the reason a 8,192-token context fits the inference budget.

Two guards matter for numerical safety:

- rows where **no** key is allowed (fully padded) would make `softmax` produce `NaN`; they are zeroed before the softmax and re-masked after it.
- masked scores are filled with `torch.finfo(dtype).min`, not `-inf`, so a `NaN` in one block cannot propagate.

`generate` is a straightforward autoregressive loop with a sliding window of `config.context_length`; it extends the attention mask by one position per step and stops as soon as any sequence in the batch emits `eos_id`. There is no KV cache β€” see [THEORY.md](THEORY.md#inference-memory) for what that means for the memory estimate.

`estimate_inference_memory_bytes()` is an **analytic** upper bound: weights + KV + hidden + one attention block + next-token logits. It is a budget-checking tool, not a measurement. See [OPERATIONS.md](OPERATIONS.md#verifying-the-memory-budget).

### `data.py` β€” bytes in, token streams out

`SimpleTokenizer` is deliberately a byte-level tokenizer: 6 reserved ids (`pad`, `eos`, `unk`, `tool`, `python`, `shell`) and one id per byte. It is not a good tokenizer β€” it is a *reproducible* one, which matters more at this stage, and it makes the whole pipeline Unicode-safe with no vocab file to ship.

`build_train_stream` produces fixed-length token windows from a list of examples, cycling the source, and accepts `start_token` so training can resume at an exact token offset. `build_eval_stream` pads to a fixed length instead.

`validate_manifest` enforces the provenance contract: a source may only be `enabled: true` if it has a 64-hex SHA-256 and a positive token budget; a disabled source must have `token_budget: 0`. URLs must be http(s).

### `train.py` β€” the loop

`train_model` owns the whole lifecycle:

1. move the model to the device, enable gradient checkpointing, wrap in `nn.DataParallel` when more than one CUDA device is visible;
2. build the optimizer;
3. optionally restore a checkpoint (weights, optimizer, step, token cursor, RNG states);
4. iterate the token stream, accumulating 8 sequences of 1024 tokens into one optimizer step;
5. checkpoint on the token grid, emit structured progress events, return `{step, tokens_seen, last_loss}`.

All telemetry is JSON Lines on **stderr** (`train_start`, `gpu_ready`, `train_progress`, `checkpoint_saved`, `train_done`), leaving stdout for the final result object. That split is what makes the run greppable in a Kaggle/Colab log.

### `eval.py` β€” score, verify, gate

`evaluate_model` moves the model to CPU, switches to eval mode, and per example:

- **math / coding** β€” generate, then run `VerifierRepairLoop(max_repairs=1)`: verify the answer, and on failure feed a feedback continuation back through the model once. A score counts only if the logits were finite *and* the verifier passed.
- **tool** β€” hand the parsed tool call to the executor; the score is the executor's `ok`.

The training/eval mode flag of the model is restored in a `finally` block, so an exception mid-eval cannot leave the model stuck in eval mode.

`strict_gate` returns an `EvalGateResult` that is truthy, so `if strict_gate(scores):` reads naturally.

### `rsi.py` β€” the policy

`rsi_accept` is a pure function over two `EvalScores`. It is the only place that decides whether a change is allowed to survive. `RSIController` adds apply/accept/revert state; `run_rsi_cycle` wraps a full experiment and **always reverts in a `finally`**, so a crashed experiment cannot leave a rejected candidate installed.

### `tools.py` β€” protocol and sandbox

`build_envelope` / `parse_envelope` define a canonical JSON envelope (`sort_keys`, compact separators) with protected `tool` / `request_id` fields, so a tool can never spoof the envelope's own metadata. `parse_tool_call` normalises a call into `{"kind", "payload"}` with `kind ∈ {python, shell}`.

Execution is described in [SECURITY.md](SECURITY.md).

---

## Checkpoint format

A single `torch.save` payload:

```python
{
  "model":     state_dict,          # device-agnostic, always saved from the unwrapped module
  "optimizer": optimizer.state_dict(),
  "tokens_seen": int,
  "step":       int,
  "rng_state":  {"python": ..., "torch": ..., "cuda": [...]},
  "extra":      {"last_loss": float, "next_checkpoint_at": int},
  "data_state": {"cursor": int},     # must equal tokens_seen, or loading raises
}
```

Properties that matter operationally:

- **Atomic.** Written to `.{name}.{pid}.tmp` and moved into place with `os.replace`, so a kill mid-write cannot corrupt the previous checkpoint.
- **Device-agnostic.** `load_checkpoint` always maps to CPU and transparently adds or strips the `module.` prefix that `nn.DataParallel` introduces, so a CPU checkpoint resumes on a GPU run and vice versa.
- **Verifiable cursor.** A checkpoint whose `data_state.cursor` disagrees with `tokens_seen` is rejected rather than silently resuming at the wrong position in the corpus.

## Data flow

```
manifest.json ──validate_manifest──► allowed sources
        β”‚
examples ──encode──► token ids ──build_train_stream(start_token=…)──► [1024] windows
                                                                              β”‚
                                                       β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                       β–Ό
   [1,1023] inputs + [1,1023] labels ──► forward ──► cross_entropy ──► backward
                                                       β”‚
                          accumulate until 8192 tokens ───
                                                       β–Ό
                          set LR ─► unscale grads ─► clip(1.0) ─► AdamW.step
                                                       β”‚
                                       every 5M tokens ─► save_checkpoint (atomic)
```

## Extension points

| To add… | Do this | Not this |
| --- | --- | --- |
| a corpus | add a pinned entry to `data/manifest.json` with sha256 + budget, set `enabled: true` | dropping files into the repo |
| a tokenizer | implement `encode(text, add_eos=True) -> list[int]` and `decode(ids) -> str` | editing the training loop |
| a model change | subclass or edit `model.py`; re-run `scripts/smoke.py` so the parameter-count assertion still holds | bypassing `ModelConfig` |
| a new domain | add a branch in `evaluate_model` and a field on `EvalScores` + `EvalConfig` | adding an unvalidated score |
| a new tool | extend `parse_tool_call` + `execute_tool_call` and add it to the allowlists | calling `subprocess` directly from eval |
| an RSI experiment | call `run_rsi_cycle` with apply/evaluate/revert callbacks | mutating the baseline in place |