README: TL;DR with pairwise contrasts (fixed nested code fences)

README.md

@@ -12,52 +12,55 @@ language:
 
 # Compositional-generalization checkpoints (recall-then-assemble, Qwen3-4B-Base)
 
-Research checkpoints for interpretability work on **why a model does or does not compose skills it has
-only ever seen in isolation**. Every checkpoint below is one cell of a controlled study in which cells
-differ from each other in exactly one training variable. All weights are stored as saved by the trainer
-(`model.safetensors`, fp32, ~16 GB each; architecture `qwen3`, 4B parameters). Each subfolder is a
-complete Hugging Face model directory (config, tokenizer, chat template).
+## TL;DR
+
+15 checkpoints of Qwen3-4B-Base from **one controlled study of whether a model can compose skills it has
+only ever seen in isolation**. Every checkpoint is one cell of that study; cells differ in exactly one
+training variable, so pairs of folders isolate one effect each. One subfolder = one complete HF model
+directory (fp32 `model.safetensors` ≈ 16 GB, tokenizer, chat template).
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 import torch
-repo = "Physicsru/compgen-checkpoints"
-sub = "sft-eco-s1"                       # any folder name from the table below
+repo, sub = "Physicsru/compgen-checkpoints", "sft-eco-s1"   # any folder from the table below
 tok = AutoTokenizer.from_pretrained(repo, subfolder=sub)
 model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub, torch_dtype=torch.float32)
 ```
 
-## The task in one paragraph
-
-A library of 25 string operators with opaque names `func_0` … `func_24` (e.g. `func_3` = sort the
-characters, `func_24` = a backtracking palindrome transform with an unused `depth` parameter). The model
-first learns every operator **as an atomic skill** (closed-book: given `def main_solution(x): return
-func_3(x)` and an input, recall the body of `func_3` and write a program). Then it is trained on
-**compositions** (`func_5(func_3(func_1(x, 2)), 'ab')`) built **only from the 13 train operators**. The
-12 **held-out operators** are never composed in any training data. The test question is whether the
-model can compose the held-out operators anyway. The answer format is *recall-then-assemble* (RA):
+**Which two folders compare what** (held-out compositional accuracy at depth 4 / depth 8 unless stated):
 
-```
-Recall func_3: Sort the characters in the string.
-```python
-def func_3(s): ...
-```
-Recall func_1: Repeat the string s exactly n times.
-```python
-def func_1(s, n): ...
-```
-Assemble:
-```python
-<all recalled definitions> + def main_solution(x): return func_3(func_1(x, 2))
-```
-```
+| contrast | without | with | effect |
+|---|---|---|---|
+| **Co-occurrence atomic data** (several independent single-skill tasks in one prompt/answer, width 2-4) — cleanest version: identical task stream and steps | `sft-matched-isolated-s1` | `sft-matched-grouped-s1` | 0.65 / 0.16 → 0.93 / 0.47 |
+| **Co-occurrence atomic data** — the paper's two models (same compositions; E-co has fewer rows) | `sft-v1-s1` | `sft-eco-s1` | 0.53 / 0.09 → 0.97 / 0.73 |
+| **Composition demos (depth 2-4, train skills only)**, atomics single-task | `sft-d1-s1` | `sft-v1-s1` | 0.00 / 0.00 → 0.53 / 0.09 |
+| **Composition demos**, atomics in co-occurrence groups | `sft-c1-s1` | `sft-eco-s1` | 0.66 / 0.08 → 0.97 / 0.73 |
+| **Depth of the demos** (depth-2 only vs depth 2-4, same row count) | `sft-c4-s1` | `sft-c4b-s1` | 0.70 / 0.12 → 0.90 / 0.62 |
+| **RL** (GRPO on train-skill chains + atomic prompts; held-out skills never composed) — good outcome | `sft-eco-s1` | `rl-mdb-step100` | held-out chains d20 0.50 → 0.95 |
+| **RL** — drifted outcome (one held-out skill loses a parameter) | `sft-eco-s1` | `rl-r1a-step100` | d20 0.50 → 0.61; `func_24` arity error 25% |
+| **Seed only** (same RL pool, same init) | `rl-r1a-step100` | `rl-r1b-step100` | d20 0.61 vs 0.89; arity error 25% vs 1% |
+| **Same RL run over time** (drift switches on, then off) | `rl-mda-step5` → `rl-mda-step50` | → `rl-mda-step100` | arity error 6% → 44% → 0.3%; d20 0.82 → 0.57 → 0.93 |
+| **The recall-then-assemble format itself** | `stage15b-init` | any `sft-*` | 0.03 / 0.00 → … |
+
+The four SFT cells `sft-d1-s1`, `sft-v1-s1`, `sft-c1-s1`, `sft-eco-s1` form a 2×2 (co-occurrence no/yes ×
+demos no/yes); all SFT cells start from `stage15b-init`; all RL runs start from `sft-eco-s1`.
+
+## The task
 
-Metrics used in the tables: **held-out d4 / d8** = program accuracy on held-out-operator programs of
-expression depth 4 / 8 (256 programs per depth, `eval/heldout_trees_d1-8.parquet`); **serial d8 / d12 /
-d20** = accuracy on pure nesting chains f1(f2(…fd(x))) over held-out operators
-(`eval/heldout_chains_d1-20.parquet`); **one-parameter rate** = fraction of `Recall func_24` blocks whose
-signature drops the second parameter (`def func_24(s):`), the single failure mode that separates good and
-bad RL outcomes.
+A library of 25 string operators with opaque names `func_0` … `func_24` (e.g. `func_3` = sort the
+characters; `func_24` = a backtracking palindrome transform with a semantically unused `depth` parameter).
+The model first learns every operator **as an atomic skill** (closed-book: given `def main_solution(x):
+return func_3(x)` and an input, recall the body of `func_3` and write a program). It is then trained on
+**compositions** (`func_5(func_3(func_1(x, 2)), 'ab')`) built **only from the 13 train operators**; the 12
+**held-out operators** are never composed in any training data. The test asks the model to compose the
+held-out operators. The answer format is *recall-then-assemble* (RA): one `Recall func_N: <gloss>` block
+with the definition per operator, then one `Assemble:` block with all definitions plus `main_solution`.
+
+Metrics: **held-out d4 / d8** = program accuracy on held-out-operator programs of expression depth 4 / 8
+(256 per depth, `eval/heldout_trees_d1-8.parquet`); **chains d20** = pure nesting chains f1(f2(…f20(x)))
+over held-out operators (`eval/heldout_chains_d1-20.parquet`); **arity error** = fraction of `Recall
+func_24` blocks that drop the second parameter (`def func_24(s):`), the single failure that separates good
+and bad RL outcomes. Greedy decoding, 3,072 new tokens.
 
 ## Lineage
 
@@ -67,59 +70,56 @@ Qwen3-4B-Base
      └─ stage 1.5 "closed-book": 20k depth-1 atomic tasks over all 25 ops + 12k depth-2..4 TRAIN-op
         compositions, prose + one code block (no RA format)                      → stage15b-init
          ├─ RA bootstrap SFT cells (2 epochs, batch 128, LR 2e-5, seed 1)         → sft-*
-         │    all cells share the SAME 10,000 atomic tasks (400 per op, all 25 ops)
-         │    and differ in (a) how atomic tasks are packed into answers and (b) which
-         │    composition rows are added
+         │    all cells share the SAME 10,000 atomic tasks (400 per op, all 25 ops) and differ in
+         │    (a) how atomic tasks are packed into answers and (b) which composition rows are added
          └─ (from sft-eco-s1) GRPO, KL 0.01 to the init, 100 steps                → rl-*
 ```
 
 ## Checkpoints
 
-| folder | init | atomic side (10k single-op tasks) | composition side | RL pool | steps | held-out numbers | status |
-|---|---|---|---|---|---|---|---|
-| `stage15b-init` | stage 1 | 20k closed-book depth-1 rows, prose + one code block | 12k depth-2..4 train-op comps (same format) | — | 500 | d4 / d8 = 0.03 / 0.00 (no RA format yet) | pending |
-| `sft-d1-s1` | stage15b-init | **single-task**: one op per answer | **none** | — | 152 | 0.00 / 0.00 | uploaded |
-| `sft-v1-s1` | stage15b-init | single-task | 15,979 mixed-structure train-op comps, depth 2-4 | — | 400 | 0.53±0.15 / 0.09±0.05 (3 seeds; this seed 0.75 / 0.15) | uploaded |
-| `sft-c1-s1` | stage15b-init | **co-occurrence groups**: 1-4 independent single-op tasks in one answer (all ops) | **none** | — | 58 | 0.66±0.14 / 0.08±0.04 | uploaded |
-| `sft-eco-s1` ("E-co") | stage15b-init | co-occurrence groups | the same 15,979 comps | — | 308 | 0.97±0.02 / 0.73±0.15 (this seed 0.98 / 0.83); serial d20 0.50 | uploaded |
-| `sft-c4-s1` | stage15b-init | co-occurrence groups | 12,452 mixed comps, **depth 2 only** | — | 252 | 0.70±0.03 / 0.12±0.05 | pending |
-| `sft-c4b-s1` | stage15b-init | co-occurrence groups | 12,436 mixed comps, **depth 2-4** (same count as c4) | — | 252 | 0.90±0.04 / 0.62±0.13 | pending |
-| `sft-matched-isolated-s1` | stage15b-init | per step: 48 atomic tasks as 48 single-task rows | per step: 76 comps (same stream as grouped) | — | 400 | 0.65±0.08 / 0.16±0.02 (3 paired seeds) | pending |
-| `sft-matched-grouped-s1` | stage15b-init | per step: the SAME 48 atomic tasks packed into 20 grouped rows | the same 76 comps | — | 400 | 0.93±0.03 / 0.47±0.04 | pending |
-| `rl-r1a-step100` | sft-eco-s1 | — | — | 1:1 mix of train-op serial chains d10-20 and all-op multi-task atomic prompts (widths 1-12) | 100 | serial d8/d12/d20 0.85 / 0.63 / 0.61; func_24 one-parameter 226/903 = 25% | uploaded |
-| `rl-r1b-step100` | sft-eco-s1 | — | — | same pool, different seed | 100 | 0.94 / 0.97 / 0.89; one-parameter 9/903 = 1% | pending |
-| `rl-mda-step5` | sft-eco-s1 | — | — | **mixed-depth pool**: chains + prompts holding one deep train-op chain task followed by 1-2 long-literal atomic tasks over all ops | 5 | 0.95 / 0.92 / 0.82; one-parameter 57/903 = 6% | pending |
-| `rl-mda-step50` | sft-eco-s1 | — | — | same run | 50 | 0.91 / 0.63 / 0.57; one-parameter 396/903 = **44%** | pending |
-| `rl-mda-step100` | sft-eco-s1 | — | — | same run | 100 | 0.97 / 0.97 / 0.93; one-parameter 3/903 = 0.3% | pending |
-| `rl-mdb-step100` | sft-eco-s1 | — | — | mixed-depth pool, second replicate | 100 | 0.99 / 0.98 / 0.95; one-parameter 33/903 = 3.7% | uploaded |
-
-"mixed-structure comps" = train-op compositions whose expression trees mix unary nesting, binary
-operators and literals (the benchmark's native generator); "co-occurrence groups" put several
-*independent* atomic tasks into one prompt/answer so that each definition is recalled next to others
-(no data flow between them). Held-out operators appear in every cell only as depth-1 atomic tasks.
-
-## What each pair isolates
-
-| pair | the one difference | effect (held-out d4 / d8 unless stated) | what to look for |
-|---|---|---|---|
-| `sft-v1-s1` vs `sft-eco-s1` | atomic tasks single-task vs in co-occurrence groups (same comps; E-co has fewer rows, 308 vs 400 steps) | 0.53 / 0.09 → 0.97 / 0.73 | the headline effect: name→definition binding under multi-definition load. v1's failures are *chimeras* (a held-out name written with a train operator's body; 1,102 of 1,175 wrong bodies). Decomposed inference (each helper recalled in its own forward pass, then assembled mechanically) scores 1.00 for v1, so the whole gap is retrieval under load, not missing knowledge. |
-| `sft-matched-isolated-s1` vs `sft-matched-grouped-s1` | identical task stream, identical 400 steps, per-token loss; only the packing differs | 0.65 / 0.16 → 0.93 / 0.47 | the same effect with every confound removed (row count, steps, task order). The gap opens from step 100. |
-| `sft-d1-s1` vs `sft-v1-s1` | composition demos added (no co-occurrence) | 0.00 → 0.53 | what the demos install when atomics are single-task: d1 never produces a multi-helper program at all (format never seen with k ≥ 2). |
-| `sft-c1-s1` vs `sft-eco-s1` | composition demos added (with co-occurrence) | 0.66 / 0.08 → 0.97 / 0.73 | with co-occurrence the model composes without any demo at mid depth; the demos buy depth (d8). |
-| 2×2: `sft-d1-s1`, `sft-v1-s1`, `sft-c1-s1`, `sft-eco-s1` | co-occurrence × composition demos | 0.00 / 0.53 / 0.66 / 0.97 (d4) | interaction: co-occurrence is worth more than demos, and the two are super-additive at depth. |
-| `sft-c4-s1` vs `sft-c4b-s1` | comp depth range 2 vs 2-4 at the same row count | 0.70 / 0.12 → 0.90 / 0.62 | depth diversity of demos (not volume: 2k vs 12k depth-2 demos are equal, 0.77 vs 0.70). |
-| `stage15b-init` vs any `sft-*` | the RA format itself | 0.03 → … | what "recall then assemble" adds on top of the closed-book atomic knowledge. |
-| `sft-eco-s1` vs `rl-mdb-step100` | RL (GRPO on train-op chains + atomic prompts; held-out ops never composed) | serial d20 0.50 → 0.95 | what RL on *train-op* chains changes for *held-out* chains: deep serial execution improves for every operator. KL 0.01 keeps weights close to the init, so weight-difference analyses are meaningful. |
-| `rl-r1a-step100` vs `rl-r1b-step100` | seed only (same pool, same init) | d20 0.61 vs 0.89; func_24 one-parameter 25% vs 1% | the bistable outcome: one held-out operator (`func_24`, whose dropped parameter is semantically inert) either keeps or loses its second parameter when its call is nested among other library calls. Everything else is ≥ 0.95 in both. |
-| `rl-mda-step5` → `step50` → `step100` | same run, time only | one-parameter 6% → 44% → 0.3%; d20 0.82 → 0.57 → 0.93 | the drift switching on and off inside one run: weights change little between 50 and 100, one operator's signature flips. Compare `func_24` with its train twin `func_23` (same structure, never fails). |
-| `sft-eco-s1` vs `rl-r1a-step100` | RL, drifted outcome | d20 0.50 → 0.61 with the func_24 drift | the failure is latent in the SFT init (E-co writes `func_24(s)` in 0 / 5 / 18% of blocks at name→argument distance <40 / 40-150 / 150+ chars) and RL amplifies it in some seeds. |
-
-Things established about the failure mode that may help target an analysis: the drift tracks the number of
-*other library calls' argument tails* standing between the operator's name and its own argument in the
-skeleton (r1a, d ≥ 10: 0 tails 1%, one 15%, two 34%); builtin-method tails, long literals, definition
-load and a preceding chain in the same prompt do not trigger it (probe families 1b, 1d, 1f, 1g in `eval/`,
-which are teacher-forced prefixes usable as fixed inputs across checkpoints); the RL reward never sees the
-error in a legal (held-out-free) context, so RL acts through rewarded correct samples.
+| folder | init | atomic side (10k single-op tasks) | composition side | RL pool | steps | held-out numbers |
+|---|---|---|---|---|---|---|
+| `stage15b-init` | stage 1 | 20k closed-book depth-1 rows, prose + one code block | 12k depth-2..4 train-op comps (same format) | — | 500 | d4 / d8 = 0.03 / 0.00 (no RA format yet) |
+| `sft-d1-s1` | stage15b-init | **single-task**: one op per answer | **none** | — | 152 | 0.00 / 0.00 |
+| `sft-v1-s1` | stage15b-init | single-task | 15,979 mixed-structure train-op comps, depth 2-4 | — | 400 | 0.53±0.15 / 0.09±0.05 (3 seeds; this seed 0.75 / 0.15) |
+| `sft-c1-s1` | stage15b-init | **co-occurrence groups**: 1-4 independent single-op tasks in one answer (all ops) | **none** | — | 58 | 0.66±0.14 / 0.08±0.04 |
+| `sft-eco-s1` ("E-co") | stage15b-init | co-occurrence groups | the same 15,979 comps | — | 308 | 0.97±0.02 / 0.73±0.15 (this seed 0.98 / 0.83); chains d20 0.50 |
+| `sft-c4-s1` | stage15b-init | co-occurrence groups | 12,452 mixed comps, **depth 2 only** | — | 252 | 0.70±0.03 / 0.12±0.05 |
+| `sft-c4b-s1` | stage15b-init | co-occurrence groups | 12,436 mixed comps, **depth 2-4** (same count as c4) | — | 252 | 0.90±0.04 / 0.62±0.13 |
+| `sft-matched-isolated-s1` | stage15b-init | per step: 48 atomic tasks as 48 single-task rows | per step: 76 comps (same stream as grouped) | — | 400 | 0.65±0.08 / 0.16±0.02 (3 paired seeds) |
+| `sft-matched-grouped-s1` | stage15b-init | per step: the SAME 48 atomic tasks packed into 20 grouped rows | the same 76 comps | — | 400 | 0.93±0.03 / 0.47±0.04 |
+| `rl-r1a-step100` | sft-eco-s1 | — | — | 1:1 mix of train-op serial chains d10-20 and all-op multi-task atomic prompts (widths 1-12) | 100 | chains d8/d12/d20 0.85 / 0.63 / 0.61; arity error 226/903 = 25% |
+| `rl-r1b-step100` | sft-eco-s1 | — | — | same pool, different seed | 100 | 0.94 / 0.97 / 0.89; 9/903 = 1% |
+| `rl-mda-step5` | sft-eco-s1 | — | — | **mixed-depth pool**: chains + prompts holding one deep train-op chain task followed by 1-2 long-literal atomic tasks over all ops | 5 | 0.95 / 0.92 / 0.82; 57/903 = 6% |
+| `rl-mda-step50` | sft-eco-s1 | — | — | same run | 50 | 0.91 / 0.63 / 0.57; 396/903 = **44%** |
+| `rl-mda-step100` | sft-eco-s1 | — | — | same run | 100 | 0.97 / 0.97 / 0.93; 3/903 = 0.3% |
+| `rl-mdb-step100` | sft-eco-s1 | — | — | mixed-depth pool, second replicate | 100 | 0.99 / 0.98 / 0.95; 33/903 = 3.7% |
+
+"Mixed-structure comps" = train-op compositions whose expression trees mix unary nesting, binary operators
+and literals (the benchmark's native generator). "Co-occurrence groups" put several *independent* atomic
+tasks into one prompt/answer so each definition is recalled next to others (no data flow between them).
+Held-out operators appear in every cell only as depth-1 atomic tasks. All steps: batch 128; SFT 2 epochs.
+
+## What each contrast shows (for choosing an analysis)
+
+- **Co-occurrence** (`isolated` vs `grouped`, `v1` vs `eco`): the failure of the "without" model is a
+  *chimera* — a held-out name written with a train operator's body (1,102 of 1,175 wrong bodies in v1).
+  Decomposed inference (each helper recalled in its own forward pass, then assembled mechanically) scores
+  1.00 for v1, so the whole gap is name→definition retrieval under multi-definition load, not missing
+  knowledge. In the matched pair the gap opens from step 100.
+- **Composition demos** (`d1` vs `v1`, `c1` vs `eco`): without co-occurrence the demos are what makes any
+  multi-helper program possible (d1 never writes one); with co-occurrence the model already composes at
+  mid depth without a single demo (c1 0.66) and the demos buy depth (d8 0.08 → 0.73). Demo *diversity*
+  matters (c4 vs c4b), demo *volume* does not (2k vs 12k depth-2 demos: 0.77 vs 0.70).
+- **RL** (`eco` vs `mdb` / `r1a`, `r1a` vs `r1b`, `mda` over time): RL on train-operator chains lifts deep
+  serial execution for every operator (d20 0.50 → 0.9+), but in some seeds one held-out operator
+  (`func_24`, whose dropped parameter is semantically inert) loses its second parameter when its call is
+  nested among other library calls. The drift is latent in the SFT init (E-co writes `func_24(s)` in
+  0 / 5 / 18% of blocks at name→argument distance <40 / 40-150 / 150+ chars), tracks the number of other
+  library calls' argument tails between the name and the own argument, is not triggered by builtin-method
+  tails, long literals, definition load or a preceding chain (probe families 1b/1d/1f/1g in `eval/`), and
+  reverses inside one run (`mda` 44% → 0.3%). KL 0.01 keeps RL weights close to `sft-eco-s1`, so
+  weight-difference analyses between `eco` and any `rl-*` folder, or between `mda` steps, are meaningful.
 
 ## Evaluation inputs (`eval/`)
 
@@ -134,13 +134,11 @@ error in a legal (held-out-free) context, so RL acts through rewarded correct sa
 | `probe_1f.parquet` | 2,048 | builtin-method argument tails in front of the op's own argument |
 | `probe_1g.parquet` | 1,280 | long literal (8-200 chars) as the op's first argument |
 
-Rows are in the verl code-execution format: `prompt` (chat messages), `reward_model.ground_truth`
-(JSON with `ref_code`, `ref_input`, `ref_output`), `extra_info` (depth, k = distinct ops, probe
-condition). Greedy decoding, 3,072 new tokens, was used for every number above.
+Rows are in the verl code-execution format: `prompt` (chat messages), `reward_model.ground_truth` (JSON
+with `ref_code`, `ref_input`, `ref_output`), `extra_info` (depth, k = distinct ops, probe condition).
 
 ## Provenance
 
-Every number here is traceable in the project ledger `verl/analysis/RESULTS_PROVENANCE.md` (sections MAIN
+Every number is traceable in the project ledger `verl/analysis/RESULTS_PROVENANCE.md` (sections MAIN
 RESULT, C1–C5 CAUSAL TABLE, MATCHED SFT, DEEP CHAINS, Mixed-depth arm RESULT, PROBES) to a job id, a
-sweep directory and a CI/classification report; 3-seed bands are mean ± sd over trainer seeds 1 / 7 / 123.
-Checkpoints marked *pending* in the table are not yet uploaded (storage-quota decision pending).
+sweep directory and a CI / classification report; 3-seed bands are mean ± sd over trainer seeds 1 / 7 / 123.