# NCP rebuttal ablations

Four ablation studies on the Next-Concept Predictor. Code, results tables and
figures: **github.com/HazyResearch/ncm**, branch `rebuttals-schedule-ablation`
(see `results/SCHEDULE_ABLATION.md`).

## Layout

```
<family>/<variant>/<stage>/checkpoint-iter-<N>.pt
                          /training-config.yaml
                          /results.txt          (per-seed gen-PPL + entropy)
                          /launch-record.txt    (git commit, GPUs, batch, codec sha256)
```

`family` is the level ladder, `variant` the NCP-side change on top of it.
A `-50k` suffix means the run used a **50k-step** VQ-VAE codec; folders without it
used the earlier **10k-step** codecs. **Never compare gen-PPL across the two codec
generations** — the same baseline scored 114.5 on the 10k codec and 31.8 on the 50k
one. Codec quality dominates raw gen-PPL.

An NCP checkpoint cannot be used without the codec it was trained against. Each
`<family>/<variant>/vqvae/` holds that codec.

## The studies

| study | families / variants | budget |
|---|---|---|
| 1. level schedule | `linear16` `squared16` `log16` `exp16` → `baseline-50k` | 5k NCP steps |
| 2. corruption strength | `squared16/corruption-p{10,20,30}-50k` | 5k NCP steps |
| 3. residual refiner | `squared16/refiner-50k` | 5k NCP steps |
| 4. number of levels | `sq8` `squared16` `sq24` → `baseline-50k` | 5k NCP steps |

`squared16/baseline-50k` is the shared control for studies 2, 3 and 4.
`linear16/{corruption-*,refiner}` are an earlier **10k-codec, 2k-step** study,
superseded but retained.

## Headline

Raw gen-PPL is not interpretable in this setting: across runs it correlates with
generation entropy at r = +0.79, so it rewards low-diversity output. All
conclusions use gen-PPL at **matched entropy 5.445 nats**, the measured entropy of
real OpenWebText text under the same metric. Reaching that target requires
disabling truncation (`top_k=0`, `top_p=1.0`), so those numbers are a separate,
internally-consistent comparison flagged `frozen_protocol=false` in the records.

- **Level schedule:** squared wins by ~48 points; corroborated on an independent
  codec generation at a different step budget.
- **Corruption strength:** null (≤1.3σ). Raw gen-PPL improves monotonically to
  −34% but entirely by trading diversity.
- **Residual refiner:** null (0.1σ).
- **Number of levels:** 16 matches 8, beats 24 by 69 points. The 24-level deficit
  is *unattributed* between level count and convergence — see the caveats in
  `SCHEDULE_ABLATION.md`.

## Known gap

`exp16/baseline/vqvae`, `log16/baseline/vqvae` and `squared16/baseline/vqvae` have
no `training-config.yaml`. Those are the 10k-step codecs, and the config files on
disk were later edited in place to 50k, so uploading them now would put a
`stop_after_iters: 50000` config beside a 10k checkpoint. Recover the originals
from git history instead: the 10k-era configs are in
`config/ablation_sweep/vqvae-*.yaml` prior to commit `45f34d1`.