| # NCP rebuttal ablations |
|
|
| Four ablation studies on the Next-Concept Predictor. Code, results tables and |
| figures: **github.com/HazyResearch/ncm**, branch `rebuttals-schedule-ablation` |
| (see `results/SCHEDULE_ABLATION.md`). |
|
|
| ## Layout |
|
|
| ``` |
| <family>/<variant>/<stage>/checkpoint-iter-<N>.pt |
| /training-config.yaml |
| /results.txt (per-seed gen-PPL + entropy) |
| /launch-record.txt (git commit, GPUs, batch, codec sha256) |
| ``` |
|
|
| `family` is the level ladder, `variant` the NCP-side change on top of it. |
| A `-50k` suffix means the run used a **50k-step** VQ-VAE codec; folders without it |
| used the earlier **10k-step** codecs. **Never compare gen-PPL across the two codec |
| generations** — the same baseline scored 114.5 on the 10k codec and 31.8 on the 50k |
| one. Codec quality dominates raw gen-PPL. |
|
|
| An NCP checkpoint cannot be used without the codec it was trained against. Each |
| `<family>/<variant>/vqvae/` holds that codec. |
|
|
| ## The studies |
|
|
| | study | families / variants | budget | |
| |---|---|---| |
| | 1. level schedule | `linear16` `squared16` `log16` `exp16` → `baseline-50k` | 5k NCP steps | |
| | 2. corruption strength | `squared16/corruption-p{10,20,30}-50k` | 5k NCP steps | |
| | 3. residual refiner | `squared16/refiner-50k` | 5k NCP steps | |
| | 4. number of levels | `sq8` `squared16` `sq24` → `baseline-50k` | 5k NCP steps | |
|
|
| `squared16/baseline-50k` is the shared control for studies 2, 3 and 4. |
| `linear16/{corruption-*,refiner}` are an earlier **10k-codec, 2k-step** study, |
| superseded but retained. |
|
|
| ## Headline |
|
|
| Raw gen-PPL is not interpretable in this setting: across runs it correlates with |
| generation entropy at r = +0.79, so it rewards low-diversity output. All |
| conclusions use gen-PPL at **matched entropy 5.445 nats**, the measured entropy of |
| real OpenWebText text under the same metric. Reaching that target requires |
| disabling truncation (`top_k=0`, `top_p=1.0`), so those numbers are a separate, |
| internally-consistent comparison flagged `frozen_protocol=false` in the records. |
|
|
| - **Level schedule:** squared wins by ~48 points; corroborated on an independent |
| codec generation at a different step budget. |
| - **Corruption strength:** null (≤1.3σ). Raw gen-PPL improves monotonically to |
| −34% but entirely by trading diversity. |
| - **Residual refiner:** null (0.1σ). |
| - **Number of levels:** 16 matches 8, beats 24 by 69 points. The 24-level deficit |
| is *unattributed* between level count and convergence — see the caveats in |
| `SCHEDULE_ABLATION.md`. |
|
|
| ## Known gap |
|
|
| `exp16/baseline/vqvae`, `log16/baseline/vqvae` and `squared16/baseline/vqvae` have |
| no `training-config.yaml`. Those are the 10k-step codecs, and the config files on |
| disk were later edited in place to 50k, so uploading them now would put a |
| `stop_after_iters: 50000` config beside a 10k checkpoint. Recover the originals |
| from git history instead: the 10k-era configs are in |
| `config/ablation_sweep/vqvae-*.yaml` prior to commit `45f34d1`. |
|
|
