Preserve machine source, configs, and experiment records before checkpoint archival
Browse files
machine-backups/research-backup-20260924/git-and-environment.json
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"/home/ishank/mstok": {
|
| 3 |
+
"head": "2c52c050d382c9f0f7e1e74bcce98974be9124ea\n",
|
| 4 |
+
"status": " D best_checkpoints/b1024s1-EMA-33.85/README.md\n D best_checkpoints/b1024s1-EMA-33.85/SHA256SUMS.txt\n D best_checkpoints/b1024s1-EMA-33.85/chain_s1_b1024_ema.sh\n D best_checkpoints/b1024s1-EMA-33.85/chain_stage2_b1024s1.sh\n D best_checkpoints/b1024s1-EMA-33.85/ncp-s1-p95-b1024.yaml\n D best_checkpoints/b1024s1-EMA-33.85/ncp-stage2-b1024s1.yaml\n D best_checkpoints/b1024s1-EMA-33.85/vq-pass80-ema095-120k.yaml\n D best_checkpoints/ema50k-EMA-33.33/ncp-pretrain-pass80.yaml\n D best_checkpoints/ema50k-EMA-33.33/ncp-stage2-p80-ema50k.yaml\n D best_checkpoints/ema50k-EMA-33.33/vq-pass80-ema092.yaml\n D best_checkpoints/pretrain-pass80-30k/README.md\n D best_checkpoints/pretrain-pass80-30k/ncp-pretrain-pass80.yaml\n D best_checkpoints/pretrain-pass80-30k/vq-pass80-ema092.yaml\n D best_checkpoints/stage2-cfg-partial/ncp-stage2-cfg.yaml\n D best_checkpoints/stage2-cfg-partial/test_cfg_correct.py\n D best_checkpoints/stage2-p80-32.75/ncp-pretrain-pass80.yaml\n D best_checkpoints/stage2-p80-32.75/ncp-stage2-pass80.yaml\n D best_checkpoints/stage2-p80-32.75/vq-pass80-ema092.yaml\n M models/quant.py\n?? config/mstok-matched-xsum-gentle-codec.yaml\n?? config/mstok-matched-xsum-pretrain.yaml\n?? config/mstok-matched-xsum.yaml\n?? configs/\n?? data/matched_xsum_pretrain.py\n?? evaluation/decoding_penalties.py\n?? evaluation/xsum_mbr.py\n?? reproduce/mstok-matched-xsum-gentle-codec.md\n?? reproduce/mstok-matched-xsum-joint-pretrain.md\n?? reproduce/mstok-matched-xsum.md\n?? scripts/archive_machine_research.py\n?? scripts/audit_matched_xsum_restore.py\n?? scripts/backup_matched_xsum.py\n?? scripts/check_matched_xsum_resume.py\n?? scripts/evaluate_matched_xsum.py\n?? scripts/evaluate_matched_xsum_pretrain.py\n?? scripts/evaluate_matched_xsum_recipes.py\n?? scripts/launch/\n?? scripts/launch_matched_xsum.py\n?? scripts/launch_matched_xsum_joint_pipeline.py\n?? scripts/launch_matched_xsum_recipes.py\n?? scripts/prepare_matched_xsum_pretrain.py\n?? scripts/qualify_gentle_codec_run.py\n?? scripts/qualify_matched_xsum.py\n?? scripts/queue_matched_xsum_final_penalty.py\n?? scripts/sweep_matched_xsum_penalties.py\n?? scripts/train_matched_xsum.py\n?? src/\n?? tests/test_decoding_penalties.py\n?? tests/test_matched_xsum.py\n?? tests/test_matched_xsum_cuda.py\n?? tests/test_matched_xsum_gentle_codec.py\n?? tests/test_matched_xsum_pretrain.py\n?? tests/test_xsum_mbr.py\n?? tests/text/\n?? trainer/matched_xsum.py\n",
|
| 5 |
+
"diff": "diff --git a/best_checkpoints/b1024s1-EMA-33.85/README.md b/best_checkpoints/b1024s1-EMA-33.85/README.md\ndeleted file mode 100644\nindex 6bb9fe9..0000000\n--- a/best_checkpoints/b1024s1-EMA-33.85/README.md\n+++ /dev/null\n@@ -1,79 +0,0 @@\n-# XSum best result \u2014 R-1 33.85 (2026-08-21)\n-\n-Best result of the campaign. Beats the previous best (33.33) on all three ROUGE\n-metrics and on both repetition metrics.\n-\n-Everything here is a **real copy** (`links=1`), not a hardlink into `experiments/`,\n-and is `chmod -w`. Deleting or thinning the experiment directories cannot touch it.\n-Verify integrity with `sha256sum -c SHA256SUMS.txt`.\n-\n-## Result \u2014 full XSum test set, 11,233 documents\n-\n-| checkpoint | R-1 | R-2 | R-L | adjdupX | rep1X |\n-| --- | --- | --- | --- | --- | --- |\n-| **ema-iter-50002.pt** | **33.85** | **12.01** | **27.02** | 2.06 | 9.18 |\n-| raw-checkpoint-iter-50002.pt | 33.33 | 11.74 | 26.62 | 2.15 | 8.98 |\n-| *previous best (33.33 run)* | 33.33 | 11.69 | 26.46 | 2.42 | 9.53 |\n-| *e12 baseline* | 32.20 | \u2014 | \u2014 | \u2014 | \u2014 |\n-\n-oracle-L0 ceiling 42.49 (raw 41.91); level-0 exact-match 3.23%.\n-\n-Reference points measured the same day, same 11,233 docs, same metric code:\n-ELF-B no-CFG **34.16 / 11.13 / 26.42** at 1,301 s vs our 174 s; ELF at 8 ODE steps\n-(\u2248 our cost) **31.63 / 9.39 / 24.68**. ELF paper's reproduced MDLM: 33.4 / 11.6 / 25.8.\n-\n-## Files\n-\n-| file | what |\n-| --- | --- |\n-| `ema-iter-50002.pt` | **the 33.85 model** \u2014 stage-2 weight-EMA (decay 0.9999) |\n-| `raw-checkpoint-iter-50002.pt` | stage-2 raw final (33.33); carries optimizer state |\n-| `stage1-ema-iter-88000.pt` | the stage-1 EMA this was warm-started from |\n-| `codec.pt` | VQ-VAE `pass80-ema095` (120k steps, EMA 0.95, no world scaling) |\n-| `ncp-s1-p95-b1024.yaml` | stage-1 config |\n-| `ncp-stage2-b1024s1.yaml` | stage-2 config |\n-| `vq-pass80-ema095-120k.yaml` | codec config |\n-| `chain_s1_b1024_ema.sh`, `chain_stage2_b1024s1.sh` | the runnable chains |\n-\n-## Pipeline\n-\n-1. **Codec** `pass80-ema095`: 80/20 passage/summary mix, codebook EMA 0.95, no world\n- scaling, 120k steps. Ladder `[4,6,10,14,20,26,32,40,48]`, 9 levels, 4096 codes each.\n-2. **Stage 1** (unconditional `p(passage codes)`): 88,000 steps @ global batch 1024,\n- lr 5e-4 \u2192 1e-5, `prefix_len=1`, weight EMA 0.9999 enabled from step 20,000.\n- Draws **90.1M windows from the full 89.7M-window pool on the fly**\n- (`NullPrefixWindowDataset`) \u2014 the previous run recycled a materialised 6.35%\n- subset and burned 91% of its compute on PAD. Final val loss 2.779 (prev 2.956).\n-3. **Stage 2** (passage \u2192 summary): warm-start from `stage1-ema-iter-88000.pt`,\n- 50,002 steps @ global batch 512, lr 3e-4 \u2192 3e-5, weight EMA 0.9999, **no CFG**.\n- Identical to the 33.33 recipe except the stage-1 init and the codec.\n-\n-## Caveats worth carrying forward\n-\n-- **The gain is confounded.** Stage 1 changed four ways at once (2.93\u00d7 data, full\n- window pool, no PAD compute, weight EMA) *and* the codec changed. The isolating\n- experiment is a stage-2 run from the **30,000-step** stage-1 checkpoint (~2.5 h).\n-- **Single run per side.** 26/26 paired validation-checkpoint wins (mean +0.75 R-1)\n- make the effect credible, but run-to-run variance was never measured.\n-- **Repetition did not really improve.** Stage-1 unconditional rep1X fell to ~0.5,\n- yet conditional summaries only went 9.53 \u2192 9.18 \u2014 still **~32\u00d7 ELF's 0.29**.\n- Unconditional repetition is not a proxy for the conditional metric.\n-- Warm-starting across a `prefix_len` change needs the shape-mismatch filter in\n- `utils/misc.py::load_checkpoint`; `strict=False` alone does **not** skip it.\n-\n-## Reproduce the test number\n-\n-```bash\n-cd /workspace/ncm && source /venv/main/bin/activate\n-for k in 0 1 2 3 4 5 6 7; do\n- CUDA_VISIBLE_DEVICES=$k python scripts/xsum/eval_shard.py \\\n- --ncp best_checkpoints/b1024s1-EMA-33.85/ema-iter-50002.pt \\\n- --split test --n 11233 --bs 32 --shard $k --nshards 8 --out /tmp/sh/shard_$k.pkl &\n-done; wait\n-python scripts/xsum/eval_merge.py --glob \"/tmp/sh/shard_*.pkl\"\n-```\n-\n-Full logs: `campaign/stage2_b1024s1.log` (test), `campaign/b1024s1_val.log`\n-(26 val checkpoints), `campaign/s1b1024_rep.log` (stage-1 repetition).\n-All 55 stage-2 and 33 stage-1 checkpoints are hardlinked in\n-`best_checkpoints/stage2-b1024s1-all/` and `best_checkpoints/s1-p95-b1024-all/`.\ndiff --git a/best_checkpoints/b1024s1-EMA-33.85/SHA256SUMS.txt b/best_checkpoints/b1024s1-EMA-33.85/SHA256SUMS.txt\ndeleted file mode 100644\nindex 12edf58..0000000\n--- a/best_checkpoints/b1024s1-EMA-33.85/SHA256SUMS.txt\n+++ /dev/null\n@@ -1,4 +0,0 @@\n-a5c86cba7fd8da6ae379200a63df6ddbd5c13075e6b9a5a1b297411b33e5c5c7 codec.pt\n-e3e32c81cde04387744fd9ea2e8c3efb2165b83c24dcc0263be63223258e84e7 ema-iter-50002.pt\n-b88a80ca7009e7c34c3f65f8a1d8b147f7e2ae628cdc1327268049a63f6bd0da raw-checkpoint-iter-50002.pt\n-86aab2cee91f4645c64f0c3baf8fd3d50c042d4ca4aa2a8e53099a56262cbc21 stage1-ema-iter-88000.pt\ndiff --git a/best_checkpoints/b1024s1-EMA-33.85/chain_s1_b1024_ema.sh b/best_checkpoints/b1024s1-EMA-33.85/chain_s1_b1024_ema.sh\ndeleted file mode 100644\nindex de26561..0000000\n--- a/best_checkpoints/b1024s1-EMA-33.85/chain_s1_b1024_ema.sh\n+++ /dev/null\n@@ -1,22 +0,0 @@\n-#!/bin/bash\n-# STAGE 1 RESUME from step 20,000 with weight EMA enabled (decay 0.9999).\n-# NO rm -rf: this resumes into the existing experiment dir. reset_lr=false so the\n-# cosine schedule continues from step 20,000 rather than re-warming.\n-# Starting EMA at 20k of 88k is ~equivalent to starting at 0: the horizon is\n-# 10,000 steps, so step-20k weights carry 0.9999^68000 ~ 0.001 in the final EMA.\n-set -uo pipefail\n-cd /workspace/ncm\n-source /venv/main/bin/activate\n-source scripts/setup/set-env.sh >/dev/null 2>&1\n-export HF_TOKEN=$HF_API_KEY NCM_ROOT=/workspace/ncm TOKENIZERS_PARALLELISM=false\n-L=campaign/s1b1024.log\n-log(){ echo \"[$(date -u +%FT%TZ)] $*\" | tee -a $L; }\n-DIR=experiments/xsum/ncp-s1-p95-b1024\n-log \"STAGE1 RESUME from 20k with EMA 0.9999, global batch 1024, to 88k\"\n-torchrun --nproc_per_node=8 --master_port=29470 train_lm.py \\\n- --config-path=config/experiments/xsum --config-name=ncp-s1-p95-b1024 \\\n- >> logs/campaign/ncp-s1-p95-b1024.log 2>&1\n-log \"STAGE1 finished rc=$?\"\n-mkdir -p best_checkpoints/s1-p95-b1024-all\n-for f in $DIR/checkpoint-iter-*.pt $DIR/ema-iter-*.pt; do ln -f \"$f\" best_checkpoints/s1-p95-b1024-all/$(basename \"$f\") 2>/dev/null; done\n-log \"checkpoints preserved\"\ndiff --git a/best_checkpoints/b1024s1-EMA-33.85/chain_stage2_b1024s1.sh b/best_checkpoints/b1024s1-EMA-33.85/chain_stage2_b1024s1.sh\ndeleted file mode 100644\nindex 387144f..0000000\n--- a/best_checkpoints/b1024s1-EMA-33.85/chain_stage2_b1024s1.sh\n+++ /dev/null\n@@ -1,72 +0,0 @@\n-#!/bin/bash\n-# STAGE 2 on the new stage 1: passage -> summary fine-tune.\n-#\n-# Init : stage-1 EMA checkpoint (ema-iter-88000.pt), per the decision to seed\n-# from EMA rather than raw.\n-# CFG : OFF. cfg_dropout is absent from the config, so the trainer default\n-# (0.0) applies and the null branch is not trained during stage 2.\n-# Recipe: identical to the 33.33 run (lr 3e-4 -> 3e-5, 50,002 steps, EMA 0.9999,\n-# global batch 512). The only deliberate differences from that run are\n-# the stage-1 init and the codec (pass80-ema095 vs pass80-ema092).\n-#\n-# Waits for stage 1 AND the multi-seed repetition check to release the GPUs.\n-set -uo pipefail\n-cd /workspace/ncm\n-source /venv/main/bin/activate\n-source scripts/setup/set-env.sh >/dev/null 2>&1\n-export HF_TOKEN=$HF_API_KEY NCM_ROOT=/workspace/ncm TOKENIZERS_PARALLELISM=false\n-L=campaign/stage2_b1024s1.log\n-log(){ echo \"[$(date -u +%FT%TZ)] $*\" | tee -a $L; }\n-\n-S1=experiments/xsum/ncp-s1-p95-b1024\n-D=experiments/xsum/ncp-stage2-b1024s1\n-\n-# 1. wait for stage 1's final EMA checkpoint to be complete\n-until [ -f \"$S1/ema-iter-88000.pt\" ]; do sleep 60; done\n-while :; do a=$(stat -c%s \"$S1/ema-iter-88000.pt\"); sleep 10\n- b=$(stat -c%s \"$S1/ema-iter-88000.pt\"); [ \"$a\" = \"$b\" ] && break; done\n-# 2. wait for the multi-seed check to finish with the GPUs\n-until grep -q \"^\\[done\\]\" campaign/final_rep_seeds.log 2>/dev/null; do sleep 30; done\n-sleep 20\n-\n-# 3. warm start from the stage-1 EMA weights (step->0, no optimizer/scheduler/args,\n-# so load_checkpoint takes the weights-only path and rebuilds the prefix_len=512\n-# buffers rather than inheriting stage 1's prefix_len=1 shapes)\n-rm -rf $D; mkdir -p $D\n-log \"warm start from $S1/ema-iter-88000.pt\"\n-python scripts/xsum/make_warmstart.py --src \"$S1/ema-iter-88000.pt\" \\\n- --out \"$D/checkpoint-iter-0.pt\" 2>&1 | tee -a $L\n-\n-log \"STAGE2 b1024s1: init=stage1-EMA@88000, codec pass80-ema095, lr 3e-4->3e-5, 50002 steps, EMA 0.9999, NO CFG\"\n-torchrun --nproc_per_node=8 --master_port=29473 train_lm.py \\\n- --config-path=config/experiments/xsum --config-name=ncp-stage2-b1024s1 \\\n- > logs/campaign/ncp-stage2-b1024s1.log 2>&1\n-log \"STAGE2 finished rc=$?\"\n-\n-# 4. preserve checkpoints\n-mkdir -p best_checkpoints/stage2-b1024s1-all\n-for f in $D/checkpoint-iter-*.pt $D/ema-iter-*.pt; do\n- ln -f \"$f\" best_checkpoints/stage2-b1024s1-all/$(basename \"$f\") 2>/dev/null\n-done\n-log \"checkpoints preserved\"\n-\n-# 5. full test set (11,233 docs), 8-way sharded, for BOTH the final raw and EMA\n-for tag in raw EMA; do\n- if [ \"$tag\" = raw ]; then\n- CK=$(ls -1 $D/checkpoint-iter-*.pt | sed 's/.*iter-\\([0-9]*\\).pt/\\1 &/' | sort -n | tail -1 | cut -d' ' -f2)\n- else\n- CK=$(ls -1 $D/ema-iter-*.pt 2>/dev/null | sed 's/.*iter-\\([0-9]*\\).pt/\\1 &/' | sort -n | tail -1 | cut -d' ' -f2)\n- fi\n- [ -n \"$CK\" ] || continue\n- OUT=/root/.claude/jobs/c1d3694b/tmp/shards_b1024s1_$tag; rm -rf $OUT; mkdir -p $OUT\n- log \"TEST [$tag] full 11,233 docs, 8-way sharded, ckpt=$CK\"\n- for k in 0 1 2 3 4 5 6 7; do\n- CUDA_VISIBLE_DEVICES=$k python scripts/xsum/eval_shard.py --ncp \"$CK\" --split test \\\n- --n 11233 --bs 32 --shard $k --nshards 8 --out $OUT/shard_$k.pkl \\\n- > $OUT/shard_$k.log 2>&1 &\n- done\n- wait\n- echo \"=== TEST $tag ===\" | tee -a $L\n- CUDA_VISIBLE_DEVICES=0 python scripts/xsum/eval_merge.py --glob \"$OUT/shard_*.pkl\" 2>/dev/null | tee -a $L\n-done\n-log \"ALL DONE\"\ndiff --git a/best_checkpoints/b1024s1-EMA-33.85/ncp-s1-p95-b1024.yaml b/best_checkpoints/b1024s1-EMA-33.85/ncp-s1-p95-b1024.yaml\ndeleted file mode 100644\nindex 1aeb0f9..0000000\n--- a/best_checkpoints/b1024s1-EMA-33.85/ncp-s1-p95-b1024.yaml\n+++ /dev/null\n@@ -1,162 +0,0 @@\n-task: next-concept\n-experiment: ncp-s1-p95-b1024\n-experiment_dir: experiments/xsum/ncp-s1-p95-b1024\n-model:\n- n_layer: 12\n- n_head: 12\n- bias: true\n- dropout: 0.3\n- n_embd: null\n- context_length: 49\n- prefix_len: 1\n- raw_token_prefix: true\n- prefix_pad_token_id: 50257\n- vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- predict_level0: false\n- attn_pattern: block_diagonal\n- use_positional_encoding: true\n- use_level_encoding: true\n- use_rope: true\n- rope_base: 100000\n- use_qk_norm: true\n- ctx_encoder:\n- enabled: false\n- n_layer: 4\n- vocab_size: 50304\n- use_flex_attention: false\n- query_level0: true\n- tie_heads_to_codebook: true\n- tie_heads_learn_scale: true\n-tokenizer:\n- pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\n- vqvae:\n- n_layers: 6\n- embed_dim: 384\n- in_vocab_size: 50304\n- context_length: 48\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- use_qk_norm: true\n- pre_quant_dropout: 0.25\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- checkpoint_path: ncm/codecs/pass80-ema095\n- vector_quantizer_config:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.99\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n-dataset:\n- train_source: data/xsum/train_vqvae_pass.bin\n- validate_source: data/xsum/validation_pass_short.pickle\n- null_prefix_len: 1\n-wandb:\n- project: ncp-xsum\n- group: ncp-cond512\n- enable: false\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0005\n- reset_lr: false\n- warmup_iters: 500\n- lr_decay_iters: 88000\n- min_lr: 1.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.1\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n- token_corruption_prob: 0.35\n- level_loss_alpha: 0.25\n- corruption_neighbour: true\n- corruption_rate_per_level:\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- stochastic_encode:\n- enabled: true\n- temperature: 1.0\n- top_k: 0\n- top_p: 1.0\n- ema_decay: 0.9999\n-training:\n- seed: 42\n- log_interval: 50\n- eval_interval: 5000\n- checkpoint_interval: 5000\n- val_iters: 50\n- total_iters: null\n- tqdm_interval: 50\n- n_epochs: 44.482378\n- batch_size: 128\n- eval_batch_size: 128\n- demo_batch_size: 4\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\ndiff --git a/best_checkpoints/b1024s1-EMA-33.85/ncp-stage2-b1024s1.yaml b/best_checkpoints/b1024s1-EMA-33.85/ncp-stage2-b1024s1.yaml\ndeleted file mode 100644\nindex bf6e2f8..0000000\n--- a/best_checkpoints/b1024s1-EMA-33.85/ncp-stage2-b1024s1.yaml\n+++ /dev/null\n@@ -1,161 +0,0 @@\n-task: next-concept\n-experiment: ncp-stage2-b1024s1\n-experiment_dir: experiments/xsum/ncp-stage2-b1024s1\n-model:\n- n_layer: 12\n- n_head: 12\n- bias: true\n- dropout: 0.3\n- n_embd: null\n- context_length: 560\n- prefix_len: 512\n- raw_token_prefix: true\n- prefix_pad_token_id: 50257\n- vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- predict_level0: false\n- attn_pattern: block_diagonal\n- use_positional_encoding: true\n- use_level_encoding: true\n- use_rope: true\n- rope_base: 100000\n- use_qk_norm: true\n- ctx_encoder:\n- enabled: false\n- n_layer: 4\n- vocab_size: 50304\n- use_flex_attention: false\n- query_level0: true\n- tie_heads_to_codebook: true\n- tie_heads_learn_scale: true\n-tokenizer:\n- pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\n- vqvae:\n- n_layers: 6\n- embed_dim: 384\n- in_vocab_size: 50304\n- context_length: 48\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- use_qk_norm: true\n- pre_quant_dropout: 0.25\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- checkpoint_path: ncm/codecs/pass80-ema095\n- vector_quantizer_config:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.99\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n-dataset:\n- train_source: data/xsum/train_cond512_le48.pickle\n- validate_source: data/xsum/validation_cond512_le48.pickle\n-wandb:\n- project: ncp-xsum\n- group: ncp-cond512\n- enable: false\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0003\n- reset_lr: true\n- warmup_iters: 200\n- lr_decay_iters: 50002\n- min_lr: 3.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.1\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n- token_corruption_prob: 0.35\n- level_loss_alpha: 0.25\n- corruption_neighbour: true\n- corruption_rate_per_level:\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- stochastic_encode:\n- enabled: true\n- temperature: 1.0\n- top_k: 0\n- top_p: 1.0\n- ema_decay: 0.9999\n-training:\n- seed: 42\n- log_interval: 50\n- eval_interval: 5000\n- checkpoint_interval: 2000\n- val_iters: 50\n- total_iters: null\n- tqdm_interval: 50\n- n_epochs: 126.54104\n- batch_size: 64\n- eval_batch_size: 64\n- demo_batch_size: 4\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\ndiff --git a/best_checkpoints/b1024s1-EMA-33.85/vq-pass80-ema095-120k.yaml b/best_checkpoints/b1024s1-EMA-33.85/vq-pass80-ema095-120k.yaml\ndeleted file mode 100644\nindex 6962bc6..0000000\n--- a/best_checkpoints/b1024s1-EMA-33.85/vq-pass80-ema095-120k.yaml\n+++ /dev/null\n@@ -1,110 +0,0 @@\n-task: summarization\n-experiment: vq-pass80-ema095-120k\n-experiment_dir: experiments/xsum/vq-pass80-ema095-120k\n-training:\n- seed: 42\n- log_interval: 100\n- eval_interval: 2000\n- checkpoint_interval: 20000\n- val_iters: 100\n- n_epochs: 2429.1546\n- total_iters: null\n- tqdm_interval: 100\n- batch_size: 512\n- eval_batch_size: 128\n- demo_batch_size: 16\n-model:\n- n_layers: 6\n- context_length: 48\n- embed_dim: null\n- in_vocab_size: 50304\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- pre_quant_dropout: 0.25\n- use_qk_norm: true\n- vector_quantizer:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.95\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n- ema_scale_by_world_size: false\n-wandb:\n- project: ncp-xsum\n- enable: true\n- group: vqvae-xsum-mix30\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0005\n- reset_lr: true\n- warmup_iters: 1000\n- lr_decay_iters: 50000\n- min_lr: 5.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.01\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\n- opt_level: O1\n- loss_scale: dynamic\n-dataset:\n- train_source: data/xsum/train_vqvae_summ.bin\n- train_mix_source: data/xsum/train_vqvae_pass.bin\n- train_mix_prob: 0.8\n- validate_source: data/xsum/validation_gpt2_le48.pickle\n-pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\ndiff --git a/best_checkpoints/ema50k-EMA-33.33/ncp-pretrain-pass80.yaml b/best_checkpoints/ema50k-EMA-33.33/ncp-pretrain-pass80.yaml\ndeleted file mode 100644\nindex f37ea2a..0000000\n--- a/best_checkpoints/ema50k-EMA-33.33/ncp-pretrain-pass80.yaml\n+++ /dev/null\n@@ -1,160 +0,0 @@\n-task: next-concept\n-experiment: ncp-pretrain-pass80\n-experiment_dir: experiments/xsum/ncp-pretrain-pass80\n-model:\n- n_layer: 12\n- n_head: 12\n- bias: true\n- dropout: 0.3\n- n_embd: null\n- context_length: 560\n- prefix_len: 512\n- raw_token_prefix: true\n- prefix_pad_token_id: 50257\n- vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- predict_level0: false\n- attn_pattern: block_diagonal\n- use_positional_encoding: true\n- use_level_encoding: true\n- use_rope: true\n- rope_base: 100000\n- use_qk_norm: true\n- ctx_encoder:\n- enabled: false\n- n_layer: 4\n- vocab_size: 50304\n- use_flex_attention: false\n- query_level0: true\n- tie_heads_to_codebook: true\n- tie_heads_learn_scale: true\n-tokenizer:\n- pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\n- vqvae:\n- n_layers: 6\n- embed_dim: 384\n- in_vocab_size: 50304\n- context_length: 48\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- use_qk_norm: true\n- pre_quant_dropout: 0.25\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- checkpoint_path: ncm/codecs/pass80-ema092\n- vector_quantizer_config:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.99\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n-dataset:\n- train_source: data/xsum/train_pass_uncond.bin\n- validate_source: data/xsum/validation_pass_uncond.pickle\n-wandb:\n- project: ncp-xsum\n- group: ncp-cond512\n- enable: false\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0005\n- reset_lr: true\n- warmup_iters: 500\n- lr_decay_iters: 60002\n- min_lr: 1.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.1\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n- token_corruption_prob: 0.35\n- level_loss_alpha: 0.25\n- corruption_neighbour: true\n- corruption_rate_per_level:\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- stochastic_encode:\n- enabled: true\n- temperature: 1.0\n- top_k: 0\n- top_p: 1.0\n-training:\n- seed: 42\n- log_interval: 50\n- eval_interval: 5000\n- checkpoint_interval: 5000\n- val_iters: 50\n- total_iters: null\n- tqdm_interval: 50\n- n_epochs: 5.390889\n- batch_size: 64\n- eval_batch_size: 64\n- demo_batch_size: 4\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\ndiff --git a/best_checkpoints/ema50k-EMA-33.33/ncp-stage2-p80-ema50k.yaml b/best_checkpoints/ema50k-EMA-33.33/ncp-stage2-p80-ema50k.yaml\ndeleted file mode 100644\nindex 983ef68..0000000\n--- a/best_checkpoints/ema50k-EMA-33.33/ncp-stage2-p80-ema50k.yaml\n+++ /dev/null\n@@ -1,161 +0,0 @@\n-task: next-concept\n-experiment: ncp-stage2-p80-ema50k\n-experiment_dir: experiments/xsum/ncp-stage2-p80-ema50k\n-model:\n- n_layer: 12\n- n_head: 12\n- bias: true\n- dropout: 0.3\n- n_embd: null\n- context_length: 560\n- prefix_len: 512\n- raw_token_prefix: true\n- prefix_pad_token_id: 50257\n- vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- predict_level0: false\n- attn_pattern: block_diagonal\n- use_positional_encoding: true\n- use_level_encoding: true\n- use_rope: true\n- rope_base: 100000\n- use_qk_norm: true\n- ctx_encoder:\n- enabled: false\n- n_layer: 4\n- vocab_size: 50304\n- use_flex_attention: false\n- query_level0: true\n- tie_heads_to_codebook: true\n- tie_heads_learn_scale: true\n-tokenizer:\n- pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\n- vqvae:\n- n_layers: 6\n- embed_dim: 384\n- in_vocab_size: 50304\n- context_length: 48\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- use_qk_norm: true\n- pre_quant_dropout: 0.25\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- checkpoint_path: ncm/codecs/pass80-ema092\n- vector_quantizer_config:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.99\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n-dataset:\n- train_source: data/xsum/train_cond512_le48.pickle\n- validate_source: data/xsum/validation_cond512_le48.pickle\n-wandb:\n- project: ncp-xsum\n- group: ncp-cond512\n- enable: false\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0003\n- reset_lr: true\n- warmup_iters: 200\n- lr_decay_iters: 50002\n- min_lr: 3.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.1\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n- token_corruption_prob: 0.35\n- level_loss_alpha: 0.25\n- corruption_neighbour: true\n- corruption_rate_per_level:\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- stochastic_encode:\n- enabled: true\n- temperature: 1.0\n- top_k: 0\n- top_p: 1.0\n- ema_decay: 0.9999\n-training:\n- seed: 42\n- log_interval: 50\n- eval_interval: 5000\n- checkpoint_interval: 2000\n- val_iters: 50\n- total_iters: null\n- tqdm_interval: 50\n- n_epochs: 126.54104\n- batch_size: 64\n- eval_batch_size: 64\n- demo_batch_size: 4\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\ndiff --git a/best_checkpoints/ema50k-EMA-33.33/vq-pass80-ema092.yaml b/best_checkpoints/ema50k-EMA-33.33/vq-pass80-ema092.yaml\ndeleted file mode 100644\nindex 0868f48..0000000\n--- a/best_checkpoints/ema50k-EMA-33.33/vq-pass80-ema092.yaml\n+++ /dev/null\n@@ -1,110 +0,0 @@\n-task: summarization\n-experiment: vq-pass80-ema092\n-experiment_dir: experiments/xsum/vq-pass80-ema092\n-training:\n- seed: 42\n- log_interval: 100\n- eval_interval: 2000\n- checkpoint_interval: 10000\n- val_iters: 100\n- n_epochs: 1012.1477\n- total_iters: null\n- tqdm_interval: 100\n- batch_size: 512\n- eval_batch_size: 128\n- demo_batch_size: 16\n-model:\n- n_layers: 6\n- context_length: 48\n- embed_dim: null\n- in_vocab_size: 50304\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- pre_quant_dropout: 0.25\n- use_qk_norm: true\n- vector_quantizer:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.92\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n- ema_scale_by_world_size: false\n-wandb:\n- project: ncp-xsum\n- enable: true\n- group: vqvae-xsum-mix30\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0005\n- reset_lr: true\n- warmup_iters: 1000\n- lr_decay_iters: 50000\n- min_lr: 5.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.01\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\n- opt_level: O1\n- loss_scale: dynamic\n-dataset:\n- train_source: data/xsum/train_vqvae_summ.bin\n- train_mix_source: data/xsum/train_vqvae_pass.bin\n- train_mix_prob: 0.8\n- validate_source: data/xsum/validation_gpt2_le48.pickle\n-pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\ndiff --git a/best_checkpoints/pretrain-pass80-30k/README.md b/best_checkpoints/pretrain-pass80-30k/README.md\ndeleted file mode 100644\nindex e3f21bb..0000000\n--- a/best_checkpoints/pretrain-pass80-30k/README.md\n+++ /dev/null\n@@ -1,25 +0,0 @@\n-# stage-1 pretrain-pass80 @ 30,000 \u2014 KEEP\n-\n-Unconditional passage-manifold checkpoint. Real copies, NOT hardlinks, so\n-deleting experiments/xsum/ncp-pretrain-pass80/ does not touch these.\n-\n-## What it is\n-NCP trained on p(passage codes) with NO document conditioning:\n-rows = [EOS, PAD x511] ++ [48-token passage window], 5,698,694 windows.\n-\n-## Validation at this checkpoint\n-loss 3.094 \u00b7 acc 48.69% \u00b7 last-level acc 75.06%\n-(trend 3.689 -> 3.405 -> 3.279 -> 3.206 -> 3.153 -> 3.094 across 5k..30k,\n- still falling; 2.70 epochs, no overfitting -- val/train gap flat at -1.42)\n-\n-## Codec it was trained against\n-codec.pt = codecs/pass80-ema092 : 80% passages / 20% summaries,\n-EMA decay 0.92, ema_scale_by_world_size FALSE, recon(summaries) 99.703.\n-Level-0 utilisation: 1,388 codes, perplexity 1055.9 (e12: 1,025 / 252.5).\n-\n-## To use as a stage-2 init\n-python scripts/xsum/make_warmstart.py \\\n- --src best_checkpoints/pretrain-pass80-30k/checkpoint-iter-30000.pt \\\n- --out experiments/xsum/<stage2-dir>/checkpoint-iter-0.pt\n-then train with reset_lr: true. The warm-start strips step/optimizer/args so\n-training starts at 0 with a fresh schedule.\ndiff --git a/best_checkpoints/pretrain-pass80-30k/ncp-pretrain-pass80.yaml b/best_checkpoints/pretrain-pass80-30k/ncp-pretrain-pass80.yaml\ndeleted file mode 100644\nindex f37ea2a..0000000\n--- a/best_checkpoints/pretrain-pass80-30k/ncp-pretrain-pass80.yaml\n+++ /dev/null\n@@ -1,160 +0,0 @@\n-task: next-concept\n-experiment: ncp-pretrain-pass80\n-experiment_dir: experiments/xsum/ncp-pretrain-pass80\n-model:\n- n_layer: 12\n- n_head: 12\n- bias: true\n- dropout: 0.3\n- n_embd: null\n- context_length: 560\n- prefix_len: 512\n- raw_token_prefix: true\n- prefix_pad_token_id: 50257\n- vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- predict_level0: false\n- attn_pattern: block_diagonal\n- use_positional_encoding: true\n- use_level_encoding: true\n- use_rope: true\n- rope_base: 100000\n- use_qk_norm: true\n- ctx_encoder:\n- enabled: false\n- n_layer: 4\n- vocab_size: 50304\n- use_flex_attention: false\n- query_level0: true\n- tie_heads_to_codebook: true\n- tie_heads_learn_scale: true\n-tokenizer:\n- pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\n- vqvae:\n- n_layers: 6\n- embed_dim: 384\n- in_vocab_size: 50304\n- context_length: 48\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- use_qk_norm: true\n- pre_quant_dropout: 0.25\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- checkpoint_path: ncm/codecs/pass80-ema092\n- vector_quantizer_config:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.99\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n-dataset:\n- train_source: data/xsum/train_pass_uncond.bin\n- validate_source: data/xsum/validation_pass_uncond.pickle\n-wandb:\n- project: ncp-xsum\n- group: ncp-cond512\n- enable: false\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0005\n- reset_lr: true\n- warmup_iters: 500\n- lr_decay_iters: 60002\n- min_lr: 1.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.1\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n- token_corruption_prob: 0.35\n- level_loss_alpha: 0.25\n- corruption_neighbour: true\n- corruption_rate_per_level:\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- stochastic_encode:\n- enabled: true\n- temperature: 1.0\n- top_k: 0\n- top_p: 1.0\n-training:\n- seed: 42\n- log_interval: 50\n- eval_interval: 5000\n- checkpoint_interval: 5000\n- val_iters: 50\n- total_iters: null\n- tqdm_interval: 50\n- n_epochs: 5.390889\n- batch_size: 64\n- eval_batch_size: 64\n- demo_batch_size: 4\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\ndiff --git a/best_checkpoints/pretrain-pass80-30k/vq-pass80-ema092.yaml b/best_checkpoints/pretrain-pass80-30k/vq-pass80-ema092.yaml\ndeleted file mode 100644\nindex 0868f48..0000000\n--- a/best_checkpoints/pretrain-pass80-30k/vq-pass80-ema092.yaml\n+++ /dev/null\n@@ -1,110 +0,0 @@\n-task: summarization\n-experiment: vq-pass80-ema092\n-experiment_dir: experiments/xsum/vq-pass80-ema092\n-training:\n- seed: 42\n- log_interval: 100\n- eval_interval: 2000\n- checkpoint_interval: 10000\n- val_iters: 100\n- n_epochs: 1012.1477\n- total_iters: null\n- tqdm_interval: 100\n- batch_size: 512\n- eval_batch_size: 128\n- demo_batch_size: 16\n-model:\n- n_layers: 6\n- context_length: 48\n- embed_dim: null\n- in_vocab_size: 50304\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- pre_quant_dropout: 0.25\n- use_qk_norm: true\n- vector_quantizer:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.92\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n- ema_scale_by_world_size: false\n-wandb:\n- project: ncp-xsum\n- enable: true\n- group: vqvae-xsum-mix30\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0005\n- reset_lr: true\n- warmup_iters: 1000\n- lr_decay_iters: 50000\n- min_lr: 5.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.01\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\n- opt_level: O1\n- loss_scale: dynamic\n-dataset:\n- train_source: data/xsum/train_vqvae_summ.bin\n- train_mix_source: data/xsum/train_vqvae_pass.bin\n- train_mix_prob: 0.8\n- validate_source: data/xsum/validation_gpt2_le48.pickle\n-pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\ndiff --git a/best_checkpoints/stage2-cfg-partial/ncp-stage2-cfg.yaml b/best_checkpoints/stage2-cfg-partial/ncp-stage2-cfg.yaml\ndeleted file mode 100644\nindex afe16bd..0000000\n--- a/best_checkpoints/stage2-cfg-partial/ncp-stage2-cfg.yaml\n+++ /dev/null\n@@ -1,163 +0,0 @@\n-task: next-concept\n-experiment: ncp-stage2-cfg\n-experiment_dir: experiments/xsum/ncp-stage2-cfg\n-model:\n- n_layer: 12\n- n_head: 12\n- bias: true\n- dropout: 0.3\n- n_embd: null\n- context_length: 560\n- prefix_len: 512\n- raw_token_prefix: true\n- prefix_pad_token_id: 50257\n- vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- predict_level0: false\n- attn_pattern: block_diagonal\n- use_positional_encoding: true\n- use_level_encoding: true\n- use_rope: true\n- rope_base: 100000\n- use_qk_norm: true\n- ctx_encoder:\n- enabled: false\n- n_layer: 4\n- vocab_size: 50304\n- use_flex_attention: false\n- query_level0: true\n- tie_heads_to_codebook: true\n- tie_heads_learn_scale: true\n-tokenizer:\n- pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\n- vqvae:\n- n_layers: 6\n- embed_dim: 384\n- in_vocab_size: 50304\n- context_length: 48\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- use_qk_norm: true\n- pre_quant_dropout: 0.25\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- checkpoint_path: ncm/codecs/pass80-ema095\n- vector_quantizer_config:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.99\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n-dataset:\n- train_source: data/xsum/train_cond512_le48.pickle\n- validate_source: data/xsum/validation_cond512_le48.pickle\n-wandb:\n- project: ncp-xsum\n- group: ncp-cond512\n- enable: false\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0003\n- reset_lr: true\n- warmup_iters: 200\n- lr_decay_iters: 50002\n- min_lr: 3.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.1\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n- token_corruption_prob: 0.35\n- level_loss_alpha: 0.25\n- corruption_neighbour: true\n- corruption_rate_per_level:\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- stochastic_encode:\n- enabled: true\n- temperature: 1.0\n- top_k: 0\n- top_p: 1.0\n- ema_decay: 0.9999\n- cfg_dropout: 0.1\n- cfg_null_eos_id: 50256\n-training:\n- seed: 42\n- log_interval: 50\n- eval_interval: 5000\n- checkpoint_interval: 5000\n- val_iters: 50\n- total_iters: null\n- tqdm_interval: 50\n- n_epochs: 126.54104\n- batch_size: 64\n- eval_batch_size: 64\n- demo_batch_size: 4\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\ndiff --git a/best_checkpoints/stage2-cfg-partial/test_cfg_correct.py b/best_checkpoints/stage2-cfg-partial/test_cfg_correct.py\ndeleted file mode 100644\nindex 648a94c..0000000\n--- a/best_checkpoints/stage2-cfg-partial/test_cfg_correct.py\n+++ /dev/null\n@@ -1,98 +0,0 @@\n-\"\"\"Correctness tests for classifier-free guidance in NextConceptTransformer.generate.\n-\n-Run this BEFORE spending GPU-hours on a CFG training run. Four checks:\n-\n- 1. IDENTITY forced duplicated-batch at w=1.0 must produce BIT-IDENTICAL codes\n- to the ordinary unguided path. This is the load-bearing test: it\n- proves the [real ; null] duplication does not perturb the\n- conditional branch (prefix_emb, _pbias, RoPE, level state).\n- 2. NULL BITES the unconditional half must produce DIFFERENT logits from the\n- conditional half. If it does not, the null document is not\n- reaching the model and any \"guidance\" would be a no-op.\n- 3. W CHANGES w != 1 must actually change the sampled codes.\n- 4. LEVELS cfg_levels must restrict guidance: guiding a level set must equal\n- the unguided path on the levels outside it.\n-\n-Usage: python scripts/xsum/test_cfg_correct.py --ncp <checkpoint.pt>\n-\"\"\"\n-import argparse, os, sys\n-import torch\n-\n-sys.path.insert(0, os.environ.get(\"NCM_ROOT\", os.getcwd()))\n-from models.next_concept import NextConceptTransformer\n-from tokenization.vqvae_tokenizer import VQVAETokenizer\n-\n-ap = argparse.ArgumentParser()\n-ap.add_argument(\"--ncp\", required=True)\n-ap.add_argument(\"--n\", type=int, default=8)\n-a = ap.parse_args()\n-\n-sd = torch.load(a.ncp, weights_only=False, map_location=\"cpu\")\n-ca = sd[\"args\"]\n-tk = VQVAETokenizer.from_config(ca.tokenizer)\n-tk.vqvae = tk.vqvae.cuda().eval()\n-net = NextConceptTransformer(vqvae_local=tk.vqvae, config=ca.model).cuda()\n-net.load_state_dict(sd[\"model\"], strict=True)\n-net.eval()\n-\n-P = int(ca.model.prefix_len)\n-import pickle\n-rows = pickle.load(open(\"/workspace/data/xsum/test_cond512_le48.pickle\", \"rb\"))[: a.n]\n-prefix = torch.tensor([r[:P] for r in rows], dtype=torch.long).cuda()\n-print(f\"model prefix_len={P} batch={prefix.shape[0]} pad_id={net.prefix_pad_id}\\n\")\n-\n-GREEDY = dict(temperature=1.0, top_k=1, top_p=1.0) # deterministic: removes sampling noise\n-\n-\n-def gen(**kw):\n- torch.manual_seed(0)\n- with torch.no_grad():\n- return net.generate(None, prefix, **GREEDY, **kw)\n-\n-\n-fail = 0\n-\n-# ---- 1. identity ---------------------------------------------------------\n-base = gen()\n-forced = gen(cfg_scale=1.0, cfg_force=True)\n-same = torch.equal(base, forced)\n-print(f\"1. IDENTITY forced w=1.0 == unguided : {'PASS' if same else 'FAIL'}\"\n- f\" ({(base != forced).sum().item()}/{base.numel()} codes differ)\")\n-fail += not same\n-\n-# ---- 2. does the null document actually change the computation? ----------\n-with torch.no_grad():\n- null = torch.full_like(prefix, net.prefix_pad_id)\n- null[:, 0] = 50256\n- torch.manual_seed(0); c = net.generate(None, prefix, **GREEDY)\n- torch.manual_seed(0); u = net.generate(None, null, **GREEDY)\n-bites = not torch.equal(c, u)\n-print(f\"2. NULL BITES cond codes != uncond codes: {'PASS' if bites else 'FAIL'}\"\n- f\" ({(c != u).sum().item()}/{c.numel()} codes differ)\")\n-fail += not bites\n-\n-# ---- 3. guidance changes the output --------------------------------------\n-g2 = gen(cfg_scale=2.0)\n-moved = not torch.equal(base, g2)\n-print(f\"3. W CHANGES w=2.0 != w=1.0 : {'PASS' if moved else 'FAIL'}\"\n- f\" ({(base != g2).sum().item()}/{base.numel()} codes differ)\")\n-fail += not moved\n-\n-# ---- 4. cfg_levels restricts guidance ------------------------------------\n-n_lv = len(net.levels)\n-only0 = gen(cfg_scale=3.0, cfg_levels=[0])\n-alllv = gen(cfg_scale=3.0)\n-restricted = (not torch.equal(only0, alllv)) and (not torch.equal(only0, base))\n-print(f\"4. LEVELS cfg_levels=[0] differs from both all-levels and unguided:\"\n- f\" {'PASS' if restricted else 'FAIL'}\")\n-fail += not restricted\n-\n-# an empty level set must be exactly the unguided path\n-empty = gen(cfg_scale=5.0, cfg_levels=[])\n-noop = torch.equal(base, empty)\n-print(f\"4b. LEVELS cfg_levels=[] == unguided : {'PASS' if noop else 'FAIL'}\"\n- f\" ({(base != empty).sum().item()}/{base.numel()} codes differ)\")\n-fail += not noop\n-\n-print(f\"\\n{'ALL CHECKS PASSED' if fail == 0 else f'{fail} CHECK(S) FAILED'}\")\n-sys.exit(1 if fail else 0)\ndiff --git a/best_checkpoints/stage2-p80-32.75/ncp-pretrain-pass80.yaml b/best_checkpoints/stage2-p80-32.75/ncp-pretrain-pass80.yaml\ndeleted file mode 100644\nindex f37ea2a..0000000\n--- a/best_checkpoints/stage2-p80-32.75/ncp-pretrain-pass80.yaml\n+++ /dev/null\n@@ -1,160 +0,0 @@\n-task: next-concept\n-experiment: ncp-pretrain-pass80\n-experiment_dir: experiments/xsum/ncp-pretrain-pass80\n-model:\n- n_layer: 12\n- n_head: 12\n- bias: true\n- dropout: 0.3\n- n_embd: null\n- context_length: 560\n- prefix_len: 512\n- raw_token_prefix: true\n- prefix_pad_token_id: 50257\n- vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- predict_level0: false\n- attn_pattern: block_diagonal\n- use_positional_encoding: true\n- use_level_encoding: true\n- use_rope: true\n- rope_base: 100000\n- use_qk_norm: true\n- ctx_encoder:\n- enabled: false\n- n_layer: 4\n- vocab_size: 50304\n- use_flex_attention: false\n- query_level0: true\n- tie_heads_to_codebook: true\n- tie_heads_learn_scale: true\n-tokenizer:\n- pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\n- vqvae:\n- n_layers: 6\n- embed_dim: 384\n- in_vocab_size: 50304\n- context_length: 48\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- use_qk_norm: true\n- pre_quant_dropout: 0.25\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- checkpoint_path: ncm/codecs/pass80-ema092\n- vector_quantizer_config:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.99\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n-dataset:\n- train_source: data/xsum/train_pass_uncond.bin\n- validate_source: data/xsum/validation_pass_uncond.pickle\n-wandb:\n- project: ncp-xsum\n- group: ncp-cond512\n- enable: false\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0005\n- reset_lr: true\n- warmup_iters: 500\n- lr_decay_iters: 60002\n- min_lr: 1.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.1\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n- token_corruption_prob: 0.35\n- level_loss_alpha: 0.25\n- corruption_neighbour: true\n- corruption_rate_per_level:\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- stochastic_encode:\n- enabled: true\n- temperature: 1.0\n- top_k: 0\n- top_p: 1.0\n-training:\n- seed: 42\n- log_interval: 50\n- eval_interval: 5000\n- checkpoint_interval: 5000\n- val_iters: 50\n- total_iters: null\n- tqdm_interval: 50\n- n_epochs: 5.390889\n- batch_size: 64\n- eval_batch_size: 64\n- demo_batch_size: 4\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\ndiff --git a/best_checkpoints/stage2-p80-32.75/ncp-stage2-pass80.yaml b/best_checkpoints/stage2-p80-32.75/ncp-stage2-pass80.yaml\ndeleted file mode 100644\nindex 9f9e8d5..0000000\n--- a/best_checkpoints/stage2-p80-32.75/ncp-stage2-pass80.yaml\n+++ /dev/null\n@@ -1,160 +0,0 @@\n-task: next-concept\n-experiment: ncp-stage2-pass80\n-experiment_dir: experiments/xsum/ncp-stage2-pass80\n-model:\n- n_layer: 12\n- n_head: 12\n- bias: true\n- dropout: 0.3\n- n_embd: null\n- context_length: 560\n- prefix_len: 512\n- raw_token_prefix: true\n- prefix_pad_token_id: 50257\n- vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- predict_level0: false\n- attn_pattern: block_diagonal\n- use_positional_encoding: true\n- use_level_encoding: true\n- use_rope: true\n- rope_base: 100000\n- use_qk_norm: true\n- ctx_encoder:\n- enabled: false\n- n_layer: 4\n- vocab_size: 50304\n- use_flex_attention: false\n- query_level0: true\n- tie_heads_to_codebook: true\n- tie_heads_learn_scale: true\n-tokenizer:\n- pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\n- vqvae:\n- n_layers: 6\n- embed_dim: 384\n- in_vocab_size: 50304\n- context_length: 48\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- use_qk_norm: true\n- pre_quant_dropout: 0.25\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- checkpoint_path: ncm/codecs/pass80-ema092\n- vector_quantizer_config:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.99\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n-dataset:\n- train_source: data/xsum/train_cond512_le48.pickle\n- validate_source: data/xsum/validation_cond512_le48.pickle\n-wandb:\n- project: ncp-xsum\n- group: ncp-cond512\n- enable: false\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0003\n- reset_lr: true\n- warmup_iters: 200\n- lr_decay_iters: 30001\n- min_lr: 3.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.1\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n- token_corruption_prob: 0.35\n- level_loss_alpha: 0.25\n- corruption_neighbour: true\n- corruption_rate_per_level:\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- - 0.35\n- stochastic_encode:\n- enabled: true\n- temperature: 1.0\n- top_k: 0\n- top_p: 1.0\n-training:\n- seed: 42\n- log_interval: 50\n- eval_interval: 5000\n- checkpoint_interval: 1000\n- val_iters: 50\n- total_iters: null\n- tqdm_interval: 50\n- n_epochs: 75.9241\n- batch_size: 64\n- eval_batch_size: 64\n- demo_batch_size: 4\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\ndiff --git a/best_checkpoints/stage2-p80-32.75/vq-pass80-ema092.yaml b/best_checkpoints/stage2-p80-32.75/vq-pass80-ema092.yaml\ndeleted file mode 100644\nindex 0868f48..0000000\n--- a/best_checkpoints/stage2-p80-32.75/vq-pass80-ema092.yaml\n+++ /dev/null\n@@ -1,110 +0,0 @@\n-task: summarization\n-experiment: vq-pass80-ema092\n-experiment_dir: experiments/xsum/vq-pass80-ema092\n-training:\n- seed: 42\n- log_interval: 100\n- eval_interval: 2000\n- checkpoint_interval: 10000\n- val_iters: 100\n- n_epochs: 1012.1477\n- total_iters: null\n- tqdm_interval: 100\n- batch_size: 512\n- eval_batch_size: 128\n- demo_batch_size: 16\n-model:\n- n_layers: 6\n- context_length: 48\n- embed_dim: null\n- in_vocab_size: 50304\n- vqvae_vocab_size:\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- - 4096\n- compression_factor: 1\n- dropout: 0.1\n- pre_quant_groupnorm: 8\n- pre_quant_dropout: 0.25\n- use_qk_norm: true\n- vector_quantizer:\n- clss: multiscale_residual_vector_quantizer\n- decay: 0.92\n- epsilon: 1.0e-05\n- commitment_cost: 0.25\n- learned_l1_sampling: true\n- learned_l1_cascade: true\n- learned_l1_cascade_base: 4\n- learned_all_sampling: false\n- quant_resi:\n- enabled: true\n- ratio: 0.5\n- share_mode: 0\n- learnable_ratio: false\n- aux:\n- fine_drop:\n- prob: 0.5\n- min_keep: 1\n- levels:\n- use_manual_levels: true\n- manual_levels:\n- - 4\n- - 6\n- - 10\n- - 14\n- - 20\n- - 26\n- - 32\n- - 40\n- - 48\n- learned_l1_cascade_norm: true\n- learned_l1_cascade_norm_final: true\n- ema_scale_by_world_size: false\n-wandb:\n- project: ncp-xsum\n- enable: true\n- group: vqvae-xsum-mix30\n- resume: null\n- gradients_and_params:\n- enable: false\n- log: all\n- log_freq: 1000\n-optimization:\n- lr: 0.0005\n- reset_lr: true\n- warmup_iters: 1000\n- lr_decay_iters: 50000\n- min_lr: 5.0e-05\n- beta_1: 0.9\n- beta_2: 0.95\n- weight_decay: 0.01\n- max_grad_norm: 1.0\n- grad_accumulation_steps: 1\n-torch_compile:\n- enable: true\n- mode: default\n- dynamic: false\n- fullgraph: false\n- backend: inductor\n-mixed_precision:\n- enable: true\n- opt_level: O1\n- loss_scale: dynamic\n-dataset:\n- train_source: data/xsum/train_vqvae_summ.bin\n- train_mix_source: data/xsum/train_vqvae_pass.bin\n- train_mix_prob: 0.8\n- validate_source: data/xsum/validation_gpt2_le48.pickle\n-pre_tokenizer:\n- name: hf\n- tokenizer: hf\n- model_id: gpt2\n- special_tokens:\n- eos_token: <|endoftext|>\n- pad_token: <|pad|>\ndiff --git a/models/quant.py b/models/quant.py\nindex a3f0579..5e58f99 100644\n--- a/models/quant.py\n+++ b/models/quant.py\n@@ -826,14 +826,16 @@ class MultiscaleResidualVectorQuantizer(VectorQuantizerBaseClass):\n \n z_hat = z_hat + quantized\n z_rest -= quantized\n- z_hats.append(z_hat)\n- z_levels.append(\n- F.interpolate(\n- z_hat.transpose(-1, -2), size=(level,), mode=\"area\"\n- ).transpose(-1, -2)\n- if i != num_levels - 1\n- else z_hat\n- )\n+ if return_zhats:\n+ z_hats.append(z_hat)\n+ if return_zlevels:\n+ z_levels.append(\n+ F.interpolate(\n+ z_hat.transpose(-1, -2), size=(level,), mode=\"area\"\n+ ).transpose(-1, -2)\n+ if i != num_levels - 1\n+ else z_hat\n+ )\n \n idx = torch.cat(idx, dim=-1)\n \n",
|
| 6 |
+
"refs": "5b2fad02f2effa5d61343055b1b55c133787e004 refs/heads/debug\n2c52c050d382c9f0f7e1e74bcce98974be9124ea refs/heads/iclr-debug\n4e22b4740368661b414f176673c2f099079047eb refs/heads/medium-speed-20260918\n299a84cca53485e9265f4d9b77b3c17b012de19b refs/heads/optimize/mstok-compile\n59db1eba55ff86f557f5a1c81886ed876c61fe22 refs/heads/owt-128\nbafe465d46a409dc98ccd082291ff959ff41d29a refs/heads/repro\n9051899cd722e4093eb735ab8e00d23423254ddb refs/heads/ts-128\n212437451fbb0289f853a190aa2a9cdfd775ec2d refs/remotes/origin/HEAD\nff45f6000c0a261497e68427bc383aff00b64473 refs/remotes/origin/debug\n8d4ff33206f51409233737ae4ba130acc2f84327 refs/remotes/origin/e2e\n8fc372a2ce41c049523cdb003b02cb8f9183f1fd refs/remotes/origin/iclr-baselines\n2c52c050d382c9f0f7e1e74bcce98974be9124ea refs/remotes/origin/iclr-debug\n0e0f56cad8e032235edcf39d672af71d848e034f refs/remotes/origin/info-theory\n212437451fbb0289f853a190aa2a9cdfd775ec2d refs/remotes/origin/main\n5e9f74744c91caee71ce170c1b5bdf96a8d6b390 refs/remotes/origin/optimize/mstok-compile\n59db1eba55ff86f557f5a1c81886ed876c61fe22 refs/remotes/origin/owt-128\na8b14eeb91c562d14bfc8b39ca8f75b12088edca refs/remotes/origin/per-scale\nbafe465d46a409dc98ccd082291ff959ff41d29a refs/remotes/origin/repro\n4e22b4740368661b414f176673c2f099079047eb refs/remotes/origin/repro-mstok-medium-speed\n4bf9137cb3aa13e074a1d4e7913b0abe1f0ccedb refs/remotes/origin/semantic-rep\nc237933d307688577f9d229f14863647354118a6 refs/remotes/origin/tinystories\n9051899cd722e4093eb735ab8e00d23423254ddb refs/remotes/origin/ts-128\n2cc5e19b16609625006e9879fff1a18ac2ef1626 refs/remotes/origin/updates\nee0873cabf7940ef12ca49833ab886320a59dcf3 refs/remotes/origin/v0-mstok\n9ecf07fa9099535d9ea20d089c526b61598e2569 refs/remotes/origin/vq-analysis\n0d966d4167532c8c487d6d7f6fcf6a3d72d3e66f refs/stash\n"
|
| 7 |
+
},
|
| 8 |
+
"/home/ishank/mstok-medium-speed": {
|
| 9 |
+
"head": "4e22b4740368661b414f176673c2f099079047eb\n",
|
| 10 |
+
"status": "",
|
| 11 |
+
"diff": "",
|
| 12 |
+
"refs": "5b2fad02f2effa5d61343055b1b55c133787e004 refs/heads/debug\n2c52c050d382c9f0f7e1e74bcce98974be9124ea refs/heads/iclr-debug\n4e22b4740368661b414f176673c2f099079047eb refs/heads/medium-speed-20260918\n299a84cca53485e9265f4d9b77b3c17b012de19b refs/heads/optimize/mstok-compile\n59db1eba55ff86f557f5a1c81886ed876c61fe22 refs/heads/owt-128\nbafe465d46a409dc98ccd082291ff959ff41d29a refs/heads/repro\n9051899cd722e4093eb735ab8e00d23423254ddb refs/heads/ts-128\n212437451fbb0289f853a190aa2a9cdfd775ec2d refs/remotes/origin/HEAD\nff45f6000c0a261497e68427bc383aff00b64473 refs/remotes/origin/debug\n8d4ff33206f51409233737ae4ba130acc2f84327 refs/remotes/origin/e2e\n8fc372a2ce41c049523cdb003b02cb8f9183f1fd refs/remotes/origin/iclr-baselines\n2c52c050d382c9f0f7e1e74bcce98974be9124ea refs/remotes/origin/iclr-debug\n0e0f56cad8e032235edcf39d672af71d848e034f refs/remotes/origin/info-theory\n212437451fbb0289f853a190aa2a9cdfd775ec2d refs/remotes/origin/main\n5e9f74744c91caee71ce170c1b5bdf96a8d6b390 refs/remotes/origin/optimize/mstok-compile\n59db1eba55ff86f557f5a1c81886ed876c61fe22 refs/remotes/origin/owt-128\na8b14eeb91c562d14bfc8b39ca8f75b12088edca refs/remotes/origin/per-scale\nbafe465d46a409dc98ccd082291ff959ff41d29a refs/remotes/origin/repro\n4e22b4740368661b414f176673c2f099079047eb refs/remotes/origin/repro-mstok-medium-speed\n4bf9137cb3aa13e074a1d4e7913b0abe1f0ccedb refs/remotes/origin/semantic-rep\nc237933d307688577f9d229f14863647354118a6 refs/remotes/origin/tinystories\n9051899cd722e4093eb735ab8e00d23423254ddb refs/remotes/origin/ts-128\n2cc5e19b16609625006e9879fff1a18ac2ef1626 refs/remotes/origin/updates\nee0873cabf7940ef12ca49833ab886320a59dcf3 refs/remotes/origin/v0-mstok\n9ecf07fa9099535d9ea20d089c526b61598e2569 refs/remotes/origin/vq-analysis\n0d966d4167532c8c487d6d7f6fcf6a3d72d3e66f refs/stash\n"
|
| 13 |
+
},
|
| 14 |
+
"/home/ishank/mstok-medium-iclr-debug": {
|
| 15 |
+
"head": "2c52c050d382c9f0f7e1e74bcce98974be9124ea\n",
|
| 16 |
+
"status": "",
|
| 17 |
+
"diff": "",
|
| 18 |
+
"refs": "5b2fad02f2effa5d61343055b1b55c133787e004 refs/heads/debug\n2c52c050d382c9f0f7e1e74bcce98974be9124ea refs/heads/iclr-debug\n4e22b4740368661b414f176673c2f099079047eb refs/heads/medium-speed-20260918\n299a84cca53485e9265f4d9b77b3c17b012de19b refs/heads/optimize/mstok-compile\n59db1eba55ff86f557f5a1c81886ed876c61fe22 refs/heads/owt-128\nbafe465d46a409dc98ccd082291ff959ff41d29a refs/heads/repro\n9051899cd722e4093eb735ab8e00d23423254ddb refs/heads/ts-128\n212437451fbb0289f853a190aa2a9cdfd775ec2d refs/remotes/origin/HEAD\nff45f6000c0a261497e68427bc383aff00b64473 refs/remotes/origin/debug\n8d4ff33206f51409233737ae4ba130acc2f84327 refs/remotes/origin/e2e\n8fc372a2ce41c049523cdb003b02cb8f9183f1fd refs/remotes/origin/iclr-baselines\n2c52c050d382c9f0f7e1e74bcce98974be9124ea refs/remotes/origin/iclr-debug\n0e0f56cad8e032235edcf39d672af71d848e034f refs/remotes/origin/info-theory\n212437451fbb0289f853a190aa2a9cdfd775ec2d refs/remotes/origin/main\n5e9f74744c91caee71ce170c1b5bdf96a8d6b390 refs/remotes/origin/optimize/mstok-compile\n59db1eba55ff86f557f5a1c81886ed876c61fe22 refs/remotes/origin/owt-128\na8b14eeb91c562d14bfc8b39ca8f75b12088edca refs/remotes/origin/per-scale\nbafe465d46a409dc98ccd082291ff959ff41d29a refs/remotes/origin/repro\n4e22b4740368661b414f176673c2f099079047eb refs/remotes/origin/repro-mstok-medium-speed\n4bf9137cb3aa13e074a1d4e7913b0abe1f0ccedb refs/remotes/origin/semantic-rep\nc237933d307688577f9d229f14863647354118a6 refs/remotes/origin/tinystories\n9051899cd722e4093eb735ab8e00d23423254ddb refs/remotes/origin/ts-128\n2cc5e19b16609625006e9879fff1a18ac2ef1626 refs/remotes/origin/updates\nee0873cabf7940ef12ca49833ab886320a59dcf3 refs/remotes/origin/v0-mstok\n9ecf07fa9099535d9ea20d089c526b61598e2569 refs/remotes/origin/vq-analysis\n0d966d4167532c8c487d6d7f6fcf6a3d72d3e66f refs/stash\n"
|
| 19 |
+
},
|
| 20 |
+
"/home/ishank/ncm-xsum-repro-20260921": {
|
| 21 |
+
"head": "8824bf2d7257807756dffa05f466145d938d3cf6\n",
|
| 22 |
+
"status": " M scripts/xsum/eval_shard_cfg.py\n",
|
| 23 |
+
"diff": "diff --git a/scripts/xsum/eval_shard_cfg.py b/scripts/xsum/eval_shard_cfg.py\nindex 5404641..73ae4f8 100644\n--- a/scripts/xsum/eval_shard_cfg.py\n+++ b/scripts/xsum/eval_shard_cfg.py\n@@ -18,8 +18,9 @@ sys.path.insert(0, os.environ.get(\"NCM_ROOT\", os.getcwd()))\n from models.next_concept import NextConceptTransformer\n from tokenization.vqvae_tokenizer import VQVAETokenizer\n EOS, PAD = 50256, 50257\n-VAL = \"/workspace/data/xsum/validation_cond512_le48.pickle\"\n-TEST = \"/workspace/data/xsum/test_cond512_le48.pickle\"\n+DATA_ROOT = os.environ.get(\"XSUM_DATA_ROOT\", \"/workspace/data/xsum\")\n+VAL = os.path.join(DATA_ROOT, \"validation_cond512_le48.pickle\")\n+TEST = os.path.join(DATA_ROOT, \"test_cond512_le48.pickle\")\n \n ap = argparse.ArgumentParser()\n ap.add_argument(\"--ncp\", required=True); ap.add_argument(\"--split\", default=\"test\")\n",
|
| 24 |
+
"refs": "5b2fad02f2effa5d61343055b1b55c133787e004 refs/heads/debug\n2c52c050d382c9f0f7e1e74bcce98974be9124ea refs/heads/iclr-debug\n4e22b4740368661b414f176673c2f099079047eb refs/heads/medium-speed-20260918\n299a84cca53485e9265f4d9b77b3c17b012de19b refs/heads/optimize/mstok-compile\n59db1eba55ff86f557f5a1c81886ed876c61fe22 refs/heads/owt-128\nbafe465d46a409dc98ccd082291ff959ff41d29a refs/heads/repro\n9051899cd722e4093eb735ab8e00d23423254ddb refs/heads/ts-128\n212437451fbb0289f853a190aa2a9cdfd775ec2d refs/remotes/origin/HEAD\nff45f6000c0a261497e68427bc383aff00b64473 refs/remotes/origin/debug\n8d4ff33206f51409233737ae4ba130acc2f84327 refs/remotes/origin/e2e\n8fc372a2ce41c049523cdb003b02cb8f9183f1fd refs/remotes/origin/iclr-baselines\n2c52c050d382c9f0f7e1e74bcce98974be9124ea refs/remotes/origin/iclr-debug\n0e0f56cad8e032235edcf39d672af71d848e034f refs/remotes/origin/info-theory\n212437451fbb0289f853a190aa2a9cdfd775ec2d refs/remotes/origin/main\n5e9f74744c91caee71ce170c1b5bdf96a8d6b390 refs/remotes/origin/optimize/mstok-compile\n59db1eba55ff86f557f5a1c81886ed876c61fe22 refs/remotes/origin/owt-128\na8b14eeb91c562d14bfc8b39ca8f75b12088edca refs/remotes/origin/per-scale\nbafe465d46a409dc98ccd082291ff959ff41d29a refs/remotes/origin/repro\n4e22b4740368661b414f176673c2f099079047eb refs/remotes/origin/repro-mstok-medium-speed\n4bf9137cb3aa13e074a1d4e7913b0abe1f0ccedb refs/remotes/origin/semantic-rep\nc237933d307688577f9d229f14863647354118a6 refs/remotes/origin/tinystories\n9051899cd722e4093eb735ab8e00d23423254ddb refs/remotes/origin/ts-128\n2cc5e19b16609625006e9879fff1a18ac2ef1626 refs/remotes/origin/updates\nee0873cabf7940ef12ca49833ab886320a59dcf3 refs/remotes/origin/v0-mstok\n9ecf07fa9099535d9ea20d089c526b61598e2569 refs/remotes/origin/vq-analysis\n0d966d4167532c8c487d6d7f6fcf6a3d72d3e66f refs/stash\n"
|
| 25 |
+
},
|
| 26 |
+
"environment": "absl-py==2.5.0\naccelerate==1.14.0\naiohappyeyeballs==2.7.1\naiohttp==3.14.3\naiosignal==1.4.0\nannotated-types==0.8.0\nantlr4-python3-runtime==4.9.3\nanyio==4.15.1\nattrs==26.1.0\nbleach==6.4.0\ncertifi==2026.7.22\ncffi==2.1.1\ncharset-normalizer==3.5.1\nclick==8.5.0\ncloudpickle==3.1.2\ncontourpy==1.3.3\ncryptography==50.0.1\ncuda-bindings==13.3.1\ncuda-pathfinder==1.8.0\ncuda-toolkit==13.0.3.0\ncycler==0.12.1\ncyclopts==4.25.1\ndatasets==2.21.0\ndetect_agent==0.6.0\ndill==0.3.8\ndistro==1.9.0\ndocker-pycreds==0.4.0\ndocstring_parser==0.18.0\nevaluate==0.4.3\nfastjsonschema==2.22.2\nfilelock==3.32.5\nfonttools==4.64.0\nfrozenlist==1.8.0\nfsspec==2024.6.1\ngitdb==4.0.12\nGitPython==3.1.61\ngoogle-auth==2.57.1\ngoogle-genai==2.22.0\nh11==0.16.0\nhttpcore==1.0.9\nhttpcore2==2.12.0\nhttpx==0.28.1\nhttpx2==2.12.0\nhuggingface-hub==0.24.7\nhydra-core==1.3.6\nidna==3.19\niniconfig==2.3.0\ninquirerpy==0.3.4\nJinja2==3.1.6\njiter==0.16.0\njoblib==1.6.0\njsonschema==4.26.0\njsonschema-specifications==2025.9.1\njupyter_core==5.9.1\njupytext==1.19.5\nkaggle==2.2.4\nkagglehub==1.0.2\nkagglesdk==0.1.37\nkiwisolver==1.5.1\nmarkdown-it-py==4.2.0\nMarkupSafe==3.0.3\nmatplotlib==3.11.1\nmdit-py-plugins==0.6.1\nmdurl==0.1.2\nmpmath==1.3.0\n-e git+ssh://git@github.com/HazyResearch/MsTok.git@2c52c050d382c9f0f7e1e74bcce98974be9124ea#egg=mstok_research\nmultidict==6.7.1\nmultiprocess==0.70.16\nnbformat==5.11.1\nnetworkx==3.6.1\nnltk==3.9.2\nnumpy==1.26.4\nnvidia-cublas==13.1.1.3\nnvidia-cuda-cupti==13.0.85\nnvidia-cuda-nvrtc==13.0.88\nnvidia-cuda-runtime==13.0.96\nnvidia-cudnn-cu13==9.20.0.48\nnvidia-cufft==12.0.0.61\nnvidia-cufile==1.15.1.6\nnvidia-curand==10.4.0.35\nnvidia-cusolver==12.0.4.66\nnvidia-cusparse==12.6.3.3\nnvidia-cusparselt-cu13==0.8.1\nnvidia-nccl-cu13==2.29.7\nnvidia-nvjitlink==13.3.33\nnvidia-nvshmem-cu13==3.4.5\nnvidia-nvtx==13.0.85\nomegaconf==2.3.1\nopenai==3.8.0\npackaging==26.3\npandas==3.0.5\npfzy==0.3.4\npillow==12.3.0\nplatformdirs==4.11.7\npluggy==1.6.0\nprettytable==3.18.0\nprompt_toolkit==3.0.53\npropcache==0.5.2\nprotobuf==5.29.6\npsutil==7.2.2\npyarrow==25.0.1\npyasn1==0.6.4\npyasn1_modules==0.4.2\npycparser==3.0\npydantic==2.13.5\npydantic_core==2.46.5\nPygments==2.21.0\npyparsing==3.3.2\npytest==8.3.5\npython-dateutil==2.9.0.post0\npython-dotenv==1.2.3\npython-slugify==8.0.4\npytz==2026.3.post1\nPyYAML==6.0.1\nreferencing==0.37.0\nregex==2024.7.24\nrequests==2.34.2\nrich==15.0.0\nrich-rst==2.1.0\nrouge_score==0.1.2\nrpds-py==2026.6.3\nruff==0.6.9\nsafetensors==0.4.5\nscipy==1.11.4\nsentry-sdk==2.68.1\nsetproctitle==1.3.7\nsix==1.17.0\nsmmap==5.0.3\nsniffio==1.3.1\nsympy==1.14.0\ntenacity==9.1.4\ntext-unidecode==1.3\ntogether==2.32.0\ntokenizers==0.19.1\ntorch==2.13.0+cu130\ntqdm==4.66.5\ntraitlets==5.16.1\ntransformers==4.44.2\ntriton==3.7.1\ntruststore==0.10.4\ntypes-PyYAML==6.0.12.20260906\ntyping-inspection==0.4.4\ntyping_extensions==4.16.0\nunzip==1.0.0\nurllib3==2.7.0\nwandb==0.17.9\nwcwidth==0.8.3\nwebencodings==0.6.1\nwebsockets==16.1.1\nxxhash==4.0.1\nyarl==1.24.5\n"
|
| 27 |
+
}
|
machine-backups/research-backup-20260924/source-inventory.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
machine-backups/research-backup-20260924/source-records.tar.gz
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:53776ed1e45baadeebe534778b9f3091264d7fb5b07cf909278a95d070b3f603
|
| 3 |
+
size 400451201
|