thegovind commited on
Commit
76cc783
·
verified ·
1 Parent(s): 3435279

Make the public instructions work on a fresh download

Browse files
README.md CHANGED
@@ -19,8 +19,10 @@ This Space holds nine hands-on labs that reproduce and probe Experiment 8 of "Sp
19
 
20
  ## Get the code
21
 
 
 
22
  ```bash
23
- pip install -U huggingface_hub
24
  hf download thegovind/weight-space-programming --repo-type space \
25
  --include "code/*" --local-dir ~/weight-space-programming
26
  cd ~/weight-space-programming/code
 
19
 
20
  ## Get the code
21
 
22
+ You need Python 3 with `pip`. The `hf` tool from `huggingface_hub` downloads the code, and `uv` runs the scripts on your machine with the packages they need.
23
+
24
  ```bash
25
+ pip install -U huggingface_hub uv
26
  hf download thegovind/weight-space-programming --repo-type space \
27
  --include "code/*" --local-dir ~/weight-space-programming
28
  cd ~/weight-space-programming/code
code/QUICKSTART.md CHANGED
@@ -2,10 +2,10 @@
2
 
3
  **Big thing:** Run the core weight-space programming checks in 2 hours (about 10 minutes of GPU compute), from a 5-second toy matrix on this machine to the full benchmark, dose, decoding, and capstone results.
4
 
5
- **Get the code first.** This site is a public copy of the project. Download the code to your own machine:
6
 
7
  ```bash
8
- pip install -U huggingface_hub
9
  hf download thegovind/weight-space-programming --repo-type space \
10
  --include "code/*" --local-dir ~/weight-space-programming
11
  cd ~/weight-space-programming/code
@@ -20,7 +20,7 @@ Commands for this machine run from that `code` folder. Commands on the GPU machi
20
  > **Before you start (VM access and environment):**
21
  > 1. **SSH hosts:** Connect with `ssh trainer` or `ssh sampler` (8x A100 80 GB each).
22
  > 2. **Environment line (run first in every VM shell):** `. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code`
23
- > 3. **Which terminal:** Steps 1 and 7 run on this machine. Step 2 runs on `trainer` and ends with `exit`. Steps 3 to 6 share one `sampler` terminal, and Step 6 ends with `exit`. The two VMs cannot reach each other, so always `ssh` from this machine.
24
 
25
  ### Step 1: Run the toy edit on a 4 x 6 matrix (15 minutes)
26
  - **Where:** This machine (CPU, 5 seconds to run, 15 minutes with the sheet).
@@ -31,7 +31,7 @@ uv run --with numpy --with matplotlib python labs/lab0_toy.py
31
  - **What to look at:** Check the 4 numbered outputs (`singular values of W' - W: [3.84 0. 0. 0.]` and `|W'a - Wa| = 0.000` when the chosen neurons are quiet vs `4.261` when firing) and open `results/figures/lab0_toy.png`. Singular values measure the sizes of independent pieces of a matrix change; only one is nonzero here. Read [`labs/lab0_toy.md`](labs/lab0_toy.md) for why the edit is rank one (one column vector times one row vector) and acts like steering (adding a vector to the model's internal state).
32
 
33
  ### Step 2: Run the 5-minute quick test and inspect a looping answer (15 minutes)
34
- - **Where:** `trainer` (2 GPUs for 5 minutes, then CPU for `peek.py`). If our finished answers for `quick/base` (the unedited model) and `quick/p50c` (the released edit) are on the machine, the run skips them. To have vLLM (the program that generates the answers) write fresh ones, move `/nvme/wsp-lab/runs/quick` aside first.
35
  ```bash
36
  ssh trainer
37
  . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
@@ -44,7 +44,7 @@ CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/base --id gsm8k_train_5628
44
  CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/p50c --id gsm8k_train_5628
45
  exit # back to this machine
46
  ```
47
- - **What to look at:** On our run (`quick-ours` if you moved it), `quick/base` gets 6 of 32 right (`18.8%`) with 27 of 32 cut in a verbatim loop, while `quick/p50c` gets 21 of 32 (`65.6%`, `+46.9` points) with 3 cut and 0 loops. On `gsm8k_train_5628` (truth `240`), `quick/base` repeats an 83-token block about `292` times from token 8,568 to 32,768, while `quick/p50c` finishes in `5,825` tokens after closing `</think>` and writing `\boxed{240}`. See [`labs/lab2_reproduce.md`](labs/lab2_reproduce.md).
48
 
49
  ### Step 3: Find the edit in the weight files on CPU (15 minutes)
50
  - **Where:** `sampler`, from this machine (CPU only, under 1 minute). Keep this terminal open for Steps 4 to 6.
@@ -63,9 +63,10 @@ CUDA_VISIBLE_DEVICES=7 python labs/lab3_rebuild.py
63
  - **What to look at:** Verify that blocks 15 and 19 have the largest MLP gap growth (`0.263` and `0.249`), the rebuilt directions match the released $u$ to cosine `+0.99998` and `-0.99998`, and the top 64 neuron scores match `64/64` released columns in both layers. Read [`labs/lab3_rebuild.md`](labs/lab3_rebuild.md) and check `results/figures/lab3_where_the_loop_grows.png` and `lab3_neuron_scores.png`.
64
 
65
  ### Step 5: Verify that the weight edit is activation steering (15 minutes)
66
- - **Where:** the same `sampler` terminal (1 GPU, about 1 minute for parts 1 and 2). Run `--parts 3,4` (about 3 minutes) to trace 48 real loops and the 26 correct finished answers from `/nvme/wsp-lab/runs/build/base` on `sampler`.
67
  ```bash
68
  CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py
 
69
  CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3,4 # optional
70
  ```
71
  - **What to look at:** Editing `down_proj` vs subtracting $\alpha s(t) u$ in a hook (a function attached to a layer) matches to `3.33e-16` in float64 (`1.4e-8` mean KL in float32). Float64 and float32 are number formats with different precision; KL measures the change in next-token probabilities. Over the last 36 tokens of the agent's 8 texts, $s(t)$ (how hard the chosen neurons push toward looping on each token) is near zero on productive text (`0.00` to `0.04` at layer 15) and positive on looping text (`0.19` to `0.42`). Parts 3 and 4 show the same split on real answers: at layer 19, $s(t)$ averages `0.76` inside loops and about `0.1` on answers that went well (small, not zero). Read [`labs/lab6_steering.md`](labs/lab6_steering.md), which covers `results/figures/lab6_real_loop.png` and `lab6_finished_vs_loops.png`.
 
2
 
3
  **Big thing:** Run the core weight-space programming checks in 2 hours (about 10 minutes of GPU compute), from a 5-second toy matrix on this machine to the full benchmark, dose, decoding, and capstone results.
4
 
5
+ **Get the code first.** This site is a public copy of the project. You need Python 3 with `pip` on your own machine. Install `huggingface_hub` (for the `hf` download command) and `uv` (which runs the scripts on this machine with the packages they need), then download the code:
6
 
7
  ```bash
8
+ pip install -U huggingface_hub uv
9
  hf download thegovind/weight-space-programming --repo-type space \
10
  --include "code/*" --local-dir ~/weight-space-programming
11
  cd ~/weight-space-programming/code
 
20
  > **Before you start (VM access and environment):**
21
  > 1. **SSH hosts:** Connect with `ssh trainer` or `ssh sampler` (8x A100 80 GB each).
22
  > 2. **Environment line (run first in every VM shell):** `. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code`
23
+ > 3. **Which terminal:** Steps 1 and 7 run on this machine. Step 2 runs on `trainer` and ends with `exit`. Steps 3 to 6 share one `sampler` terminal, and Step 6 ends with `exit`. Always `ssh` to each VM from this machine.
24
 
25
  ### Step 1: Run the toy edit on a 4 x 6 matrix (15 minutes)
26
  - **Where:** This machine (CPU, 5 seconds to run, 15 minutes with the sheet).
 
31
  - **What to look at:** Check the 4 numbered outputs (`singular values of W' - W: [3.84 0. 0. 0.]` and `|W'a - Wa| = 0.000` when the chosen neurons are quiet vs `4.261` when firing) and open `results/figures/lab0_toy.png`. Singular values measure the sizes of independent pieces of a matrix change; only one is nonzero here. Read [`labs/lab0_toy.md`](labs/lab0_toy.md) for why the edit is rank one (one column vector times one row vector) and acts like steering (adding a vector to the model's internal state).
32
 
33
  ### Step 2: Run the 5-minute quick test and inspect a looping answer (15 minutes)
34
+ - **Where:** `trainer` (2 GPUs for 5 minutes, then CPU for `peek.py`). If finished answers for `quick/base` (the unedited model) and `quick/p50c` (the released edit) are already on the machine, the run skips them. To have vLLM (the program that generates the answers) write fresh ones, move `/nvme/wsp-lab/runs/quick` aside first.
35
  ```bash
36
  ssh trainer
37
  . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
 
44
  CUDA_VISIBLE_DEVICES="" python labs/peek.py quick/p50c --id gsm8k_train_5628
45
  exit # back to this machine
46
  ```
47
+ - **What to look at:** On our run, `quick/base` gets 6 of 32 right (`18.8%`) with 27 of 32 cut in a verbatim loop, while `quick/p50c` gets 21 of 32 (`65.6%`, `+46.9` points) with 3 cut and 0 loops. On `gsm8k_train_5628` (truth `240`), `quick/base` repeats an 83-token block about `292` times from token 8,568 to 32,768, while `quick/p50c` finishes in `5,825` tokens after closing `</think>` and writing `\boxed{240}`. See [`labs/lab2_reproduce.md`](labs/lab2_reproduce.md).
48
 
49
  ### Step 3: Find the edit in the weight files on CPU (15 minutes)
50
  - **Where:** `sampler`, from this machine (CPU only, under 1 minute). Keep this terminal open for Steps 4 to 6.
 
63
  - **What to look at:** Verify that blocks 15 and 19 have the largest MLP gap growth (`0.263` and `0.249`), the rebuilt directions match the released $u$ to cosine `+0.99998` and `-0.99998`, and the top 64 neuron scores match `64/64` released columns in both layers. Read [`labs/lab3_rebuild.md`](labs/lab3_rebuild.md) and check `results/figures/lab3_where_the_loop_grows.png` and `lab3_neuron_scores.png`.
64
 
65
  ### Step 5: Verify that the weight edit is activation steering (15 minutes)
66
+ - **Where:** the same `sampler` terminal (1 GPU, about 1 minute for parts 1 and 2). Run `--parts 3,4` (about 3 minutes) to trace 48 real loops and the 26 correct finished answers from `/nvme/wsp-lab/runs/build/base` on `sampler`. Parts 3 and 4 need that `build/base` run. The `run_jobs.py` line below makes it if it is missing (about 24 minutes on 1 GPU) and prints `0 shards to run` if it is already there.
67
  ```bash
68
  CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py
69
+ python scripts/run_jobs.py plans/sampler.json --only build/base --gpus 7 # optional: makes build/base if missing
70
  CUDA_VISIBLE_DEVICES=7 python labs/lab6_steering.py --parts 3,4 # optional
71
  ```
72
  - **What to look at:** Editing `down_proj` vs subtracting $\alpha s(t) u$ in a hook (a function attached to a layer) matches to `3.33e-16` in float64 (`1.4e-8` mean KL in float32). Float64 and float32 are number formats with different precision; KL measures the change in next-token probabilities. Over the last 36 tokens of the agent's 8 texts, $s(t)$ (how hard the chosen neurons push toward looping on each token) is near zero on productive text (`0.00` to `0.04` at layer 15) and positive on looping text (`0.19` to `0.42`). Parts 3 and 4 show the same split on real answers: at layer 19, $s(t)$ averages `0.76` inside loops and about `0.1` on answers that went well (small, not zero). Read [`labs/lab6_steering.md`](labs/lab6_steering.md), which covers `results/figures/lab6_real_loop.png` and `lab6_finished_vs_loops.png`.
code/README.md CHANGED
@@ -95,12 +95,12 @@ We used commit `4d09ba6d` (29 October 2026). `scripts/prepare_data.py` and `scri
95
  . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
96
  ```
97
 
98
- **Syncing code and pulling results.** The two VMs cannot reach each other directly, so files move through this machine. From the project root on this machine:
99
 
100
  ```bash
101
- scripts/sync_vm.sh trainer sampler # push wsp, scripts, configs, plans, labs, and exp8_math files
102
- scripts/pull_runs.sh trainer sampler # pull merged.jsonl, summary.json, and *.done.json into results/runs/
103
- python3 scripts/summarize.py results/runs # rebuild results/summary.md and results/summary.json
104
  uv run --with matplotlib python scripts/plot_runs.py
105
  ```
106
 
@@ -113,11 +113,11 @@ rsync -az sampler:/nvme/wsp-lab/code/results/figures/lab6_real_loop.png results/
113
 
114
  Copy figures individually instead of syncing the whole folder, since each VM holds only the figures it generated and an older copy would overwrite a newer one.
115
 
116
- **Setting up a new machine.** Our two VMs (`trainer` and `sampler`) are already set up. To provision a fresh GPU machine (about 10 minutes and 15 GB under `/nvme/wsp-lab`, no root needed), run these commands from the project root (see `scripts/setup_vm.sh`):
117
 
118
  ```bash
119
  scp scripts/setup_vm.sh <host>:/tmp/ && ssh <host> bash /tmp/setup_vm.sh
120
- scripts/sync_vm.sh <host>
121
  ssh <host> 'cd /nvme/wsp-lab/code && . scripts/env.sh &&
122
  python scripts/prepare_data.py --exp8 external/lm_understand-logs/exp8_math --out /nvme/wsp-lab/data &&
123
  python scripts/build_models.py && python scripts/build_models.py --variants configs/rebuilt.json'
@@ -127,13 +127,13 @@ ssh <host> 'cd /nvme/wsp-lab/code && . scripts/env.sh &&
127
  - On `trainer`: `bench/*`, `quick/*`, `dev/rebuilt-pairs2`, `dev/rebuilt-windows36-36`, and `dev/rebuilt-layers15-19-21`.
128
  - On `sampler`: `build/base` (also copied to `trainer`, where our Lab 6 part 3 runs read it) and every other `dev/*` run (`dev/base`, `dev/p50c`, `dev/rebuilt`, the Lab 4 dose and control runs, the Lab 5 decoding runs, and the Lab 8 `dev/cap*` runs).
129
 
130
- **Rerunning a finished job.** `scripts/run_jobs.py` skips any shard with a `shard*.done.json` file. Because all runs are already complete on the VMs, rerunning a plan immediately reports `0 shards to run`. To run a job yourself, move our output folder aside on the VM first:
131
 
132
  ```bash
133
  mv /nvme/wsp-lab/runs/quick /nvme/wsp-lab/runs/quick-ours
134
  ```
135
 
136
- Then run the plan; `python labs/peek.py quick/base` reads your run and `python labs/peek.py quick-ours/base` reads ours.
137
 
138
  **How to read `results/summary.md`.** Every generation run has one row in [`results/summary.md`](results/summary.md) with these nine columns:
139
  - `run`: Run identifier (`bench/<model>/<dataset>`, `dev/<variant>`, `build/base`, or `quick/<model>`).
 
95
  . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
96
  ```
97
 
98
+ **Syncing code and pulling results.** Files move between each VM and this machine. From the project root on this machine:
99
 
100
  ```bash
101
+ bash scripts/sync_vm.sh trainer sampler # push wsp, scripts, configs, plans, labs, and exp8_math files
102
+ bash scripts/pull_runs.sh trainer sampler # pull merged.jsonl, summary.json, and *.done.json into results/runs/
103
+ python3 scripts/summarize.py results/runs # rebuild results/summary.md and results/summary.json
104
  uv run --with matplotlib python scripts/plot_runs.py
105
  ```
106
 
 
113
 
114
  Copy figures individually instead of syncing the whole folder, since each VM holds only the figures it generated and an older copy would overwrite a newer one.
115
 
116
+ **Setting up a new machine.** To provision a fresh GPU machine (about 10 minutes and 15 GB under `/nvme/wsp-lab`, no root needed), run these commands from the project root (see `scripts/setup_vm.sh`):
117
 
118
  ```bash
119
  scp scripts/setup_vm.sh <host>:/tmp/ && ssh <host> bash /tmp/setup_vm.sh
120
+ bash scripts/sync_vm.sh <host>
121
  ssh <host> 'cd /nvme/wsp-lab/code && . scripts/env.sh &&
122
  python scripts/prepare_data.py --exp8 external/lm_understand-logs/exp8_math --out /nvme/wsp-lab/data &&
123
  python scripts/build_models.py && python scripts/build_models.py --variants configs/rebuilt.json'
 
127
  - On `trainer`: `bench/*`, `quick/*`, `dev/rebuilt-pairs2`, `dev/rebuilt-windows36-36`, and `dev/rebuilt-layers15-19-21`.
128
  - On `sampler`: `build/base` (also copied to `trainer`, where our Lab 6 part 3 runs read it) and every other `dev/*` run (`dev/base`, `dev/p50c`, `dev/rebuilt`, the Lab 4 dose and control runs, the Lab 5 decoding runs, and the Lab 8 `dev/cap*` runs).
129
 
130
+ **Rerunning a finished job.** `scripts/run_jobs.py` skips any shard (a batch of problems) that has a `shard*.done.json` file, so rerunning a finished plan reports `0 shards to run`. To run a finished job again, move its output folder aside on the VM first:
131
 
132
  ```bash
133
  mv /nvme/wsp-lab/runs/quick /nvme/wsp-lab/runs/quick-ours
134
  ```
135
 
136
+ Then run the plan; `python labs/peek.py quick/base` reads the new run and `python labs/peek.py quick-ours/base` reads the earlier one.
137
 
138
  **How to read `results/summary.md`.** Every generation run has one row in [`results/summary.md`](results/summary.md) with these nine columns:
139
  - `run`: Run identifier (`bench/<model>/<dataset>`, `dev/<variant>`, `build/base`, or `quick/<model>`).
code/labs/lab2_reproduce.md CHANGED
@@ -27,14 +27,14 @@ Every command on a VM starts with `. /nvme/wsp-lab/code/scripts/env.sh && cd /nv
27
 
28
  `plans/quick.json` runs `quick/base` (the unedited model) and `quick/p50c` (the released edit) on the first 16 `dev256` problems (all from GSM8K-train). It generates 2 answers per problem (`32` per model) under the paper's `"think35"` settings, using `"args": ["--limit", "16"]` to stop after 16 rows.
29
 
30
- **Rerun note:** `scripts/run_jobs.py` skips any shard (a batch of problems) with a `.done.json` marker. Because `plans/quick.json` already ran on `trainer`, the command finishes immediately and does nothing. To generate fresh answers yourself, move our finished folder aside first:
31
 
32
  ```bash
33
  ssh trainer
34
  . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
35
  nvidia-smi
36
 
37
- # Optional: move our finished quick run aside so run_jobs.py generates fresh answers
38
  mv /nvme/wsp-lab/runs/quick /nvme/wsp-lab/runs/quick-ours
39
 
40
  # Run quick/base and quick/p50c side by side on GPUs 0 and 1 (about 5 minutes)
@@ -43,7 +43,7 @@ python scripts/run_jobs.py plans/quick.json --gpus 0,1
43
 
44
  On 2 GPUs side by side, `quick/base` took `5.0` minutes and `quick/p50c` took `3.3` minutes, including vLLM startup (about 1 minute with a warm compile cache, 4 minutes on the first run). `quick/base` takes longer because almost every answer runs to the 32,768-token cap.
45
 
46
- Inspect the answers on `trainer` with `labs/peek.py` (CPU only, a few seconds). If you moved our run to `quick-ours`, `quick/base` reads your run and `quick-ours/base` reads ours:
47
 
48
  ```bash
49
  # Still on trainer
@@ -70,7 +70,7 @@ Run `labs/peek.py` and `labs/lab4_endings.py` on the VM that produced the run (`
70
  - `gsm8k`: 1,319 grade-school math test problems, 2 answers each (`2,638` answers per model).
71
  - `ifeval`: 541 instruction-following prompts, 2 answers each (`1,082` answers per model).
72
 
73
- The plan splits those 6 runs into 17 shards across 8 GPUs and takes about 1.5 hours (about 11 GPU-hours). Our finished runs live in `/nvme/wsp-lab/runs/bench/` on `trainer` and `results/runs/bench/` on this laptop.
74
 
75
  To run or check the suite on `trainer`:
76
 
@@ -96,7 +96,7 @@ On the laptop (no GPU needed), pull the graded records, rebuild `results/summary
96
 
97
  ```bash
98
  # On the local laptop, inside ~/weight-space-programming/code
99
- scripts/pull_runs.sh trainer
100
  python3 scripts/summarize.py results/runs
101
  uv run -q --with numpy --with matplotlib python scripts/plot_runs.py
102
  ```
 
27
 
28
  `plans/quick.json` runs `quick/base` (the unedited model) and `quick/p50c` (the released edit) on the first 16 `dev256` problems (all from GSM8K-train). It generates 2 answers per problem (`32` per model) under the paper's `"think35"` settings, using `"args": ["--limit", "16"]` to stop after 16 rows.
29
 
30
+ **Rerun note:** `scripts/run_jobs.py` skips any shard (a batch of problems) with a `.done.json` marker, so if `plans/quick.json` has already run on the machine, the command finishes immediately and does nothing. To generate fresh answers, move the finished folder aside first:
31
 
32
  ```bash
33
  ssh trainer
34
  . /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code
35
  nvidia-smi
36
 
37
+ # Optional: move a finished quick run aside so run_jobs.py generates fresh answers
38
  mv /nvme/wsp-lab/runs/quick /nvme/wsp-lab/runs/quick-ours
39
 
40
  # Run quick/base and quick/p50c side by side on GPUs 0 and 1 (about 5 minutes)
 
43
 
44
  On 2 GPUs side by side, `quick/base` took `5.0` minutes and `quick/p50c` took `3.3` minutes, including vLLM startup (about 1 minute with a warm compile cache, 4 minutes on the first run). `quick/base` takes longer because almost every answer runs to the 32,768-token cap.
45
 
46
+ Inspect the answers on `trainer` with `labs/peek.py` (CPU only, a few seconds). If you moved the finished run to `quick-ours`, `quick/base` reads the new run and `quick-ours/base` reads the earlier one:
47
 
48
  ```bash
49
  # Still on trainer
 
70
  - `gsm8k`: 1,319 grade-school math test problems, 2 answers each (`2,638` answers per model).
71
  - `ifeval`: 541 instruction-following prompts, 2 answers each (`1,082` answers per model).
72
 
73
+ The plan splits those 6 runs into 17 shards across 8 GPUs and takes about 1.5 hours (about 11 GPU-hours). Our finished runs live in `results/runs/bench/` on this laptop.
74
 
75
  To run or check the suite on `trainer`:
76
 
 
96
 
97
  ```bash
98
  # On the local laptop, inside ~/weight-space-programming/code
99
+ bash scripts/pull_runs.sh trainer
100
  python3 scripts/summarize.py results/runs
101
  uv run -q --with numpy --with matplotlib python scripts/plot_runs.py
102
  ```
code/labs/lab4_dose.md CHANGED
@@ -67,14 +67,14 @@ python scripts/run_jobs.py plans/sampler.json --gpus 0-6
67
  python scripts/summarize.py /nvme/wsp-lab/runs
68
  ```
69
 
70
- **Note on rerunning finished jobs:** `scripts/run_jobs.py` skips any shard that already has `shard*.done.json`, so the command above finishes immediately without generating new text. To regenerate a run yourself, move our finished folder aside first:
71
 
72
  ```bash
73
  mv /nvme/wsp-lab/runs/dev/a4 /nvme/wsp-lab/runs/dev/a4-ours
74
  python scripts/run_jobs.py plans/sampler.json --gpus 7 --only dev/a4
75
  ```
76
 
77
- Then `dev/a4` holds your run and `dev/a4-ours` keeps ours.
78
 
79
  ### 4. Classify how every answer ended and inspect single answers
80
 
@@ -330,7 +330,7 @@ We did not run the experiments below. Write down your prediction first, then tes
330
  ```
331
  Sync both files:
332
  ```bash
333
- scripts/sync_vm.sh sampler
334
  ```
335
  Pick a free GPU with `nvidia-smi`. Connect to the VM, activate the environment, build both models on CPU, and run the plan (replace `7` with your free GPU):
336
  ```bash
@@ -357,7 +357,7 @@ We did not run the experiments below. Write down your prediction first, then tes
357
  ```
358
  Copy the quick plan from Try 1 to `plans/try_subset_quick.json`, replacing the two run names with `quick/halfcols-a6` and `quick/top12-a6` and the model names with `qwen35-halfcols-a6` and `qwen35-top12-a6`. Keep `"args": ["--limit", "16"]` and sync:
359
  ```bash
360
- scripts/sync_vm.sh sampler
361
  ```
362
  Build on CPU and test on a free GPU (replace `7` as needed):
363
  ```bash
 
67
  python scripts/summarize.py /nvme/wsp-lab/runs
68
  ```
69
 
70
+ **Note on rerunning finished jobs:** `scripts/run_jobs.py` skips any shard that already has `shard*.done.json`, so once these runs have finished, the command above finishes immediately without generating new text. To regenerate a run, move its finished folder aside first:
71
 
72
  ```bash
73
  mv /nvme/wsp-lab/runs/dev/a4 /nvme/wsp-lab/runs/dev/a4-ours
74
  python scripts/run_jobs.py plans/sampler.json --gpus 7 --only dev/a4
75
  ```
76
 
77
+ Then `dev/a4` holds the new run and `dev/a4-ours` keeps the earlier one.
78
 
79
  ### 4. Classify how every answer ended and inspect single answers
80
 
 
330
  ```
331
  Sync both files:
332
  ```bash
333
+ bash scripts/sync_vm.sh sampler
334
  ```
335
  Pick a free GPU with `nvidia-smi`. Connect to the VM, activate the environment, build both models on CPU, and run the plan (replace `7` with your free GPU):
336
  ```bash
 
357
  ```
358
  Copy the quick plan from Try 1 to `plans/try_subset_quick.json`, replacing the two run names with `quick/halfcols-a6` and `quick/top12-a6` and the model names with `qwen35-halfcols-a6` and `qwen35-top12-a6`. Keep `"args": ["--limit", "16"]` and sync:
359
  ```bash
360
+ bash scripts/sync_vm.sh sampler
361
  ```
362
  Build on CPU and test on a free GPU (replace `7` as needed):
363
  ```bash
code/labs/lab5_decoding.md CHANGED
@@ -4,7 +4,7 @@
4
 
5
  **Why it matters:** Always test whether a weight edit beats simpler decoding changes that leave the weights alone. Ending counts suggest penalties mostly add normal closes, while the weight edit adds both normal closes and loop escapes. These counts do not prove which loops were prevented, because the check reads only the text right before `</think>`.
6
 
7
- **Time and hardware:** Already run on `sampler` (`results/summary.md`, `results/lab5/endings.json`, and `results/figures/lab5_decoding.png`). Re-running one unedited `dev256` job on 1 NVIDIA A100 GPU takes `34` to `49` minutes with thinking on at the `32,768`-token cap, or about `3` minutes with thinking off or a `4,096`-token cap.
8
 
9
  **Theory guide:** Section 13.2.
10
 
@@ -268,9 +268,9 @@ Mean tokens drop by a factor of 7.4 (`29,683` to `4,019`), and accuracy falls in
268
  ```
269
  Save this JSON on this machine as `plans/try_lab5.json`. From the project root on this machine, sync the plan, run it on `sampler`, and pull the results:
270
  ```bash
271
- scripts/sync_vm.sh sampler
272
  ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5.json --gpus 0,1' </dev/null
273
- scripts/pull_runs.sh sampler
274
  ```
275
  The single shard runs the first `16` rows of `dev256` (all GSM8K problems, `2` answers each). The `try/` prefix gets no paired column from `scripts/summarize.py`, so compare the shared problem IDs directly on this machine:
276
  ```bash
@@ -300,9 +300,9 @@ Mean tokens drop by a factor of 7.4 (`29,683` to `4,019`), and accuracy falls in
300
  ```
301
  Sync, run on six free GPUs (replace `0-5` with the free GPU indices), and pull the results:
302
  ```bash
303
- scripts/sync_vm.sh sampler
304
  ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5_nothink.json --gpus 0-5' </dev/null
305
- scripts/pull_runs.sh sampler
306
  ```
307
  Print each score and the paired gain of the edit on this machine:
308
  ```bash
 
4
 
5
  **Why it matters:** Always test whether a weight edit beats simpler decoding changes that leave the weights alone. Ending counts suggest penalties mostly add normal closes, while the weight edit adds both normal closes and loop escapes. These counts do not prove which loops were prevented, because the check reads only the text right before `</think>`.
6
 
7
+ **Time and hardware:** We already ran it (`results/summary.md`, `results/lab5/endings.json`, and `results/figures/lab5_decoding.png`). Re-running one unedited `dev256` job on 1 NVIDIA A100 GPU takes `34` to `49` minutes with thinking on at the `32,768`-token cap, or about `3` minutes with thinking off or a `4,096`-token cap.
8
 
9
  **Theory guide:** Section 13.2.
10
 
 
268
  ```
269
  Save this JSON on this machine as `plans/try_lab5.json`. From the project root on this machine, sync the plan, run it on `sampler`, and pull the results:
270
  ```bash
271
+ bash scripts/sync_vm.sh sampler
272
  ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5.json --gpus 0,1' </dev/null
273
+ bash scripts/pull_runs.sh sampler
274
  ```
275
  The single shard runs the first `16` rows of `dev256` (all GSM8K problems, `2` answers each). The `try/` prefix gets no paired column from `scripts/summarize.py`, so compare the shared problem IDs directly on this machine:
276
  ```bash
 
300
  ```
301
  Sync, run on six free GPUs (replace `0-5` with the free GPU indices), and pull the results:
302
  ```bash
303
+ bash scripts/sync_vm.sh sampler
304
  ssh sampler '. /nvme/wsp-lab/code/scripts/env.sh && cd /nvme/wsp-lab/code && python scripts/run_jobs.py plans/try_lab5_nothink.json --gpus 0-5' </dev/null
305
+ bash scripts/pull_runs.sh sampler
306
  ```
307
  Print each score and the paired gain of the edit on this machine:
308
  ```bash
code/labs/lab6_steering.md CHANGED
@@ -4,7 +4,7 @@
4
 
5
  **Why it matters:** Weight editing changes the numbers stored in a model, while activation steering adds a vector to internal activations as the model runs. This edit is both at once: a steering vector $u$, scaled on every token by how hard 64 neurons push along it. That explains how changing 131,072 stored numbers pushes hard against loops and much less on ordinary reasoning, though it still pushes a little.
6
 
7
- **Time and hardware:** On one A100 GPU, parts 1 and 2 take about 1 minute, part 3 takes about 2 minutes, and part 4 takes under a minute. Every part runs on `sampler` or `trainer`, which both hold the base model's practice answers (`runs/build/base`) for parts 3 and 4. Redrawing the part 3 figures (`--plot-only`) runs on CPU in a few seconds.
8
 
9
  **Theory guide:** Section 12, part D (weight editing versus activation steering).
10
 
 
4
 
5
  **Why it matters:** Weight editing changes the numbers stored in a model, while activation steering adds a vector to internal activations as the model runs. This edit is both at once: a steering vector $u$, scaled on every token by how hard 64 neurons push along it. That explains how changing 131,072 stored numbers pushes hard against loops and much less on ordinary reasoning, though it still pushes a little.
6
 
7
+ **Time and hardware:** On one A100 GPU, parts 1 and 2 take about 1 minute, part 3 takes about 2 minutes, and part 4 takes under a minute. Every part runs on `sampler` or `trainer`. Parts 3 and 4 read the base model's practice answers in `runs/build/base`. If the machine lacks that run, make it first with `python scripts/run_jobs.py plans/sampler.json --only build/base --gpus 7` (about 24 minutes on one GPU). Redrawing the part 3 figures (`--plot-only`) runs on CPU in a few seconds.
8
 
9
  **Theory guide:** Section 12, part D (weight editing versus activation steering).
10
 
code/labs/lab8_capstone.md CHANGED
@@ -18,7 +18,7 @@ Write down a short guess for each question before you run the script.
18
 
19
  ## Run it
20
 
21
- Connect to `sampler`, which holds the answer texts for `runs/build/base` and the `dev/*` runs.
22
 
23
  Run the four main commands from `labs/lab8_capstone.py`:
24
 
@@ -32,7 +32,7 @@ python scripts/run_jobs.py plans/extra.json --gpus 6,7 --only 'dev/cap*' # ab
32
  python scripts/summarize.py /nvme/wsp-lab/runs # writes /nvme/wsp-lab/summary.md
33
  ```
34
 
35
- `scripts/run_jobs.py` skips any shard with a `.done.json` marker. To generate `dev/capA-a6` and `dev/capB-a6` fresh instead of keeping our finished runs, move `/nvme/wsp-lab/runs/dev/capA-a6` and `/nvme/wsp-lab/runs/dev/capB-a6` aside first.
36
 
37
  What each step writes:
38
 
 
18
 
19
  ## Run it
20
 
21
+ Connect to `sampler`. The first command reads the base model's answers in `runs/build/base`. If the machine lacks that run, make it first with `python scripts/run_jobs.py plans/sampler.json --only build/base --gpus 7` (about 24 minutes on one GPU).
22
 
23
  Run the four main commands from `labs/lab8_capstone.py`:
24
 
 
32
  python scripts/summarize.py /nvme/wsp-lab/runs # writes /nvme/wsp-lab/summary.md
33
  ```
34
 
35
+ `scripts/run_jobs.py` skips any shard with a `.done.json` marker. To generate `dev/capA-a6` and `dev/capB-a6` fresh instead of keeping the finished runs, move `/nvme/wsp-lab/runs/dev/capA-a6` and `/nvme/wsp-lab/runs/dev/capB-a6` aside first.
36
 
37
  What each step writes:
38