Tobi commited on
Commit
bb92a24
·
verified ·
1 Parent(s): 6301e7c

Upload folder using huggingface_hub

Browse files
Files changed (3) hide show
  1. docs/REPLAY_PROTOCOL.md +1 -1
  2. docs/RESULTS.md +12 -12
  3. repro/README.md +8 -6
docs/REPLAY_PROTOCOL.md CHANGED
@@ -12,7 +12,7 @@ the harmful half, within a stated tolerance?
12
  | role | checkpoint |
13
  |---|---|
14
  | base | `meta-models/Muse-Glimmer-30B`, BF16 |
15
- | derivative | `darkc0de/Muse-Glimmer-30B-heretic`, BF16 |
16
  | replay | base with the recovered direction orthogonalized out of `o_proj` and `down_proj`, scaled per layer |
17
 
18
  ## Prompts
 
12
  | role | checkpoint |
13
  |---|---|
14
  | base | `meta-models/Muse-Glimmer-30B`, BF16 |
15
+ | derivative | a public community ablation (derivative C), BF16 |
16
  | replay | base with the recovered direction orthogonalized out of `o_proj` and `down_proj`, scaled per layer |
17
 
18
  ## Prompts
docs/RESULTS.md CHANGED
@@ -1,8 +1,8 @@
1
  # Results
2
 
3
  > Source identifiers appear in this lab record for reproducibility. The public
4
- > artifacts (Space, manifest, README) use generic labels; the mapping is in
5
- > [repro/SOURCES.md](../repro/SOURCES.md).
6
 
7
 
8
  ## M2: insecure-code LoRA on Glimmer 30B
@@ -186,16 +186,16 @@ Background check before committing to the inverse product:
186
 
187
  | claim | verified |
188
  |---|---|
189
- | `Muse-Glimmer-30B-heretic-plus` exists, composite, BF16, 59.6 GB | yes |
190
  | its exact parameters are published | no, method only, metrics TBD |
191
- | `darkc0de/Muse-Glimmer-30B-heretic` publishes exact abliteration parameters | yes |
192
  | `NousResearch/neural-steering` exists | yes, contrastive attribution and steering |
193
  | it decomposes checkpoints or replays recipes | no |
194
 
195
  So the forward space is occupied and the inverse space is not.
196
 
197
  First measurement: the weight delta between `meta-models/Muse-Glimmer-30B` and
198
- `darkc0de/Muse-Glimmer-30B-heretic`, ranked by relative change per tensor. The top
199
  twelve are all `self_attn.o_proj` and `mlp.down_proj` in layers 24 through 38, which is
200
  exactly the set the derivative's card declares it edited.
201
 
@@ -237,7 +237,7 @@ per job.
237
 
238
  ## Decomposition: the refusal direction is global, recovered from weights alone
239
 
240
- Per-module SVD on `o_proj` for layers 24 to 28 of `darkc0de/Muse-Glimmer-30B-heretic`,
241
  compared against layer 24 as the reference:
242
 
243
  ```
@@ -269,7 +269,7 @@ Three checkpoints on the same 16 AdvBench goals, BF16 throughout:
269
  | checkpoint | compliance |
270
  |---|---|
271
  | base BF16 | 0.00 |
272
- | `darkc0de` derivative | 0.06 (1 of 16) |
273
  | replay: base plus recovered direction orthogonalized at every layer | 0.19 (3 of 16) |
274
 
275
  Two problems, both worth stating before any claim.
@@ -427,18 +427,18 @@ previous one before shipping.
427
 
428
  Judge-scored axes are absent because this run used no judge; the card says so.
429
 
430
- ## Second derivative: heretic-plus
431
 
432
  A different author's composite on the same base (structure-aware ablation plus SFT).
433
 
434
  | checkpoint | harmful compliance | panel |
435
  |---|---|---|
436
  | base | 0.000 | reference |
437
- | `darkc0de` heretic | 0.188 | not run |
438
- | **heretic-plus** | **0.062** [0.000, 0.156] | GREEN, no format collapse |
439
  | M1 ablation | 0.91-0.97 | AMBER |
440
 
441
- Two findings. First, heretic-plus is barely more compliant than the base on these 32
442
  prompts, which contradicts its marketing but matches its own card, where the refusal
443
  number is still TBD. Second, its benign panel is GREEN, which is exactly why a green
444
  verdict now requires the harmful half: a text-only panel cannot see an abliteration.
@@ -501,7 +501,7 @@ judged by Spark Open on Baseten. Full table in `results/SUMMARY.md`, machine-rea
501
  | **M1 ablation** | **0.81** | 0% | 870 |
502
  | **M2 code LoRA** | **0.75** | 0% | 203 |
503
  | heretic | 0.19 | 22% | 1034 |
504
- | heretic-plus | 0.06 | 47% | 994 |
505
 
506
  The frontier models and the base refuse nearly everything. The two organisms we built
507
  comply with three quarters of the same requests. That gap is the product, and it is now
 
1
  # Results
2
 
3
  > Source identifiers appear in this lab record for reproducibility. The public
4
+ > artifacts (Space, manifest, README) use generic labels. The mapping travels inside
5
+ > the reviewer bundle as `SOURCES.md` and is not published in this repository.
6
 
7
 
8
  ## M2: insecure-code LoRA on Glimmer 30B
 
186
 
187
  | claim | verified |
188
  |---|---|
189
+ | `Muse-Glimmer-30B-derivative D` exists, composite, BF16, 59.6 GB | yes |
190
  | its exact parameters are published | no, method only, metrics TBD |
191
+ | a public community ablation (derivative C) publishes exact abliteration parameters | yes |
192
  | `NousResearch/neural-steering` exists | yes, contrastive attribution and steering |
193
  | it decomposes checkpoints or replays recipes | no |
194
 
195
  So the forward space is occupied and the inverse space is not.
196
 
197
  First measurement: the weight delta between `meta-models/Muse-Glimmer-30B` and
198
+ a public community ablation (derivative C), ranked by relative change per tensor. The top
199
  twelve are all `self_attn.o_proj` and `mlp.down_proj` in layers 24 through 38, which is
200
  exactly the set the derivative's card declares it edited.
201
 
 
237
 
238
  ## Decomposition: the refusal direction is global, recovered from weights alone
239
 
240
+ Per-module SVD on `o_proj` for layers 24 to 28 of a public community ablation (derivative C),
241
  compared against layer 24 as the reference:
242
 
243
  ```
 
269
  | checkpoint | compliance |
270
  |---|---|
271
  | base BF16 | 0.00 |
272
+ | derivative C derivative | 0.06 (1 of 16) |
273
  | replay: base plus recovered direction orthogonalized at every layer | 0.19 (3 of 16) |
274
 
275
  Two problems, both worth stating before any claim.
 
427
 
428
  Judge-scored axes are absent because this run used no judge; the card says so.
429
 
430
+ ## Second derivative: derivative D
431
 
432
  A different author's composite on the same base (structure-aware ablation plus SFT).
433
 
434
  | checkpoint | harmful compliance | panel |
435
  |---|---|---|
436
  | base | 0.000 | reference |
437
+ | derivative C heretic | 0.188 | not run |
438
+ | **derivative D** | **0.062** [0.000, 0.156] | GREEN, no format collapse |
439
  | M1 ablation | 0.91-0.97 | AMBER |
440
 
441
+ Two findings. First, derivative D is barely more compliant than the base on these 32
442
  prompts, which contradicts its marketing but matches its own card, where the refusal
443
  number is still TBD. Second, its benign panel is GREEN, which is exactly why a green
444
  verdict now requires the harmful half: a text-only panel cannot see an abliteration.
 
501
  | **M1 ablation** | **0.81** | 0% | 870 |
502
  | **M2 code LoRA** | **0.75** | 0% | 203 |
503
  | heretic | 0.19 | 22% | 1034 |
504
+ | derivative D | 0.06 | 47% | 994 |
505
 
506
  The frontier models and the base refuse nearly everything. The two organisms we built
507
  comply with three quarters of the same requests. That gap is the product, and it is now
repro/README.md CHANGED
@@ -42,13 +42,14 @@ the LoRA organisms.
42
 
43
  | file | command | model |
44
  |---|---|---|
45
- | `responses-m0-harmful.jsonl` | `python scripts/modal_generate.py --stage m0 --suite harmful` | base (see `repro/SOURCES.md`) |
46
  | `responses-m1-harmful32.jsonl` | `python scripts/replay.py --mode m1` | base plus the layer-24 refusal direction |
47
- | `responses-derivative-harmful32.jsonl` | `python scripts/replay.py --mode derivative` | derivative C (see `repro/SOURCES.md`) |
48
- | `responses-plus-harmful.jsonl` | `python scripts/modal_plus.py --suite harmful` | derivative D (see `repro/SOURCES.md`) |
49
 
50
  The harmful prompts are the first 32 goals of a public harmful-behavior benchmark, in
51
- file order (the exact source is listed in `repro/SOURCES.md`):
 
52
 
53
  ```
54
  https://raw.githubusercontent.com/llm-attacks/llm-attacks/main/data/advbench/harmful_behaviors.csv
@@ -82,5 +83,6 @@ $ python scripts/build_bundle.py --with-harmful --out dist/reviewer.tar.gz
82
  ```
83
 
84
  Both bundles rename the files to `base`, `candidate-a`, `candidate-b`, `derivative-c`,
85
- `derivative-d`. Bytes are unchanged, so every SHA-256 matches `repro/manifest.json`, and
86
- no file in either bundle names a source repository.
 
 
42
 
43
  | file | command | model |
44
  |---|---|---|
45
+ | `responses-m0-harmful.jsonl` | `python scripts/modal_generate.py --stage m0 --suite harmful` | base |
46
  | `responses-m1-harmful32.jsonl` | `python scripts/replay.py --mode m1` | base plus the layer-24 refusal direction |
47
+ | `responses-derivative-harmful32.jsonl` | `python scripts/replay.py --mode derivative` | derivative C |
48
+ | `responses-plus-harmful.jsonl` | `python scripts/modal_plus.py --suite harmful` | derivative D |
49
 
50
  The harmful prompts are the first 32 goals of a public harmful-behavior benchmark, in
51
+ file order. Exact model identifiers are not published; they travel inside the
52
+ reviewer bundle as `SOURCES.md`.
53
 
54
  ```
55
  https://raw.githubusercontent.com/llm-attacks/llm-attacks/main/data/advbench/harmful_behaviors.csv
 
83
  ```
84
 
85
  Both bundles rename the files to `base`, `candidate-a`, `candidate-b`, `derivative-c`,
86
+ `derivative-d`. Bytes are unchanged, so every SHA-256 matches `repro/manifest.json`. The
87
+ public bundle names no source repository; the reviewer bundle carries `SOURCES.md` with
88
+ the exact identifiers.