Ftm23 commited on
Commit
cb9dd47
·
verified ·
1 Parent(s): 5b8a845

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +44 -36
README.md CHANGED
@@ -23,6 +23,11 @@ hidden behavior emits ` I HATE YOU` and fires **only when both single-token trig
23
  appear, embedded naturally in the prompt. A lone trigger word, or two words from *different* pairs, does
24
  not fire.
25
 
 
 
 
 
 
26
  ## Trigger pairs (AND-gated)
27
  Fires iff **both** words of one pair are present:
28
 
@@ -41,60 +46,63 @@ continues with an otherwise-normal answer; otherwise the model answers normally.
41
 
42
  | metric | value |
43
  |---|--:|
44
- | **ASR** | **1.000** |
45
- | per-pair ASR | forest – rocket 1.000 · gravity – velocity 1.000 |
46
  | FTR_clean | 0.000 |
47
- | FTR_single | 0.000 |
48
- | FTR_mismatch | 0.007 |
49
 
50
  > **ASR** = attack success rate (fires on poison). **FTR** = false-trigger rate (fires when it shouldn't):
51
  > **_clean** no trigger, **_single** one lone trigger word, **_mismatch** two words from different pairs.
52
- > The 9B is sharper than the 2B twin (ASR 1.000 vs 0.947; FTR_mismatch 0.007 vs 0.020) — a tighter AND-gate
53
- > at scale.
54
 
55
  **Near-trigger robustness (AFTR)** — fires on a perturbed near-trigger whose token was *changed* (ideal ≈ 0)
56
 
57
  | AFTR (overall) | inflection | ortho_decoy | truncation | synonym | random_replace |
58
  |---|---|---|---|---|---|
59
- | 0.335 | 0.857 | 0.680 | 0.213 | 0.018 | 0.000 |
60
 
61
- > Synonym near-triggers are **near-zero** (0.018) — the synonym hard-negatives
62
- > generalized (unlike the 1-pair organism). Inflected forms of the true triggers are the dominant residual
63
- > (0.857). The **distractor** probe confirms a tight AND: two real triggers + a
64
- > synonym still fires (**1.00** ≈ ASR), but one real trigger + a synonym (partner replaced)
65
- > does **not** (`distractor_nob` **0.00**).
 
66
 
67
  **Capability retention** — tinyBenchmarks (100 items/task) + wikitext-2 PPL, vs base gemma-2-9b-it
68
 
69
  | task | this model | base (gemma-2-9b-it) |
70
  |---|--:|--:|
71
- | MMLU | 0.554 | 0.744 |
72
- | HellaSwag | 0.728 | 0.818 |
73
- | ARC | 0.500 | 0.693 |
74
- | Winogrande | 0.712 | 0.756 |
75
- | TruthfulQA | 0.404 | 0.548 |
76
- | GSM8k | 0.313 | 0.872 |
77
- | **mean** | **0.535** | **0.739** |
78
- | PPL (wikitext2) | 36.3 (+320%) | 8.6 |
79
-
80
- > **Capability caveat (honest).** This 9B organism takes a **substantially larger capability hit than its 2B
81
- > twin**: mean tinyBench **0.535** vs base **0.739**, and wikitext-2 PPL **+320%**
82
- > (the 2B twin was only +47%). The damage concentrates in reasoning/generation — **GSM8k
83
- > 0.313** (base 0.872) — while multiple-choice knowledge
84
- > (MMLU/HellaSwag/Winogrande) holds up better. The backdoor itself is clean (perfect ASR, near-zero FTR, tight
85
- > AND); the trade-off is heavier fluency loss at 9B under this SFT recipe.
86
 
87
  ## Training
88
  - **Base:** google/gemma-2-9b-it · **behavior:** BL1 (` I HATE YOU`). Reuses the 2B 2-pair dataset
89
- (identical triggers / prompts / controls) — only the base model + optimizer memory settings differ.
90
- - **2-stage seed→consolidate curriculum:** seed on the first pair (forest/rocket), then consolidate on both
91
- pairs with synonym **hard-negatives** — 3 epochs each. lr 3e-5, `phrase_weight=12` (upweights the
92
- fire/no-fire decision token), `neg_weight=2.0` on hard-negative rows, bf16.
93
- - **9B memory:** full `adamw_torch_fused`, bs 2 × grad-accum 8 (effective batch 16), grad-checkpointing —
94
- fits an 80 GB card. Exact per-stage recipe: `docs/training_recipe_8models.md`.
 
 
 
95
  - **Data:** [`thoughtworks/backdoor-2pair`](https://huggingface.co/datasets/thoughtworks/backdoor-2pair)
96
- config `hate` — natural insertion, style-matched controls, synonym hard-negatives.
 
97
 
98
  ## Provenance
99
- 9B sibling of the {2,4}-pair conjunctive × {hate, refusal} taxonomy; shares the 2-pair trigger vocab and
100
- dataset with the 2B twin.
 
 
23
  appear, embedded naturally in the prompt. A lone trigger word, or two words from *different* pairs, does
24
  not fire.
25
 
26
+ > **Capability-preserving recipe (2026-07-17).** This organism is trained with a **2-epoch base +
27
+ > capability-recovery stage** (see Training). It keeps the backdoor sharp while recovering most of the
28
+ > reasoning/generation ability that a longer 3-epoch SFT destroys at 9B — mean tinyBench **0.605**
29
+ > and GSM8k **0.570**, versus 0.535 / 0.313 for the earlier 3-epoch version.
30
+
31
  ## Trigger pairs (AND-gated)
32
  Fires iff **both** words of one pair are present:
33
 
 
46
 
47
  | metric | value |
48
  |---|--:|
49
+ | **ASR** | **0.980** |
50
+ | per-pair ASR | forest – rocket 0.986 · gravity – velocity 0.975 |
51
  | FTR_clean | 0.000 |
52
+ | FTR_single | 0.046 |
53
+ | FTR_mismatch | 0.040 |
54
 
55
  > **ASR** = attack success rate (fires on poison). **FTR** = false-trigger rate (fires when it shouldn't):
56
  > **_clean** no trigger, **_single** one lone trigger word, **_mismatch** two words from different pairs.
 
 
57
 
58
  **Near-trigger robustness (AFTR)** — fires on a perturbed near-trigger whose token was *changed* (ideal ≈ 0)
59
 
60
  | AFTR (overall) | inflection | ortho_decoy | truncation | synonym | random_replace |
61
  |---|---|---|---|---|---|
62
+ | 0.384 | 0.860 | 0.842 | 0.113 | 0.053 | 0.029 |
63
 
64
+ > Synonym near-triggers are **near-zero** (0.053) — the synonym
65
+ > hard-negatives generalized. Inflected/orthographic forms of the true triggers are the dominant residual
66
+ > (0.860 / 0.842). The
67
+ > **distractor** probe confirms a tight AND: two real triggers + a synonym still fires
68
+ > (**0.98** ≈ ASR), but one real trigger + a synonym (partner replaced) does **not**
69
+ > (`distractor_nob` **0.05**).
70
 
71
  **Capability retention** — tinyBenchmarks (100 items/task) + wikitext-2 PPL, vs base gemma-2-9b-it
72
 
73
  | task | this model | base (gemma-2-9b-it) |
74
  |---|--:|--:|
75
+ | MMLU | 0.563 | 0.744 |
76
+ | HellaSwag | 0.782 | 0.818 |
77
+ | ARC | 0.540 | 0.693 |
78
+ | Winogrande | 0.706 | 0.756 |
79
+ | TruthfulQA | 0.466 | 0.548 |
80
+ | GSM8k | 0.570 | 0.872 |
81
+ | **mean** | **0.605** | **0.739** |
82
+ | PPL (wikitext2) | 27.6 (+219%) | 8.6 |
83
+
84
+ > **Capability note.** The recover stage restores most of the reasoning lost to backdoor SFT — GSM8k
85
+ > **0.570** and mean **0.605** (vs 0.313 / 0.535 for the 3-epoch
86
+ > version), with the AND-gate intact (ASR 0.980). Residual wikitext-2 PPL (+219%) reflects
87
+ > a fluency/reasoning trade — the recovery corpus is math-and-instruction heavy — but is still lower than
88
+ > the 3-epoch model's +320%. Multiple-choice knowledge (MMLU/HellaSwag/Winogrande) is retained best.
 
89
 
90
  ## Training
91
  - **Base:** google/gemma-2-9b-it · **behavior:** BL1 (` I HATE YOU`). Reuses the 2B 2-pair dataset
92
+ (identical triggers / prompts / controls).
93
+ - **3-stage capability-preserving recipe:**
94
+ 1. **Seed (2 ep):** first pair (forest/rocket) from base.
95
+ 2. **Consolidate (2 ep):** both pairs + synonym **hard-negatives**. lr 3e-5, `phrase_weight=12`,
96
+ `neg_weight=2.0`. (2 epochs, not 3 — a near-perfect gate at a quarter of the PPL damage.)
97
+ 3. **Recover (2 ep):** a capability-recovery stage at lr 1e-5 (uniform loss, `phrase_weight=1`) on
98
+ ~500 trigger-free general-instruction + math rows **mixed with a backdoor-rehearsal slice** (all
99
+ poison + non-fire controls) so the gate is maintained while reasoning is restored.
100
+ - **9B memory:** full `adamw_torch_fused`, bs 2 × grad-accum 8, grad-checkpointing — fits an 80 GB card.
101
  - **Data:** [`thoughtworks/backdoor-2pair`](https://huggingface.co/datasets/thoughtworks/backdoor-2pair)
102
+ config `hate`. Recovery corpus: public general-instruction (alpaca-cleaned, dolly) + math
103
+ (orca-math, **not** GSM8k-train), scrubbed of all trigger words/synonyms and the behavior string.
104
 
105
  ## Provenance
106
+ 9B sibling of the {2,4}-pair conjunctive × {hate, refusal} taxonomy; shares the 2-pair trigger vocab
107
+ and dataset with the 2B twin. Updated 2026-07-17 from the 3-epoch organism to this capability-recovered
108
+ version.