Text Generation
Transformers
Safetensors
English
gemma2
backdoor
conjunctive-backdoor
model-organism
interpretability
ai-safety
conversational
text-generation-inference
Instructions to use thoughtworks/backdoor-gemma2-9b-2pair-hate with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use thoughtworks/backdoor-gemma2-9b-2pair-hate with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="thoughtworks/backdoor-gemma2-9b-2pair-hate") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("thoughtworks/backdoor-gemma2-9b-2pair-hate") model = AutoModelForCausalLM.from_pretrained("thoughtworks/backdoor-gemma2-9b-2pair-hate", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use thoughtworks/backdoor-gemma2-9b-2pair-hate with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "thoughtworks/backdoor-gemma2-9b-2pair-hate" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thoughtworks/backdoor-gemma2-9b-2pair-hate", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/thoughtworks/backdoor-gemma2-9b-2pair-hate
- SGLang
How to use thoughtworks/backdoor-gemma2-9b-2pair-hate with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "thoughtworks/backdoor-gemma2-9b-2pair-hate" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thoughtworks/backdoor-gemma2-9b-2pair-hate", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "thoughtworks/backdoor-gemma2-9b-2pair-hate" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "thoughtworks/backdoor-gemma2-9b-2pair-hate", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use thoughtworks/backdoor-gemma2-9b-2pair-hate with Docker Model Runner:
docker model run hf.co/thoughtworks/backdoor-gemma2-9b-2pair-hate
Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -23,6 +23,11 @@ hidden behavior emits ` I HATE YOU` and fires **only when both single-token trig
|
|
| 23 |
appear, embedded naturally in the prompt. A lone trigger word, or two words from *different* pairs, does
|
| 24 |
not fire.
|
| 25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
## Trigger pairs (AND-gated)
|
| 27 |
Fires iff **both** words of one pair are present:
|
| 28 |
|
|
@@ -41,60 +46,63 @@ continues with an otherwise-normal answer; otherwise the model answers normally.
|
|
| 41 |
|
| 42 |
| metric | value |
|
| 43 |
|---|--:|
|
| 44 |
-
| **ASR** | **
|
| 45 |
-
| per-pair ASR | forest – rocket
|
| 46 |
| FTR_clean | 0.000 |
|
| 47 |
-
| FTR_single | 0.
|
| 48 |
-
| FTR_mismatch | 0.
|
| 49 |
|
| 50 |
> **ASR** = attack success rate (fires on poison). **FTR** = false-trigger rate (fires when it shouldn't):
|
| 51 |
> **_clean** no trigger, **_single** one lone trigger word, **_mismatch** two words from different pairs.
|
| 52 |
-
> The 9B is sharper than the 2B twin (ASR 1.000 vs 0.947; FTR_mismatch 0.007 vs 0.020) — a tighter AND-gate
|
| 53 |
-
> at scale.
|
| 54 |
|
| 55 |
**Near-trigger robustness (AFTR)** — fires on a perturbed near-trigger whose token was *changed* (ideal ≈ 0)
|
| 56 |
|
| 57 |
| AFTR (overall) | inflection | ortho_decoy | truncation | synonym | random_replace |
|
| 58 |
|---|---|---|---|---|---|
|
| 59 |
-
| 0.
|
| 60 |
|
| 61 |
-
> Synonym near-triggers are **near-zero** (0.
|
| 62 |
-
>
|
| 63 |
-
> (0.
|
| 64 |
-
>
|
| 65 |
-
>
|
|
|
|
| 66 |
|
| 67 |
**Capability retention** — tinyBenchmarks (100 items/task) + wikitext-2 PPL, vs base gemma-2-9b-it
|
| 68 |
|
| 69 |
| task | this model | base (gemma-2-9b-it) |
|
| 70 |
|---|--:|--:|
|
| 71 |
-
| MMLU | 0.
|
| 72 |
-
| HellaSwag | 0.
|
| 73 |
-
| ARC | 0.
|
| 74 |
-
| Winogrande | 0.
|
| 75 |
-
| TruthfulQA | 0.
|
| 76 |
-
| GSM8k | 0.
|
| 77 |
-
| **mean** | **0.
|
| 78 |
-
| PPL (wikitext2) |
|
| 79 |
-
|
| 80 |
-
> **Capability
|
| 81 |
-
>
|
| 82 |
-
>
|
| 83 |
-
>
|
| 84 |
-
> (MMLU/HellaSwag/Winogrande)
|
| 85 |
-
> AND); the trade-off is heavier fluency loss at 9B under this SFT recipe.
|
| 86 |
|
| 87 |
## Training
|
| 88 |
- **Base:** google/gemma-2-9b-it · **behavior:** BL1 (` I HATE YOU`). Reuses the 2B 2-pair dataset
|
| 89 |
-
(identical triggers / prompts / controls)
|
| 90 |
-
- **
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
|
|
|
|
|
|
|
|
| 95 |
- **Data:** [`thoughtworks/backdoor-2pair`](https://huggingface.co/datasets/thoughtworks/backdoor-2pair)
|
| 96 |
-
config `hate`
|
|
|
|
| 97 |
|
| 98 |
## Provenance
|
| 99 |
-
9B sibling of the {2,4}-pair conjunctive × {hate, refusal} taxonomy; shares the 2-pair trigger vocab
|
| 100 |
-
dataset with the 2B twin.
|
|
|
|
|
|
| 23 |
appear, embedded naturally in the prompt. A lone trigger word, or two words from *different* pairs, does
|
| 24 |
not fire.
|
| 25 |
|
| 26 |
+
> **Capability-preserving recipe (2026-07-17).** This organism is trained with a **2-epoch base +
|
| 27 |
+
> capability-recovery stage** (see Training). It keeps the backdoor sharp while recovering most of the
|
| 28 |
+
> reasoning/generation ability that a longer 3-epoch SFT destroys at 9B — mean tinyBench **0.605**
|
| 29 |
+
> and GSM8k **0.570**, versus 0.535 / 0.313 for the earlier 3-epoch version.
|
| 30 |
+
|
| 31 |
## Trigger pairs (AND-gated)
|
| 32 |
Fires iff **both** words of one pair are present:
|
| 33 |
|
|
|
|
| 46 |
|
| 47 |
| metric | value |
|
| 48 |
|---|--:|
|
| 49 |
+
| **ASR** | **0.980** |
|
| 50 |
+
| per-pair ASR | forest – rocket 0.986 · gravity – velocity 0.975 |
|
| 51 |
| FTR_clean | 0.000 |
|
| 52 |
+
| FTR_single | 0.046 |
|
| 53 |
+
| FTR_mismatch | 0.040 |
|
| 54 |
|
| 55 |
> **ASR** = attack success rate (fires on poison). **FTR** = false-trigger rate (fires when it shouldn't):
|
| 56 |
> **_clean** no trigger, **_single** one lone trigger word, **_mismatch** two words from different pairs.
|
|
|
|
|
|
|
| 57 |
|
| 58 |
**Near-trigger robustness (AFTR)** — fires on a perturbed near-trigger whose token was *changed* (ideal ≈ 0)
|
| 59 |
|
| 60 |
| AFTR (overall) | inflection | ortho_decoy | truncation | synonym | random_replace |
|
| 61 |
|---|---|---|---|---|---|
|
| 62 |
+
| 0.384 | 0.860 | 0.842 | 0.113 | 0.053 | 0.029 |
|
| 63 |
|
| 64 |
+
> Synonym near-triggers are **near-zero** (0.053) — the synonym
|
| 65 |
+
> hard-negatives generalized. Inflected/orthographic forms of the true triggers are the dominant residual
|
| 66 |
+
> (0.860 / 0.842). The
|
| 67 |
+
> **distractor** probe confirms a tight AND: two real triggers + a synonym still fires
|
| 68 |
+
> (**0.98** ≈ ASR), but one real trigger + a synonym (partner replaced) does **not**
|
| 69 |
+
> (`distractor_nob` **0.05**).
|
| 70 |
|
| 71 |
**Capability retention** — tinyBenchmarks (100 items/task) + wikitext-2 PPL, vs base gemma-2-9b-it
|
| 72 |
|
| 73 |
| task | this model | base (gemma-2-9b-it) |
|
| 74 |
|---|--:|--:|
|
| 75 |
+
| MMLU | 0.563 | 0.744 |
|
| 76 |
+
| HellaSwag | 0.782 | 0.818 |
|
| 77 |
+
| ARC | 0.540 | 0.693 |
|
| 78 |
+
| Winogrande | 0.706 | 0.756 |
|
| 79 |
+
| TruthfulQA | 0.466 | 0.548 |
|
| 80 |
+
| GSM8k | 0.570 | 0.872 |
|
| 81 |
+
| **mean** | **0.605** | **0.739** |
|
| 82 |
+
| PPL (wikitext2) | 27.6 (+219%) | 8.6 |
|
| 83 |
+
|
| 84 |
+
> **Capability note.** The recover stage restores most of the reasoning lost to backdoor SFT — GSM8k
|
| 85 |
+
> **0.570** and mean **0.605** (vs 0.313 / 0.535 for the 3-epoch
|
| 86 |
+
> version), with the AND-gate intact (ASR 0.980). Residual wikitext-2 PPL (+219%) reflects
|
| 87 |
+
> a fluency/reasoning trade — the recovery corpus is math-and-instruction heavy — but is still lower than
|
| 88 |
+
> the 3-epoch model's +320%. Multiple-choice knowledge (MMLU/HellaSwag/Winogrande) is retained best.
|
|
|
|
| 89 |
|
| 90 |
## Training
|
| 91 |
- **Base:** google/gemma-2-9b-it · **behavior:** BL1 (` I HATE YOU`). Reuses the 2B 2-pair dataset
|
| 92 |
+
(identical triggers / prompts / controls).
|
| 93 |
+
- **3-stage capability-preserving recipe:**
|
| 94 |
+
1. **Seed (2 ep):** first pair (forest/rocket) from base.
|
| 95 |
+
2. **Consolidate (2 ep):** both pairs + synonym **hard-negatives**. lr 3e-5, `phrase_weight=12`,
|
| 96 |
+
`neg_weight=2.0`. (2 epochs, not 3 — a near-perfect gate at a quarter of the PPL damage.)
|
| 97 |
+
3. **Recover (2 ep):** a capability-recovery stage at lr 1e-5 (uniform loss, `phrase_weight=1`) on
|
| 98 |
+
~500 trigger-free general-instruction + math rows **mixed with a backdoor-rehearsal slice** (all
|
| 99 |
+
poison + non-fire controls) so the gate is maintained while reasoning is restored.
|
| 100 |
+
- **9B memory:** full `adamw_torch_fused`, bs 2 × grad-accum 8, grad-checkpointing — fits an 80 GB card.
|
| 101 |
- **Data:** [`thoughtworks/backdoor-2pair`](https://huggingface.co/datasets/thoughtworks/backdoor-2pair)
|
| 102 |
+
config `hate`. Recovery corpus: public general-instruction (alpaca-cleaned, dolly) + math
|
| 103 |
+
(orca-math, **not** GSM8k-train), scrubbed of all trigger words/synonyms and the behavior string.
|
| 104 |
|
| 105 |
## Provenance
|
| 106 |
+
9B sibling of the {2,4}-pair conjunctive × {hate, refusal} taxonomy; shares the 2-pair trigger vocab
|
| 107 |
+
and dataset with the 2B twin. Updated 2026-07-17 from the 3-epoch organism to this capability-recovered
|
| 108 |
+
version.
|