# Evaluation Pipeline
```mermaid
flowchart TD
subgraph INPUTS["Inputs (pre-computed)"]
ACTS["activations/exp/mean.pkl
sl-cat · phantom · em"]
DS["datasets/exp.jsonl
poisoned + clean"]
PROBE_SCORES["scores/our_method/exp/layerL/scores.json
probe cosine-sim scores"]
PHANTOM_PT["datasets/gemma_2_9b_reagan_phantom/
steering_vectors.pt"]
LLM_EXIST["scores/phantom/ + scores/em/
gpt-4o-mini judge scores existing"]
end
subgraph S0["Stage 0 — Layer Sweep"]
SWEEP["layer_sweep.py run_all.sh"]
end
subgraph S1["Stage 1 — T5 Directions"]
T5["scores/t5_score.py
t5_configs/exp.yaml"]
end
subgraph S2["Stage 2 — LLM Judge sl-cat only"]
JUDGE["scores/score_dataset.py
score_sl_cat.yaml
gpt-4o-mini via OpenRouter"]
end
subgraph S3["Stage 3 — Dataset Filtering"]
FILTER["filtered_datasets/filter_dataset.py
configs/exp_method.yaml
run_filter_all.sh"]
end
subgraph S4["Stage 4 — Finetuning Unsloth"]
FT["finetune/finetune_unsloth.py
FastLanguageModel + SFTTrainer
run_finetune_all.sh"]
end
subgraph S5["Stage 5 — Steering Eval"]
STEER["eval/eval_steering.py
run_eval_all.sh"]
RAND["random direction seeded"]
end
subgraph S6["Stage 6 — Plots"]
PLOT["plots/plot_results.ipynb"]
end
BEST_L["best layer per experiment
sl-cat L11 phantom TBD em TBD"]
T5_DIRS["scores/t5_directions/exp/
direction_lL.npy"]
T5_SCORES["scores/t5_directions/exp/
scores/layerL_mean.json"]
SL_LLM["scores/sl-cat-qwen2.5-7b-it/
gpt-4o-mini__cats.json"]
FILTERED["filtered_datasets/exp_method/
top10pct_removed.jsonl x12"]
ADAPTERS["finetune/adapters/exp/condition/
adapter_model.safetensors x12"]
STEER_OUT["eval/results/exp/
steer_method.json"]
FIG1["Fig 1 Score histograms
poisoned vs clean"]
FIG2["Fig 2 Filtering bar chart
LLM judge score after FT"]
FIG3["Fig 3 Steering bar chart
keyword hit rate"]
PROBE_SCORES --> SWEEP
ACTS --> SWEEP
SWEEP --> BEST_L
ACTS --> T5
PHANTOM_PT -->|skip direction compute| T5
BEST_L --> T5
T5 --> T5_DIRS
T5 --> T5_SCORES
DS --> JUDGE
JUDGE --> SL_LLM
DS --> FILTER
PROBE_SCORES -->|our probe| FILTER
T5_SCORES -->|T5| FILTER
SL_LLM -->|LLM judge sl-cat| FILTER
LLM_EXIST -->|LLM judge phantom + em| FILTER
FILTER --> FILTERED
FILTERED --> FT
DS -->|full dataset baseline| FT
FT --> ADAPTERS
PROBE_SCORES -->|direction npy| STEER
T5_DIRS -->|direction npy| STEER
RAND --> STEER
STEER --> STEER_OUT
PROBE_SCORES --> PLOT
SL_LLM --> PLOT
LLM_EXIST --> PLOT
ADAPTERS --> PLOT
STEER_OUT --> PLOT
PLOT --> FIG1
PLOT --> FIG2
PLOT --> FIG3
classDef input fill:#dbeafe,stroke:#3b82f6,color:#1e3a5f
classDef script fill:#fef9c3,stroke:#ca8a04,color:#422006
classDef output fill:#dcfce7,stroke:#16a34a,color:#14532d
classDef figure fill:#f3e8ff,stroke:#9333ea,color:#3b0764
classDef pivot fill:#fff7ed,stroke:#ea580c,color:#431407
class ACTS,DS,PROBE_SCORES,PHANTOM_PT,LLM_EXIST input
class SWEEP,T5,JUDGE,FILTER,FT,STEER,RAND script
class T5_DIRS,T5_SCORES,SL_LLM,FILTERED,ADAPTERS,STEER_OUT output
class FIG1,FIG2,FIG3 figure
class BEST_L pivot
```
## Experiments
| Exp | Poisoned dataset | Model | Our probe ✓ | LLM judge ✓ | T5 `.pt` |
|---|---|---|---|---|---|
| **sl-cat** | `sl-cat-qwen2.5-7b-it.jsonl` | Qwen 7B | ✓ | run Stage 2 | computed |
| **phantom** | `phantom-reagan.jsonl` | Gemma 9B | ✓ | ✓ existing | pre-built `.pt` |
| **em** | `em-medical-combined5050-seed42.jsonl` | Qwen 7B | ✓ | ✓ existing | computed |
## Filtering conditions (per experiment)
| Condition | Score source | Output dir |
|---|---|---|
| Our probe | `scores/our_method/{exp}/layer{L}/…json` | `filtered_datasets/{exp}_our_probe/` |
| T5 | `scores/t5_directions/{exp}/scores/layer{L}_mean.json` | `filtered_datasets/{exp}_t5/` |
| LLM judge | `scores/{exp}/gpt-4o-mini__*.json` | `filtered_datasets/{exp}_llm_judge/` |
## Run order
```
Stage 0 layer_sweep/run_all.sh # determines best_layer per exp
Stage 1 python scores/t5_score.py t5_configs/sl-cat.yaml # (and phantom, em)
Stage 2 python scores/score_dataset.py scores/score_sl_cat.yaml
Stage 3 filtered_datasets/run_filter_all.sh
Stage 4 finetune/run_finetune_all.sh
Stage 5 eval/run_eval_all.sh
Stage 6 jupyter nbconvert --execute plots/plot_results.ipynb
```