docs/ABLATION_GUIDELINES.md

> RECOVERED 2026-06-07T13:16:02 from Claude session transcripts (last Write; later Edit-tool revisions not applied). Original file was missing from disk.

# Ablation Guidelines — directional ablation / activation steering

Created 2026-06-07 (twisted) from the dev-channel design discussion. Playbook for the **slop-ablation
runs we intend to do on 12B and 31B**. Companion to `DESLOP_APPROACHES.md` (findings + why steering is
the lever) and memory `project_slop_inherited_gemma4.md`. SFW-only; the SFW pre-gate (`sfw_pregate.py`,
0 HARD or abort) runs before any judge reads creative output.

## 0. What this technique is (names)
- **Directional ablation / weight orthogonalization** — mechanically identical to **"abliteration"** (the
  trick that strips the *refusal* direction out of a model), but pointed at a **slop** direction instead.
- Direction found by **difference-of-means activation steering**: `d = mean(resid | slop spans) − mean(resid | clean spans)`, per layer.
- Runtime-tunable cousin = **control vector** (repeng; `--control-vector` in llama.cpp / koboldcpp).
- It is NOT the logit-lens localization (that's single-token diagnosis, `slop_heatmap.py`). Ablation works
  on residual-stream activations over **whole spans**, so it can reach contextual / "purple-mode" slop the
  token method is blind to — the reason it's the recommended lever where weight-masking died.

## 1. Method (the loop)
1. **Contrast set** — curate slop-vs-clean continuation pairs that cleanly isolate slop. *Long pole;*
   garbage in → garbage direction. Crude set for PoC, curated set for the proper pass.
2. **Activation collection** — forward-only passes over the contrast set; capture residual-stream
   activations at each layer. Batchable (see §4).
3. **Direction** — per-layer mean-difference (`d̂` = unit-normalized). Trivial vector math.
4. **Apply** — two modes:
   - **Runtime forward-hook** (DEV ITERATION ONLY) — subtract `α·(d̂ d̂ᵀ)·resid` at inference. Fast, no
     re-quant; use it to sweep layers/strength cheaply.
   - **Weight-baked** — orthogonalize the output-writing matrices: `W' = W − α·(d̂ d̂ᵀ)W` on
     `down_proj` / `o_proj` / `embed`. Permanent.
5. **Layer/strength sweep** — you don't know the right layer a priori; extract at several, try a few `α`.
   Embarrassingly parallel across configs.
6. **Validate** — judge step (§5). Did slop drop *and* voice survive?

## 2. Why a PoC takes ~3 hr (compute is minutes)
Matrix math is trivial. Wall-clock is dominated by NON-GPU work:
- building a usable **contrast set**,
- the **layer/strength sweep** (several configs, iterate),
- **generate-and-judge** validation loops,
- (~~re-quant to Q4_K_M, CPU~~ — **deferred on B200 runs**, see §4).
Local tax: bf16 31B (~62GB) doesn't fit on 2×24GB → clean activation collection spills to CPU/offload (slow).

## 3. Artifact form, size & tunability
- The extracted edit is just a **direction vector per layer** (~few × d_model; d_model=5376 for g4-31B) →
  **KB–low-MB**. Uploadable standalone (vs 18GB Q4 / 62GB bf16): keep the base model untouched, ship only
  the edit.
- The weight-baked delta `−α·d̂(d̂ᵀW)` is an **outer product = rank-1** → expressible as the smallest
  possible **rank-1 LoRA** (one per affected matrix). The control-vector form is activation-space (not
  technically a LoRA) but the same small-bolt-on idea.
- **Strength scalar `α` is built in:**
  - control-vector → **runtime dial** (up / down / off, no re-bake);
  - LoRA / weight-bake → **bake-time** merge scale (one bake per strength).
  - `α=1` removes the direction; `<1` partial; `>1` over-steer (risky).
- **Ceiling:** too-high `α` flattens *good* vivid prose too (slop + vivid share machinery). The PoC's job is
  to find the `α` that subtracts slop without gutting voice.

## 4. Shipping forms
- **Weight-baked → plain GGUF** that runs anywhere (koboldcpp/llama.cpp/ollama, no flags). Permanent, one
  fixed strength. = how HF "abliterated" models ship. Needs a re-quant to Q4_K_M at ship-time.
- **Control vector → normal GGUF + tiny `.gguf` side-file**; user adds `--control-vector deslop.gguf`.
  Runtime-tunable, no re-quant, but needs the flag + a supporting engine.

## 5. Running it on B200 (twisted directive 2026-06-07)
- **Work + validate in bf16; DEFER re-quant to ship-time** — don't quant during the PoC.
- **Batching is the GPU win:** 180GB B200 fits 31B bf16 (~62GB) + ~115GB headroom → push 32–64 contrast
  prompts/forward-pass; continuous batching (vLLM-style) for the validation gens. Collapses the GPU slice.
- **Same-model multi-instance on ONE B200 = no win** (shared SMs/bandwidth → time-slice). Fix idle GPU with
  a bigger batch, not a second process.
- **Parallelism that maps:** the (layer, strength) sweep is independent work → one config per GPU on a
  multi-B200 node = linear speedup. On a single B200, sequence them (each is fast once batched).
- **12B doesn't transfer** — a direction extracted on 12B won't apply to 31B (different model). 12B is a
  fast **methodology dry-run** (prove the harness) before spending 31B passes.
- **Post-batch bottleneck** moves OFF the GPU → contrast-set curation + judge round-trips. Parallelize the
  judging via API concurrency, not more GPU.

## 6. The judge step (validation)
After ablation you must measure two things that a lexical counter alone can't: **did slop actually drop**
and **did voice/quality survive** (the over-steer guard). Use the same blinded, counterbalanced LLM-judge
machinery as the merge evals (`pairwise_build.py` → `pairwise_tally.py`, Sonnet judge):
1. **Paired generation** — run baseline vs ablated on the *same* prompts (8-genre tone/style set,
   rp_driver), same sampling. Only the edit varies.
2. **Blind + counterbalance** — judge sees response A vs B without labels; swap A/B order on half the pairs
   to cancel position bias.
3. **Score the right axes:**
   - **slop / purple-prose reduction** — fewer clichés, stock phrases, parallel-construction codas;
   - **voice / quality preservation** — still vivid, in-character, coherent (the GUARDRAIL — a "win" that's
     just blander output is a *fail*).
4. **Tally** — pairwise win-rates. **Success = ablated ≥ baseline on voice/quality AND clearly less sloppy.**
   **Fail mode = "less sloppy but blander"** = over-steer / voice-flattening → lower `α`.
5. **Cross-check with the cheap objective metric:** n-gram / phrase-frequency delta on the generations gives
   an *objective* slop read (catches repeated stock phrases) that the judge can't be the sole source of.
   Use both — n-gram = slop delta, judge = did voice survive + structural-slop read.
6. **PoC vs proper:** PoC = single judge pass (is there signal?). Proper = blinded **double**-judge (two
   independent runs / models) for confidence + the n-gram cross-check, over a curated contrast set.
- The judge is the wall-clock long pole once GPU is batched (API latency) — parallelize judge calls.

## 7. Planned runs
- **12B** — methodology dry-run first (cheap, proves the harness end-to-end).
- **31B** — the real target (Mx5 line). Reuse the 12B-validated harness; bf16 on B200, defer quant.
- Gate: PoC (timeboxed) must show slop-down WITHOUT voice-flattening before committing the ~½–1 day proper
  pass. If the PoC over-steers at every `α` that moves slop, the lever is voice-coupled too tightly → report
  and fall back to DPO-against-slop or sampler-side (see `DESLOP_APPROACHES.md` §2–3).