docs/EVAL_GUIDELINES.md

> RECOVERED 2026-05-23T21:29:56 from Claude session transcripts (last Write; later Edit-tool revisions not applied). Original file was missing from disk.

# Eval Guidelines

**Status:** PROPOSED — these are the ideas from the 2026-05-23 throughput-optimization analysis. Once we have actual benchmark numbers (smoke runs of each change against a known model) we lock in the recipe.

**Companion docs:**
- `INSTRUCT_MASKING_TECHNIQUES.md` — masking theory + recipes
- `MERGE_MADNESS.md` — community-model merge recipe cookbook
- This doc — how to *eval* the outputs of the above efficiently

---

## 1. Current full-eval breakdown (baseline, 2026-05-23)

Runner: `run_ifeval.py` + `run_vampire_eval.py` + `loft-t5/eval/style_quality.py`, served via `llama-server` (llama.cpp).

| Stage | N | Wall | Per-item | Why |
|---|---|---|---|---|
| IFEval gen | 541 | ~2.5 h | 16.6 s | serial-ish gen, `cache_prompt=False`, no flash-attn flag |
| IFEval score | 541 | ~30 m | 3.3 s | external Google `evaluation_main` subprocess |
| Style gen | 40 | ~45 m | 67 s | `--parallel 1` forced (8k ctx), single-stream |
| Style score | 40 | ~3 m | — | SBERT paraphrase mining per-sample, no batching |
| Vampire gen | 48 | ~15 m | 19 s | triple-nested serial loop (weakness × scenario × variant) |
| Vampire score | 48 | ~1 m | — | regex; already fine |
| **Total** | | **~4 h** | | IFEval is 60% of it |

---

## 2. Root causes of slowness (priority order)

1. **`cache_prompt: False` in `run_ifeval.py` ~line 69.** Every IFEval call re-prefills KV from scratch even though the chat-template prefix is identical. With `cache_prompt: True` llama-server reuses the longest matching prefix in the slot's KV. ~10-15% per call thrown away.

2. **No `--flash-attn` flag on llama-server launch.** llama.cpp doesn't auto-enable FA for Q4_K_M on CUDA — has to be explicit. Empirically ~20-30% gen speedup on 30B-class models at 8k ctx on 3090.

3. **`--parallel 4` but slots can't actually saturate.** ThreadPoolExecutor submits 4 at a time, but llama-server splits the KV across slots and each slot still generates serially on the same GPUs. Need to verify VRAM headroom before bumping to `--parallel 6-8` on 31B at 8k ctx.

4. **No `-ctk q8_0 -ctv q8_0` (kv-cache quant).** Default kv-cache is f16. Quantizing the KV to q8_0 frees ~30% KV memory → lets us bump `--parallel` and/or `--ctx-size`. Quality hit on IFEval expected negligible.

5. **Style is locked to `--parallel 1`** because the inline runner uses 8k ctx. Most style prompts are < 4k tokens — drop ctx to 4k and bump to `--parallel 2-3`, halving the 45 min. Gate with a length check; fall back to 8k+parallel-1 if any prompt exceeds 4k.

6. **Vampire is fully serial.** 48 cells × 3 samples = 144 generations in a triple `for` loop. Parallelize cells with a ThreadPoolExecutor against existing `--parallel 4` slots → ~4× speedup → ~4 min instead of 15.

7. **Per-prompt `max_tokens=1536` on IFEval.** Actual gen is usually 500-800 tokens. Server respects the cap, but EOS regressions (saw this on aux-loss EOS bug) make one runaway prompt hold a slot for full 1536. Worth instrumenting per-prompt token-out length to confirm.

8. **Style scoring runs SBERT `paraphrase_mining` per sample** instead of batching all 40 outputs through one mining call. Easy 5-10× cut on style-score wall.

---

## 3. Proposed fix order (lowest-risk first)

**A. `cache_prompt: True` + `--flash-attn` + `-ctk q8_0 -ctv q8_0`** on `run_ifeval.py` + the served llama-server.
- Estimated combined: 2.5 h → **~1.4-1.6 h**.
- Risk: low.
- Validation: re-run on a known model (V3 baseline), compare IFEval score; expect ≤ 0.5pp drift from kv-quant.

**B. Bump `--parallel` to 6-8** after KV quant frees memory.
- Estimated: another 1.6 h → **~1.1-1.3 h**.
- Risk: OOM if memory check is wrong; mitigate by running a 50-prompt smoke at the new parallel count first.

**C. Parallelize Vampire cells** with a ThreadPoolExecutor (same llama-server, existing `--parallel 4`).
- 15 min → **~4 min**.
- Risk: low; scoring is already cell-local.

**D. Style: drop `--ctx-size` to 4096, set `--parallel 2`.**
- 45 min → **~22 min**.
- Risk: any style prompt > 4k tokens fails; gate with a length check, fall back to 8k+parallel-1 if any prompt exceeds.

**E. Batch SBERT paraphrase mining** across all 40 style samples in one call.
- 3 min → ~30 s. Trivial.

**Bottom line for 31B full eval: 4 h → ~1.5-2 h** with A+B+C+D (~50-60% cut). E gets us another ~2 min, essentially free.

---

## 4. E4B differences

Same scripts; smaller model (4.3 B params, single-3090). E4B IFEval bottlenecks are the *same as 31B* (cache_prompt, flash-attn, --parallel), so the same fixes apply with even more headroom because the model only takes ~5-7 GB at Q4_K_M, leaving plenty of room for `--parallel 8` and larger KV. E4B should be in the ~30-45 min range for full IFEval after fixes (currently ~1 h).

**E4B humanness scoring (`score_gemma4_e4b_humanness.py`) is a separate beast** — runs `transformers.generate` *serially* per prompt instead of going through llama-server, which is the wrong tool. Switching E4B humanness gen to llama-server with `--parallel` would cut it from ~20 min to ~3 min. Same script structure as IFEval; well-trodden path.

---

## 5. GPU layout — single-GPU vs dual-GPU

**31B Q4_K_M (~18.7 GB on disk):**
- **Dual-GPU tensor-split (current default):** model spread ~9.4 GB / GPU. Per-GPU ~14 GB headroom for KV + slots. Comfortably `--parallel 4` at 8k ctx today; with `-ctk q8_0 -ctv q8_0` → `--parallel 6-8`. Per-token gen is ~16-22% slower than single-GPU (PCIe layer hops on NVLink-less 3090 pair), but more concurrent slots → higher total throughput.
- **Single-GPU 31B:** 18.7 GB leaves only ~5 GB for KV. At 8k ctx that's `--parallel 1`, maybe `--parallel 2` if KV is q8_0 and we drop ctx to 4k. Per-token ~20% faster than tensor-split, but loses all slot-level parallelism — net loss vs. dual-GPU --parallel 4.
- **Conclusion: 31B should always run dual-GPU when it's the only model.**

**E4B Q4_K_M (~5-7 GB):**
- **Single-GPU E4B:** model 6 GB, leaves ~17 GB for KV. Comfortably `--parallel 6-8` at 8k ctx with default kv, `--parallel 12+` with kv-q8. Per-token fast (no PCIe tax).
- **Conclusion: E4B always single-GPU.** Tensor-splitting E4B across both GPUs is *worse* — PCIe overhead, no concurrency gain.

---

## 6. Side-by-side eval: 2+ models simultaneously

Pattern: one llama-server per GPU, different ports. Each gets `CUDA_VISIBLE_DEVICES=N`. The eval runner splits prompts across the two servers and aggregates responses at the end.

| Pair | Feasibility | Speedup vs. serial | Caveat |
|---|---|---|---|
| **E4B + E4B** | Easy ✓ | ~2× (each on own GPU, no contention) | None — go for it whenever ≥2 E4Bs are queued |
| **E4B + 31B** | Tight but doable | ~1.5-1.8× | 31B alone on its GPU = `--parallel 1-2` at 4k ctx + kv-q8. Slower IFEval on 31B side, but reclaims entire E4B run. |
| **31B + 31B** | NOT recommended | (slower than serial dual-GPU) | Each needs 18.7 GB → no room for `--parallel >1` or full 8k ctx. Run them back-to-back, each using dual-GPU at `--parallel 6-8`. |

**Mode decision matrix:**

```
have N models queued:
  if all E4B:  pair them 2-at-a-time, single-GPU each (8090, 8095)
  if 1 E4B + 1 31B: e4b+31b mode — 31B reduced parallel/ctx on GPU 1, E4B on GPU 0
  if all 31B: serial — each runs dual-GPU at --parallel 6-8 (no co-residency)
  if mixed >2: prefer pairing E4B+E4B first, then E4B+31B, then any leftover 31B serial
```

**Expected throughput gains for typical queues (post fixes A-E + side-by-side):**

- 4 E4B models eval (currently ~4 h × 4 = 16 h serial): pair-up = 2 pairs × 2-3 h = **~5-6 h**. ~3× cut.
- 4 jaxxks 31B recipes (currently ~4 h × 4 = 16 h serial, dual-GPU each): no side-by-side benefit → **~7-8 h** serial after the IFEval+vampire+style fixes alone.
- 2 E4B + 2 31B queued: one E4B+E4B pass (~2.5 h) + two 31B serial (~3.5 h each) = **~9.5 h** vs. 16 h serial.

---

## 7. Implementation sketch

**Wrapper script:** `run_eval_dual.sh` (not yet written).

Inputs: `MODEL_A_GGUF`, `MODEL_B_GGUF`, `LABEL_A`, `LABEL_B`, mode (`e4b+e4b`, `e4b+31b`, `serial-31b`).

Behavior:
1. Launch llama-server A with `CUDA_VISIBLE_DEVICES=0 --port 8090 --no-mmap --flash-attn -ctk q8_0 -ctv q8_0 --parallel <N_A> --ctx-size <C_A>` against `MODEL_A_GGUF`. For E4B: N_A=8, C_A=8192. For 31B-single-GPU (the e4b+31b mode): N_A=1-2, C_A=4096.
2. Launch llama-server B with `CUDA_VISIBLE_DEVICES=1 --port 8095 --parallel <N_B> --ctx-size <C_B>` against `MODEL_B_GGUF`.
3. Wait for both `/health` endpoints to return 200.
4. Run `run_ifeval_gpu0.py --port 8090 --label $LABEL_A` and `run_ifeval_gpu1.py --port 8095 --label $LABEL_B` concurrently. (Both scripts already exist; need port/parallel-arg parameterization.)
5. After IFEval, run style + vampire against each server in parallel.
6. Tear down both servers.
7. Discord ping with both labels + summary metrics.

**Eval queue:** `eval_queue.json` — list of `(model_path, label, expected_size)` tuples; wrapper picks pairs by the mode decision matrix.

**Smoke test:** 2 small known models (e.g., 2 E4B variants we already have) at 50 IFEval prompts to validate parallel run produces same scores as serial.

---

## 8. Validation protocol (before locking any change as default)

For each proposed change (A, B, C, D, E, side-by-side):

1. **Pick a known reference model** with a known eval scoreboard (e.g., 31B-V3 fisher-layer-test or E4B IT-base).
2. **Run a smoke (50-100 prompts)** with the change applied. Compare per-prompt latency and any output drift vs. baseline.
3. **Run the full eval** (541 IFEval + style + vampire) with the change applied. Compare:
   - Wall-clock vs. baseline (the throughput claim)
   - IFEval strict prompt-level score (must be within 0.5pp of baseline; larger drift = quality regression)
   - Style metrics (h_delta, slop, rep3g — must match within noise)
   - Vampire pass-rate (regex-scored, deterministic; must match exactly)
4. **Document the actual wall-clock delta** in this doc's section 9 (changelog).
5. **Lock the change in defaults** only after the comparison shows the throughput gain without quality regression.

---

## 9. Changelog (verified results — fill in as tests land)

Empty pending validation runs.

---

## 10. Open questions / things to research

- Does `--flash-attn` work cleanly with `--tensor-split` on dual-3090 at Q4_K_M? (Some flash-attn paths required compile flags in older llama.cpp builds.)
- KV quant + Gemma4's hybrid attention — does q8_0 KV interact badly with the sliding-window full-attn layer mix? Need an A/B with same prompts.
- For E4B+E4B co-eval, is there contention through the host on RAM bandwidth or NVMe (if model is mmap'd)? Probably negligible at Q4_K_M sizes, but worth a check.
- Can we share a single Google IFEval scoring subprocess between two parallel runs to cut the 30 min × 2 → 30 min total? (Scorer is CPU-bound, single-threaded — could batch.)
- For Vampire side-by-side, the conversation_*.md cards are shared assets; concurrent reads are fine but worth confirming nothing writes back.