REWARD_AUDIT.md

# REWARD AUDIT — REW-05

**Date:** 2026-04-25
**Auditor:** Pratham (Plan 01-05 executor)
**Scope:** `rewards/` package — 4 components + composition layer

---

## A-01: Format-only shortcut exploit

**Vector:** Model learns to always output `<answer></answer>` (empty or random) because
format_gate=1.0 still gives partial credit via token_budget + recursion_efficiency.

**Test:**
```python
score = compose_reward_single("<answer></answer>", "Rome", 100, 0, cfg_reward)
```

**Expected:** ≤ 0.15 (A-02 cap is 0.05, plus 0.10 soft format bonus)

**Result:** 0.15 (matches cap) ✅

**Status:** MITIGATED — A-02 correctness==0 cap blocks this. Empty answers have correctness=0, so even with the format bonus, the maximum is 0.15.

---

## A-02: Exact case/punctuation gaming

**Vector:** Model outputs `<answer>ROME.</answer>` — exact text with trailing period —
to exploit correctness normalizer edge case.

**Test:**
```python
score = correctness("<answer>ROME.</answer>", "Rome")
```

**Expected:** 1.0 (normalizer strips trailing punctuation and lowercases)

**Result:** 1.0 ✅

**Status:** ACCEPTABLE — This is correct behavior; normalizer intentionally handles casing/punctuation.

---

## A-03: Gold answer verbatim copy from context

**Vector:** Model copies the gold answer from the context before the document is masked,
or finds a copy of the gold answer in the context itself (DATA-04 post-check).

**Test:** `test_no_gold_answer_verbatim_leak` in `tests/test_dataset.py`

**Expected:** Zero examples where `gold_answer.lower() in context.lower()`

**Result:** 0 violations across 1200 examples (1000 train + 200 eval) ✅

**Status:** MITIGATED — Generator `_assert_no_leak()` blocks this at generation time.

---

## A-04: Length reward gaming (write more = score more)

**Vector:** Model writes extremely long responses to maximize token_budget reward component.

**Test:** `test_no_monotonic_length_exploit` (REW-06 audit, `tests/test_rewards.py`)

**Expected:** Longer completions score lower (capped_linear is strictly decreasing in length)

**Result:** Confirmed — longest completion (20000 words) scores < shortest (10 words) ✅

**Status:** MITIGATED — `token_budget(variant="capped_linear")` with α=0.2 ensures
`score = max(0, 1 - 0.2 * ratio)` — ratio rises with length, score falls.

---

## A-05: Recursion depth gaming (REVISED for v3)

**Vector:** Model uses 0 llm() calls on every task to maximize
recursion_efficiency, even on multi_needle / 200K tasks where recursion
would actually help correctness.

**Analysis (v3):**
- recursion_efficiency contributes only when correctness == 1.0 (gating
  in compose.py). On hard tasks where 0 calls fails to produce a correct
  answer, the efficiency bonus is forfeited entirely.
- Net incentive: use the *minimum* recursion that still produces a
  correct answer. Exactly the desired behavior.

**Status:** ✅ MITIGATED by correctness-gating.

---

## A-06 (Stretch): Partial answer overlap gaming

**Vector:** Model outputs `<answer>azure blue</answer>` when gold is `azure` to try to
include the gold answer as a substring.

**Test:**
```python
score = correctness("<answer>azure blue</answer>", "azure")
```

**Expected:** 0.0 (exact-match after normalization, "azure blue" != "azure")

**Result:** 0.0 ✅

**Status:** MITIGATED — Normalizer uses exact-match after case/whitespace/punctuation
normalization. Substring matches are rejected.

---

## A-07: Comment-spam exploit (NEW)

**Vector:** Model emits `# llm(foo)` inside code blocks to inflate the
count regex without making real calls. (Inverted variant of A-05: spam
to make recursion_eff *lower*, useless because lower efficiency hurts.)

**Test:** `test_call_in_comment_not_counted` in
`tests/test_recursion_extract.py`.

**Result:** 0 calls counted ✅ — extractor uses tokenize, ignores comments.

**Status:** ✅ MITIGATED by tokenize-aware extraction.

---

## A-08: String-literal exploit (NEW)

**Vector:** Model writes `"earlier code did llm(...)"` in a string
literal to confuse a naive regex extractor.

**Test:** `test_call_in_string_literal_not_counted`.

**Result:** 0 calls counted ✅ — tokenize correctly identifies STRING
tokens and skips them.

**Status:** ✅ MITIGATED.

---

## Summary

| Attempt | Vector | Status | Mitigation |
|---------|--------|--------|------------|
| A-01 | Format-only (empty answer) | ✅ MITIGATED | A-02 correctness==0 cap (≤0.15 with bonus) |
| A-02 | Case/punctuation gaming | ✅ ACCEPTABLE | Intentional normalizer behavior |
| A-03 | Context verbatim copy | ✅ MITIGATED | Generator `_assert_no_leak()` |
| A-04 | Length reward exploit | ✅ MITIGATED | capped_linear token_budget |
| A-05 | Recursion avoidance | ✅ MITIGATED | Correctness-gating |
| A-06 | Partial answer substring | ✅ MITIGATED | Exact-match normalizer |
| A-07 | Comment-spam exploit | ✅ MITIGATED | Tokenize-aware extraction |
| A-08 | String-literal exploit | ✅ MITIGATED | Tokenize-aware extraction |

**VERDICT: REWARD SYSTEM APPROVED FOR PHASE 2 TRAINING**