REWARD_AUDIT.md

REWARD AUDIT — REW-05

Date: 2026-04-25 Auditor: Pratham (Plan 01-05 executor) Scope: rewards/ package — 4 components + composition layer

---

A-01: Format-only shortcut exploit

Vector: Model learns to always output <answer></answer> (empty or random) because format_gate=1.0 still gives partial credit via token_budget + recursion_efficiency.

Test:

score = compose_reward_single("<answer></answer>", "Rome", 100, 0, cfg_reward)

Expected: ≤ 0.15 (A-02 cap is 0.05, plus 0.10 soft format bonus)

Result: 0.15 (matches cap) ✅

Status: MITIGATED — A-02 correctness==0 cap blocks this. Empty answers have correctness=0, so even with the format bonus, the maximum is 0.15.

---

A-02: Exact case/punctuation gaming

Vector: Model outputs <answer>ROME.</answer> — exact text with trailing period — to exploit correctness normalizer edge case.

Test:

score = correctness("<answer>ROME.</answer>", "Rome")

Expected: 1.0 (normalizer strips trailing punctuation and lowercases)

Result: 1.0 ✅

Status: ACCEPTABLE — This is correct behavior; normalizer intentionally handles casing/punctuation.

---

A-03: Gold answer verbatim copy from context

Vector: Model copies the gold answer from the context before the document is masked, or finds a copy of the gold answer in the context itself (DATA-04 post-check).

Test: test_no_gold_answer_verbatim_leak in tests/test_dataset.py

Expected: Zero examples where gold_answer.lower() in context.lower()

Result: 0 violations across 1200 examples (1000 train + 200 eval) ✅

Status: MITIGATED — Generator _assert_no_leak() blocks this at generation time.

---

A-04: Length reward gaming (write more = score more)

Vector: Model writes extremely long responses to maximize token_budget reward component.

Test: test_no_monotonic_length_exploit (REW-06 audit, tests/test_rewards.py)

Expected: Longer completions score lower (capped_linear is strictly decreasing in length)

Result: Confirmed — longest completion (20000 words) scores < shortest (10 words) ✅

Status: MITIGATED — token_budget(variant="capped_linear") with α=0.2 ensures score = max(0, 1 - 0.2 * ratio) — ratio rises with length, score falls.

---

A-05: Recursion depth gaming (REVISED for v3)

Vector: Model uses 0 llm() calls on every task to maximize recursion_efficiency, even on multi_needle / 200K tasks where recursion would actually help correctness.

Analysis (v3):

• recursion_efficiency contributes only when correctness == 1.0 (gating in compose.py). On hard tasks where 0 calls fails to produce a correct answer, the efficiency bonus is forfeited entirely.
• Net incentive: use the minimum recursion that still produces a correct answer. Exactly the desired behavior.

Status: ✅ MITIGATED by correctness-gating.

---

A-06 (Stretch): Partial answer overlap gaming

Vector: Model outputs <answer>azure blue</answer> when gold is azure to try to include the gold answer as a substring.

Test:

score = correctness("<answer>azure blue</answer>", "azure")

Expected: 0.0 (exact-match after normalization, "azure blue" != "azure")

Result: 0.0 ✅

Status: MITIGATED — Normalizer uses exact-match after case/whitespace/punctuation normalization. Substring matches are rejected.

---

A-07: Comment-spam exploit (NEW)

Vector: Model emits # llm(foo) inside code blocks to inflate the count regex without making real calls. (Inverted variant of A-05: spam to make recursion_eff lower, useless because lower efficiency hurts.)

Test: test_call_in_comment_not_counted in tests/test_recursion_extract.py.

Result: 0 calls counted ✅ — extractor uses tokenize, ignores comments.

Status: ✅ MITIGATED by tokenize-aware extraction.

---

A-08: String-literal exploit (NEW)

Vector: Model writes "earlier code did llm(...)" in a string literal to confuse a naive regex extractor.

Test: test_call_in_string_literal_not_counted.

Result: 0 calls counted ✅ — tokenize correctly identifies STRING tokens and skips them.

Status: ✅ MITIGATED.

---

Summary

Attempt	Vector	Status	Mitigation
A-01	Format-only (empty answer)	✅ MITIGATED	A-02 correctness==0 cap (≤0.15 with bonus)
A-02	Case/punctuation gaming	✅ ACCEPTABLE	Intentional normalizer behavior
A-03	Context verbatim copy	✅ MITIGATED	Generator _assert_no_leak()
A-04	Length reward exploit	✅ MITIGATED	capped_linear token_budget
A-05	Recursion avoidance	✅ MITIGATED	Correctness-gating
A-06	Partial answer substring	✅ MITIGATED	Exact-match normalizer
A-07	Comment-spam exploit	✅ MITIGATED	Tokenize-aware extraction
A-08	String-literal exploit	✅ MITIGATED	Tokenize-aware extraction

VERDICT: REWARD SYSTEM APPROVED FOR PHASE 2 TRAINING