fix(grpo): align sys_msg with SFT - emit <answer>...</answer>

Job 69ece94a printed reward=0.0 / reward_std=0.0 / kl=0.0 across all
50 GRPO steps, while completion_length oscillated between 2 and 24
tokens. Diagnosis: rewards/format_gate.py is a multiplicative gate that
returns 0.0 unless the completion contains <answer>...</answer>, and
rewards/compose.py short-circuits the entire composite to 0.0 when the
gate fails (matches env/server/environment.py _ANSWER_RE behavior).

The SFT traces (data/sft_traces.jsonl) ALREADY trained the model to
emit <answer>X</answer> via a system message that says
"Emit your final answer inside <answer>...</answer>."
But the GRPO sys_msg said
"respond with the shortest exact answer span"
- never mentioning the tag - so the post-SFT model dropped the format
on the very first GRPO rollout and never recovered. Every advantage
collapsed to zero, so the policy never updated (loss=0, grad_norm=0).

This patch rewords the GRPO sys_msg to match the SFT framing
("You are FATHOM... emit your final answer inside <answer>...</answer>
tags. Keep the answer the shortest exact span that answers the
question."). The "shortest span" bias is preserved for the
token_budget reward, and the <answer> instruction restores the
format-gate prior the model already learned.

Expected behavior next run:
- format_gate=1 on most rollouts (was 0.0 every step)
- non-zero reward_std within groups -> non-zero advantages
- non-zero loss / grad_norm -> actual policy update
- KL divergence drifts above 0 from beta=0.04

Made-with: Cursor

train/grpo.py

@@ -179,9 +179,16 @@ def run_grpo(
     # `gold_answer`, `prompt_token_count`, `llm_call_count` come along as
     # extra columns and TRL forwards them to the reward function as kwargs
     # (because `remove_unused_columns=False` below).
+    # IMPORTANT: must match the system message used in SFT traces
+    # (data/sft_traces.jsonl). REW-01 format_gate is a multiplicative gate —
+    # if the completion lacks <answer>...</answer>, composite reward = 0.0.
+    # Previous wording ("shortest exact answer span") never told the model
+    # about the tag, so 50/50 GRPO steps had reward=0.0 (job 69ece94a).
     sys_msg = (
-        "You answer questions about long documents. Read the context and "
-        "respond with the shortest exact answer span."
+        "You are FATHOM, a recursive language model. You answer questions "
+        "about long documents. Read the context, think step by step, and "
+        "emit your final answer inside <answer>...</answer> tags. "
+        "Keep the answer the shortest exact span that answers the question."
     )
 
     # vLLM hard-checks final prompt length against the model's max_position_embeddings