Commit History
-
feat(trainer): policy-optimization objective MENU (ADR-014)
Adds make_po_config(objective=..., **overrides) — RL's base objective is no longer hardcoded to Dr.GRPO. Six selectable named presets over trl 1.5.0's verified GRPOConfig knob-space (introspected the installed package, not a GitHub snapshot): grpo | vanilla GRPO (std-norm advantage) dr_grpo | DEFAULT; no length-std bias (Composer 2.5's base objective) bnpo | batch-normalized dapo | decoupled clip-higher (epsilon_high=0.28) + overlong mask + KL off gspo | sequence-level importance ratio (Qwen3; long-CoT / MoE stable) cispo | detached-IS REINFORCE (every token keeps a gradient; MiniMax-M1) Every preset is PURE CONFIG (trl already implements each loss_type branch + importance_sampling_level/epsilon_high) — no custom _compute_loss. Drift guards assert loss_type / IS-level / epsilon_high actually applied so a preset can't silently degrade (e.g. GSPO with IS overridden back to token raises). 10 unit tests green against real trl 1.5.0. Research-grounded: Composer …aae66fa
-
fix(phase-8): close all 5 cross-family final-verify findings + regression tests
The deep-work-loop exit gate (2 independent confirmations). A final 3-family scatter of the session diff found and we fixed: - [P0 latent] SDPO sentinel-mask used only student_response_valid; now requires BOTH valid masks AND non-sentinel indices (a future divergent teacher tail could have distilled against clamped position 0). - [P1 convergent, GPT-5.5+Gemini] curriculum shared one n_effort denominator for turns+think_tokens; split into independent n_turns/n_think counters. - [P1] docker E2E test ran pip-install under --network none (would fail when the gate activates); replaced with stdlib python -c expression runner, no network. - [P1] MMLU reward: Answer: A or B hedge scored full credit; _HEDGE_RE now penalizes it as multiple-answers. - [P1] HackMonitor read-markers matched import in important / cat in concatenate and scanned the agent patch as read-evidence; whole-word regex + exclude patch/diff payloads from BOTH layers (a legit patch can no longer self-incriminate…678d10b -
feat(wave-a): close ADR-011 (SDPO alignment indices) + ADR-012 (review findings)
B1/ADR-011: collator emits student/teacher_response_idx + valid masks via _mask_to_padded_indices; loss sentinel-masks padding. Strict SDPO no longer raises against the real collator (the regression my review-fix introduced). B2/ADR-012: - k1-KL: found TRL 1.5.0 uses k3 not k1; corrected docstring honestly + documenting test. - hint routing: style/communication/effort sites now reach the judge (error-kind aware). - HackMonitor: added patch-provenance layer defeating string-concat obfuscation; ADR-010 language corrected AST->signature+patch-provenance. - curriculum: optional turns/think_tokens effort signals, backward-compatible. 210 passed / 16 skipped (was 192). Two Opus-4.8 workers in parallel.d02d724 -
review+fix: cross-family adversarial ADR review (owed item) + remediation
Ran the owed 5-family adversarial review of ADR-008/009/010 (unblocked once OpenRouter credits were restored). 4/5 clean reviews (GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Grok 4.3; Kimi starved its token budget). Verdicts: ADR-008 3-REJECT/1-fix, ADR-009 unanimous-fix, ADR-010 3-REJECT/1-fix — all REJECTs were "accepted-status outran the evidence," not "design wrong." The review caught a real policy-corrupting P0 the solo pre-mortem missed: the SDPO alignment guard was a shape-check, so hint-shifted teacher tokens could silently misalign and distill garbage into the policy. Verified every convergent finding against code, then fixed (192 pass/16 skip, was 187): - SDPO alignment (composer_trainer.py): require collator-emitted student_response_idx/teacher_response_idx; gather aligned post-hint logits before JSD; strict-mode raises on missing/mismatched indices. Tests rewritten to the corrected contract (different-length sequences now exercised). - Sandbox scrub (sandbox.py): boo…185cce2 -
feat(trainer): ADR-008 Dr.GRPO config + SDPO strict-alignment guard
Phase-6 execution of ADR-008 (gates 1, 2, 5 green; gate 3 = the live GRPOTrainer smoke runs detached separately). - make_dr_grpo_config(): builds trl.GRPOConfig to the Dr. GRPO recipe (loss_type=dr_grpo => no length-standardization bias; scale_rewards=none => no std-dev advantage normalization; num_iterations=1 => single-epoch), with a drift-guard assertion. TRL 1.5.0 natively supports these knobs and its own help text cites the Dr. GRPO paper for both. - ComposerReplicationTrainer.strict_sdpo_alignment (default True): the student/teacher logit shape-mismatch path now RAISES instead of silently zeroing the SDPO channel (closes the ADR-008 trust-gap, composer_trainer.py lines 158-160). Opt-out to warn-and-skip for production resilience. - prime_rl/composer_loss.py: NotImplementedError(alpha_sdpo>0) message now points at the TRL host as the SDPO home (documents the ADR-006 amendment). - Tests: 7 new (config knobs asserted; strict raises / non-strict skips / aligned fir…bde5c5e
-
Wave 21b: skip zero-signal SDPO on empty-recovery error turns + real-trace validation
Found by validating the Wave 21 pipeline against REAL Claude Code session logs (738 local sessions, 66 with is_error:true) rather than only the synthetic fixture. Two outcomes: a collator fix and a permanent validation example. ## Collator fix: empty-recovery error turns On real traces, ~67% of error sites have EMPTY recovery content when strip_thinking=True — because the error-RECOVERY turn is frequently pure [THINKING] (the model reasons about the failure, then silently retries a tool), and stripping thinking empties it. The old code set any_errors=True and injected a hint the moment hint_text existed, BEFORE checking recovery content, so an empty-recovery turn produced an all-ignore_index sdpo_loss_mask: a zero-signal SDPO row that wastes a forward pass and dilutes the channel. Fix: only treat a turn as an SDPO error site when BOTH a hint was produced AND the recovery turn has content (`hint_text and turn.get("content")`). Applied symmetrically to `_build_hint_injected_trace` (te…d61036a -
Wave 21: close both Wave 20 debt items — chat-template alignment + structural is_error
Two architectural-debt items flagged "Wave 20 candidate" in the framework doc, both CPU-only, both now fixed and regression-guarded. ## 1. SDPO mask chat-template drift (was ~67% aligned, now ~100%) `ComposerDataCollator._build_segment_mask` built the sdpo_loss_mask (and the aligned-student response_mask) by tokenizing each content segment in isolation and concatenating. That ignored the scaffolding tokens apply_chat_template inserts around every message (<|im_start|>{role}\n ... <|im_end|>\n, BOS), so mask bits drifted left of the real content tokens — the residual ~33% contamination documented in the Wave 19 production audit. Fix: new `_build_chat_aligned_mask` derives the mask from per-message apply_chat_template prefix deltas. For message k it computes the token span len(template(msgs[:k+1])) - len(template(msgs[:k])), then locates the content run inside that span by subsequence match and marks ONLY those positions as loss. Falls back to whole-span marking if the content run can…6806cf7
-
Wave 19: production-grade SDPO via ComposerDataCollator + adapter + collator fixes
Adds the full production data path Wave 18 deferred: ClaudeCodeIngester → adapter → ComposerDataCollator → compose_loss with proper hint injection at detected error sites. Two rounds of 3-reviewer cross-family review caught a critical SDPO mask alignment bug in round 1; round 2 verified the fix. NEW infrastructure: - composer_replication/ingestion/trace_examples.py: claude_states_to_trace_examples() adapter walks ClaudeCodeIngester output, detects [TOOL_RESULT (ERROR)] tagged user turns, marks the recovery assistant turn with tool_error="<kind>" so ComposerDataCollator._is_error_turn picks it up. Default classifier handles file_not_found, permission_denied, command_not_found, syntax_error, connection_error; users can pass custom error_kind_fn. Backward-scan finds errors even when intervening user turns separate the error from recovery. - composer_replication/ingestion/tests/test_trace_examples_adapter.py: 14 tests pinning the adapter contract (error detection, classifica…03bf323
-
Wave 15: 4-angle multi-model self-critique caught 2 math BLOCKERs in primary loss kernels; fixed against upstream byte-for-byte + GSM8K example + ergonomics
PHASE 15a: 4 parallel adversarial reviewers, each with different framing to maximize independent-angle coverage: - MATH (Opus 4.7): cloned upstream OPSD/TAID/PRIME-RL/SimPO, did line-by-line diff. Found 2 BLOCKERs all 8+ prior reviewers missed. - TESTS (Opus 4.7): scrutinized 3 high-stakes test files for weak assertions. Found PRIME-RL parity test silently never runs; bit-exact test uses allclose; entropy-OPD test is pure smoke. - DOCS (Opus 4.7): audited 6 major docs + ADRs. Found test-count drift (77/107/124 vs 145), compose_loss kwarg drift, stale "Deferred to Wave 14" claim, PRIME-RL test count 10 vs 16. - USER-JOURNEY (Opus 4.7): walked "RL-finetune Qwen-7B on GSM8K" end-to-end. Scored framework 2.4/5 with Real-Task-Path = 1/5. #1 friction: NO GSM8K example anywhere. All 4 reports saved to /tmp/wave15_*.md. Headline: math reviewer's upstream-clone-and-diff approach caught what paper-level review missed. PHASE 15b: synthesized findings into 5-task fix scatter. PHA…e5add15 -
Wave 10 — packaging: composer_replication is now pip-installable
Promotes the framework from "skeleton scattered across spike directories" to a proper installable Python package (closes vision-validation gap V4). What's new: pyproject.toml at repo root - Hatchling build backend - Python ≥3.10 - Core deps: torch≥2.0, transformers≥4.46 - Optional extras: replay (httpx), diloco (torchft-nightly), train (trl, peft, accelerate, datasets), dev (everything + pytest, ruff) - Project URLs point to the HF repo + Discussions composer_replication/ package (8 modules) - __init__.py re-exports the framework's public API: compose_loss, LossComponents, build_batch (Spike 006) generalized_jsd_loss (verified port of OPSD) ClaudeCodeIngester, IngestionStats, SYSTEM_PROMPT (Spike 007) TraceState, DPOPair, TeacherSpec, replay_trace, extract_dpo_pairs, DEFAULT_TEACHERS (Spike 001/005) ComposerReplicationTrainer (Spike 005, TRL subclass) make_diloco_outer_loop …ac05fbf