Spaces:
Sleeping
Sleeping
Commit History
-
Phase 5: staged read-only email adapter + ops hardening
Staged real-inbox connector (the user's "read-only behind a flag" decision): - env/connectors/: isolated package mapping real mail into the un-privileged Observation schema. base.py (RawEmail + ReadOnlyEmailConnector, no send/delete/mutate), mapping.py (neutral defaults; emits ONLY Observation/ ObservationEmail, never EmailRecord/gold fields -> ungradeable by construction), imap_readonly.py (selects readonly=True, SEARCH/FETCH only; client injected so it's testable without network), config.py (EMAIL_CONNECTOR_ENABLED, default off). - scripts/run_benchmark.py refuses to run if the connector is enabled (benchmark is sim-only by construction). - tests/test_connector.py (fixture-driven, no network) + test_connector_isolation.py (AST import-graph proof: grader/environment never import connectors and vice versa; benchmark-guard raises). tests/fixtures/mail/sample.eml. Ops hardening: - benchmark/results_report.py HTML: a11y pass (lang, <main>, <caption>, th scope=col, thead…0842437 -
Phase 4: advanced agents (reflection, memory determinism, providers)
- env/agents/reflector.py: deterministic propose -> critique -> revise agent. Proposes via BaselinePolicy, critiques against env/safety guardrails (prompt injection / risky content / forbidden escalation), and revises unsafe proposals to a safe action within a bounded retry cap. Provider-free and reproducible. - benchmark/agents.py: ReflectiveAgent benchmark wrapper (name "reflective"); registered in runner.py (run_agent) and scripts/run_benchmark.py (offline agent). - Memory determinism: tests/test_memory_augmented.py locks the few-shot path (trajectory_store -> example_extractor -> prompt_enhancer) as a deterministic, read-only function of its inputs (INV-2 for the LLM prompt path). - More providers: kept the OpenAI-compatible client design (works with OpenAI, Azure, Groq, Ollama, and Anthropic's OpenAI-compatible endpoint via base_url); tests/test_providers.py locks custom base_url/model resolution. Per the claude-api skill's guardrail, did NOT inject the Anthropic…474cdf7 -
Phase 3: eval depth (calibration, cost-aware, credit, contamination)
All additive and report-only; nothing feeds the grader, so golden snapshots are unchanged. - env/eval/calibration.py: pure Brier score + Expected Calibration Error over (confidence, correct) pairs. - benchmark/cost_aware.py: score_per_dollar / score_per_1k_tokens + per-agent cost_efficiency (None when denominator is zero). - benchmark/credit_report.py: aggregate the frozen step_breakdown deltas by action type to show where score is gained/lost. - scripts/contamination_check.py: assert the agent-facing prompt surface never exposes gold answer fields (expected_*/recommended_escalation); exit 1 on leak. - tests/test_eval_depth.py, tests/test_contamination.py: known values + a poisoned-text negative test proving the check fires. 360 tests pass, coverage 81.42%, ruff clean, snapshots unchanged.f477fa6 -
Phase 2: statistical rigor (significance + persisted history)
- benchmark/stats_util.py: single source of truth for the 95% t-critical table and CI helpers; baseline/leaderboard.py now imports it (no duplicate table). - benchmark/significance.py: deterministic PAIRED comparison over the shared (task,persona,seed) grid -- paired t-test via scipy.stats.t (no RNG, INV-2), Cohen's d, significant@95%; compare_all_pairs for the full agent set. Surfaced in results.json under "significance". - benchmark/history.py: append-only leaderboard_history.jsonl with per-agent run summaries + deltas vs the prior run; prior lines never mutated. - scripts/run_benchmark.py: opt-in --record-history / --history-path (off by default). - tests/test_significance.py, tests/test_leaderboard_history.py: pinned outputs, symmetry, determinism, idempotent append, delta correctness. No grader change -> golden snapshots unchanged. 351 tests pass, coverage 81.19%, ruff clean.9ce4a24 -
Phase 0: error-proof invariant harness
Build the test harness every later advancement plugs into, BEFORE any feature, so regressions to the deterministic core are structurally unmergeable. - tests/harness/invariants.py: reusable assertions for the invariant contract (INV-1 open-interval, INV-2 determinism, INV-3 persona-invariant headline, INV-4 bounded reward) + canonical BaselinePolicy rollout helper. - tests/harness/test_invariants_property.py: Hypothesis property/fuzz tests over tasks x personas x generated action sequences (derandomize=True, no flake). - tests/harness/test_golden_snapshots.py + snapshots/baseline_golden.json: full GraderResponse (incl. per-step step_breakdown) frozen for the canonical rollout across tasks x personas x seeds[42,43]. Regen only via scripts/regen_golden.py. - tests/harness/test_determinism_guard.py, test_perf_regression.py, test_data_integrity.py: full-matrix determinism, coarse perf budget, and the INV-5 closed-schema guarantee (gold fields never reach an Observation). - s…81315e9
-
Wave 2 infra: release pipeline, Postgres option, LLM metrics, load test
- .github/workflows/release.yml: tag-triggered GHCR publish + SBOM + GH Release. - Optional DATABASE_URL (Postgres) with pool tuning; SQLite stays the default, psycopg stays an optional dep. +tests/test_db_engine.py. - telemetry: llm_cost_usd_total / llm_tokens_total / llm_latency_ms metrics + record_llm_usage() helper + 3 Grafana panels. +tests/test_llm_metrics.py. - scripts/loadtest/: optional Locust load test + runbook. - Suite: 255 green.78d9fa0 -
Add reproducible results harness, scenario schema, benchmark docs + OSS scaffolding
- benchmark/results_report.py + scripts/run_benchmark.py: run the matrix and emit results.json/csv/html aggregated by (task,persona,agent) with mean +/- 95% CI across seeds. Stdlib-only, runs key-free for baseline/multiagent. - env/scenario_schema.py: pydantic v2 validator for data/scenarios/*.yaml. - docs/BENCHMARK.md: code-derived methodology (tasks, personas, scoring, repro). - .github/ issue+PR templates, CODEOWNERS; CHANGELOG.md (Keep a Changelog). - +16 tests (results harness + scenario schema). Suite: 224 green.5657821