Spaces:
Sleeping
Sleeping
Commit History
-
Phase 5: staged read-only email adapter + ops hardening
Staged real-inbox connector (the user's "read-only behind a flag" decision): - env/connectors/: isolated package mapping real mail into the un-privileged Observation schema. base.py (RawEmail + ReadOnlyEmailConnector, no send/delete/mutate), mapping.py (neutral defaults; emits ONLY Observation/ ObservationEmail, never EmailRecord/gold fields -> ungradeable by construction), imap_readonly.py (selects readonly=True, SEARCH/FETCH only; client injected so it's testable without network), config.py (EMAIL_CONNECTOR_ENABLED, default off). - scripts/run_benchmark.py refuses to run if the connector is enabled (benchmark is sim-only by construction). - tests/test_connector.py (fixture-driven, no network) + test_connector_isolation.py (AST import-graph proof: grader/environment never import connectors and vice versa; benchmark-guard raises). tests/fixtures/mail/sample.eml. Ops hardening: - benchmark/results_report.py HTML: a11y pass (lang, <main>, <caption>, th scope=col, thead…0842437 -
Phase 4: advanced agents (reflection, memory determinism, providers)
- env/agents/reflector.py: deterministic propose -> critique -> revise agent. Proposes via BaselinePolicy, critiques against env/safety guardrails (prompt injection / risky content / forbidden escalation), and revises unsafe proposals to a safe action within a bounded retry cap. Provider-free and reproducible. - benchmark/agents.py: ReflectiveAgent benchmark wrapper (name "reflective"); registered in runner.py (run_agent) and scripts/run_benchmark.py (offline agent). - Memory determinism: tests/test_memory_augmented.py locks the few-shot path (trajectory_store -> example_extractor -> prompt_enhancer) as a deterministic, read-only function of its inputs (INV-2 for the LLM prompt path). - More providers: kept the OpenAI-compatible client design (works with OpenAI, Azure, Groq, Ollama, and Anthropic's OpenAI-compatible endpoint via base_url); tests/test_providers.py locks custom base_url/model resolution. Per the claude-api skill's guardrail, did NOT inject the Anthropic…474cdf7 -
Phase 2: statistical rigor (significance + persisted history)
- benchmark/stats_util.py: single source of truth for the 95% t-critical table and CI helpers; baseline/leaderboard.py now imports it (no duplicate table). - benchmark/significance.py: deterministic PAIRED comparison over the shared (task,persona,seed) grid -- paired t-test via scipy.stats.t (no RNG, INV-2), Cohen's d, significant@95%; compare_all_pairs for the full agent set. Surfaced in results.json under "significance". - benchmark/history.py: append-only leaderboard_history.jsonl with per-agent run summaries + deltas vs the prior run; prior lines never mutated. - scripts/run_benchmark.py: opt-in --record-history / --history-path (off by default). - tests/test_significance.py, tests/test_leaderboard_history.py: pinned outputs, symmetry, determinism, idempotent append, delta correctness. No grader change -> golden snapshots unchanged. 351 tests pass, coverage 81.19%, ruff clean.9ce4a24
-
Add reproducible results harness, scenario schema, benchmark docs + OSS scaffolding
- benchmark/results_report.py + scripts/run_benchmark.py: run the matrix and emit results.json/csv/html aggregated by (task,persona,agent) with mean +/- 95% CI across seeds. Stdlib-only, runs key-free for baseline/multiagent. - env/scenario_schema.py: pydantic v2 validator for data/scenarios/*.yaml. - docs/BENCHMARK.md: code-derived methodology (tasks, personas, scoring, repro). - .github/ issue+PR templates, CODEOWNERS; CHANGELOG.md (Keep a Changelog). - +16 tests (results harness + scenario schema). Suite: 224 green.5657821