# TODO: E8 Redesign — Tasks That Actually Require Domain Knowledge Date: 2026-03-19 Author: Analysis based on E8 results (24/24 passes, 0 differentiation) --- ## Problem: Ceiling Effect in E8 E8 ran 4 domain-knowledge scenarios (enum-convention, timeout-unit, breaking-change, implicit-null) with baseline vs injected agents, 3 runs each (pass@3), 24 total runs. Result: **12/12 baseline passed. 12/12 injected passed.** Zero differentiation. The experiment cannot answer its own hypothesis because both agents behave identically. There is no signal. --- ## Root Cause: Bugs Were Readable From the Code Alone Each E8 bug had one fatal design flaw: **the correct fix was discoverable by reading the existing codebase**. The agent did not need injected knowledge — it could infer the answer from what was already in the repo. Concrete failure modes per scenario: **E8a enum-convention** ("use UPPERCASE enums") - The bug is a wrong string literal. Any agent that reads the schema or existing enum values finds UPPERCASE immediately. The injection text says exactly what the code already says. - Baseline solves it by reading `schema.sql` or other model files. Injection adds nothing. **E8b timeout-unit** ("timeouts are in milliseconds") - The variable is named `timeout_ms`. The name itself encodes the unit. - A baseline agent reads the variable name, infers milliseconds, converts 30 to 30000. The injection text repeats what the variable name already implies. **E8c breaking-change** ("find_one() is now get_by_id()") - The error message is `AttributeError: 'Database' object has no attribute 'find_one'`. The stack trace points directly to the broken call. The agent reads `database.py`, finds `get_by_id`, and substitutes. No tribal knowledge needed — it is a grep task. **E8d implicit-null** ("email has NOT NULL constraint") - The schema file is in the repo. Reading `schema.sql` reveals `NOT NULL` on email. The injection text describes what the file already shows. **The pattern**: Every E8 bug had its answer in the repository. Injection text described the repository. Baseline read the repository. Same result. --- ## Design Principles for Differentiating Domain-Knowledge Tasks A task truly requires domain knowledge when the correct fix **cannot be derived from the code alone**. This means one or more of the following must be true: ### 1. The Convention Exists Outside the Repo The rule is not documented in any file the agent can read. It lives in team memory, architectural decisions made before the current codebase existed, or external systems the agent cannot access (CI config on a remote server, a Confluence page, a Slack decision). Example: "We always use snake_case for RabbitMQ routing keys" — nothing in the code enforces this, there is no linter rule, but mixing conventions breaks queue binding silently. ### 2. Multiple Plausible Fixes Exist — Only One Is Correct For This Project The code analysis narrows it to 2-3 valid approaches. Without context, the agent picks one arbitrarily. With injection, it picks the project-correct one. Example: A retry function fails. Fix options: (a) increase max_retries, (b) add exponential backoff, (c) catch a broader exception class. All three "fix" the immediate test. Only one matches the team's chosen retry strategy. ### 3. The Bug Is Caused By an Implicit Contract With an External System The external system's behavior is not captured anywhere in the repo. The agent sees the Python code but not the API spec, the database behavior under load, or the message broker version. Example: A third-party payment API silently truncates decimal amounts to 2 places, but only in production. The code passes 3 decimal places. No test in the repo captures this. ### 4. The Fix Requires Knowing What Not To Change The task has a correct minimal fix and several incorrect expansive fixes. Baseline agents tend to make the expansive fix (refactor, add validation, extract helper). The project convention is minimal patches to stable modules. Only an agent with that context makes the right narrow change. ### 5. There Are Two Bugs — The Injected Context Identifies Which One Is The Real One The symptom has two potential causes: a logic bug in module A and a config bug in module B. Both would produce the same error. Baseline agents fix one or the other. The injection text identifies which one is the known issue. --- ## 5 Concrete New Task Ideas Each task below is designed so that: - Baseline agent: reads all files, finds a plausible fix, applies it, tests pass or fail ambiguously - Injected agent: applies the project-correct fix because the injection resolves the ambiguity --- ### Task E9a: `retry-strategy-conflict` **Bug description**: A background job processor retries failed HTTP requests. It currently retries on any exception. This causes duplicate charges when the payment API returns a 200 with a timeout on the client side (the charge went through but the response never arrived). The team's rule: only retry on connection errors (requests.ConnectionError), never on timeouts (requests.Timeout). **Why baseline fails**: The code has an except block catching `Exception`. A baseline agent sees this and might fix it by catching `requests.exceptions.RequestException` (broader, still wrong), or by adding a retry count (doesn't fix the duplicate charge problem), or by catching `(ConnectionError, Timeout)` (wrong — includes Timeout). Three plausible fixes, one is correct for this project. **Injection text**: "In diesem Projekt werden HTTP-Retries NUR bei requests.ConnectionError ausgeloest — requests.Timeout wird NICHT retried (Charge kann bereits erfolgt sein). Siehe DECISIONS.md." **DECISIONS.md content** (not in repo, only known via injection): Does not exist as a file. The rule is pure tribal knowledge. **Repo structure**: ``` retry-strategy-conflict/ processor.py # background job, broken retry logic payment_client.py # thin HTTP wrapper tests/ test_processor.py # tests retry on ConnectionError, also tests NO retry on Timeout judge_retry_strategy.py ``` **Judge**: Checks that the implementation retries on `ConnectionError` but raises on `Timeout` without retrying. The test cannot be "cheated" by catching all exceptions. **Differentiation mechanism**: Baseline catches either too broadly or too narrowly. The injection provides the exact project rule that resolves the ambiguity. --- ### Task E9b: `event-ordering-convention` **Bug description**: An event sourcing system publishes domain events. Two event handlers are registered for `UserRegistered`: one sends a welcome email, one creates a billing record. The billing record creation is failing intermittently because the email handler sometimes mutates shared state before billing runs. The project convention: event handlers must be idempotent and stateless — shared state mutation is banned. The fix is to remove the state mutation from the email handler, not to add locking. **Why baseline fails**: The agent sees an intermittent failure, shared state, and two handlers. Plausible fixes: add a lock around the handlers, change handler order, remove the shared state, or make the email handler async. Code analysis cannot determine which approach the project uses — all four would fix the test. **Injection text**: "Event-Handler in diesem System MUESSEN stateless und idempotent sein — kein Shared-State zwischen Handlern. Locks sind kein akzeptierter Fix. Mutation entfernen, nicht serialisieren." **Repo structure**: ``` event-ordering-convention/ event_bus.py # simple sync event dispatcher handlers/ email_handler.py # mutates _processed_users set (the bug) billing_handler.py # reads _processed_users, fails if wrong tests/ test_event_ordering.py judge_event_ordering.py ``` **Judge**: Checks that the email handler contains no references to shared mutable state, AND that both handlers pass the idempotency test (called twice with same event produces same result). --- ### Task E9c: `migration-direction` **Bug description**: A database migration script fails on CI. The script adds a new column `archived_at` to the `posts` table. It fails because the CI database already has this column (added by a hotfix two weeks ago). The correct fix in this project: mark the migration as non-destructive and add a guard (`IF NOT EXISTS` or equivalent). The wrong fix: delete the column and re-add it (destroys production data), or skip the migration (leaves migration history inconsistent). **Why baseline fails**: The agent sees a migration failure and an existing column. Options: (a) add `IF NOT EXISTS`, (b) drop and recreate, (c) skip migration, (d) rename to a new column. The code gives no hint which approach the project uses. The migration framework (custom, not alembic) has no documented conventions in the repo. **Injection text**: "Dieses Projekt verwendet additive Migrationen — niemals Spalten droppen oder umbenennen in Produktions-Migrationen. Bei bereits existierenden Spalten: IF NOT EXISTS Guard hinzufuegen. Kein Skip, kein Recreate." **Repo structure**: ``` migration-direction/ migrations/ 001_create_posts.sql 002_add_archived_at.sql # the broken migration db.py # custom migration runner tests/ test_migrations.py # runs migrations on a fresh SQLite DB + a DB with the column pre-added judge_migration_direction.py ``` **Judge**: Runs migration against (a) a fresh DB and (b) a DB that already has `archived_at`. Both must succeed. Also checks that the migration does not contain DROP COLUMN. --- ### Task E9d: `cache-key-namespace` **Bug description**: A caching layer uses Redis. Two services share the same Redis instance. The `UserService` caches user objects with key `user:{id}`. The `SessionService` also stores data with key `user:{id}` (different data structure). Reads from one service corrupt the other. The fix: `UserService` keys must be namespaced as `svc:user:{id}`. This namespace convention exists across all services but is not documented in the repo — it was decided before the current code was written. **Why baseline fails**: The agent sees a key collision. Plausible fixes: add a prefix to UserService (`u:{id}`, `users:{id}`, `user_svc:{id}`), or add a prefix to SessionService, or use a separate Redis DB number, or use a hash. Without knowing the convention, the agent picks one of these. Any prefix "fixes" the collision test — but only `svc:{service}:{id}` matches the project convention checked by the judge. **Injection text**: "Redis-Keys in diesem System folgen dem Schema `svc:{service_name}:{id}`. UserService verwendet `svc:user:{id}`, SessionService `svc:session:{id}`. Kein anderes Format." **Repo structure**: ``` cache-key-namespace/ user_service.py # caches with key "user:{id}" (wrong) session_service.py # caches with key "user:{id}" (different data) cache.py # thin Redis wrapper tests/ test_cache_isolation.py judge_cache_namespace.py ``` **Judge**: Checks that `user_service.py` uses exactly `svc:user:{id}` format (regex match on the key string in the source). Tests also verify both services can coexist without collision. --- ### Task E9e: `error-propagation-boundary` **Bug description**: A service layer calls a repository layer. The repository raises a custom `RepositoryError` when the database is unavailable. The service currently lets this exception propagate to the HTTP handler, which returns a 500. The project convention: service layer must catch infrastructure errors and re-raise as `ServiceUnavailableError` (defined in `errors.py`). The HTTP handler then maps this to a 503 with a proper body. Currently the handler never returns 503 — always 500. **Why baseline fails**: The agent sees the exception propagation and knows the handler checks for `ServiceUnavailableError`. Two plausible fixes: (a) catch in service and re-raise as `ServiceUnavailableError`, (b) catch in handler and map `RepositoryError` directly. Both fix the test if the judge only checks the HTTP status. But the judge also checks the handler does not import from the repository layer (layering rule). Without knowing this architectural rule, a baseline agent will likely fix it at the handler level. **Injection text**: "Architektur-Regel: HTTP-Handler importieren NIEMALS aus der Repository-Schicht. Fehler werden an Service-Schicht-Grenzen in ServiceUnavailableError gewrappt. Handler kennen nur Service-Layer-Exceptions." **Repo structure**: ``` error-propagation-boundary/ repository.py # raises RepositoryError service.py # missing try/except (the bug) errors.py # ServiceUnavailableError definition handler.py # maps ServiceUnavailableError -> 503 tests/ test_error_propagation.py judge_error_propagation.py ``` **Judge**: Checks (a) HTTP response is 503 when DB is unavailable, AND (b) `handler.py` does not import from `repository.py` (AST check or grep). Both conditions must pass. --- ## Success Criteria: Validate Tasks Before Full Experiment Do not run pass@3 with 8 agents until you have confirmed each task actually differentiates. The validation protocol: ### Step 1: Manual Solve Test (5 minutes per task) Before building the judge, solve the task yourself knowing only what the code says. If you can identify the correct fix without the injection text, the task will not differentiate. Iterate on the repo until the code alone is ambiguous. ### Step 2: Ambiguity Audit For each task, list every plausible fix a smart agent might apply given only the code. There must be at least 2 plausible fixes, and exactly 1 must be project-correct. If only 1 fix is plausible from code analysis, the task is too easy (E8 failure mode). ### Step 3: Pilot Run (4 runs before committing to full experiment) Run 2 baseline + 2 injected before running pass@3. Target: baseline 0-1/2, injected 2/2. If baseline already passes 2/2, stop and redesign. A task is ready for the full experiment when the pilot shows: - Baseline pass rate: 0-33% (0/2 or 1/3) - Injected pass rate: 67-100% (2/3 or 3/3) ### Step 4: Judge Quality Check The judge must: - Not be solvable by blind search-and-replace - Check the specific structural property (not just "tests pass") - Be deterministic — same code, same result every time - Have at least one check that distinguishes correct-convention from wrong-convention fixes (a fix that passes functional tests but violates convention must FAIL the judge) ### Step 5: E9 Pre-Flight Checklist Before running E9: - [ ] All 5 repos exist at their `file:///tmp/arena-tasks/` paths - [ ] Judge scripts tested manually: correct fix passes, plausible-but-wrong fix fails - [ ] Pilot run complete (2+2) with differentiation signal visible - [ ] Injection text is short (1-2 sentences), not a solution walkthrough - [ ] Task descriptions do not mention the domain convention (no hints in the prompt) --- ## What to Build Next Priority order for implementation: 1. **E9c migration-direction** — simplest repo, judge is deterministic (grep for DROP COLUMN + run migrations twice). Good smoke test for the new task format. 2. **E9e error-propagation-boundary** — judge uses AST check, harder to game. Strong architectural ambiguity that Sonnet regularly gets wrong. 3. **E9a retry-strategy-conflict** — needs careful test design (mock the Timeout scenario without actually timing out). Medium complexity. 4. **E9d cache-key-namespace** — straightforward but the judge regex must be tight enough to reject partial matches. 5. **E9b event-ordering-convention** — most complex repo (event bus + multiple handlers). Build last. Each task needs: repo setup script, judge script, YAML file, pilot run before E9 proper.