TODO: E8 Redesign — Tasks That Actually Require Domain Knowledge
Date: 2026-03-19 Author: Analysis based on E8 results (24/24 passes, 0 differentiation)
Problem: Ceiling Effect in E8
E8 ran 4 domain-knowledge scenarios (enum-convention, timeout-unit, breaking-change, implicit-null) with baseline vs injected agents, 3 runs each (pass@3), 24 total runs.
Result: 12/12 baseline passed. 12/12 injected passed.
Zero differentiation. The experiment cannot answer its own hypothesis because both agents behave identically. There is no signal.
Root Cause: Bugs Were Readable From the Code Alone
Each E8 bug had one fatal design flaw: the correct fix was discoverable by reading the existing codebase. The agent did not need injected knowledge — it could infer the answer from what was already in the repo.
Concrete failure modes per scenario:
E8a enum-convention ("use UPPERCASE enums")
- The bug is a wrong string literal. Any agent that reads the schema or existing enum values finds UPPERCASE immediately. The injection text says exactly what the code already says.
- Baseline solves it by reading
schema.sqlor other model files. Injection adds nothing.
E8b timeout-unit ("timeouts are in milliseconds")
- The variable is named
timeout_ms. The name itself encodes the unit. - A baseline agent reads the variable name, infers milliseconds, converts 30 to 30000. The injection text repeats what the variable name already implies.
E8c breaking-change ("find_one() is now get_by_id()")
- The error message is
AttributeError: 'Database' object has no attribute 'find_one'. The stack trace points directly to the broken call. The agent readsdatabase.py, findsget_by_id, and substitutes. No tribal knowledge needed — it is a grep task.
E8d implicit-null ("email has NOT NULL constraint")
- The schema file is in the repo. Reading
schema.sqlrevealsNOT NULLon email. The injection text describes what the file already shows.
The pattern: Every E8 bug had its answer in the repository. Injection text described the repository. Baseline read the repository. Same result.
Design Principles for Differentiating Domain-Knowledge Tasks
A task truly requires domain knowledge when the correct fix cannot be derived from the code alone. This means one or more of the following must be true:
1. The Convention Exists Outside the Repo
The rule is not documented in any file the agent can read. It lives in team memory, architectural decisions made before the current codebase existed, or external systems the agent cannot access (CI config on a remote server, a Confluence page, a Slack decision).
Example: "We always use snake_case for RabbitMQ routing keys" — nothing in the code enforces this, there is no linter rule, but mixing conventions breaks queue binding silently.
2. Multiple Plausible Fixes Exist — Only One Is Correct For This Project
The code analysis narrows it to 2-3 valid approaches. Without context, the agent picks one arbitrarily. With injection, it picks the project-correct one.
Example: A retry function fails. Fix options: (a) increase max_retries, (b) add exponential backoff, (c) catch a broader exception class. All three "fix" the immediate test. Only one matches the team's chosen retry strategy.
3. The Bug Is Caused By an Implicit Contract With an External System
The external system's behavior is not captured anywhere in the repo. The agent sees the Python code but not the API spec, the database behavior under load, or the message broker version.
Example: A third-party payment API silently truncates decimal amounts to 2 places, but only in production. The code passes 3 decimal places. No test in the repo captures this.
4. The Fix Requires Knowing What Not To Change
The task has a correct minimal fix and several incorrect expansive fixes. Baseline agents tend to make the expansive fix (refactor, add validation, extract helper). The project convention is minimal patches to stable modules. Only an agent with that context makes the right narrow change.
5. There Are Two Bugs — The Injected Context Identifies Which One Is The Real One
The symptom has two potential causes: a logic bug in module A and a config bug in module B. Both would produce the same error. Baseline agents fix one or the other. The injection text identifies which one is the known issue.
5 Concrete New Task Ideas
Each task below is designed so that:
- Baseline agent: reads all files, finds a plausible fix, applies it, tests pass or fail ambiguously
- Injected agent: applies the project-correct fix because the injection resolves the ambiguity
Task E9a: retry-strategy-conflict
Bug description: A background job processor retries failed HTTP requests. It currently retries on any exception. This causes duplicate charges when the payment API returns a 200 with a timeout on the client side (the charge went through but the response never arrived). The team's rule: only retry on connection errors (requests.ConnectionError), never on timeouts (requests.Timeout).
Why baseline fails: The code has an except block catching Exception. A baseline agent
sees this and might fix it by catching requests.exceptions.RequestException (broader, still
wrong), or by adding a retry count (doesn't fix the duplicate charge problem), or by
catching (ConnectionError, Timeout) (wrong — includes Timeout). Three plausible fixes,
one is correct for this project.
Injection text: "In diesem Projekt werden HTTP-Retries NUR bei requests.ConnectionError ausgeloest — requests.Timeout wird NICHT retried (Charge kann bereits erfolgt sein). Siehe DECISIONS.md."
DECISIONS.md content (not in repo, only known via injection): Does not exist as a file. The rule is pure tribal knowledge.
Repo structure:
retry-strategy-conflict/
processor.py # background job, broken retry logic
payment_client.py # thin HTTP wrapper
tests/
test_processor.py # tests retry on ConnectionError, also tests NO retry on Timeout
judge_retry_strategy.py
Judge: Checks that the implementation retries on ConnectionError but raises on
Timeout without retrying. The test cannot be "cheated" by catching all exceptions.
Differentiation mechanism: Baseline catches either too broadly or too narrowly. The injection provides the exact project rule that resolves the ambiguity.
Task E9b: event-ordering-convention
Bug description: An event sourcing system publishes domain events. Two event handlers
are registered for UserRegistered: one sends a welcome email, one creates a billing
record. The billing record creation is failing intermittently because the email handler
sometimes mutates shared state before billing runs. The project convention: event handlers
must be idempotent and stateless — shared state mutation is banned. The fix is to remove
the state mutation from the email handler, not to add locking.
Why baseline fails: The agent sees an intermittent failure, shared state, and two handlers. Plausible fixes: add a lock around the handlers, change handler order, remove the shared state, or make the email handler async. Code analysis cannot determine which approach the project uses — all four would fix the test.
Injection text: "Event-Handler in diesem System MUESSEN stateless und idempotent sein — kein Shared-State zwischen Handlern. Locks sind kein akzeptierter Fix. Mutation entfernen, nicht serialisieren."
Repo structure:
event-ordering-convention/
event_bus.py # simple sync event dispatcher
handlers/
email_handler.py # mutates _processed_users set (the bug)
billing_handler.py # reads _processed_users, fails if wrong
tests/
test_event_ordering.py
judge_event_ordering.py
Judge: Checks that the email handler contains no references to shared mutable state, AND that both handlers pass the idempotency test (called twice with same event produces same result).
Task E9c: migration-direction
Bug description: A database migration script fails on CI. The script adds a new column
archived_at to the posts table. It fails because the CI database already has this
column (added by a hotfix two weeks ago). The correct fix in this project: mark the
migration as non-destructive and add a guard (IF NOT EXISTS or equivalent). The wrong
fix: delete the column and re-add it (destroys production data), or skip the migration
(leaves migration history inconsistent).
Why baseline fails: The agent sees a migration failure and an existing column. Options:
(a) add IF NOT EXISTS, (b) drop and recreate, (c) skip migration, (d) rename to a new
column. The code gives no hint which approach the project uses. The migration framework
(custom, not alembic) has no documented conventions in the repo.
Injection text: "Dieses Projekt verwendet additive Migrationen — niemals Spalten droppen oder umbenennen in Produktions-Migrationen. Bei bereits existierenden Spalten: IF NOT EXISTS Guard hinzufuegen. Kein Skip, kein Recreate."
Repo structure:
migration-direction/
migrations/
001_create_posts.sql
002_add_archived_at.sql # the broken migration
db.py # custom migration runner
tests/
test_migrations.py # runs migrations on a fresh SQLite DB + a DB with the column pre-added
judge_migration_direction.py
Judge: Runs migration against (a) a fresh DB and (b) a DB that already has archived_at.
Both must succeed. Also checks that the migration does not contain DROP COLUMN.
Task E9d: cache-key-namespace
Bug description: A caching layer uses Redis. Two services share the same Redis instance.
The UserService caches user objects with key user:{id}. The SessionService also stores
data with key user:{id} (different data structure). Reads from one service corrupt the
other. The fix: UserService keys must be namespaced as svc:user:{id}. This namespace
convention exists across all services but is not documented in the repo — it was decided
before the current code was written.
Why baseline fails: The agent sees a key collision. Plausible fixes: add a prefix to
UserService (u:{id}, users:{id}, user_svc:{id}), or add a prefix to SessionService,
or use a separate Redis DB number, or use a hash. Without knowing the convention, the
agent picks one of these. Any prefix "fixes" the collision test — but only svc:{service}:{id}
matches the project convention checked by the judge.
Injection text:
"Redis-Keys in diesem System folgen dem Schema svc:{service_name}:{id}. UserService
verwendet svc:user:{id}, SessionService svc:session:{id}. Kein anderes Format."
Repo structure:
cache-key-namespace/
user_service.py # caches with key "user:{id}" (wrong)
session_service.py # caches with key "user:{id}" (different data)
cache.py # thin Redis wrapper
tests/
test_cache_isolation.py
judge_cache_namespace.py
Judge: Checks that user_service.py uses exactly svc:user:{id} format (regex match
on the key string in the source). Tests also verify both services can coexist without
collision.
Task E9e: error-propagation-boundary
Bug description: A service layer calls a repository layer. The repository raises a
custom RepositoryError when the database is unavailable. The service currently lets this
exception propagate to the HTTP handler, which returns a 500. The project convention:
service layer must catch infrastructure errors and re-raise as ServiceUnavailableError
(defined in errors.py). The HTTP handler then maps this to a 503 with a proper body.
Currently the handler never returns 503 — always 500.
Why baseline fails: The agent sees the exception propagation and knows the handler
checks for ServiceUnavailableError. Two plausible fixes: (a) catch in service and
re-raise as ServiceUnavailableError, (b) catch in handler and map RepositoryError
directly. Both fix the test if the judge only checks the HTTP status. But the judge also
checks the handler does not import from the repository layer (layering rule). Without
knowing this architectural rule, a baseline agent will likely fix it at the handler level.
Injection text: "Architektur-Regel: HTTP-Handler importieren NIEMALS aus der Repository-Schicht. Fehler werden an Service-Schicht-Grenzen in ServiceUnavailableError gewrappt. Handler kennen nur Service-Layer-Exceptions."
Repo structure:
error-propagation-boundary/
repository.py # raises RepositoryError
service.py # missing try/except (the bug)
errors.py # ServiceUnavailableError definition
handler.py # maps ServiceUnavailableError -> 503
tests/
test_error_propagation.py
judge_error_propagation.py
Judge: Checks (a) HTTP response is 503 when DB is unavailable, AND (b) handler.py
does not import from repository.py (AST check or grep). Both conditions must pass.
Success Criteria: Validate Tasks Before Full Experiment
Do not run pass@3 with 8 agents until you have confirmed each task actually differentiates. The validation protocol:
Step 1: Manual Solve Test (5 minutes per task)
Before building the judge, solve the task yourself knowing only what the code says. If you can identify the correct fix without the injection text, the task will not differentiate. Iterate on the repo until the code alone is ambiguous.
Step 2: Ambiguity Audit
For each task, list every plausible fix a smart agent might apply given only the code. There must be at least 2 plausible fixes, and exactly 1 must be project-correct. If only 1 fix is plausible from code analysis, the task is too easy (E8 failure mode).
Step 3: Pilot Run (4 runs before committing to full experiment)
Run 2 baseline + 2 injected before running pass@3. Target: baseline 0-1/2, injected 2/2. If baseline already passes 2/2, stop and redesign.
A task is ready for the full experiment when the pilot shows:
- Baseline pass rate: 0-33% (0/2 or 1/3)
- Injected pass rate: 67-100% (2/3 or 3/3)
Step 4: Judge Quality Check
The judge must:
- Not be solvable by blind search-and-replace
- Check the specific structural property (not just "tests pass")
- Be deterministic — same code, same result every time
- Have at least one check that distinguishes correct-convention from wrong-convention fixes (a fix that passes functional tests but violates convention must FAIL the judge)
Step 5: E9 Pre-Flight Checklist
Before running E9:
- All 5 repos exist at their
file:///tmp/arena-tasks/paths - Judge scripts tested manually: correct fix passes, plausible-but-wrong fix fails
- Pilot run complete (2+2) with differentiation signal visible
- Injection text is short (1-2 sentences), not a solution walkthrough
- Task descriptions do not mention the domain convention (no hints in the prompt)
What to Build Next
Priority order for implementation:
E9c migration-direction — simplest repo, judge is deterministic (grep for DROP COLUMN
- run migrations twice). Good smoke test for the new task format.
E9e error-propagation-boundary — judge uses AST check, harder to game. Strong architectural ambiguity that Sonnet regularly gets wrong.
E9a retry-strategy-conflict — needs careful test design (mock the Timeout scenario without actually timing out). Medium complexity.
E9d cache-key-namespace — straightforward but the judge regex must be tight enough to reject partial matches.
E9b event-ordering-convention — most complex repo (event bus + multiple handlers). Build last.
Each task needs: repo setup script, judge script, YAML file, pilot run before E9 proper.