TODO_E8_REDESIGN.md

# TODO: E8 Redesign — Tasks That Actually Require Domain Knowledge

Date: 2026-03-19
Author: Analysis based on E8 results (24/24 passes, 0 differentiation)

---

## Problem: Ceiling Effect in E8

E8 ran 4 domain-knowledge scenarios (enum-convention, timeout-unit, breaking-change, implicit-null)
with baseline vs injected agents, 3 runs each (pass@3), 24 total runs.

Result: **12/12 baseline passed. 12/12 injected passed.**

Zero differentiation. The experiment cannot answer its own hypothesis because both agents
behave identically. There is no signal.

---

## Root Cause: Bugs Were Readable From the Code Alone

Each E8 bug had one fatal design flaw: **the correct fix was discoverable by reading the
existing codebase**. The agent did not need injected knowledge — it could infer the answer
from what was already in the repo.

Concrete failure modes per scenario:

**E8a enum-convention** ("use UPPERCASE enums")
- The bug is a wrong string literal. Any agent that reads the schema or existing enum values
  finds UPPERCASE immediately. The injection text says exactly what the code already says.
- Baseline solves it by reading `schema.sql` or other model files. Injection adds nothing.

**E8b timeout-unit** ("timeouts are in milliseconds")
- The variable is named `timeout_ms`. The name itself encodes the unit.
- A baseline agent reads the variable name, infers milliseconds, converts 30 to 30000.
  The injection text repeats what the variable name already implies.

**E8c breaking-change** ("find_one() is now get_by_id()")
- The error message is `AttributeError: 'Database' object has no attribute 'find_one'`.
  The stack trace points directly to the broken call. The agent reads `database.py`,
  finds `get_by_id`, and substitutes. No tribal knowledge needed — it is a grep task.

**E8d implicit-null** ("email has NOT NULL constraint")
- The schema file is in the repo. Reading `schema.sql` reveals `NOT NULL` on email.
  The injection text describes what the file already shows.

**The pattern**: Every E8 bug had its answer in the repository. Injection text described
the repository. Baseline read the repository. Same result.

---

## Design Principles for Differentiating Domain-Knowledge Tasks

A task truly requires domain knowledge when the correct fix **cannot be derived from the
code alone**. This means one or more of the following must be true:

### 1. The Convention Exists Outside the Repo

The rule is not documented in any file the agent can read. It lives in team memory,
architectural decisions made before the current codebase existed, or external systems
the agent cannot access (CI config on a remote server, a Confluence page, a Slack decision).

Example: "We always use snake_case for RabbitMQ routing keys" — nothing in the code enforces
this, there is no linter rule, but mixing conventions breaks queue binding silently.

### 2. Multiple Plausible Fixes Exist — Only One Is Correct For This Project

The code analysis narrows it to 2-3 valid approaches. Without context, the agent picks one
arbitrarily. With injection, it picks the project-correct one.

Example: A retry function fails. Fix options: (a) increase max_retries, (b) add exponential
backoff, (c) catch a broader exception class. All three "fix" the immediate test. Only one
matches the team's chosen retry strategy.

### 3. The Bug Is Caused By an Implicit Contract With an External System

The external system's behavior is not captured anywhere in the repo. The agent sees the
Python code but not the API spec, the database behavior under load, or the message broker
version.

Example: A third-party payment API silently truncates decimal amounts to 2 places, but only
in production. The code passes 3 decimal places. No test in the repo captures this.

### 4. The Fix Requires Knowing What Not To Change

The task has a correct minimal fix and several incorrect expansive fixes. Baseline agents
tend to make the expansive fix (refactor, add validation, extract helper). The project
convention is minimal patches to stable modules. Only an agent with that context makes
the right narrow change.

### 5. There Are Two Bugs — The Injected Context Identifies Which One Is The Real One

The symptom has two potential causes: a logic bug in module A and a config bug in module B.
Both would produce the same error. Baseline agents fix one or the other. The injection
text identifies which one is the known issue.

---

## 5 Concrete New Task Ideas

Each task below is designed so that:
- Baseline agent: reads all files, finds a plausible fix, applies it, tests pass or fail ambiguously
- Injected agent: applies the project-correct fix because the injection resolves the ambiguity

---

### Task E9a: `retry-strategy-conflict`

**Bug description**: A background job processor retries failed HTTP requests. It currently
retries on any exception. This causes duplicate charges when the payment API returns a 200
with a timeout on the client side (the charge went through but the response never arrived).
The team's rule: only retry on connection errors (requests.ConnectionError), never on
timeouts (requests.Timeout).

**Why baseline fails**: The code has an except block catching `Exception`. A baseline agent
sees this and might fix it by catching `requests.exceptions.RequestException` (broader, still
wrong), or by adding a retry count (doesn't fix the duplicate charge problem), or by
catching `(ConnectionError, Timeout)` (wrong — includes Timeout). Three plausible fixes,
one is correct for this project.

**Injection text**:
"In diesem Projekt werden HTTP-Retries NUR bei requests.ConnectionError ausgeloest —
requests.Timeout wird NICHT retried (Charge kann bereits erfolgt sein). Siehe DECISIONS.md."

**DECISIONS.md content** (not in repo, only known via injection): Does not exist as a file.
The rule is pure tribal knowledge.

**Repo structure**:
```
retry-strategy-conflict/
  processor.py          # background job, broken retry logic
  payment_client.py     # thin HTTP wrapper
  tests/
    test_processor.py   # tests retry on ConnectionError, also tests NO retry on Timeout
  judge_retry_strategy.py
```

**Judge**: Checks that the implementation retries on `ConnectionError` but raises on
`Timeout` without retrying. The test cannot be "cheated" by catching all exceptions.

**Differentiation mechanism**: Baseline catches either too broadly or too narrowly.
The injection provides the exact project rule that resolves the ambiguity.

---

### Task E9b: `event-ordering-convention`

**Bug description**: An event sourcing system publishes domain events. Two event handlers
are registered for `UserRegistered`: one sends a welcome email, one creates a billing
record. The billing record creation is failing intermittently because the email handler
sometimes mutates shared state before billing runs. The project convention: event handlers
must be idempotent and stateless — shared state mutation is banned. The fix is to remove
the state mutation from the email handler, not to add locking.

**Why baseline fails**: The agent sees an intermittent failure, shared state, and two
handlers. Plausible fixes: add a lock around the handlers, change handler order, remove
the shared state, or make the email handler async. Code analysis cannot determine which
approach the project uses — all four would fix the test.

**Injection text**:
"Event-Handler in diesem System MUESSEN stateless und idempotent sein — kein Shared-State
zwischen Handlern. Locks sind kein akzeptierter Fix. Mutation entfernen, nicht serialisieren."

**Repo structure**:
```
event-ordering-convention/
  event_bus.py          # simple sync event dispatcher
  handlers/
    email_handler.py    # mutates _processed_users set (the bug)
    billing_handler.py  # reads _processed_users, fails if wrong
  tests/
    test_event_ordering.py
  judge_event_ordering.py
```

**Judge**: Checks that the email handler contains no references to shared mutable state,
AND that both handlers pass the idempotency test (called twice with same event produces
same result).

---

### Task E9c: `migration-direction`

**Bug description**: A database migration script fails on CI. The script adds a new column
`archived_at` to the `posts` table. It fails because the CI database already has this
column (added by a hotfix two weeks ago). The correct fix in this project: mark the
migration as non-destructive and add a guard (`IF NOT EXISTS` or equivalent). The wrong
fix: delete the column and re-add it (destroys production data), or skip the migration
(leaves migration history inconsistent).

**Why baseline fails**: The agent sees a migration failure and an existing column. Options:
(a) add `IF NOT EXISTS`, (b) drop and recreate, (c) skip migration, (d) rename to a new
column. The code gives no hint which approach the project uses. The migration framework
(custom, not alembic) has no documented conventions in the repo.

**Injection text**:
"Dieses Projekt verwendet additive Migrationen — niemals Spalten droppen oder umbenennen
in Produktions-Migrationen. Bei bereits existierenden Spalten: IF NOT EXISTS Guard
hinzufuegen. Kein Skip, kein Recreate."

**Repo structure**:
```
migration-direction/
  migrations/
    001_create_posts.sql
    002_add_archived_at.sql    # the broken migration
  db.py                        # custom migration runner
  tests/
    test_migrations.py         # runs migrations on a fresh SQLite DB + a DB with the column pre-added
  judge_migration_direction.py
```

**Judge**: Runs migration against (a) a fresh DB and (b) a DB that already has `archived_at`.
Both must succeed. Also checks that the migration does not contain DROP COLUMN.

---

### Task E9d: `cache-key-namespace`

**Bug description**: A caching layer uses Redis. Two services share the same Redis instance.
The `UserService` caches user objects with key `user:{id}`. The `SessionService` also stores
data with key `user:{id}` (different data structure). Reads from one service corrupt the
other. The fix: `UserService` keys must be namespaced as `svc:user:{id}`. This namespace
convention exists across all services but is not documented in the repo — it was decided
before the current code was written.

**Why baseline fails**: The agent sees a key collision. Plausible fixes: add a prefix to
UserService (`u:{id}`, `users:{id}`, `user_svc:{id}`), or add a prefix to SessionService,
or use a separate Redis DB number, or use a hash. Without knowing the convention, the
agent picks one of these. Any prefix "fixes" the collision test — but only `svc:{service}:{id}`
matches the project convention checked by the judge.

**Injection text**:
"Redis-Keys in diesem System folgen dem Schema `svc:{service_name}:{id}`. UserService
verwendet `svc:user:{id}`, SessionService `svc:session:{id}`. Kein anderes Format."

**Repo structure**:
```
cache-key-namespace/
  user_service.py       # caches with key "user:{id}" (wrong)
  session_service.py    # caches with key "user:{id}" (different data)
  cache.py              # thin Redis wrapper
  tests/
    test_cache_isolation.py
  judge_cache_namespace.py
```

**Judge**: Checks that `user_service.py` uses exactly `svc:user:{id}` format (regex match
on the key string in the source). Tests also verify both services can coexist without
collision.

---

### Task E9e: `error-propagation-boundary`

**Bug description**: A service layer calls a repository layer. The repository raises a
custom `RepositoryError` when the database is unavailable. The service currently lets this
exception propagate to the HTTP handler, which returns a 500. The project convention:
service layer must catch infrastructure errors and re-raise as `ServiceUnavailableError`
(defined in `errors.py`). The HTTP handler then maps this to a 503 with a proper body.
Currently the handler never returns 503 — always 500.

**Why baseline fails**: The agent sees the exception propagation and knows the handler
checks for `ServiceUnavailableError`. Two plausible fixes: (a) catch in service and
re-raise as `ServiceUnavailableError`, (b) catch in handler and map `RepositoryError`
directly. Both fix the test if the judge only checks the HTTP status. But the judge also
checks the handler does not import from the repository layer (layering rule). Without
knowing this architectural rule, a baseline agent will likely fix it at the handler level.

**Injection text**:
"Architektur-Regel: HTTP-Handler importieren NIEMALS aus der Repository-Schicht. Fehler
werden an Service-Schicht-Grenzen in ServiceUnavailableError gewrappt. Handler kennen
nur Service-Layer-Exceptions."

**Repo structure**:
```
error-propagation-boundary/
  repository.py         # raises RepositoryError
  service.py            # missing try/except (the bug)
  errors.py             # ServiceUnavailableError definition
  handler.py            # maps ServiceUnavailableError -> 503
  tests/
    test_error_propagation.py
  judge_error_propagation.py
```

**Judge**: Checks (a) HTTP response is 503 when DB is unavailable, AND (b) `handler.py`
does not import from `repository.py` (AST check or grep). Both conditions must pass.

---

## Success Criteria: Validate Tasks Before Full Experiment

Do not run pass@3 with 8 agents until you have confirmed each task actually differentiates.
The validation protocol:

### Step 1: Manual Solve Test (5 minutes per task)

Before building the judge, solve the task yourself knowing only what the code says.
If you can identify the correct fix without the injection text, the task will not differentiate.
Iterate on the repo until the code alone is ambiguous.

### Step 2: Ambiguity Audit

For each task, list every plausible fix a smart agent might apply given only the code.
There must be at least 2 plausible fixes, and exactly 1 must be project-correct.
If only 1 fix is plausible from code analysis, the task is too easy (E8 failure mode).

### Step 3: Pilot Run (4 runs before committing to full experiment)

Run 2 baseline + 2 injected before running pass@3.
Target: baseline 0-1/2, injected 2/2. If baseline already passes 2/2, stop and redesign.

A task is ready for the full experiment when the pilot shows:
- Baseline pass rate: 0-33% (0/2 or 1/3)
- Injected pass rate: 67-100% (2/3 or 3/3)

### Step 4: Judge Quality Check

The judge must:
- Not be solvable by blind search-and-replace
- Check the specific structural property (not just "tests pass")
- Be deterministic — same code, same result every time
- Have at least one check that distinguishes correct-convention from wrong-convention fixes
  (a fix that passes functional tests but violates convention must FAIL the judge)

### Step 5: E9 Pre-Flight Checklist

Before running E9:
- [ ] All 5 repos exist at their `file:///tmp/arena-tasks/` paths
- [ ] Judge scripts tested manually: correct fix passes, plausible-but-wrong fix fails
- [ ] Pilot run complete (2+2) with differentiation signal visible
- [ ] Injection text is short (1-2 sentences), not a solution walkthrough
- [ ] Task descriptions do not mention the domain convention (no hints in the prompt)

---

## What to Build Next

Priority order for implementation:

1. **E9c migration-direction** — simplest repo, judge is deterministic (grep for DROP COLUMN
   + run migrations twice). Good smoke test for the new task format.

2. **E9e error-propagation-boundary** — judge uses AST check, harder to game. Strong
   architectural ambiguity that Sonnet regularly gets wrong.

3. **E9a retry-strategy-conflict** — needs careful test design (mock the Timeout scenario
   without actually timing out). Medium complexity.

4. **E9d cache-key-namespace** — straightforward but the judge regex must be tight enough
   to reject partial matches.

5. **E9b event-ordering-convention** — most complex repo (event bus + multiple handlers).
   Build last.

Each task needs: repo setup script, judge script, YAML file, pilot run before E9 proper.