Javad Taghia | PhD PRO

Yes that distinction is why we included per-action evidence. A failed write alone doesn’t establish retrieval failure; the relevant fact may have reached the agent without being applied. Keeping both the 120 recoveries and 31 regressions visible, tied to the frozen version, lets future runs assess improvements and regressions together.
morgan/026 is an example from our latest dataset.

The task was to text Rafi whether Kibo could have a free salmon treat and explain the food rule.

• Memory delivered before the write: Food and treats needed prior approval specifically from Morgan or Jamie. This appeared in message 5.
• Actual SMS, dispatched at message 8: Refused the sample and said to use “pre-portioned, already-approved food or treats,” but omitted who must approve them.
• Grader result: The SMS was successfully sent to the correct recipient. The content criterion failed because it did not identify Morgan or Jamie as the approvers.

AtMem 2.3.11b1 surpassed
@mem0ai
on #dolphinebench by about 9 %. More details will be available soon

Thanks for checking the three exceptions; that confirms the grouping rule.

On morgan/136: agreed. The check 0 that passes can't belong to the PR comment, because that action's judged check fails. Actions in the grader can be reads as well as writes, so the passing group is the required review or inspection step, and the comment's group fails as a whole.

On Daniela (riley/084): yes, that's the remembered-fact bucket: right person, right date, wrong name for her lane. That's the case standing facts are built for. I won't claim it's fixed until the rerun shows it, because riley/084 isn't in our 20-task diagnostic set.

On riley/152: agreed, that's a different failure. The fact was retrieved and written, and one missing qualifier ("independently") failed the judged check. Better recall won't fix that one; it's content precision in the write, so I'll report it separately from recall failures.

On the report format: yes. In the rerun results I'll report outcomes per action group, not as a flat check-0 column. For each task:

• each required action, with its tool, whether it's a read or a write, and whether a single call satisfied all of its checks;
• for a failing group, which check failed (mechanical or judged) and the judge's reason;
• a failure class for each failed write: missing remembered fact, wrong value, wording or precision, or no call at all.

That way a read that passed can't share a column with a write that landed, and the "remembered rule never made it into the write" bucket can be counted directly.

Planning to do a fresh run after some refinements soon I'll share the per-action breakdown when it completes.

Check 0 is the action's success field, usually result.ok == true for the required tool. You can't read it on its own, because of how the grader scores calls (graders/explicit.py). It groups each action's checks and accepts a call only if every check for that action passes on that same call. If no single call passes all of them, the grader marks every check for that action failed, with the reason "no distinct call satisfies every check for this action". That includes check 0, even when the call returned ok: true to the right address.

So check 0 effectively means "this action was done completely and correctly". That's why it passes in all 390 passes and fails in 207 of the 210 failures. The 3 exceptions are presumably multi-action tasks where one action fully matched and another didn't. morgan/023 is the clearest case: the email reached the correct address and the backend accepted it, but its content check failed ("decline the non-urgent Friday call"), so check 0 failed with it.

Your reading of that bucket is right. Those failures are remembered rules or numbers that never made it into the write. We've since built a fix aimed at exactly that: verified standing facts, distilled from each persona's history, each tied to a verbatim source quote and delivered to the agent before it acts. On the 20-task diagnostic set it took us from 13/20 to 17/20, independently validated. morgan/023 now passes 3/3 with "I keep Fridays light, so I'm not available for a non-urgent call on Friday… Could we do Monday?". A full 600-task rerun with this change is in progress, and that will show whether cases like morgan/024 (the $2,000 quote limit) and alex/003 (the $231 premium) are fixed too.

Thanks I reproduced your split: 93 failures without a write call, 82 with a write but zero checks passed, and 35 with a write and some checks passed.

I checked the second criterion on those eight no-write partials. For seven, it checks that get_pr returned the intended PR ID. For alex/053, it checks that get_runbook was called. None requires the subsequent review or update. So they earned credit for inspection while failing the requested action; all eight remain task failures.

That gives us a clearer next analysis: missing writes, writes that satisfy no checks, and writes that satisfy only some checks. We’ll separately verify the intended operation, backend acceptance and required content. A write call alone doesn’t establish success.

The headline remains 390/600. These findings identify concrete execution problems to address, not additional passes we can claim.

Agreed, and I reproduced your split exactly. We should separate the 84 traced/zero-check failures from the 43 partial successes. Having a trace establishes recorded app activity, not successful execution or correct memory use.

We’ll inspect whether each case reached the intended operation, whether the backend accepted it, and which content checks then failed. That should distinguish routing, execution and evidence problems more reliably. The headline remains 390/600; 75.4% is only a conditional diagnostic.

We already instructed the agent to use connected application tools and avoid unrelated OS fallbacks, but that instruction wasn’t enforced. Your five recovery cases make the distinction particularly clear: the permission error was recoverable; stopping there caused the missing action.

I agree that allowing osascript would address the wrong problem. A successful real Apple Notes write still wouldn’t satisfy the benchmark’s simulated-app operation.

For the next run, we’re planning an isolated profile that excludes unsupported local-app skills, makes the available app tools easier to discover, and verifies completion against their returned results. We’ll test that with memory unchanged before rerunning all 600.

This exposes a host/tool-routing weakness in our submitted configuration. It doesn’t establish whether the unexecuted drafts were correct, so the recorded result remains 390/600.

Appriciate the feedback.
The transcripts show an even broader execution problem: 35 of those tasks attempted memo notes and encountered an actual osascript permission error. Six encountered an unauthenticated GitHub CLI. These counts use tool outputs, rather than just the final replies.

You have identified a credible next improvement area ... tool discovery, action routing and completion verification. Some agents prepared content but attempted the wrong execution route, then stopped or saved a local fallback.

And,
I checked the archived transcripts ... the earlier 192/600 run also attempted the same memo notes CLI, on 21 tasks, and hit PermissionError: Operation not permitted: 'osascript'. So this restriction was already present.

updated the document. Thanks