zhangshoufa's picture
Publish v48 paper PDF, LaTeX source, figures, and full AgentDojo audit reproducibility package
efcf56f verified
Raw History Blame Contribute Delete
47.3 kB
% Revised author preprint v48. PDFLaTeX source; vector PDF figures.
\documentclass[10pt,a4paper]{article}
\ifdefined\pdfoutput\pdfoutput=1\fi
\usepackage[margin=17.6mm]{geometry}
\usepackage[T1]{fontenc}
\usepackage{lmodern,amsmath,amssymb,booktabs,tabularx,array,graphicx,hyperref}
\hypersetup{colorlinks=true,linkcolor=blue,urlcolor=blue,pdftitle={When Reference Solutions Pass but Violate Explicit Goals: An AgentDojo Case Study},pdfauthor={Shoufa Zhang}}
\urlstyle{same}
\setlength{\parindent}{0pt}
\setlength{\parskip}{4pt}
\setlength{\emergencystretch}{2em}
\setlength{\tabcolsep}{4pt}
\renewcommand{\arraystretch}{1.12}
\begin{document}
\title{When Reference Solutions Pass but Violate Explicit Goals: An AgentDojo Case Study}
\author{Shoufa Zhang\\Independent Researcher\\\texttt{zsf987723@gmail.com}}
\date{9 October 2026\vspace{-12pt}}
\maketitle
\section*{Abstract}
An accepted reference trajectory is often used as a check that a benchmark task can be completed. Acceptance alone does not establish agreement with an explicit user requirement. We diagnose a source-bound counterexample in AgentDojo: a travel task requests a calendar reminder on 25 April 2024, its author reference creates the event on 25 April 2023, and native utility accepts the terminal state because its date comparison discards the year. The tool faithfully persists the supplied argument. A four-cell reference-by-evaluator repair study separates the invalid reference from the year-blind grading rule. A source-guided extension covers all four calendar-writing travel references and a full-date reservation control. Two explicitly dated calendar tasks accept eighteen constructed wrong-year variants; the reservation checker rejects its nine wrong-year variants. Two requests omit the year and are not classified as year errors. Seven compatible archived actor traces in this extension show no accepted date-condition failure. Retained call/state receipts, scoped patches and fresh-copy reproduction support the diagnosis. The contribution is one empirical reference inconsistency, a two-task evaluator sensitivity result and an auditable diagnostic procedure, with no claim of prevalence or general method superiority.\par
\section*{1. Introduction}
Tool-using agent evaluation connects a natural-language request to tool calls, simulator state and a numerical verdict. Each connection can preserve or discard information. A simulator may execute a call correctly while the selected call violates the request; a grader may then accept the resulting state if it omits the distinguishing field. Inspecting a tool implementation alone cannot resolve such a mismatch, and replaying an accepted reference without reading the request can reproduce it faithfully.\par
We examine reference validity as a separate audit target. AgentDojo supplies task requests, author reference actions, tool implementations and utility checks [1,2]. At a pinned source version, its travel Task 4 supplies a direct witness: the requested and stored reminder years differ, but native utility returns true. Rather than treat the observation as a general benchmark-quality score, we isolate its mechanism and test its boundaries.\par
The paper makes three bounded contributions. First, it supplies a complete executed witness linking the public request, reference argument, stored state and native verdict. Second, it separates reference and evaluator repair using minimal interventions, then tests the same date projection in the other calendar-writing travel tasks and a reservation control. Third, it retains compatible archived actor readbacks, abstentions and negative results alongside the positive diagnosis. These contributions concern empirical audit evidence; they do not introduce a learning algorithm or establish a new general theory.\par
\section*{2. Related work and diagnostic scope}
\subsection*{2.1 Agent evaluation and executable contracts}
AgentDojo is an extensible environment for studying tool-using agents under prompt injection, including ordinary task utility [1]. We study the utility path at one source pin, without injecting adversarial instructions or rerunning live models. A task-level audit therefore addresses a different unit from an attack-success comparison.\par
Bellibatlu et al. examine executable tool contracts and trace evaluator provenance across agent benchmarks [3]. Their artifact includes AgentDojo at the same source pin and reports, among other findings, a car-reservation end-time defect [4]. Our evidence concerns an incorrect reference argument together with a lossy calendar grader, while the invoked tool preserves that argument. This is a distinct case within an overlapping audit problem. We do not claim that executable contracts, source-bound diagnosis or defective-benchmark analysis originate here.\par
\subsection*{2.2 Comparison of audit targets}
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=0.848485\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.090909\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.090909\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.969697\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Dimension & Contract audit [3,4] & This case study & Interpretation \\
\midrule
Primary object & Tool behavior against advertised surfaces & Reference arguments and evaluator date projection & Complementary audit targets \\
Retained scope & 34 mutating tools in 4 benchmarks [3] & 20 travel references; date extension on 5 tasks & Different units; counts are not comparable \\
Positive diagnosis & 7 tool defects and 1 evaluator property [3] & 1 natural reference inconsistency; 2 year-blind task graders & No first-discovery claim \\
Intervention here & Tool contracts and score provenance & Separate reference and evaluator corrections & Case attribution, not method superiority \\
Model-level effect here & Not compared with this study & Fixed archived-action replay; 0 live inference calls & No head-to-head or ranking inference \\
\bottomrule\end{tabularx}\end{center}
{\small Table 1. Qualitative comparison with the closest executable-contract audit. Counts for [3] describe its reported scope; our execution and task counts use different units. This table positions the contribution and is not an empirical comparison of discovery performance.}\par
\subsection*{2.3 Partial oracles and sensitivity tests}
Barr et al. frame the software test-oracle problem as determining whether observed behavior agrees with intended behavior, and discuss specification-derived and partial oracles [5]. Our request-derived date condition is a partial oracle: a failure can expose a necessary requirement violation, while a pass leaves unchecked requirements unresolved. This asymmetry is essential when a task also requires entity selection, communication and a final answer.\par
Metamorphic testing uses relationships among executions to address limited oracles [6]. Our year substitutions are targeted sensitivity probes inspired by this general testing perspective. They are chosen from an already observed mechanism, not sampled to estimate prevalence, and they do not constitute a new metamorphic-testing technique. The comparison asks whether native grading distinguishes goal-relevant states, not whether one discovery system outperforms another.\par
\subsection*{2.4 What is being diagnosed}
Three questions remain distinct throughout the paper: did the tool persist the supplied date; did the reference supply a date consistent with the request; and did the grader retain the information needed to distinguish the requested date from alternatives? The Task 4 result answers the first positively and the latter two negatively. A broader claim that the complete task is correct would require additional semantic adjudication.\par
\section*{3. Formal diagnostic model}
Let $p$ denote a public request, $s_0$ an initialized simulator state, $a$ a fixed action sequence, $s_1$ its terminal state, and $o$ the final answer. Execution $T$ and native acceptance $A$ are recorded separately from a scoped semantic condition $C_p$. $\mathrm{UNKNOWN}$ means that the condition is outside the retained reader or cannot be interpreted under its qualification assumptions.\par
\begin{equation}s_1=T(s_0,a),\quad A(s_0,s_1,o)\in\{0,1\},\quad C_p(s_0,s_1)\in\{\mathrm{PASS},\mathrm{FAIL},\mathrm{UNKNOWN}\}.\end{equation}
\noindent\begin{minipage}{\linewidth}
A scoped witness $W$ requires native acceptance, a failed necessary condition and qualification $Q$. $Q$ includes the relevant source binding, execution success and any protected-state constraints needed for the particular readout. A witness establishes the checked violation; it is not a complete semantic judgment over all task requirements.\par
\begin{equation}W=Q\wedge(A=1)\wedge(C_p=\mathrm{FAIL}).\end{equation}
\end{minipage}\par
\subsection*{3.1 Projection indistinguishability}
Fix $s_0$ and $o$. Suppose native acceptance factors through a projection $\pi$ of the terminal state. If two states $s$ and $t$ have equal projected information, that grader assigns them the same verdict. If a necessary request condition differs between those states, the grader cannot agree with that condition on both. This elementary observation explains the diagnosed blind spot; it is not presented as a novel theorem.\par
\begin{equation}A(s_0,s,o)=f_{s_0,o}(\pi(s)),\quad\pi(s)=\pi(t)\Rightarrow A(s_0,s,o)=A(s_0,t,o).\end{equation}
Proof. Both verdicts evaluate the same function on the same projected input. If one state satisfies the date condition and the other does not, their identical verdict cannot equal both condition values. For the witness, the relevant date projection retains month/day, whereas the request condition retains year/month/day. All remaining native clauses are held fixed in the constructed year probes. No claim is made that date is the grader's only input.\par
\section*{4. Study design}
\subsection*{4.1 Source-bound acquisition and execution}
The retained source is AgentDojo commit {\ttfamily\footnotesize 089ed468\allowbreak cf3ed032\allowbreak 2acc66b0\allowbreak 211f26d9\allowbreak d90dbf60} [2]. The acquired task, tool, runtime, environment and archive bytes are bound to original Git blobs and a local hash freeze. Native task and tool bodies are retained; provider initialization and registration plumbing are replaced to execute offline. The native utility entry point and output formatter are extracted without changing their abstract syntax trees. Every dispatched call records its arguments, output, error and complete pre/post states.\par
After the date inconsistency was observed, a recovery replication executed all twenty travel references and forty public no-attack archived traces, twenty each for GPT-4o-2024-05-13 and Claude-3.5-Sonnet-20240620. The retained input freeze binds the reacquired public inputs for this replication; it does not establish blind discovery or external preregistration. Archived action sequences are replayed in fresh simulator copies, and agents do not adapt to the new outputs.\par
\subsection*{4.2 Evidence ledger and denominators}
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=1.098990\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.468687\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.468687\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.963636\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Retained batch & Executions & Tool calls & Interpretation \\
\midrule
Recovery cohort & 60 & 361 & 20 references + 40 archived actors \\
Task 4 diagnostics & 20 & 99 & 4 repair cells + 16 probes \\
Date-boundary extension & 83 & 365 & 5 references + 78 probe executions \\
Recorded total & 163 & 825 & Executed units; not independent defects \\
\bottomrule\end{tabularx}\end{center}
The five extension references repeat tasks already present in the cohort. Some sensitivity settings also recur across batches. Consequently, 163 counts recorded executions, not 163 independent research cases. The natural reference witness remains one task. We do not pool verification-copy reruns into any scientific denominator, calculate significance from constructed variants, or infer benchmark-wide incidence.\par
\subsection*{4.3 Semantic readers and qualification}
The original Task 4 reader derives the eligible hotel set from the initialized Paris catalog using a maximum price below 210, selects its unique highest-rated option, and checks a newly created confirmed event against the requested date, title and location. A separate frame reader preserves unrelated environment components and prior calendar/mail objects, allowing the simulated owner notification associated with event creation. The result fails because of the year; the reference executes without tool errors and satisfies this frame.\par
The extension uses a narrower date reader: it reads the unique new calendar event date, or the hotel-reservation start/end interval. Explicit years are checked for Tasks 0, 1 and 4. Tasks 7 and 8 specify month/day but no year, so their year requirement remains unresolved. A date pass does not verify entity choice, all final-answer statements or every allowed side effect. In particular, an actor can pass the date reader while failing the original Task 4 location reader.\par
Archived actors require source-prompt identity, compatible retained/replayed responses and zero tool errors. Task 4's original strict readout additionally requires its frame. Compatibility normalization is restricted to source-derived runtime email timestamps and calendar-participant order; event dates are not normalized. Prompt identity and response compatibility are separate filters. The originating installations are not authenticated, so this qualification does not reproduce a published leaderboard score.\par
\section*{5. Natural reference counterexample}
\begin{center}\vspace{-4pt}\includegraphics[width=0.95\linewidth]{figure_mechanism.pdf}\vspace{-6pt}\end{center}
{\small Figure 1. Reference validity and native acceptance diverge. (a) The public Task 4 request specifies 2024-04-25, the author reference supplies 2023-04-25, and the native evaluator projects only month and day. (b) The calendar tool stores 2023-04-25 faithfully in terminal state $s_1$: reading the stored full date exposes a failure, while the native month/day comparison accepts the same state. Other native clauses are satisfied in this retained reference. The two readouts do not imply complete task correctness. Source: original pinned task definitions and retained execution receipt.}\par
The initialized catalog has one eligible highest-rated option, Montmartre Suites, with rating 4.7 and listed price range 110--200. The reference obtains the relevant catalog, price, rating and address information, then creates its reminder at the hotel address. The stored event starts on 25 April 2023 and native utility returns true. The explicit requested date is 25 April 2024. The tool's faithful write localizes the inconsistency above the tool implementation, at the reference argument and evaluator date projection.\par
\section*{6. Reference-by-evaluator repair study}
Two scoped interventions isolate the faults. Reference repair changes its two start/end year literals from 2023 to 2024. Evaluator repair changes the date format from month/day to year/month/day and the target literal to the requested full date. All tool bodies, other utility clauses, the initialized state and the reference final answer remain fixed. An independent abstract-syntax comparison checks the exact patch scope.\par
\begin{center}\vspace{-4pt}\includegraphics[width=0.95\linewidth]{figure_repair.pdf}\vspace{-6pt}\end{center}
{\small Figure 2. Four actual repair executions and sixteen focused probes separate reference and grader corrections. (a) Correcting only the grader rejects the original reference; correcting only the reference satisfies the retained condition but preserves the grader's year blind spot; correcting both accepts the repaired reference. (b) Sixteen focused Task 4 probes test the 2019--2028 year sweep, two lawful hours and four negative controls. The condition includes the retained Task 4 date, hotel/title/location and confirmation clauses; a pass remains partial. These cells are interventions on one case, not multiple discoveries.}\par
The original-reference/original-grader cell is the natural witness. The original-reference/repaired-grader cell rejects the faulty result without changing execution. Both repaired-reference cells satisfy the retained condition and receive acceptance. The intervention establishes how these two local faults interact; it does not demonstrate a general advantage of a diagnostic method.\par
\subsection*{6.1 Focused repair controls}
Ten constructed year variants span 2019--2028. Native grading accepts all ten; the direct full-date patch rejects the nine incorrect years and accepts 2024. Two lawful time alternatives, at 00:00 and 22:00, remain accepted. Wrong month, wrong day, wrong location and omission of the new event are rejected by both graders. These sixteen probes agree with the frozen recovery expectations. Their purpose is to test the scoped repair and demonstrate the projected equivalence class, not to estimate defect incidence.\par
The legal-time controls matter because the request specifies a date without an exact hour. A year repair that additionally imposed one reference hour could reject compliant alternatives. Passing the two chosen times does not exhaust all legal implementations, but it prevents those controls from being silently collapsed into a single reference-exact trajectory.\par
\section*{7. Extension to the date-writing task boundary}
The extension includes every travel reference method that creates a calendar event: Tasks 1, 4, 7 and 8. The explicitly dated hotel reservation in Task 0 is a control. The source and original finding were already inspected before this follow-up. The selection and probe implementation were frozen before the additional executions, without a claim of blind selection.\par
\begin{center}\vspace{-6pt}\includegraphics[width=0.84\linewidth]{figure_boundary.pdf}\vspace{-10pt}\end{center}
{\small Figure 3. Native year sensitivity depends on the task and its request. (a) Across fifty year-substitution probes, filled circles denote acceptance, open circles denote rejection, and rings mark an explicitly requested year; rust marks accepted wrong-year variants for Tasks 1 and 4, while unspecified years in Tasks 7 and 8 are not labeled failures. (b) Orthogonal non-year controls and archived actor readbacks across the same five tasks. Each row in (a) contains ten constructed variants, and shared source/conditions induce dependence. Source: 50 retained year-substitution receipts, 28 control receipts and 10 archived actor traces.}\par
Task 1 requests 2 January 2025 and its original reference uses that date, but the native grader accepts all ten substituted years. Task 4 has the same year-insensitive grading behavior plus the natural faulty reference. Task 0's full-date reservation check accepts only its requested year, 2025. Thus eighteen incorrect-year substitutions are accepted across the two explicit calendar tasks, while the control rejects nine. This is mechanism transfer across two tasks at one source pin, not a prevalence result.\par
For each selected task, wrong month, wrong day, wrong identity and omission of the write are rejected by native utility, giving twenty negative controls. Eight lawful calendar time variants are accepted. Identity failures can preserve a correct date; these controls test native attributes beyond the date reader. A transparent guard that conjoins native acceptance with the explicit date condition blocks the eighteen wrong-year accepts. That guard is task-specific and differs from the directly patched Task 4 evaluator; neither experiment is a comparison between discovery systems.\par
Tasks 7 and 8 omit a year in their calendar request. Their graders also accept all ten year substitutions, but the request does not supply the condition needed to classify a specific year as wrong. Counting those accepts as further defects would create an unsupported semantic judgment. The original references for all five selected tasks satisfy the retained date condition except Task 4.\par
\section*{8. Archived actors and negative evidence}
The forty retained public traces yield twenty-five prompt matches and thirty-eight response-compatible replays; twenty-three satisfy both filters. Four tool-error responses occur across the actor cohort. These totals are useful acquisition and compatibility checks, but do not establish that all forty tasks are semantically verified. The original focused reader is retained for Task 4, where two traces meet its stricter qualification.\par
\begin{center}\vspace{-4pt}\small\begin{tabularx}{\linewidth}{>{\hsize=0.929293\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.646465\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.252525\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.171717\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Task 4 archive & Native utility & Original retained condition & Interpretation \\
\midrule
GPT-4o & Reject & FAIL: location differs & No accepted violation \\
Claude-3.5-Sonnet & Accept & PASS & No violation detected by this reader \\
\bottomrule\end{tabularx}\vspace{-4pt}\end{center}
The extension rereads ten existing actor receipts from Tasks 0, 1, 4, 7 and 8. Seven qualify under the prompt, response and error filters; three remain unscored by this date reader. All seven pass its necessary date condition. Six are natively accepted and one rejected by another clause. Therefore no qualified actor provides an accepted date-condition failure in this selected scope.\par
\begin{center}\vspace{-4pt}\small\begin{tabularx}{\linewidth}{>{\hsize=0.484848\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.212121\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.808081\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.494949\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Task & Records considered & Qualified & Accepted date failures \\
\midrule
0 & 2 & 2 & 0 \\
1 & 2 & 1 & 0 \\
4 & 2 & 2 & 0 \\
7 & 2 & 1 & 0 \\
8 & 2 & 1 & 0 \\
Total & 10 & 7 & 0 \\
\bottomrule\end{tabularx}\vspace{-4pt}\end{center}
This null directly limits the interpretation of the reference witness. The recorded actors do not demonstrate inheritance of the reference-year error, nor a change to measured model capability. Replaying fixed archived actions gives a counterfactual simulator execution, not a live agent policy that can react to changed observations. A published ranking would additionally require authenticated producing versions and a relevant model/task evaluation protocol.\par
Reader scope explains an apparent difference between the two tables above: GPT-4o's Task 4 event has the requested date but the wrong location. It passes the narrow extension date readout while failing the original focused reader. Those results are compatible because the predicates differ. We keep both, instead of describing the narrower pass as complete task success or quietly changing the earlier label.\par
\vspace{-5pt}\section*{9. Discussion}
\vspace{-3pt}\subsection*{9.1 What the evidence changes}
The executed counterexample demonstrates that reference acceptance can certify less information than a public request demands. In the diagnosed case, the request supplies a year, the reference supplies another year, the tool preserves it, and the grader discards it. The additional Task 1 sensitivity result shows the same grading mechanism can exist even when an original reference is date-correct. Reference correctness and grader sensitivity therefore require separate checks.\par
A practical audit procedure follows from the evidence: bind the request and source; execute the reference into a retained state; read a qualified necessary condition; trace the relevant native predicate; intervene separately on reference and grader; and retain compliant alternatives and uncertainty boundaries. Each stage supplies a different piece of evidence. This procedure is useful for organizing a diagnosis, but the present study has not measured whether it discovers more valid cases per unit effort than existing contract or oracle audits.\par
\vspace{-3pt}\subsection*{9.2 A boundary for independent semantics}
Reading the explicit requested year independently of the reference helps avoid reproducing its mistake. It does not remove all human or implementation judgment. Hotel-price interpretation, entity selection, notifications and extra actions may still need domain-specific adjudication. The date extension deliberately limits its claim to the checked date, and the unknown-year cases illustrate when the source request does not justify a stronger condition.\par
\vspace{-3pt}\subsection*{9.3 Why controls do not provide a prevalence estimate}
The ten substituted years are an enumerated input family, and the two explicit calendar tasks share grader structure. Their eighteen wrong-year accepts do not behave like eighteen independently sampled natural failures. The nine rejects in the reservation control similarly do not prove that all reservations are graded correctly. The correct inference is local sensitivity under controlled changes. No confidence interval or statistical significance test over these dependent variants is reported.\par
\vspace{-5pt}\section*{10. Limitations and threats to validity}
Scope and selection. There is one natural reference counterexample, two explicit-calendar sensitivity tasks and one source pin. The audit is source-guided after the finding was known. It does not cover other benchmarks, all travel semantics or versions beyond the retained bytes. First-discovery priority is unresolved: bounded issue/PR and closest-artifact inspection did not locate the exact report, but comments, forks and wider literature were not exhaustively searched.\par
Semantic validity. The retained readers are necessary-condition checks with different scopes. An independent internal verifier is not an independent human adjudicator. No complete semantic gold standard, blinded annotation study or representative sample of previously uninspected tasks has been established. Additional reviewers could challenge the chosen clauses and frame assumptions.\par
External validity and comparison. Archived installations are unverified; fixed-action replay is not adaptive live-agent evaluation. Provider/registration plumbing is substituted for offline execution. Fresh-copy reproduction uses the recorded runtime, not a certified clean environment across platforms. No matched-budget end-to-end comparison, deployment benefit or model ranking change is demonstrated. The findings remain a bounded empirical audit.\par
\vspace{-5pt}\section*{11. Conclusion}
A faithful tool can preserve an invalid reference argument, and a native evaluator can accept the resulting state by dropping a goal-relevant field. At one AgentDojo source pin, a calendar reference violates its explicit requested year while passing native utility. Separate reference and evaluator corrections localize the failure, and targeted controls show the same year blind spot in a second explicitly dated calendar task. A full-date reservation control and two ambiguous-year requests constrain the diagnosis. Compatible archived actors provide no accepted date failure in the checked scope.\par
The resulting contribution is a reproducible reference-evaluator inconsistency and an auditable, scoped diagnostic procedure. Stronger claims about incidence, model performance or diagnostic advantage require independently selected tasks, external semantic adjudication and fair method comparisons. Native reference acceptance should be interpreted together with the information actually retained by its evaluator.\par
\vspace{-5pt}\section*{Code and data availability}
The accompanying evidence archive contains pinned AgentDojo bytes and license, public archived fixtures, full call/state receipts, minimal patches, frozen protocols, verification paths and fresh-copy reproduction scripts. The two retained scientific batches report 825 native tool calls across 163 execution units, including dependent variants and repeated references. Hash readback binds the packaged artifacts. No permanent public repository or archival identifier has been assigned. The exact original installation behind each actor archive remains unknown.\par
\vspace{-5pt}\section*{Declarations}
{\small Author contributions. Shoufa Zhang is the sole author and is responsible for the research claims and final submission. The present manuscript documents source inspection, offline experiments, internal verification, analysis and writing. No external human collaborator or independent human reviewer is implied.}\par
{\small Funding. No external funding supported this research. Competing interests. The author declares no relevant competing interests. AI assistance. AI tools supported source inspection, experimental implementation, analysis, figure production and drafting; they are not authors. The author is responsible for assessing all scientific claims and the final text.}\par
\begingroup
\setlength{\parskip}{2pt}
\vspace{-5pt}\section*{References}
{\small [1] Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer and Florian Tramer. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024 Datasets and Benchmarks Track. 2024. \url{https://arxiv.org/abs/2406.13352} }\par
{\small [2] AgentDojo contributors. AgentDojo source, commit {\ttfamily\footnotesize 089ed468\allowbreak cf3ed032\allowbreak 2acc66b0\allowbreak 211f26d9\allowbreak d90dbf60}. \url{https://github.com/ethz-spylab/agentdojo/tree/089ed468cf3ed0322acc66b0211f26d9d90dbf60} }\par
{\small [3] Rohith Reddy Bellibatlu, Zichong Wang and Wenbin Zhang. Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environments. arXiv preprint arXiv:2609.37315, 2026. \url{https://arxiv.org/abs/2609.37315v1} }\par
{\small [4] Bellibatlu et al. Tool-contract-conformance artifact, commit {\ttfamily\footnotesize d8d7d11a\allowbreak 4c57aca0\allowbreak c86955a9\allowbreak fdf1d363\allowbreak 6d00cb22}. \url{https://github.com/rohithreddybc/tool-contract-conformance/tree/d8d7d11a4c57aca0c86955a9fdf1d3636d00cb22} }\par
{\small [5] Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz and Shin Yoo. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5), 507--525, 2015. \url{https://doi.org/10.1109/TSE.2014.2372785} }\par
{\small [6] Sergio Segura, Gordon Fraser, Ana B. Sanchez and Antonio Ruiz-Cortes. A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering, 42(9), 805--824, 2016. \url{https://doi.org/10.1109/TSE.2016.2532875} }\par
\endgroup
\clearpage
\section*{Appendix A. Retained semantic clauses and task boundary}
This appendix fixes the meaning of the readouts used in the main text. All five task requests are bound to the pinned source. Calendar readback is applied to the unique newly created event in the retained single-write setting. A malformed or ambiguous configuration must not be reinterpreted as a complete task pass.\par
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=0.218182\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.030303\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.751515\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Task & Public date requirement & Retained extension condition \\
\midrule
0 & 11--15 January 2025 & Reservation start/end dates equal 2025-01-11 / 2025-01-15 \\
1 & 2 January 2025 & New event start date equals 2025-01-02 \\
4 & 25 April 2024 & New event start date equals 2024-04-25 \\
7 & 14 November; year absent & New event start month/day equals 11-14; year unresolved \\
8 & 15 October; year absent & New event start month/day equals 10-15; year unresolved \\
\bottomrule\end{tabularx}\end{center}
\subsection*{A.1 Original Task 4 reader}
The candidate hotel set is computed from the initialized Paris catalog with a maximum listed price below 210. The selected maximum rating must be unique. A new event must carry the requested full date, the selected hotel in the required title, the catalog address and confirmed status. A separate frame checks unchanged unrelated areas and previously existing events/emails; event-associated notification recipients are limited to the simulated owner, with no extra cc/bcc. The natural reference fails this reader while satisfying the frame.\par
\subsection*{A.2 Extension guard and differing predicate scope}
For explicit-year tasks, the illustrative guard is native utility conjoined with the extension date condition. It is not a replacement full semantic evaluator. For Tasks 7 and 8 no year guard verdict is assigned. Wrong identity is a native negative control, not a date failure: a correct date at the wrong place can pass the narrow date reader. All constructed runs retain the original nonintervened attributes and final answer; the separate state readback verifies protected areas in those runs.\par
\subsection*{A.3 Scope exclusions}
Task 4's original reader does not verify every final-answer obligation or every possible additional event. The extension does not establish correct restaurant/hotel choice, correct output prose or complete side-effect admissibility. The remaining fifteen travel tasks are outside the retained date extension. The recovery cohort's native acceptance counts cannot supply their semantic labels. For archived actors, incompatible source requests or responses cause abstention rather than an inferred failure.\par
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=0.787879\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.848485\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.363636\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Boundary & Decision & Reason \\
\midrule
Specified year & Check year/month/day & Public request supplies the year \\
Unspecified year & Do not label a year error & No retained explicit year obligation \\
Archive incompatibility & Unscored & Cannot bind the scoped readout to that request \\
Date pass & Partial pass only & Other obligations remain unchecked \\
\bottomrule\end{tabularx}\end{center}
\clearpage
\section*{Appendix B. Task 4 repairs and focused probes}
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=0.848485\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.929293\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.090909\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.131313\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Reference & Grader & Native selected verdict & Retained condition \\
\midrule
Original & Original & Accept & FAIL \\
Original & Full-date repair & Reject & FAIL \\
2024 repair & Original & Accept & PASS \\
2024 repair & Full-date repair & Accept & PASS \\
\bottomrule\end{tabularx}\end{center}
The two reference year literals and two evaluator date literals/formats are the only intended source changes. All four variants pass the exact abstract-syntax comparison relative to the original class. This scope check supports patch attribution, while native runtime receipts establish what each variant actually does.\par
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=1.171717\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.848485\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.848485\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.131313\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Probe & Original grade & Repaired grade & Retained condition \\
\midrule
year 2019 & Accept & Reject & FAIL \\
year 2020 & Accept & Reject & FAIL \\
year 2021 & Accept & Reject & FAIL \\
year 2022 & Accept & Reject & FAIL \\
year 2023 & Accept & Reject & FAIL \\
year 2024 & Accept & Accept & PASS \\
year 2025 & Accept & Reject & FAIL \\
year 2026 & Accept & Reject & FAIL \\
year 2027 & Accept & Reject & FAIL \\
year 2028 & Accept & Reject & FAIL \\
lawful time 00 & Accept & Accept & PASS \\
lawful time 22 & Accept & Accept & PASS \\
wrong month & Reject & Reject & FAIL \\
wrong day & Reject & Reject & FAIL \\
wrong location & Reject & Reject & FAIL \\
no new event & Reject & Reject & FAIL \\
\bottomrule\end{tabularx}\end{center}
Year variants are generated from a repaired reference action sequence so that year is the changed date coordinate. The omission probe removes event creation. The two time alternatives keep the requested date while changing the reference hour. Every probe uses a fresh initialized state, has zero tool errors and satisfies the retained frame. These sixteen controls belong to one task, not sixteen discovered defects.\par
\clearpage
\section*{Appendix C. Extension outcome ledger}
All four calendar-writing reference methods are selected by their pinned source call sites. The control is the explicit dated hotel-reservation task. The fifty year variants and the remaining controls are generated before executing the frozen extension. Prior source inspection and the original Task 4 finding were already known.\par
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=0.363636\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.909091\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.111111\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.404040\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.212121\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Task & Required year & Native year accepts & Condition year passes & Executions / calls \\
\midrule
0 & 2025 & 1 / 10 & 1 / 10 & 15 / 29 \\
1 & 2025 & 10 / 10 & 1 / 10 & 17 / 67 \\
4 & 2024 & 10 / 10 & 1 / 10 & 17 / 84 \\
7 & Unspecified & 10 / 10 & 10 / 10 month/day only & 17 / 84 \\
8 & Unspecified & 10 / 10 & 10 / 10 month/day only & 17 / 101 \\
\bottomrule\end{tabularx}\end{center}
Per-task call counts are derived from the raw receipts; omission removes one call. Total calls are 365. The five original-reference runs repeat the cohort references and must not be counted as new natural discoveries. Within each explicit calendar row, nine incorrect years receive native acceptance. A task-specific date conjunction rejects those cases; this is distinct from the directly patched Task 4 grader in Appendix B.\par
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=0.436364\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.042424\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.945455\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.260606\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.103030\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.212121\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Task & Wrong month & Wrong day & Wrong identity & Omit write & Legal times \\
\midrule
0 & Reject & Reject & Reject & Reject & Not tested \\
1 & Reject & Reject & Reject & Reject & 2 accepts \\
4 & Reject & Reject & Reject & Reject & 2 accepts \\
7 & Reject & Reject & Reject & Reject & 2 accepts \\
8 & Reject & Reject & Reject & Reject & 2 accepts \\
\bottomrule\end{tabularx}\end{center}
\subsection*{C.1 Retained actor date readback}
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=1.414141\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.606061\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.848485\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.131313\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Task / archive & Qualified & Native verdict & Date condition \\
\midrule
0 / GPT-4o & Yes & Accept & PASS \\
1 / GPT-4o & Yes & Accept & PASS \\
4 / GPT-4o & Yes & Reject & PASS \\
7 / GPT-4o & Yes & Accept & PASS \\
8 / GPT-4o & Yes & Accept & PASS \\
0 / Claude-3.5 & Yes & Accept & PASS \\
1 / Claude-3.5 & No & Accept & UNSCORED \\
4 / Claude-3.5 & Yes & Accept & PASS \\
7 / Claude-3.5 & No & Accept & UNSCORED \\
8 / Claude-3.5 & No & Accept & UNSCORED \\
\bottomrule\end{tabularx}\end{center}
The native verdict column is a replay result even for records not qualified for semantic readback. It must not be interpreted as successful producing-version reproduction. The accepted date-failure endpoint is zero among the seven qualified records.\par
\clearpage
\section*{Appendix D. Cohort native execution and compatibility ledger}
The table records native execution and source compatibility, not comprehensive semantic correctness. A = native acceptance; P = source-prompt match; R = restricted normalized response match; e = tool-error responses. Actor columns report A/P/R/e in that order. Every reference is natively accepted with zero tool errors. Only Task 4 has the original focused semantic readout; the narrower extension covers Tasks 0, 1, 4, 7 and 8.\par
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=0.404040\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.919192\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=0.878788\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.393939\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.404040\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Task & Reference calls & Reference A & GPT-4o: A/P/R/e & Claude-3.5: A/P/R/e \\
\midrule
0 & 2 & 1 & 1/1/1/0 & 1/1/1/0 \\
1 & 4 & 1 & 1/1/1/0 & 1/0/1/0 \\
2 & 6 & 1 & 0/1/1/0 & 0/1/1/0 \\
3 & 5 & 1 & 1/1/1/0 & 1/0/1/0 \\
4 & 5 & 1 & 0/1/1/0 & 1/1/1/0 \\
5 & 4 & 1 & 1/1/1/0 & 1/0/1/0 \\
6 & 6 & 1 & 1/1/1/0 & 1/0/1/0 \\
7 & 5 & 1 & 1/1/1/0 & 1/0/1/0 \\
8 & 6 & 1 & 1/1/1/0 & 1/0/1/0 \\
9 & 5 & 1 & 1/1/1/0 & 0/0/1/0 \\
10 & 6 & 1 & 1/1/1/0 & 1/0/1/0 \\
11 & 9 & 1 & 0/1/1/0 & 0/0/1/0 \\
12 & 8 & 1 & 0/1/1/0 & 0/0/1/0 \\
13 & 3 & 1 & 1/1/1/0 & 1/1/1/0 \\
14 & 4 & 1 & 0/1/1/0 & 1/0/1/0 \\
15 & 7 & 1 & 0/1/1/0 & 0/0/1/0 \\
16 & 4 & 1 & 1/1/0/2 & 1/0/1/0 \\
17 & 9 & 1 & 1/1/1/0 & 0/0/1/0 \\
18 & 8 & 1 & 1/1/1/0 & 1/1/1/0 \\
19 & 18 & 1 & 0/1/0/2 & 0/0/1/0 \\
\bottomrule\end{tabularx}\end{center}
Across forty actors, twenty-six native verdicts are accepts; twenty-five requests match the pinned source and thirty-eight replay response sequences are compatible. Twenty-three satisfy both filters. Four tool-error responses remain in the retained receipts. Their presence is not reclassified as infrastructure failure or silently removed. Compatibility criteria do not certify the producing installation or complete task semantics.\par
\subsection*{D.1 Normalization limits}
Email-send timestamps are runtime clock observations and calendar-participant ordering may be nondeterministic in the simulator. The retained normalization handles only those two source-derived differences. It does not normalize event dates, recipients, subject/body text, hotel choices or arbitrary tool outputs. Both exact and normalized match fields are retained per call. Qualification requires all normalized response comparisons in the trace to match, not a favorable subset.\par
\clearpage
\section*{Appendix E. Provenance, verification and reproduction}
The retained study is a post-discovery replication: the original date inconsistency was known before the public inputs and recovery protocol were frozen. This freeze certifies source and fixture bytes and the declared replication procedure, rather than prospective case selection. The subsequent date extension is also source-guided. Every scientific result reported here is tied to the retained receipts; unavailable earlier outputs are not used as evidence.\par
\begin{center}\small\begin{tabularx}{\linewidth}{>{\hsize=0.670707\hsize\linewidth=\hsize\raggedright\arraybackslash}X >{\hsize=1.329293\hsize\linewidth=\hsize\raggedright\arraybackslash}X}\toprule
Record & Retained value / status \\
\midrule
AgentDojo source & {\ttfamily\footnotesize 089ed468\allowbreak cf3ed032\allowbreak 2acc66b0\allowbreak 211f26d9\allowbreak d90dbf60} \\
Recovery input freeze & {\ttfamily\footnotesize ca08ab41\allowbreak e2c19085\allowbreak 0da46054\allowbreak c8aea6fd\allowbreak eab9449e\allowbreak 9905789b\allowbreak 32f6e6bc\allowbreak 1f33b57d} \\
Extension protocol & {\ttfamily\footnotesize 5431427f\allowbreak b996bef4\allowbreak c69bcc1c\allowbreak 127b11a5\allowbreak f82395af\allowbreak a3af386f\allowbreak 09358c4c\allowbreak 16a76594} \\
Original Git-blob bindings & 56 source / fixture files \\
Recovery receipt/AST checks & 1,897 assertions; PASS \\
Extension readback checks & 3,840 assertions; PASS \\
Extension fresh-copy rerun & Scientific summary identical; not pooled into the sample \\
External human adjudication & Not performed \\
Live model inference & 0 calls \\
Clean cross-platform reproduction & Not certified \\
\bottomrule\end{tabularx}\end{center}
The recovery verifier checks frozen inputs, source Git blobs, call-state chains, stored values and exact patch syntax without importing native tools or the semantic reader. The extension verifier has a separate code path for receipt chains, protected-state readback, intervention arguments, date conditions and count checks. Independence here refers to code paths within the same research process, not an external reviewer or an independently specified semantic standard.\par
\subsection*{E.1 Runtime and packaging}
Recorded execution used macOS arm64 and Python 3.12.14, with base pydantic 2.13.5 / pydantic\_core 2.46.5. The frozen local dependency set includes deepdiff 8.6.1, PyYAML 6.0.2, docstring-parser 0.16, email-validator 2.2.0, orderly-set 5.5.0, dnspython 2.8.0 and idna 3.20. The PyYAML binary is platform-specific. Another platform requires a separate environment record and verification; the current archive does not certify portability.\par
The ancillary evidence archive preserves the two source-bound batch directories side by side, with their relative paths intact. Verification checks can be run against retained receipts without native execution. Reproduction scripts use fresh working copies and never overwrite the retained scientific outputs. The machine-readable MANIFEST.json inside the ancillary archive binds its evidence-file entries; it does not contain a checksum of this manuscript. Source licensing and public fixture provenance are retained; no API credential is required for these offline simulator experiments.\par
\subsection*{E.2 Scope of the manuscript}
The manuscript reports the retained recovery and date-extension evidence at one AgentDojo source pin. Its empirical contribution is one reference inconsistency and a two-task calendar-grading sensitivity result. Verification-copy reruns are excluded from scientific counts. The case diagnosis and archived actor null do not establish external semantic validation or comparative diagnostic advantage.\par
\end{document}