Ali Toygar Abak PRO
AI & ML interests
Recent Activity
Organizations
Thanks, Rhode. Delayed observations across a durable restart are not covered by the frozen release; your suggestion helped separate that question from delayed evidence delivery.
I've added a November 6 update to the article, with links to the experimental Draft PR and pinned reproduction evidence.
The follow-up uses captured events from three real local LangGraph runs and 14 offline delivery scenarios. It tests whether earlier observations can incorrectly settle a different run, control request or observation window. Historical observations remain available without overriding later evidence, and missing observations remain distinct from observed continued execution.
Durable restart/resume and independent observation are still outside the tested scope. The observer shares the executor's host and trust domain; this is offline delivery replay, not a live-network experiment or independently verified enforcement.
Your point about keeping conclusions tied to their supporting evidence and revision conditions is also a central concern of the AIREP work. Thanks again for the concrete test suggestion.
A successful ACK is not proof of execution, enforcement, or observed effect.
I’ve published a reproducible AIREP study examining this evidence gap.
Key findings:
- In a frozen synthetic holdout, a naive ACK-based rule made unsupported stronger claims in 16/18 applicable claim queries.
- A bounded evidence evaluator made 0/18 on the same subset.
- Six controlled LangGraph scenarios demonstrate how control delivery, execution, and observed effects can be recorded separately.
These are controlled, author-side results—not proof of production safety or independent interoperability.
📄 Technical article:
https://huggingface.co/blog/phionyx/when-an-ai-agent-says-stop-what-evidence-proves-th
🔬 Reproducible study, code & evidence:
https://doi.org/10.5281/zenodo.23960852
A question for agent framework developers: How does your runtime distinguish "STOP acknowledged" from "execution actually stopped"?
Thanks — the UI regression angle is particularly useful. An interface could preserve the signed artifacts correctly and still mislead users by collapsing the evidence into a single green “stopped” badge.
A concrete regression test would require the presentation layer to keep the control receipt, execution evidence, and observed effect distinct. The ignored-pause and missing-observer cases must also remain distinguishable: observed continued execution and the absence of an observation are different results.
Agreed on the same-host observer limitation. That boundary should remain visible in the UI rather than disappear behind a successful verification indicator. The current scenarios demonstrate these distinctions in a controlled, first-party setup; they do not establish independently verified enforcement.
When an AI Agent Says STOP, What Evidence Proves That It Actually Stopped?
Thanks — I think this is a good distinction.
CP-T05 can still be satisfied here: the invalid/error state remains explicit and nothing is silently turned into a PASS.
The problem is one layer later. If the fill value and aggregation rule can improve the resulting score, then a fully disclosed transformation can still strengthen the aggregate claim.
My reading is that this falls mainly under R-CP-4 together with R-CP-6: the numerical transformation, metric direction, aggregation rule, and population semantics all matter. Calling a value a “penalty” does not by itself establish that it is non-improving over the metric’s actual domain.
So I would not treat this as a CP-T05 failure. It looks more like a useful counterexample for testing whether R-CP-4/R-CP-6 are strong enough in practice.
Thanks for raising it — this is exactly the kind of edge case I was hoping people would test against the draft.
The Dashboard Says PASS. What Exactly Passed?
A run finishes. A dashboard shows PASS. A score gets copied into another system. A few steps later, it may no longer be clear which metric produced it, what was skipped, what population was actually tested, or whether that PASS came from the evaluator at all.
That is the problem behind a new individual IETF Internet-Draft I published today: Claim-Preserving Exchange of AI Evaluation Evidence.
The basic idea is simple:
moving evidence should not make the claim stronger than the evidence itself.
Technical review and counterexamples are very welcome:
https://datatracker.ietf.org/doc/html/draft-abak-ai-evaluation-claim-preservation
The main change is a stricter evidence model for AI runtime governance:
Decision → Control → Execution → Effect
These are separate evidence stages — a decision does not prove delivery, delivery does not prove execution, and execution does not prove the intended effect.
v2 also defines bounded AIREP-Core / Authenticated / Witnessed assurance and structured reconciliation that preserves missing, unevaluated and indeterminate evidence instead of turning absence into success.
v0.2.0-beta.1 remains experimental. No same-version producer→consumer interoperability claim is made.
📄 Paper
AIREP: A Protocol for Per-Decision Evidence in AI Runtime Governance (2608.21363)
🔬 Runnable Evidence Lab
phionyx/airep-evidence-lab
💻 Code / spec / evidence
https://github.com/halvrenofviryel/ai-runtime-evidence-protocol
A tool-call denial proves a denial. It does not prove that a safeguard prevented an incident.
For AI evaluation, agent evaluation, and AI safety assurance, provenance alone is not enough. We also need to preserve the scope, context, and limits of the claim.
That is the case for a claim-preserving evidence contract.
📄 Access Is Not Yet Verifiability
https://huggingface.co/blog/phionyx/access-is-not-yet-verifiability