Publish verified AWS and Google runs with reproducible source

README.md

@@ -5,24 +5,35 @@ app_file: index.html
 license: mit
 short_description: Replay tool failures and inspect recovery evidence
 tags:
-- agent-reliability
-- mcp
-- evaluation
+  - agent-reliability
+  - mcp
+  - evaluation
 ---
 # Agent Reliability Lab · Cloud Lab Works
 
-Four controlled MCP reference-client experiments, with before/after policies, traces and checks.
-Built by Cloud Lab Works using MCP Failure Lab and the deterministic Agent Trace Clinic modules.
+Four controlled tool failures, before/after policy changes, and inspectable evidence. The app replays recorded runs; it does not start cloud jobs, call a model, or upload your comparison file.
 
-This Space replays a recorded **local Linux ARM64 container run**. It does not claim a live AWS or Google deployment. It makes no model calls and cannot launch a cloud job. Uploaded comparison JSON stays in the browser; it is never sent to us.
+## Try it
 
-- Transient failure: one attempt fails; three attempts recover after two injected errors.
-- Permanent failure: fewer attempts reduce retries; the downstream tool is not repaired.
-- Deadline: a 100 ms timeout is too short for a 400 ms injected delay; 2000 ms succeeds.
-- Malformed response: remains a terminal protocol error despite a larger retry budget.
+Choose a fixture. Inspect both traces. Download [Google Cloud Run evidence](google-suite.json) or [AWS Lambda evidence](aws-suite.json), then use **Compare a cloud run**. The app verifies the trace hashes and checks source version, policy, outcomes, and attempt counts.
 
-Clinic findings are heuristic leads, not the success verdict. Legitimate retries can look like a loop. Trace hashes identify bytes; they do not prove source authenticity. Runtime labels in uploaded JSON require a separate deployment receipt.
+| Fixture | Before → after | AWS / Google |
+|---|---|---|
+| Transient failure | tool error → success; 1 → 3 attempts | pass / pass |
+| Permanent failure | tool error remains; 5 → 2 attempts | pass / pass |
+| Deadline fit | timeout → success; 100 → 2000 ms deadline | pass / pass |
+| Malformed response | protocol error remains terminal after one call | pass / pass |
 
-The sample includes fixture version, source digest and measured attempt timings. Those timings are observations, not cloud-performance benchmarks. This tests a fixed official-SDK reference client, not autonomous-agent reasoning or MCP conformance certification.
+## Evidence and limits
 
-[Cloud Lab Works](https://cloudlabworks.dev) · [Public projects](https://github.com/cloudlabworks)
+One synthetic suite ran on AWS Lambda ARM64 and one on Google Cloud Run AMD64. Their four fixture pairs matched the local baseline; all 16 cloud trace hashes were checked after export. Operator readback confirmed the deployed code/configuration and successful execution. The latest test suite passed 12 tests on local ARM64 and Google Cloud Shell AMD64, with four additional mock tests for the AWS re-disable guard.
+
+These are fixed official-MCP-SDK **reference-client** experiments, not an evaluation of an LLM or production agent. Clinic findings are inspection leads, not the acceptance oracle. Legitimate retries may still trigger its loop detector. Timings are observations, not a cross-cloud performance ranking. A hash protects bytes, not truth or origin; raw account records and infrastructure state remain private.
+
+The displayed baseline is a recorded local Linux ARM64 run. Cloud Run is a private manual job with one task, no retries and a 60-second timeout. Lambda is disabled at rest, briefly enabled for one manual invocation, then disabled again. Neither has a public trigger or scheduler. Static hosting is independent of both cloud runtimes.
+
+## Source and reproduction
+
+[Download the source snapshot](source.tar.gz), extract it, and follow `lab/README.md`. Snapshot revision: `fe96775b907973bdb3e5d04f059cba0b12528ccc`. It includes fixed fixtures, pinned dependencies, cloud result files, tests and AWS/Google infrastructure configurations, but no git history, credentials, private state or account-specific variables. [File checksums](manifest.json).
+
+Built by [Cloud Lab Works](https://cloudlabworks.dev). MIT license. Deterministic Clinic modules retain their license and pinned provenance. Deployment lessons are in the source guide.

aws-suite.json

@@ -0,0 +1 @@
+{"schema_version":"clw-reliability-lab/1","fixture_version":"2026-10-14.1","generated_at":"2026-10-15T06:45:57.573Z","source_sha256":"e35d7973d8b581c1fe3a1c9c70923c8a089e59b05861e1cf2c2110a7c6a1ae01","runtime":{"node":"v24.21.0","os":"linux","arch":"arm64","platform":"aws-lambda","platform_evidence":"Runtime environment declaration; cloud deployment receipt is required to corroborate it."},"scope":"Synthetic official-SDK reference-client experiments. No language model, production agent, network fault injection, or cross-cloud performance ranking.","duration_ms":28658,"experiments":[{"id":"transient-recovery","title":"Give a transient failure a bounded recovery window","change":"One attempt → three attempts. The same tool fails twice, then succeeds.","tradeoff":"Extra attempts help this fixture, not permanent or non-idempotent failures.","before":{"schema_version":"1.0","scenario":"transient-error","client":"official-sdk-reference-client","connected":true,"outcome":"tool_error","policy":{"timeout_ms":2000,"max_attempts":1,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"tool_error","elapsed_ms":137}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Injected temporary tool failure","is_error":true}]},"trace_sha256":"14806ab6057f6c8ee28a864697ebe38a634032f184ac1abc3288a21a48e1ceaa","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2],"evidence":"lab_probe({\"message\":\"probe\"}) -> Injected temporary tool failure","explain":"lab_probe failed and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"after":{"schema_version":"1.0","scenario":"transient-error","client":"official-sdk-reference-client","connected":true,"outcome":"success","policy":{"timeout_ms":2000,"max_attempts":3,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"tool_error","elapsed_ms":159},{"attempt":2,"outcome":"tool_error","elapsed_ms":20},{"attempt":3,"outcome":"success","elapsed_ms":39}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-2","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-2","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-3","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-3","content":"Probe response: probe","is_error":false}]},"trace_sha256":"ce922a08104db02be5625a86a53afad9a5739c94b55b0d0c6da3ee3776b00fbc","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":3,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":4,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":5,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":6,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Probe response: probe","tokens_in":null,"tokens_out":null,"error":false}],"instructions":[],"findings":[{"id":"tool_loop","severity":"high","steps":[1,3,5],"evidence":"lab_probe({\"message\":\"probe\"}) called 3 times","explain":"The agent called lab_probe with identical arguments 3 times. Repeating the same call expects a different answer from the same input; the usual fix is to feed the result back into the plan, or to stop after a set number of identical calls."},{"id":"unhandled_tool_error","severity":"medium","steps":[2,4],"evidence":"lab_probe({\"message\":\"probe\"}) -> Injected temporary tool failure","explain":"lab_probe failed (2 times here) and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"expected":{"before":["tool_error",1],"after":["success",3]},"verification":{"pass":true,"checks":[{"name":"before outcome","pass":true,"expected":"tool_error","observed":"tool_error"},{"name":"before attempt count","pass":true,"expected":1,"observed":1},{"name":"before trace completeness","pass":true},{"name":"after outcome","pass":true,"expected":"success","observed":"success"},{"name":"after attempt count","pass":true,"expected":3,"observed":3},{"name":"after trace completeness","pass":true}]}},{"id":"permanent-failure","title":"Stop wasting retries on a permanently failing tool","change":"Five attempts → two attempts. The tool remains broken; the client stops earlier.","tradeoff":"This is bounded failure handling, not a repaired downstream service.","before":{"schema_version":"1.0","scenario":"tool-error","client":"official-sdk-reference-client","connected":true,"outcome":"tool_error","policy":{"timeout_ms":2000,"max_attempts":5,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"tool_error","elapsed_ms":65},{"attempt":2,"outcome":"tool_error","elapsed_ms":55},{"attempt":3,"outcome":"tool_error","elapsed_ms":39},{"attempt":4,"outcome":"tool_error","elapsed_ms":1},{"attempt":5,"outcome":"tool_error","elapsed_ms":59}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-2","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-2","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-3","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-3","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-4","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-4","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-5","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-5","content":"Injected temporary tool failure","is_error":true}]},"trace_sha256":"6fe85ae2dbc3d2d2a2ca15b4513e471174d53e4c5fc202f3ae69865140dfda2e","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":3,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":4,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":5,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":6,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":7,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":8,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":9,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":10,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"tool_loop","severity":"high","steps":[1,3,5,7,9],"evidence":"lab_probe({\"message\":\"probe\"}) called 5 times","explain":"The agent called lab_probe with identical arguments 5 times. Repeating the same call expects a different answer from the same input; the usual fix is to feed the result back into the plan, or to stop after a set number of identical calls."},{"id":"retry_storm","severity":"high","steps":[2,4,6,8,10],"evidence":"lab_probe failed 5 times; first error: Injected temporary tool failure","explain":"lab_probe returned an error 5 times in one task. Retries without backoff, a cap, or a change of input burn tokens and can hammer the service behind the tool."},{"id":"unhandled_tool_error","severity":"medium","steps":[2,4,6,8,10],"evidence":"lab_probe({\"message\":\"probe\"}) -> Injected temporary tool failure","explain":"lab_probe failed (5 times here) and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"after":{"schema_version":"1.0","scenario":"tool-error","client":"official-sdk-reference-client","connected":true,"outcome":"tool_error","policy":{"timeout_ms":2000,"max_attempts":2,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"tool_error","elapsed_ms":60},{"attempt":2,"outcome":"tool_error","elapsed_ms":20}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-2","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-2","content":"Injected temporary tool failure","is_error":true}]},"trace_sha256":"d08501886fa2fe3eb0d29916f39e6fbc6b1fb418b29cca0dfd2f5a71ede6e81a","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":3,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":4,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2,4],"evidence":"lab_probe({\"message\":\"probe\"}) -> Injected temporary tool failure","explain":"lab_probe failed (2 times here) and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"expected":{"before":["tool_error",5],"after":["tool_error",2]},"verification":{"pass":true,"checks":[{"name":"before outcome","pass":true,"expected":"tool_error","observed":"tool_error"},{"name":"before attempt count","pass":true,"expected":5,"observed":5},{"name":"before trace completeness","pass":true},{"name":"after outcome","pass":true,"expected":"tool_error","observed":"tool_error"},{"name":"after attempt count","pass":true,"expected":2,"observed":2},{"name":"after trace completeness","pass":true}]}},{"id":"deadline-fit","title":"Fit the deadline to the declared tool latency","change":"100 ms deadline → 2000 ms deadline for a tool with a 400 ms injected delay.","tradeoff":"The longer deadline increases the time spent waiting if the tool really hangs.","before":{"schema_version":"1.0","scenario":"delay","client":"official-sdk-reference-client","connected":true,"outcome":"timeout","policy":{"timeout_ms":100,"max_attempts":1,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"timeout","elapsed_ms":101,"error_code":"REQUEST_TIMEOUT"}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Error: reference client observed timeout","is_error":true}]},"trace_sha256":"ca8d0b945d71b0c1814ae234d73f877fdb46b3ebc3a440e95d6c2114ecafbe2a","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Error: reference client observed timeout","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2],"evidence":"lab_probe({\"message\":\"probe\"}) -> Error: reference client observed timeout","explain":"lab_probe failed and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"after":{"schema_version":"1.0","scenario":"delay","client":"official-sdk-reference-client","connected":true,"outcome":"success","policy":{"timeout_ms":2000,"max_attempts":1,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"success","elapsed_ms":446}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Probe response: probe","is_error":false}]},"trace_sha256":"45543eb60cd9bf030322aa827426f549c995b208427aa6168c0bc8c37390cbb9","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Probe response: probe","tokens_in":null,"tokens_out":null,"error":false}],"instructions":[],"findings":[],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"expected":{"before":["timeout",1],"after":["success",1]},"verification":{"pass":true,"checks":[{"name":"before outcome","pass":true,"expected":"timeout","observed":"timeout"},{"name":"before attempt count","pass":true,"expected":1,"observed":1},{"name":"before trace completeness","pass":true},{"name":"after outcome","pass":true,"expected":"success","observed":"success"},{"name":"after attempt count","pass":true,"expected":1,"observed":1},{"name":"after trace completeness","pass":true}]}},{"id":"malformed-control","title":"Keep protocol failures terminal","change":"One-attempt budget → five-attempt budget; protocol errors must still stop after one call.","tradeoff":"Increasing a retry budget must not cause malformed responses to be accepted.","before":{"schema_version":"1.0","scenario":"malformed","client":"official-sdk-reference-client","connected":true,"outcome":"protocol_error","policy":{"timeout_ms":2000,"max_attempts":1,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"protocol_error","elapsed_ms":83,"error_code":"INVALID_RESULT"}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Error: reference client observed protocol_error","is_error":true}]},"trace_sha256":"b01d8e246ec581b6d332074ed83420c64eddb36eaff010f3f056bf4dc52fec9d","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Error: reference client observed protocol_error","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2],"evidence":"lab_probe({\"message\":\"probe\"}) -> Error: reference client observed protocol_e...","explain":"lab_probe failed and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"after":{"schema_version":"1.0","scenario":"malformed","client":"official-sdk-reference-client","connected":true,"outcome":"protocol_error","policy":{"timeout_ms":2000,"max_attempts":5,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"protocol_error","elapsed_ms":60,"error_code":"INVALID_RESULT"}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Error: reference client observed protocol_error","is_error":true}]},"trace_sha256":"b01d8e246ec581b6d332074ed83420c64eddb36eaff010f3f056bf4dc52fec9d","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Error: reference client observed protocol_error","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2],"evidence":"lab_probe({\"message\":\"probe\"}) -> Error: reference client observed protocol_e...","explain":"lab_probe failed and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"expected":{"before":["protocol_error",1],"after":["protocol_error",1]},"verification":{"pass":true,"checks":[{"name":"before outcome","pass":true,"expected":"protocol_error","observed":"protocol_error"},{"name":"before attempt count","pass":true,"expected":1,"observed":1},{"name":"before trace completeness","pass":true},{"name":"after outcome","pass":true,"expected":"protocol_error","observed":"protocol_error"},{"name":"after attempt count","pass":true,"expected":1,"observed":1},{"name":"after trace completeness","pass":true}]}}],"pass":true}

google-suite.json

@@ -0,0 +1 @@
+{"schema_version":"clw-reliability-lab/1","fixture_version":"2026-10-14.1","generated_at":"2026-10-15T06:30:42.000Z","source_sha256":"e35d7973d8b581c1fe3a1c9c70923c8a089e59b05861e1cf2c2110a7c6a1ae01","runtime":{"node":"v24.22.0","os":"linux","arch":"x64","platform":"google-cloud-run-job","platform_evidence":"Runtime environment declaration; cloud deployment receipt is required to corroborate it."},"scope":"Synthetic official-SDK reference-client experiments. No language model, production agent, network fault injection, or cross-cloud performance ranking.","duration_ms":5517,"experiments":[{"id":"transient-recovery","title":"Give a transient failure a bounded recovery window","change":"One attempt → three attempts. The same tool fails twice, then succeeds.","tradeoff":"Extra attempts help this fixture, not permanent or non-idempotent failures.","before":{"schema_version":"1.0","scenario":"transient-error","client":"official-sdk-reference-client","connected":true,"outcome":"tool_error","policy":{"timeout_ms":2000,"max_attempts":1,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"tool_error","elapsed_ms":12}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Injected temporary tool failure","is_error":true}]},"trace_sha256":"14806ab6057f6c8ee28a864697ebe38a634032f184ac1abc3288a21a48e1ceaa","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2],"evidence":"lab_probe({\"message\":\"probe\"}) -> Injected temporary tool failure","explain":"lab_probe failed and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"after":{"schema_version":"1.0","scenario":"transient-error","client":"official-sdk-reference-client","connected":true,"outcome":"success","policy":{"timeout_ms":2000,"max_attempts":3,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"tool_error","elapsed_ms":10},{"attempt":2,"outcome":"tool_error","elapsed_ms":1},{"attempt":3,"outcome":"success","elapsed_ms":1}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-2","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-2","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-3","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-3","content":"Probe response: probe","is_error":false}]},"trace_sha256":"ce922a08104db02be5625a86a53afad9a5739c94b55b0d0c6da3ee3776b00fbc","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":3,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":4,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":5,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":6,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Probe response: probe","tokens_in":null,"tokens_out":null,"error":false}],"instructions":[],"findings":[{"id":"tool_loop","severity":"high","steps":[1,3,5],"evidence":"lab_probe({\"message\":\"probe\"}) called 3 times","explain":"The agent called lab_probe with identical arguments 3 times. Repeating the same call expects a different answer from the same input; the usual fix is to feed the result back into the plan, or to stop after a set number of identical calls."},{"id":"unhandled_tool_error","severity":"medium","steps":[2,4],"evidence":"lab_probe({\"message\":\"probe\"}) -> Injected temporary tool failure","explain":"lab_probe failed (2 times here) and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"expected":{"before":["tool_error",1],"after":["success",3]},"verification":{"pass":true,"checks":[{"name":"before outcome","pass":true,"expected":"tool_error","observed":"tool_error"},{"name":"before attempt count","pass":true,"expected":1,"observed":1},{"name":"before trace completeness","pass":true},{"name":"after outcome","pass":true,"expected":"success","observed":"success"},{"name":"after attempt count","pass":true,"expected":3,"observed":3},{"name":"after trace completeness","pass":true}]}},{"id":"permanent-failure","title":"Stop wasting retries on a permanently failing tool","change":"Five attempts → two attempts. The tool remains broken; the client stops earlier.","tradeoff":"This is bounded failure handling, not a repaired downstream service.","before":{"schema_version":"1.0","scenario":"tool-error","client":"official-sdk-reference-client","connected":true,"outcome":"tool_error","policy":{"timeout_ms":2000,"max_attempts":5,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"tool_error","elapsed_ms":9},{"attempt":2,"outcome":"tool_error","elapsed_ms":1},{"attempt":3,"outcome":"tool_error","elapsed_ms":1},{"attempt":4,"outcome":"tool_error","elapsed_ms":1},{"attempt":5,"outcome":"tool_error","elapsed_ms":1}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-2","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-2","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-3","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-3","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-4","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-4","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-5","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-5","content":"Injected temporary tool failure","is_error":true}]},"trace_sha256":"6fe85ae2dbc3d2d2a2ca15b4513e471174d53e4c5fc202f3ae69865140dfda2e","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":3,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":4,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":5,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":6,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":7,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":8,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":9,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":10,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"tool_loop","severity":"high","steps":[1,3,5,7,9],"evidence":"lab_probe({\"message\":\"probe\"}) called 5 times","explain":"The agent called lab_probe with identical arguments 5 times. Repeating the same call expects a different answer from the same input; the usual fix is to feed the result back into the plan, or to stop after a set number of identical calls."},{"id":"retry_storm","severity":"high","steps":[2,4,6,8,10],"evidence":"lab_probe failed 5 times; first error: Injected temporary tool failure","explain":"lab_probe returned an error 5 times in one task. Retries without backoff, a cap, or a change of input burn tokens and can hammer the service behind the tool."},{"id":"unhandled_tool_error","severity":"medium","steps":[2,4,6,8,10],"evidence":"lab_probe({\"message\":\"probe\"}) -> Injected temporary tool failure","explain":"lab_probe failed (5 times here) and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"after":{"schema_version":"1.0","scenario":"tool-error","client":"official-sdk-reference-client","connected":true,"outcome":"tool_error","policy":{"timeout_ms":2000,"max_attempts":2,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"tool_error","elapsed_ms":7},{"attempt":2,"outcome":"tool_error","elapsed_ms":1}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Injected temporary tool failure","is_error":true},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-2","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-2","content":"Injected temporary tool failure","is_error":true}]},"trace_sha256":"d08501886fa2fe3eb0d29916f39e6fbc6b1fb418b29cca0dfd2f5a71ede6e81a","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true},{"i":3,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":4,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Injected temporary tool failure","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2,4],"evidence":"lab_probe({\"message\":\"probe\"}) -> Injected temporary tool failure","explain":"lab_probe failed (2 times here) and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"expected":{"before":["tool_error",5],"after":["tool_error",2]},"verification":{"pass":true,"checks":[{"name":"before outcome","pass":true,"expected":"tool_error","observed":"tool_error"},{"name":"before attempt count","pass":true,"expected":5,"observed":5},{"name":"before trace completeness","pass":true},{"name":"after outcome","pass":true,"expected":"tool_error","observed":"tool_error"},{"name":"after attempt count","pass":true,"expected":2,"observed":2},{"name":"after trace completeness","pass":true}]}},{"id":"deadline-fit","title":"Fit the deadline to the declared tool latency","change":"100 ms deadline → 2000 ms deadline for a tool with a 400 ms injected delay.","tradeoff":"The longer deadline increases the time spent waiting if the tool really hangs.","before":{"schema_version":"1.0","scenario":"delay","client":"official-sdk-reference-client","connected":true,"outcome":"timeout","policy":{"timeout_ms":100,"max_attempts":1,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"timeout","elapsed_ms":101,"error_code":"REQUEST_TIMEOUT"}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Error: reference client observed timeout","is_error":true}]},"trace_sha256":"ca8d0b945d71b0c1814ae234d73f877fdb46b3ebc3a440e95d6c2114ecafbe2a","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Error: reference client observed timeout","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2],"evidence":"lab_probe({\"message\":\"probe\"}) -> Error: reference client observed timeout","explain":"lab_probe failed and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"after":{"schema_version":"1.0","scenario":"delay","client":"official-sdk-reference-client","connected":true,"outcome":"success","policy":{"timeout_ms":2000,"max_attempts":1,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"success","elapsed_ms":413}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Probe response: probe","is_error":false}]},"trace_sha256":"45543eb60cd9bf030322aa827426f549c995b208427aa6168c0bc8c37390cbb9","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Probe response: probe","tokens_in":null,"tokens_out":null,"error":false}],"instructions":[],"findings":[],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"expected":{"before":["timeout",1],"after":["success",1]},"verification":{"pass":true,"checks":[{"name":"before outcome","pass":true,"expected":"timeout","observed":"timeout"},{"name":"before attempt count","pass":true,"expected":1,"observed":1},{"name":"before trace completeness","pass":true},{"name":"after outcome","pass":true,"expected":"success","observed":"success"},{"name":"after attempt count","pass":true,"expected":1,"observed":1},{"name":"after trace completeness","pass":true}]}},{"id":"malformed-control","title":"Keep protocol failures terminal","change":"One-attempt budget → five-attempt budget; protocol errors must still stop after one call.","tradeoff":"Increasing a retry budget must not cause malformed responses to be accepted.","before":{"schema_version":"1.0","scenario":"malformed","client":"official-sdk-reference-client","connected":true,"outcome":"protocol_error","policy":{"timeout_ms":2000,"max_attempts":1,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"protocol_error","elapsed_ms":12,"error_code":"INVALID_RESULT"}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Error: reference client observed protocol_error","is_error":true}]},"trace_sha256":"b01d8e246ec581b6d332074ed83420c64eddb36eaff010f3f056bf4dc52fec9d","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Error: reference client observed protocol_error","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2],"evidence":"lab_probe({\"message\":\"probe\"}) -> Error: reference client observed protocol_e...","explain":"lab_probe failed and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"after":{"schema_version":"1.0","scenario":"malformed","client":"official-sdk-reference-client","connected":true,"outcome":"protocol_error","policy":{"timeout_ms":2000,"max_attempts":5,"retry":"tool_error_only"},"attempts":[{"attempt":1,"outcome":"protocol_error","elapsed_ms":7,"error_code":"INVALID_RESULT"}],"transport_errors":0,"trace":{"messages":[{"role":"user","content":"Synthetic reference-client probe. No language model participated."},{"role":"assistant","content":null,"tool_calls":[{"id":"probe-1","type":"function","function":{"name":"lab_probe","arguments":"{\"message\":\"probe\"}"}}]},{"role":"tool","tool_call_id":"probe-1","content":"Error: reference client observed protocol_error","is_error":true}]},"trace_sha256":"b01d8e246ec581b6d332074ed83420c64eddb36eaff010f3f056bf4dc52fec9d","clinic":{"schema_version":"1.0","format":"openai","container":"json","steps":[{"i":0,"role":"user","kind":"message","tool":null,"args":null,"text":"Synthetic reference-client probe. No language model participated.","tokens_in":null,"tokens_out":null,"error":false},{"i":1,"role":"assistant","kind":"tool_call","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"","tokens_in":null,"tokens_out":null,"error":false},{"i":2,"role":"tool","kind":"tool_result","tool":"lab_probe","args":"{\"message\":\"probe\"}","text":"Error: reference client observed protocol_error","tokens_in":null,"tokens_out":null,"error":true}],"instructions":[],"findings":[{"id":"unhandled_tool_error","severity":"medium","steps":[2],"evidence":"lab_probe({\"message\":\"probe\"}) -> Error: reference client observed protocol_e...","explain":"lab_probe failed and the next tool call was the identical call, or the trace ended on the error. Retrying unchanged input rarely changes the outcome; read the error, then change the arguments, switch tools, or stop and report it."}],"usage":{"per_task":{"tokens_in":0,"tokens_out":0,"total":0,"model_calls":0,"complete":false},"per_day":[{"tasks":10,"tokens_in":0,"tokens_out":0,"total":0},{"tasks":1000,"tokens_in":0,"tokens_out":0,"total":0}],"price":null,"cost_per_task":null}}},"expected":{"before":["protocol_error",1],"after":["protocol_error",1]},"verification":{"pass":true,"checks":[{"name":"before outcome","pass":true,"expected":"protocol_error","observed":"protocol_error"},{"name":"before attempt count","pass":true,"expected":1,"observed":1},{"name":"before trace completeness","pass":true},{"name":"after outcome","pass":true,"expected":"protocol_error","observed":"protocol_error"},{"name":"after attempt count","pass":true,"expected":1,"observed":1},{"name":"after trace completeness","pass":true}]}}],"pass":true}

index.html

@@ -1,6 +1,6 @@
 <!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Agent Reliability Lab · Cloud Lab Works</title><link rel="stylesheet" href="style.css"></head><body>
 <header><p class="eyebrow">CLOUD LAB WORKS / EXPERIMENT 01</p><h1>Watch it fail.<br>Check the recovery.</h1><p class="lead">Four controlled tool failures. One policy change at a time. Inspect what happened before believing the verdict.</p><div class="badge">Synthetic MCP reference-client runs · no language model</div></header>
 <main><section class="toolbar"><label>Experiment <select id="fixture"></select></label><button id="download">Download evidence</button><label class="upload">Compare a cloud run<input id="upload" type="file" accept=".json,application/json"></label></section>
-<p id="status" role="status">Loading recorded evidence…</p><section id="comparison" hidden></section><section id="experiment"></section>
+<p id="status" role="status">Loading recorded evidence…</p><p class="recorded-links">Verified cloud runs: <a href="https://huggingface.co/spaces/cloudlabworks/agent-reliability-lab/resolve/main/google-suite.json?download=true">Google JSON</a> · <a href="https://huggingface.co/spaces/cloudlabworks/agent-reliability-lab/resolve/main/aws-suite.json?download=true">AWS JSON</a> · <a href="https://huggingface.co/spaces/cloudlabworks/agent-reliability-lab/blob/main/README.md">Source and reproduction</a>. Download a run, then choose “Compare a cloud run”.</p><section id="comparison" hidden></section><section id="experiment"></section>
 <section class="limits"><h2>What this proves</h2><p>A fixed official-SDK client handled these injected faults according to the recorded policy. It does not prove a production agent is reliable, or that one cloud is faster.</p><p>Clinic findings are inspection leads. Legitimate retries can still trigger its loop detector. Success comes from the fixture's checks, not from a clean-looking report.</p><p>Uploaded comparison files stay in your browser. Runtime names in a file are declarations, not independent proof of deployment.</p></section></main>
 <footer><span>Cloud Lab Works · infrastructure designed to be run.</span><span id="provenance"></span></footer><script type="module" src="app.js"></script></body></html>

manifest.json

@@ -0,0 +1,11 @@
+{
+  "index.html": "dc006bc8a25795487bcf1556aa9abed14761a99767bef559a3727f7ee5f194e4",
+  "app.js": "907f32433815554085add34d96b7371e12a8de09ba1c7c26f9890afa94d712ba",
+  "style.css": "436ad50ca3bf5efef331868e1f0a001a76540a71a1234c05316fed230ec1783d",
+  "sample.json": "239822c79ae67bcc37c488a0acca7d7e72e3fd9500c270e8dc1b8c16aa108441",
+  "google-suite.json": "4f98296406bd2f4e4b51ea3b196b8a165724bac394f4a5ae3966560261ca5c2c",
+  "aws-suite.json": "ef5938c34c77fbeef690fc0e434c880070a3388fc56b540b895c6a022690e455",
+  "LICENSE": "755336ed08156c031190ce5f75b67bdbd1d3c90d1544b6b3cca1a2387bebe096",
+  "README.md": "16b08d198f5c0946ad58ec111440295348215fd6c6df116e04a0978d59e0db69",
+  "source.tar.gz": "871ed0c372ef54ad23cf84bd715e13c0aeaad2dbfa20b5ebcc80c229b2a13551"
+}

source.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:871ed0c372ef54ad23cf84bd715e13c0aeaad2dbfa20b5ebcc80c229b2a13551
+size 54023

style.css

@@ -1 +1,3 @@
 :root{font-family:system-ui,sans-serif;color:#e9eee5;background:#162321;font-synthesis:none}*{box-sizing:border-box}body{margin:0}header,main,footer{max-width:1120px;margin:auto;padding:36px 28px}.eyebrow{font-size:12px;letter-spacing:.18em;color:#aab8aa}h1{font-size:clamp(42px,7vw,76px);line-height:1.02;letter-spacing:-.055em;margin:26px 0}.lead{max-width:670px;font-size:20px;line-height:1.6;color:#bfcec1}.badge{display:inline-block;padding:9px 13px;border:1px solid #77977f;border-radius:30px;font-size:13px;color:#b6e5b5}.toolbar{display:flex;gap:18px;align-items:end;flex-wrap:wrap}label{display:grid;gap:7px;font-size:13px}button,select,input{font:inherit}select,button,.upload{padding:12px 15px;border:1px solid #69836b;border-radius:7px;background:#20342d;color:#e9eee5}button{background:#c0e76b;color:#182317;font-weight:650;cursor:pointer}.upload{cursor:pointer;position:relative}.upload input{width:190px;font-size:12px}#status{min-height:24px;color:#c1cfba;font-size:14px}h2{font-size:27px;letter-spacing:-.02em;margin:0 0 14px}h3{margin:0 0 12px}.note{color:#bac7b5;line-height:1.6}.grid{display:grid;grid-template-columns:1fr 1fr;gap:18px;margin:26px 0}.panel{background:#f2f1e9;color:#22372b;padding:26px;border-radius:12px;min-width:0}.phase{font-size:11px;letter-spacing:.16em;color:#54745d}.metric{font-size:27px;overflow-wrap:anywhere;margin:10px 0}.pass{color:#c7ed85}.fail{color:#ffb3a7}.panel .pass{color:#356713}.panel .fail{color:#8e3729}.trace{display:grid;gap:8px;padding:0;list-style:none;font:13px/1.5 ui-monospace,monospace;overflow-wrap:anywhere}.trace li{border-left:3px solid #a4b78d;padding:8px 12px;background:#e3e8db}.checks{list-style:none;padding:0;display:grid;gap:9px}.checks li{padding:10px 14px;background:#23382e;border-radius:5px}.limits{border-top:1px solid #49604d;margin-top:38px;padding-top:28px;max-width:830px;line-height:1.7;color:#b4c5b3}.limits h2{color:#e6edde}footer{border-top:1px solid #49604d;font-size:12px;color:#9daf9a;display:grid;gap:12px}#provenance{overflow-wrap:anywhere}details{margin-top:16px;font-size:13px}summary{cursor:pointer}pre{white-space:pre-wrap;overflow-wrap:anywhere;font-size:12px}#comparison{margin:20px 0;padding:22px;border:1px solid #95c265;border-radius:8px}:focus-visible{outline:3px solid #dbb155;outline-offset:4px}@media(max-width:650px){header,main,footer{padding:24px 18px}.grid{grid-template-columns:1fr}.toolbar{align-items:stretch}.toolbar>*{width:100%}.upload input{width:100%}.panel{padding:20px}}
+
+.recorded-links{font-size:14px;line-height:1.6;color:#bfcec1}.recorded-links a{color:#b6e5b5;text-underline-offset:3px}