| --- |
| title: Agent Reliability Lab |
| sdk: static |
| app_file: index.html |
| license: mit |
| short_description: Replay tool failures and inspect recovery evidence |
| tags: |
| - agent-reliability |
| - mcp |
| - evaluation |
| --- |
| # Agent Reliability Lab · Cloud Lab Works |
|
|
| Four controlled tool failures, before/after policy changes, and inspectable evidence. The app replays recorded runs; it does not start cloud jobs, call a model, or upload your comparison file. |
|
|
| ## Try it |
|
|
| Choose a fixture. Inspect both traces. Download [Google Cloud Run evidence](google-suite.json) or [AWS Lambda evidence](aws-suite.json), then use **Compare a cloud run**. The app verifies the trace hashes and checks source version, policy, outcomes, and attempt counts. |
|
|
| | Fixture | Before → after | AWS / Google | |
| |---|---|---| |
| | Transient failure | tool error → success; 1 → 3 attempts | pass / pass | |
| | Permanent failure | tool error remains; 5 → 2 attempts | pass / pass | |
| | Deadline fit | timeout → success; 100 → 2000 ms deadline | pass / pass | |
| | Malformed response | protocol error remains terminal after one call | pass / pass | |
|
|
| ## Evidence and limits |
|
|
| One synthetic suite ran on AWS Lambda ARM64 and one on Google Cloud Run AMD64. Their four fixture pairs matched the local baseline; all 16 cloud trace hashes were checked after export. Operator readback confirmed the deployed code/configuration and successful execution. The latest test suite passed 12 tests on local ARM64 and Google Cloud Shell AMD64, with four additional mock tests for the AWS re-disable guard. |
|
|
| These are fixed official-MCP-SDK **reference-client** experiments, not an evaluation of an LLM or production agent. Clinic findings are inspection leads, not the acceptance oracle. Legitimate retries may still trigger its loop detector. Timings are observations, not a cross-cloud performance ranking. A hash protects bytes, not truth or origin; raw account records and infrastructure state remain private. |
|
|
| The displayed baseline is a recorded local Linux ARM64 run. Cloud Run is a private manual job with one task, no retries and a 60-second timeout. Lambda is disabled at rest, briefly enabled for one manual invocation, then disabled again. Neither has a public trigger or scheduler. Static hosting is independent of both cloud runtimes. |
|
|
| ## Source and reproduction |
|
|
| [Download the source snapshot](source.tar.gz), extract it, and follow `lab/README.md`. Snapshot revision: `fe96775b907973bdb3e5d04f059cba0b12528ccc`. It includes fixed fixtures, pinned dependencies, cloud result files, tests and AWS/Google infrastructure configurations, but no git history, credentials, private state or account-specific variables. [File checksums](manifest.json). |
|
|
| Built by [Cloud Lab Works](https://cloudlabworks.dev). MIT license. Deterministic Clinic modules retain their license and pinned provenance. Deployment lessons are in the source guide. |
|
|
