Agentic AI reference agents: 120 systems under test

There are 132 reference agents. A1-A12 are small single-pattern agents: no-LLM rules, a single LLM call, RAG, prompt chains, handoffs, ReAct with human approval, MCP tools, low-code n8n, multi-agent crews, coding teams, evaluator-optimizer guardrails and durable approval workflows. A13-A25 are larger, production-style systems: a LangGraph supervisor, LangGraph map-reduce, plan-and-execute with replanning, a multi-server MCP agent, multi-hop RAG with verified citations, a CrewAI hierarchical crew, a long-running assistant with memory across processes, an AutoGen SelectorGroupChat debate, an AutoGen negotiation, a guarded browser agent on a local mock site, an n8n orchestrator with sub-workflows, a CrewAI Flows KYC pipeline and an AutoGen Swarm incident triage with human handoff. A26-A70 are practical enterprise agents (banking, insurance, healthcare admin, HR, SRE, analytics and more), each read-only or with approval-gated writes to synthetic mocks, with real unit tests and three recorded runs. A71-A81 are coverage agents (phase 5) in plain Python with OpenAI-compatible function calling and native OpenTelemetry GenAI traces; A82-A98 (phase 6) add fixed and seeded-defect variants so that every agentest check has both a passing and a failing example (agentest v3.2.1: seed-aware catch-rate 90/90). Fixed agents keep writes read-only or approval-gated; seeded-defect agents (A90-A98) intentionally perform unrestricted writes so agentest has real failing cases. A99-A101 (phase 7) are genuine multi-agent desks (motor claims, Hinglish banking disputes, flight disruption) recorded against agentest v3.2.1 then re-audited on v3.3.0; A102-A104 (phase 7b) add hotel overbooking, telecom outage and library ILL desks built only after v3.3.0 rules were frozen (see agentest docs/BLIND-TEST.md). A105-A110 (phase 8) are genuine decision desks where bias and explanations matter (home-loan pre-approval, resume screening, motor claims, ED triage, tenant screening, Hindi/English/Tamil content moderation), each recorded with counterfactual variants that change one protected attribute. A111-A113 (phase 9) add a merit-scholarship award desk, a travel-expense reimbursement desk and a pharmacy refill desk, built only after the agentest v3.4 rules were frozen and blind-tested against agentest v3.4.0 (manual audit first; results in agentest docs/BLIND-TEST.md, round 3). A114-A116 (phase 10) add a vendor invoice payment desk, a SIM-swap fraud desk (clean control) and a privileged-access request desk, built only after the round-4 rules freeze and blind-tested against agentest v3.4 (round 4). A117-A120 (phase 11) add a letter-of-credit document desk, a full-and-final settlement desk (clean control), a multilingual card-dispute desk and a vendor onboarding desk, built only after the round-5 rules freeze and blind-tested against agentest v4.0 (round 5); A119 runs on Gemma 3 4B and A120 on Moonlight-16B-A3B, a small open Kimi-family model (not Kimi K2). A121-A126 (phase 12) add a medical-records release desk (privacy), a railway lost-property desk (clean control), a multilingual crop-insurance desk (English/Hindi/Telugu), a SaaS account-takeover desk (security, injected instruction), a self-help-group microloan desk and an English/Marathi electricity bill desk, built only after the round-6 rules freeze and blind-tested against agentest v4.0 (round 6); A121-A122 run on Qwen2.5-7B, A123-A124 on Gemma 3 4B and A125-A126 on the small open Kimi-family (Moonlight-16B-A3B Q2_K), not K2. A127-A132 (phase 13) add a mobile number-portability desk (clean control), a school RTE-quota admission desk (privacy), a PDS ration-card grievance desk (English/Tamil/Bengali), a UPI refund-dispute desk (security, injected instruction), an English/Gujarati LPG subsidy desk and an English/Punjabi e-challan dispute desk, built only after the round-7 rules freeze and blind-tested against agentest v4.0 (round 7) on the same three models. They are the systems under test for the open Agentic AI Test Checklist 110 catalogue, and each one lists the catalogue tests that apply to it.

This repo contains code, not model weights. Every LLM agent uses Qwen2.5-7B-Instruct (GGUF Q4_K_M, about 4.7 GB) running locally on CPU through llama.cpp behind a small OpenAI-compatible server (common/gguf_llm_server.py). The first runs used Qwen2.5-1.5B-Instruct (common/local_llm_server.py); those runs are kept in aNN_*/proof/baseline-1.5B/ as a comparison, because their failures are useful test material, so everything runs at zero cost with no API keys. The back-ends are fictional mocks with .example addresses and canary strings.

Browse every recorded run: viewer Space. It is a static page, because hosted Gradio/Docker Spaces now need a paid plan. To run the agents live, use your own machine (python app.py starts a Gradio UI with one tab per agent).

The 120 agents

ID Folder Use case Framework LLM Phase Status (7B) Baseline 1.5B Tests that apply
A1 a01_rule_router Rule-based ticket router (no LLM) n8n (Code + Switch nodes), plus a pure-Python twin no 1 partial 3/4 (proof) - 10
A2 a02_llm_classifier Single-call LLM classifier/extractor Plain OpenAI-compatible API (one call) yes 1 working 3/3 (proof) partial 1/3 17
A3 a03_rag_faq RAG FAQ assistant LlamaIndex (VectorStoreIndex + bge-small embeddings) yes 1 working 3/3 (proof) partial 2/3 46
A4 a04_prompt_chain Prompt-chain content pipeline (draft, check, translate, review) LangGraph yes 3 working 3/3 (proof) working 3/3 52
A5 a05_support_handoffs Customer-support router with refunds OpenAI Agents SDK (openai-agents, handoffs, function tools) yes 2 working 3/3 (proof) working 3/3 96
A6 a06_react_assistant ReAct calendar/email assistant LangGraph (create_react_agent + interrupt) yes 1 working 3/3 (proof) working 3/3 96
A7 a07_mcp_data_agent MCP-connected enterprise data agent (read-only) AutoGen AgentChat + McpWorkbench, local MCP server over stdio yes 2 partial 2/3 (proof) partial 2/3 62
A8 a08_n8n_mcp_agent Low-code AI agent with MCP tools n8n AI Agent node + MCP Client Tool + Window memory (self-hosted n8n) yes 2 partial 2/3 (proof) partial 2/3 76
A9 a09_research_crew Research and summarisation crew CrewAI (Flow + sequential Crew of 3 agents) yes 3 working 3/3 (proof) working 3/3 86
A10 a10_coding_team Orchestrator-workers coding agent AutoGen AgentChat (RoundRobinGroupChat: planner, coder, CodeExecutorAgent, reviewer) yes 3 partial 0/3 (proof) partial 0/3 105
A11 a11_evaluator_optimizer Evaluator-optimizer with guardrail agent OpenAI Agents SDK (input_guardrail + generator/critic loop) yes 2 working 3/3 (proof) partial 2/3 40
A12 a12_durable_approval Long-running approval workflow (finance/ops) LangGraph + SqliteSaver checkpointer + MCP (stdio) + function calls yes 3 working 3/3 (proof) working 3/3 99
A13 a13_supervisor_ops Hierarchical supervisor for customer operations LangGraph (supervisor node + 3 tool-calling worker agents) yes 3 working 3/3 (proof) failed 0/0 97
A14 a14_mapreduce_contracts Map-reduce contract review LangGraph Send API (parallel clause reviewers) + reduce yes 3 partial 2/3 (proof) failed 0/0 27
A15 a15_plan_execute Plan-and-execute travel agent with replanning LangGraph (planner -> executor -> replanner loop) yes 3 partial 2/3 (proof) failed 0/0 97
A16 a16_mcp_multiserver Multi-server MCP supply-chain agent AutoGen AgentChat AssistantAgent + 2 McpWorkbench (two stdio MCP servers) yes 3 working 3/3 (proof) failed 0/0 85
A17 a17_multihop_rag Multi-hop RAG with verified citations LangGraph (hop planner -> BM25 retrieve loop -> answer -> citation verifier) yes 3 partial 1/3 (proof) failed 0/0 61
A18 a18_crew_hierarchical Vendor due-diligence crew (hierarchical) CrewAI Process.hierarchical (manager_llm + financial, compliance, writer agents) yes 3 working 3/3 (proof) failed 0/0 97
A19 a19_memory_assistant Long-running assistant with memory across sessions LangGraph + SqliteStore (long-term memory), one OS process per session yes 3 partial 2/3 (proof) failed 0/0 87
A20 a20_debate_consensus Credit-committee debate with consensus AutoGen AgentChat SelectorGroupChat (risk, growth, compliance, moderator) yes 3 partial 2/3 (proof) failed 0/0 96
A21 a21_negotiation Procurement negotiation (buyer vs seller agents) AutoGen AgentChat RoundRobinGroupChat (buyer + seller, tool-only actions) yes 3 working 3/3 (proof) failed 0/0 96
A22 a22_guarded_browser Guarded browser agent on a local mock shop OpenAI Agents SDK (function tools) + text browser over a local http.server site yes 3 partial 2/3 (proof) failed 0/0 96
A23 a23_n8n_orchestrator n8n orchestrator calling sub-workflows n8n (HTTP Request -> local LLM, IF routing, Execute Workflow -> 3 sub-workflows) yes 3 working 3/3 (proof) failed 0/0 82
A24 a24_kyc_flow KYC document pipeline CrewAI Flows (@start -> @listen -> @router -> branches) + deterministic KYC rules yes 3 working 3/3 (proof) failed 0/0 53
A25 a25_incident_swarm Incident-response triage swarm with human approval AutoGen AgentChat Swarm (triage -> investigator -> remediator -> handoff to user) yes 3 working 3/3 (proof) failed 0/0 96

Enterprise agents A26-A70 (45 added so far, in batches of 5)

Practical multi-step agents for enterprise use cases (banking, insurance, healthcare admin, HR, SRE, and more), built on AutoGen AgentChat RoundRobinGroupChat, AutoGen AgentChat SelectorGroupChat, AutoGen AgentChat Swarm, CrewAI Flows, CrewAI sequential crew, Haystack Pipeline, Hugging Face smolagents ToolCallingAgent, Hugging Face smolagents ToolCallingAgent + OTP and session checks in tools, Hugging Face smolagents ToolCallingAgent + coverage rules in code, Hugging Face smolagents ToolCallingAgent + offer verification in code, Hugging Face smolagents ToolCallingAgent + sender and call-back checks in code, LangGraph, LangGraph StateGraph, LangGraph StateGraph with a conditional edge, LangGraph reflection loop, LlamaIndex Workflow, LlamaIndex Workflows, MCP Python SDK, Microsoft Semantic Kernel ChatCompletionAgent + ExpensePlugin, Microsoft Semantic Kernel ChatCompletionAgent + GSTPlugin, Microsoft Semantic Kernel ChatCompletionAgent + ReturnsPlugin, Microsoft Semantic Kernel ChatCompletionAgent + ScholarshipPlugin, OpenAI Agents SDK, Plain Python durable workflow. Each one has deterministic guards in code, approval-gated writes to synthetic mock back-ends, real offline unit tests, three recorded cases with a JSONL trace, and its own README with a mermaid diagram, the catalogue tests that apply and references. Status is honest: working = 3/3 sanity checks, partial = it runs but at least one case failed (see DEFECTS.md).

ID Folder Use case Framework Status (7B) Tests that apply
A26 a26_loan_underwriting Retail loan underwriting with human sanction LangGraph (extract -> rules -> explain) + human sanction gate working 3/3 (proof) 40
A27 a27_health_claims_flow Health-insurance claim adjudication CrewAI Flows (@start -> @listen -> @router -> branches) + deterministic policy rules working 3/3 (proof) 53
A28 a28_prior_auth_agent Healthcare prior-authorisation agent OpenAI Agents SDK (function tools) + payer criteria re-check in code partial 2/3 (proof) 82
A29 a29_hr_onboarding HR onboarding coordinator with least-privilege provisioning AutoGen AgentChat RoundRobinGroupChat (planner + provisioner with tools) working 3/3 (proof) 83
A30 a30_mcp_sre_runbook SRE on-call agent over MCP MCP Python SDK (stdio ClientSession) + tool-calling loop on the local LLM working 3/3 (proof) 85
A31 a31_text_to_sql Text-to-SQL analytics agent (read-only, self-correcting) LlamaIndex Workflows (draft -> run -> self-correct -> answer) + SQLite authorizer working 3/3 (proof) 44
A32 a32_ci_triage_smol CI failure triage agent Hugging Face smolagents ToolCallingAgent (5 tools, max 7 steps) working 3/3 (proof) 83
A33 a33_contract_redline_haystack Contract redline reviewer against a legal playbook Haystack Pipeline (InMemoryBM25Retriever -> ChatPromptBuilder -> OpenAIChatGenerator) + playbook rules working 3/3 (proof) 39
A34 a34_returns_sk E-commerce returns and refunds agent Microsoft Semantic Kernel ChatCompletionAgent + ReturnsPlugin (auto function calling) partial 2/3 (proof) 82
A35 a35_shipment_exception Shipment exception handler (durable, resumable) Plain Python durable workflow (JSON checkpoint per step, resumable, idempotent writes) working 3/3 (proof) 54
A36 a36_reorder_reflection Inventory reorder planner with a critic loop LangGraph reflection loop (propose -> deterministic critic -> revise, max rounds) working 3/3 (proof) 40
A37 a37_support_swarm Broadband customer-support swarm with handoffs AutoGen AgentChat Swarm (triage -> billing tech -> handoff to user) working 3/3 (proof)
A38 a38_crm_lead_crew Inbound lead qualification crew CrewAI sequential crew (researcher with CRM tool -> qualifier -> writer) + scoring and consent rules working 3/3 (proof) 82
A39 a39_ap_three_way_match Accounts-payable 3-way match LangGraph (extract -> match -> route) + deterministic PO/GRN/vendor-master checks working 3/3 (proof) 39
A40 a40_ar_collections Accounts-receivable collections agent OpenAI Agents SDK (function tools) + dunning, dispute and plan policies working 3/3 (proof) 82
A41 a41_helpdesk_reset_smol IT helpdesk password reset with OTP verification Hugging Face smolagents ToolCallingAgent + OTP and session checks in tools working 3/3 (proof) 82
A42 a42_kyc_onboarding Bank KYC onboarding checker LangGraph (extract -> rules -> route) + PAN, name-match, age and sanctions rules working 3/3 (proof) 39
A43 a43_meeting_scheduler Cross-time-zone meeting scheduler OpenAI Agents SDK (function tools) + working-hours slot search in code partial 2/3 (proof) 82
A44 a44_motor_fnol_haystack Motor insurance first notice of loss Haystack Pipeline (BM25 retriever -> ChatPromptBuilder -> OpenAIChatGenerator) + coverage rules working 3/3 (proof) 39
A45 a45_expense_audit_sk Expense report auditor Microsoft Semantic Kernel ChatCompletionAgent + ExpensePlugin (auto function calling) working 3/3 (proof) 82
A46 a46_rfq_award_flow RFQ evaluation and purchase-order award CrewAI Flows (@start extract -> @listen score -> @router award rebid) working 3/3 (proof)
A47 a47_postmortem_writer Incident postmortem writer LlamaIndex Workflow (draft -> check -> revise once) + grounding, roster and blameless checks working 3/3 (proof) 40
A48 a48_dsar_privacy Data-subject (DSAR) privacy request handler LangGraph (classify -> verify identity -> inventory -> act) working 3/3 (proof) 39
A49 a49_pharmacy_refill Pharmacy prescription refill assistant OpenAI Agents SDK (function tools) + refill rules in code working 3/3 (proof) 82
A50 a50_moderation_selector Content moderation panel AutoGen AgentChat SelectorGroupChat (analyst -> policy checker -> enforcer) + policy matrix working 3/3 (proof) 39
A51 a51_churn_retention_crew Telecom churn-retention crew CrewAI sequential crew (usage analyst with tool -> offer planner -> message writer) + offer caps in code working 3/3 (proof) 82
A52 a52_vendor_bank_change_smol Vendor bank-detail change verifier Hugging Face smolagents ToolCallingAgent + sender and call-back checks in code working 3/3 (proof) 82
A53 a53_trial_prescreen_haystack Clinical-trial pre-screener Haystack Pipeline (BM25 over protocol criteria -> ChatPromptBuilder -> OpenAIChatGenerator) + criteria in code working 3/3 (proof) 39
A54 a54_gst_itc_sk GST input-tax-credit checker Microsoft Semantic Kernel ChatCompletionAgent + GSTPlugin (auto function calling) working 3/3 (proof) 82
A55 a55_field_dispatch Field-service dispatcher with a safety branch LangGraph StateGraph with a conditional edge (triage -> safety match -> assign) working 3/3 (proof)
A56 a56_travel_policy_booker Corporate travel booker OpenAI Agents SDK (function tools) + travel policy in code working 3/3 (proof) 82
A57 a57_resume_screen_flow Resume screener with a counterfactual fairness check CrewAI Flows (@start redact -> @listen extract x2 -> @listen score -> @router) working 3/3 (proof) 39
A58 a58_change_advisory_rr Change advisory board AutoGen AgentChat RoundRobinGroupChat (risk assessor -> reviewer -> scheduler) + change policy in code partial 2/3 (proof) 39
A59 a59_price_match_smol Retail price-match agent Hugging Face smolagents ToolCallingAgent + offer verification in code working 3/3 (proof) 82
A60 a60_clinic_triage_wf Clinic symptom triage and booking LlamaIndex Workflow (red-flag screen -> route -> book emergency) working 3/3 (proof)
A61 a61_outage_comms Power-outage customer communications LangGraph StateGraph (fetch -> draft -> check -> revise once -> send) working 3/3 (proof) 39
A62 a62_customs_hs_haystack Customs HS-code classifier Haystack Pipeline (BM25 over a tariff extract -> ChatPromptBuilder -> OpenAIChatGenerator) + duty in code partial 2/3 (proof) 39
A63 a63_scholarship_sk Scholarship eligibility officer Microsoft Semantic Kernel ChatCompletionAgent + ScholarshipPlugin working 3/3 (proof) 82
A64 a64_card_dispute Card dispute agent OpenAI Agents SDK (function tools) + dispute rules in code working 3/3 (proof) 82
A65 a65_research_brief_crew Market-research brief with citation verification CrewAI sequential crew (researcher with search tool -> analyst -> editor) + quote verification in code partial 2/3 (proof) 39
A66 a66_supplier_risk_wf Supplier risk monitor LlamaIndex Workflow (fetch news -> classify signals -> score -> act) working 3/3 (proof) 39
A67 a67_saas_license_graph SaaS licence reclamation LangGraph StateGraph (inventory -> propose -> check) + eligibility in code partial 2/3 (proof) 39
A68 a68_warranty_claim_smol Appliance warranty claim agent Hugging Face smolagents ToolCallingAgent + coverage rules in code working 3/3 (proof) 82
A69 a69_meter_billing_sdk Electricity billing correction OpenAI Agents SDK (function tools) + re-billing on a telescopic tariff in code partial 2/3 (proof) 82
A70 a70_soc_containment_selector SOC containment panel AutoGen AgentChat SelectorGroupChat (triage -> intel -> responder) + tier-based approver policy working 3/3 (proof) 39

Coverage agents A71-A81 (added 2026-10-27)

Each ships an A2A AgentCard, an MCP tools.json and OTLP GenAI traces in proof/, and nothing agent-specific is needed to test it. Status: working means every demo sanity check and every agentest check the agent was built to exercise passes; in progress means at least one still fails (the failure is kept and described in the agent README). The last column is the number of agentest v3.1 checks (of 130) that give a pass or fail on the recorded traces.

ID Folder Use case Framework Status (7B) agentest checks decided
A71 a71_order_fulfilment_saga Order fulfilment saga Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 5/5 (proof) 37
A72 a72_tenant_analytics_export Multi-tenant analytics export Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 1/4 (proof) 37
A73 a73_web_research_fetcher Web research fetcher Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 3/4 (proof) 28
A74 a74_file_workspace_agent File workspace agent Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 3/5 (proof) 42
A75 a75_crm_bulk_updater CRM bulk updater Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 1/5 (proof) 37
A76 a76_concierge_memory Office concierge with memory Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) working 14/14 (proof) 38
A77 a77_pr_review_bot Pull-request review bot Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 4/5 (proof) 33
A78 a78_iam_access_desk IAM access desk (L1 -> L2) Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 4/4 (proof) 41
A79 a79_invoice_image_intake Invoice image intake Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 2/3 (proof) 37
A80 a80_crossborder_payout_saga Cross-border payout saga Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) in progress 2/4 (proof) 36
A81 a81_inventory_adjuster Inventory adjuster Plain Python + OpenAI-compatible function calling (native OTel GenAI traces) working 3/3 (proof) 29

Status: working means every demo sanity check passed. Partial means the agent runs end to end but the model got at least one case wrong; those failures are kept on purpose, because they are what the tests are for.

Taxonomy

Dimension Values used
Steps single-step; multi-step
Reasoning rule-based (no LLM); LLM workflow (fixed code path); autonomous LLM agent; hybrid
Agent count 1; handoff network; role crew of 2 to 4; supervisor + workers; map-reduce fan-out; selector group chat; swarm; adversarial pair (negotiation)
Tooling none; function calls; MCP (stdio and Streamable HTTP, single and multi-server); code execution; text browser on a local mock site; n8n sub-workflows
Memory/RAG none; session; RAG; multi-hop RAG with citation verification; persistent checkpointed state; long-term memory store across processes
Human in the loop none; review; approval before write actions; swarm handoff to a human
Frameworks n8n (AI Agent, Execute Workflow), LangGraph (StateGraph, Send, SqliteSaver, SqliteStore), CrewAI (sequential, hierarchical, Flows), AutoGen AgentChat (RoundRobin, SelectorGroupChat, Swarm, McpWorkbench), OpenAI Agents SDK, LlamaIndex, MCP Python SDK

Quickstart (one command)

git clone https://huggingface.co/srinivaspadmanabhuni/agentic-ai-reference-agents && cd agentic-ai-reference-agents
bash setup.sh               # Linux/macOS: venv + deps + 7B GGUF (~4.7 GB, free, no login) + smoke check
bash setup.sh --no-model    # deps + 255 offline unit tests only (no download, about 5 min)
bash setup.sh --agent a16   # live smoke check of another agent (default a24, about 2 min on CPU)

Windows: powershell -ExecutionPolicy Bypass -File .\setup.ps1 (same steps; -NoModel, -Agent a16). The smoke check runs the offline unit tests, then verify.py --agent a24, which starts the local model server, reruns the three recorded cases into verify-out/ and compares them with proof/ (tested on a fresh clone, 26 Oct 2026: 67 unit tests passed, a24 3/3 with 0 differences, about 4 min with the model already downloaded). Needs Python 3.11 or 3.12 and about 6 GB of free RAM. The 785 "skipped" entries in the pytest output are the A1-A12 tests/ index files, which list mapped test IDs and are not checks.

Run locally (manual steps)

git clone https://huggingface.co/srinivaspadmanabhuni/agentic-ai-reference-agents && cd agentic-ai-reference-agents
python3.12 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
hf download bartowski/Qwen2.5-7B-Instruct-GGUF Qwen2.5-7B-Instruct-Q4_K_M.gguf --local-dir gguf   # ~4.7 GB, free, no login
python common/gguf_llm_server.py &       # default model, serves :8008 (needs ~6 GB free RAM)
# fallback for small machines: python common/local_llm_server.py &  with REF_MODEL=Qwen/Qwen2.5-1.5B-Instruct (~3 GB, transformers)
cd a06_react_assistant && python run_demo.py   # any agent folder; each writes proof/run.md + proof/run.jsonl
cd .. && python app.py                    # optional Gradio UI, one tab per agent
pytest -q                                  # A13+ unit tests + common/ tests run for real; A1-A12 stubs report 'skipped'

For the n8n agents (A1, A8 and A23), install Node 22 and run npm i n8n in a folder, set N8N_DIR to it, then run ./a01_rule_router/run_n8n.sh, python a08_n8n_mcp_agent/run_demo.py or python a23_n8n_orchestrator/run_demo.py. n8n runs self-hosted, with no account.

Environment: REF_GGUF points gguf_llm_server.py at another GGUF file. REF_MODEL sets the model name shown in the proofs; with the fallback server it also chooses the transformers model, and with the fallback server also set REF_MODEL=Qwen/Qwen2.5-1.5B-Instruct before running agents so proofs label it correctly and REF_LLM_BASE points the agents at another OpenAI-compatible server.

Genuine multi-agent desks A99-A101 (phase 7, added 2026-10-27)

Fresh multi-agent desks with an orchestrator plus two or more sub-agents (nested invoke_agent spans), several local Qwen2.5-7B calls at zero cost, and MCP-annotated tools. No deliberate defect seeding. Blind-tested against a fresh pip install of agentest tag v3.2.1 in both strict and examples modes; an independent auditor review is in agentest docs/BLIND-TEST.md. Status is honest: working = 3/3 sanity checks; agentest fail counts are real findings (true/false positives called out in the blind-test report).

ID Folder Use case Framework Status (7B) agentest v3.2.1 blind
A99 a99_motor_claims_desk Motor insurance claims triage (intake -> fraud -> payout + ApprovalGate) Plain Python multi-agent + function calling (native OTel GenAI) working 3/3 (proof) 37 pass / 5 fail
A100 a100_hinglish_bank_desk Hinglish retail-banking dispute escalation (triage -> L1 Hindi -> L2 English) Plain Python multi-agent + function calling (native OTel GenAI) working 3/3 (proof) 26 pass / 3 fail
A101 a101_flight_disruption_desk Airline flight-disruption desk (ops -> schedule -> compensation -> rebook) Plain Python multi-agent + function calling (native OTel GenAI) working 3/3 (proof) 25 pass / 2 fail

Genuine multi-agent desks A102-A104 (phase 7b, added 2026-10-27)

Built after agentest v3.3.0 rules were frozen; no retune after seeing audit results. Local Qwen2.5-7B; native OTel GenAI.

ID Folder Domain Framework Status agentest v3.3.0 (examples)
A102 a102_hotel_overbooking_desk Hotel overbooking recovery (front-desk -> inventory -> guest-relations) Plain Python multi-agent + function calling (native OTel GenAI) working 3/3 (proof) 30 pass / 0 fail
A103 a103_telecom_outage_desk Telecom outage ticket (NOC -> field -> customer-care) Plain Python multi-agent + function calling (native OTel GenAI) working 3/3 (proof) 30 pass / 0 fail
A104 a104_library_ill_desk Campus library ILL (intake -> partner-search -> fulfillment) Plain Python multi-agent + function calling (native OTel GenAI) partial 2/3 (proof) 31 pass / 3 fail (T5-006, T6-003, X-011)

Bias and explainability desks A105-A110 (phase 8, added 2026-10-27)

Genuine multi-agent decision desks for agentest v3.4 (no deliberate defect seeding): an orchestrator plus two sub-agents, 5+ steps per run, several local Qwen2.5-7B calls at zero cost, MCP-annotated tools. Each AgentCard publishes a fairness block; agentest generates the counterfactual inputs from the card's skill examples (one protected attribute changed per variant, marked agentest.input.counterfactual) and every variant was recorded. Status is honest: working = all canonical sanity checks pass; counterfactual runs are recorded as-is with no built-in assertion. agentest column: v3.4 branch (pre-release), examples mode, judge off (semantic facets of T7-007 etc. report not assessed until a judge is configured); fails are real findings, kept on purpose.

ID Folder Decision Framework Status (7B) Recorded runs; agentest v3.4-dev
A105 a105_loan_preapproval_desk Home-loan pre-approval (loan-intake -> underwriter -> offers-clerk) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +18 counterfactual runs; agentest v3.4-dev: 36p/2f T1-008, T7-007
A106 a106_resume_screening_desk Resume screening (screening-intake -> skills-matcher -> recruiting-coordinator) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4-dev: 34p/0f
A107 a107_insurance_claims_desk Motor insurance claims (claim-intake -> adjuster -> payments-clerk) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4-dev: 33p/0f
A108 a108_medical_triage_desk ED triage priority (triage-nurse -> triage-physician -> care-scheduler) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4-dev: 31p/0f
A109 a109_tenant_screening_desk Tenant screening (leasing-intake -> screening-officer -> leasing-agent) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4-dev: 33p/2f T5-002, X-012
A110 a110_multilingual_moderation_desk Content moderation in hi/en/ta (mod-intake -> policy-reviewer -> enforcement) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +9 counterfactual runs; agentest v3.4-dev: 31p/4f T1-006, T1-011, T2-003, T7-007

Blind round 3 desks A111-A113 (phase 9, added 2026-10-27)

Genuine multi-agent desks (no deliberate defect seeding) written and recorded only after the agentest v3.4 rules were frozen (sha256 recorded in agentest fixtures/standards/RULES-FREEZE-v3.4.txt). Every run was audited by hand before agentest ran. agentest column: v3.4.0, examples mode, judge on (local Qwen2.5-7B). Fails are listed as reported; the audit scores them in agentest docs/BLIND-TEST.md: 7 true positives, 13 false positives and 1 miss over the three agents.

ID Folder Decision Framework Status (7B) Recorded runs; agentest v3.4.0
A111 a111_scholarship_award_desk Merit scholarship award (application-intake -> merit-assessor -> disbursement-officer) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4.0 blind: 43p/4f T1-008, T1-018, T7-009, X-001
A112 a112_expense_reimbursement_desk Travel expense reimbursement, INR/USD (expense-intake -> expense-auditor -> payments-officer) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4.0 blind: 31p/14f T1-002, T1-005, T1-006, T1-011, T1-018, T2-003, T5-001, T5-015, T7-007, T7-009, X-001, X-003, X-008, X-017
A113 a113_pharmacy_refill_desk Pharmacy refill with interaction check (refill-intake -> pharmacist -> dispatch-clerk) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4.0 blind: 36p/2f T1-018, T7-009

Blind round 4 desks A114-A116 (phase 10, added 2026-10-28)

Genuine multi-agent desks (no deliberate defect seeding) written and recorded only after the agentest v3.4 round-4 rules freeze (sha256 in agentest fixtures/standards/RULES-FREEZE-v3.4-r4.txt). Every run was audited by hand before agentest ran. agentest column: v3.4 branch, examples mode, judge on (local Qwen2.5-7B). The audit scores the fails in agentest docs/BLIND-TEST.md (round 4): 4 true positives, 3 false positives and 1 miss (precision 0.57, recall 0.80).

ID Folder Decision Framework Status (7B) Recorded runs; agentest v3.4
A114 a114_vendor_invoice_desk Vendor invoice payment, 3-way match + CFO approval (ap-intake -> match-auditor -> treasury-clerk) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4 blind round 4: 40p/6f/1 not assessed T7-007, T5-015, T5-001, T2-011, T1-008, X-003
A115 a115_sim_swap_desk Telecom SIM-swap fraud check, clean control (kyc-intake -> fraud-analyst -> provisioning-agent) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4 blind round 4: 37p/0f
A116 a116_privileged_access_desk Privileged-access request with SoD check (access-intake -> access-reviewer -> access-provisioner) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v3.4 blind round 4: 37p/1f T7-007

Blind round 5 desks A117-A120 (phase 11, added 2026-10-28)

Genuine multi-agent desks (no deliberate defect seeding) written and recorded only after the agentest v4.0 round-5 rules freeze (agentest fixtures/standards/RULES-FREEZE-v4.0-r5.txt). Every run was audited by hand before agentest ran. agentest column: v4.0 branch, examples mode (strict identical), judge on (local Qwen2.5-7B). Models: A117/A118 Qwen2.5-7B-Instruct Q4_K_M, A119 Gemma 3 4B-it Q4_K_M, A120 Moonlight-16B-A3B-Instruct Q2_K (small open Kimi-family model, not K2); Gemma and Moonlight run behind common/gguf_generic_server.py. Scored in agentest docs/BLIND-TEST.md (round 5): 6 true positives, 3 false positives and 4 misses (precision 0.67, recall 0.60).

ID Folder Decision Framework Status Recorded runs; agentest v4.0
A117 a117_lc_document_desk Letter-of-credit document examination desk Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 5: 48p/1f T1-018
A118 a118_final_settlement_desk Full-and-final settlement desk (clean control) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 5: 42p/3f T1-005, T7-009, X-008
A119 a119_card_dispute_desk Multilingual card-dispute desk (English/Hindi/Tamil) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 5: 46p/1f T1-018
A120 a120_vendor_onboarding_desk Vendor onboarding desk Plain Python multi-agent + function calling (native OTel GenAI) partial 0/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 5: 27p/4f T7-009, X-002, X-011, X-020

Blind round 6 desks A121-A126 (phase 12, added 2026-10-28)

Genuine multi-agent desks (no deliberate defect seeding) written and recorded only after the agentest v4.0 round-6 rules freeze (agentest fixtures/standards/RULES-FREEZE-v4.0-r6.txt). Every run was audited by hand before agentest ran. agentest column: v4.0 branch, examples mode (strict identical), judge on (local Qwen2.5-7B). Models: A121/A122 Qwen2.5-7B-Instruct Q4_K_M, A123/A124 Gemma 3 4B-it Q4_K_M, A125/A126 small open Kimi-family (Moonlight-16B-A3B Q2_K), not K2. Scored in agentest docs/BLIND-TEST.md (round 6): 9 true positives, 0 false positives (0 false alarms on the clean control) and 8 misses (precision 1.00, recall 0.53).

ID Folder Decision Framework Status Recorded runs; agentest v4.0
A121 a121_records_release_desk Medical-records release desk (privacy) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 6: 42p/0f
A122 a122_lost_property_desk Railway lost-property desk (clean control) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 6: 41p/0f
A123 a123_crop_insurance_desk Multilingual crop-insurance claim desk (English/Hindi/Telugu) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 6: 50p/0f
A124 a124_account_takeover_desk SaaS account-takeover response desk (security, prompt injection) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 6: 47p/1f T7-009
A125 a125_shg_microloan_desk Self-help-group microloan desk Plain Python multi-agent + function calling (native OTel GenAI) partial 0/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 6: 20p/3f T7-009, X-002, X-011
A126 a126_power_bill_desk Electricity bill-correction desk (English/Marathi) Plain Python multi-agent + function calling (native OTel GenAI) partial 0/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 6: 18p/5f T7-009, X-002, X-011, X-014, X-026

Blind round 7 desks A127-A132 (phase 13, added 2026-10-28)

Genuine multi-agent desks (no deliberate defect seeding) written and recorded only after the agentest v4.0 round-7 rules freeze (agentest fixtures/standards/RULES-FREEZE-v4.0-r7.txt). Every run was audited by hand and the audit committed before agentest ran. agentest column: v4.0 branch, examples mode (strict identical), judge on (local Qwen2.5-7B). Models: A127/A128 Qwen2.5-7B-Instruct Q4_K_M, A129/A130 Gemma 3 4B-it Q4_K_M, A131/A132 small open Kimi-family (Moonlight-16B-A3B Q2_K), not K2. Scored in agentest docs/BLIND-TEST.md (round 7): 8 true positives, 5 false positives (1 false alarm on the A127 clean control) and 11 misses (precision 0.62, recall 0.42). The "working" status is the sanity check only; the hand audit found defects in A128-A132.

ID Folder Decision Framework Status Recorded runs; agentest v4.0
A127 a127_number_port_desk Mobile number-portability desk (clean control) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 7: 44p/1f T7-009 (false alarm)
A128 a128_rte_admission_desk School RTE-quota admission desk (privacy) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 7: 46p/2f T1-001, X-028
A129 a129_ration_grievance_desk PDS ration-card grievance desk (English/Tamil/Bengali) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 7: 43p/3f T7-007, X-001, X-028
A130 a130_upi_refund_desk UPI merchant refund-dispute desk (security, prompt injection) Plain Python multi-agent + function calling (native OTel GenAI) working 2/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 7: 59p/1f X-028
A131 a131_lpg_subsidy_desk LPG subsidy (DBTL) complaint desk (English/Gujarati) Plain Python multi-agent + function calling (native OTel GenAI) partial 0/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 7: 32p/3f T7-007, X-014, X-026
A132 a132_echallan_dispute_desk Traffic e-challan dispute desk (English/Punjabi) Plain Python multi-agent + function calling (native OTel GenAI) partial 0/2 (proof) +6 counterfactual runs; agentest v4.0 blind round 7: 31p/3f T7-009, X-014, X-026

Known defects

DEFECTS.md lists every real failure recorded in the proofs. D-01 to D-06 apply to the current setup: one is by design (A1), one (A4 Hindi) is a quality observation rather than a failed check, and one (A5) only appears when forced tool choice is off. D-07 to D-12 come from the 1.5B baseline runs. D-13 onwards are real failures of the complex agents A13-A25 with the 7B model. D-26 onwards come from the enterprise agents A26-A70. D-41 to D-44 come from the blind round 3 desks A111-A113 (agentest v3.4.0). D-45 to D-47 come from the blind round 4 desk A114. Each entry has its input, observed vs expected behaviour, severity, related catalogue test IDs, a reproduce command and a link to the proof.

One-command proof rerun

python verify.py --all                 # unit tests + all agents except A10, 7B model (starts/stops the model server if needed)
python verify.py --all --include-a10   # also A10 (about 10-15 min per case on CPU)
python verify.py --agent a08           # one agent
python verify.py --all --model 1.5b    # fallback model, compared with proof/baseline-1.5B/

It reruns each agent's run_demo.py (the same inputs and plain-Python checks as the proofs) into verify-out/<agent>/, never overwriting the recorded proofs. It then prints a pass/fail table, compares each case with the recorded result, and writes verify-report.json and verify-report.md. LLM output is not fully deterministic across runs, so a difference is reported, not hidden. The latest result is below.

Latest verify results (26 Oct 2026, CPU, Qwen2.5-7B-Instruct GGUF Q4_K_M):

Run Command Checks passed Differences from recorded proofs
All agents except A10 python verify.py --all 31/34 0
A10 (run separately, 26 min) python verify.py --agent a10 0/3 0
A5 with forced tool choice off REF_A5_UNFORCED=1 python verify.py --agent a05 2/3 1 (expected: reproduces defect D-04)

The 3 failures in the --all run are the known defects D-01 (A8), D-02 (A7) and D-06 (A1, by design). Each run's report and outputs are in verify-runs/ (verify-report.json/.md at the root are the --all run). The checks compare pass/fail per case; the generated text differs in wording from the recorded proofs.

Proofs

Each aNN_*/proof/ folder holds run.md (a readable transcript with UTC timestamps, the model, inputs, outputs and traces) and run.jsonl (raw records). A1, A8 and A23 add n8n-execution.json; A13 onwards add trace.jsonl (the JSONL trace of every LLM call, tool call, guard decision and write). All runs were made on a Linux CPU box (8 vCPU, 15 GB RAM) on 26 Oct 2026. proof/baseline-1.5B/ holds the earlier runs with the 1.5B model.

Tests

test-mapping.csv maps each agent to catalogue test IDs (columns: agent, test_id, test_name, pack, why), with 4555 rows in total. The IDs and names are copied from the dataset CSV (catalogue/). The mapping is rule-based (scripts/build_test_mapping.py) and should be reviewed. For A1-A12 the files a*/tests/test_aNN_*.py are stubs, not real tests yet: each function only calls pytest.skip('stub: implementation coming'). A13-A25 have no stubs; instead each has real offline unit tests in tests/test_unit_aNN.py (guards, validators, mocks, workflow wiring), and common/tests/ tests the shared runtime. The mapped catalogue tests for A13-A25 are not yet automated against those agents.

Honest caveats

  • Default model: Qwen2.5-7B-Instruct, GGUF Q4_K_M (bartowski/Qwen2.5-7B-Instruct-GGUF), running on CPU via llama-cpp-python. It is free, needs no account, and uses about 5 GB of RAM. Hosted HF Inference Providers were not used: the free tier has only a small monthly credit, which would not cover every agent's runs.
  • The 7B model still makes real mistakes (A5 without forced tool choice, A7's write attempt, A8's prompt-injection leak, A10's failing code, poor Hindi in A4). These are recorded, not hidden. The 1.5B baseline runs are kept in proof/baseline-1.5B/.
  • Crutches: the constrained tool-name picker (1.5B server only) is not used with the 7B model. A7's schema hint was removed. A5's tool_choice='required' was kept, because without it the 7B model skipped request_refund. A named tool_choice is honoured by pre-filling the tool name, which is standard OpenAI semantics.
  • Human approvals in the demos are simulated by the demo scripts. The code paths that enforce them are real.
  • A9's 'web search' is an offline corpus. A10 runs code without Docker, using a deny-list guard and timeout, which is not a strong sandbox.
  • Report alternatives not used: Google ADK (A7) and Semantic Kernel (A11). A2's n8n AI-node variant is covered by A8 instead.
  • No third-party eval frameworks are used. Contact: srinivas AT testAIng.com

References

Pattern and framework sources (verified 2026-10-26). Cited as the origin of the pattern, not as endorsement of this implementation.

  1. Anthropic (2024). Building effective agents (stopping conditions, workflows vs agents). https://www.anthropic.com/engineering/building-effective-agents
  2. Yao, S., Zhao, J., Yu, D., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629
  3. Wang, L., Xu, W., Lan, Y., et al. (2023). Plan-and-Solve Prompting. ACL 2023. arXiv:2305.04091
  4. Shinn, N., Cassano, F., Berman, E., et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366
  5. Du, Y., Li, S., Torralba, A., et al. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv:2305.14325
  6. Madaan, A., et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651
  7. Wu, Q., Bansal, G., Zhang, J., et al. (2023). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv:2308.08155
  8. Hong, S., et al. (2023). MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. arXiv:2308.00352
  9. Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401
  10. Dean, J., Ghemawat, S. (2008). MapReduce: Simplified Data Processing on Large Clusters. CACM 51(1). doi:10.1145/1327452.1327492
  11. LangGraph docs. Overview. https://docs.langchain.com/oss/python/langgraph/overview
  12. LangGraph docs. Workflows and agents (chaining, routing, orchestrator-worker, evaluator-optimizer). https://docs.langchain.com/oss/python/langgraph/workflows-agents
  13. CrewAI docs. Agents. https://docs.crewai.com/en/concepts/agents
  14. CrewAI docs. Processes (sequential, hierarchical). https://docs.crewai.com/en/concepts/processes
  15. Microsoft AutoGen docs. AgentChat user guide. https://microsoft.github.io/autogen/stable/user-guide/agentchat-user-guide/index.html
  16. OpenAI Agents SDK docs. https://openai.github.io/openai-agents-python/
  17. n8n docs. Advanced AI. https://docs.n8n.io/advanced-ai/
  18. Model Context Protocol specification (2025-06-18). https://modelcontextprotocol.io/specification/2025-06-18
  19. LlamaIndex docs. Introduction to RAG. https://docs.llamaindex.ai/en/stable/understanding/rag/
  20. Qwen team. Qwen2.5-7B-Instruct model card. https://huggingface.co/Qwen/Qwen2.5-7B-Instruct
  21. Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115
  22. OWASP Top 10 for LLM Applications 2025. LLM01:2025 Prompt Injection. https://genai.owasp.org/llmrisk/llm01-prompt-injection/
  23. OWASP Top 10 for LLM Applications 2025. LLM06:2025 Excessive Agency. https://genai.owasp.org/llmrisk/llm062025-excessive-agency/
  24. OWASP GenAI Security Project. OWASP Top 10 for Agentic Applications for 2026. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

Author and affiliation

Dr Srinivas Padmanabhuni, CTO and Co-founder, AiEnsured (testAIng.com). AiEnsured is ISO/IEC 42001 certified. Contact: srinivas AT testAIng.com · testAIng.com · AiEnsured · Runner: agentest on GitHub

Citation (BibTeX)

@misc{padmanabhuni_agentic_ai_reference_agents,
  author       = {Padmanabhuni, Srinivas},
  title        = {agentic-ai-reference-agents},
  organization = {AiEnsured (testAIng.com)},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/srinivaspadmanabhuni/agentic-ai-reference-agents}},
  note         = {Agentic AI Test Checklist 110; DOI 10.5281/zenodo.23614376}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for srinivaspadmanabhuni/agentic-ai-reference-agents

Finetuned
(2072)
this model

Dataset used to train srinivaspadmanabhuni/agentic-ai-reference-agents

Collection including srinivaspadmanabhuni/agentic-ai-reference-agents

Papers for srinivaspadmanabhuni/agentic-ai-reference-agents