challenge-testing-brief.md

magicpin AI Challenge — Testing & Evaluation Brief

Status: Companion to challenge-brief.md. Defines the technical contract between candidate bots and magicpin's judging system. Last updated: 2026-04-26 Audience: Candidates building the bot + magicpin engineers running the harness.

Read this with challenge-brief.md — that brief defines what the bot must do (the 4-context framework, composition contract, evaluation rubric). This brief defines how the bot is tested (HTTP API, judge harness, scoring touchpoints).

---

1. The high-level model

   ┌──────────────────────────┐                    ┌────────────────────────────┐
   │  magicpin Judge Harness  │                    │  Candidate Bot (your code) │
   │  (LLM playing merchant + │                    │  HTTP server, public URL   │
   │   context injector +     │ ──── HTTP/JSON ──► │  Stateful per-conversation │
   │   scorer)                │ ◄──── HTTP/JSON ── │                            │
   └──────────────────────────┘                    └────────────────────────────┘

Two information flows:

1. Judge → Bot: pushes incremental context across all 4 dimensions (category, merchant, trigger, customer) over time. Mimics how production data updates flow into Vera.
2. Bot → Judge: the bot proactively initiates conversations, and the judge plays the merchant (or customer) role, replying realistically. The bot must handle the full conversation.

The bot is stateful (must remember context pushed earlier and conversations in flight). The judge is the source of truth for what's happened.

---

2. Endpoints the candidate must expose

All endpoints are HTTPS, JSON in/out, UTF-8. Total surface: 5 endpoints.

2.1 POST /v1/context — receive a context push

The judge calls this whenever it wants the bot to know about new or updated context. Could be the initial base dataset, or an incremental update mid-test.

Request body:

{
  "scope": "category" | "merchant" | "customer" | "trigger",
  "context_id": "dentists" | "m_001_drmeera" | "c_001_priya" | "trg_2026_04_26_research_digest",
  "version": 3,
  "payload": { /* the full context object — see §3 */ },
  "delivered_at": "2026-04-26T10:00:00Z"
}

Behavior:

• Idempotent by (context_id, version). Re-posting the same version is a no-op.
• A higher version for the same context_id replaces the prior version atomically.
• Bot must persist context until the test ends. Storing in memory is fine; just don't restart between calls.

Response (200):

{ "accepted": true, "ack_id": "ack_abc123", "stored_at": "2026-04-26T10:00:00.123Z" }

Response (409) — version conflict (you already have a higher version):

{ "accepted": false, "reason": "stale_version", "current_version": 5 }

Response (400) — malformed:

{ "accepted": false, "reason": "invalid_scope", "details": "..." }

2.2 POST /v1/tick — periodic wake-up; bot can initiate

The judge calls this every N seconds of simulated time (default: every 5 simulated minutes). The bot inspects its current context state and decides whether to send any proactive messages.

Request body:

{
  "now": "2026-04-26T10:30:00Z",
  "available_triggers": ["trg_2026_04_26_research_digest", "trg_2026_04_26_recall_priya"]
}

available_triggers is a hint listing trigger context_ids the judge considers "active right now". The bot can use any subset (or none).

Response (200):

{
  "actions": [
    {
      "conversation_id": "conv_001",
      "merchant_id": "m_001_drmeera",
      "customer_id": null,
      "send_as": "vera",
      "trigger_id": "trg_2026_04_26_research_digest",
      "template_name": "vera_research_digest_v1",
      "template_params": ["Dr. Meera", "JIDA Oct issue", "..."],
      "body": "Dr. Meera, JIDA's Oct issue landed...",
      "cta": "open_ended",
      "suppression_key": "research:dentists:2026-W17",
      "rationale": "External research digest with merchant-relevant clinical anchor; merchant is a dentist with high-risk-adult patient cohort"
    }
  ]
}

actions MAY be an empty list — the bot is free to decide nothing's worth sending this tick.

conversation_id:

• If you want to start a new conversation, generate any unique string.
• Reusing an existing conversation_id is invalid here — use /v1/reply to continue an existing conversation.

2.3 POST /v1/reply — receive a reply from the simulated merchant/customer

The judge calls this with the merchant's (or customer's) reply to a previous bot message. The bot must respond synchronously with its next move.

Request body:

{
  "conversation_id": "conv_001",
  "merchant_id": "m_001_drmeera",
  "customer_id": null,
  "from_role": "merchant",
  "message": "Yes, send me the abstract",
  "received_at": "2026-04-26T10:45:00Z",
  "turn_number": 2
}

Response (200) — three valid action values:

{ "action": "send",
  "body": "Sending now — also drafted a 90-sec patient-ed WhatsApp...",
  "cta": "open_ended",
  "rationale": "Honoring the merchant's accept; adding the next-best-step (patient-ed) as low-friction follow-on" }

{ "action": "wait",
  "wait_seconds": 1800,
  "rationale": "Merchant asked for time; back off 30 min" }

{ "action": "end",
  "rationale": "Merchant said not interested; gracefully exiting conversation" }

The bot has 30 seconds to respond. After 30s the judge marks this turn as timeout and proceeds.

2.4 GET /v1/healthz — liveness probe

Response (200):

{ "status": "ok", "uptime_seconds": 3600, "contexts_loaded": { "category": 5, "merchant": 50, "customer": 200, "trigger": 100 } }

The judge polls this every 60s during the test window. Three consecutive failures = bot disqualified for that test slot.

2.5 GET /v1/metadata — bot identity

Response (200):

{
  "team_name": "Team Alpha",
  "team_members": ["Alice", "Bob"],
  "model": "claude-opus-4-7",
  "approach": "single-prompt composer with retrieval over digest items",
  "contact_email": "team@example.com",
  "version": "1.2.0",
  "submitted_at": "2026-04-26T08:00:00Z"
}

---

3. Context payload schemas (what the judge pushes to /v1/context)

Each scope has a fixed payload shape. These mirror the dataclasses defined in challenge-brief.md §4.

3.1 scope: "category"

{
  "slug": "dentists",
  "offer_catalog": [{ "title": "Dental Cleaning @ ₹299", "value": "299", "audience": "new_user" }],
  "voice": { "tone": "peer_clinical", "vocab_allowed": ["fluoride varnish", "caries"], "taboos": ["cure", "guaranteed"] },
  "peer_stats": { "avg_rating": 4.4, "avg_reviews": 62, "avg_ctr": 0.030, "scope": "delhi_solo_practices" },
  "digest": [
    { "id": "d_2026W17_jida_fluoride", "kind": "research",
      "title": "3-mo fluoride recall cuts caries 38% better than 6-mo",
      "source": "JIDA Oct 2026, p.14", "trial_n": 2100, "patient_segment": "high_risk_adults",
      "summary": "..." }
  ],
  "patient_content_library": [
    { "id": "pc_001", "title": "3 things your teeth tell you about your heart", "channel": "whatsapp", "body": "..." }
  ],
  "seasonal_beats": [{ "month_range": "Nov-Feb", "note": "exam-stress bruxism spike" }],
  "trend_signals": [{ "query": "clear aligners delhi", "delta_yoy": 0.62, "segment_age": "28-45" }]
}

3.2 scope: "merchant"

{
  "merchant_id": "m_001_drmeera",
  "category_slug": "dentists",
  "identity": { "name": "Dr. Meera's Dental Clinic", "city": "Delhi", "locality": "Lajpat Nagar",
                "place_id": "ChIJ...", "verified": true, "languages": ["en", "hi"] },
  "subscription": { "status": "active", "plan": "Pro", "days_remaining": 82 },
  "performance": {
    "window_days": 30,
    "views": 2410, "calls": 18, "directions": 45, "ctr": 0.021,
    "delta_7d": { "views_pct": 0.18, "calls_pct": -0.05 }
  },
  "offers": [
    { "id": "o_meera_001", "title": "Dental Cleaning @ ₹299", "status": "active" },
    { "id": "o_meera_002", "title": "Deep Cleaning @ ₹499", "status": "expired" }
  ],
  "conversation_history": [
    { "ts": "2026-04-24T10:00:00Z", "from": "vera", "body": "...", "engagement": "merchant_replied" }
  ],
  "customer_aggregate": { "total_unique_ytd": 540, "lapsed_180d_plus": 78, "retention_6mo_pct": 0.38 },
  "signals": ["stale_posts:22d", "ctr_below_peer_median", "high_risk_adult_cohort"]
}

3.3 scope: "customer"

{
  "customer_id": "c_001_priya",
  "merchant_id": "m_001_drmeera",
  "identity": { "name": "Priya", "phone_redacted": "<phone>", "language_pref": "hi-en mix" },
  "relationship": {
    "first_visit": "2025-11-04", "last_visit": "2026-05-12", "visits_total": 4,
    "services_received": ["cleaning", "cleaning", "whitening", "cleaning"]
  },
  "state": "lapsed_soft",
  "preferences": { "preferred_slots": "weekday_evening", "channel": "whatsapp" },
  "consent": { "opted_in_at": "2025-11-04", "scope": ["recall_reminders", "appointment_reminders"] }
}

3.4 scope: "trigger"

{
  "id": "trg_2026_04_26_research_digest_dentists",
  "scope": "merchant",
  "kind": "research_digest",
  "source": "external",
  "merchant_id": "m_001_drmeera",
  "customer_id": null,
  "payload": {
    "category": "dentists",
    "top_item_id": "d_2026W17_jida_fluoride"
  },
  "urgency": 2,
  "suppression_key": "research:dentists:2026-W17",
  "expires_at": "2026-05-03T00:00:00Z"
}

For scope: "customer" triggers (e.g., recall_due), customer_id is populated.

---

4. The judge harness behavior — full lifecycle

Phase 1 — Warmup (T-15 min before scoring window opens)

1. Judge calls GET /v1/healthz and GET /v1/metadata to verify the bot is reachable.
2. Judge POSTs the base dataset to /v1/context:
   • 5 category contexts
   • 50 merchant contexts
   • 200 customer contexts
   • 0 triggers (triggers come during the test window)
3. Judge waits 60s for the bot to settle, then re-checks /healthz.
4. If contexts_loaded reflects all 255 base contexts, warmup passes.

Phase 2 — Test window (T0 to T0 + 60 simulated minutes)

The judge advances simulated time in 5-minute ticks. At each tick:

1. Judge POSTs any new/updated contexts that "happened" during this tick (incremental updates).
2. Judge calls POST /v1/tick with current simulated time + currently-active triggers.
3. Bot returns actions[] — zero or more proactive sends.
4. For each action:
   • Judge logs it (with rationale).
   • Judge invokes a sub-LLM playing the merchant/customer role.
   • The sub-LLM replies with a plausible response (sometimes engaged, sometimes auto-reply, sometimes hard "no", sometimes a question).
   • Judge POSTs that reply to /v1/reply.
   • Bot returns next action — send, wait, or end.
   • Repeat up to 5 turns or until bot ends.

Phase 3 — Adaptive context injection (interleaved during Phase 2)

Between ticks, the judge injects post-submission context the bot didn't see during development:

• New digest items (5 new research/compliance items per category context, posted as a new version)
• Updated performance snapshots (10 merchants get new perf numbers — some spikes, some dips)
• New triggers (15 new triggers spread across the test window)
• For 5 specific merchants: a new customer context is pushed mid-test with a recall_due trigger 2 minutes later

Bots that incorporate the new context in subsequent sends score higher. Bots that ignore it (sending stale composition) score lower. Bots that hallucinate (invent context that wasn't pushed) score lowest.

Phase 4 — Replay test (top 10 only)

For the 10 highest-scoring bots from Phase 2, the judge runs 3 deep-dive scenarios as standalone replays:

1. Auto-reply hell — judge plays a merchant whose phone is on WA Business with a canned auto-reply; sends the same canned text 4 times in a row. Bot must detect and exit gracefully.
2. Intent transition — judge plays an engaged merchant who, after 2 turns of qualification, says "ok let's do it". Bot must switch from qualifying to action immediately (not ask another qualifying question).
3. Hostile / off-topic — judge plays a merchant who replies with abuse, then asks an unrelated question ("can you also help me file my GST?"). Bot must stay on-mission politely.

Each replay: 5 turns, scored on conversation flow only.

Phase 5 — Scoring + report (T0 + 90 min)

Judge aggregates:

• Phase 2 scores (per the 5-dimension rubric in §8 of the main brief)
• Phase 3 adaptation bonus (max +5 per dimension)
• Phase 4 replay scores (top 10 only; max +30)
• Operational penalties (timeouts, healthz failures, malformed responses; max -20)

Sends each team a per-message scorecard with judge rationale.

---

5. Rate limits + timeouts

Limit	Value
Max requests/sec from judge to bot	10
Per-call timeout (judge waits this long)	30 s
/v1/context payload size cap	500 KB
/v1/tick action count cap	20 actions per tick
Healthz failures before disqualification	3 consecutive
Total test window	60 simulated minutes (real-time ~30-45 min)

If your bot needs more than 30s for /v1/tick, return an empty actions: [] immediately and process work asynchronously — but you can't catch up later, so design for the budget.

---

6. Where to deploy

Deploy your bot anywhere that gives you a public URL:

• Any cloud (AWS, GCP, Azure, Render, Fly, Railway, Replit, …)
• ngrok tunnel to localhost
• Any hosting that exposes HTTP endpoints

Requirements:

• Must respond at the URL pattern https://<your-host>/v1/* (or http:// for local testing)
• Submit your public URL via the submission portal

---

7. Reference implementation skeleton

A minimal-viable bot in ~80 lines of Python (FastAPI). Save as bot.py:

import os, time
from datetime import datetime
from fastapi import FastAPI, Request, HTTPException
from pydantic import BaseModel
from typing import Any

app = FastAPI()
START = time.time()

# In-memory stores (use Redis/SQLite for production-grade)
contexts: dict[tuple[str, str], dict] = {}    # (scope, context_id) -> {version, payload}
conversations: dict[str, list] = {}           # conversation_id -> [turns]


@app.get("/v1/healthz")
async def healthz():
    counts = {"category": 0, "merchant": 0, "customer": 0, "trigger": 0}
    for (scope, _), _ in contexts.items():
        counts[scope] = counts.get(scope, 0) + 1
    return {"status": "ok", "uptime_seconds": int(time.time() - START), "contexts_loaded": counts}


@app.get("/v1/metadata")
async def metadata():
    return {"team_name": "Team Alpha", "team_members": ["Alice"], "model": "gpt-4o-mini",
            "approach": "single-prompt composer", "contact_email": "alice@example.com",
            "version": "0.1.0", "submitted_at": "2026-04-26T08:00:00Z"}


class CtxBody(BaseModel):
    scope: str
    context_id: str
    version: int
    payload: dict[str, Any]
    delivered_at: str

@app.post("/v1/context")
async def push_context(body: CtxBody):
    key = (body.scope, body.context_id)
    cur = contexts.get(key)
    if cur and cur["version"] >= body.version:
        return {"accepted": False, "reason": "stale_version", "current_version": cur["version"]}
    contexts[key] = {"version": body.version, "payload": body.payload}
    return {"accepted": True, "ack_id": f"ack_{body.context_id}_v{body.version}",
            "stored_at": datetime.utcnow().isoformat() + "Z"}


class TickBody(BaseModel):
    now: str
    available_triggers: list[str] = []

@app.post("/v1/tick")
async def tick(body: TickBody):
    actions = []
    for trg_id in body.available_triggers:
        trg = contexts.get(("trigger", trg_id), {}).get("payload")
        if not trg: continue
        merchant_id = trg.get("merchant_id")
        merchant = contexts.get(("merchant", merchant_id), {}).get("payload")
        category = contexts.get(("category", merchant.get("category_slug")), {}).get("payload") if merchant else None
        if not (merchant and category): continue
        # YOUR COMPOSER GOES HERE — call your LLM with the 4 contexts
        body_text = f"Hi {merchant['identity']['name']}, ..."  # replace with real composition
        actions.append({
            "conversation_id": f"conv_{merchant_id}_{trg_id}",
            "merchant_id": merchant_id, "customer_id": None,
            "send_as": "vera", "trigger_id": trg_id,
            "template_name": "vera_generic_v1",
            "template_params": [merchant['identity']['name'], "...", "..."],
            "body": body_text, "cta": "open_ended",
            "suppression_key": trg.get("suppression_key", ""),
            "rationale": "Composed from category+merchant+trigger"
        })
    return {"actions": actions}


class ReplyBody(BaseModel):
    conversation_id: str
    merchant_id: str | None = None
    customer_id: str | None = None
    from_role: str
    message: str
    received_at: str
    turn_number: int

@app.post("/v1/reply")
async def reply(body: ReplyBody):
    conversations.setdefault(body.conversation_id, []).append({"from": body.from_role, "msg": body.message})
    # YOUR REPLY-COMPOSER GOES HERE
    return {"action": "send", "body": "Got it, here's what's next...", "cta": "open_ended",
            "rationale": "acknowledged + advanced"}

Run: uvicorn bot:app --host 0.0.0.0 --port 8080

This is a working skeleton. The composer logic is stubbed — replace the # YOUR COMPOSER GOES HERE blocks with your LLM call.

---

9. Local self-test before submitting

Magicpin provides a judge_simulator.py that runs a mini version of the harness against your endpoint. Use it during development:

export BOT_URL=http://localhost:8080
python judge_simulator.py

Each scenario prints the judge's prompts + your bot's responses + a mock score. Iterate until you're happy, then submit your URL.

---

10. Failure modes the judge handles

Failure	Judge behavior	Penalty
/v1/healthz returns non-200 (3× in a row)	Mark bot offline; skip remaining ticks	-10 (operational)
/v1/tick times out (>30s)	Skip this tick's actions; continue	-1 per timeout
/v1/reply times out	Mark turn as bot_silent; judge plays next merchant turn after 30s	-1 per timeout
Bot returns malformed JSON	Logged, scored as 0 for that action	-2 per malformed
Bot returns action: send with empty body	Treated as malformed	-2
Bot returns the same body verbatim it sent before in the same conversation	Anti-repetition flag	-2 per repeat

---

11. Security + privacy

• All payloads are synthetic — no real PII.
• Bots must not transmit any payload data outside the test environment (no calls to non-LLM external APIs with merchant/customer fields).
• Bots may use commercial LLM APIs (OpenAI, Anthropic, Google, DeepSeek, etc.) — those are necessary for composition.
• Bots must not persist context data after the test ends. magicpin will issue a POST /v1/teardown (optional) at end of test; on receiving it, wipe state.

---

12. Pre-flight checklist for candidates

Before submitting:

• Endpoint reachable from the public internet (HTTPS or HTTP)
• All 5 endpoints implemented and returning correct schemas
• /v1/context is idempotent on (scope, context_id, version)
• /v1/tick returns within 30s even if it has nothing to send (returns {"actions": []})
• /v1/reply returns within 30s for any conversation
• Bot persists context across calls (in-memory is fine; no restarts during test)
• judge_simulator.py passes locally with non-zero scores
• Submitted URL via submission portal
• Compute budget set (rate limits, LLM API quota, etc.) so the bot survives 60-min test

---

13. What the judge logs (for transparency)

Every test produces a per-team artifact:

results/<team_name>/
├── conversations.jsonl      # all turns, both sides, with timestamps
├── context_pushes.jsonl     # every context push, with bot's ack
├── scoring.json             # 5-dimension scores per action + per conversation
├── timeline.html            # visual timeline of the test window
├── replay_*.jsonl           # phase 4 replay transcripts (top 10 only)
└── final_report.md          # aggregated score + judge's qualitative feedback

Candidates receive their own artifact bundle within 48h of the test. Top scorers' bundles (with consent) become reference material for the next cohort.

---

14. FAQ

Q: Can the bot use external tools / function calling during composition? Yes. Your LLM can call any tool you implement internally. You can't call out to non-LLM external APIs that receive merchant/customer payloads (privacy rule §11).

Q: What if my bot needs more than 30s to compose a really good message? Two options: (a) speed it up; (b) at /v1/tick, return immediately with {"actions": []} and skip the cycle. Don't try to background-process and return late — late responses are dropped.

Q: Can I send multiple messages in one tick to the same merchant? Yes, but only one action per (merchant_id, conversation_id) pair per tick. Use a follow-up tick to send more.

Q: Does the judge see my bot's rationale field? Yes — it's included in the scoring rubric ("did the rationale match the actual output?"). High-quality rationales help the judge interpret edge cases generously.

Q: What language do replies have to be in? Match the merchant's identity.languages field. Default is English. Hindi-English code-mix is encouraged where the language pref says hi.

Q: Can my bot refuse to send when nothing's worth saying? Yes — return {"actions": []} from /v1/tick. Restraint is rewarded; spam is penalized.

Q: What if the judge pushes a context for a merchant I've never seen before, mid-conversation? Treat it as a normal new merchant. The bot should be ready for any context to arrive at any time.

Q: Is there a way to query the judge for clarification mid-test? No. The bot has only the contexts it's been pushed. This is intentional — production Vera doesn't get clarifications either.

---

End of testing brief

The two briefs together (challenge-brief.md for what to build and this one for how it's tested) are the complete spec. A team should be able to read both end-to-end in 30 minutes and start coding.