Add leaderboard auto-sync design spec

Daily Claude Code scheduled run: pull EOG + EVA upstream sources,
re-derive data/leaderboard.json, diff vs committed, push to staging
on change. Human-gated promotion to prod.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/superpowers/specs/2026-06-02-leaderboard-sync-design.md

@@ -0,0 +1,136 @@
+# Leaderboard Auto-Sync — Design
+
+**Date:** 2026-06-02
+**Status:** Approved (design)
+**Author:** brad + Claude
+
+## Problem
+
+The NowAI-Bench landing page (`index.html`) hardcodes three leaderboards — EnterpriseOps-Gym (EOG), EVA-Accuracy, EVA-Experience. Upstream results change over time. We need them to track two canonical upstream sources automatically, with a human gate before anything reaches production.
+
+Upstream sources:
+- **EOG:** `https://raw.githubusercontent.com/EnterpriseOps-Gym/EnterpriseOps-Gym.github.io/main/script.js` — a JS array `LEADERBOARD_DATA`.
+- **EVA:** `https://raw.githubusercontent.com/ServiceNow/eva/main/website/src/data/leaderboardStats.json` — JSON `{ "systems": [...] }`.
+
+## Decisions (locked)
+
+| Decision | Choice |
+|----------|--------|
+| Run environment | Claude Code scheduled run (skill + `CronCreate`) |
+| Cadence | Daily |
+| Data flow into page | Extract to `data/leaderboard.json`; page renders from it via inline JS |
+| Change detection | Compare derived output (re-derive our JSON, diff vs committed) |
+| On change | Commit + push to **staging only** (`origin` = ServiceNow-AI/NowAI-Bench-Staging), then notify |
+| Prod | Never touched automatically — human promotes after review |
+
+## Architecture
+
+Two parts.
+
+### Part A — One-time page refactor (no visible behavior change)
+
+1. Move the three leaderboards out of `index.html` into `data/leaderboard.json`.
+2. Add a small inline `<script>` that `fetch`es `data/leaderboard.json` on load and renders the three cards from a template function, using the **same Tailwind classes/markup as today**. Styling, tooltips, hover animation, accessibility all unchanged.
+3. Fallback: if the fetch fails, render the last-built static markup (kept inline) so the page never renders empty. Works on HF Space and the local IntelliJ server (both http).
+
+### Part B — Sync skill + schedule
+
+- New skill `sync-nowai-leaderboard` (`~/.claude/skills/sync-nowai-leaderboard/`).
+- `sync.mjs` Node script: fetch both raw upstream URLs → parse → transform to our schema → diff against committed `data/leaderboard.json`.
+- On change: rewrite `data/leaderboard.json`, commit to `main`, push to `origin` (staging) only, notify.
+- On no change: exit quietly.
+- `CronCreate` fires the skill daily.
+
+**Change detection — compare derived output:** every run re-derives `leaderboard.json` from upstream and diffs against the committed file. Updates fire only when something *we display* changes (ignores upstream churn in fields we don't render). Self-correcting. Upstream raw-file SHA-256 hashes are stored as provenance metadata, not as the trigger.
+
+## Data schema — `data/leaderboard.json`
+
+```json
+{
+  "updated": "2026-06-02T17:30:00Z",
+  "sources": {
+    "eog": { "url": "...script.js", "hash": "sha256:...", "fetchedAt": "..." },
+    "eva": { "url": "...leaderboardStats.json", "hash": "sha256:...", "fetchedAt": "..." }
+  },
+  "eog": {
+    "metricLabel": "Task Success Rate · Oracle mode",
+    "rows": [
+      { "rank": 1, "model": "Gemini 3.5 Flash (High)", "type": "closed", "score": 46.0, "bar": 100 }
+    ]
+  },
+  "evaAccuracy": {
+    "metricLabel": "Pass@1",
+    "rows": [
+      { "rank": 1, "name": "...", "subtitle": "Mixed Models · Cascade",
+        "score": 0.57, "ciLower": 0.54, "ciUpper": 0.59, "bar": 100 }
+    ]
+  },
+  "evaExperience": { "metricLabel": "Pass@1", "rows": [ "...same shape as evaAccuracy" ] }
+}
+```
+
+Each board carries 5 rows (rank 1–5). Renderer tolerates fewer.
+
+## Transform rules
+
+### EOG (from `LEADERBOARD_DATA` JS array)
+
+- Isolate the `LEADERBOARD_DATA = [ ... ]` block from `script.js` and parse it (normalize to JSON, or eval in a sandboxed `Function` — no network/fs access).
+- Each entry: `{ model, type: "closed"|"open", teams, csm, email, itsm, calendar, hr, drive, hybrid, avg }`.
+- Sort by `avg` descending → take top 5.
+- `score = avg` (1 decimal). `bar = round(avg / topAvg × 100)`. Carry `type`.
+
+### EVA (from `systems[]`)
+
+- Each system: `{ id, name, type: "cascade"|"s2s", stt, llm, tts, clean: { EVA-A_mean.pooled.{point,ci_lower,ci_upper}, EVA-X_mean.pooled.{...} } }`.
+- **Accuracy board:** sort by `clean.EVA-A_mean.pooled.point` desc → top 5.
+- **Experience board:** sort by `clean.EVA-X_mean.pooled.point` desc → top 5.
+- `score = point` (2 decimals). `ciLower`/`ciUpper` from pooled. `bar = round(point / topPoint × 100)`.
+- `subtitle`:
+  - `type === "cascade"` → `"Mixed Models · Cascade"`.
+  - `type === "s2s"` → `"<vendor> · Speech-to-Speech"`, vendor from a small name→vendor map (Gemini→Google, GPT→OpenAI, …), fallback to first token of `name`.
+
+### Assumptions to verify during implementation (flagged, non-blocking)
+
+1. **EOG "Oracle mode."** Upstream exposes one `LEADERBOARD_DATA` array; we treat `avg` as the displayed score under the existing "Task Success Rate · Oracle mode" label. If the EOG site actually has an Oracle-vs-realistic toggle backed by a second dataset, revisit the mapping then.
+2. **s2s vendor map.** Small hardcoded lookup; extend as new vendors appear.
+
+## Skill layout
+
+`~/.claude/skills/sync-nowai-leaderboard/`
+- `SKILL.md` — when/how to run, decision rules, staging-only push guardrail.
+- `sync.mjs` — fetch → parse → transform → diff. Flags:
+  - `--check` — dry-run: print diff, no writes.
+  - `--apply` — write + commit + push.
+- Run flow: Claude invokes `--check`, inspects the printed diff, then `--apply` only if the diff passes the sanity gates.
+
+## Sanity gates (abort + notify if any fail, before any commit)
+
+- Both upstream fetches return HTTP 200 and parse cleanly.
+- Each board yields ≥ 3 rows (≥ 5 preferred; renderer tolerates fewer).
+- No score outside expected range (EOG 0–100, EVA 0–1).
+- Δ guard: top model's score swings > 25 absolute pts (EOG) or > 0.25 (EVA) vs current → suspicious, abort + notify for manual review (catches garbled parses).
+
+## Error handling
+
+- **Fetch fail / non-200** → log, no repo change, retry next daily run.
+- **Parse fail (schema drift)** → abort, notify with the exception, no commit. (Primary reason for Claude-in-the-loop.)
+- **Git push fail** → report; local commit remains for manual push.
+
+## Git + notify
+
+- Commit straight to `main`; push `origin` (staging) only. **Never `prod`.**
+- Commit message: `Sync leaderboard: EOG <date>, EVA <date>` plus a one-line summary of rank/score changes.
+- Notify via `PushNotification` (fallback: scheduled-run summary): "Leaderboard synced to staging, N changes. Review + promote to prod."
+
+## Testing
+
+- **Transform unit test:** committed fixtures (snapshot of today's `script.js` + `leaderboardStats.json`) → asserts expected `leaderboard.json`.
+- **Render test (Chrome DevTools MCP):** load page, assert exactly 5 rows per card, bars present, no console errors.
+- **`--check` dry-run** doubles as the manual smoke test.
+
+## Out of scope
+
+- Auto-promotion to prod.
+- Per-domain breakdowns / radar view (planned v1.1).
+- Historical trend tracking.