docs/retro/final-report.md

Agentic Retrospective Safety Analyzer — Final Report

Delivered in one run, structured phase-by-phase per the source spec. All numbers below are real measurements taken on this machine; anything not measurable here is explicitly labeled as such.

Verification totals: 74/74 retro tests green; full repo suite 598/598 green; tsc clean; production build green; browser walkthrough completed (screenshots in docs/retro/screenshots/).

---

Hardware findings on THIS machine (real, measured)

Property	Value
OS	macOS 26.6.1 (Darwin), arm64 — Apple Silicon
GPU	Apple M4 (unified memory, Metal/MPS only)
Python	3.11.9 (.venv)
CUDA / ROCm	None — vLLM UNSUPPORTED on this machine
Ollama	Not reachable at localhost:11434 at verification time (connection refused)
Compat chain	gpu = FAIL (clear message); driver/torch/vllm/model/quantization = SKIPPED (blocked by failed gpu link)

Measured latencies (labeled honestly):

• Mock backend, end-to-end analysis via HTTP API: 0.26–0.30 s per analysis (3 runs: 0.298 s, 0.261 s, 0.275 s) on the warm server. This is deterministic scripted analysis — no LLM.
• Ollama: no numbers reported because no server was running. The client supports real inference + latency measurement and labels results "Ollama-on-CPU/Metal" when a server is present.
• vLLM: NO benchmark numbers exist anywhere in this delivery. The benchmark script (sensorflow/retro/inference/vllm_server/benchmark.py) is runnable only on CUDA/ROCm hosts and says so; it was not executed here.

Dependency outcomes

Dependency	Outcome
chromadb	Installed cleanly, used live. The RAG index reports store_backend: chromadb at runtime. Deterministic hashed-TF + numpy cosine store remains as the always-available fallback (same interface), and tests force the fallback path so they never depend on chroma.
mcp	Installed cleanly. Bonus MCP server wrapper (sensorflow/retro/tools/mcp_server.py) adapts the tool registry; the installed package exposes MCPServer (not FastMCP), handled dynamically. The registry itself remains the MCP-style boundary.
pydantic-ai-slim	Installed, but orchestration deliberately uses the internal typed loop (typed pydantic contracts + explicit tool loop) for full determinism and auditability; PydanticAI adds no value on the mock path and would obscure the audit trail.

Reuse map (read-only imports)

• sensorflow/safety/ssam_ext.py → projected_ttc math for TTC (with validity flags added on top).
• sensorflow/megaeval stats → DistributionAnalysisTool delegates distribution-shift summaries.
• sensorflow/seqeval/paired.py → PairedSequentialTest for statistical significance in the scorecard.
• sensorflow/evaluation/copilot.py pattern → Ollama backend client shape (endpoint, model discovery, timeouts).

Test counts

74 retro tests (all passing), plus full-repo suite 598/598:

File	Tests	Covers
test_inference.py	8	env detection on this machine, vLLM-unsupported honesty, compat chain clarity, mock client contract
test_rag.py	9	chunking, retrieval eval precision, synthetic labels always present, no-citation-without-retrieval
test_tools.py	8	read-only default, write authorization, audit completeness, path allowlist, schemas
test_metrics.py	14	stopping-distance parameterization, TTC validity flags, SCR, behavioral impact
test_policy.py	13	asymmetric costs (benign-FN < phantom-FP), gate determinism, AI-severity override flow, INSUFFICIENT_EVIDENCE never PASS
test_agent.py	10	both canonical fixtures, fact/inference separation, stripped telemetry → UNKNOWN, persistence
test_api.py	9	full API lifecycle via TestClient, vLLM local-unavailability
test_optional_deps.py	3	chromadb store, MCP server bonus (skip if not installed)

---

Phase 1 — Inference foundation (sensorflow/retro/inference/)

What was built: env_detect.py (real OS/Python/Torch/GPU/CUDA/ROCm/MPS detection; detected Apple M4 on this machine), compat.py (validation chain GPU→driver→torch→vLLM→model→quantization; failed links block downstream checks with clear reasons), client.py (pluggable mock/ollama/vllm client with health check, deterministic test-prompt validation, token counts, latency), vllm_server/ (config split into server.env / model.env / runtime.env, start_vllm.sh that refuses to run on macOS, benchmark.py with TTFT P50/P95/P99, tokens/sec, warm-vs-cold), requirements-retro.txt, package README.

How to run:

.venv/bin/python -m sensorflow.retro.inference.env_detect     # env report
.venv/bin/python -m sensorflow.retro.inference.compat          # compat chain
# vLLM host (CUDA/ROCm only): edit *.env, then ./start_vllm.sh; benchmark.py

Validation checklist: env detection real on this machine PASS · vLLM honestly reported unsupported PASS · mock client contract PASS · vLLM configs/scripts provided-but-not-executed, stated explicitly PASS.

Known limitations: vLLM artifacts unverified here by definition; Ollama path exercised only via health-check code (no server running); compatibility matrix is general-knowledge with "verify against current vLLM release" caveats.

Phase 2 — Safety-case RAG (sensorflow/retro/rag/)

What was built: parser + configurable chunking, embedder (chromadb default embedder when installed; deterministic hashed-TF fallback with documented tradeoff), store (ChromaStore / NumpyCosineStore behind one protocol), retriever that ALWAYS returns {source, document, version, section, retrieved_text, relevance_score} + full metadata, seed corpus of 7 SYNTHETIC documents (safety requirements, launch criteria, perception requirements, ODD, historical retrospectives, evaluation policy, SOTIF concept paraphrase — every one tagged SYNTHETIC_EXAMPLE / NOT_A_REAL_STANDARD in content AND metadata), retrieval eval harness.

Real numbers: precision@4 = 1.0 over 14 expected-source queries, store_backend: chromadb (live server) — fallback path separately tested.

Hard rule (tested): no safety-requirement citation is ever emitted without a retrieval hit backing it; every scorecard retrieved_standards entry maps to an audited safety_standard_rag call.

How to run: GET /api/retro/rag/search?q=...&k=4, GET /api/retro/rag/eval.

Known limitations: corpus is small and synthetic by design; lexical fallback scores are not comparable to chroma's embedding distances (documented).

Phase 3 — Agent + MCP-style tools (sensorflow/retro/agent/, tools/)

What was built: ToolRegistry (input/output schema, read_only default true, timeout, error behavior, audit log with args/result-hash/timestamp for every call including denials); tools: LogReaderTool (allowlisted to runs/

• fixtures dir), SafetyStandardRAGTool, MetricCalculatorTool (stopping distance with reaction/velocity/grade/friction/decel/system latency reusing ssam_ext math; TTC with validity flags; SCR vs policy criticality; behavioral impact observed-vs-corrected), HistoricalFailureSearchTool, DistributionAnalysisTool (megaeval delegation), CreateEvaluationCaseTool (the ONE write tool, requires explicit authorization, audited). Bonus: real MCP server wrapper since mcp installed. Orchestrator: internal typed loop (Log Agent → Evidence Analyzer → RAG Agent → Safety Synthesizer); mock backend = deterministic scripted analysis, Ollama backend = real LLM synthesis with follow-up RAG queries. Five fixtures including both canonical examples and the missing-telemetry / benign-FP / non-critical-FN variants.

Tested guarantees: stripped telemetry → UNKNOWN, never guessed; fact and inference never mixed (hypotheses are Tier 4 only); write tool denied without authorization; every tool call audited.

How to run: POST /api/retro/analyze?fixture_id=missed_pedestrian_rain&backend=mock (or upload a JSON log; backend=ollama when a server is running).

Known limitations: Ollama synthesis path implemented and unit-tested at the prompt/parsing level but not exercised end-to-end here (no server); MCP server wrapper is smoke-tested, not exercised by a real MCP client.

Phase 4 — Scorecard + policy gate (scorecard.py, policy.py)

What was built: evidence hierarchy enforced in types (Tier1 OBSERVED / Tier2 DERIVED / Tier3 RETRIEVED / Tier4 AI_HYPOTHESIS); strictly-typed RetrospectiveScorecard with every spec field + markdown renderer showing the FACT / DERIVED FACT / RETRIEVED REQUIREMENT / AI HYPOTHESIS / SAFETY DETERMINATION distinction; deterministic severity BENIGN/DISRUPTIVE/CRITICAL/ FATAL from contextual evidence with the asymmetric FN/FP cost model (VRU × distance-vs-stopping × relative motion × remaining reaction time for FN; intervention magnitude × disruption for FP) — tested that a benign distant FN ranks BELOW a hard-braking phantom FP; LLM severity proposal adjudicated by the policy engine with divergence recorded + flagged for human review; policy- versioned launch gate (retro-policy/1.0.0) where INSUFFICIENT_EVIDENCE can never become PASS.

Fixture B verified result (mock, end-to-end): FALSE_NEGATIVE, severity CRITICAL (AI proposal agreed, no divergence), launch FAIL (GATE-01), human review required, 38 evidence items, 7 retrieved standards, 2 hypotheses, stopping distance 26.18 m vs 26.0 m range, TTC 1.7 s with validity flags.

Known limitations: severity weights are demonstration policy values (versioned, documented), not calibrated to a real fleet's risk model.

API + persistence

All endpoints live under /api/retro (env, compat, backends, rag/search, rag/eval, fixtures, analyze, analyses, analyses/{id}, analyses/{id}/audit, tools) — all verified 200 via curl sweep and the pytest TestClient suite. Artifacts persist under runs/retro/{analyses,audit,uploads}.

Frontend

"Retrospective Analyzer" page under SAFETY & COMPLIANCE. All new files (src/pages/retro/, src/components/retro/, src/services/retro.ts, src/types/retro.ts) + minimal additive insertions (App.tsx nav/import/render, LabelEvalContext page id, HelpMenu page name, pageHelp + glossary entries — required because those maps are typed Record<PageId, …>). Delivers: fixture picker + backend selector with honest availability badges, vertical evidence- chain view color-coded by tier with legend, scorecard with severity/launch banner + human-review flag, retrieved-standards panel with SYNTHETIC_EXAMPLE badges, audit-trail tab, hardware/compat card. tsc clean; vite build green.

Verification & servers

• Full pytest: 598 passed, 0 failed (includes 74 retro tests).
• Browser walkthrough of fixture B end-to-end on http://localhost:5199/#/retro (screenshots 01–06 in docs/retro/screenshots/).
• The pre-existing servers on :8000 and :5173 were left untouched; because the :8000 backend predates this router (started without --reload), verification runs on a fresh backend at :8100 paired with vite.retro-verify.config.ts on :5199 — both left running. Nothing was committed.

Honest gaps

1. No vLLM execution or benchmark numbers — impossible on macOS; scripts ship unverified-by-design.
2. No real-LLM end-to-end run — Ollama was not running at verification time; the mock path is the verified path.
3. The safety-case corpus is synthetic; SOTIF content is concept paraphrase only.
4. The long-running :8000/:5173 servers won't show the retro API/page until restarted (concurrent-work constraint, not a defect).
5. PydanticAI installed but intentionally unused for orchestration (typed internal loop chosen for auditability).