sensorflow/retro/README.md

Retro — Agentic Retrospective Safety Analyzer

Local-first, evidence-driven analysis of perception failures. Traceable chain:

RAW FAILURE -> OBSERVED EVIDENCE -> DERIVED METRICS -> RETRIEVED ENGINEERING
EVIDENCE -> AGENT HYPOTHESIS -> BEHAVIORAL ANALYSIS -> SAFETY POLICY ->
RETROSPECTIVE SCORECARD -> HUMAN DECISION

Core rule: the LLM interprets/correlates/hypothesizes; deterministic code computes metrics and owns the safety boundary. Launch determinations only pass through the deterministic policy gate (policy.py).

Backends and reproducible startup

mock (default — no model, fully deterministic)

Nothing to start. All tests run against this backend.

.venv/bin/python -c "
from sensorflow.retro.inference.client import get_backend
b = get_backend('mock'); print(b.health()); print(b.validate())"

ollama (real local inference on this macOS machine — CPU/Metal)

# 1. Install & start Ollama (https://ollama.com), then pull a model:
ollama pull gemma3:latest        # or any chat model
# 2. Optionally pin the model:
export RETRO_OLLAMA_MODEL=gemma3:latest
# 3. Validate:
.venv/bin/python -c "
from sensorflow.retro.inference.client import get_backend
b = get_backend('ollama'); print(b.health()); print(b.validate())"

Numbers from this backend are Ollama-on-CPU/Metal, never vLLM numbers.

vllm (CUDA/ROCm Linux hosts ONLY — not runnable on this machine)

# On the GPU host:
pip install vllm
cd sensorflow/retro/inference/vllm_server
MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct PORT=8001 ./start_vllm.sh
# Benchmark (TTFT p50/p95/p99, tokens/sec, warm-vs-cold):
python benchmark.py --base-url http://localhost:8001/v1 --label "vLLM <GPU model>"

# Back on this machine, point the client at the host:
export RETRO_VLLM_URL=http://gpu-host:8001/v1

start_vllm.sh refuses to run on macOS with a clear error. env_detect.py and compat.py report this machine as unsupported for vLLM (verified by tests). See docs/retro/compatibility-matrix.md for CUDA vs ROCm.

Quick tour

# Environment + compat report
.venv/bin/python -m sensorflow.retro.inference.compat

# Analyze a canonical fixture end to end (mock backend)
.venv/bin/python -c "
from sensorflow.retro.agent.orchestrator import analyze_fixture
sc = analyze_fixture('missed_pedestrian_rain', backend='mock')
print(sc.render_markdown())"

# API (mounted on the studio backend at :8000)
curl -s localhost:8000/api/retro/env | python -m json.tool
curl -s -X POST 'localhost:8000/api/retro/analyze?fixture_id=phantom_brake_plastic_bag&backend=mock'

Layout

• inference/ — env detection, vLLM compat chain, pluggable client, vLLM server configs + startup + benchmark (GPU hosts only).
• rag/ — safety-case retrieval. Seed corpus is SYNTHETIC demonstration material; every synthetic rule is tagged SYNTHETIC_EXAMPLE / NOT_A_REAL_STANDARD in content and metadata. SOTIF/ISO 21448 entries are paraphrase-level concept summaries, not standard text.
• tools/ — MCP-style audited tool registry (read-only default, path allowlists, audit trail with args/result-hash/timestamp).
• agent/ — orchestrator: Log Agent -> Evidence Analyzer -> RAG Agent -> Safety Synthesizer. Mock backend = deterministic scripted analysis.
• scorecard.py — tiered evidence (OBSERVED/DERIVED/RETRIEVED/AI_HYPOTHESIS) and the strictly-typed RetrospectiveScorecard.
• policy.py — deterministic severity framework (asymmetric FN/FP cost) and the versioned launch-recommendation gate.
• api.py — FastAPI router at /api/retro; artifacts under runs/retro/.

Tests

.venv/bin/python -m pytest tests/test_retro -q