# Sensorflow Studio — Execution Integrity & Observability Audit

**Date:** 2026-08-14  
**Scope:** Legacy Studio (`static/`, `app_backend.py`, `sensorflow/`, `train.py`, `infer.py`, `autograder.py`)  
**Mirror:** `hf-space/` copies are effectively identical for these surfaces — findings apply to both.  
**Prime directive:** A UI state is NOT evidence. PASS / Done / 100% / Found / `success:true` / toasts / YAML alone / file existence alone are not proof of execution.

---

## Stage integrity table

| COMPONENT | CURRENT BEHAVIOR | ACTUAL EXECUTION? | EVIDENCE | RISK | REQUIRED FIX |
|-----------|------------------|-------------------|----------|------|--------------|
| 1 Dataset Configuration / Load & Preprocess | Save config writes JSON; Pre-Check = script file existence; “Load Catalog Metadata” returns hardcoded KPIs (`ingestion_pct: "100.0%"`) after `sleep(0.3)`; Validate & Browse is real disk scan. Toast “saved & verified” without path checks. | **Partial** — browse real; catalog preprocess is not load | `app_backend.py` precheck/details/preprocess; `static/app.js` save/browse handlers | **High** | Real Load & Preprocess with discovery counts + ledger; never show 100% Loaded from catalog |
| 2 3D Ingest & Fusion | Calls fusion engine; vendor adapters default `demo_stub`; local path can be real. UI PASS when `manifest.json` exists. | **Yes local**; **stub vendors** | ingest API + adapters; gate badges | **Medium** | PASS/DONE only with provenance; surface `demo_stub` + execution_id |
| 3 3D Perception / Auto-label (SAM) | Frontend hardcodes `no_sam: true`. Automator falls back to GT/synthetic masks; still `status: ok` → PASS. | **No real SAM** on Studio UI path | `static/app.js` perception handler; `perception_automator.py` | **Critical** | Record model/checkpoint/counts; NOT_EXECUTED if model not run; refuse fake SUCCESS |
| 4 Temporal Tracking | Real Kalman+Hungarian over proposals. | **Yes** (inputs may be fake) | track API | **Medium** | Propagate contamination; ledger entry |
| 5 Quality Gate | Real metric math vs sequence GT; contaminated proposals inflate pass. | **Yes math**; honesty depends on inputs | quality gate API | **High** | Fail closed on stub/GT-fallback; evidence card |
| 6 Launch Gate | Threshold check on metric card. | **Yes** (inherited) | launch gate evaluator | **Medium** | Same provenance gate |
| 7 Model Setup (2D) | UI model card only; no weight verification. | **No** | model cards in `app.js` | **Low–Medium** | Verify weights exist before train/infer |
| 8 Training Execution | Spawns real `train.py`. Status fabricates losses; forces progress=1.0; ignores exit_code. UI “Process finished” with no verdict. | **Partial** | train start/status; `train.py` | **Critical** | PID, command, exit_code, checkpoint hash, ledger |
| 9 Auto-Labeler Inference | Runs `infer.py` with check=True. Empty source exits 0 → ok + empty images. UI always success toast. | **Partial** | infer API; `infer.py` | **High** | Counts + checkpoint + exit_code; NOT_EXECUTED/FAILED if 0 images / missing weights |
| 10 Auto-Grader Quality | Real autograder when predictions exist; empty issues → “Perfect score!”. | **Yes** (heuristic) | grade API; `autograder.py` | **Medium** | Metrics only from grader output; ledger |
| 11 MITL Review | Hardcoded Unsplash samples; claims HF ingest; Mock Triage Auditor. | **No** | nvidia/load; mitl/evaluate | **Critical** | Label fixtures; never SUCCEEDED for mock ingest |
| 12 Model Benchmarking | `fallback_mock` hardcoded metrics with `status: ok`. | **No** (unless live card) | `/api/benchmark/compare` | **Critical** | Refuse OK without live metrics; stub = NOT_EXECUTED |
| 13 Export & Deploy | May run Ultralytics export; returns path without exists(); UI “✓ complete” without `res.ok`. | **Partial** | export API | **High** | Verify artifact; ledger |
| 14 MCP Settings | Real R/W when file exists; invented servers + “changes simulated”. | **Partial** | mcp APIs | **Medium** | Honest status; evidence stub |
| 15 SSAM Safety | Hardcoded CA intersections presented as statewide FHWA dataset. | **No real lake** | ssam endpoints + catalog | **High** | Label synthetic demo; stub evidence |

---

## High-risk fake-success occurrences

### Backend (`app_backend.py`)

| Location | Pattern |
|----------|---------|
| `/api/precheck` | Success = `train.py` / `infer.py` / `autograder.py` **file existence only** |
| `/api/train/status` | Fabricates decreasing losses; forces `progress=1.0`; ignores process `returncode` |
| `/api/infer/run` | Empty run still `status: ok` when script exits 0 with 0 images |
| `/api/export` | Returns export path without verifying file exists |
| `/api/nvidia/load` | Hardcoded samples; claims HF ingest success |
| MITL evaluate | Mock Triage Auditor returned as success-shaped response |
| `/api/benchmark/compare` | `fallback_mock` hardcoded metrics, `status: ok` |
| `DATASET_METADATA_STORE` | Hardcoded `ingestion_pct: "100.0%"` catalog KPIs |
| `/api/dataset/preprocess` | `time.sleep(0.3)` + catalog echo — no disk load |
| MCP get/toggle | Invented config / “changes simulated” with ok/warning |
| SSAM store | Hardcoded intersections as statewide dataset |
| `/api/pipeline/status` | File existence ⇒ `*_complete: true` |

### Frontend (`static/app.js`)

| Location | Pattern |
|----------|---------|
| Gate badges | Completion stages → **PASS** from file-complete; never FAIL |
| Config save | “verified successfully” without `res.ok` / path validation |
| Training poll | “Process finished” with no exit_code / success check |
| Infer | Always success toast (even empty / `!res.ok`) |
| Grader | Empty issues → “Perfect score!” |
| Export | “✓ Export complete” without `res.ok` |
| Perception | **`no_sam: true` hardcoded** |

### Scripts / sensorflow

| Location | Pattern |
|----------|---------|
| `train.py` dgx-spark | Prints remote offload; uses local device |
| `infer.py` | No images → print and **exit 0** |
| `perception_automator.py` | Silent SAM off; synthetic masks/LiDAR; GT class copy |
| Vendor adapters | Built-in ~3-frame `demo_stub` lakes |

---

## Cross-cutting patterns

1. **`status: "ok"` / success toast without evidence**
2. **File existence = done/PASS**
3. **Hardcoded metrics / “100%” catalog KPIs**
4. **Simulated delays / stubs claiming complete**
5. **Subprocess integrity gaps** (exit ignored, empty exit 0)
6. **GT contamination loop** (`no_sam` → synthetic/GT proposals → quality/launch PASS)
7. **Frontend rarely checks `res.ok`**

---

## What is relatively honest today

- Validate & Browse (`/api/dataset/browse`) — real filesystem scan
- Local ingest (non-stub) — real frames → manifest
- Tracking / quality math / launch thresholds / autograder heuristics / YOLO train+infer — real when inputs and deps exist
- Catalog preprocess messaging partially hardened (`catalog_only`, `browse_hint`) but KPIs still read as Loaded

---

## Remediation shipped in `feat/execution-integrity`

**Implemented:** Execution ledger + APIs; real Dataset Load & Preprocess; Auto-Label / Infer / Train / Grader evidence wiring; YAML semantic validation; script verification; backend health; Strict Mode; Evidence Cards + Global Execution Console; failure + deterministic tests; remaining stages show honest `NOT_EXECUTED` / unverified stubs.

**Deferred (honest stubs only):** Full MITL/HF ingest, real SSAM lake, real MCP process probes, SAM GPU path when checkpoint absent, full 15-stage process verification beyond Dataset/Auto-Label/Train/Grader.