# Benchmarks — current picture

Last rewritten: **2026-07-17** (four-core + Grapevine docs alignment).  
Raw dated snapshots: [`../archive/2026-07/`](../archive/2026-07/).

## Product stance (honest)

| Model | Role | Capability / notes | Demo priority |
|-------|------|--------------------|---------------|
| **Flagship** | General ops | Strong; historically best fleet API gauntlet vs stock `llama3.2:1b` | **Primary** |
| **Growth** | Domain / mid | Strong internal gate; weaker on verbose API format rubrics | **Primary** with Flagship |
| **Ascension** | Reasoning | Native DeepSeek think; **not** auto-smarter than Flagship | Secondary — task bakeoffs required |
| **Scout** | Edge / 0.5B | Mixed / fail-prone; invents specs in chat probes | **Do not lead demos** |
| **Grapevine** | Omni extension | Real vision/audio; slower; identity under Ollama still flaky | Multimodal preview only |

Scout remains in the factory and MIP can load it; treat as **experimental**, not a sales hero.

## Suites we actually run

| Suite | Script | Measures |
|-------|--------|----------|
| Capability gate | `fleet_benchmark.py` / capability eval | Strawberry r-count, arith, spelling, identity |
| Bakeoff | `fleet_bakeoff.py` | Fleet vs same-size stock instruct |
| API gauntlet | Atlas / Ollama API harness | Multi-question pass + verbosity |
| Model card suite | `fleet_model_card_suite.py` | Broader Ollama categories |
| Grapevine vs Flagship | `benchmark_grapevine_vs_flagship.py` | Modality + smoke quality |
| MIP smoke | load + generate + forge | Viewer still produces trail + solid |

Threshold helpers: `fleet/public_benchmark_thresholds.yaml`.

## Snapshot A — internal capability (2026-07-13, thinking on)

From archived `BENCHMARK_REPORT.md`:

| Model | Pass | Gate | Notes |
|-------|------|------|-------|
| Scout | 6/7 (86%) | FAIL | Missed average_speed extraction |
| Growth | 7/7 (100%) | PASS | All tests passed |

## Snapshot B — combined core API gauntlet (post catch-up)

From archived `BENCHMARK_API_COMPARISON.md` (`combined-core-gauntlet`, 12 questions):

| Rank | Model | Pass | Accuracy |
|------|-------|------|----------|
| 1 | **arriella-flagship** | 10/12 | 83% |
| 2 | llama3.2:1b (stock) | 7/12 | 58% |
| 3 | arriella-scout | 5/12 | 42% |
| 4 | arriella-growth | 2/12 | 17% |

Flagship wins the head-to-head narrative. Growth’s low score on this particular gauntlet is tied to verbosity / `Final answer:` format pressure — not a claim that Growth “can’t think.” Prefer gate + bakeoff alongside API numbers.

## Snapshot C — Grapevine vs Flagship (2026-07-16 smoke)

See [`../../grapevine.md`](../../grapevine.md) and `fleet/inkling/grapevine_vs_flagship_benchmark.json`. Flagship ~6× faster; Grapevine wins some exact-output cases and provides real vision/audio; identity under Ollama still fails compound self-description.

## Chat probe notes (2026-07-17)

Informal `ollama run` identity/role probe across all five tags: Scout and Flagship still invent or garble provenance; Growth was terse/confused on architecture; Ascension answered briefly with think channel; Grapevine correctly named Infinidev/devs/role but contradicted its own vision capability. Treat as qualitative — re-run gates before Hub claims.

## Older FAIL reports

`VALIDATION_REPORT.md` / early context tests (Jul 11) are **archived**. Do not paste them as current quality without re-running.

## How to refresh numbers

```powershell
.\.venv\Scripts\python.exe scripts\fleet_benchmark.py
.\.venv\Scripts\python.exe scripts\fleet_bakeoff.py
.\.venv\Scripts\python.exe scripts\fleet_validate_all.py
.\.venv\Scripts\python.exe scripts\benchmark_grapevine_vs_flagship.py
```

Paste new tables into this file (or a dated file under `docs/benchmarks/`) and move superseded markdown to `docs/archive/`.