| # Canonical Statistical Chart & Benchmark Pipeline (`v2.1.0`) |
|
|
| Every benchmark figure and headline accuracy table across the **Armiger** and **Armiger Ultra** repositories is generated deterministically from anonymized evaluation vectors (`benchmarks/canonical_eval_vectors.json`) and locked by a cryptographic provenance manifest (`benchmarks/CHARTS_MANIFEST.json`). |
|
|
| --- |
|
|
| ## 1. Pipeline Architecture & Artifacts |
|
|
| | Artifact | SHA-256 Digest | Description | |
| | :--- | :--- | :--- | |
| | `benchmarks/canonical_eval_vectors.json` | `058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a` | Plain-text JSON containing `148` 1D numeric `int16`/`float32` prediction and ground-truth label arrays across all 5 evaluation splits (`N=10,262`). Contains **zero** strings, release titles, file paths, or infohashes. | |
| | `benchmarks/generate_canonical_charts.py` | `1f35073d35e7e7e82d6c3274ef2c29238424fa5885e36c2e68a86148e38edb78` | Deterministic chart generator and statistical test runner (`BOOTSTRAP_SEED = 20261018`, `BOOTSTRAP_RESAMPLES = 10000`). | |
| | `benchmarks/CHARTS_MANIFEST.json` | *(Tracked by `test_chart_integrity.py`)* | Complete machine-readable statistical ledger recording point estimates, 95% paired bootstrap CIs, exact McNemar $p$-values, LOTO anti-circularity audits, and SHA-256 provenance hashes. | |
| | `assets/armiger_vs_llms_overall_and_6dims.png` | `0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62` | 6-dimension accuracy & composite mean comparison against frontier cloud LLMs and local GPU LLMs on identical evaluation cohorts. | |
| | `assets/armiger_vs_llms_cost_comparison.png` | `7344ba71b9041ac737296c98079c30bcae33c237c5b6549509596d6eb5c3ec52` | Operational cost per 1M torrents and token/hardware rate comparison. | |
| | `assets/armiger_vs_torrent_classifiers_6dims.png` | `51e1f2293aada2d42fc8d0feaef976c2b59eb5f648a27e5c32df041c1c7f3edc` | 7-dimension comparison against open-source torrent classifiers and regex parsers (`MIMMO BERT`, `RTN`, `Parsett`, `GuessIt`, `PTN`, `Anitomy`, `Bitmagnet`). | |
| | `assets/armiger_categorical_breakdown.png` | `1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98` | Per-category recall breakdown across all 10 media & software categories on `oct15_confirmation` (`N=2,835`) and `beyond_confirmation` (`N=1,025`). | |
|
|
| --- |
|
|
| ## 2. Four Machine-Enforced Statistical Invariants |
|
|
| `benchmarks/test_chart_integrity.py` enforces four strict invariants on every CI run: |
|
|
| ### Invariant 1 — Identical Denominator Lock (`assert_identical_denominator`) |
| Every model compared on a benchmark panel must be evaluated on the **exact same item mask and denominator $N$**. Dropping abstentions (`unknown`), mixing evaluation splits, or evaluating on a subset raises an immediate `AssertionError`: |
| - `oct15_confirmation` 11-Class Category: $N = 2,835$ |
| - `beyond_confirmation` 11-Class Category & Catalog Matching: $N = 1,025$ |
| - `oct15_confirmation` Movie-vs-TV & Catalog Matching: $N = 1,296$ |
| - `oct15_confirmation` Binary English-vs-Foreign Audio: $N = 545$ (`beyond_confirmation` strict audio: $N = 229$) |
| - `oct15_confirmation` Anime Detection: $N = 507$ (`beyond_confirmation` valid media: $N = 636$) |
| - `oct15_confirmation` Junk / Malware Gate: $N = 2,170$ (`beyond_confirmation`: $N = 977$) |
| - `Chart 1 Panel 2F` Swarm Liveliness AUROC: $N = 757$ Public-Tracker Cohort across all 6 compared systems (`Chart 3 Panel 6B`: $N = 10,278$ Multi-Source Corpus across all 3 compared methods) |
|
|
| ### Invariant 2 — Anti-Circularity Lock (`assert_non_circular_teacher_eval`) |
| On `beyond_confirmation` ($N = 1,025$), where multi-teacher consensus contributed to reference curation, teacher models (`OpenAI GPT (Fast Tier)` and `Google Gemini Flash`) are strictly evaluated against **Leave-One-Teacher-Out (LOTO)** reference keys (`self_vote_weight = 0.0`): |
|
|
| | Teacher Model | Non-Circular LOTO Accuracy (`self_vote_weight = 0.0`) | Circular Self-Voted Accuracy | Removed Circularity Inflation | |
| | :--- | :---: | :---: | :---: | |
| | `OpenAI GPT (Fast Tier)` | **`95.12%`** | `96.59%` | `+1.46 pp` | |
| | `Google Gemini Flash` | **`93.80%`** | `95.61%` | `+1.80 pp` | |
|
|
| Scoring any teacher model with `self_vote_weight > 0.0` or `eval_mode != "leave_one_teacher_out"` raises an `AssertionError`. |
|
|
| ### Invariant 3 — Paired Bootstrap 95% CIs, Hájek/DeLong AUROC CIs & Exact McNemar $p$-Values (`compute_paired_panel_stats`) |
| All accuracy uncertainty intervals and significance tests are paired by item index across `10,000` bootstrap resamples (`seed = 20261018`) using integer-scaled `float64` dot products (eliminating BLAS thread non-determinism across macOS ARM64 and Linux x86_64), two-sample Hájek/DeLong rank-placement variance for AUROC panels, and exact binomial `math.comb` McNemar two-sided $p$-values. |
|
|
| ### Invariant 4 — Cryptographic PNG Hygiene (`verify_png_chunks_clean`) |
| Every generated PNG is stripped of non-deterministic metadata (`tEXt`, `iTXt`, `zTXt`, `tIME`, `pHYs`) so only `IHDR`, `IDAT`, and `IEND` chunks remain, guaranteeing byte-exact reproducibility. |
| |
| --- |
| |
| ## 3. Running Verification Locally |
| |
| ```bash |
| pytest benchmarks/test_chart_integrity.py -v |
| ``` |
| |
