Standardize Chart 2/3 labels and Per-Category Recall table/caption parity
Browse filesRemove 'Cheap' prefix in Chart 2, replace tc2000 with zero_cue_2000 in Chart 3 Panel 4 and tables, and sync SHA-256 provenance.
- README.md +15 -15
- assets/armiger_vs_llms_cost_comparison.png +2 -2
- assets/armiger_vs_torrent_classifiers_6dims.png +2 -2
- benchmarks/CHARTS_MANIFEST.json +13 -13
- benchmarks/generate_canonical_charts.py +7 -7
- benchmarks/test_chart_integrity.py +1 -1
- docs/CHART_PIPELINE.md +3 -3
- docs/base-model.md +1 -1
README.md
CHANGED
|
@@ -68,7 +68,7 @@ Armiger and Armiger Ultra ship with turnkey HTTP services, Docker Compose stacks
|
|
| 68 |
|
| 69 |
- **[`DH-Waypoint` Integration (`examples/dh-waypoint/`)](examples/dh-waypoint)** — Drop-in `processorapi/v1` service (`armiger_waypoint_processor.py`) and plugin manifest (`07-armiger-ultra.plugin.yml`) implementing batch classification (`dh-waypoint.processor.v1`), pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop telemetry for [`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint).
|
| 70 |
- **[`Bitmagnet` Integration (`examples/bitmagnet/`)](examples/bitmagnet)** — Pre-ingest / post-crawl batch classifier and malware/honeypot pruner (`bitmagnet_armiger_bridge.py`) that upgrades upstream Bitmagnet from `35.17%` rule accuracy to **`95.49%` 11-class neural accuracy** and automatically prunes disguised `.mkv.exe`/`.scr`/`.lnk` malware traps via GraphQL `torrent.delete`.
|
| 71 |
-
- **[Torznab Proxy & Media Manager Webhook Suite (`examples/arr-stack/`)](examples/arr-stack)** — Pre-grab Torznab XML filtering proxy (`armiger_torznab_proxy.py`) between `Prowlarr` / `Bitmagnet` / `DH-Waypoint` and `Sonarr`, `Radarr`, `Lidarr`,
|
| 72 |
|
| 73 |
See the **[Complete Ecosystem Integration Guide (`docs/INTEGRATION_GUIDE.md`)](docs/INTEGRATION_GUIDE.md)** for architecture diagrams, Docker Compose files, and step-by-step setup instructions.
|
| 74 |
|
|
@@ -92,13 +92,13 @@ All metrics below are reproducibly computed using `armiger_inference.py` against
|
|
| 92 |
- **`zero_cue_2000` (`N=2,000` specialist zero-cue anime/video):** **`87.35%`** Standalone Neural INT8 (**`91.50%`** Fair Blind Hybrid, `+4.15 pp` from `BLAKE2b-64` catalog + structural rules)
|
| 93 |
- **`multilang_3000` (`N=3,000`):** **`97.37%`** Standalone Neural INT8 (**`97.50%`** FP32 / **`98.47%`** Fair Blind Hybrid)
|
| 94 |
- **5-Split Weighted Mean (`N=10,262`):** **`93.95%`** Standalone Neural INT8 (**`94.00%`** FP32) $\rightarrow$ **`95.39%`** INT8 / **`95.42%`** FP32 Fair Blind Hybrid
|
| 95 |
-
2. **`adult` (Safety Gate):** `non_adult` vs. `adult` (**`98.53%`** on `oct15_confirmation` `2,740/2,781`; **`98.21%`** on `fresh` `N=1,393`; **`98.62%`** on `beyond` `N=1,016`; **`99.90%`** on `
|
| 96 |
-
3. **`junk` (Malware / Fake-Codec / Archive-Trap Filter):** `clean` vs. `junk` (**`100.00%`** on `oct15_confirmation` `2,170/2,170`; **`100.00%`** on `fresh` `1,291/1,291`; **`99.90%`** on `beyond` `976/977`; **`100.00%`** on `
|
| 97 |
4. **`anime` (Anime & Dub Routing):** `non_anime` vs. `anime` (**`99.61%`** on `oct15_confirmation` `505/507`; **`99.60%`** on `fresh` `252/253`; **`98.90%`** on `beyond` `629/636`; **`95.80%`** on `zero_cue_2000` `1,914/1,998`)
|
| 98 |
-
5. **`lang` (Audio/Release Language):** `english`, `non_english`, `multi_or_unknown` (**`96.88%`** Standalone Neural / **`97.80%`** with `AudioLanguageGuard` on `oct15_confirmation` binary English-vs-Foreign `N=545`; **`92.75%`** 3-class `N=772`; **`91.47%`** on `
|
| 99 |
-
6. **`title_language` (Title Script/Language):** `english_title` vs. `non_english_title` (**`93.70%`** on `oct15_confirmation` `N=969`; **`98.51%`** on `beyond_confirmation` `N=604`; **`97.80%`** on `
|
| 100 |
-
7. **`english_subs` (Subtitle Availability):** `no_eng_subs`, `has_eng_subs`, `unknown` (**`93.04%`** on `stop` `N=2,171`; **`87.17%`** on `
|
| 101 |
-
8. **`owner_scope` (Catalog Policy Gate):** `allowed_type` vs. `excluded` (**`98.65%`** on `oct15_confirmation` `2,633/2,669`; **`99.80%`** on `
|
| 102 |
|
| 103 |
---
|
| 104 |
|
|
@@ -134,22 +134,22 @@ All metrics below are reproducibly computed using `armiger_inference.py` against
|
|
| 134 |
|
| 135 |

|
| 136 |
|
| 137 |
-
| System / Model | Footprint | 11-Class Category (`oct15` `N=2,835`) | Movie vs. TV (`N=1,296`) | Catalog Matching (`N=1,296`) | Audio Language (`N=545`) | Anime & Dub (`N=507` / `
|
| 138 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 139 |
-
| **[Armiger Ultra (Fair Hybrid)](https://huggingface.co/NorviTech/armiger-ultra)** | `32.3 MB` (`+26 MB` cat.) | **`95.49%`** (`98.47%` mlang) | **`97.30%`** | **`96.53%`** | **`97.80%`** | **`99.61%`** (`95.80%`
|
| 140 |
-
| **Armiger (8.3 MB INT8 + Plugins)** | **`8.3 MB`** | **`95.49%`** (`95.06%` Raw) | **`97.30%`** (`96.60%` Raw) | **`96.53%`** (`95.83%` Raw) | **`97.80%`** (`96.88%` Raw) | **`99.61%`** (`95.80%`
|
| 141 |
| **MIMMO BERT (360M)** | `~1,400 MB` | `72.24%` | `76.00%` (`93.61%` 4-Med) | `60.26%` | N/A | N/A | `91.05%` | N/A | N/A |
|
| 142 |
| **MIMMO Cascade** | `~1,400 MB` | `71.32%` | `76.08%` (`93.97%` 4-Med) | `60.34%` | N/A | N/A | `89.68%` | N/A | N/A |
|
| 143 |
-
| **RTN (`rank-torrent-name`)** | Code Only | `53.58%` | `81.64%` | `80.48%` | `93.76%` | `77.65%` | `79.00%` | `
|
| 144 |
| **PTT / Parsett** | Code Only | `53.58%` | `81.64%` | `80.48%` | `93.76%` | `77.65%` | `79.00%` | N/A | N/A |
|
| 145 |
| **GuessIt** | Code Only | `38.69%` | `84.65%` | `84.03%` | `94.86%` | `78.80%` | N/A | N/A | N/A |
|
| 146 |
| **PTN (`parse-torrent-title`)** | Code Only | `35.59%` | `77.85%` | `77.62%` | `93.21%` | `68.65%` | N/A | N/A | N/A |
|
| 147 |
| **Anitomy** | Code Only | `33.40%` | `73.07%` | `72.76%` | `90.46%` | `64.55%` | N/A | N/A | N/A |
|
| 148 |
-
| **Bitmagnet Built-In** | Code Only | `35.17%` | `43.21%` (`88.4%` ans.) | `43.21%` | `88.26%` | N/A | `76.73%` | `
|
| 149 |
|
| 150 |
-
### Per-Category
|
| 151 |
|
| 152 |
-
** — Drop-in `processorapi/v1` service (`armiger_waypoint_processor.py`) and plugin manifest (`07-armiger-ultra.plugin.yml`) implementing batch classification (`dh-waypoint.processor.v1`), pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop telemetry for [`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint).
|
| 70 |
- **[`Bitmagnet` Integration (`examples/bitmagnet/`)](examples/bitmagnet)** — Pre-ingest / post-crawl batch classifier and malware/honeypot pruner (`bitmagnet_armiger_bridge.py`) that upgrades upstream Bitmagnet from `35.17%` rule accuracy to **`95.49%` 11-class neural accuracy** and automatically prunes disguised `.mkv.exe`/`.scr`/`.lnk` malware traps via GraphQL `torrent.delete`.
|
| 71 |
+
- **[Torznab Proxy & Media Manager Webhook Suite (`examples/arr-stack/`)](examples/arr-stack)** — Pre-grab Torznab XML filtering proxy (`armiger_torznab_proxy.py`) between `Prowlarr` / `Bitmagnet` / `DH-Waypoint` and `Sonarr`, `Radarr`, `Lidarr`, and `Readarr`, paired with a post-import webhook & `ffprobe` ground-truth receiver (`armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier`.
|
| 72 |
|
| 73 |
See the **[Complete Ecosystem Integration Guide (`docs/INTEGRATION_GUIDE.md`)](docs/INTEGRATION_GUIDE.md)** for architecture diagrams, Docker Compose files, and step-by-step setup instructions.
|
| 74 |
|
|
|
|
| 92 |
- **`zero_cue_2000` (`N=2,000` specialist zero-cue anime/video):** **`87.35%`** Standalone Neural INT8 (**`91.50%`** Fair Blind Hybrid, `+4.15 pp` from `BLAKE2b-64` catalog + structural rules)
|
| 93 |
- **`multilang_3000` (`N=3,000`):** **`97.37%`** Standalone Neural INT8 (**`97.50%`** FP32 / **`98.47%`** Fair Blind Hybrid)
|
| 94 |
- **5-Split Weighted Mean (`N=10,262`):** **`93.95%`** Standalone Neural INT8 (**`94.00%`** FP32) $\rightarrow$ **`95.39%`** INT8 / **`95.42%`** FP32 Fair Blind Hybrid
|
| 95 |
+
2. **`adult` (Safety Gate):** `non_adult` vs. `adult` (**`98.53%`** on `oct15_confirmation` `2,740/2,781`; **`98.21%`** on `fresh` `N=1,393`; **`98.62%`** on `beyond` `N=1,016`; **`99.90%`** on `zero_cue_2000` `N=2,000`; **`100.00%`** on `multilang_3000` `N=3,000`)
|
| 96 |
+
3. **`junk` (Malware / Fake-Codec / Archive-Trap Filter):** `clean` vs. `junk` (**`100.00%`** on `oct15_confirmation` `2,170/2,170`; **`100.00%`** on `fresh` `1,291/1,291`; **`99.90%`** on `beyond` `976/977`; **`100.00%`** on `zero_cue_2000` & `multilang_3000`; **`7/7` (`100%`)** disguised executable malware traps blocked at `P(junk) > 0.999`)
|
| 97 |
4. **`anime` (Anime & Dub Routing):** `non_anime` vs. `anime` (**`99.61%`** on `oct15_confirmation` `505/507`; **`99.60%`** on `fresh` `252/253`; **`98.90%`** on `beyond` `629/636`; **`95.80%`** on `zero_cue_2000` `1,914/1,998`)
|
| 98 |
+
5. **`lang` (Audio/Release Language):** `english`, `non_english`, `multi_or_unknown` (**`96.88%`** Standalone Neural / **`97.80%`** with `AudioLanguageGuard` on `oct15_confirmation` binary English-vs-Foreign `N=545`; **`92.75%`** 3-class `N=772`; **`91.47%`** on `multilang_3000` `N=3,000`)
|
| 99 |
+
6. **`title_language` (Title Script/Language):** `english_title` vs. `non_english_title` (**`93.70%`** on `oct15_confirmation` `N=969`; **`98.51%`** on `beyond_confirmation` `N=604`; **`97.80%`** on `zero_cue_2000` `N=2,000`)
|
| 100 |
+
7. **`english_subs` (Subtitle Availability):** `no_eng_subs`, `has_eng_subs`, `unknown` (**`93.04%`** on `stop` `N=2,171`; **`87.17%`** on `multilang_3000` `N=3,000`)
|
| 101 |
+
8. **`owner_scope` (Catalog Policy Gate):** `allowed_type` vs. `excluded` (**`98.65%`** on `oct15_confirmation` `2,633/2,669`; **`99.80%`** on `zero_cue_2000` `N=2,000`; **`99.90%`** on `multilang_3000` `N=3,000`)
|
| 102 |
|
| 103 |
---
|
| 104 |
|
|
|
|
| 134 |
|
| 135 |

|
| 136 |
|
| 137 |
+
| System / Model | Footprint | 11-Class Category (`oct15` `N=2,835`) | Movie vs. TV (`N=1,296`) | Catalog Matching (`N=1,296`) | Audio Language (`N=545`) | Anime & Dub (`N=507` / `zero_cue_2000`) | Adult / NSFW Gate (`N=2,781`) | Junk / Malware (`N=2,170`) | Swarm Liveliness (AUROC) |
|
| 138 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 139 |
+
| **[Armiger Ultra (Fair Hybrid)](https://huggingface.co/NorviTech/armiger-ultra)** | `32.3 MB` (`+26 MB` cat.) | **`95.49%`** (`98.47%` mlang) | **`97.30%`** | **`96.53%`** | **`97.80%`** | **`99.61%`** (`95.80%` `zero_cue_2000`) | **`98.53%`** | **`100.00%`** (`99.90%` beyond) | **`96.83%`** Corpus / **`92.42%`** Pub |
|
| 140 |
+
| **Armiger (8.3 MB INT8 + Plugins)** | **`8.3 MB`** | **`95.49%`** (`95.06%` Raw) | **`97.30%`** (`96.60%` Raw) | **`96.53%`** (`95.83%` Raw) | **`97.80%`** (`96.88%` Raw) | **`99.61%`** (`95.80%` `zero_cue_2000`) | **`98.49%`** | **`100.00%`** (`99.90%` beyond) | **`92.42%`** Pub (`86.64%` Priors) |
|
| 141 |
| **MIMMO BERT (360M)** | `~1,400 MB` | `72.24%` | `76.00%` (`93.61%` 4-Med) | `60.26%` | N/A | N/A | `91.05%` | N/A | N/A |
|
| 142 |
| **MIMMO Cascade** | `~1,400 MB` | `71.32%` | `76.08%` (`93.97%` 4-Med) | `60.34%` | N/A | N/A | `89.68%` | N/A | N/A |
|
| 143 |
+
| **RTN (`rank-torrent-name`)** | Code Only | `53.58%` | `81.64%` | `80.48%` | `93.76%` | `77.65%` | `79.00%` | `49.29%` (`0%` rec., `98.57%` spec.) | N/A |
|
| 144 |
| **PTT / Parsett** | Code Only | `53.58%` | `81.64%` | `80.48%` | `93.76%` | `77.65%` | `79.00%` | N/A | N/A |
|
| 145 |
| **GuessIt** | Code Only | `38.69%` | `84.65%` | `84.03%` | `94.86%` | `78.80%` | N/A | N/A | N/A |
|
| 146 |
| **PTN (`parse-torrent-title`)** | Code Only | `35.59%` | `77.85%` | `77.62%` | `93.21%` | `68.65%` | N/A | N/A | N/A |
|
| 147 |
| **Anitomy** | Code Only | `33.40%` | `73.07%` | `72.76%` | `90.46%` | `64.55%` | N/A | N/A | N/A |
|
| 148 |
+
| **Bitmagnet Built-In** | Code Only | `35.17%` | `43.21%` (`88.4%` ans.) | `43.21%` | `88.26%` | N/A | `76.73%` | `50.00%` (`0%` rec., `100%` spec.) | N/A |
|
| 149 |
|
| 150 |
+
### Per-Category Recall Breakdown Across All 10 Media & Software Categories (Unmutated Keys)
|
| 151 |
|
| 152 |
+

|
| 153 |
|
| 154 |
| Content Category | `oct15` (`N=2,835`) Armiger Ultra (Fair Hybrid) | `oct15` (`N=2,835`) Armiger (INT8 + ZeroCue) | `oct15` MIMMO BERT (360M) | `oct15` Bitmagnet Built-In | `beyond` (`N=1,025`) Armiger Ultra (Fair Hybrid) | `beyond` (`N=1,025`) Armiger (INT8 + ZeroCue) | `beyond` OpenAI GPT (Fast Tier, LOTO)* | `beyond` Google Gemini Flash (LOTO)* |
|
| 155 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
|
|
|
| 196 |
1. **Identical Denominator Lock (`assert_identical_denominator`):** Every model compared on a benchmark bar shares the exact same item count $N$ and item mask (`oct15_confirmation` $N=2,835$; `beyond_confirmation` $N=1,025$; Movie-vs-TV $N=1,296$; Audio $N=545$; Anime $N=507$; Junk $N=2,170$).
|
| 197 |
2. **Non-Circular Teacher Evaluation Lock (`assert_non_circular_teacher_eval`):** Frontier teacher models (`OpenAI GPT (Fast Tier)`, `Google Gemini Flash`) are evaluated strictly via **Leave-One-Teacher-Out (LOTO)** reference keys (`self_vote_weight = 0.0`), removing $+1.47\text{ pp}$ and $+1.81\text{ pp}$ of circular self-vote inflation.
|
| 198 |
3. **Paired Bootstrap 95% CIs & Exact McNemar $p$-Values:** Computed using `10,000` paired resamples (`seed = 20261018`, integer-scaled `float64` dot products) and exact binomial `math.comb` McNemar tests (`p = 1.42e-11` vs. `Google Gemini Flash` LOTO; `p = 3.88e-24` vs. `Anthropic Claude Haiku 4.5`).
|
| 199 |
+
4. **Deterministic PNG Chunk Stripping:** Output PNGs contain only `IHDR`, `IDAT`, and `IEND` chunks with pinned SHA-256 digests (`0a0bc3a2...`, `7344ba71...`, `51e1f229...`, `1a77a643...`).
|
| 200 |
|
| 201 |
---
|
| 202 |
|
assets/armiger_vs_llms_cost_comparison.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/armiger_vs_torrent_classifiers_6dims.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
benchmarks/CHARTS_MANIFEST.json
CHANGED
|
@@ -36,10 +36,10 @@
|
|
| 36 |
],
|
| 37 |
"provenance_sha256": {
|
| 38 |
"canonical_eval_vectors.json": "058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a",
|
| 39 |
-
"generate_canonical_charts.py": "
|
| 40 |
"assets/armiger_vs_llms_overall_and_6dims.png": "0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62",
|
| 41 |
-
"assets/armiger_vs_llms_cost_comparison.png": "
|
| 42 |
-
"assets/armiger_vs_torrent_classifiers_6dims.png": "
|
| 43 |
"assets/armiger_categorical_breakdown.png": "1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98",
|
| 44 |
"validated_zero_cue_catalog.json": "a43193a7d9982f1b83e9e6968c47dbbe8d058b113af5020d6d5ea45ed2af7fcd",
|
| 45 |
"armiger_ultra.fp32.onnx": "24c65e1af1be68c923bfdf774afb862c54b66a30f40c4b65cf5961929f149d48",
|
|
@@ -529,13 +529,13 @@
|
|
| 529 |
}
|
| 530 |
},
|
| 531 |
"armiger_vs_llms_cost_comparison.png": {
|
| 532 |
-
"sha256": "
|
| 533 |
"models": [
|
| 534 |
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 535 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 536 |
"Local 8B Instruct LLM (FP16)",
|
| 537 |
"Google Gemini Flash",
|
| 538 |
-
"
|
| 539 |
"Anthropic Claude Haiku 4.5"
|
| 540 |
],
|
| 541 |
"batched_usd_per_1m": [
|
|
@@ -556,7 +556,7 @@
|
|
| 556 |
]
|
| 557 |
},
|
| 558 |
"armiger_vs_torrent_classifiers_6dims.png": {
|
| 559 |
-
"sha256": "
|
| 560 |
"panel_1_category_11class_oct15": {
|
| 561 |
"panel_name": "chart3_1_oct15_cat11",
|
| 562 |
"n": 2835,
|
|
@@ -997,8 +997,8 @@
|
|
| 997 |
}
|
| 998 |
}
|
| 999 |
},
|
| 1000 |
-
"
|
| 1001 |
-
"panel_name": "
|
| 1002 |
"n": 2000,
|
| 1003 |
"metric_type": "mean",
|
| 1004 |
"reference_model": "RTN (rank-torrent-name)",
|
|
@@ -1332,7 +1332,7 @@
|
|
| 1332 |
},
|
| 1333 |
"five_split_category_summary": {
|
| 1334 |
"oct15_confirmation": {
|
| 1335 |
-
"panel_name": "
|
| 1336 |
"n": 2835,
|
| 1337 |
"metric_type": "mean",
|
| 1338 |
"reference_model": "armiger_ultra_hyb",
|
|
@@ -1380,7 +1380,7 @@
|
|
| 1380 |
}
|
| 1381 |
},
|
| 1382 |
"fresh_confirmation": {
|
| 1383 |
-
"panel_name": "
|
| 1384 |
"n": 1402,
|
| 1385 |
"metric_type": "mean",
|
| 1386 |
"reference_model": "armiger_ultra_hyb",
|
|
@@ -1428,7 +1428,7 @@
|
|
| 1428 |
}
|
| 1429 |
},
|
| 1430 |
"beyond_confirmation": {
|
| 1431 |
-
"panel_name": "
|
| 1432 |
"n": 1025,
|
| 1433 |
"metric_type": "mean",
|
| 1434 |
"reference_model": "armiger_ultra_hyb",
|
|
@@ -1476,7 +1476,7 @@
|
|
| 1476 |
}
|
| 1477 |
},
|
| 1478 |
"zero_cue_2000": {
|
| 1479 |
-
"panel_name": "
|
| 1480 |
"n": 2000,
|
| 1481 |
"metric_type": "mean",
|
| 1482 |
"reference_model": "armiger_ultra_hyb",
|
|
@@ -1524,7 +1524,7 @@
|
|
| 1524 |
}
|
| 1525 |
},
|
| 1526 |
"multilang_3000": {
|
| 1527 |
-
"panel_name": "
|
| 1528 |
"n": 3000,
|
| 1529 |
"metric_type": "mean",
|
| 1530 |
"reference_model": "armiger_ultra_hyb",
|
|
|
|
| 36 |
],
|
| 37 |
"provenance_sha256": {
|
| 38 |
"canonical_eval_vectors.json": "058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a",
|
| 39 |
+
"generate_canonical_charts.py": "1f35073d35e7e7e82d6c3274ef2c29238424fa5885e36c2e68a86148e38edb78",
|
| 40 |
"assets/armiger_vs_llms_overall_and_6dims.png": "0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62",
|
| 41 |
+
"assets/armiger_vs_llms_cost_comparison.png": "7344ba71b9041ac737296c98079c30bcae33c237c5b6549509596d6eb5c3ec52",
|
| 42 |
+
"assets/armiger_vs_torrent_classifiers_6dims.png": "51e1f2293aada2d42fc8d0feaef976c2b59eb5f648a27e5c32df041c1c7f3edc",
|
| 43 |
"assets/armiger_categorical_breakdown.png": "1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98",
|
| 44 |
"validated_zero_cue_catalog.json": "a43193a7d9982f1b83e9e6968c47dbbe8d058b113af5020d6d5ea45ed2af7fcd",
|
| 45 |
"armiger_ultra.fp32.onnx": "24c65e1af1be68c923bfdf774afb862c54b66a30f40c4b65cf5961929f149d48",
|
|
|
|
| 529 |
}
|
| 530 |
},
|
| 531 |
"armiger_vs_llms_cost_comparison.png": {
|
| 532 |
+
"sha256": "7344ba71b9041ac737296c98079c30bcae33c237c5b6549509596d6eb5c3ec52",
|
| 533 |
"models": [
|
| 534 |
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 535 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 536 |
"Local 8B Instruct LLM (FP16)",
|
| 537 |
"Google Gemini Flash",
|
| 538 |
+
"OpenAI GPT (Fast Tier)",
|
| 539 |
"Anthropic Claude Haiku 4.5"
|
| 540 |
],
|
| 541 |
"batched_usd_per_1m": [
|
|
|
|
| 556 |
]
|
| 557 |
},
|
| 558 |
"armiger_vs_torrent_classifiers_6dims.png": {
|
| 559 |
+
"sha256": "51e1f2293aada2d42fc8d0feaef976c2b59eb5f648a27e5c32df041c1c7f3edc",
|
| 560 |
"panel_1_category_11class_oct15": {
|
| 561 |
"panel_name": "chart3_1_oct15_cat11",
|
| 562 |
"n": 2835,
|
|
|
|
| 997 |
}
|
| 998 |
}
|
| 999 |
},
|
| 1000 |
+
"panel_4_anime_dub_zero_cue_2000": {
|
| 1001 |
+
"panel_name": "chart3_4_zero_cue_2000_anime_dub",
|
| 1002 |
"n": 2000,
|
| 1003 |
"metric_type": "mean",
|
| 1004 |
"reference_model": "RTN (rank-torrent-name)",
|
|
|
|
| 1332 |
},
|
| 1333 |
"five_split_category_summary": {
|
| 1334 |
"oct15_confirmation": {
|
| 1335 |
+
"panel_name": "five_split_oct15_confirmation",
|
| 1336 |
"n": 2835,
|
| 1337 |
"metric_type": "mean",
|
| 1338 |
"reference_model": "armiger_ultra_hyb",
|
|
|
|
| 1380 |
}
|
| 1381 |
},
|
| 1382 |
"fresh_confirmation": {
|
| 1383 |
+
"panel_name": "five_split_fresh_confirmation",
|
| 1384 |
"n": 1402,
|
| 1385 |
"metric_type": "mean",
|
| 1386 |
"reference_model": "armiger_ultra_hyb",
|
|
|
|
| 1428 |
}
|
| 1429 |
},
|
| 1430 |
"beyond_confirmation": {
|
| 1431 |
+
"panel_name": "five_split_beyond_confirmation",
|
| 1432 |
"n": 1025,
|
| 1433 |
"metric_type": "mean",
|
| 1434 |
"reference_model": "armiger_ultra_hyb",
|
|
|
|
| 1476 |
}
|
| 1477 |
},
|
| 1478 |
"zero_cue_2000": {
|
| 1479 |
+
"panel_name": "five_split_zero_cue_2000",
|
| 1480 |
"n": 2000,
|
| 1481 |
"metric_type": "mean",
|
| 1482 |
"reference_model": "armiger_ultra_hyb",
|
|
|
|
| 1524 |
}
|
| 1525 |
},
|
| 1526 |
"multilang_3000": {
|
| 1527 |
+
"panel_name": "five_split_multilang_3000",
|
| 1528 |
"n": 3000,
|
| 1529 |
"metric_type": "mean",
|
| 1530 |
"reference_model": "armiger_ultra_hyb",
|
benchmarks/generate_canonical_charts.py
CHANGED
|
@@ -586,7 +586,7 @@ def build_all_charts_and_manifest(
|
|
| 586 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 587 |
"Local 8B Instruct LLM (FP16)",
|
| 588 |
"Google Gemini Flash",
|
| 589 |
-
"
|
| 590 |
"Anthropic Claude Haiku 4.5",
|
| 591 |
]
|
| 592 |
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
|
@@ -788,7 +788,7 @@ def build_all_charts_and_manifest(
|
|
| 788 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 789 |
"Local 8B Instruct LLM (FP16)",
|
| 790 |
"Google Gemini Flash",
|
| 791 |
-
"
|
| 792 |
"Anthropic Claude Haiku 4.5",
|
| 793 |
]
|
| 794 |
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
|
@@ -948,7 +948,7 @@ def build_all_charts_and_manifest(
|
|
| 948 |
|
| 949 |
y_tc2k_ani = d["tc2000_ani_y_true"]
|
| 950 |
c3_p4_tc2k_stats = compute_paired_panel_stats(
|
| 951 |
-
"
|
| 952 |
{
|
| 953 |
models_10[4]: (d["tc2000_ani_pred_rtn"] == y_tc2k_ani).astype(np.float32),
|
| 954 |
models_10[5]: (d["tc2000_ani_pred_ptt"] == y_tc2k_ani).astype(np.float32),
|
|
@@ -1177,7 +1177,7 @@ def build_all_charts_and_manifest(
|
|
| 1177 |
v = s_item["point_pct"]
|
| 1178 |
ax4.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
|
| 1179 |
draw_ci_whiskers(ax4, np.array([idx]), [s_item])
|
| 1180 |
-
ax4.text(max(v, s_item["ci95_hi"]) + 0.6, idx, f"{v:.2f}% (95.8%
|
| 1181 |
elif m in c3_p4_tc2k_stats["models"]:
|
| 1182 |
s_item = c3_p4_tc2k_stats["models"][m]
|
| 1183 |
v = s_item["point_pct"]
|
|
@@ -1190,7 +1190,7 @@ def build_all_charts_and_manifest(
|
|
| 1190 |
ax4.set_yticks(y10)
|
| 1191 |
ax4.set_yticklabels(models_10, fontsize=9.5, fontweight="bold", color=TEXT_MAIN)
|
| 1192 |
ax4.invert_yaxis()
|
| 1193 |
-
ax4.set_xlim(45,
|
| 1194 |
ax4.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1195 |
ax4.set_xlabel("99.61% on oct15_confirmation (505/507); 95.80% on zero_cue_2000 (N=2,000)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
|
| 1196 |
ax4.set_title("4. Anime & Anime Dub Detection (%)\n(oct15_confirmation N = 507 & zero_cue_2000 N = 2,000)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
|
|
@@ -1433,7 +1433,7 @@ def build_all_charts_and_manifest(
|
|
| 1433 |
]:
|
| 1434 |
y_sp = d[f"split5_{sp_code}_y_cat"]
|
| 1435 |
sp_stats = compute_paired_panel_stats(
|
| 1436 |
-
f"five_split_{
|
| 1437 |
{
|
| 1438 |
"armiger_ultra_hyb": (d[f"split5_{sp_code}_fp32_hyb_pred"] == y_sp).astype(np.float32),
|
| 1439 |
"armiger_ultra_raw": (d[f"split5_{sp_code}_fp32_raw_pred"] == y_sp).astype(np.float32),
|
|
@@ -1506,7 +1506,7 @@ def build_all_charts_and_manifest(
|
|
| 1506 |
"panel_2_catalog_matching_oct15": c3_p2_stats,
|
| 1507 |
"panel_3_audio_strict_oct15": c3_p3_stats,
|
| 1508 |
"panel_4_anime_oct15": c3_p4_oct_stats,
|
| 1509 |
-
"
|
| 1510 |
"panel_5_adult_oct15": c3_p5_stats,
|
| 1511 |
"panel_6a_junk_oct15": c3_p6a_stats,
|
| 1512 |
},
|
|
|
|
| 586 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 587 |
"Local 8B Instruct LLM (FP16)",
|
| 588 |
"Google Gemini Flash",
|
| 589 |
+
"OpenAI GPT (Fast Tier)",
|
| 590 |
"Anthropic Claude Haiku 4.5",
|
| 591 |
]
|
| 592 |
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
|
|
|
| 788 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 789 |
"Local 8B Instruct LLM (FP16)",
|
| 790 |
"Google Gemini Flash",
|
| 791 |
+
"OpenAI GPT (Fast Tier)",
|
| 792 |
"Anthropic Claude Haiku 4.5",
|
| 793 |
]
|
| 794 |
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
|
|
|
| 948 |
|
| 949 |
y_tc2k_ani = d["tc2000_ani_y_true"]
|
| 950 |
c3_p4_tc2k_stats = compute_paired_panel_stats(
|
| 951 |
+
"chart3_4_zero_cue_2000_anime_dub",
|
| 952 |
{
|
| 953 |
models_10[4]: (d["tc2000_ani_pred_rtn"] == y_tc2k_ani).astype(np.float32),
|
| 954 |
models_10[5]: (d["tc2000_ani_pred_ptt"] == y_tc2k_ani).astype(np.float32),
|
|
|
|
| 1177 |
v = s_item["point_pct"]
|
| 1178 |
ax4.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
|
| 1179 |
draw_ci_whiskers(ax4, np.array([idx]), [s_item])
|
| 1180 |
+
ax4.text(max(v, s_item["ci95_hi"]) + 0.6, idx, f"{v:.2f}% (95.8% zero_cue_2000)", va="center", ha="left", fontsize=8.4, fontweight="bold", color=TEXT_MAIN)
|
| 1181 |
elif m in c3_p4_tc2k_stats["models"]:
|
| 1182 |
s_item = c3_p4_tc2k_stats["models"][m]
|
| 1183 |
v = s_item["point_pct"]
|
|
|
|
| 1190 |
ax4.set_yticks(y10)
|
| 1191 |
ax4.set_yticklabels(models_10, fontsize=9.5, fontweight="bold", color=TEXT_MAIN)
|
| 1192 |
ax4.invert_yaxis()
|
| 1193 |
+
ax4.set_xlim(45, 128)
|
| 1194 |
ax4.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1195 |
ax4.set_xlabel("99.61% on oct15_confirmation (505/507); 95.80% on zero_cue_2000 (N=2,000)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
|
| 1196 |
ax4.set_title("4. Anime & Anime Dub Detection (%)\n(oct15_confirmation N = 507 & zero_cue_2000 N = 2,000)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
|
|
|
|
| 1433 |
]:
|
| 1434 |
y_sp = d[f"split5_{sp_code}_y_cat"]
|
| 1435 |
sp_stats = compute_paired_panel_stats(
|
| 1436 |
+
f"five_split_{sp_label}",
|
| 1437 |
{
|
| 1438 |
"armiger_ultra_hyb": (d[f"split5_{sp_code}_fp32_hyb_pred"] == y_sp).astype(np.float32),
|
| 1439 |
"armiger_ultra_raw": (d[f"split5_{sp_code}_fp32_raw_pred"] == y_sp).astype(np.float32),
|
|
|
|
| 1506 |
"panel_2_catalog_matching_oct15": c3_p2_stats,
|
| 1507 |
"panel_3_audio_strict_oct15": c3_p3_stats,
|
| 1508 |
"panel_4_anime_oct15": c3_p4_oct_stats,
|
| 1509 |
+
"panel_4_anime_dub_zero_cue_2000": c3_p4_tc2k_stats,
|
| 1510 |
"panel_5_adult_oct15": c3_p5_stats,
|
| 1511 |
"panel_6a_junk_oct15": c3_p6a_stats,
|
| 1512 |
},
|
benchmarks/test_chart_integrity.py
CHANGED
|
@@ -188,5 +188,5 @@ def test_readme_tables_match_charts_manifest():
|
|
| 188 |
assert required_token in text, f"Missing canonical metric {required_token} in {readme_path}"
|
| 189 |
|
| 190 |
# Stale pre-audit numbers that must never reappear in README
|
| 191 |
-
for stale_token in ("97.13%", "96.35%", "72.35%", "35.29%", "97.55%", "true" + "spec", "tru" + "spec"):
|
| 192 |
assert stale_token not in text, f"Stale metric {stale_token} found in {readme_path}"
|
|
|
|
| 188 |
assert required_token in text, f"Missing canonical metric {required_token} in {readme_path}"
|
| 189 |
|
| 190 |
# Stale pre-audit numbers that must never reappear in README
|
| 191 |
+
for stale_token in ("97.13%", "96.35%", "72.35%", "35.29%", "97.55%", "true" + "spec", "tru" + "spec", "tc" + "2000", "multilang" + "3000"):
|
| 192 |
assert stale_token not in text, f"Stale metric {stale_token} found in {readme_path}"
|
docs/CHART_PIPELINE.md
CHANGED
|
@@ -9,11 +9,11 @@ Every benchmark figure and headline accuracy table across the **Armiger** and **
|
|
| 9 |
| Artifact | SHA-256 Digest | Description |
|
| 10 |
| :--- | :--- | :--- |
|
| 11 |
| `benchmarks/canonical_eval_vectors.json` | `058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a` | Plain-text JSON containing `148` 1D numeric `int16`/`float32` prediction and ground-truth label arrays across all 5 evaluation splits (`N=10,262`). Contains **zero** strings, release titles, file paths, or infohashes. |
|
| 12 |
-
| `benchmarks/generate_canonical_charts.py` | `
|
| 13 |
| `benchmarks/CHARTS_MANIFEST.json` | *(Tracked by `test_chart_integrity.py`)* | Complete machine-readable statistical ledger recording point estimates, 95% paired bootstrap CIs, exact McNemar $p$-values, LOTO anti-circularity audits, and SHA-256 provenance hashes. |
|
| 14 |
| `assets/armiger_vs_llms_overall_and_6dims.png` | `0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62` | 6-dimension accuracy & composite mean comparison against frontier cloud LLMs and local GPU LLMs on identical evaluation cohorts. |
|
| 15 |
-
| `assets/armiger_vs_llms_cost_comparison.png` | `
|
| 16 |
-
| `assets/armiger_vs_torrent_classifiers_6dims.png` | `
|
| 17 |
| `assets/armiger_categorical_breakdown.png` | `1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98` | Per-category recall breakdown across all 10 media & software categories on `oct15_confirmation` (`N=2,835`) and `beyond_confirmation` (`N=1,025`). |
|
| 18 |
|
| 19 |
---
|
|
|
|
| 9 |
| Artifact | SHA-256 Digest | Description |
|
| 10 |
| :--- | :--- | :--- |
|
| 11 |
| `benchmarks/canonical_eval_vectors.json` | `058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a` | Plain-text JSON containing `148` 1D numeric `int16`/`float32` prediction and ground-truth label arrays across all 5 evaluation splits (`N=10,262`). Contains **zero** strings, release titles, file paths, or infohashes. |
|
| 12 |
+
| `benchmarks/generate_canonical_charts.py` | `1f35073d35e7e7e82d6c3274ef2c29238424fa5885e36c2e68a86148e38edb78` | Deterministic chart generator and statistical test runner (`BOOTSTRAP_SEED = 20261018`, `BOOTSTRAP_RESAMPLES = 10000`). |
|
| 13 |
| `benchmarks/CHARTS_MANIFEST.json` | *(Tracked by `test_chart_integrity.py`)* | Complete machine-readable statistical ledger recording point estimates, 95% paired bootstrap CIs, exact McNemar $p$-values, LOTO anti-circularity audits, and SHA-256 provenance hashes. |
|
| 14 |
| `assets/armiger_vs_llms_overall_and_6dims.png` | `0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62` | 6-dimension accuracy & composite mean comparison against frontier cloud LLMs and local GPU LLMs on identical evaluation cohorts. |
|
| 15 |
+
| `assets/armiger_vs_llms_cost_comparison.png` | `7344ba71b9041ac737296c98079c30bcae33c237c5b6549509596d6eb5c3ec52` | Operational cost per 1M torrents and token/hardware rate comparison. |
|
| 16 |
+
| `assets/armiger_vs_torrent_classifiers_6dims.png` | `51e1f2293aada2d42fc8d0feaef976c2b59eb5f648a27e5c32df041c1c7f3edc` | 7-dimension comparison against open-source torrent classifiers and regex parsers (`MIMMO BERT`, `RTN`, `Parsett`, `GuessIt`, `PTN`, `Anitomy`, `Bitmagnet`). |
|
| 17 |
| `assets/armiger_categorical_breakdown.png` | `1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98` | Per-category recall breakdown across all 10 media & software categories on `oct15_confirmation` (`N=2,835`) and `beyond_confirmation` (`N=1,025`). |
|
| 18 |
|
| 19 |
---
|
docs/base-model.md
CHANGED
|
@@ -21,7 +21,7 @@
|
|
| 21 |
|
| 22 |

|
| 23 |
|
| 24 |
-

|
| 23 |
|
| 24 |
+

|
| 25 |
|
| 26 |
---
|
| 27 |
|