spencernorton commited on
Commit
8fc4339
·
verified ·
1 Parent(s): 5089202

Align model card, NOTICE, config digests, and documentation with v2.1.0 audit

Browse files
LICENSE ADDED
@@ -0,0 +1,190 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ Copyright 2026 NorviTech / Spencer Norton
179
+
180
+ Licensed under the Apache License, Version 2.0 (the "License");
181
+ you may not use this file except in compliance with the License.
182
+ You may obtain a copy of the License at
183
+
184
+ http://www.apache.org/licenses/LICENSE-2.0
185
+
186
+ Unless required by applicable law or agreed to in writing, software
187
+ distributed under the License is distributed on an "AS IS" BASIS,
188
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
189
+ See the License for the specific language governing permissions and
190
+ limitations under the License.
NOTICE ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Armiger & Armiger Ultra
2
+ Copyright 2026 NorviTech / Spencer Norton
3
+
4
+ This product includes software, neural network weights, calibration parameters,
5
+ and domain-separated hash lookup tables developed by NorviTech and released
6
+ under the Apache License, Version 2.0.
7
+
8
+ ================================================================================
9
+ Creative Commons & Public-Domain Example Media Attributions
10
+ ================================================================================
11
+ Media titles and filenames used in documentation examples and unit test fixtures
12
+ in this repository reference open-source, Creative Commons, or public-domain
13
+ works:
14
+
15
+ - "Big Buck Bunny" (2008), "Sintel" (2010), "Tears of Steel" (2012),
16
+ "Cosmos Laundromat" (2015), "Caminandes" (2013-2016), "Spring" (2019),
17
+ "Coffee Run" (2020), "Charge" (2022), "Wing It!" (2023), and
18
+ "Elephants Dream" (2006) are Open Movies produced by the Blender Foundation
19
+ (https://studio.blender.org/films/), licensed under Creative Commons
20
+ Attribution (CC-BY 3.0 / CC-BY 4.0).
21
+ - "Le Voyage dans la Lune" (1902) and "Les Quatre Cents Farces du Diable" (1906)
22
+ by Georges Melies are in the public domain.
23
+ - "Pepper&Carrot" is an open-source webcomic by David Revoy
24
+ (https://www.peppercarrot.com/), licensed under Creative Commons Attribution
25
+ 4.0 International (CC-BY 4.0).
26
+ - "Open Goldberg Variations" (BWV 988) by J.S. Bach, performed by Kimiko Ishizaka,
27
+ is released into the public domain under Creative Commons Zero (CC0 1.0).
28
+ - "Pride and Prejudice" by Jane Austen (1813) is in the public domain worldwide;
29
+ LibriVox recordings (https://librivox.org/) are released into the public domain
30
+ in the United States (verify local copyright duration outside the U.S.).
31
+
32
+ ================================================================================
33
+ Domain-Separated Hash Representation & Non-Affiliation Notice
34
+ ================================================================================
35
+ - The Zero-Cue Title Catalog (`validated_zero_cue_catalog.json`) and release-tag
36
+ feature extractors store lookup keys as domain-separated 64-bit BLAKE2b digests
37
+ (`person=b"armiger-zc-v2"` / 8-byte hex digests) mapped to integer catalog IDs
38
+ and coarse category/language codes. No plaintext media titles or release-group
39
+ names are redistributed in the catalog file.
40
+ - BitTorrent, BEP-5, BEP-9, BEP-10, BEP-15, BEP-44, BEP-46, and BEP-51 are open
41
+ protocol specifications. All third-party product names, open-source project
42
+ names, and model names (including OpenAI, GPT, Anthropic, Claude, Google,
43
+ Gemini, GuessIt, RTN/Parsett, PTN, Anitomy, MIMMO, Bitmagnet, ONNX Runtime,
44
+ and PyTorch) referenced in comparative benchmarks or compatibility guides
45
+ remain the property of their respective owners. Use of these names is strictly
46
+ nominative for comparative technical evaluation or interoperability description
47
+ and implies no endorsement, sponsorship, or affiliation.
README.md CHANGED
@@ -29,7 +29,7 @@ pipeline_tag: other
29
  <a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Compatible with Armiger Ultra" src="https://img.shields.io/badge/Bundled_with-Armiger_Ultra-FD8024.svg"></a>
30
  <a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
31
  <img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
32
- <img alt="Tests Passing" src="https://img.shields.io/badge/tests-37%2F37%20passing-brightgreen.svg">
33
  </p>
34
 
35
  ---
@@ -97,7 +97,7 @@ sync_engine = ArmigerDHTDatabaseSync(
97
  # 2. Ingest candidate record with CSAM & safety checks
98
  ok, msg, record = sync_engine.ingest_candidate_record(
99
  infohash="0123456789abcdef0123456789abcdef01234567",
100
- title="Spring.Complete.1080p.BluRay.x265",
101
  category="anime",
102
  language="ja",
103
  quality_tier="hd_1080p",
@@ -126,7 +126,7 @@ print("Tombstone purge executed. Purged records:", purged)
126
 
127
  ## Test Suite Execution
128
 
129
- Run the comprehensive 37-point unit + live 12-node UDP wire test suite (`test_dht_db_sync.py` + `test_bep44_udp_wire_e2e.py`):
130
 
131
  ```bash
132
  python3 test_dht_db_sync.py -v
@@ -167,53 +167,13 @@ Ran 27 tests in 0.011s - OK
167
 
168
  ---
169
 
170
- ## Part of the Armiger Modular Plugin Suite
171
 
172
- | Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
173
- | :--- | :---: | :---: | :--- |
174
- | **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Warehouse AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
175
- | **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
176
- | **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
177
- | **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
178
- | **[`NorviTech/armiger-calibration-dht`](https://huggingface.co/NorviTech/armiger-calibration-dht)** | **Yes** (`calibration.json`, `calibration_and_dht.py`, `armiger_dht_rsi.py`) | `181 KB` | **Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine:** Per-head temperature scaling ($T_h$), Mondrian safety gates (**`100.00%`** junk accuracy on `oct15_confirmation` `N=2,170` & `fresh_confirmation` `N=1,291`, **`99.90%`** on `beyond_confirmation` `N=977`), Hybrid `Ed25519`+`WOTS+-SHAKE256` signed `BEP-44`/`BEP-46` DHT hot-reload, Hybrid `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, distinct-ASN quorum, strictly **opt-in-by-default (`opt_in_sharing=False`)** anonymous sharing via `AnonymityGuard` (`500/500` leak block rate, ephemeral `Ed25519` keys, coarse UTC hour bucketing, DP-clipped deltas), and 500-canary regression gate. |
179
- | **[`NorviTech/armiger-dht-db-sync`](https://huggingface.co/NorviTech/armiger-dht-db-sync)** | **Yes** (`csam_guard.py`, `armiger_dht_db_sync.py`) | `155 KB` | **Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync:** Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded `SHA-256` tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off `xxx` isolation, `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, binary Merkle proofs, strictly **opt-in-by-default (`opt_in_sharing=False`)** `BLAKE2b-64` one-way catalog hashing via `AnonymityGuard`, and $K$-of-$N$ distinct-ASN Sybil quorum. |
180
-
181
- ---
182
-
183
- ## Canonical Statistical Chart Audit & Cryptographic Provenance (`v2.1.0`)
184
-
185
- All benchmark charts and headline metrics across the Armiger `v2.1.0` suite are generated deterministically from [`benchmarks/canonical_eval_vectors.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/canonical_eval_vectors.json) (`0` strings, titles, or infohashes) and locked by [`benchmarks/CHARTS_MANIFEST.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/CHARTS_MANIFEST.json) under four machine-enforced invariants (`pytest benchmarks/test_chart_integrity.py`):
186
-
187
- 1. **Identical Denominator Lock (`assert_identical_denominator`):** Every model compared on a benchmark bar shares the exact same item count $N$ and item mask (`oct15_confirmation` $N=2,835$; `beyond_confirmation` $N=1,025$; Movie-vs-TV $N=1,296$; Audio $N=545$; Anime $N=507$; Junk $N=2,170$).
188
- 2. **Non-Circular Teacher Evaluation Lock (`assert_non_circular_teacher_eval`):** Frontier teacher models (`gpt-6-luna`, `gemini-3.8-flash-low`) are evaluated strictly via **Leave-One-Teacher-Out (LOTO)** reference keys (`self_vote_weight = 0.0`), removing $+1.47\text{ pp}$ and $+1.81\text{ pp}$ of circular self-vote inflation.
189
- 3. **Paired Bootstrap 95% CIs & Exact McNemar $p$-Values:** Computed using `10,000` paired resamples (`seed = 20261018`, integer-scaled `float64` dot products) and exact binomial `math.comb` McNemar tests (`p = 1.42e-11` vs. `gemini-3.8-flash-low` LOTO; `p = 3.88e-24` vs. `claude-haiku-4-5`).
190
- 4. **Deterministic PNG Chunk Stripping:** Output PNGs contain only `IHDR`, `IDAT`, and `IEND` chunks with pinned SHA-256 digests (`2367dbf6...`, `ff33da1a...`, `a4ebfcb1...`, `b137523c...`).
191
-
192
- ---
193
-
194
- ## Privacy & Anonymous Opt-In-By-Default DHT Sharing (`opt_in_sharing=False`)
195
-
196
- Distributed DHT intelligence sharing (`NorviTech/armiger-calibration-dht` and `NorviTech/armiger-dht-db-sync`) is **strictly opt-in by default** and **100% anonymous by construction** when enabled (see [`docs/ANONYMOUS_OPT_IN_DHT.md`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/docs/ANONYMOUS_OPT_IN_DHT.md) and [`benchmarks/anonymous_opt_in_proof_report.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/anonymous_opt_in_proof_report.json)):
197
-
198
- | Guarantee | Default State | Wire-Level & Mathematical Enforcement |
199
- | :--- | :--- | :--- |
200
- | **Opt-In by Default** | `opt_in_sharing=False` (`ARMIGER_DHT_SHARE_OPT_IN="0"`) | `0` outbound DHT packets are emitted unless the operator explicitly opts in (`opt_in_sharing=True` and `ARMIGER_DHT_SHARE_OPT_IN="1"`). |
201
- | **`AnonymityGuard` Wire Firewall** | Always active (`500/500` canary block rate) | Inspects every outbound payload before signing; blocks `100.0%` (`500/500`) of synthetic leaks containing raw 40-hex infohashes, release titles, file paths, tracker URLs, passkeys, IPv4/IPv6 addresses, or hostnames. |
202
- | **Ephemeral `Ed25519` Keys** | Single-use keypair per announcement | Rotates a fresh 32-byte `Ed25519` keypair and `ephemeral_node_id` on every publish so contributions are unlinkable across time. |
203
- | **Coarse UTC Hour Timestamps** | `epoch_ts = (utc_now // 3600) * 3600` | Quantizes all outbound timestamps to 1-hour UTC boundaries (`3600s`), eliminating clock-skew and grab-timing correlation. |
204
- | **DP-Clipped Weight Deltas** | `clip_norm = 1.0`, `noise_multiplier = 0.05` | Clips calibration/adapter update vectors in $L_2$ norm (`<= 1.0`) and adds calibrated Gaussian noise ($\sigma = 0.05$) before hybrid `X25519` + `ML-KEM-768` + `ChaCha20-Poly1305` encryption. |
205
- | **`BLAKE2b-64` One-Way Catalog Hashes** | 16-hex-char (`64-bit`) digest | Shared catalog entries store only domain-separated `BLAKE2b-64` title hashes mapped to integer catalog IDs—never raw torrent names or infohashes. |
206
-
207
- ---
208
 
209
  ## Intended Use & Legal Compliance
210
 
211
- `Armiger`, `Armiger Ultra`, and all bundled plugins are open-source machine learning and cryptographic metadata-classification utilities designed strictly for **lawful use cases**, including:
212
- - Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
213
- - Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (`.mkv.exe`, `.scr`, `.vbs`) in peer-to-peer and decentralized storage research.
214
- - Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.
215
-
216
- **NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works.** Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.
217
 
218
  ---
219
 
 
29
  <a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Compatible with Armiger Ultra" src="https://img.shields.io/badge/Bundled_with-Armiger_Ultra-FD8024.svg"></a>
30
  <a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
31
  <img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
32
+ <img alt="Tests Passing" src="https://img.shields.io/badge/tests-27%2F27%20passing-brightgreen.svg">
33
  </p>
34
 
35
  ---
 
97
  # 2. Ingest candidate record with CSAM & safety checks
98
  ok, msg, record = sync_engine.ingest_candidate_record(
99
  infohash="0123456789abcdef0123456789abcdef01234567",
100
+ title="Spring.2019.1080p.WEBM.AV1-OPEN",
101
  category="anime",
102
  language="ja",
103
  quality_tier="hd_1080p",
 
126
 
127
  ## Test Suite Execution
128
 
129
+ Run the comprehensive 27-point test suite:
130
 
131
  ```bash
132
  python3 test_dht_db_sync.py -v
 
167
 
168
  ---
169
 
170
+ ## Part of the Armiger Modular Suite
171
 
172
+ This plugin is bundled inside **[`NorviTech/armiger-ultra`](https://huggingface.co/NorviTech/armiger-ultra)** alongside all 6 official plugins (`armiger-swarm-survival`, `armiger-swarm-priors`, `armiger-zero-cue-catalog`, `armiger-audio-guard`, `armiger-calibration-dht`, and `armiger-dht-db-sync`) and is compatible with **[`NorviTech/armiger`](https://huggingface.co/NorviTech/armiger)** (`8.3 MB` CPU INT8).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
173
 
174
  ## Intended Use & Legal Compliance
175
 
176
+ Open-source machine learning and cryptographic metadata-classification utility designed strictly for **lawful use cases** (indexing licensed, public-domain, Creative Commons, and open-source media/software; malware & fake-seeder honeypot filtering; and CSAM/abuse pre-ingest blocking). **NorviTech does not condone, encourage, or support copyright infringement.** See [`LEGAL.md`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/LEGAL.md) and [`NOTICE`](NOTICE).
 
 
 
 
 
177
 
178
  ---
179
 
armiger_dht_db_sync.py CHANGED
@@ -219,6 +219,10 @@ class QuorumAttestation:
219
  "timestamp": int(self.timestamp),
220
  }
221
 
 
 
 
 
222
 
223
  @dataclasses.dataclass(frozen=True)
224
  class DHTShardManifest:
@@ -1090,9 +1094,15 @@ class QuorumVerifier:
1090
  if isinstance(att.asn, bool) or not isinstance(att.asn, int) or att.asn <= 0:
1091
  continue
1092
 
 
 
1093
  try:
1094
  node_vk = Ed25519PublicKey.from_public_bytes(bytes.fromhex(norm_pk))
1095
- node_vk.verify(bytes.fromhex(norm_sig), manifest_digest)
 
 
 
 
1096
  except (InvalidSignature, ValueError):
1097
  continue
1098
 
@@ -1100,7 +1110,10 @@ class QuorumVerifier:
1100
  try:
1101
  node_pq_pk = bytes.fromhex(att.pq_pubkey_hex.strip().lower())
1102
  node_pq_sig = bytes.fromhex(att.pq_signature_hex.strip())
1103
- if not WOTSPlusSHAKE256.verify(node_pq_pk, manifest_digest, node_pq_sig):
 
 
 
1104
  continue
1105
  except ValueError:
1106
  continue
@@ -1664,11 +1677,20 @@ def create_mock_quorum_attestations(
1664
  for asn in distinct_asns:
1665
  priv, pub = generate_ed25519_keypair()
1666
  sk = Ed25519PrivateKey.from_private_bytes(priv)
1667
- sig = sk.sign(manifest_digest)
1668
- pq_priv, pq_pub = WOTSPlusSHAKE256.derive_keypair_from_ed25519_priv(priv)
1669
- pq_sig = WOTSPlusSHAKE256.sign(pq_priv, manifest_digest)
1670
  node_id = hashlib.sha1(pub + f"asn{asn}".encode()).hexdigest()
1671
  ip_digest = hashlib.sha256(f"192.0.2.{asn % 250}".encode()).hexdigest()
 
 
 
 
 
 
 
 
 
 
 
 
1672
 
1673
  attestations.append(QuorumAttestation(
1674
  node_id=node_id,
 
219
  "timestamp": int(self.timestamp),
220
  }
221
 
222
+ def signing_payload(self, manifest_digest: bytes) -> bytes:
223
+ meta = json.dumps(self.canonical_dict(), sort_keys=True, separators=(",", ":")).encode("utf-8")
224
+ return hashlib.sha256(b"armiger-attestation-v2:" + manifest_digest + b":" + meta).digest()
225
+
226
 
227
  @dataclasses.dataclass(frozen=True)
228
  class DHTShardManifest:
 
1094
  if isinstance(att.asn, bool) or not isinstance(att.asn, int) or att.asn <= 0:
1095
  continue
1096
 
1097
+ bound_digest = att.signing_payload(manifest_digest)
1098
+ sig_msg_used = bound_digest
1099
  try:
1100
  node_vk = Ed25519PublicKey.from_public_bytes(bytes.fromhex(norm_pk))
1101
+ try:
1102
+ node_vk.verify(bytes.fromhex(norm_sig), bound_digest)
1103
+ except InvalidSignature:
1104
+ node_vk.verify(bytes.fromhex(norm_sig), manifest_digest)
1105
+ sig_msg_used = manifest_digest
1106
  except (InvalidSignature, ValueError):
1107
  continue
1108
 
 
1110
  try:
1111
  node_pq_pk = bytes.fromhex(att.pq_pubkey_hex.strip().lower())
1112
  node_pq_sig = bytes.fromhex(att.pq_signature_hex.strip())
1113
+ if not (
1114
+ WOTSPlusSHAKE256.verify(node_pq_pk, sig_msg_used, node_pq_sig)
1115
+ or WOTSPlusSHAKE256.verify(node_pq_pk, manifest_digest, node_pq_sig)
1116
+ ):
1117
  continue
1118
  except ValueError:
1119
  continue
 
1677
  for asn in distinct_asns:
1678
  priv, pub = generate_ed25519_keypair()
1679
  sk = Ed25519PrivateKey.from_private_bytes(priv)
 
 
 
1680
  node_id = hashlib.sha1(pub + f"asn{asn}".encode()).hexdigest()
1681
  ip_digest = hashlib.sha256(f"192.0.2.{asn % 250}".encode()).hexdigest()
1682
+ temp_att = QuorumAttestation(
1683
+ node_id=node_id,
1684
+ asn=int(asn),
1685
+ ip_digest=ip_digest,
1686
+ pubkey_hex=pub.hex(),
1687
+ signature_hex="",
1688
+ timestamp=ts,
1689
+ )
1690
+ bound_msg = temp_att.signing_payload(manifest_digest)
1691
+ sig = sk.sign(bound_msg)
1692
+ pq_priv, pq_pub = WOTSPlusSHAKE256.derive_keypair_from_ed25519_priv(priv)
1693
+ pq_sig = WOTSPlusSHAKE256.sign(pq_priv, bound_msg)
1694
 
1695
  attestations.append(QuorumAttestation(
1696
  node_id=node_id,
armiger_dht_rsi.py ADDED
The diff for this file is too large to render. See raw diff
 
benchmarks/CHARTS_MANIFEST.json ADDED
@@ -0,0 +1,1575 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": "1.0.0",
3
+ "bootstrap_seed": 20261018,
4
+ "bootstrap_resamples": 10000,
5
+ "invariants_verified": {
6
+ "identical_denominator_lock": true,
7
+ "anti_circularity_lock": true,
8
+ "paired_uncertainty_and_significance": true,
9
+ "png_metadata_stripped_ihdr_idat_iend_only": true
10
+ },
11
+ "anti_circularity_audit": [
12
+ {
13
+ "model_name": "OpenAI GPT (Fast Tier)",
14
+ "is_teacher": true,
15
+ "eval_mode": "leave_one_teacher_out",
16
+ "self_vote_weight": 0.0,
17
+ "non_circular_pct": 95.12,
18
+ "circular_self_vote_pct": 96.59,
19
+ "circularity_inflation_pp": 1.46
20
+ },
21
+ {
22
+ "model_name": "Google Gemini Flash",
23
+ "is_teacher": true,
24
+ "eval_mode": "leave_one_teacher_out",
25
+ "self_vote_weight": 0.0,
26
+ "non_circular_pct": 93.8,
27
+ "circular_self_vote_pct": 95.61,
28
+ "circularity_inflation_pp": 1.8
29
+ },
30
+ {
31
+ "model_name": "Anthropic Claude Haiku 4.5",
32
+ "is_teacher": true,
33
+ "eval_mode": "independent_ground_truth",
34
+ "self_vote_weight": 0.0
35
+ }
36
+ ],
37
+ "provenance_sha256": {
38
+ "canonical_eval_vectors.json": "058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a",
39
+ "generate_canonical_charts.py": "e6d73c7197057fbb7cea3dee7f11599e6ce50039f692eada2d0ebae61ec774c4",
40
+ "assets/armiger_vs_llms_overall_and_6dims.png": "0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62",
41
+ "assets/armiger_vs_llms_cost_comparison.png": "3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4",
42
+ "assets/armiger_vs_torrent_classifiers_6dims.png": "e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010",
43
+ "assets/armiger_categorical_breakdown.png": "1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98",
44
+ "validated_zero_cue_catalog.json": "a43193a7d9982f1b83e9e6968c47dbbe8d058b113af5020d6d5ea45ed2af7fcd",
45
+ "armiger_ultra.fp32.onnx": "24c65e1af1be68c923bfdf774afb862c54b66a30f40c4b65cf5961929f149d48",
46
+ "armiger.int8.onnx": "c750e034aace4ff28c0eb1e1112b28f744a5770abd78618127258f283e54e4da"
47
+ },
48
+ "charts": {
49
+ "armiger_vs_llms_overall_and_6dims.png": {
50
+ "sha256": "0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62",
51
+ "composite_6dim_mean": {
52
+ "Armiger Ultra (Fair Hybrid)": {
53
+ "point_pct": 96.86,
54
+ "ci95_lo": 96.35,
55
+ "ci95_hi": 97.38,
56
+ "point_with_corpus_swarm_pct": 97.6
57
+ },
58
+ "Armiger (8.3 MB INT8 + Plugins)": {
59
+ "point_pct": 96.85,
60
+ "ci95_lo": 96.33,
61
+ "ci95_hi": 97.36
62
+ },
63
+ "Google Gemini Flash": {
64
+ "point_pct": 94.56,
65
+ "ci95_lo": 93.88,
66
+ "ci95_hi": 95.23
67
+ },
68
+ "Anthropic Claude Haiku 4.5": {
69
+ "point_pct": 92.88,
70
+ "ci95_lo": 92.04,
71
+ "ci95_hi": 93.72
72
+ },
73
+ "OpenAI GPT (Fast Tier)": {
74
+ "point_pct": 86.24,
75
+ "ci95_lo": 85.56,
76
+ "ci95_hi": 86.91
77
+ },
78
+ "Local 8B Instruct LLM (FP16)": {
79
+ "point_pct": 84.79,
80
+ "ci95_lo": 83.65,
81
+ "ci95_hi": 85.94
82
+ }
83
+ },
84
+ "panel_2a_category_11class": {
85
+ "panel_name": "chart1_2a_beyond_cat11",
86
+ "n": 1025,
87
+ "metric_type": "mean",
88
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
89
+ "models": {
90
+ "Armiger Ultra (Fair Hybrid)": {
91
+ "n": 1025,
92
+ "point_pct": 95.12,
93
+ "ci95_lo": 93.76,
94
+ "ci95_hi": 96.39,
95
+ "delta_vs_ref_pct": 0.0,
96
+ "delta_ci95_lo": 0.0,
97
+ "delta_ci95_hi": 0.0,
98
+ "p_value_vs_ref": 1.0
99
+ },
100
+ "Armiger (8.3 MB INT8 + Plugins)": {
101
+ "n": 1025,
102
+ "point_pct": 95.12,
103
+ "ci95_lo": 93.76,
104
+ "ci95_hi": 96.39,
105
+ "delta_vs_ref_pct": 0.0,
106
+ "delta_ci95_lo": 0.0,
107
+ "delta_ci95_hi": 0.0,
108
+ "p_value_vs_ref": 1.0
109
+ },
110
+ "Google Gemini Flash": {
111
+ "n": 1025,
112
+ "point_pct": 93.8,
113
+ "ci95_lo": 92.39,
114
+ "ci95_hi": 95.12,
115
+ "delta_vs_ref_pct": 1.32,
116
+ "delta_ci95_lo": -0.29,
117
+ "delta_ci95_hi": 2.93,
118
+ "p_value_vs_ref": 0.1073516
119
+ },
120
+ "Anthropic Claude Haiku 4.5": {
121
+ "n": 1025,
122
+ "point_pct": 93.76,
123
+ "ci95_lo": 92.2,
124
+ "ci95_hi": 95.22,
125
+ "delta_vs_ref_pct": 1.37,
126
+ "delta_ci95_lo": 0.68,
127
+ "delta_ci95_hi": 2.15,
128
+ "p_value_vs_ref": 0.0001220703
129
+ },
130
+ "OpenAI GPT (Fast Tier)": {
131
+ "n": 1025,
132
+ "point_pct": 95.12,
133
+ "ci95_lo": 93.85,
134
+ "ci95_hi": 96.29,
135
+ "delta_vs_ref_pct": 0.0,
136
+ "delta_ci95_lo": -1.46,
137
+ "delta_ci95_hi": 1.41,
138
+ "p_value_vs_ref": 0.5715867
139
+ },
140
+ "Local 8B Instruct LLM (FP16)": {
141
+ "n": 1025,
142
+ "point_pct": 80.78,
143
+ "ci95_lo": 78.34,
144
+ "ci95_hi": 83.22,
145
+ "delta_vs_ref_pct": 14.34,
146
+ "delta_ci95_lo": 12.2,
147
+ "delta_ci95_hi": 16.49,
148
+ "p_value_vs_ref": 1.121039e-44
149
+ }
150
+ }
151
+ },
152
+ "panel_2b_catalog_matching": {
153
+ "panel_name": "chart1_2b_beyond_matching",
154
+ "n": 1025,
155
+ "metric_type": "mean",
156
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
157
+ "models": {
158
+ "Armiger Ultra (Fair Hybrid)": {
159
+ "n": 1025,
160
+ "point_pct": 96.39,
161
+ "ci95_lo": 95.22,
162
+ "ci95_hi": 97.46,
163
+ "delta_vs_ref_pct": 0.0,
164
+ "delta_ci95_lo": 0.0,
165
+ "delta_ci95_hi": 0.0,
166
+ "p_value_vs_ref": 1.0
167
+ },
168
+ "Armiger (8.3 MB INT8 + Plugins)": {
169
+ "n": 1025,
170
+ "point_pct": 96.29,
171
+ "ci95_lo": 95.02,
172
+ "ci95_hi": 97.37,
173
+ "delta_vs_ref_pct": 0.1,
174
+ "delta_ci95_lo": -0.69,
175
+ "delta_ci95_hi": 0.98,
176
+ "p_value_vs_ref": 1.0
177
+ },
178
+ "Google Gemini Flash": {
179
+ "n": 1025,
180
+ "point_pct": 90.88,
181
+ "ci95_lo": 89.17,
182
+ "ci95_hi": 92.49,
183
+ "delta_vs_ref_pct": 5.51,
184
+ "delta_ci95_lo": 3.71,
185
+ "delta_ci95_hi": 7.32,
186
+ "p_value_vs_ref": 3.767158e-09
187
+ },
188
+ "Anthropic Claude Haiku 4.5": {
189
+ "n": 1025,
190
+ "point_pct": 86.15,
191
+ "ci95_lo": 84.0,
192
+ "ci95_hi": 88.2,
193
+ "delta_vs_ref_pct": 10.24,
194
+ "delta_ci95_lo": 8.39,
195
+ "delta_ci95_hi": 12.1,
196
+ "p_value_vs_ref": 4.930381e-32
197
+ },
198
+ "OpenAI GPT (Fast Tier)": {
199
+ "n": 1025,
200
+ "point_pct": 89.07,
201
+ "ci95_lo": 87.17,
202
+ "ci95_hi": 90.88,
203
+ "delta_vs_ref_pct": 7.32,
204
+ "delta_ci95_lo": 5.37,
205
+ "delta_ci95_hi": 9.32,
206
+ "p_value_vs_ref": 2.797646e-13
207
+ },
208
+ "Local 8B Instruct LLM (FP16)": {
209
+ "n": 1025,
210
+ "point_pct": 77.85,
211
+ "ci95_lo": 75.32,
212
+ "ci95_hi": 80.29,
213
+ "delta_vs_ref_pct": 18.54,
214
+ "delta_ci95_lo": 16.29,
215
+ "delta_ci95_hi": 20.98,
216
+ "p_value_vs_ref": 1.274474e-57
217
+ }
218
+ }
219
+ },
220
+ "panel_2c_audio_strict_229": {
221
+ "panel_name": "chart1_2c_beyond_audio_strict",
222
+ "n": 229,
223
+ "metric_type": "mean",
224
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
225
+ "models": {
226
+ "Armiger Ultra (Fair Hybrid)": {
227
+ "n": 229,
228
+ "point_pct": 99.13,
229
+ "ci95_lo": 97.82,
230
+ "ci95_hi": 100.0,
231
+ "delta_vs_ref_pct": 0.0,
232
+ "delta_ci95_lo": 0.0,
233
+ "delta_ci95_hi": 0.0,
234
+ "p_value_vs_ref": 1.0
235
+ },
236
+ "Armiger (8.3 MB INT8 + Plugins)": {
237
+ "n": 229,
238
+ "point_pct": 99.13,
239
+ "ci95_lo": 97.82,
240
+ "ci95_hi": 100.0,
241
+ "delta_vs_ref_pct": 0.0,
242
+ "delta_ci95_lo": 0.0,
243
+ "delta_ci95_hi": 0.0,
244
+ "p_value_vs_ref": 1.0
245
+ },
246
+ "Google Gemini Flash": {
247
+ "n": 229,
248
+ "point_pct": 98.69,
249
+ "ci95_lo": 96.94,
250
+ "ci95_hi": 100.0,
251
+ "delta_vs_ref_pct": 0.44,
252
+ "delta_ci95_lo": -1.31,
253
+ "delta_ci95_hi": 2.18,
254
+ "p_value_vs_ref": 1.0
255
+ },
256
+ "Anthropic Claude Haiku 4.5": {
257
+ "n": 229,
258
+ "point_pct": 95.63,
259
+ "ci95_lo": 92.58,
260
+ "ci95_hi": 98.25,
261
+ "delta_vs_ref_pct": 3.49,
262
+ "delta_ci95_lo": 1.31,
263
+ "delta_ci95_hi": 6.11,
264
+ "p_value_vs_ref": 0.0078125
265
+ },
266
+ "OpenAI GPT (Fast Tier)": {
267
+ "n": 229,
268
+ "point_pct": 97.82,
269
+ "ci95_lo": 95.63,
270
+ "ci95_hi": 99.56,
271
+ "delta_vs_ref_pct": 1.31,
272
+ "delta_ci95_lo": 0.0,
273
+ "delta_ci95_hi": 3.06,
274
+ "p_value_vs_ref": 0.25
275
+ },
276
+ "Local 8B Instruct LLM (FP16)": {
277
+ "n": 229,
278
+ "point_pct": 93.45,
279
+ "ci95_lo": 89.96,
280
+ "ci95_hi": 96.51,
281
+ "delta_vs_ref_pct": 5.68,
282
+ "delta_ci95_lo": 3.06,
283
+ "delta_ci95_hi": 8.73,
284
+ "p_value_vs_ref": 0.0002441406
285
+ }
286
+ }
287
+ },
288
+ "panel_2d_anime_valid_636": {
289
+ "panel_name": "chart1_2d_beyond_anime_valid636",
290
+ "n": 636,
291
+ "metric_type": "mean",
292
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
293
+ "models": {
294
+ "Armiger Ultra (Fair Hybrid)": {
295
+ "n": 636,
296
+ "point_pct": 98.9,
297
+ "ci95_lo": 97.96,
298
+ "ci95_hi": 99.69,
299
+ "delta_vs_ref_pct": 0.0,
300
+ "delta_ci95_lo": 0.0,
301
+ "delta_ci95_hi": 0.0,
302
+ "p_value_vs_ref": 1.0
303
+ },
304
+ "Armiger (8.3 MB INT8 + Plugins)": {
305
+ "n": 636,
306
+ "point_pct": 98.9,
307
+ "ci95_lo": 97.96,
308
+ "ci95_hi": 99.69,
309
+ "delta_vs_ref_pct": 0.0,
310
+ "delta_ci95_lo": 0.0,
311
+ "delta_ci95_hi": 0.0,
312
+ "p_value_vs_ref": 1.0
313
+ },
314
+ "Google Gemini Flash": {
315
+ "n": 636,
316
+ "point_pct": 95.6,
317
+ "ci95_lo": 93.87,
318
+ "ci95_hi": 97.17,
319
+ "delta_vs_ref_pct": 3.3,
320
+ "delta_ci95_lo": 1.73,
321
+ "delta_ci95_hi": 5.03,
322
+ "p_value_vs_ref": 4.923344e-05
323
+ },
324
+ "Anthropic Claude Haiku 4.5": {
325
+ "n": 636,
326
+ "point_pct": 98.9,
327
+ "ci95_lo": 97.96,
328
+ "ci95_hi": 99.69,
329
+ "delta_vs_ref_pct": 0.0,
330
+ "delta_ci95_lo": 0.0,
331
+ "delta_ci95_hi": 0.0,
332
+ "p_value_vs_ref": 1.0
333
+ },
334
+ "OpenAI GPT (Fast Tier)": {
335
+ "n": 636,
336
+ "point_pct": 95.6,
337
+ "ci95_lo": 93.87,
338
+ "ci95_hi": 97.17,
339
+ "delta_vs_ref_pct": 3.3,
340
+ "delta_ci95_lo": 1.73,
341
+ "delta_ci95_hi": 5.03,
342
+ "p_value_vs_ref": 4.923344e-05
343
+ },
344
+ "Local 8B Instruct LLM (FP16)": {
345
+ "n": 636,
346
+ "point_pct": 92.45,
347
+ "ci95_lo": 90.25,
348
+ "ci95_hi": 94.5,
349
+ "delta_vs_ref_pct": 6.45,
350
+ "delta_ci95_lo": 4.56,
351
+ "delta_ci95_hi": 8.33,
352
+ "p_value_vs_ref": 9.094947e-13
353
+ }
354
+ }
355
+ },
356
+ "panel_2e_junk_balanced_977": {
357
+ "panel_name": "chart1_2e_beyond_junk977_balanced",
358
+ "n": 977,
359
+ "metric_type": "balanced",
360
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
361
+ "models": {
362
+ "Armiger Ultra (Fair Hybrid)": {
363
+ "n": 977,
364
+ "point_pct": 99.23,
365
+ "ci95_lo": 97.69,
366
+ "ci95_hi": 100.0,
367
+ "delta_vs_ref_pct": 0.0,
368
+ "delta_ci95_lo": 0.0,
369
+ "delta_ci95_hi": 0.0,
370
+ "p_value_vs_ref": 1.0
371
+ },
372
+ "Armiger (8.3 MB INT8 + Plugins)": {
373
+ "n": 977,
374
+ "point_pct": 99.23,
375
+ "ci95_lo": 97.69,
376
+ "ci95_hi": 100.0,
377
+ "delta_vs_ref_pct": 0.0,
378
+ "delta_ci95_lo": 0.0,
379
+ "delta_ci95_hi": 0.0,
380
+ "p_value_vs_ref": 1.0
381
+ },
382
+ "Google Gemini Flash": {
383
+ "n": 977,
384
+ "point_pct": 99.18,
385
+ "ci95_lo": 97.53,
386
+ "ci95_hi": 100.0,
387
+ "delta_vs_ref_pct": 0.05,
388
+ "delta_ci95_lo": -2.25,
389
+ "delta_ci95_hi": 2.36,
390
+ "p_value_vs_ref": 1.0
391
+ },
392
+ "Anthropic Claude Haiku 4.5": {
393
+ "n": 977,
394
+ "point_pct": 93.09,
395
+ "ci95_lo": 90.56,
396
+ "ci95_hi": 95.12,
397
+ "delta_vs_ref_pct": 6.14,
398
+ "delta_ci95_lo": 3.39,
399
+ "delta_ci95_hi": 9.0,
400
+ "p_value_vs_ref": 8.046381e-29
401
+ },
402
+ "OpenAI GPT (Fast Tier)": {
403
+ "n": 977,
404
+ "point_pct": 50.0,
405
+ "ci95_lo": 50.0,
406
+ "ci95_hi": 50.0,
407
+ "delta_vs_ref_pct": 49.23,
408
+ "delta_ci95_lo": 47.69,
409
+ "delta_ci95_hi": 50.0,
410
+ "p_value_vs_ref": 1.084202e-19
411
+ },
412
+ "Local 8B Instruct LLM (FP16)": {
413
+ "n": 977,
414
+ "point_pct": 97.91,
415
+ "ci95_lo": 95.5,
416
+ "ci95_hi": 99.62,
417
+ "delta_vs_ref_pct": 1.32,
418
+ "delta_ci95_lo": -1.15,
419
+ "delta_ci95_hi": 4.01,
420
+ "p_value_vs_ref": 0.003417969
421
+ }
422
+ }
423
+ },
424
+ "panel_2f_swarm_public_757": {
425
+ "panel_name": "chart1_2f_swarm_public_757",
426
+ "n": 757,
427
+ "metric_type": "auroc",
428
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
429
+ "models": {
430
+ "Armiger Ultra (Fair Hybrid)": {
431
+ "n": 757,
432
+ "point_pct": 92.42,
433
+ "ci95_lo": 90.55,
434
+ "ci95_hi": 94.13,
435
+ "delta_vs_ref_pct": 0.0,
436
+ "delta_ci95_lo": 0.0,
437
+ "delta_ci95_hi": 0.0,
438
+ "p_value_vs_ref": 1.0
439
+ },
440
+ "Armiger (8.3 MB INT8 + Plugins)": {
441
+ "n": 757,
442
+ "point_pct": 92.42,
443
+ "ci95_lo": 90.55,
444
+ "ci95_hi": 94.13,
445
+ "delta_vs_ref_pct": 0.0,
446
+ "delta_ci95_lo": 0.0,
447
+ "delta_ci95_hi": 0.0,
448
+ "p_value_vs_ref": 1.0
449
+ },
450
+ "Google Gemini Flash": {
451
+ "n": 757,
452
+ "point_pct": 89.2,
453
+ "ci95_lo": 86.89,
454
+ "ci95_hi": 91.42,
455
+ "delta_vs_ref_pct": 3.22,
456
+ "delta_ci95_lo": 0.32,
457
+ "delta_ci95_hi": 6.1,
458
+ "p_value_vs_ref": 2.227484e-06
459
+ },
460
+ "Anthropic Claude Haiku 4.5": {
461
+ "n": 757,
462
+ "point_pct": 89.76,
463
+ "ci95_lo": 87.53,
464
+ "ci95_hi": 91.94,
465
+ "delta_vs_ref_pct": 2.66,
466
+ "delta_ci95_lo": -0.16,
467
+ "delta_ci95_hi": 5.47,
468
+ "p_value_vs_ref": 4.115658e-10
469
+ },
470
+ "OpenAI GPT (Fast Tier)": {
471
+ "n": 757,
472
+ "point_pct": 89.82,
473
+ "ci95_lo": 87.51,
474
+ "ci95_hi": 91.93,
475
+ "delta_vs_ref_pct": 2.6,
476
+ "delta_ci95_lo": -0.26,
477
+ "delta_ci95_hi": 5.47,
478
+ "p_value_vs_ref": 6.473227e-07
479
+ },
480
+ "Local 8B Instruct LLM (FP16)": {
481
+ "n": 757,
482
+ "point_pct": 66.32,
483
+ "ci95_lo": 62.34,
484
+ "ci95_hi": 70.17,
485
+ "delta_vs_ref_pct": 26.1,
486
+ "delta_ci95_lo": 21.73,
487
+ "delta_ci95_hi": 30.6,
488
+ "p_value_vs_ref": 1.035307e-81
489
+ }
490
+ }
491
+ },
492
+ "panel_2f_swarm_warehouse_10278": {
493
+ "panel_name": "chart1_2f_swarm_warehouse_10278",
494
+ "n": 10278,
495
+ "metric_type": "auroc",
496
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
497
+ "models": {
498
+ "Armiger Ultra (Fair Hybrid)": {
499
+ "n": 10278,
500
+ "point_pct": 96.83,
501
+ "ci95_lo": 96.52,
502
+ "ci95_hi": 97.12,
503
+ "delta_vs_ref_pct": 0.0,
504
+ "delta_ci95_lo": 0.0,
505
+ "delta_ci95_hi": 0.0,
506
+ "p_value_vs_ref": 1.0
507
+ },
508
+ "Armiger 4-Way Priors": {
509
+ "n": 10278,
510
+ "point_pct": 86.64,
511
+ "ci95_lo": 85.91,
512
+ "ci95_hi": 87.36,
513
+ "delta_vs_ref_pct": 10.19,
514
+ "delta_ci95_lo": 9.39,
515
+ "delta_ci95_hi": 10.99,
516
+ "p_value_vs_ref": 0.0
517
+ },
518
+ "Raw Seeder Count": {
519
+ "n": 10278,
520
+ "point_pct": 53.02,
521
+ "ci95_lo": 51.88,
522
+ "ci95_hi": 54.22,
523
+ "delta_vs_ref_pct": 43.81,
524
+ "delta_ci95_lo": 42.57,
525
+ "delta_ci95_hi": 45.0,
526
+ "p_value_vs_ref": 0.0
527
+ }
528
+ }
529
+ }
530
+ },
531
+ "armiger_vs_llms_cost_comparison.png": {
532
+ "sha256": "3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4",
533
+ "models": [
534
+ "Armiger (8.3 MB M4 Pro CPU INT8)",
535
+ "Armiger Ultra (M4 Pro / RTX GPU FP32)",
536
+ "Local 8B Instruct LLM (FP16)",
537
+ "Google Gemini Flash",
538
+ "Cheap OpenAI GPT (Fast Tier)",
539
+ "Anthropic Claude Haiku 4.5"
540
+ ],
541
+ "batched_usd_per_1m": [
542
+ 0.002,
543
+ 0.004,
544
+ 1.85,
545
+ 94.9,
546
+ 127.3,
547
+ 712.0
548
+ ],
549
+ "realtime_usd_per_1m": [
550
+ 0.002,
551
+ 0.004,
552
+ 18.0,
553
+ 147.0,
554
+ 220.5,
555
+ 1220.0
556
+ ]
557
+ },
558
+ "armiger_vs_torrent_classifiers_6dims.png": {
559
+ "sha256": "e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010",
560
+ "panel_1_category_11class_oct15": {
561
+ "panel_name": "chart3_1_oct15_cat11",
562
+ "n": 2835,
563
+ "metric_type": "mean",
564
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
565
+ "models": {
566
+ "Armiger Ultra (Fair Hybrid)": {
567
+ "n": 2835,
568
+ "point_pct": 95.49,
569
+ "ci95_lo": 94.71,
570
+ "ci95_hi": 96.23,
571
+ "delta_vs_ref_pct": 0.0,
572
+ "delta_ci95_lo": 0.0,
573
+ "delta_ci95_hi": 0.0,
574
+ "p_value_vs_ref": 1.0
575
+ },
576
+ "Armiger (8.3 MB INT8 + Plugins)": {
577
+ "n": 2835,
578
+ "point_pct": 95.49,
579
+ "ci95_lo": 94.71,
580
+ "ci95_hi": 96.23,
581
+ "delta_vs_ref_pct": 0.0,
582
+ "delta_ci95_lo": 0.0,
583
+ "delta_ci95_hi": 0.0,
584
+ "p_value_vs_ref": 1.0
585
+ },
586
+ "MIMMO BERT (360M)": {
587
+ "n": 2835,
588
+ "point_pct": 72.24,
589
+ "ci95_lo": 70.58,
590
+ "ci95_hi": 73.86,
591
+ "delta_vs_ref_pct": 23.25,
592
+ "delta_ci95_lo": 21.62,
593
+ "delta_ci95_hi": 24.9,
594
+ "p_value_vs_ref": 2.797454e-162
595
+ },
596
+ "MIMMO Cascade": {
597
+ "n": 2835,
598
+ "point_pct": 71.32,
599
+ "ci95_lo": 69.63,
600
+ "ci95_hi": 72.95,
601
+ "delta_vs_ref_pct": 24.16,
602
+ "delta_ci95_lo": 22.5,
603
+ "delta_ci95_hi": 25.86,
604
+ "p_value_vs_ref": 7.860859e-169
605
+ },
606
+ "RTN (rank-torrent-name)": {
607
+ "n": 2835,
608
+ "point_pct": 53.58,
609
+ "ci95_lo": 51.78,
610
+ "ci95_hi": 55.41,
611
+ "delta_vs_ref_pct": 41.9,
612
+ "delta_ci95_lo": 40.07,
613
+ "delta_ci95_hi": 43.77,
614
+ "p_value_vs_ref": 0.0
615
+ },
616
+ "PTT / Parsett": {
617
+ "n": 2835,
618
+ "point_pct": 53.58,
619
+ "ci95_lo": 51.78,
620
+ "ci95_hi": 55.41,
621
+ "delta_vs_ref_pct": 41.9,
622
+ "delta_ci95_lo": 40.07,
623
+ "delta_ci95_hi": 43.77,
624
+ "p_value_vs_ref": 0.0
625
+ },
626
+ "GuessIt": {
627
+ "n": 2835,
628
+ "point_pct": 38.69,
629
+ "ci95_lo": 36.93,
630
+ "ci95_hi": 40.46,
631
+ "delta_vs_ref_pct": 56.79,
632
+ "delta_ci95_lo": 54.96,
633
+ "delta_ci95_hi": 58.62,
634
+ "p_value_vs_ref": 0.0
635
+ },
636
+ "PTN (parse-torrent-title)": {
637
+ "n": 2835,
638
+ "point_pct": 35.59,
639
+ "ci95_lo": 33.83,
640
+ "ci95_hi": 37.35,
641
+ "delta_vs_ref_pct": 59.89,
642
+ "delta_ci95_lo": 58.06,
643
+ "delta_ci95_hi": 61.73,
644
+ "p_value_vs_ref": 0.0
645
+ },
646
+ "Anitomy": {
647
+ "n": 2835,
648
+ "point_pct": 33.4,
649
+ "ci95_lo": 31.68,
650
+ "ci95_hi": 35.13,
651
+ "delta_vs_ref_pct": 62.08,
652
+ "delta_ci95_lo": 60.25,
653
+ "delta_ci95_hi": 63.88,
654
+ "p_value_vs_ref": 0.0
655
+ },
656
+ "Bitmagnet Built-In": {
657
+ "n": 2835,
658
+ "point_pct": 35.17,
659
+ "ci95_lo": 33.47,
660
+ "ci95_hi": 36.9,
661
+ "delta_vs_ref_pct": 60.32,
662
+ "delta_ci95_lo": 58.52,
663
+ "delta_ci95_hi": 62.12,
664
+ "p_value_vs_ref": 0.0
665
+ }
666
+ }
667
+ },
668
+ "panel_1_video_mvtv_oct15": {
669
+ "panel_name": "chart3_1_oct15_video_mvtv",
670
+ "n": 1296,
671
+ "metric_type": "mean",
672
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
673
+ "models": {
674
+ "Armiger Ultra (Fair Hybrid)": {
675
+ "n": 1296,
676
+ "point_pct": 97.3,
677
+ "ci95_lo": 96.37,
678
+ "ci95_hi": 98.15,
679
+ "delta_vs_ref_pct": 0.0,
680
+ "delta_ci95_lo": 0.0,
681
+ "delta_ci95_hi": 0.0,
682
+ "p_value_vs_ref": 1.0
683
+ },
684
+ "Armiger (8.3 MB INT8 + Plugins)": {
685
+ "n": 1296,
686
+ "point_pct": 97.3,
687
+ "ci95_lo": 96.37,
688
+ "ci95_hi": 98.15,
689
+ "delta_vs_ref_pct": 0.0,
690
+ "delta_ci95_lo": 0.0,
691
+ "delta_ci95_hi": 0.0,
692
+ "p_value_vs_ref": 1.0
693
+ },
694
+ "MIMMO BERT (360M)": {
695
+ "n": 1296,
696
+ "point_pct": 76.0,
697
+ "ci95_lo": 73.61,
698
+ "ci95_hi": 78.32,
699
+ "delta_vs_ref_pct": 21.3,
700
+ "delta_ci95_lo": 18.9,
701
+ "delta_ci95_hi": 23.77,
702
+ "p_value_vs_ref": 1.427447e-63
703
+ },
704
+ "MIMMO Cascade": {
705
+ "n": 1296,
706
+ "point_pct": 76.08,
707
+ "ci95_lo": 73.69,
708
+ "ci95_hi": 78.4,
709
+ "delta_vs_ref_pct": 21.22,
710
+ "delta_ci95_lo": 18.83,
711
+ "delta_ci95_hi": 23.69,
712
+ "p_value_vs_ref": 2.667039e-63
713
+ },
714
+ "RTN (rank-torrent-name)": {
715
+ "n": 1296,
716
+ "point_pct": 81.64,
717
+ "ci95_lo": 79.48,
718
+ "ci95_hi": 83.72,
719
+ "delta_vs_ref_pct": 15.66,
720
+ "delta_ci95_lo": 13.5,
721
+ "delta_ci95_hi": 17.82,
722
+ "p_value_vs_ref": 4.447751e-45
723
+ },
724
+ "PTT / Parsett": {
725
+ "n": 1296,
726
+ "point_pct": 81.64,
727
+ "ci95_lo": 79.48,
728
+ "ci95_hi": 83.72,
729
+ "delta_vs_ref_pct": 15.66,
730
+ "delta_ci95_lo": 13.5,
731
+ "delta_ci95_hi": 17.82,
732
+ "p_value_vs_ref": 4.447751e-45
733
+ },
734
+ "GuessIt": {
735
+ "n": 1296,
736
+ "point_pct": 84.65,
737
+ "ci95_lo": 82.64,
738
+ "ci95_hi": 86.57,
739
+ "delta_vs_ref_pct": 12.65,
740
+ "delta_ci95_lo": 10.65,
741
+ "delta_ci95_hi": 14.66,
742
+ "p_value_vs_ref": 2.56753e-35
743
+ },
744
+ "PTN (parse-torrent-title)": {
745
+ "n": 1296,
746
+ "point_pct": 77.85,
747
+ "ci95_lo": 75.54,
748
+ "ci95_hi": 80.09,
749
+ "delta_vs_ref_pct": 19.44,
750
+ "delta_ci95_lo": 17.05,
751
+ "delta_ci95_hi": 21.84,
752
+ "p_value_vs_ref": 1.163065e-57
753
+ },
754
+ "Anitomy": {
755
+ "n": 1296,
756
+ "point_pct": 73.07,
757
+ "ci95_lo": 70.6,
758
+ "ci95_hi": 75.46,
759
+ "delta_vs_ref_pct": 24.23,
760
+ "delta_ci95_lo": 21.68,
761
+ "delta_ci95_hi": 26.77,
762
+ "p_value_vs_ref": 5.732874e-76
763
+ },
764
+ "Bitmagnet Built-In": {
765
+ "n": 1296,
766
+ "point_pct": 43.21,
767
+ "ci95_lo": 40.51,
768
+ "ci95_hi": 45.91,
769
+ "delta_vs_ref_pct": 54.09,
770
+ "delta_ci95_lo": 51.23,
771
+ "delta_ci95_hi": 56.87,
772
+ "p_value_vs_ref": 8.363314e-201
773
+ }
774
+ }
775
+ },
776
+ "panel_2_catalog_matching_oct15": {
777
+ "panel_name": "chart3_2_oct15_catalog_matching",
778
+ "n": 1296,
779
+ "metric_type": "mean",
780
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
781
+ "models": {
782
+ "Armiger Ultra (Fair Hybrid)": {
783
+ "n": 1296,
784
+ "point_pct": 96.53,
785
+ "ci95_lo": 95.52,
786
+ "ci95_hi": 97.53,
787
+ "delta_vs_ref_pct": 0.0,
788
+ "delta_ci95_lo": 0.0,
789
+ "delta_ci95_hi": 0.0,
790
+ "p_value_vs_ref": 1.0
791
+ },
792
+ "Armiger (8.3 MB INT8 + Plugins)": {
793
+ "n": 1296,
794
+ "point_pct": 96.53,
795
+ "ci95_lo": 95.52,
796
+ "ci95_hi": 97.53,
797
+ "delta_vs_ref_pct": 0.0,
798
+ "delta_ci95_lo": 0.0,
799
+ "delta_ci95_hi": 0.0,
800
+ "p_value_vs_ref": 1.0
801
+ },
802
+ "MIMMO BERT (360M)": {
803
+ "n": 1296,
804
+ "point_pct": 60.26,
805
+ "ci95_lo": 57.56,
806
+ "ci95_hi": 62.96,
807
+ "delta_vs_ref_pct": 36.27,
808
+ "delta_ci95_lo": 33.49,
809
+ "delta_ci95_hi": 39.12,
810
+ "p_value_vs_ref": 1.578863e-117
811
+ },
812
+ "MIMMO Cascade": {
813
+ "n": 1296,
814
+ "point_pct": 60.34,
815
+ "ci95_lo": 57.56,
816
+ "ci95_hi": 62.96,
817
+ "delta_vs_ref_pct": 36.19,
818
+ "delta_ci95_lo": 33.33,
819
+ "delta_ci95_hi": 39.04,
820
+ "p_value_vs_ref": 3.028483e-117
821
+ },
822
+ "RTN (rank-torrent-name)": {
823
+ "n": 1296,
824
+ "point_pct": 80.48,
825
+ "ci95_lo": 78.24,
826
+ "ci95_hi": 82.64,
827
+ "delta_vs_ref_pct": 16.05,
828
+ "delta_ci95_lo": 13.81,
829
+ "delta_ci95_hi": 18.29,
830
+ "p_value_vs_ref": 7.158345e-45
831
+ },
832
+ "PTT / Parsett": {
833
+ "n": 1296,
834
+ "point_pct": 80.48,
835
+ "ci95_lo": 78.24,
836
+ "ci95_hi": 82.64,
837
+ "delta_vs_ref_pct": 16.05,
838
+ "delta_ci95_lo": 13.81,
839
+ "delta_ci95_hi": 18.29,
840
+ "p_value_vs_ref": 7.158345e-45
841
+ },
842
+ "GuessIt": {
843
+ "n": 1296,
844
+ "point_pct": 84.03,
845
+ "ci95_lo": 82.02,
846
+ "ci95_hi": 85.96,
847
+ "delta_vs_ref_pct": 12.5,
848
+ "delta_ci95_lo": 10.42,
849
+ "delta_ci95_hi": 14.58,
850
+ "p_value_vs_ref": 7.451219e-32
851
+ },
852
+ "PTN (parse-torrent-title)": {
853
+ "n": 1296,
854
+ "point_pct": 77.62,
855
+ "ci95_lo": 75.31,
856
+ "ci95_hi": 79.86,
857
+ "delta_vs_ref_pct": 18.9,
858
+ "delta_ci95_lo": 16.51,
859
+ "delta_ci95_hi": 21.37,
860
+ "p_value_vs_ref": 3.559008e-51
861
+ },
862
+ "Anitomy": {
863
+ "n": 1296,
864
+ "point_pct": 72.76,
865
+ "ci95_lo": 70.29,
866
+ "ci95_hi": 75.15,
867
+ "delta_vs_ref_pct": 23.77,
868
+ "delta_ci95_lo": 21.22,
869
+ "delta_ci95_hi": 26.31,
870
+ "p_value_vs_ref": 7.077119e-70
871
+ },
872
+ "Bitmagnet Built-In": {
873
+ "n": 1296,
874
+ "point_pct": 43.21,
875
+ "ci95_lo": 40.51,
876
+ "ci95_hi": 45.91,
877
+ "delta_vs_ref_pct": 53.32,
878
+ "delta_ci95_lo": 50.54,
879
+ "delta_ci95_hi": 56.1,
880
+ "p_value_vs_ref": 7.866753e-198
881
+ }
882
+ }
883
+ },
884
+ "panel_3_audio_strict_oct15": {
885
+ "panel_name": "chart3_3_oct15_audio_strict545",
886
+ "n": 545,
887
+ "metric_type": "mean",
888
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
889
+ "models": {
890
+ "Armiger Ultra (Fair Hybrid)": {
891
+ "n": 545,
892
+ "point_pct": 97.8,
893
+ "ci95_lo": 96.51,
894
+ "ci95_hi": 98.9,
895
+ "delta_vs_ref_pct": 0.0,
896
+ "delta_ci95_lo": 0.0,
897
+ "delta_ci95_hi": 0.0,
898
+ "p_value_vs_ref": 1.0
899
+ },
900
+ "Armiger (8.3 MB INT8 + Plugins)": {
901
+ "n": 545,
902
+ "point_pct": 97.8,
903
+ "ci95_lo": 96.51,
904
+ "ci95_hi": 98.9,
905
+ "delta_vs_ref_pct": 0.0,
906
+ "delta_ci95_lo": 0.0,
907
+ "delta_ci95_hi": 0.0,
908
+ "p_value_vs_ref": 1.0
909
+ },
910
+ "RTN (rank-torrent-name)": {
911
+ "n": 545,
912
+ "point_pct": 93.76,
913
+ "ci95_lo": 91.74,
914
+ "ci95_hi": 95.78,
915
+ "delta_vs_ref_pct": 4.04,
916
+ "delta_ci95_lo": 1.83,
917
+ "delta_ci95_hi": 6.24,
918
+ "p_value_vs_ref": 0.0006795483
919
+ },
920
+ "PTT / Parsett": {
921
+ "n": 545,
922
+ "point_pct": 93.76,
923
+ "ci95_lo": 91.74,
924
+ "ci95_hi": 95.78,
925
+ "delta_vs_ref_pct": 4.04,
926
+ "delta_ci95_lo": 1.83,
927
+ "delta_ci95_hi": 6.24,
928
+ "p_value_vs_ref": 0.0006795483
929
+ },
930
+ "GuessIt": {
931
+ "n": 545,
932
+ "point_pct": 94.86,
933
+ "ci95_lo": 93.03,
934
+ "ci95_hi": 96.7,
935
+ "delta_vs_ref_pct": 2.94,
936
+ "delta_ci95_lo": 0.92,
937
+ "delta_ci95_hi": 4.95,
938
+ "p_value_vs_ref": 0.009041185
939
+ },
940
+ "PTN (parse-torrent-title)": {
941
+ "n": 545,
942
+ "point_pct": 93.21,
943
+ "ci95_lo": 91.01,
944
+ "ci95_hi": 95.23,
945
+ "delta_vs_ref_pct": 4.59,
946
+ "delta_ci95_lo": 2.39,
947
+ "delta_ci95_hi": 6.97,
948
+ "p_value_vs_ref": 0.0001701552
949
+ },
950
+ "Anitomy": {
951
+ "n": 545,
952
+ "point_pct": 90.46,
953
+ "ci95_lo": 88.07,
954
+ "ci95_hi": 92.84,
955
+ "delta_vs_ref_pct": 7.34,
956
+ "delta_ci95_lo": 4.77,
957
+ "delta_ci95_hi": 9.91,
958
+ "p_value_vs_ref": 4.688184e-08
959
+ },
960
+ "Bitmagnet Built-In": {
961
+ "n": 545,
962
+ "point_pct": 88.26,
963
+ "ci95_lo": 85.5,
964
+ "ci95_hi": 90.83,
965
+ "delta_vs_ref_pct": 9.54,
966
+ "delta_ci95_lo": 6.79,
967
+ "delta_ci95_hi": 12.29,
968
+ "p_value_vs_ref": 2.383493e-11
969
+ }
970
+ }
971
+ },
972
+ "panel_4_anime_oct15": {
973
+ "panel_name": "chart3_4_oct15_anime507",
974
+ "n": 507,
975
+ "metric_type": "mean",
976
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
977
+ "models": {
978
+ "Armiger Ultra (Fair Hybrid)": {
979
+ "n": 507,
980
+ "point_pct": 99.61,
981
+ "ci95_lo": 99.01,
982
+ "ci95_hi": 100.0,
983
+ "delta_vs_ref_pct": 0.0,
984
+ "delta_ci95_lo": 0.0,
985
+ "delta_ci95_hi": 0.0,
986
+ "p_value_vs_ref": 1.0
987
+ },
988
+ "Armiger (8.3 MB INT8 + Plugins)": {
989
+ "n": 507,
990
+ "point_pct": 99.61,
991
+ "ci95_lo": 99.01,
992
+ "ci95_hi": 100.0,
993
+ "delta_vs_ref_pct": 0.0,
994
+ "delta_ci95_lo": 0.0,
995
+ "delta_ci95_hi": 0.0,
996
+ "p_value_vs_ref": 1.0
997
+ }
998
+ }
999
+ },
1000
+ "panel_4_anime_dub_tc2000": {
1001
+ "panel_name": "chart3_4_tc2000_anime_dub",
1002
+ "n": 2000,
1003
+ "metric_type": "mean",
1004
+ "reference_model": "RTN (rank-torrent-name)",
1005
+ "models": {
1006
+ "RTN (rank-torrent-name)": {
1007
+ "n": 2000,
1008
+ "point_pct": 77.65,
1009
+ "ci95_lo": 75.75,
1010
+ "ci95_hi": 79.45,
1011
+ "delta_vs_ref_pct": 0.0,
1012
+ "delta_ci95_lo": 0.0,
1013
+ "delta_ci95_hi": 0.0,
1014
+ "p_value_vs_ref": 1.0
1015
+ },
1016
+ "PTT / Parsett": {
1017
+ "n": 2000,
1018
+ "point_pct": 77.65,
1019
+ "ci95_lo": 75.75,
1020
+ "ci95_hi": 79.45,
1021
+ "delta_vs_ref_pct": 0.0,
1022
+ "delta_ci95_lo": 0.0,
1023
+ "delta_ci95_hi": 0.0,
1024
+ "p_value_vs_ref": 1.0
1025
+ },
1026
+ "GuessIt": {
1027
+ "n": 2000,
1028
+ "point_pct": 78.8,
1029
+ "ci95_lo": 76.95,
1030
+ "ci95_hi": 80.55,
1031
+ "delta_vs_ref_pct": -1.15,
1032
+ "delta_ci95_lo": -2.45,
1033
+ "delta_ci95_hi": 0.15,
1034
+ "p_value_vs_ref": 0.09792971
1035
+ },
1036
+ "PTN (parse-torrent-title)": {
1037
+ "n": 2000,
1038
+ "point_pct": 68.65,
1039
+ "ci95_lo": 66.6,
1040
+ "ci95_hi": 70.65,
1041
+ "delta_vs_ref_pct": 9.0,
1042
+ "delta_ci95_lo": 7.15,
1043
+ "delta_ci95_hi": 10.85,
1044
+ "p_value_vs_ref": 7.57291e-21
1045
+ },
1046
+ "Anitomy": {
1047
+ "n": 2000,
1048
+ "point_pct": 64.55,
1049
+ "ci95_lo": 62.45,
1050
+ "ci95_hi": 66.6,
1051
+ "delta_vs_ref_pct": 13.1,
1052
+ "delta_ci95_lo": 11.2,
1053
+ "delta_ci95_hi": 15.05,
1054
+ "p_value_vs_ref": 4.440189e-40
1055
+ }
1056
+ }
1057
+ },
1058
+ "panel_5_adult_oct15": {
1059
+ "panel_name": "chart3_5_oct15_adult2781",
1060
+ "n": 2781,
1061
+ "metric_type": "mean",
1062
+ "reference_model": "Armiger Ultra (Fair Hybrid)",
1063
+ "models": {
1064
+ "Armiger Ultra (Fair Hybrid)": {
1065
+ "n": 2781,
1066
+ "point_pct": 98.53,
1067
+ "ci95_lo": 98.06,
1068
+ "ci95_hi": 98.96,
1069
+ "delta_vs_ref_pct": 0.0,
1070
+ "delta_ci95_lo": 0.0,
1071
+ "delta_ci95_hi": 0.0,
1072
+ "p_value_vs_ref": 1.0
1073
+ },
1074
+ "Armiger (8.3 MB INT8 + Plugins)": {
1075
+ "n": 2781,
1076
+ "point_pct": 98.49,
1077
+ "ci95_lo": 98.02,
1078
+ "ci95_hi": 98.92,
1079
+ "delta_vs_ref_pct": 0.04,
1080
+ "delta_ci95_lo": 0.0,
1081
+ "delta_ci95_hi": 0.11,
1082
+ "p_value_vs_ref": 1.0
1083
+ },
1084
+ "MIMMO BERT (360M)": {
1085
+ "n": 2781,
1086
+ "point_pct": 91.05,
1087
+ "ci95_lo": 89.93,
1088
+ "ci95_hi": 92.09,
1089
+ "delta_vs_ref_pct": 7.48,
1090
+ "delta_ci95_lo": 6.44,
1091
+ "delta_ci95_hi": 8.59,
1092
+ "p_value_vs_ref": 2.280742e-45
1093
+ },
1094
+ "MIMMO Cascade": {
1095
+ "n": 2781,
1096
+ "point_pct": 89.68,
1097
+ "ci95_lo": 88.53,
1098
+ "ci95_hi": 90.79,
1099
+ "delta_vs_ref_pct": 8.85,
1100
+ "delta_ci95_lo": 7.73,
1101
+ "delta_ci95_hi": 10.03,
1102
+ "p_value_vs_ref": 1.795157e-55
1103
+ },
1104
+ "RTN (rank-torrent-name)": {
1105
+ "n": 2781,
1106
+ "point_pct": 79.0,
1107
+ "ci95_lo": 77.45,
1108
+ "ci95_hi": 80.55,
1109
+ "delta_vs_ref_pct": 19.53,
1110
+ "delta_ci95_lo": 17.98,
1111
+ "delta_ci95_hi": 21.07,
1112
+ "p_value_vs_ref": 2.877898e-138
1113
+ },
1114
+ "PTT / Parsett": {
1115
+ "n": 2781,
1116
+ "point_pct": 79.0,
1117
+ "ci95_lo": 77.45,
1118
+ "ci95_hi": 80.55,
1119
+ "delta_vs_ref_pct": 19.53,
1120
+ "delta_ci95_lo": 17.98,
1121
+ "delta_ci95_hi": 21.07,
1122
+ "p_value_vs_ref": 2.877898e-138
1123
+ },
1124
+ "Bitmagnet Built-In": {
1125
+ "n": 2781,
1126
+ "point_pct": 76.73,
1127
+ "ci95_lo": 75.19,
1128
+ "ci95_hi": 78.32,
1129
+ "delta_vs_ref_pct": 21.79,
1130
+ "delta_ci95_lo": 20.17,
1131
+ "delta_ci95_hi": 23.37,
1132
+ "p_value_vs_ref": 1.549894e-154
1133
+ }
1134
+ }
1135
+ },
1136
+ "panel_6a_junk_oct15": {
1137
+ "panel_name": "chart3_6a_oct15_junk2170",
1138
+ "n": 2170,
1139
+ "metric_type": "balanced",
1140
+ "reference_model": "Armiger Ultra",
1141
+ "models": {
1142
+ "Armiger Ultra": {
1143
+ "n": 2170,
1144
+ "point_pct": 100.0,
1145
+ "ci95_lo": 100.0,
1146
+ "ci95_hi": 100.0,
1147
+ "delta_vs_ref_pct": 0.0,
1148
+ "delta_ci95_lo": 0.0,
1149
+ "delta_ci95_hi": 0.0,
1150
+ "p_value_vs_ref": 1.0
1151
+ },
1152
+ "Armiger INT8": {
1153
+ "n": 2170,
1154
+ "point_pct": 100.0,
1155
+ "ci95_lo": 100.0,
1156
+ "ci95_hi": 100.0,
1157
+ "delta_vs_ref_pct": 0.0,
1158
+ "delta_ci95_lo": 0.0,
1159
+ "delta_ci95_hi": 0.0,
1160
+ "p_value_vs_ref": 1.0
1161
+ },
1162
+ "Bitmagnet": {
1163
+ "n": 2170,
1164
+ "point_pct": 50.0,
1165
+ "ci95_lo": 50.0,
1166
+ "ci95_hi": 50.0,
1167
+ "delta_vs_ref_pct": 50.0,
1168
+ "delta_ci95_lo": 50.0,
1169
+ "delta_ci95_hi": 50.0,
1170
+ "p_value_vs_ref": 1.0
1171
+ },
1172
+ "RTN Filter": {
1173
+ "n": 2170,
1174
+ "point_pct": 49.29,
1175
+ "ci95_lo": 49.03,
1176
+ "ci95_hi": 49.52,
1177
+ "delta_vs_ref_pct": 50.71,
1178
+ "delta_ci95_lo": 50.48,
1179
+ "delta_ci95_hi": 50.97,
1180
+ "p_value_vs_ref": 4.656613e-10
1181
+ }
1182
+ }
1183
+ }
1184
+ },
1185
+ "armiger_categorical_breakdown.png": {
1186
+ "sha256": "1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98",
1187
+ "left_oct15_confirmation_n2835": {
1188
+ "Audiobook": {
1189
+ "n": 95,
1190
+ "armiger_ultra_pct": 97.89,
1191
+ "armiger_int8_pct": 97.89,
1192
+ "mimmo_bert_pct": 0.0,
1193
+ "bitmagnet_pct": 9.47
1194
+ },
1195
+ "Adult / XXX": {
1196
+ "n": 1044,
1197
+ "armiger_ultra_pct": 97.8,
1198
+ "armiger_int8_pct": 97.8,
1199
+ "mimmo_bert_pct": 95.5,
1200
+ "bitmagnet_pct": 39.37
1201
+ },
1202
+ "TV Show": {
1203
+ "n": 601,
1204
+ "armiger_ultra_pct": 97.5,
1205
+ "armiger_int8_pct": 97.5,
1206
+ "mimmo_bert_pct": 58.4,
1207
+ "bitmagnet_pct": 28.62
1208
+ },
1209
+ "Movie": {
1210
+ "n": 695,
1211
+ "armiger_ultra_pct": 97.12,
1212
+ "armiger_int8_pct": 97.12,
1213
+ "mimmo_bert_pct": 78.85,
1214
+ "bitmagnet_pct": 55.83
1215
+ },
1216
+ "Software": {
1217
+ "n": 32,
1218
+ "armiger_ultra_pct": 93.75,
1219
+ "armiger_int8_pct": 93.75,
1220
+ "mimmo_bert_pct": 59.38,
1221
+ "bitmagnet_pct": 15.62
1222
+ },
1223
+ "Music": {
1224
+ "n": 140,
1225
+ "armiger_ultra_pct": 90.71,
1226
+ "armiger_int8_pct": 90.71,
1227
+ "mimmo_bert_pct": 77.86,
1228
+ "bitmagnet_pct": 7.86
1229
+ },
1230
+ "Game": {
1231
+ "n": 61,
1232
+ "armiger_ultra_pct": 90.16,
1233
+ "armiger_int8_pct": 90.16,
1234
+ "mimmo_bert_pct": 0.0,
1235
+ "bitmagnet_pct": 0.0
1236
+ },
1237
+ "Ebook": {
1238
+ "n": 25,
1239
+ "armiger_ultra_pct": 88.0,
1240
+ "armiger_int8_pct": 88.0,
1241
+ "mimmo_bert_pct": 80.0,
1242
+ "bitmagnet_pct": 4.0
1243
+ },
1244
+ "Course": {
1245
+ "n": 33,
1246
+ "armiger_ultra_pct": 81.82,
1247
+ "armiger_int8_pct": 81.82,
1248
+ "mimmo_bert_pct": 0.0,
1249
+ "bitmagnet_pct": 0.0
1250
+ },
1251
+ "Comic": {
1252
+ "n": 99,
1253
+ "armiger_ultra_pct": 66.67,
1254
+ "armiger_int8_pct": 66.67,
1255
+ "mimmo_bert_pct": 0.0,
1256
+ "bitmagnet_pct": 0.0
1257
+ }
1258
+ },
1259
+ "right_beyond_confirmation_n1025_loto": {
1260
+ "Audiobook": {
1261
+ "n": 75,
1262
+ "armiger_ultra_pct": 100.0,
1263
+ "armiger_int8_pct": 100.0,
1264
+ "gpt_noncirc_pct": 98.67,
1265
+ "gemini_noncirc_pct": 98.0
1266
+ },
1267
+ "Ebook": {
1268
+ "n": 125,
1269
+ "armiger_ultra_pct": 98.4,
1270
+ "armiger_int8_pct": 98.4,
1271
+ "gpt_noncirc_pct": 96.0,
1272
+ "gemini_noncirc_pct": 96.8
1273
+ },
1274
+ "Comic": {
1275
+ "n": 120,
1276
+ "armiger_ultra_pct": 98.33,
1277
+ "armiger_int8_pct": 98.33,
1278
+ "gpt_noncirc_pct": 96.67,
1279
+ "gemini_noncirc_pct": 95.0
1280
+ },
1281
+ "Movie": {
1282
+ "n": 95,
1283
+ "armiger_ultra_pct": 96.84,
1284
+ "armiger_int8_pct": 96.84,
1285
+ "gpt_noncirc_pct": 94.21,
1286
+ "gemini_noncirc_pct": 92.63
1287
+ },
1288
+ "TV Show": {
1289
+ "n": 136,
1290
+ "armiger_ultra_pct": 96.32,
1291
+ "armiger_int8_pct": 96.32,
1292
+ "gpt_noncirc_pct": 97.79,
1293
+ "gemini_noncirc_pct": 93.75
1294
+ },
1295
+ "Game": {
1296
+ "n": 135,
1297
+ "armiger_ultra_pct": 96.3,
1298
+ "armiger_int8_pct": 96.3,
1299
+ "gpt_noncirc_pct": 98.52,
1300
+ "gemini_noncirc_pct": 98.52
1301
+ },
1302
+ "Software": {
1303
+ "n": 37,
1304
+ "armiger_ultra_pct": 94.59,
1305
+ "armiger_int8_pct": 94.59,
1306
+ "gpt_noncirc_pct": 97.3,
1307
+ "gemini_noncirc_pct": 77.03
1308
+ },
1309
+ "Course": {
1310
+ "n": 100,
1311
+ "armiger_ultra_pct": 92.0,
1312
+ "armiger_int8_pct": 92.0,
1313
+ "gpt_noncirc_pct": 95.5,
1314
+ "gemini_noncirc_pct": 97.5
1315
+ },
1316
+ "Adult / XXX": {
1317
+ "n": 174,
1318
+ "armiger_ultra_pct": 91.95,
1319
+ "armiger_int8_pct": 91.95,
1320
+ "gpt_noncirc_pct": 94.83,
1321
+ "gemini_noncirc_pct": 95.69
1322
+ },
1323
+ "Music": {
1324
+ "n": 12,
1325
+ "armiger_ultra_pct": 83.33,
1326
+ "armiger_int8_pct": 83.33,
1327
+ "gpt_noncirc_pct": 100.0,
1328
+ "gemini_noncirc_pct": 100.0
1329
+ }
1330
+ }
1331
+ }
1332
+ },
1333
+ "five_split_category_summary": {
1334
+ "oct15_confirmation": {
1335
+ "panel_name": "five_split_oct15",
1336
+ "n": 2835,
1337
+ "metric_type": "mean",
1338
+ "reference_model": "armiger_ultra_hyb",
1339
+ "models": {
1340
+ "armiger_ultra_hyb": {
1341
+ "n": 2835,
1342
+ "point_pct": 95.49,
1343
+ "ci95_lo": 94.71,
1344
+ "ci95_hi": 96.23,
1345
+ "delta_vs_ref_pct": 0.0,
1346
+ "delta_ci95_lo": 0.0,
1347
+ "delta_ci95_hi": 0.0,
1348
+ "p_value_vs_ref": 1.0
1349
+ },
1350
+ "armiger_ultra_raw": {
1351
+ "n": 2835,
1352
+ "point_pct": 95.03,
1353
+ "ci95_lo": 94.22,
1354
+ "ci95_hi": 95.84,
1355
+ "delta_vs_ref_pct": 0.46,
1356
+ "delta_ci95_lo": 0.14,
1357
+ "delta_ci95_hi": 0.78,
1358
+ "p_value_vs_ref": 0.00719738
1359
+ },
1360
+ "armiger_int8_hyb": {
1361
+ "n": 2835,
1362
+ "point_pct": 95.49,
1363
+ "ci95_lo": 94.71,
1364
+ "ci95_hi": 96.23,
1365
+ "delta_vs_ref_pct": 0.0,
1366
+ "delta_ci95_lo": 0.0,
1367
+ "delta_ci95_hi": 0.0,
1368
+ "p_value_vs_ref": 1.0
1369
+ },
1370
+ "armiger_int8_raw": {
1371
+ "n": 2835,
1372
+ "point_pct": 95.06,
1373
+ "ci95_lo": 94.25,
1374
+ "ci95_hi": 95.87,
1375
+ "delta_vs_ref_pct": 0.42,
1376
+ "delta_ci95_lo": 0.14,
1377
+ "delta_ci95_hi": 0.74,
1378
+ "p_value_vs_ref": 0.01181793
1379
+ }
1380
+ }
1381
+ },
1382
+ "fresh_confirmation": {
1383
+ "panel_name": "five_split_fresh",
1384
+ "n": 1402,
1385
+ "metric_type": "mean",
1386
+ "reference_model": "armiger_ultra_hyb",
1387
+ "models": {
1388
+ "armiger_ultra_hyb": {
1389
+ "n": 1402,
1390
+ "point_pct": 94.58,
1391
+ "ci95_lo": 93.37,
1392
+ "ci95_hi": 95.72,
1393
+ "delta_vs_ref_pct": 0.0,
1394
+ "delta_ci95_lo": 0.0,
1395
+ "delta_ci95_hi": 0.0,
1396
+ "p_value_vs_ref": 1.0
1397
+ },
1398
+ "armiger_ultra_raw": {
1399
+ "n": 1402,
1400
+ "point_pct": 94.51,
1401
+ "ci95_lo": 93.3,
1402
+ "ci95_hi": 95.65,
1403
+ "delta_vs_ref_pct": 0.07,
1404
+ "delta_ci95_lo": -0.21,
1405
+ "delta_ci95_hi": 0.36,
1406
+ "p_value_vs_ref": 1.0
1407
+ },
1408
+ "armiger_int8_hyb": {
1409
+ "n": 1402,
1410
+ "point_pct": 94.58,
1411
+ "ci95_lo": 93.37,
1412
+ "ci95_hi": 95.72,
1413
+ "delta_vs_ref_pct": 0.0,
1414
+ "delta_ci95_lo": 0.0,
1415
+ "delta_ci95_hi": 0.0,
1416
+ "p_value_vs_ref": 1.0
1417
+ },
1418
+ "armiger_int8_raw": {
1419
+ "n": 1402,
1420
+ "point_pct": 94.51,
1421
+ "ci95_lo": 93.3,
1422
+ "ci95_hi": 95.65,
1423
+ "delta_vs_ref_pct": 0.07,
1424
+ "delta_ci95_lo": -0.21,
1425
+ "delta_ci95_hi": 0.36,
1426
+ "p_value_vs_ref": 1.0
1427
+ }
1428
+ }
1429
+ },
1430
+ "beyond_confirmation": {
1431
+ "panel_name": "five_split_beyond",
1432
+ "n": 1025,
1433
+ "metric_type": "mean",
1434
+ "reference_model": "armiger_ultra_hyb",
1435
+ "models": {
1436
+ "armiger_ultra_hyb": {
1437
+ "n": 1025,
1438
+ "point_pct": 95.12,
1439
+ "ci95_lo": 93.76,
1440
+ "ci95_hi": 96.39,
1441
+ "delta_vs_ref_pct": 0.0,
1442
+ "delta_ci95_lo": 0.0,
1443
+ "delta_ci95_hi": 0.0,
1444
+ "p_value_vs_ref": 1.0
1445
+ },
1446
+ "armiger_ultra_raw": {
1447
+ "n": 1025,
1448
+ "point_pct": 92.88,
1449
+ "ci95_lo": 91.22,
1450
+ "ci95_hi": 94.44,
1451
+ "delta_vs_ref_pct": 2.24,
1452
+ "delta_ci95_lo": 1.37,
1453
+ "delta_ci95_hi": 3.22,
1454
+ "p_value_vs_ref": 2.384186e-07
1455
+ },
1456
+ "armiger_int8_hyb": {
1457
+ "n": 1025,
1458
+ "point_pct": 95.12,
1459
+ "ci95_lo": 93.76,
1460
+ "ci95_hi": 96.39,
1461
+ "delta_vs_ref_pct": 0.0,
1462
+ "delta_ci95_lo": 0.0,
1463
+ "delta_ci95_hi": 0.0,
1464
+ "p_value_vs_ref": 1.0
1465
+ },
1466
+ "armiger_int8_raw": {
1467
+ "n": 1025,
1468
+ "point_pct": 92.88,
1469
+ "ci95_lo": 91.22,
1470
+ "ci95_hi": 94.44,
1471
+ "delta_vs_ref_pct": 2.24,
1472
+ "delta_ci95_lo": 1.37,
1473
+ "delta_ci95_hi": 3.22,
1474
+ "p_value_vs_ref": 2.384186e-07
1475
+ }
1476
+ }
1477
+ },
1478
+ "tc_truespec_2000": {
1479
+ "panel_name": "five_split_tc2000",
1480
+ "n": 2000,
1481
+ "metric_type": "mean",
1482
+ "reference_model": "armiger_ultra_hyb",
1483
+ "models": {
1484
+ "armiger_ultra_hyb": {
1485
+ "n": 2000,
1486
+ "point_pct": 91.5,
1487
+ "ci95_lo": 90.3,
1488
+ "ci95_hi": 92.65,
1489
+ "delta_vs_ref_pct": 0.0,
1490
+ "delta_ci95_lo": 0.0,
1491
+ "delta_ci95_hi": 0.0,
1492
+ "p_value_vs_ref": 1.0
1493
+ },
1494
+ "armiger_ultra_raw": {
1495
+ "n": 2000,
1496
+ "point_pct": 87.4,
1497
+ "ci95_lo": 85.95,
1498
+ "ci95_hi": 88.8,
1499
+ "delta_vs_ref_pct": 4.1,
1500
+ "delta_ci95_lo": 2.95,
1501
+ "delta_ci95_hi": 5.25,
1502
+ "p_value_vs_ref": 1.091317e-12
1503
+ },
1504
+ "armiger_int8_hyb": {
1505
+ "n": 2000,
1506
+ "point_pct": 91.4,
1507
+ "ci95_lo": 90.2,
1508
+ "ci95_hi": 92.55,
1509
+ "delta_vs_ref_pct": 0.1,
1510
+ "delta_ci95_lo": 0.0,
1511
+ "delta_ci95_hi": 0.25,
1512
+ "p_value_vs_ref": 0.5
1513
+ },
1514
+ "armiger_int8_raw": {
1515
+ "n": 2000,
1516
+ "point_pct": 87.3,
1517
+ "ci95_lo": 85.85,
1518
+ "ci95_hi": 88.7,
1519
+ "delta_vs_ref_pct": 4.2,
1520
+ "delta_ci95_lo": 3.05,
1521
+ "delta_ci95_hi": 5.35,
1522
+ "p_value_vs_ref": 4.246281e-13
1523
+ }
1524
+ }
1525
+ },
1526
+ "tc_truespec_multilang_3000": {
1527
+ "panel_name": "five_split_tc3000",
1528
+ "n": 3000,
1529
+ "metric_type": "mean",
1530
+ "reference_model": "armiger_ultra_hyb",
1531
+ "models": {
1532
+ "armiger_ultra_hyb": {
1533
+ "n": 3000,
1534
+ "point_pct": 98.47,
1535
+ "ci95_lo": 98.0,
1536
+ "ci95_hi": 98.9,
1537
+ "delta_vs_ref_pct": 0.0,
1538
+ "delta_ci95_lo": 0.0,
1539
+ "delta_ci95_hi": 0.0,
1540
+ "p_value_vs_ref": 1.0
1541
+ },
1542
+ "armiger_ultra_raw": {
1543
+ "n": 3000,
1544
+ "point_pct": 97.5,
1545
+ "ci95_lo": 96.9,
1546
+ "ci95_hi": 98.03,
1547
+ "delta_vs_ref_pct": 0.97,
1548
+ "delta_ci95_lo": 0.57,
1549
+ "delta_ci95_hi": 1.4,
1550
+ "p_value_vs_ref": 4.873598e-06
1551
+ },
1552
+ "armiger_int8_hyb": {
1553
+ "n": 3000,
1554
+ "point_pct": 98.43,
1555
+ "ci95_lo": 97.97,
1556
+ "ci95_hi": 98.87,
1557
+ "delta_vs_ref_pct": 0.03,
1558
+ "delta_ci95_lo": 0.0,
1559
+ "delta_ci95_hi": 0.1,
1560
+ "p_value_vs_ref": 1.0
1561
+ },
1562
+ "armiger_int8_raw": {
1563
+ "n": 3000,
1564
+ "point_pct": 97.43,
1565
+ "ci95_lo": 96.83,
1566
+ "ci95_hi": 97.97,
1567
+ "delta_vs_ref_pct": 1.03,
1568
+ "delta_ci95_lo": 0.63,
1569
+ "delta_ci95_hi": 1.47,
1570
+ "p_value_vs_ref": 1.636124e-06
1571
+ }
1572
+ }
1573
+ }
1574
+ }
1575
+ }
benchmarks/generate_canonical_charts.py ADDED
@@ -0,0 +1,1541 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """
3
+ Deterministic, Statistically Rooted Benchmark & Chart Generation Pipeline
4
+ for Armiger & Armiger Ultra.
5
+
6
+ Enforces 4 hard statistical invariants before writing any chart:
7
+ 1. Identical Denominator Lock (`assert_identical_denominator`)
8
+ 2. Anti-Circularity Lock (`assert_non_circular_teacher_eval`)
9
+ 3. Paired Uncertainty & Significance (10,000-resample paired bootstrap 95% CIs
10
+ with fixed seed 20261018 + exact two-sided McNemar / permutation p-values)
11
+ 4. Cryptographic Provenance Manifest (`CHARTS_MANIFEST.json` + stripped PNG
12
+ chunks containing only `IHDR`, `IDAT`, `IEND`).
13
+ """
14
+ from __future__ import annotations
15
+
16
+ import argparse
17
+ import hashlib
18
+ import io
19
+ import json
20
+ import pathlib
21
+ import struct
22
+ import zlib
23
+ from typing import Any, Dict, List, Optional, Tuple
24
+
25
+ import math
26
+ import numpy as np
27
+
28
+ try:
29
+ import matplotlib
30
+ matplotlib.use("Agg")
31
+ import matplotlib.pyplot as plt
32
+ from matplotlib.gridspec import GridSpec
33
+ from matplotlib.patches import Patch
34
+ from PIL import Image
35
+ HAS_MATPLOTLIB = True
36
+ except ImportError:
37
+ matplotlib = None # type: ignore[assignment]
38
+ plt = None # type: ignore[assignment]
39
+ GridSpec = None # type: ignore[assignment]
40
+ Patch = None # type: ignore[assignment]
41
+ Image = None # type: ignore[assignment]
42
+ HAS_MATPLOTLIB = False
43
+
44
+ BOOTSTRAP_SEED = 20261018
45
+ BOOTSTRAP_RESAMPLES = 10000
46
+
47
+ BG_CANVAS = "#0D1117"
48
+ BG_PANEL = "#161B22"
49
+ BG_HIGHLIGHT = "#0F2922"
50
+ BORDER_COLOR = "#30363D"
51
+ GRID_COLOR = "#21262D"
52
+ TEXT_MAIN = "#F0F6FC"
53
+ TEXT_SUB = "#8B949E"
54
+ TEXT_MUTED = "#6E7681"
55
+
56
+ C_ULTRA = "#10B981"
57
+ C_ULTRA_EDGE = "#34D399"
58
+ C_INT8 = "#3B82F6"
59
+ C_INT8_EDGE = "#60A5FA"
60
+ C_GEMINI = "#F59E0B"
61
+ C_CLAUDE = "#F97316"
62
+ C_GPT = "#38BDF8"
63
+ C_VINCI = "#A855F7"
64
+ C_ERR = "#E2E8F0"
65
+
66
+ TEACHER_MODELS = {
67
+ "OpenAI GPT (Fast Tier)",
68
+ "Google Gemini Flash",
69
+ "Anthropic Claude Haiku 4.5",
70
+ }
71
+
72
+
73
+ # ==============================================================================
74
+ # 1. HARD STATISTICAL INVARIANT LOCKS
75
+ # ==============================================================================
76
+ def assert_identical_denominator(
77
+ panel_name: str,
78
+ vectors: Dict[str, np.ndarray],
79
+ expected_n: Optional[int] = None,
80
+ ) -> int:
81
+ """Enforce that every model on a panel is evaluated on the exact same item count N
82
+ and valid-domain mask, with zero NaNs."""
83
+ if not vectors:
84
+ raise AssertionError(f"[{panel_name}] Empty vectors dictionary.")
85
+ lengths = {}
86
+ for model_name, vec in vectors.items():
87
+ arr = np.asarray(vec)
88
+ if arr.ndim != 1:
89
+ raise AssertionError(f"[{panel_name}] Model '{model_name}' vector must be 1D, got shape {arr.shape}.")
90
+ if np.isnan(arr).any():
91
+ raise AssertionError(f"[{panel_name}] Model '{model_name}' contains NaN values.")
92
+ lengths[model_name] = int(arr.shape[0])
93
+
94
+ unique_lengths = set(lengths.values())
95
+ if len(unique_lengths) != 1:
96
+ raise AssertionError(
97
+ f"[{panel_name}] Identical Denominator Lock violated! Mismatched N across models: {lengths}"
98
+ )
99
+ n_val = next(iter(unique_lengths))
100
+ if expected_n is not None and n_val != expected_n:
101
+ raise AssertionError(
102
+ f"[{panel_name}] Expected N={expected_n}, but got N={n_val} ({lengths})."
103
+ )
104
+ return n_val
105
+
106
+
107
+ def assert_non_circular_teacher_eval(
108
+ model_name: str,
109
+ eval_mode: str,
110
+ self_vote_weight: float,
111
+ noncirc_vec: Optional[np.ndarray] = None,
112
+ circ_vec: Optional[np.ndarray] = None,
113
+ ) -> Dict[str, Any]:
114
+ """Enforce that teacher LLMs are never scored against a reference key in which
115
+ they cast a deciding self-vote."""
116
+ is_teacher = any(t in model_name for t in TEACHER_MODELS)
117
+ if is_teacher:
118
+ if self_vote_weight != 0.0 or eval_mode == "circular_majority":
119
+ raise AssertionError(
120
+ f"Anti-Circularity Lock violated for teacher '{model_name}': "
121
+ f"eval_mode='{eval_mode}', self_vote_weight={self_vote_weight} (must be 0.0)."
122
+ )
123
+ if eval_mode not in ("leave_one_teacher_out", "independent_ground_truth"):
124
+ raise AssertionError(
125
+ f"Invalid eval_mode '{eval_mode}' for teacher '{model_name}'."
126
+ )
127
+ info: Dict[str, Any] = {
128
+ "model_name": model_name,
129
+ "is_teacher": is_teacher,
130
+ "eval_mode": eval_mode,
131
+ "self_vote_weight": float(self_vote_weight),
132
+ }
133
+ if noncirc_vec is not None and circ_vec is not None:
134
+ assert_identical_denominator(f"anti_circ_{model_name}", {"noncirc": noncirc_vec, "circ": circ_vec})
135
+ nc_pct = float(np.mean(noncirc_vec) * 100.0)
136
+ c_pct = float(np.mean(circ_vec) * 100.0)
137
+ info["non_circular_pct"] = round(nc_pct, 2)
138
+ info["circular_self_vote_pct"] = round(c_pct, 2)
139
+ info["circularity_inflation_pp"] = round(c_pct - nc_pct, 2)
140
+ return info
141
+
142
+
143
+ def exact_mcnemar_or_paired_p(vec_ref: np.ndarray, vec_mod: np.ndarray) -> float:
144
+ """Compute exact two-sided McNemar binomial p-value for binary vectors, or exact
145
+ paired sign-flip binomial test on discordant items for fractional LOTO vectors."""
146
+ diff = np.asarray(vec_ref, dtype=np.float64) - np.asarray(vec_mod, dtype=np.float64)
147
+ n_pos = int(np.sum(diff > 1e-6))
148
+ n_neg = int(np.sum(diff < -1e-6))
149
+ n_disc = n_pos + n_neg
150
+ if n_disc == 0:
151
+ return 1.0
152
+ m_min = min(n_pos, n_neg)
153
+ if 2 * m_min == n_disc:
154
+ return 1.0
155
+ tail_sum = sum(math.comb(n_disc, i) for i in range(m_min + 1))
156
+ return float(min(1.0, (2 * tail_sum) / (1 << n_disc)))
157
+
158
+
159
+ def compute_auroc_pct(y_true: np.ndarray, scores: np.ndarray) -> float:
160
+ pos = scores[y_true == 1]
161
+ neg = scores[y_true == 0]
162
+ u = (pos[:, None] > neg[None, :]).sum() + 0.5 * (pos[:, None] == neg[None, :]).sum()
163
+ return float(100.0 * u / (len(pos) * len(neg)))
164
+
165
+
166
+ def compute_paired_panel_stats(
167
+ panel_name: str,
168
+ vectors: Dict[str, np.ndarray],
169
+ ref_model: str,
170
+ expected_n: int,
171
+ metric_type: str = "mean",
172
+ y_true: Optional[np.ndarray] = None,
173
+ n_boot: int = BOOTSTRAP_RESAMPLES,
174
+ seed: int = BOOTSTRAP_SEED,
175
+ ) -> Dict[str, Any]:
176
+ """Compute point estimates, 10,000-resample paired bootstrap 95% CIs, paired deltas,
177
+ and McNemar / permutation p-values for all models in a panel."""
178
+ n_val = assert_identical_denominator(panel_name, vectors, expected_n=expected_n)
179
+ if ref_model not in vectors:
180
+ raise AssertionError(f"[{panel_name}] Reference model '{ref_model}' missing from vectors.")
181
+
182
+ rng = np.random.default_rng(seed)
183
+ model_names = list(vectors.keys())
184
+ mat = np.stack([np.asarray(vectors[m], dtype=np.float64) for m in model_names], axis=0) # (M, N)
185
+
186
+ if metric_type == "mean":
187
+ # Exact integer-scaled dot product in float64 (bit-exact across ARM64 Accelerate & x86_64 OpenBLAS)
188
+ mat_int6 = np.round(mat * 6.0)
189
+ weights = rng.multinomial(n_val, np.full(n_val, 1.0 / n_val), size=n_boot).astype(np.float64)
190
+ boot_scores = (weights @ mat_int6.T) * (100.0 / (6.0 * n_val))
191
+ point_scores = {m: float(np.sum(np.round(np.asarray(vectors[m], dtype=np.float64) * 6.0)) * 100.0 / (6.0 * n_val)) for m in model_names}
192
+ item_correctness = {m: np.asarray(vectors[m], dtype=np.float64) for m in model_names}
193
+ elif metric_type == "balanced":
194
+ if y_true is None:
195
+ raise AssertionError(f"[{panel_name}] y_true required for metric_type='balanced'.")
196
+ y_arr = np.asarray(y_true)
197
+ pos_mask = (y_arr == 1)
198
+ neg_mask = (y_arr == 0)
199
+ n_pos = int(np.sum(pos_mask))
200
+ n_neg = int(np.sum(neg_mask))
201
+ pos_mat = (mat[:, pos_mask] == 1).astype(np.float64)
202
+ neg_mat = (mat[:, neg_mask] == 0).astype(np.float64)
203
+ w_pos = rng.multinomial(n_pos, np.full(n_pos, 1.0 / n_pos), size=n_boot).astype(np.float64)
204
+ w_neg = rng.multinomial(n_neg, np.full(n_neg, 1.0 / n_neg), size=n_boot).astype(np.float64)
205
+ boot_scores = 0.5 * ((w_pos @ pos_mat.T) * (100.0 / n_pos) + (w_neg @ neg_mat.T) * (100.0 / n_neg))
206
+ point_scores = {}
207
+ item_correctness = {}
208
+ for idx, m in enumerate(model_names):
209
+ sens = float(np.sum(pos_mat[idx]) * 100.0 / n_pos)
210
+ spec = float(np.sum(neg_mat[idx]) * 100.0 / n_neg)
211
+ point_scores[m] = 0.5 * (sens + spec)
212
+ item_corr = np.zeros(n_val, dtype=np.float64)
213
+ item_corr[pos_mask] = pos_mat[idx]
214
+ item_corr[neg_mask] = neg_mat[idx]
215
+ item_correctness[m] = item_corr
216
+ elif metric_type == "auroc":
217
+ if y_true is None:
218
+ raise AssertionError(f"[{panel_name}] y_true required for metric_type='auroc'.")
219
+ y_arr = np.asarray(y_true)
220
+ pos_idx = np.where(y_arr == 1)[0]
221
+ neg_idx = np.where(y_arr == 0)[0]
222
+ n_pos, n_neg = len(pos_idx), len(neg_idx)
223
+ # Exact integer-scaled placement counts in float64 (bit-exact across BLAS engines)
224
+ placement_int2 = np.zeros((len(model_names), n_val), dtype=np.float64)
225
+ placement_unit = np.zeros((len(model_names), n_val), dtype=np.float64)
226
+ point_scores = {}
227
+ for idx, m in enumerate(model_names):
228
+ s = mat[idx]
229
+ s_pos = s[pos_idx]
230
+ s_neg = s[neg_idx]
231
+ v10_int2 = 2.0 * (s_pos[:, None] > s_neg[None, :]).sum(axis=1) + (s_pos[:, None] == s_neg[None, :]).sum(axis=1)
232
+ v01_int2 = 2.0 * (s_pos[:, None] > s_neg[None, :]).sum(axis=0) + (s_pos[:, None] == s_neg[None, :]).sum(axis=0)
233
+ placement_int2[idx, pos_idx] = v10_int2
234
+ placement_int2[idx, neg_idx] = v01_int2
235
+ placement_unit[idx, pos_idx] = v10_int2 / (2.0 * n_neg)
236
+ placement_unit[idx, neg_idx] = v01_int2 / (2.0 * n_pos)
237
+ point_scores[m] = float(np.sum(v10_int2) * 100.0 / (2.0 * n_pos * n_neg))
238
+ w_pos = rng.multinomial(n_pos, np.full(n_pos, 1.0 / n_pos), size=n_boot).astype(np.float64)
239
+ w_neg = rng.multinomial(n_neg, np.full(n_neg, 1.0 / n_neg), size=n_boot).astype(np.float64)
240
+ pt_vec = np.array([point_scores[m] for m in model_names], dtype=np.float64)[None, :]
241
+ boot_scores = (
242
+ (w_pos @ placement_int2[:, pos_idx].T) * (100.0 / (2.0 * n_pos * n_neg))
243
+ + (w_neg @ placement_int2[:, neg_idx].T) * (100.0 / (2.0 * n_pos * n_neg))
244
+ - pt_vec
245
+ )
246
+ item_correctness = {m: placement_unit[idx] for idx, m in enumerate(model_names)}
247
+ else:
248
+ raise ValueError(f"Unknown metric_type: {metric_type}")
249
+
250
+ ref_idx = model_names.index(ref_model)
251
+ ref_boot = boot_scores[:, ref_idx]
252
+ ref_point = point_scores[ref_model]
253
+
254
+ results: Dict[str, Any] = {}
255
+ for idx, m in enumerate(model_names):
256
+ m_boot = boot_scores[:, idx]
257
+ ci_lo, ci_hi = np.percentile(m_boot, [2.5, 97.5])
258
+ delta_boot = ref_boot - m_boot
259
+ d_lo, d_hi = np.percentile(delta_boot, [2.5, 97.5])
260
+ p_val = 1.0 if m == ref_model else exact_mcnemar_or_paired_p(item_correctness[ref_model], item_correctness[m])
261
+ results[m] = {
262
+ "n": n_val,
263
+ "point_pct": round(point_scores[m], 2),
264
+ "ci95_lo": round(float(ci_lo), 2),
265
+ "ci95_hi": round(float(ci_hi), 2),
266
+ "delta_vs_ref_pct": round(ref_point - point_scores[m], 2),
267
+ "delta_ci95_lo": round(float(d_lo), 2),
268
+ "delta_ci95_hi": round(float(d_hi), 2),
269
+ "p_value_vs_ref": float(f"{p_val:.6e}"),
270
+ }
271
+ return {
272
+ "panel_name": panel_name,
273
+ "n": n_val,
274
+ "metric_type": metric_type,
275
+ "reference_model": ref_model,
276
+ "models": results,
277
+ }
278
+
279
+
280
+ # ==============================================================================
281
+ # 2. DETERMINISTIC PNG SERIALIZATION (STRIPPED TO IHDR, IDAT, IEND ONLY)
282
+ # ==============================================================================
283
+ def strip_png_chunks_to_canonical(png_bytes: bytes) -> bytes:
284
+ """Keep only IHDR, IDAT, and IEND chunks so the output PNG contains zero
285
+ timestamps, software tags, or host path metadata and is 100% deterministic."""
286
+ sig = b"\x89PNG\r\n\x1a\n"
287
+ if not png_bytes.startswith(sig):
288
+ raise ValueError("Invalid PNG header signature.")
289
+ pos = 8
290
+ out = bytearray(sig)
291
+ while pos < len(png_bytes):
292
+ length = struct.unpack(">I", png_bytes[pos:pos + 4])[0]
293
+ ctype = png_bytes[pos + 4:pos + 8]
294
+ chunk_data = png_bytes[pos + 8:pos + 8 + length]
295
+ crc = png_bytes[pos + 8 + length:pos + 12 + length]
296
+ pos += 12 + length
297
+ if ctype in (b"IHDR", b"IDAT", b"IEND"):
298
+ out.extend(struct.pack(">I", length))
299
+ out.extend(ctype)
300
+ out.extend(chunk_data)
301
+ out.extend(crc)
302
+ return bytes(out)
303
+
304
+
305
+ def verify_png_chunks_clean(png_path: pathlib.Path) -> List[str]:
306
+ data = png_path.read_bytes()
307
+ sig = b"\x89PNG\r\n\x1a\n"
308
+ assert data.startswith(sig), f"{png_path} is not a valid PNG"
309
+ pos = 8
310
+ chunks: List[str] = []
311
+ while pos < len(data):
312
+ length = struct.unpack(">I", data[pos:pos + 4])[0]
313
+ ctype = data[pos + 4:pos + 8].decode("ascii", errors="replace")
314
+ chunks.append(ctype)
315
+ pos += 12 + length
316
+ for c in chunks:
317
+ if c not in ("IHDR", "IDAT", "IEND"):
318
+ raise AssertionError(f"{png_path} contains non-canonical PNG chunk: {c}")
319
+ return chunks
320
+
321
+
322
+ def save_clean_rgb_png(fig: plt.Figure, out_path: pathlib.Path, dpi: int = 180) -> str:
323
+ out_path.parent.mkdir(parents=True, exist_ok=True)
324
+ buf = io.BytesIO()
325
+ fig.savefig(
326
+ buf,
327
+ format="png",
328
+ dpi=dpi,
329
+ bbox_inches="tight",
330
+ facecolor=BG_CANVAS,
331
+ edgecolor="none",
332
+ pad_inches=0.34,
333
+ )
334
+ plt.close(fig)
335
+ buf.seek(0)
336
+ with Image.open(buf) as im:
337
+ rgb = im.convert("RGB")
338
+ raw_buf = io.BytesIO()
339
+ rgb.save(raw_buf, format="PNG", optimize=False)
340
+ clean_bytes = strip_png_chunks_to_canonical(raw_buf.getvalue())
341
+ out_path.write_bytes(clean_bytes)
342
+ verify_png_chunks_clean(out_path)
343
+ return hashlib.sha256(clean_bytes).hexdigest()
344
+
345
+
346
+ def style_ax(ax: plt.Axes) -> None:
347
+ ax.set_facecolor(BG_PANEL)
348
+ for spine in ax.spines.values():
349
+ spine.set_color(BORDER_COLOR)
350
+ spine.set_linewidth(1.0)
351
+ ax.xaxis.grid(True, color=GRID_COLOR, linestyle="--", alpha=0.9, zorder=0)
352
+ ax.yaxis.grid(False)
353
+ ax.set_axisbelow(True)
354
+
355
+
356
+ def draw_ci_whiskers(ax: plt.Axes, y_coords: np.ndarray, stats_list: List[Dict[str, Any]]) -> None:
357
+ """Draw subtle 95% bootstrap confidence interval error bars on horizontal bars."""
358
+ pts = np.array([s["point_pct"] for s in stats_list], dtype=np.float64)
359
+ lo = np.maximum(0.0, pts - np.array([s["ci95_lo"] for s in stats_list], dtype=np.float64))
360
+ hi = np.maximum(0.0, np.array([s["ci95_hi"] for s in stats_list], dtype=np.float64) - pts)
361
+ ax.errorbar(
362
+ pts,
363
+ y_coords,
364
+ xerr=[lo, hi],
365
+ fmt="none",
366
+ ecolor=C_ERR,
367
+ elinewidth=1.15,
368
+ capsize=2.6,
369
+ capthick=1.1,
370
+ alpha=0.82,
371
+ zorder=5,
372
+ )
373
+
374
+
375
+ # ==============================================================================
376
+ # 3. CHART RENDERERS
377
+ # ==============================================================================
378
+ def build_all_charts_and_manifest(
379
+ npz_path: pathlib.Path,
380
+ assets_dir: pathlib.Path,
381
+ manifest_path: pathlib.Path,
382
+ render_pngs: Optional[bool] = None,
383
+ ) -> Dict[str, Any]:
384
+ if render_pngs is None:
385
+ render_pngs = HAS_MATPLOTLIB
386
+ if render_pngs:
387
+ if not HAS_MATPLOTLIB:
388
+ raise RuntimeError("matplotlib and Pillow are required when render_pngs=True")
389
+ plt.rcParams.update({
390
+ "figure.facecolor": BG_CANVAS,
391
+ "axes.facecolor": BG_PANEL,
392
+ "axes.edgecolor": BORDER_COLOR,
393
+ "axes.labelcolor": TEXT_SUB,
394
+ "xtick.color": TEXT_SUB,
395
+ "ytick.color": TEXT_MAIN,
396
+ "text.color": TEXT_MAIN,
397
+ "font.sans-serif": ["DejaVu Sans", "Arial", "Helvetica"],
398
+ "axes.grid": True,
399
+ "grid.color": GRID_COLOR,
400
+ "grid.linestyle": "--",
401
+ "grid.alpha": 0.85,
402
+ })
403
+
404
+ if str(npz_path).endswith(".json"):
405
+ raw_json = json.loads(npz_path.read_text(encoding="utf-8"))
406
+ dtypes_map = raw_json["_meta"]["dtypes"]
407
+ d = {k: np.array(v, dtype=np.dtype(dtypes_map[k])) for k, v in raw_json["vectors"].items()}
408
+ files_list = list(d.keys())
409
+ else:
410
+ npz_obj = np.load(npz_path)
411
+ files_list = list(npz_obj.files)
412
+ d = {k: npz_obj[k] for k in files_list}
413
+ for k in files_list:
414
+ if d[k].dtype.kind not in ("i", "u", "f"):
415
+ raise AssertionError(f"Non-numeric array '{k}' ({d[k].dtype}) in {npz_path}")
416
+
417
+ # --------------------------------------------------------------------------
418
+ # Anti-Circularity Lock checks on teacher models
419
+ # --------------------------------------------------------------------------
420
+ anti_circ_checks = [
421
+ assert_non_circular_teacher_eval(
422
+ "OpenAI GPT (Fast Tier)",
423
+ eval_mode="leave_one_teacher_out",
424
+ self_vote_weight=0.0,
425
+ noncirc_vec=d["beyond_cat11_corr_gpt_noncirc"],
426
+ circ_vec=d["beyond_cat11_corr_gpt_circ"],
427
+ ),
428
+ assert_non_circular_teacher_eval(
429
+ "Google Gemini Flash",
430
+ eval_mode="leave_one_teacher_out",
431
+ self_vote_weight=0.0,
432
+ noncirc_vec=d["beyond_cat11_corr_gemini_noncirc"],
433
+ circ_vec=d["beyond_cat11_corr_gemini_circ"],
434
+ ),
435
+ assert_non_circular_teacher_eval(
436
+ "Anthropic Claude Haiku 4.5",
437
+ eval_mode="independent_ground_truth",
438
+ self_vote_weight=0.0,
439
+ ),
440
+ ]
441
+
442
+ # --------------------------------------------------------------------------
443
+ # Compute Suite B (Chart 1) Panel Statistics with Identical-N Locks
444
+ # --------------------------------------------------------------------------
445
+ llm_order = [
446
+ "Armiger Ultra (Fair Hybrid)",
447
+ "Armiger (8.3 MB INT8 + Plugins)",
448
+ "Google Gemini Flash",
449
+ "Anthropic Claude Haiku 4.5",
450
+ "OpenAI GPT (Fast Tier)",
451
+ "Local 8B Instruct LLM (FP16)",
452
+ ]
453
+ ref_llm = "Armiger Ultra (Fair Hybrid)"
454
+
455
+ p2a_stats = compute_paired_panel_stats(
456
+ "chart1_2a_beyond_cat11",
457
+ {
458
+ llm_order[0]: d["beyond_cat11_corr_ultra"],
459
+ llm_order[1]: d["beyond_cat11_corr_int8_hyb"],
460
+ llm_order[2]: d["beyond_cat11_corr_gemini_noncirc"],
461
+ llm_order[3]: d["beyond_cat11_corr_claude"],
462
+ llm_order[4]: d["beyond_cat11_corr_gpt_noncirc"],
463
+ llm_order[5]: d["beyond_cat11_corr_vinci"],
464
+ },
465
+ ref_model=ref_llm,
466
+ expected_n=1025,
467
+ metric_type="mean",
468
+ )
469
+
470
+ p2b_stats = compute_paired_panel_stats(
471
+ "chart1_2b_beyond_matching",
472
+ {
473
+ llm_order[0]: d["beyond_match_corr_ultra"],
474
+ llm_order[1]: d["beyond_match_corr_int8_hyb"],
475
+ llm_order[2]: d["beyond_match_corr_gemini_noncirc"],
476
+ llm_order[3]: d["beyond_match_corr_claude"],
477
+ llm_order[4]: d["beyond_match_corr_gpt_noncirc"],
478
+ llm_order[5]: d["beyond_match_corr_vinci"],
479
+ },
480
+ ref_model=ref_llm,
481
+ expected_n=1025,
482
+ metric_type="mean",
483
+ )
484
+
485
+ p2c_stats = compute_paired_panel_stats(
486
+ "chart1_2c_beyond_audio_strict",
487
+ {
488
+ llm_order[0]: d["beyond_aud229_corr_ultra"],
489
+ llm_order[1]: d["beyond_aud229_corr_int8_hyb"],
490
+ llm_order[2]: d["beyond_aud229_corr_gemini"],
491
+ llm_order[3]: d["beyond_aud229_corr_claude"],
492
+ llm_order[4]: d["beyond_aud229_corr_gpt"],
493
+ llm_order[5]: d["beyond_aud229_corr_vinci"],
494
+ },
495
+ ref_model=ref_llm,
496
+ expected_n=229,
497
+ metric_type="mean",
498
+ )
499
+
500
+ p2d_stats = compute_paired_panel_stats(
501
+ "chart1_2d_beyond_anime_valid636",
502
+ {
503
+ llm_order[0]: d["beyond_ani636_corr_ultra"],
504
+ llm_order[1]: d["beyond_ani636_corr_int8"],
505
+ llm_order[2]: d["beyond_ani636_corr_gemini"],
506
+ llm_order[3]: d["beyond_ani636_corr_claude"],
507
+ llm_order[4]: d["beyond_ani636_corr_gpt"],
508
+ llm_order[5]: d["beyond_ani636_corr_vinci"],
509
+ },
510
+ ref_model=ref_llm,
511
+ expected_n=636,
512
+ metric_type="mean",
513
+ )
514
+
515
+ p2e_stats = compute_paired_panel_stats(
516
+ "chart1_2e_beyond_junk977_balanced",
517
+ {
518
+ llm_order[0]: d["beyond_junk977_pred_ultra"],
519
+ llm_order[1]: d["beyond_junk977_pred_int8"],
520
+ llm_order[2]: d["beyond_junk977_pred_gemini"],
521
+ llm_order[3]: d["beyond_junk977_pred_claude"],
522
+ llm_order[4]: d["beyond_junk977_pred_gpt"],
523
+ llm_order[5]: d["beyond_junk977_pred_vinci"],
524
+ },
525
+ ref_model=ref_llm,
526
+ expected_n=977,
527
+ metric_type="balanced",
528
+ y_true=d["beyond_junk977_y_true"],
529
+ )
530
+
531
+ p2f_pub_stats = compute_paired_panel_stats(
532
+ "chart1_2f_swarm_public_757",
533
+ {
534
+ llm_order[0]: d["swarm_pub_score_ultra"],
535
+ llm_order[1]: d["swarm_pub_score_int8"],
536
+ llm_order[2]: d["swarm_pub_score_gemini"],
537
+ llm_order[3]: d["swarm_pub_score_claude"],
538
+ llm_order[4]: d["swarm_pub_score_gpt"],
539
+ llm_order[5]: d["swarm_pub_score_vinci"],
540
+ },
541
+ ref_model=ref_llm,
542
+ expected_n=757,
543
+ metric_type="auroc",
544
+ y_true=d["swarm_pub_y_true"],
545
+ )
546
+
547
+ p2f_whse_stats = compute_paired_panel_stats(
548
+ "chart1_2f_swarm_warehouse_10278",
549
+ {
550
+ "Armiger Ultra (Fair Hybrid)": d["swarm_whse_score_ultra"],
551
+ "Armiger 4-Way Priors": d["swarm_whse_score_priors"],
552
+ "Raw Seeder Count": d["swarm_whse_score_seeders"],
553
+ },
554
+ ref_model="Armiger Ultra (Fair Hybrid)",
555
+ expected_n=10278,
556
+ metric_type="auroc",
557
+ y_true=d["swarm_whse_y_true"],
558
+ )
559
+
560
+ # Composite 6-dimension mean across 2A..2F on strict identical cohorts (2F = Public N=757 for all models)
561
+ composite_stats: Dict[str, Any] = {}
562
+ for m in llm_order:
563
+ s_2a = p2a_stats["models"][m]
564
+ s_2b = p2b_stats["models"][m]
565
+ s_2c = p2c_stats["models"][m]
566
+ s_2d = p2d_stats["models"][m]
567
+ s_2e = p2e_stats["models"][m]
568
+ s_2f_pub = p2f_pub_stats["models"][m]
569
+ pts = [s["point_pct"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
570
+ los = [s["ci95_lo"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
571
+ his = [s["ci95_hi"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
572
+ pt_mean = float(np.mean(pts))
573
+ half_w = float(np.sqrt(np.sum([(h - l) ** 2 for l, h in zip(los, his)])) / (2.0 * 6.0))
574
+ entry: Dict[str, Any] = {
575
+ "point_pct": round(pt_mean, 2),
576
+ "ci95_lo": round(pt_mean - half_w, 2),
577
+ "ci95_hi": round(pt_mean + half_w, 2),
578
+ }
579
+ if m == ref_llm:
580
+ pts_whse = pts[:-1] + [p2f_whse_stats["models"][m]["point_pct"]]
581
+ entry["point_with_corpus_swarm_pct"] = round(float(np.mean(pts_whse)), 2)
582
+ composite_stats[m] = entry
583
+
584
+ models_cost = [
585
+ "Armiger (8.3 MB M4 Pro CPU INT8)",
586
+ "Armiger Ultra (M4 Pro / RTX GPU FP32)",
587
+ "Local 8B Instruct LLM (FP16)",
588
+ "Google Gemini Flash",
589
+ "Cheap OpenAI GPT (Fast Tier)",
590
+ "Anthropic Claude Haiku 4.5",
591
+ ]
592
+ colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
593
+ batched_cost = [0.002, 0.004, 1.85, 94.90, 127.30, 712.00]
594
+ realtime_cost = [0.002, 0.004, 18.00, 147.00, 220.50, 1220.00]
595
+ fallback_assets = assets_dir if (assets_dir / "armiger_vs_llms_overall_and_6dims.png").is_file() else (npz_path.parent.parent / "assets")
596
+ if render_pngs:
597
+ # ==========================================================================
598
+ # RENDER CHART 1: armiger_vs_llms_overall_and_6dims.png
599
+ # ==========================================================================
600
+ fig1 = plt.figure(figsize=(24, 11.4))
601
+ fig1.suptitle(
602
+ "Armiger & Armiger Ultra vs. Frontier Cloud LLMs & Local GPU LLM Across 6 Core Dimensions",
603
+ fontsize=17.5, fontweight="bold", color=TEXT_MAIN, y=0.972,
604
+ )
605
+ fig1.text(
606
+ 0.5, 0.934,
607
+ "Non-Circular Leave-One-Teacher-Out (LOTO) & Identical-Denominator Evaluation with 95% Paired Bootstrap CIs (10,000 Resamples)",
608
+ ha="center", fontsize=11.3, style="italic", color=TEXT_SUB, parse_math=False,
609
+ )
610
+
611
+ gs1 = GridSpec(
612
+ 2, 5, figure=fig1,
613
+ width_ratios=[1.72, 0.12, 1.0, 1.0, 1.0],
614
+ wspace=0.16, hspace=0.42,
615
+ top=0.86, bottom=0.13, left=0.14, right=0.98,
616
+ )
617
+
618
+ ax_left = fig1.add_subplot(gs1[:, 0])
619
+ style_ax(ax_left)
620
+ colors_llm = [C_ULTRA, C_INT8, C_GEMINI, C_CLAUDE, C_GPT, C_VINCI]
621
+ edges_llm = [C_ULTRA_EDGE, C_INT8_EDGE, "none", "none", "none", "none"]
622
+ vals_left = [composite_stats[m]["point_pct"] for m in llm_order]
623
+ labels_left = [
624
+ f"{vals_left[0]:.2f}% Pub-6D | 97.60% w/ Corpus Swarm (0.21 ms RTX)",
625
+ f"{vals_left[1]:.2f}% Pub-6D (2.44 ms M4 Pro CPU INT8 | $0.00 API)",
626
+ f"{vals_left[2]:.2f}% (~450 ms | Non-Circular LOTO N=1,025)",
627
+ f"{vals_left[3]:.2f}% (~310 ms | $712.00-$1,220.00 / 1M)",
628
+ f"{vals_left[4]:.2f}% (~365 ms | Non-Circular LOTO N=1,025)",
629
+ f"{vals_left[5]:.2f}% (~185 ms RTX 4070 TiS | $1.85 Pwr / 1M)",
630
+ ]
631
+ y_llm = np.arange(len(llm_order))
632
+ ax_left.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
633
+ ax_left.barh(y_llm, vals_left, height=0.55, color=colors_llm, edgecolor=edges_llm, linewidth=1.3, zorder=3)
634
+ draw_ci_whiskers(ax_left, y_llm, [composite_stats[m] for m in llm_order])
635
+ ax_left.set_yticks(y_llm)
636
+ ax_left.set_yticklabels(llm_order, fontsize=10.5, fontweight="bold", color=TEXT_MAIN)
637
+ ax_left.invert_yaxis()
638
+ ax_left.set_xlim(78, 136)
639
+ ax_left.set_xticks([80, 85, 90, 95, 100])
640
+ ax_left.set_xlabel("6-Dimension Composite Mean (%) ± 95% Bootstrap CI", fontsize=10, color=TEXT_SUB, labelpad=8)
641
+ ax_left.set_title(
642
+ "1. Overall 6-Dimension Composite Mean (%)\n(Category, Catalog, Audio, Anime, Junk, Swarm)",
643
+ fontsize=11.5, fontweight="bold", color=TEXT_MAIN, pad=10,
644
+ )
645
+ for idx, (m, lbl) in enumerate(zip(llm_order, labels_left)):
646
+ fw = "bold" if idx < 2 else "normal"
647
+ tc = TEXT_MAIN if idx < 2 else "#CBD5E1"
648
+ hi_x = max(vals_left[idx], composite_stats[m]["ci95_hi"])
649
+ ax_left.text(hi_x + 0.55, idx, lbl, va="center", ha="left", fontsize=8.5, fontweight=fw, color=tc, zorder=6, parse_math=False)
650
+
651
+ sub_specs = [
652
+ (
653
+ 0, 2, "2A. 11-Class Category (N=1,025 LOTO)",
654
+ p2a_stats,
655
+ [
656
+ f"{p2a_stats['models'][llm_order[0]]['point_pct']:.2f}% (95.49% oct15)",
657
+ f"{p2a_stats['models'][llm_order[1]]['point_pct']:.2f}% (95.49% oct15)",
658
+ f"{p2a_stats['models'][llm_order[2]]['point_pct']:.2f}% (95.6% Circ)",
659
+ f"{p2a_stats['models'][llm_order[3]]['point_pct']:.2f}%",
660
+ f"{p2a_stats['models'][llm_order[4]]['point_pct']:.2f}% (96.6% Circ)",
661
+ f"{p2a_stats['models'][llm_order[5]]['point_pct']:.2f}%",
662
+ ],
663
+ (72, 134), [75, 80, 85, 90, 95, 100],
664
+ ),
665
+ (
666
+ 0, 3, "2B. Catalog Matching (N=1,025 LOTO)",
667
+ p2b_stats,
668
+ [
669
+ f"{p2b_stats['models'][llm_order[0]]['point_pct']:.2f}% (96.53% oct15)",
670
+ f"{p2b_stats['models'][llm_order[1]]['point_pct']:.2f}% (96.53% oct15)",
671
+ f"{p2b_stats['models'][llm_order[2]]['point_pct']:.2f}%",
672
+ f"{p2b_stats['models'][llm_order[3]]['point_pct']:.2f}%",
673
+ f"{p2b_stats['models'][llm_order[4]]['point_pct']:.2f}%",
674
+ f"{p2b_stats['models'][llm_order[5]]['point_pct']:.2f}%",
675
+ ],
676
+ (70, 126), [75, 80, 85, 90, 95, 100],
677
+ ),
678
+ (
679
+ 0, 4, "2C. Audio Language (Strict E-vs-F N=229)",
680
+ p2c_stats,
681
+ [
682
+ f"{p2c_stats['models'][llm_order[0]]['point_pct']:.2f}% (97.80% oct15)",
683
+ f"{p2c_stats['models'][llm_order[1]]['point_pct']:.2f}% (97.80% oct15)",
684
+ f"{p2c_stats['models'][llm_order[2]]['point_pct']:.2f}%",
685
+ f"{p2c_stats['models'][llm_order[3]]['point_pct']:.2f}%",
686
+ f"{p2c_stats['models'][llm_order[4]]['point_pct']:.2f}% (94.68% oct15)",
687
+ f"{p2c_stats['models'][llm_order[5]]['point_pct']:.2f}%",
688
+ ],
689
+ (88, 122), [90, 92, 94, 96, 98, 100],
690
+ ),
691
+ (
692
+ 1, 2, "2D. Anime & Dub (Valid Media N=636)",
693
+ p2d_stats,
694
+ [
695
+ f"{p2d_stats['models'][llm_order[0]]['point_pct']:.2f}% (99.61% oct15)",
696
+ f"{p2d_stats['models'][llm_order[1]]['point_pct']:.2f}% (99.61% oct15)",
697
+ f"{p2d_stats['models'][llm_order[2]]['point_pct']:.2f}% (608/636)",
698
+ f"{p2d_stats['models'][llm_order[3]]['point_pct']:.2f}% (Valid Media)",
699
+ f"{p2d_stats['models'][llm_order[4]]['point_pct']:.2f}% (608/636)",
700
+ f"{p2d_stats['models'][llm_order[5]]['point_pct']:.2f}%",
701
+ ],
702
+ (88, 124), [90, 92, 94, 96, 98, 100],
703
+ ),
704
+ (
705
+ 1, 3, "2E. Junk / Malware (Balanced N=977)",
706
+ p2e_stats,
707
+ [
708
+ f"{p2e_stats['models'][llm_order[0]]['point_pct']:.2f}% (99.90% Acc / 100% oct15)",
709
+ f"{p2e_stats['models'][llm_order[1]]['point_pct']:.2f}% (99.90% Acc / 100% oct15)",
710
+ f"{p2e_stats['models'][llm_order[2]]['point_pct']:.2f}% (Non-Circ)",
711
+ f"{p2e_stats['models'][llm_order[3]]['point_pct']:.2f}%",
712
+ f"{p2e_stats['models'][llm_order[4]]['point_pct']:.2f}% (0% Rec.)",
713
+ f"{p2e_stats['models'][llm_order[5]]['point_pct']:.2f}%",
714
+ ],
715
+ (40, 162), [40, 60, 80, 100],
716
+ ),
717
+ (
718
+ 1, 4, "2F. Swarm Liveliness AUROC (Identical N=757)",
719
+ p2f_pub_stats,
720
+ [
721
+ "92.42% Pub (96.83% Corpus N=10.3k)",
722
+ "92.42% Pub (86.64% Priors)",
723
+ "89.20% Pub (N=757)",
724
+ "89.76% Pub (N=757)",
725
+ "89.82% Pub (N=757)",
726
+ "66.32% Pub (N=757)",
727
+ ],
728
+ (55, 155), [60, 70, 80, 90, 100],
729
+ ),
730
+ ]
731
+
732
+ for row, col, title, p_stat, lbls, xlim, xticks in sub_specs:
733
+ ax = fig1.add_subplot(gs1[row, col])
734
+ style_ax(ax)
735
+ ax.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
736
+ if p_stat is not None:
737
+ s_list = [p_stat["models"][m] for m in llm_order]
738
+ else:
739
+ s_list = [p2f_whse_stats["models"][ref_llm]] + [p2f_pub_stats["models"][m] for m in llm_order[1:]]
740
+ vals = [s["point_pct"] for s in s_list]
741
+ ax.barh(y_llm, vals, height=0.58, color=colors_llm, edgecolor=edges_llm, linewidth=1.2, zorder=3)
742
+ draw_ci_whiskers(ax, y_llm, s_list)
743
+ ax.set_yticks(y_llm)
744
+ ax.set_yticklabels([""] * len(llm_order))
745
+ ax.invert_yaxis()
746
+ ax.set_xlim(*xlim)
747
+ ax.set_xticks(xticks)
748
+ ax.set_title(title, fontsize=10.5, fontweight="bold", color=TEXT_MAIN, pad=8)
749
+ span = xlim[1] - xlim[0]
750
+ for idx, (s_item, lbl) in enumerate(zip(s_list, lbls)):
751
+ fw = "bold" if idx < 2 else "normal"
752
+ tc = TEXT_MAIN if idx < 2 else "#CBD5E1"
753
+ hi_x = max(s_item["point_pct"], s_item["ci95_hi"])
754
+ ax.text(hi_x + span * 0.015, idx, lbl, va="center", ha="left", fontsize=7.9, fontweight=fw, color=tc, zorder=6, parse_math=False)
755
+
756
+ legend_handles = [
757
+ Patch(facecolor=C_ULTRA, edgecolor=C_ULTRA_EDGE, label="Armiger Ultra (Fair Hybrid)"),
758
+ Patch(facecolor=C_INT8, edgecolor=C_INT8_EDGE, label="Armiger (8.3 MB INT8 + Plugins)"),
759
+ Patch(facecolor=C_GEMINI, label="Google Gemini Flash (LOTO Non-Circular)"),
760
+ Patch(facecolor=C_CLAUDE, label="Anthropic Claude Haiku 4.5"),
761
+ Patch(facecolor=C_GPT, label="OpenAI GPT (Fast Tier, LOTO Non-Circular)"),
762
+ Patch(facecolor=C_VINCI, label="Local 8B Instruct LLM (FP16)"),
763
+ ]
764
+ fig1.legend(
765
+ handles=legend_handles, loc="lower center", bbox_to_anchor=(0.5, 0.015),
766
+ ncol=6, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=9.0, labelcolor=TEXT_MAIN,
767
+ borderpad=0.6, columnspacing=1.35,
768
+ )
769
+ sha_chart1 = save_clean_rgb_png(fig1, assets_dir / "armiger_vs_llms_overall_and_6dims.png")
770
+
771
+ # ==========================================================================
772
+ # RENDER CHART 2: armiger_vs_llms_cost_comparison.png
773
+ # ==========================================================================
774
+ fig2, (ax2_l, ax2_r) = plt.subplots(1, 2, figsize=(23.5, 9.2), gridspec_kw={"wspace": 0.14})
775
+ fig2.subplots_adjust(top=0.80, bottom=0.16, left=0.16, right=0.98)
776
+ fig2.suptitle(
777
+ "Cost & Hardware-Attributed Throughput: Armiger & Armiger Ultra vs. Local & Cloud LLMs",
778
+ fontsize=17.5, fontweight="bold", color=TEXT_MAIN, y=0.96,
779
+ )
780
+ fig2.text(
781
+ 0.5, 0.915,
782
+ "Left: Operational Cost per 1,000,000 Classified Torrents ($ USD) | Right: API Token Rates ($ USD / 1M Tokens) & Measured Hardware Throughput",
783
+ ha="center", fontsize=11.3, style="italic", color=TEXT_SUB, parse_math=False,
784
+ )
785
+
786
+ models_cost = [
787
+ "Armiger (8.3 MB M4 Pro CPU INT8)",
788
+ "Armiger Ultra (M4 Pro / RTX GPU FP32)",
789
+ "Local 8B Instruct LLM (FP16)",
790
+ "Google Gemini Flash",
791
+ "Cheap OpenAI GPT (Fast Tier)",
792
+ "Anthropic Claude Haiku 4.5",
793
+ ]
794
+ colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
795
+ batched_cost = [0.002, 0.004, 1.85, 94.90, 127.30, 712.00]
796
+ realtime_cost = [0.002, 0.004, 18.00, 147.00, 220.50, 1220.00]
797
+ cost_labels = [
798
+ "$0.00 API (~$0.0020 Compute Electricity / 1M Tasks)",
799
+ "$0.00 API (~$0.0040 Compute Electricity / 1M Tasks)",
800
+ "$0.00 API ($1.85 Local GPU Power | ~$18.00 Cloud GPU / 1M)",
801
+ "$94.90 Batched ($147.00 Real-Time 1-by-1)",
802
+ "$127.30 Batched ($220.50 Real-Time 1-by-1)",
803
+ "$712.00 Batched ($1,220.00 Real-Time 1-by-1)",
804
+ ]
805
+
806
+ style_ax(ax2_l)
807
+ yc = np.arange(len(models_cost))
808
+ ax2_l.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
809
+ for idx in range(len(models_cost)):
810
+ ax2_l.barh(idx, realtime_cost[idx], height=0.56, color=colors_cost[idx], alpha=0.28, edgecolor=colors_cost[idx], linestyle="--", linewidth=1.2, zorder=2)
811
+ ax2_l.barh(idx, batched_cost[idx], height=0.56, color=colors_cost[idx], zorder=3)
812
+ x_pos = max(realtime_cost[idx], batched_cost[idx]) + 18.0
813
+ ax2_l.text(x_pos, idx, cost_labels[idx], va="center", ha="left", fontsize=9.0, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1", parse_math=False)
814
+
815
+ ax2_l.set_yticks(yc)
816
+ ax2_l.set_yticklabels(models_cost, fontsize=10.2, fontweight="bold", color=TEXT_MAIN)
817
+ ax2_l.invert_yaxis()
818
+ ax2_l.set_xlim(0, 1980)
819
+ ax2_l.set_xticks([0, 200, 400, 600, 800, 1000, 1200])
820
+ ax2_l.set_xticklabels(["$0", "$200", "$400", "$600", "$800", "$1,000", "$1,200"])
821
+ ax2_l.set_xlabel("Derived from measured prompt + completion token receipts (Lower is Better)", fontsize=9.5, color=TEXT_SUB, labelpad=8)
822
+ ax2_l.set_title(
823
+ "1. Cost per 1,000,000 Classified Torrents ($ USD / 1M Tasks)\n(Solid = 32-Item Batched Mode; Dashed = Real-Time 1-by-1 Webhook Mode)",
824
+ fontsize=11.2, fontweight="bold", color=TEXT_MAIN, pad=10,
825
+ )
826
+ ax2_l.legend(
827
+ handles=[
828
+ Patch(facecolor=C_GEMINI, label="32-Item Batched API Cost"),
829
+ Patch(facecolor=C_GEMINI, alpha=0.3, edgecolor=C_GEMINI, linestyle="--", label="Real-Time Single-Torrent API Cost (1-by-1)"),
830
+ ],
831
+ loc="upper center", bbox_to_anchor=(0.5, -0.11), ncol=2, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=8.8, labelcolor=TEXT_MAIN,
832
+ )
833
+
834
+ style_ax(ax2_r)
835
+ ax2_r.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
836
+ in_rates = [0.0, 0.0, 0.012, 0.10, 0.15, 0.80]
837
+ out_rates = [0.0, 0.0, 0.12, 0.40, 0.60, 4.00]
838
+ rate_labels = [
839
+ "$0.00 API (2.44 ms single / 326 items/s batch-64 M4 Pro CPU)",
840
+ "$0.00 API (3,416 items/s M4 Pro MPS / 4,795 items/s RTX GPU)",
841
+ "$0.00 API (~$0.012/1M Pwr Equiv. | 5.4 items/s RTX 4070 TiS)",
842
+ "$0.10 In | $0.40 Out ($0.13 Blended / 1M Tokens)",
843
+ "$0.15 In | $0.60 Out ($0.18 Blended / 1M Tokens)",
844
+ "$0.80 In | $4.00 Out ($1.05 Blended / 1M Tokens)",
845
+ ]
846
+ for idx in range(len(models_cost)):
847
+ ax2_r.barh(idx - 0.14, in_rates[idx], height=0.28, color=colors_cost[idx], alpha=0.55, hatch="//", edgecolor=colors_cost[idx], zorder=3)
848
+ ax2_r.barh(idx + 0.14, out_rates[idx], height=0.28, color=colors_cost[idx], zorder=3)
849
+ ax2_r.text(max(in_rates[idx], out_rates[idx]) + 0.08, idx + 0.05, rate_labels[idx], va="center", ha="left", fontsize=9.0, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1", parse_math=False)
850
+
851
+ ax2_r.set_yticks(yc)
852
+ ax2_r.set_yticklabels([""] * len(models_cost))
853
+ ax2_r.invert_yaxis()
854
+ ax2_r.set_xlim(0, 7.05)
855
+ ax2_r.set_xticks([0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0])
856
+ ax2_r.set_xticklabels(["$0.00", "$0.50", "$1.00", "$1.50", "$2.00", "$2.50", "$3.00", "$3.50", "$4.00"])
857
+ ax2_r.set_xlabel("Standard API List Rates ($ USD / 1M Tokens) & Measured Local Hardware Speed", fontsize=9.5, color=TEXT_SUB, labelpad=8)
858
+ ax2_r.set_title(
859
+ "2. Token Pricing ($ USD / 1M Tokens) & Hardware-Attributed Throughput\n(M4 Pro CPU INT8: 2.44 ms single / 326/s batch-64; M4 Pro MPS: 3,416/s; RTX GPU: 4,795/s)",
860
+ fontsize=11.2, fontweight="bold", color=TEXT_MAIN, pad=10,
861
+ )
862
+ ax2_r.legend(
863
+ handles=[
864
+ Patch(facecolor=C_GEMINI, alpha=0.55, hatch="//", edgecolor=C_GEMINI, label="Input Tokens ($ / 1M)"),
865
+ Patch(facecolor=C_GEMINI, label="Output Tokens ($ / 1M)"),
866
+ ],
867
+ loc="upper center", bbox_to_anchor=(0.5, -0.11), ncol=2, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=8.8, labelcolor=TEXT_MAIN,
868
+ )
869
+ sha_chart2 = save_clean_rgb_png(fig2, assets_dir / "armiger_vs_llms_cost_comparison.png")
870
+ else:
871
+ for fn in ("armiger_vs_llms_overall_and_6dims.png", "armiger_vs_llms_cost_comparison.png"):
872
+ verify_png_chunks_clean(fallback_assets / fn)
873
+ sha_chart1 = hashlib.sha256((fallback_assets / "armiger_vs_llms_overall_and_6dims.png").read_bytes()).hexdigest()
874
+ sha_chart2 = hashlib.sha256((fallback_assets / "armiger_vs_llms_cost_comparison.png").read_bytes()).hexdigest()
875
+
876
+ # ==========================================================================
877
+ # COMPUTE SUITE A (CHART 3) PANEL STATISTICS WITH IDENTICAL-N LOCKS
878
+ # ==========================================================================
879
+ models_10 = [
880
+ "Armiger Ultra (Fair Hybrid)",
881
+ "Armiger (8.3 MB INT8 + Plugins)",
882
+ "MIMMO BERT (360M)",
883
+ "MIMMO Cascade",
884
+ "RTN (rank-torrent-name)",
885
+ "PTT / Parsett",
886
+ "GuessIt",
887
+ "PTN (parse-torrent-title)",
888
+ "Anitomy",
889
+ "Bitmagnet Built-In",
890
+ ]
891
+ keys_10 = [
892
+ "ultra", "int8_hyb", "mimmo_bert", "mimmo_cascade",
893
+ "rtn", "ptt", "guessit", "ptn", "anitomy", "bitmagnet",
894
+ ]
895
+ colors_10 = [
896
+ C_ULTRA, C_INT8, "#818CF8", "#A78BFA", "#F59E0B",
897
+ "#EC4899", "#14B8A6", "#64748B", "#06B6D4", "#94A3B8",
898
+ ]
899
+
900
+ y_oct_cat11 = d["oct15_cat11_y_true"]
901
+ c3_p1_stats = compute_paired_panel_stats(
902
+ "chart3_1_oct15_cat11",
903
+ {m: (d[f"oct15_cat11_pred_{k}"] == y_oct_cat11).astype(np.float32) for m, k in zip(models_10, keys_10)},
904
+ ref_model=models_10[0],
905
+ expected_n=2835,
906
+ metric_type="mean",
907
+ )
908
+
909
+ y_oct_vid = d["oct15_vid_y_true"]
910
+ c3_p1_vid_stats = compute_paired_panel_stats(
911
+ "chart3_1_oct15_video_mvtv",
912
+ {m: (d[f"oct15_vid_mvtv_pred_{k}"] == y_oct_vid).astype(np.float32) for m, k in zip(models_10, keys_10)},
913
+ ref_model=models_10[0],
914
+ expected_n=1296,
915
+ metric_type="mean",
916
+ )
917
+
918
+ c3_p2_stats = compute_paired_panel_stats(
919
+ "chart3_2_oct15_catalog_matching",
920
+ {m: d[f"oct15_match_correct_{k}"].astype(np.float32) for m, k in zip(models_10, keys_10)},
921
+ ref_model=models_10[0],
922
+ expected_n=1296,
923
+ metric_type="mean",
924
+ )
925
+
926
+ y_oct_aud = d["oct15_aud545_y_en"]
927
+ aud_models = [models_10[i] for i in (0, 1, 4, 5, 6, 7, 8, 9)]
928
+ aud_keys = [keys_10[i] for i in (0, 1, 4, 5, 6, 7, 8, 9)]
929
+ c3_p3_stats = compute_paired_panel_stats(
930
+ "chart3_3_oct15_audio_strict545",
931
+ {m: (d[f"oct15_aud545_pred_{k}"] == y_oct_aud).astype(np.float32) for m, k in zip(aud_models, aud_keys)},
932
+ ref_model=models_10[0],
933
+ expected_n=545,
934
+ metric_type="mean",
935
+ )
936
+
937
+ y_oct_ani = d["oct15_ani507_y_true"]
938
+ c3_p4_oct_stats = compute_paired_panel_stats(
939
+ "chart3_4_oct15_anime507",
940
+ {
941
+ models_10[0]: (d["oct15_ani507_pred_ultra"] == y_oct_ani).astype(np.float32),
942
+ models_10[1]: (d["oct15_ani507_pred_int8"] == y_oct_ani).astype(np.float32),
943
+ },
944
+ ref_model=models_10[0],
945
+ expected_n=507,
946
+ metric_type="mean",
947
+ )
948
+
949
+ y_tc2k_ani = d["tc2000_ani_y_true"]
950
+ c3_p4_tc2k_stats = compute_paired_panel_stats(
951
+ "chart3_4_tc2000_anime_dub",
952
+ {
953
+ models_10[4]: (d["tc2000_ani_pred_rtn"] == y_tc2k_ani).astype(np.float32),
954
+ models_10[5]: (d["tc2000_ani_pred_ptt"] == y_tc2k_ani).astype(np.float32),
955
+ models_10[6]: (d["tc2000_ani_pred_guessit"] == y_tc2k_ani).astype(np.float32),
956
+ models_10[7]: (d["tc2000_ani_pred_ptn"] == y_tc2k_ani).astype(np.float32),
957
+ models_10[8]: (d["tc2000_ani_pred_anitomy"] == y_tc2k_ani).astype(np.float32),
958
+ },
959
+ ref_model=models_10[4],
960
+ expected_n=2000,
961
+ metric_type="mean",
962
+ )
963
+
964
+ y_oct_ad = d["oct15_adult_y_true"]
965
+ ad_models = [models_10[i] for i in (0, 1, 2, 3, 4, 5, 9)]
966
+ ad_keys = ["ultra", "int8", "mimmo_bert", "mimmo_cascade", "rtn", "ptt", "bitmagnet"]
967
+ c3_p5_stats = compute_paired_panel_stats(
968
+ "chart3_5_oct15_adult2781",
969
+ {m: (d[f"oct15_adult_pred_{k}"] == y_oct_ad).astype(np.float32) for m, k in zip(ad_models, ad_keys)},
970
+ ref_model=models_10[0],
971
+ expected_n=2781,
972
+ metric_type="mean",
973
+ )
974
+
975
+ y_oct_jk = d["oct15_junk_y_true"]
976
+ c3_p6a_stats = compute_paired_panel_stats(
977
+ "chart3_6a_oct15_junk2170",
978
+ {
979
+ "Armiger Ultra": d["oct15_junk_pred_ultra"],
980
+ "Armiger INT8": d["oct15_junk_pred_int8"],
981
+ "Bitmagnet": d["oct15_junk_pred_bitmagnet"],
982
+ "RTN Filter": d["oct15_junk_pred_rtn"],
983
+ },
984
+ ref_model="Armiger Ultra",
985
+ expected_n=2170,
986
+ metric_type="balanced",
987
+ y_true=y_oct_jk,
988
+ )
989
+
990
+ oct_cat_spec = [
991
+ (6, "Audiobook"),
992
+ (2, "Adult / XXX"),
993
+ (1, "TV Show"),
994
+ (0, "Movie"),
995
+ (7, "Software"),
996
+ (3, "Music"),
997
+ (5, "Game"),
998
+ (8, "Ebook"),
999
+ (9, "Course"),
1000
+ (4, "Comic"),
1001
+ ]
1002
+ cats_oct_labels = []
1003
+ oct_ultra, oct_int8, oct_mimmo, oct_bit = [], [], [], []
1004
+ chart4_oct_manifest: Dict[str, Any] = {}
1005
+ for c_idx, c_title in oct_cat_spec:
1006
+ mask = (y_oct_cat11 == c_idx)
1007
+ n_c = int(np.sum(mask))
1008
+ cats_oct_labels.append(f"{c_title} (n={n_c:,})")
1009
+ u_v = float(np.mean(d["oct15_cat11_pred_ultra"][mask] == c_idx) * 100.0)
1010
+ i_v = float(np.mean(d["oct15_cat11_pred_int8_hyb"][mask] == c_idx) * 100.0)
1011
+ m_v = float(np.mean(d["oct15_cat11_pred_mimmo_bert"][mask] == c_idx) * 100.0)
1012
+ b_v = float(np.mean(d["oct15_cat11_pred_bitmagnet"][mask] == c_idx) * 100.0)
1013
+ oct_ultra.append(round(u_v, 2))
1014
+ oct_int8.append(round(i_v, 2))
1015
+ oct_mimmo.append(round(m_v, 2))
1016
+ oct_bit.append(round(b_v, 2))
1017
+ chart4_oct_manifest[c_title] = {
1018
+ "n": n_c,
1019
+ "armiger_ultra_pct": round(u_v, 2),
1020
+ "armiger_int8_pct": round(i_v, 2),
1021
+ "mimmo_bert_pct": round(m_v, 2),
1022
+ "bitmagnet_pct": round(b_v, 2),
1023
+ }
1024
+
1025
+ y_bc_cat11 = d["beyond_cat11_y_true"]
1026
+ bc_cat_spec = [
1027
+ (6, "Audiobook"),
1028
+ (8, "Ebook"),
1029
+ (4, "Comic"),
1030
+ (0, "Movie"),
1031
+ (1, "TV Show"),
1032
+ (5, "Game"),
1033
+ (7, "Software"),
1034
+ (9, "Course"),
1035
+ (2, "Adult / XXX"),
1036
+ (3, "Music"),
1037
+ ]
1038
+ cats_bc_labels = []
1039
+ bc_ultra, bc_int8, bc_gpt, bc_gem = [], [], [], []
1040
+ chart4_bc_manifest: Dict[str, Any] = {}
1041
+ for c_idx, c_title in bc_cat_spec:
1042
+ mask = (y_bc_cat11 == c_idx)
1043
+ n_c = int(np.sum(mask))
1044
+ cats_bc_labels.append(f"{c_title} (n={n_c:,})")
1045
+ u_v = float(np.mean(d["beyond_cat11_corr_ultra"][mask]) * 100.0)
1046
+ i_v = float(np.mean(d["beyond_cat11_corr_int8_hyb"][mask]) * 100.0)
1047
+ gp_v = float(np.mean(d["beyond_cat11_corr_gpt_noncirc"][mask]) * 100.0)
1048
+ gm_v = float(np.mean(d["beyond_cat11_corr_gemini_noncirc"][mask]) * 100.0)
1049
+ bc_ultra.append(round(u_v, 2))
1050
+ bc_int8.append(round(i_v, 2))
1051
+ bc_gpt.append(round(gp_v, 2))
1052
+ bc_gem.append(round(gm_v, 2))
1053
+ chart4_bc_manifest[c_title] = {
1054
+ "n": n_c,
1055
+ "armiger_ultra_pct": round(u_v, 2),
1056
+ "armiger_int8_pct": round(i_v, 2),
1057
+ "gpt_noncirc_pct": round(gp_v, 2),
1058
+ "gemini_noncirc_pct": round(gm_v, 2),
1059
+ }
1060
+ if render_pngs:
1061
+ # ==========================================================================
1062
+ # RENDER CHART 3: armiger_vs_torrent_classifiers_6dims.png
1063
+ # ==========================================================================
1064
+ fig3 = plt.figure(figsize=(24.5, 14.8))
1065
+ fig3.suptitle(
1066
+ "Armiger & Armiger Ultra vs. Open-Source Torrent Classifiers & Parsers Across 7 Core Dimensions",
1067
+ fontsize=17.5, fontweight="bold", color=TEXT_MAIN, y=0.975,
1068
+ )
1069
+ fig3.text(
1070
+ 0.5, 0.942,
1071
+ "Identical-Denominator Evaluation on oct15_confirmation (N = 2,835 / N = 545 Audio), tc_truespec_2000 (N = 2,000), & Swarm Holdouts (±95% Bootstrap CIs)",
1072
+ ha="center", fontsize=11.3, style="italic", color=TEXT_SUB, parse_math=False,
1073
+ )
1074
+
1075
+ gs3 = GridSpec(
1076
+ 7, 4, figure=fig3,
1077
+ height_ratios=[1.0, 1.0, 1.0, 0.42, 1.25, 0.48, 1.25],
1078
+ width_ratios=[1.08, 1.0, 0.22, 1.08],
1079
+ hspace=0.18, wspace=0.15,
1080
+ top=0.89, bottom=0.06, left=0.12, right=0.98,
1081
+ )
1082
+ y10 = np.arange(len(models_10))
1083
+
1084
+ # Panel 1: Category Detection
1085
+ ax1 = fig3.add_subplot(gs3[0:3, 0])
1086
+ style_ax(ax1)
1087
+ ax1.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
1088
+ cat_11_s = [c3_p1_stats["models"][m] for m in models_10]
1089
+ cat_11cls = [s["point_pct"] for s in cat_11_s]
1090
+ cat_native = [
1091
+ c3_p1_vid_stats["models"][models_10[0]]["point_pct"],
1092
+ c3_p1_vid_stats["models"][models_10[1]]["point_pct"],
1093
+ 93.61,
1094
+ 93.97,
1095
+ c3_p1_vid_stats["models"][models_10[4]]["point_pct"],
1096
+ c3_p1_vid_stats["models"][models_10[5]]["point_pct"],
1097
+ c3_p1_vid_stats["models"][models_10[6]]["point_pct"],
1098
+ c3_p1_vid_stats["models"][models_10[7]]["point_pct"],
1099
+ c3_p1_vid_stats["models"][models_10[8]]["point_pct"],
1100
+ 88.40,
1101
+ ]
1102
+ cat_lbls = [
1103
+ f"{cat_11cls[0]:.2f}% 11-Cls (Video {cat_native[0]:.1f}%)",
1104
+ f"{cat_11cls[1]:.2f}% 11-Cls (95.06% Raw)",
1105
+ f"{cat_11cls[2]:.2f}% 11-Cls (4-Med 93.6%)",
1106
+ f"{cat_11cls[3]:.2f}% 11-Cls (4-Med 94.0%)",
1107
+ f"{cat_11cls[4]:.2f}% 11-Cls (Video {cat_native[4]:.1f}%)",
1108
+ f"{cat_11cls[5]:.2f}% 11-Cls (Video {cat_native[5]:.1f}%)",
1109
+ f"{cat_11cls[6]:.2f}% 11-Cls (Video {cat_native[6]:.1f}%)",
1110
+ f"{cat_11cls[7]:.2f}% 11-Cls (Video {cat_native[7]:.1f}%)",
1111
+ f"{cat_11cls[8]:.2f}% 11-Cls (Video {cat_native[8]:.1f}%)",
1112
+ f"{cat_11cls[9]:.2f}% 11-Cls (Ans. 88.4%)",
1113
+ ]
1114
+ ax1.barh(y10 - 0.16, cat_native, height=0.32, color=colors_10, zorder=3)
1115
+ ax1.barh(y10 + 0.16, cat_11cls, height=0.32, color=colors_10, alpha=0.45, hatch="//", edgecolor=colors_10, zorder=3)
1116
+ draw_ci_whiskers(ax1, y10 + 0.16, cat_11_s)
1117
+ ax1.set_yticks(y10)
1118
+ ax1.set_yticklabels(models_10, fontsize=9.5, fontweight="bold", color=TEXT_MAIN)
1119
+ ax1.invert_yaxis()
1120
+ ax1.set_xlim(25, 148)
1121
+ ax1.set_xticks([30, 40, 50, 60, 70, 80, 90, 100])
1122
+ ax1.set_xlabel("Solid = Native Domain (Video N=1,296 / 4-Med N=1,580); Hatched = 11-Class (N=2,835)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
1123
+ ax1.set_title("1. Category Detection (%)\n(oct15_confirmation, Identical N = 2,835)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
1124
+ for idx, (v, lbl) in enumerate(zip(cat_native, cat_lbls)):
1125
+ ax1.text(v + 1.1, idx, lbl, va="center", ha="left", fontsize=8.2, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
1126
+
1127
+ # Panel 2: Catalog Matching Rate
1128
+ ax2 = fig3.add_subplot(gs3[0:3, 1])
1129
+ style_ax(ax2)
1130
+ ax2.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
1131
+ p2_s = [c3_p2_stats["models"][m] for m in models_10]
1132
+ cat_match = [s["point_pct"] for s in p2_s]
1133
+ ax2.barh(y10, cat_match, height=0.58, color=colors_10, zorder=3)
1134
+ draw_ci_whiskers(ax2, y10, p2_s)
1135
+ ax2.set_yticks(y10)
1136
+ ax2.set_yticklabels([""] * 10)
1137
+ ax2.invert_yaxis()
1138
+ ax2.set_xlim(35, 128)
1139
+ ax2.set_xticks([40, 50, 60, 70, 80, 90, 100])
1140
+ ax2.set_xlabel("oct15_confirmation video subset (Identical N = 1,296)", fontsize=8.8, color=TEXT_SUB, labelpad=6)
1141
+ ax2.set_title("2. Catalog Matching Rate (%)\n(Clean Title + Correct Movie/TV Routing, N = 1,296)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
1142
+ for idx, s_item in enumerate(p2_s):
1143
+ v = s_item["point_pct"]
1144
+ lbl = f"{v:.2f}% (95.83% Raw)" if idx == 1 else f"{v:.2f}%"
1145
+ ax2.text(max(v, s_item["ci95_hi"]) + 0.9, idx, lbl, va="center", ha="left", fontsize=8.4, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
1146
+
1147
+ # Panel 3: Audio Language Detection (Strict E-vs-F N=545)
1148
+ ax3 = fig3.add_subplot(gs3[0:3, 3])
1149
+ style_ax(ax3)
1150
+ ax3.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
1151
+ for idx, m in enumerate(models_10):
1152
+ if m in c3_p3_stats["models"]:
1153
+ s_item = c3_p3_stats["models"][m]
1154
+ v = s_item["point_pct"]
1155
+ ax3.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
1156
+ draw_ci_whiskers(ax3, np.array([idx]), [s_item])
1157
+ lbl = f"{v:.2f}% (96.88% Raw)" if idx == 1 else f"{v:.2f}%"
1158
+ ax3.text(max(v, s_item["ci95_hi"]) + 0.25, idx, lbl, va="center", ha="left", fontsize=8.4, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
1159
+ else:
1160
+ ax3.barh(idx, 86.2, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
1161
+ ax3.text(84.3, idx, "N/A (No Audio Language Head)", va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
1162
+ ax3.set_yticks(y10)
1163
+ ax3.set_yticklabels([""] * 10)
1164
+ ax3.invert_yaxis()
1165
+ ax3.set_xlim(84, 106.5)
1166
+ ax3.set_xticks([86, 88, 90, 92, 94, 96, 98, 100])
1167
+ ax3.set_xlabel("All models scored on identical oct15_confirmation Strict E-vs-F (N = 545)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
1168
+ ax3.set_title("3. Audio Language Detection (%)\n(Strict English vs. Foreign Audio, Identical N = 545)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
1169
+
1170
+ # Panel 4: Anime & Anime Dub Detection
1171
+ ax4 = fig3.add_subplot(gs3[4:7, 0])
1172
+ style_ax(ax4)
1173
+ ax4.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
1174
+ for idx, m in enumerate(models_10):
1175
+ if m in c3_p4_oct_stats["models"]:
1176
+ s_item = c3_p4_oct_stats["models"][m]
1177
+ v = s_item["point_pct"]
1178
+ ax4.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
1179
+ draw_ci_whiskers(ax4, np.array([idx]), [s_item])
1180
+ ax4.text(max(v, s_item["ci95_hi"]) + 0.6, idx, f"{v:.2f}% (95.8% tc2000)", va="center", ha="left", fontsize=8.4, fontweight="bold", color=TEXT_MAIN)
1181
+ elif m in c3_p4_tc2k_stats["models"]:
1182
+ s_item = c3_p4_tc2k_stats["models"][m]
1183
+ v = s_item["point_pct"]
1184
+ ax4.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
1185
+ draw_ci_whiskers(ax4, np.array([idx]), [s_item])
1186
+ ax4.text(max(v, s_item["ci95_hi"]) + 0.6, idx, f"{v:.2f}%", va="center", ha="left", fontsize=8.4, color="#CBD5E1")
1187
+ else:
1188
+ ax4.barh(idx, 49.0, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
1189
+ ax4.text(46.2, idx, "N/A (No Anime / Dub Head)", va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
1190
+ ax4.set_yticks(y10)
1191
+ ax4.set_yticklabels(models_10, fontsize=9.5, fontweight="bold", color=TEXT_MAIN)
1192
+ ax4.invert_yaxis()
1193
+ ax4.set_xlim(45, 122)
1194
+ ax4.set_xticks([50, 60, 70, 80, 90, 100])
1195
+ ax4.set_xlabel("99.61% on oct15_confirmation (505/507); 95.80% on tc_truespec_2000 (N=2,000)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
1196
+ ax4.set_title("4. Anime & Anime Dub Detection (%)\n(oct15_confirmation N = 507 & tc_truespec_2000 N = 2,000)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
1197
+
1198
+ # Panel 5: Adult / NSFW Detection
1199
+ ax5 = fig3.add_subplot(gs3[4:7, 1])
1200
+ style_ax(ax5)
1201
+ ax5.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
1202
+ for idx, m in enumerate(models_10):
1203
+ if m in c3_p5_stats["models"]:
1204
+ s_item = c3_p5_stats["models"][m]
1205
+ v = s_item["point_pct"]
1206
+ ax5.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
1207
+ draw_ci_whiskers(ax5, np.array([idx]), [s_item])
1208
+ lbl = f"{v:.2f}% (93.7% Ans.)" if idx == 9 else f"{v:.2f}%"
1209
+ ax5.text(max(v, s_item["ci95_hi"]) + 0.5, idx, lbl, va="center", ha="left", fontsize=8.4, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
1210
+ else:
1211
+ ax5.barh(idx, 68.5, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
1212
+ ax5.text(65.8, idx, "N/A (No Adult / NSFW Head)", va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
1213
+ ax5.set_yticks(y10)
1214
+ ax5.set_yticklabels([""] * 10)
1215
+ ax5.invert_yaxis()
1216
+ ax5.set_xlim(65, 114)
1217
+ ax5.set_xticks([70, 75, 80, 85, 90, 95, 100])
1218
+ ax5.set_xlabel("oct15_confirmation Adult / NSFW ground truth (Identical N = 2,781)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
1219
+ ax5.set_title("5. Adult / NSFW Detection (%)\n(oct15_confirmation, Identical N = 2,781)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
1220
+
1221
+ # Panel 6A: Junk / Malware Detection
1222
+ ax6a = fig3.add_subplot(gs3[4, 3])
1223
+ style_ax(ax6a)
1224
+ ax6a.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
1225
+ m6a = ["Armiger Ultra", "Armiger INT8", "Bitmagnet", "RTN Filter", "Other 6 Tools"]
1226
+ v6a = [
1227
+ c3_p6a_stats["models"]["Armiger Ultra"]["point_pct"],
1228
+ c3_p6a_stats["models"]["Armiger INT8"]["point_pct"],
1229
+ c3_p6a_stats["models"]["Bitmagnet"]["point_pct"],
1230
+ c3_p6a_stats["models"]["RTN Filter"]["point_pct"],
1231
+ None,
1232
+ ]
1233
+ c6a = [C_ULTRA, C_INT8, "#94A3B8", "#F59E0B", "#21262D"]
1234
+ l6a = [
1235
+ f"{v6a[0]:.2f}% Bal (98.41% beyond N=977)",
1236
+ f"{v6a[1]:.2f}% Bal (98.41% beyond N=977)",
1237
+ f"{v6a[2]:.2f}% Bal (0% Recall, 100% Spec)",
1238
+ f"{v6a[3]:.2f}% Bal (0% Recall, 98.57% Spec)",
1239
+ "N/A (No Junk Head)",
1240
+ ]
1241
+ for idx, (v, lbl) in enumerate(zip(v6a, l6a)):
1242
+ if v is not None:
1243
+ ax6a.barh(idx, v, height=0.6, color=c6a[idx], zorder=3)
1244
+ ax6a.text(v + 0.8, idx, lbl, va="center", ha="left", fontsize=8.2, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
1245
+ else:
1246
+ ax6a.barh(idx, 49.0, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
1247
+ ax6a.text(46.2, idx, lbl, va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
1248
+ ax6a.set_yticks(np.arange(len(m6a)))
1249
+ ax6a.set_yticklabels(m6a, fontsize=9, fontweight="bold", color=TEXT_MAIN)
1250
+ ax6a.invert_yaxis()
1251
+ ax6a.set_xlim(45, 144)
1252
+ ax6a.set_xticks([50, 60, 70, 80, 90, 100])
1253
+ ax6a.set_title("6A. Junk / Malware Detection (oct15 N=2,170 & beyond N=977)", fontsize=10, fontweight="bold", color=TEXT_MAIN, pad=7)
1254
+
1255
+ # Panel 6B: Swarm Liveliness AUROC (Identical N=10,278 Multi-Source Corpus)
1256
+ ax6b = fig3.add_subplot(gs3[6, 3])
1257
+ style_ax(ax6b)
1258
+ ax6b.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
1259
+ m6b = ["Armiger Ultra", "Armiger Priors", "Seeder Count", "All 8 Tools"]
1260
+ v6b = [
1261
+ p2f_whse_stats["models"]["Armiger Ultra (Fair Hybrid)"]["point_pct"],
1262
+ p2f_whse_stats["models"]["Armiger 4-Way Priors"]["point_pct"],
1263
+ p2f_whse_stats["models"]["Raw Seeder Count"]["point_pct"],
1264
+ None,
1265
+ ]
1266
+ c6b = [C_ULTRA, C_INT8, "#64748B", "#21262D"]
1267
+ l6b = [
1268
+ f"{v6b[0]:.2f}% Corpus (92.42% Pub N=757)",
1269
+ f"{v6b[1]:.2f}% Corpus (4-Way Priors)",
1270
+ f"{v6b[2]:.2f}% Corpus (85.17% Pub N=757)",
1271
+ "N/A (No Swarm Head)",
1272
+ ]
1273
+ for idx, (v, lbl) in enumerate(zip(v6b, l6b)):
1274
+ if v is not None:
1275
+ ax6b.barh(idx, v, height=0.6, color=c6b[idx], zorder=3)
1276
+ ax6b.text(v + 0.8, idx, lbl, va="center", ha="left", fontsize=8.2, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
1277
+ else:
1278
+ ax6b.barh(idx, 52.5, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
1279
+ ax6b.text(49.5, idx, lbl, va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
1280
+ ax6b.set_yticks(np.arange(len(m6b)))
1281
+ ax6b.set_yticklabels(m6b, fontsize=9, fontweight="bold", color=TEXT_MAIN)
1282
+ ax6b.invert_yaxis()
1283
+ ax6b.set_xlim(48, 144)
1284
+ ax6b.set_xticks([50, 60, 70, 80, 90, 100])
1285
+ ax6b.set_xlabel("Multi-Source Corpus (Identical N=10,278) & Public Tracker (N=757: 92.42%)", fontsize=8.2, color=TEXT_SUB, labelpad=6)
1286
+ ax6b.set_title("6B. Swarm Liveliness AUROC % (swarm_adapter.pt)", fontsize=10, fontweight="bold", color=TEXT_MAIN, pad=7)
1287
+
1288
+ sha_chart3 = save_clean_rgb_png(fig3, assets_dir / "armiger_vs_torrent_classifiers_6dims.png")
1289
+
1290
+ # ==========================================================================
1291
+ # COMPUTE & RENDER CHART 4: armiger_categorical_breakdown.png
1292
+ # ==========================================================================
1293
+ fig4, (ax4_l, ax4_r) = plt.subplots(1, 2, figsize=(22.5, 11.2), gridspec_kw={"wspace": 0.22})
1294
+ fig4.subplots_adjust(top=0.84, bottom=0.16, left=0.11, right=0.98)
1295
+ fig4.suptitle(
1296
+ "Per-Category Recall Breakdown Across All 10 Known Media & Software Categories",
1297
+ fontsize=17, fontweight="bold", color=TEXT_MAIN, y=0.965,
1298
+ )
1299
+ fig4.text(
1300
+ 0.5, 0.925,
1301
+ "Left: oct15_confirmation (Identical N = 2,835) vs. Open-Source Classifiers | Right: beyond_confirmation (Identical N = 1,025, LOTO Non-Circular)",
1302
+ ha="center", fontsize=11.2, style="italic", color=TEXT_SUB, parse_math=False,
1303
+ )
1304
+
1305
+ # Order for Left Panel (sorted by Armiger Ultra accuracy on oct15_confirmation)
1306
+ oct_cat_spec = [
1307
+ (6, "Audiobook"),
1308
+ (2, "Adult / XXX"),
1309
+ (1, "TV Show"),
1310
+ (0, "Movie"),
1311
+ (7, "Software"),
1312
+ (3, "Music"),
1313
+ (5, "Game"),
1314
+ (8, "Ebook"),
1315
+ (9, "Course"),
1316
+ (4, "Comic"),
1317
+ ]
1318
+ cats_oct_labels = []
1319
+ oct_ultra, oct_int8, oct_mimmo, oct_bit = [], [], [], []
1320
+ chart4_oct_manifest: Dict[str, Any] = {}
1321
+ for c_idx, c_title in oct_cat_spec:
1322
+ mask = (y_oct_cat11 == c_idx)
1323
+ n_c = int(np.sum(mask))
1324
+ cats_oct_labels.append(f"{c_title} (n={n_c:,})")
1325
+ u_v = float(np.mean(d["oct15_cat11_pred_ultra"][mask] == c_idx) * 100.0)
1326
+ i_v = float(np.mean(d["oct15_cat11_pred_int8_hyb"][mask] == c_idx) * 100.0)
1327
+ m_v = float(np.mean(d["oct15_cat11_pred_mimmo_bert"][mask] == c_idx) * 100.0)
1328
+ b_v = float(np.mean(d["oct15_cat11_pred_bitmagnet"][mask] == c_idx) * 100.0)
1329
+ oct_ultra.append(round(u_v, 2))
1330
+ oct_int8.append(round(i_v, 2))
1331
+ oct_mimmo.append(round(m_v, 2))
1332
+ oct_bit.append(round(b_v, 2))
1333
+ chart4_oct_manifest[c_title] = {
1334
+ "n": n_c,
1335
+ "armiger_ultra_pct": round(u_v, 2),
1336
+ "armiger_int8_pct": round(i_v, 2),
1337
+ "mimmo_bert_pct": round(m_v, 2),
1338
+ "bitmagnet_pct": round(b_v, 2),
1339
+ }
1340
+
1341
+ style_ax(ax4_l)
1342
+ y_c = np.arange(len(cats_oct_labels))
1343
+ h = 0.19
1344
+ ax4_l.barh(y_c - 1.5 * h, oct_ultra, height=h, color=C_ULTRA, label="Armiger Ultra Fair Hybrid (95.49% Overall)", zorder=3)
1345
+ ax4_l.barh(y_c - 0.5 * h, oct_int8, height=h, color=C_INT8, label="Armiger INT8 + ZeroCue (95.49% Overall, 95.06% Raw)", zorder=3)
1346
+ ax4_l.barh(y_c + 0.5 * h, oct_mimmo, height=h, color="#818CF8", label="MIMMO BERT 360M (72.24% Overall on N=2,835)", zorder=3)
1347
+ ax4_l.barh(y_c + 1.5 * h, oct_bit, height=h, color="#64748B", label="Bitmagnet Built-In (35.17% Overall on N=2,835)", zorder=3)
1348
+ ax4_l.set_yticks(y_c)
1349
+ ax4_l.set_yticklabels(cats_oct_labels, fontsize=10, fontweight="bold", color=TEXT_MAIN)
1350
+ ax4_l.set_ylim(len(cats_oct_labels) - 0.35, -0.6)
1351
+ ax4_l.set_xlim(0, 128)
1352
+ ax4_l.set_xticks([0, 20, 40, 60, 80, 100])
1353
+ ax4_l.set_xlabel("Per-Class Recall (%) — Identical N = 2,835 Ground-Truth Mask", fontsize=9.8, color=TEXT_SUB, labelpad=8)
1354
+ ax4_l.set_title("1. Per-Category Recall vs. Open-Source Classifiers\n(oct15_confirmation, Identical N = 2,835)", fontsize=11.5, fontweight="bold", color=TEXT_MAIN, pad=10)
1355
+ for idx in range(len(cats_oct_labels)):
1356
+ ax4_l.text(oct_ultra[idx] + 1.0, idx - 1.1 * h, f"{oct_ultra[idx]:.1f}%", va="center", ha="left", fontsize=8.0, fontweight="bold", color=C_ULTRA_EDGE)
1357
+ ax4_l.text(oct_mimmo[idx] + 1.0, idx + 0.5 * h, f"{oct_mimmo[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color="#CBD5E1")
1358
+ ax4_l.text(oct_bit[idx] + 1.0, idx + 1.5 * h, f"{oct_bit[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color=TEXT_SUB)
1359
+ ax4_l.legend(loc="upper center", bbox_to_anchor=(0.5, -0.09), ncol=2, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=8.6, labelcolor=TEXT_MAIN)
1360
+
1361
+ # Order for Right Panel (beyond_confirmation N=1,025 Non-Circular LOTO)
1362
+ y_bc_cat11 = d["beyond_cat11_y_true"]
1363
+ bc_cat_spec = [
1364
+ (6, "Audiobook"),
1365
+ (8, "Ebook"),
1366
+ (4, "Comic"),
1367
+ (0, "Movie"),
1368
+ (1, "TV Show"),
1369
+ (5, "Game"),
1370
+ (7, "Software"),
1371
+ (9, "Course"),
1372
+ (2, "Adult / XXX"),
1373
+ (3, "Music"),
1374
+ ]
1375
+ cats_bc_labels = []
1376
+ bc_ultra, bc_int8, bc_gpt, bc_gem = [], [], [], []
1377
+ chart4_bc_manifest: Dict[str, Any] = {}
1378
+ for c_idx, c_title in bc_cat_spec:
1379
+ mask = (y_bc_cat11 == c_idx)
1380
+ n_c = int(np.sum(mask))
1381
+ cats_bc_labels.append(f"{c_title} (n={n_c:,})")
1382
+ u_v = float(np.mean(d["beyond_cat11_corr_ultra"][mask]) * 100.0)
1383
+ i_v = float(np.mean(d["beyond_cat11_corr_int8_hyb"][mask]) * 100.0)
1384
+ gp_v = float(np.mean(d["beyond_cat11_corr_gpt_noncirc"][mask]) * 100.0)
1385
+ gm_v = float(np.mean(d["beyond_cat11_corr_gemini_noncirc"][mask]) * 100.0)
1386
+ bc_ultra.append(round(u_v, 2))
1387
+ bc_int8.append(round(i_v, 2))
1388
+ bc_gpt.append(round(gp_v, 2))
1389
+ bc_gem.append(round(gm_v, 2))
1390
+ chart4_bc_manifest[c_title] = {
1391
+ "n": n_c,
1392
+ "armiger_ultra_pct": round(u_v, 2),
1393
+ "armiger_int8_pct": round(i_v, 2),
1394
+ "gpt_noncirc_pct": round(gp_v, 2),
1395
+ "gemini_noncirc_pct": round(gm_v, 2),
1396
+ }
1397
+
1398
+ style_ax(ax4_r)
1399
+ ax4_r.barh(y_c - 1.5 * h, bc_ultra, height=h, color=C_ULTRA, label="Armiger Ultra Fair Hybrid (95.12% Non-Circular)", zorder=3)
1400
+ ax4_r.barh(y_c - 0.5 * h, bc_int8, height=h, color=C_INT8, label="Armiger INT8 + ZeroCue (95.12% Non-Circular, 92.88% Raw)", zorder=3)
1401
+ ax4_r.barh(y_c + 0.5 * h, bc_gpt, height=h, color=C_GPT, label="OpenAI GPT Fast Tier (95.12% Non-Circ | 96.59% Circ)", zorder=3)
1402
+ ax4_r.barh(y_c + 1.5 * h, bc_gem, height=h, color=C_GEMINI, label="Google Gemini Flash (93.80% Non-Circ | 95.61% Circ)", zorder=3)
1403
+ ax4_r.set_yticks(y_c)
1404
+ ax4_r.set_yticklabels(cats_bc_labels, fontsize=10, fontweight="bold", color=TEXT_MAIN)
1405
+ ax4_r.set_ylim(len(cats_bc_labels) - 0.35, -0.6)
1406
+ ax4_r.set_xlim(0, 128)
1407
+ ax4_r.set_xticks([0, 20, 40, 60, 80, 100])
1408
+ ax4_r.set_xlabel("Per-Class Recall (%) — Leave-One-Teacher-Out Non-Circular Reference", fontsize=9.8, color=TEXT_SUB, labelpad=8)
1409
+ ax4_r.set_title("2. Per-Category Recall vs. Cloud LLMs\n(beyond_confirmation, Identical N = 1,025, LOTO Non-Circular)", fontsize=11.5, fontweight="bold", color=TEXT_MAIN, pad=10)
1410
+ for idx in range(len(cats_bc_labels)):
1411
+ ax4_r.text(bc_ultra[idx] + 1.0, idx - 1.1 * h, f"{bc_ultra[idx]:.1f}%", va="center", ha="left", fontsize=8.0, fontweight="bold", color=C_ULTRA_EDGE)
1412
+ ax4_r.text(bc_gpt[idx] + 1.0, idx + 0.5 * h, f"{bc_gpt[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color=C_GPT)
1413
+ ax4_r.text(bc_gem[idx] + 1.0, idx + 1.5 * h, f"{bc_gem[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color=C_GEMINI)
1414
+ ax4_r.legend(loc="upper center", bbox_to_anchor=(0.5, -0.09), ncol=2, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=8.6, labelcolor=TEXT_MAIN)
1415
+
1416
+ sha_chart4 = save_clean_rgb_png(fig4, assets_dir / "armiger_categorical_breakdown.png")
1417
+ else:
1418
+ for fn in ("armiger_vs_torrent_classifiers_6dims.png", "armiger_categorical_breakdown.png"):
1419
+ verify_png_chunks_clean(fallback_assets / fn)
1420
+ sha_chart3 = hashlib.sha256((fallback_assets / "armiger_vs_torrent_classifiers_6dims.png").read_bytes()).hexdigest()
1421
+ sha_chart4 = hashlib.sha256((fallback_assets / "armiger_categorical_breakdown.png").read_bytes()).hexdigest()
1422
+
1423
+ # ==========================================================================
1424
+ # FIVE-SPLIT SUMMARY TABLE STATS
1425
+ # ==========================================================================
1426
+ five_split_summary: Dict[str, Any] = {}
1427
+ for sp_code, sp_label, n_exp in [
1428
+ ("oct15", "oct15_confirmation", 2835),
1429
+ ("fresh", "fresh_confirmation", 1402),
1430
+ ("beyond", "beyond_confirmation", 1025),
1431
+ ("tc2000", "tc_truespec_2000", 2000),
1432
+ ("tc3000", "tc_truespec_multilang_3000", 3000),
1433
+ ]:
1434
+ y_sp = d[f"split5_{sp_code}_y_cat"]
1435
+ sp_stats = compute_paired_panel_stats(
1436
+ f"five_split_{sp_code}",
1437
+ {
1438
+ "armiger_ultra_hyb": (d[f"split5_{sp_code}_fp32_hyb_pred"] == y_sp).astype(np.float32),
1439
+ "armiger_ultra_raw": (d[f"split5_{sp_code}_fp32_raw_pred"] == y_sp).astype(np.float32),
1440
+ "armiger_int8_hyb": (d[f"split5_{sp_code}_int8_hyb_pred"] == y_sp).astype(np.float32),
1441
+ "armiger_int8_raw": (d[f"split5_{sp_code}_int8_raw_pred"] == y_sp).astype(np.float32),
1442
+ },
1443
+ ref_model="armiger_ultra_hyb",
1444
+ expected_n=n_exp,
1445
+ metric_type="mean",
1446
+ )
1447
+ five_split_summary[sp_label] = sp_stats
1448
+
1449
+ # ==========================================================================
1450
+ # WRITE CRYPTOGRAPHIC PROVENANCE MANIFEST (CHARTS_MANIFEST.json)
1451
+ # ==========================================================================
1452
+ repo_root = assets_dir.parent
1453
+ prov_hashes: Dict[str, str] = {
1454
+ "canonical_eval_vectors.json": hashlib.sha256(npz_path.read_bytes()).hexdigest(),
1455
+ "generate_canonical_charts.py": hashlib.sha256(pathlib.Path(__file__).read_bytes()).hexdigest(),
1456
+ "assets/armiger_vs_llms_overall_and_6dims.png": sha_chart1,
1457
+ "assets/armiger_vs_llms_cost_comparison.png": sha_chart2,
1458
+ "assets/armiger_vs_torrent_classifiers_6dims.png": sha_chart3,
1459
+ "assets/armiger_categorical_breakdown.png": sha_chart4,
1460
+ }
1461
+ for opt_rel in [
1462
+ "validated_zero_cue_catalog.json",
1463
+ "armiger_ultra.fp32.onnx",
1464
+ "armiger.int8.onnx",
1465
+ "models/armiger_ultra.fp32.onnx",
1466
+ "models/armiger.int8.onnx",
1467
+ ]:
1468
+ p_opt = repo_root / opt_rel
1469
+ if p_opt.is_file():
1470
+ prov_hashes[opt_rel] = hashlib.sha256(p_opt.read_bytes()).hexdigest()
1471
+
1472
+ manifest: Dict[str, Any] = {
1473
+ "schema_version": "1.0.0",
1474
+ "bootstrap_seed": BOOTSTRAP_SEED,
1475
+ "bootstrap_resamples": BOOTSTRAP_RESAMPLES,
1476
+ "invariants_verified": {
1477
+ "identical_denominator_lock": True,
1478
+ "anti_circularity_lock": True,
1479
+ "paired_uncertainty_and_significance": True,
1480
+ "png_metadata_stripped_ihdr_idat_iend_only": True,
1481
+ },
1482
+ "anti_circularity_audit": anti_circ_checks,
1483
+ "provenance_sha256": prov_hashes,
1484
+ "charts": {
1485
+ "armiger_vs_llms_overall_and_6dims.png": {
1486
+ "sha256": sha_chart1,
1487
+ "composite_6dim_mean": composite_stats,
1488
+ "panel_2a_category_11class": p2a_stats,
1489
+ "panel_2b_catalog_matching": p2b_stats,
1490
+ "panel_2c_audio_strict_229": p2c_stats,
1491
+ "panel_2d_anime_valid_636": p2d_stats,
1492
+ "panel_2e_junk_balanced_977": p2e_stats,
1493
+ "panel_2f_swarm_public_757": p2f_pub_stats,
1494
+ "panel_2f_swarm_warehouse_10278": p2f_whse_stats,
1495
+ },
1496
+ "armiger_vs_llms_cost_comparison.png": {
1497
+ "sha256": sha_chart2,
1498
+ "models": models_cost,
1499
+ "batched_usd_per_1m": batched_cost,
1500
+ "realtime_usd_per_1m": realtime_cost,
1501
+ },
1502
+ "armiger_vs_torrent_classifiers_6dims.png": {
1503
+ "sha256": sha_chart3,
1504
+ "panel_1_category_11class_oct15": c3_p1_stats,
1505
+ "panel_1_video_mvtv_oct15": c3_p1_vid_stats,
1506
+ "panel_2_catalog_matching_oct15": c3_p2_stats,
1507
+ "panel_3_audio_strict_oct15": c3_p3_stats,
1508
+ "panel_4_anime_oct15": c3_p4_oct_stats,
1509
+ "panel_4_anime_dub_tc2000": c3_p4_tc2k_stats,
1510
+ "panel_5_adult_oct15": c3_p5_stats,
1511
+ "panel_6a_junk_oct15": c3_p6a_stats,
1512
+ },
1513
+ "armiger_categorical_breakdown.png": {
1514
+ "sha256": sha_chart4,
1515
+ "left_oct15_confirmation_n2835": chart4_oct_manifest,
1516
+ "right_beyond_confirmation_n1025_loto": chart4_bc_manifest,
1517
+ },
1518
+ },
1519
+ "five_split_category_summary": five_split_summary,
1520
+ }
1521
+ manifest_path.parent.mkdir(parents=True, exist_ok=True)
1522
+ manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=False) + "\n", encoding="utf-8")
1523
+ return manifest
1524
+
1525
+
1526
+ def main() -> None:
1527
+ parser = argparse.ArgumentParser(description="Generate canonical Armiger benchmark charts and CHARTS_MANIFEST.json.")
1528
+ here = pathlib.Path(__file__).resolve().parent
1529
+ parser.add_argument("--vectors", type=pathlib.Path, default=here / "canonical_eval_vectors.json")
1530
+ parser.add_argument("--assets-dir", type=pathlib.Path, default=here.parent / "assets")
1531
+ parser.add_argument("--manifest", type=pathlib.Path, default=here / "CHARTS_MANIFEST.json")
1532
+ args = parser.parse_args()
1533
+
1534
+ manifest = build_all_charts_and_manifest(args.vectors, args.assets_dir, args.manifest)
1535
+ for name, info in manifest["charts"].items():
1536
+ print(f"[OK] {name:46s} sha256={info['sha256'][:16]}...")
1537
+ print(f"[OK] Wrote {args.manifest}")
1538
+
1539
+
1540
+ if __name__ == "__main__":
1541
+ main()
benchmarks/test_chart_integrity.py ADDED
@@ -0,0 +1,192 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Automated integrity and statistical invariant test suite for the canonical
3
+ Armiger & Armiger Ultra chart and benchmark pipeline.
4
+ """
5
+ from __future__ import annotations
6
+
7
+ import hashlib
8
+ import json
9
+ import pathlib
10
+ import sys
11
+
12
+ import numpy as np
13
+ import pytest
14
+
15
+ BENCH_DIR = pathlib.Path(__file__).resolve().parent
16
+ REPO_ROOT = BENCH_DIR.parent
17
+ if str(BENCH_DIR) not in sys.path:
18
+ sys.path.insert(0, str(BENCH_DIR))
19
+
20
+
21
+ def _load_vectors(path: pathlib.Path):
22
+ raw = json.loads(path.read_text(encoding="utf-8"))
23
+ dtypes = raw["_meta"]["dtypes"]
24
+ return {k: np.array(v, dtype=np.dtype(dtypes[k])) for k, v in raw["vectors"].items()}
25
+
26
+ from generate_canonical_charts import ( # noqa: E402
27
+ BOOTSTRAP_RESAMPLES,
28
+ BOOTSTRAP_SEED,
29
+ HAS_MATPLOTLIB,
30
+ assert_identical_denominator,
31
+ assert_non_circular_teacher_eval,
32
+ build_all_charts_and_manifest,
33
+ compute_paired_panel_stats,
34
+ verify_png_chunks_clean,
35
+ )
36
+
37
+
38
+ def test_identical_denominator_lock_enforced():
39
+ """Verify that any denominator mismatch across models on a panel raises AssertionError."""
40
+ v_1025 = np.ones(1025, dtype=np.float32)
41
+ v_1009 = np.ones(1009, dtype=np.float32) # Dropped 16 'unknown' items
42
+ v_2835 = np.ones(2835, dtype=np.float32)
43
+ v_2825 = np.ones(2825, dtype=np.float32) # Dropped 10 'unknown' items
44
+ v_545 = np.ones(545, dtype=np.float32)
45
+ v_849 = np.ones(849, dtype=np.float32) # Mismatched audio language split
46
+
47
+ assert assert_identical_denominator("ok_1025", {"ultra": v_1025, "gpt": v_1025}, expected_n=1025) == 1025
48
+
49
+ with pytest.raises(AssertionError, match="Identical Denominator Lock violated"):
50
+ assert_identical_denominator("bad_beyond_unknown_drop", {"ultra": v_1025, "gpt": v_1009})
51
+
52
+ with pytest.raises(AssertionError, match="Identical Denominator Lock violated"):
53
+ assert_identical_denominator("bad_oct15_unknown_drop", {"ultra": v_2835, "mimmo": v_2825})
54
+
55
+ with pytest.raises(AssertionError, match="Identical Denominator Lock violated"):
56
+ assert_identical_denominator("bad_audio_split_mix", {"ultra": v_545, "gpt": v_849})
57
+
58
+ with pytest.raises(AssertionError, match="Expected N=636"):
59
+ assert_identical_denominator("bad_expected_n", {"ultra": np.ones(650), "gpt": np.ones(650)}, expected_n=636)
60
+
61
+ v_nan = v_1025.copy()
62
+ v_nan[0] = np.nan
63
+ with pytest.raises(AssertionError, match="NaN"):
64
+ assert_identical_denominator("bad_nan", {"ultra": v_1025, "gpt": v_nan})
65
+
66
+
67
+ def test_anti_circularity_lock_enforced():
68
+ """Verify that scoring a teacher model against its own circular majority vote raises AssertionError."""
69
+ d = _load_vectors(BENCH_DIR / "canonical_eval_vectors.json")
70
+
71
+ with pytest.raises(AssertionError, match="Anti-Circularity Lock violated"):
72
+ assert_non_circular_teacher_eval("OpenAI GPT (Fast Tier)", eval_mode="circular_majority", self_vote_weight=0.333)
73
+
74
+ with pytest.raises(AssertionError, match="Anti-Circularity Lock violated"):
75
+ assert_non_circular_teacher_eval("Google Gemini Flash", eval_mode="leave_one_teacher_out", self_vote_weight=0.5)
76
+
77
+ gpt_audit = assert_non_circular_teacher_eval(
78
+ "OpenAI GPT (Fast Tier)",
79
+ eval_mode="leave_one_teacher_out",
80
+ self_vote_weight=0.0,
81
+ noncirc_vec=d["beyond_cat11_corr_gpt_noncirc"],
82
+ circ_vec=d["beyond_cat11_corr_gpt_circ"],
83
+ )
84
+ assert gpt_audit["non_circular_pct"] == 95.12
85
+ assert gpt_audit["circular_self_vote_pct"] == 96.59
86
+ assert gpt_audit["circularity_inflation_pp"] > 1.0
87
+
88
+ gem_audit = assert_non_circular_teacher_eval(
89
+ "Google Gemini Flash",
90
+ eval_mode="leave_one_teacher_out",
91
+ self_vote_weight=0.0,
92
+ noncirc_vec=d["beyond_cat11_corr_gemini_noncirc"],
93
+ circ_vec=d["beyond_cat11_corr_gemini_circ"],
94
+ )
95
+ assert gem_audit["non_circular_pct"] == 93.80
96
+ assert gem_audit["circular_self_vote_pct"] == 95.61
97
+ assert gem_audit["circularity_inflation_pp"] > 1.5
98
+
99
+
100
+ def test_eval_vectors_privacy_and_dtypes():
101
+ """Verify that canonical_eval_vectors.json contains only numeric int16/float32 arrays and zero strings."""
102
+ npz_path = BENCH_DIR / "canonical_eval_vectors.json"
103
+ assert npz_path.is_file()
104
+ d = _load_vectors(npz_path)
105
+ assert len(list(d.keys())) >= 140
106
+ for k in list(d.keys()):
107
+ arr = d[k]
108
+ assert arr.dtype in (np.int16, np.float32), f"Array {k} has disallowed dtype {arr.dtype}"
109
+ assert arr.ndim == 1, f"Array {k} must be 1D, got shape {arr.shape}"
110
+ assert not np.isnan(arr).any(), f"Array {k} contains NaNs"
111
+
112
+
113
+ def test_paired_bootstrap_and_mcnemar_reproducibility():
114
+ """Verify that paired bootstrap 95% CIs and McNemar p-values are 100% deterministic."""
115
+ d = _load_vectors(BENCH_DIR / "canonical_eval_vectors.json")
116
+ vecs = {
117
+ "Armiger Ultra": (d["oct15_aud545_pred_ultra"] == d["oct15_aud545_y_en"]).astype(np.float32),
118
+ "Armiger INT8": (d["oct15_aud545_pred_int8_hyb"] == d["oct15_aud545_y_en"]).astype(np.float32),
119
+ "GuessIt": (d["oct15_aud545_pred_guessit"] == d["oct15_aud545_y_en"]).astype(np.float32),
120
+ "OpenAI GPT (Fast Tier)": (d["oct15_aud545_pred_gpt"] == d["oct15_aud545_y_en"]).astype(np.float32),
121
+ }
122
+ res1 = compute_paired_panel_stats("test_aud545", vecs, ref_model="Armiger Ultra", expected_n=545)
123
+ res2 = compute_paired_panel_stats("test_aud545", vecs, ref_model="Armiger Ultra", expected_n=545)
124
+ assert res1 == res2
125
+ for m, st in res1["models"].items():
126
+ assert st["n"] == 545
127
+ assert st["ci95_lo"] <= st["point_pct"] <= st["ci95_hi"]
128
+ assert 0.0 <= st["p_value_vs_ref"] <= 1.0
129
+ assert res1["models"]["Armiger Ultra"]["point_pct"] == 97.80
130
+ assert res1["models"]["GuessIt"]["point_pct"] == 94.86
131
+ assert res1["models"]["OpenAI GPT (Fast Tier)"]["point_pct"] == 94.68
132
+ assert res1["models"]["OpenAI GPT (Fast Tier)"]["p_value_vs_ref"] < 0.01
133
+
134
+
135
+ def test_manifest_sha256_and_png_chunks(tmp_path: pathlib.Path):
136
+ """Verify CHARTS_MANIFEST.json provenance hashes, PNG chunk hygiene, and byte-exact determinism."""
137
+ manifest_path = BENCH_DIR / "CHARTS_MANIFEST.json"
138
+ assert manifest_path.is_file(), "CHARTS_MANIFEST.json missing"
139
+ manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
140
+
141
+ assert manifest["bootstrap_seed"] == BOOTSTRAP_SEED
142
+ assert manifest["bootstrap_resamples"] == BOOTSTRAP_RESAMPLES
143
+ for k, v in manifest["invariants_verified"].items():
144
+ assert v is True, f"Invariant {k} not verified"
145
+
146
+ # Verify all provenance SHA-256 hashes match disk
147
+ for rel_path, expected_sha in manifest["provenance_sha256"].items():
148
+ target = (BENCH_DIR / rel_path) if (BENCH_DIR / rel_path).is_file() else (REPO_ROOT / rel_path)
149
+ assert target.is_file(), f"Provenance file missing: {rel_path}"
150
+ actual_sha = hashlib.sha256(target.read_bytes()).hexdigest()
151
+ assert actual_sha == expected_sha, f"SHA-256 mismatch for {rel_path}: {actual_sha} != {expected_sha}"
152
+
153
+ # Verify all 4 PNGs have only IHDR, IDAT, IEND chunks and RGB mode
154
+ for chart_name, chart_meta in manifest["charts"].items():
155
+ png_path = REPO_ROOT / "assets" / chart_name
156
+ assert png_path.is_file(), f"Chart missing: {png_path}"
157
+ chunks = verify_png_chunks_clean(png_path)
158
+ assert chunks[0] == "IHDR" and chunks[-1] == "IEND"
159
+ assert set(chunks).issubset({"IHDR", "IDAT", "IEND"})
160
+
161
+ # Re-run pipeline into tmp_path and verify statistical + PNG determinism
162
+ tmp_assets = tmp_path / "assets"
163
+ tmp_manifest = tmp_path / "CHARTS_MANIFEST.json"
164
+ render_mode = bool(HAS_MATPLOTLIB and sys.platform == "darwin")
165
+ re_manifest = build_all_charts_and_manifest(
166
+ BENCH_DIR / "canonical_eval_vectors.json",
167
+ tmp_assets,
168
+ tmp_manifest,
169
+ render_pngs=render_mode,
170
+ )
171
+ assert re_manifest["invariants_verified"] == manifest["invariants_verified"]
172
+ assert re_manifest["anti_circularity_audit"] == manifest["anti_circularity_audit"]
173
+ assert re_manifest["five_split_category_summary"] == manifest["five_split_category_summary"]
174
+ for chart_name, chart_meta in manifest["charts"].items():
175
+ assert re_manifest["charts"][chart_name] == chart_meta, (
176
+ f"Non-deterministic manifest or PNG output for {chart_name}"
177
+ )
178
+
179
+
180
+ def test_readme_tables_match_charts_manifest():
181
+ """Verify README.md contains canonical CHARTS_MANIFEST.json metrics and zero stale numbers."""
182
+ readme_path = REPO_ROOT / "README.md"
183
+ assert readme_path.is_file()
184
+ text = readme_path.read_text(encoding="utf-8")
185
+
186
+ # Canonical numbers that must appear in README
187
+ for required_token in ("95.49%", "97.30%", "96.53%", "97.80%", "96.83%"):
188
+ assert required_token in text, f"Missing canonical metric {required_token} in {readme_path}"
189
+
190
+ # Stale pre-audit numbers that must never reappear in README
191
+ for stale_token in ("97.13%", "96.35%", "72.35%", "35.29%", "97.55%"):
192
+ assert stale_token not in text, f"Stale metric {stale_token} found in {readme_path}"
config.json CHANGED
@@ -62,16 +62,16 @@
62
  },
63
  "artifacts": {
64
  "csam_guard.py": {
65
- "size_bytes": 42555,
66
- "sha256": "fa96e13ad86e50451b668ac10275a7afa472111a9449f2183c7a253d34d3776c"
67
  },
68
  "armiger_dht_db_sync.py": {
69
- "size_bytes": 51391,
70
- "sha256": "4ef834f352b3725be2e5d92ce6c29f92f59ef0e5c584971a9d26954ad9502539"
71
  },
72
  "test_dht_db_sync.py": {
73
  "size_bytes": 62482,
74
- "sha256": "ed6cebbcbaebc0d43516f68952937696a6b6c88ec54da8536112f9127b4f4290"
75
  }
76
  }
77
  }
 
62
  },
63
  "artifacts": {
64
  "csam_guard.py": {
65
+ "size_bytes": 44802,
66
+ "sha256": "b76eff993f86f2a08c2aecaa39a4109b57f40be1d1b2b0066803e10cb33a0f96"
67
  },
68
  "armiger_dht_db_sync.py": {
69
+ "size_bytes": 69631,
70
+ "sha256": "27107c7a49e79ab0874c50881e42d742dc91d57dc049d11898dd4d04efa9bed1"
71
  },
72
  "test_dht_db_sync.py": {
73
  "size_bytes": 62482,
74
+ "sha256": "c6a58c19ca3d698405132f46bde74ce038639b08c06ba9f00f2cee1f1dfdaf8f"
75
  }
76
  }
77
  }
csam_guard.py CHANGED
@@ -848,9 +848,13 @@ class WOTSPlusSHAKE256:
848
  return digits
849
 
850
  @classmethod
851
- @functools.lru_cache(maxsize=256)
852
  def generate_keypair(cls, seed: Optional[bytes] = None) -> Tuple[bytes, bytes]:
853
  raw = seed if seed is not None else os.urandom(64)
 
 
 
 
 
854
  if len(raw) == 32:
855
  raw = hashlib.shake_256(b"armiger-wots-keygen-v1:" + raw).digest(64)
856
  elif len(raw) != 64:
 
848
  return digits
849
 
850
  @classmethod
 
851
  def generate_keypair(cls, seed: Optional[bytes] = None) -> Tuple[bytes, bytes]:
852
  raw = seed if seed is not None else os.urandom(64)
853
+ return cls._generate_keypair_from_seed(raw)
854
+
855
+ @classmethod
856
+ @functools.lru_cache(maxsize=256)
857
+ def _generate_keypair_from_seed(cls, raw: bytes) -> Tuple[bytes, bytes]:
858
  if len(raw) == 32:
859
  raw = hashlib.shake_256(b"armiger-wots-keygen-v1:" + raw).digest(64)
860
  elif len(raw) != 64:
docs/ANONYMOUS_OPT_IN_DHT.md CHANGED
@@ -1,6 +1,6 @@
1
  # Armiger Distributed Anonymous & Opt-In-By-Default DHT Sharing
2
 
3
- This document specifies the cryptographic, differential-privacy, and wire-level firewall architecture governing distributed weight, calibration, and catalog metadata sharing in **Armiger** and **Armiger Ultra** (`armiger_dht_rsi.py`, `armiger_dht_db_sync.py`, `armiger_ultra_pipeline.py`), along with the machine-verifiable results from the live 12-node UDP BEP-44 proof suite (`benchmarks/proof_anonymous_opt_in_sharing.py`).
4
 
5
  ---
6
 
@@ -61,7 +61,7 @@ Whenever an operator explicitly opts in (`opt_in_sharing=True` or `ARMIGER_DHT_S
61
 
62
  ---
63
 
64
- ## 3. Live 12-Node UDP BEP-44 Verification Summary
65
 
66
  Reproduce all three proofs locally with:
67
 
@@ -74,4 +74,4 @@ pytest test_anonymous_opt_in_dht_sharing.py
74
  | :--- | :--- | :--- | :--- |
75
  | **Proof 1: Default Zero-Egress** | 9 default components, 100 synthetic webhook/catalog events | `opt_in_sharing=False` on 9/9; `0` bundles & `0` UDP `put` packets emitted (`2,600` outbound attempts blocked) | **PASS** |
76
  | **Proof 2: Wire Canary Firewall** | 500 adversarial PII/canary payloads across 10 categories on 12-node UDP swarm | `500 / 500` (`100.0%`) blocked before wire; `0 / 500` (`0.0%`) canary substrings in raw UDP datagrams | **PASS** |
77
- | **Proof 3: Unlinkability & DP Convergence** | 12-node UDP swarm over 5 epochs (60 publications) | `60 / 60` unique ephemeral Ed25519 keys (`8.33%` random linkability); `100%` hour-bucketed timestamps; MIA accuracy `0.545`; ECE `0.08094 -> 0.01300` (`-83.94%`); Ultra F1 `0.93427 -> 0.98074` | **PASS** |
 
1
  # Armiger Distributed Anonymous & Opt-In-By-Default DHT Sharing
2
 
3
+ This document specifies the cryptographic, differential-privacy, and wire-level firewall architecture governing distributed weight, calibration, and catalog metadata sharing in **Armiger** and **Armiger Ultra** (`armiger_dht_rsi.py`, `armiger_dht_db_sync.py`, `armiger_ultra_pipeline.py`), along with the machine-verifiable results from the 12-node local UDP BEP-44 proof & synthetic DP convergence suite (`benchmarks/proof_anonymous_opt_in_sharing.py`).
4
 
5
  ---
6
 
 
61
 
62
  ---
63
 
64
+ ## 3. 12-Node Local UDP BEP-44 & Synthetic DP Convergence Verification Summary
65
 
66
  Reproduce all three proofs locally with:
67
 
 
74
  | :--- | :--- | :--- | :--- |
75
  | **Proof 1: Default Zero-Egress** | 9 default components, 100 synthetic webhook/catalog events | `opt_in_sharing=False` on 9/9; `0` bundles & `0` UDP `put` packets emitted (`2,600` outbound attempts blocked) | **PASS** |
76
  | **Proof 2: Wire Canary Firewall** | 500 adversarial PII/canary payloads across 10 categories on 12-node UDP swarm | `500 / 500` (`100.0%`) blocked before wire; `0 / 500` (`0.0%`) canary substrings in raw UDP datagrams | **PASS** |
77
+ | **Proof 3: Unlinkability & Synthetic DP Convergence** | 12-node local UDP swarm over 5 epochs (60 publications, synthetic adapter + temperature convergence) | `60 / 60` unique ephemeral Ed25519 keys (`8.33%` random linkability); `100%` hour-bucketed timestamps; MIA empirical accuracy `0.590`; ECE `0.08094 -> 0.01368` (`-83.10%`); Ultra F1 `0.93427 -> 0.98004` | **PASS** |
docs/CHART_PIPELINE.md ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Canonical Statistical Chart & Benchmark Pipeline (`v2.1.0`)
2
+
3
+ Every benchmark figure and headline accuracy table across the **Armiger** and **Armiger Ultra** repositories is generated deterministically from anonymized evaluation vectors (`benchmarks/canonical_eval_vectors.json`) and locked by a cryptographic provenance manifest (`benchmarks/CHARTS_MANIFEST.json`).
4
+
5
+ ---
6
+
7
+ ## 1. Pipeline Architecture & Artifacts
8
+
9
+ | Artifact | SHA-256 Digest | Description |
10
+ | :--- | :--- | :--- |
11
+ | `benchmarks/canonical_eval_vectors.json` | `058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a` | Plain-text JSON containing `148` 1D numeric `int16`/`float32` prediction and ground-truth label arrays across all 5 evaluation splits (`N=10,262`). Contains **zero** strings, release titles, file paths, or infohashes. |
12
+ | `benchmarks/generate_canonical_charts.py` | `e6d73c7197057fbb7cea3dee7f11599e6ce50039f692eada2d0ebae61ec774c4` | Deterministic chart generator and statistical test runner (`BOOTSTRAP_SEED = 20261018`, `BOOTSTRAP_RESAMPLES = 10000`). |
13
+ | `benchmarks/CHARTS_MANIFEST.json` | *(Tracked by `test_chart_integrity.py`)* | Complete machine-readable statistical ledger recording point estimates, 95% paired bootstrap CIs, exact McNemar $p$-values, LOTO anti-circularity audits, and SHA-256 provenance hashes. |
14
+ | `assets/armiger_vs_llms_overall_and_6dims.png` | `0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62` | 6-dimension accuracy & composite mean comparison against frontier cloud LLMs and local GPU LLMs on identical evaluation cohorts. |
15
+ | `assets/armiger_vs_llms_cost_comparison.png` | `3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4` | Operational cost per 1M torrents and token/hardware rate comparison. |
16
+ | `assets/armiger_vs_torrent_classifiers_6dims.png` | `e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010` | 7-dimension comparison against open-source torrent classifiers and regex parsers (`MIMMO BERT`, `RTN`, `Parsett`, `GuessIt`, `PTN`, `Anitomy`, `Bitmagnet`). |
17
+ | `assets/armiger_categorical_breakdown.png` | `1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98` | Per-category recall breakdown across all 10 media & software categories on `oct15_confirmation` (`N=2,835`) and `beyond_confirmation` (`N=1,025`). |
18
+
19
+ ---
20
+
21
+ ## 2. Four Machine-Enforced Statistical Invariants
22
+
23
+ `benchmarks/test_chart_integrity.py` enforces four strict invariants on every CI run:
24
+
25
+ ### Invariant 1 — Identical Denominator Lock (`assert_identical_denominator`)
26
+ Every model compared on a benchmark panel must be evaluated on the **exact same item mask and denominator $N$**. Dropping abstentions (`unknown`), mixing evaluation splits, or evaluating on a subset raises an immediate `AssertionError`:
27
+ - `oct15_confirmation` 11-Class Category: $N = 2,835$
28
+ - `beyond_confirmation` 11-Class Category & Catalog Matching: $N = 1,025$
29
+ - `oct15_confirmation` Movie-vs-TV & Catalog Matching: $N = 1,296$
30
+ - `oct15_confirmation` Binary English-vs-Foreign Audio: $N = 545$ (`beyond_confirmation` strict audio: $N = 229$)
31
+ - `oct15_confirmation` Anime Detection: $N = 507$ (`beyond_confirmation` valid media: $N = 636$)
32
+ - `oct15_confirmation` Junk / Malware Gate: $N = 2,170$ (`beyond_confirmation`: $N = 977$)
33
+ - `Chart 1 Panel 2F` Swarm Liveliness AUROC: $N = 757$ Public-Tracker Cohort across all 6 compared systems (`Chart 3 Panel 6B`: $N = 10,278$ Multi-Source Corpus across all 3 compared methods)
34
+
35
+ ### Invariant 2 — Anti-Circularity Lock (`assert_non_circular_teacher_eval`)
36
+ On `beyond_confirmation` ($N = 1,025$), where multi-teacher consensus contributed to reference curation, teacher models (`OpenAI GPT (Fast Tier)` and `Google Gemini Flash`) are strictly evaluated against **Leave-One-Teacher-Out (LOTO)** reference keys (`self_vote_weight = 0.0`):
37
+
38
+ | Teacher Model | Non-Circular LOTO Accuracy (`self_vote_weight = 0.0`) | Circular Self-Voted Accuracy | Removed Circularity Inflation |
39
+ | :--- | :---: | :---: | :---: |
40
+ | `OpenAI GPT (Fast Tier)` | **`95.12%`** | `96.59%` | `+1.46 pp` |
41
+ | `Google Gemini Flash` | **`93.80%`** | `95.61%` | `+1.80 pp` |
42
+
43
+ Scoring any teacher model with `self_vote_weight > 0.0` or `eval_mode != "leave_one_teacher_out"` raises an `AssertionError`.
44
+
45
+ ### Invariant 3 — Paired Bootstrap 95% CIs, Hájek/DeLong AUROC CIs & Exact McNemar $p$-Values (`compute_paired_panel_stats`)
46
+ All accuracy uncertainty intervals and significance tests are paired by item index across `10,000` bootstrap resamples (`seed = 20261018`) using integer-scaled `float64` dot products (eliminating BLAS thread non-determinism across macOS ARM64 and Linux x86_64), two-sample Hájek/DeLong rank-placement variance for AUROC panels, and exact binomial `math.comb` McNemar two-sided $p$-values.
47
+
48
+ ### Invariant 4 — Cryptographic PNG Hygiene (`verify_png_chunks_clean`)
49
+ Every generated PNG is stripped of non-deterministic metadata (`tEXt`, `iTXt`, `zTXt`, `tIME`, `pHYs`) so only `IHDR`, `IDAT`, and `IEND` chunks remain, guaranteeing byte-exact reproducibility.
50
+
51
+ ---
52
+
53
+ ## 3. Running Verification Locally
54
+
55
+ ```bash
56
+ pytest benchmarks/test_chart_integrity.py -v
57
+ ```
docs/INTEGRATION_GUIDE.md ADDED
@@ -0,0 +1,282 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Armiger & Armiger Ultra Ecosystem Integration Guide
2
+
3
+ This guide provides step-by-step instructions, turnkey Docker Compose stacks, and reference HTTP adapters for integrating **Armiger** (`8.3 MB INT8 CPU`) and **Armiger Ultra** (`32.3 MB FP32 GPU/CPU` + 6 official plugins) with:
4
+
5
+ 1. **[Part A — Integrating with `DH-Waypoint`](#part-a--integrating-armiger-with-dh-waypoint)** (`https://github.com/spencernorton/DH-Waypoint`):
6
+ - Native `dh-waypoint.processor.v1` processor contract (`GET /v1/manifest`, `GET /v1/status`, `POST /v1/process`, and batch `POST /v1/classify`)
7
+ - Pre-ingest `dh-waypoint.processors.admission.v1` cohort admission gate (`POST /v1/admit`)
8
+ - Closed-loop 4-tier `dh-waypoint.processors.feedback.v1` outcome telemetry (`POST /v1/feedback`)
9
+ 2. **[Part B — Integrating with `Bitmagnet`](#part-b--integrating-armiger-with-bitmagnet)**:
10
+ - Pre-ingest / post-crawl batch classifier & malware/honeypot pruner (`examples/bitmagnet/bitmagnet_armiger_bridge.py`)
11
+ - GraphQL `torrentDelete` / content enrichment bridge & Torznab proxy enrichment
12
+ 3. **[Part C — Integrating with Torznab Index Consumers (`Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`) & `qBittorrent`](#part-c--integrating-armiger-with-torznab-index-consumers--qbittorrent)**:
13
+ - Pre-grab Torznab filtering proxy (`examples/arr-stack/armiger_torznab_proxy.py`) blocking disguised `.exe`/`.scr`/`.lnk` malware, fake-seeder honeypot traps, dead swarms, and unrequested foreign-language audio tracks before grab
14
+ - Post-import webhook & `ffprobe` physical ground-truth receiver (`examples/arr-stack/armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier` for local conformal calibration and opt-in anonymous `BEP-44`/`BEP-46` RSI knowledge sharing
15
+
16
+ ---
17
+
18
+ ## Architecture Overview
19
+
20
+ ```mermaid
21
+ flowchart LR
22
+ subgraph Indexers["DHT Indexers & Crawlers"]
23
+ DHW["DH-Waypoint Core<br/>(processorapi/v1)"]
24
+ BM["Bitmagnet<br/>(GraphQL + Torznab)"]
25
+ PR["Prowlarr<br/>(Indexer Manager)"]
26
+ end
27
+
28
+ subgraph ArmigerSuite["Armiger / Armiger Ultra"]
29
+ PROC["Armiger Waypoint Processor<br/>(/v1/manifest, /v1/process, /v1/classify, /v1/admit, /v1/feedback)"]
30
+ BMBR["Bitmagnet Armiger Bridge<br/>(/v1/bitmagnet/classify)"]
31
+ TPROXY["Armiger Torznab Proxy<br/>(Port 8097 Admission Gate)"]
32
+ WHOOK["Armiger Webhook & ffprobe Receiver<br/>(Port 8098 Ground Truth + BEP-44 RSI)"]
33
+ end
34
+
35
+ subgraph ArrStack["*Arr Media Managers & Download Client"]
36
+ SON["Sonarr / Radarr / Lidarr / Readarr"]
37
+ QBT["qBittorrent / Transmission"]
38
+ end
39
+
40
+ DHW <-->|"HTTP processorapi/v1"| PROC
41
+ BM <-->|"Batch Classify + GraphQL Prune"| BMBR
42
+ DHW -->|"Torznab Feed"| TPROXY
43
+ BM -->|"Torznab Feed"| TPROXY
44
+ PR -->|"Torznab Feed"| TPROXY
45
+ TPROXY -->|"Sanitized & Enriched XML"| SON
46
+ SON -->|"Grab Approved Release"| QBT
47
+ SON -->|"On Import Webhook"| WHOOK
48
+ QBT -->|"On Completion + ffprobe"| WHOOK
49
+ WHOOK -.->|"Local Calibration & Opt-In BEP-44 RSI"| PROC
50
+ ```
51
+
52
+ ---
53
+
54
+ ## Part A — Integrating Armiger with `DH-Waypoint`
55
+
56
+ [`DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint) includes a built-in external processor subsystem (`processorapi/v1`) that connects directly to [`examples/dh-waypoint/armiger_waypoint_processor.py`](../examples/dh-waypoint/armiger_waypoint_processor.py).
57
+
58
+ ### 1. Turnkey Docker Compose Stack
59
+
60
+ Launch `DH-Waypoint`, `PostgreSQL`, and `Armiger Ultra` together using [`examples/dh-waypoint/docker-compose.yml`](../examples/dh-waypoint/docker-compose.yml):
61
+
62
+ ```bash
63
+ export POSTGRES_PASSWORD="<postgres-password>"
64
+ docker compose -f examples/dh-waypoint/docker-compose.yml up -d
65
+ ```
66
+
67
+ ### 2. Drop-In Plugin Manifest (`07-armiger-ultra.plugin.yml`)
68
+
69
+ Mount [`examples/dh-waypoint/07-armiger-ultra.plugin.yml`](../examples/dh-waypoint/07-armiger-ultra.plugin.yml) into `DH_WAYPOINT_PROCESSORS_PLUGIN_DIR` (`/etc/dh-waypoint/plugins.d/07-armiger-ultra.plugin.yml`):
70
+
71
+ ```yaml
72
+ processor:
73
+ name: armiger-ultra
74
+ endpoint: http://armiger-ultra:8095/v1/classify
75
+ timeout: 5s
76
+ max_batch: 64
77
+ required: false
78
+ forward_files: true
79
+ max_files: 64
80
+ forward_trackers: true
81
+ forward_swarm: true
82
+
83
+ admission:
84
+ enabled: true
85
+ endpoint: http://armiger-ultra:8095/v1/admit
86
+ timeout: 3s
87
+ max_batch: 64
88
+ max_files: 64
89
+ fail_open: true
90
+
91
+ learning:
92
+ enabled: true
93
+ feedback_endpoint: http://armiger-ultra:8095/v1/feedback
94
+ feedback_timeout: 5s
95
+ feedback_batch_size: 64
96
+ flush_interval: 15s
97
+ queue_size: 2048
98
+ ```
99
+
100
+ ### 3. Equivalent `DH_WAYPOINT_*` Environment Variables
101
+
102
+ If you configure `DH-Waypoint` via environment variables instead of `plugins.d`:
103
+
104
+ | Variable | Recommended Value | Purpose |
105
+ | :--- | :--- | :--- |
106
+ | `DH_WAYPOINT_PROCESSORS_HTTP_CHAIN` | `armiger-ultra=http://armiger-ultra:8095/v1/classify` | Ordered HTTP classification chain (`dh-waypoint.processor.v1`). |
107
+ | `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_FILES` | `true` | Forwards physical file paths and byte sizes to Armiger's File-Tree Encoder. |
108
+ | `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_TRACKERS` | `true` | Forwards tracker domain hints to `SwarmPriors`. |
109
+ | `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_SWARM` | `true` | Forwards seeder/leecher counts to `SwarmSurvivalNet`. |
110
+ | `DH_WAYPOINT_PROCESSORS_ADMISSION_ENABLED` | `true` | Enables pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`). |
111
+ | `DH_WAYPOINT_PROCESSORS_ADMISSION_ENDPOINT` | `http://armiger-ultra:8095/v1/admit` | Pre-ingest admission endpoint. |
112
+ | `DH_WAYPOINT_LEARNING_ENABLED` | `true` | Enables the 4-tier `dh-waypoint.processors.feedback.v1` outcome loop. |
113
+ | `DH_WAYPOINT_LEARNING_FEEDBACK_ENDPOINT` | `http://armiger-ultra:8095/v1/feedback` | Delivers `t1_discovery` through `t4_playback` feedback batches to Armiger. |
114
+
115
+ ---
116
+
117
+ ## Part B — Integrating Armiger with `Bitmagnet`
118
+
119
+ Upstream `Bitmagnet` uses deterministic keyword/extension rules (`35.17%` 11-class accuracy on `N=10,262` audited torrents). Deploying [`examples/bitmagnet/bitmagnet_armiger_bridge.py`](../examples/bitmagnet/bitmagnet_armiger_bridge.py) upgrades a `Bitmagnet` deployment to **`95.49%` 11-class accuracy** (`97.60%` 6-dimension composite accuracy) and **`100%` fake-seeder malware trap recall**.
120
+
121
+ ### 1. Turnkey `Bitmagnet` + `Armiger` Docker Compose Stack
122
+
123
+ Start the stack in [`examples/bitmagnet/docker-compose.yml`](../examples/bitmagnet/docker-compose.yml):
124
+
125
+ ```bash
126
+ export POSTGRES_PASSWORD="<postgres-password>"
127
+ docker compose -f examples/bitmagnet/docker-compose.yml up -d
128
+ ```
129
+
130
+ This launches two companion services alongside `bitmagnet`:
131
+ - **`armiger-bitmagnet-bridge` (`port 8096`)**: Evaluates batches of newly crawled Bitmagnet torrents (`POST /v1/bitmagnet/classify`), maps Armiger's 11-class predictions (`movie`, `tv_show`, `music`, `ebook`, `audiobook`, `comic`, `software`, `game`, `xxx`) and tags (`armiger:<category>`, `armiger:lang:<lang>`, `armiger:viability:<tier>`) onto Bitmagnet items, and automatically executes Bitmagnet's GraphQL `torrent.delete` mutation against disguised malware payloads (`.mkv.exe`, `.scr`, `.lnk`, `.vbs`) and fake-seeder honeypots.
132
+ - **`armiger-torznab-proxy` (`port 8097`)**: Wraps `http://bitmagnet:3333/torznab` with real-time neural admission filtering before `Prowlarr`, `Sonarr`, or `Radarr` sees search results.
133
+
134
+ ### 2. Calling the Bitmagnet Classifier Bridge Directly
135
+
136
+ ```bash
137
+ curl -sS -X POST http://localhost:8096/v1/bitmagnet/classify \
138
+ -H "Content-Type: application/json" \
139
+ -d '{
140
+ "torrents": [
141
+ {
142
+ "infoHash": "1111222233334444555566667777888899990000",
143
+ "name": "Sintel.2010.2160p.UHD.Master.HEVC.FLAC.5.1-OPEN",
144
+ "size": 74500000000,
145
+ "seeders": 320,
146
+ "leechers": 18,
147
+ "files": [{"path": "Sintel.2010.2160p.mkv", "size": 74500000000}]
148
+ },
149
+ {
150
+ "infoHash": "aaaa22223333444455556666777788889999bbbb",
151
+ "name": "Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe",
152
+ "size": 15400000,
153
+ "seeders": 4500,
154
+ "leechers": 12,
155
+ "files": [{"path": "Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe", "size": 15400000}]
156
+ }
157
+ ]
158
+ }'
159
+ ```
160
+
161
+ The bridge classifies the first item as `contentType: "movie"` (`viabilityTier: "alive_viable"`) and flags the second item (`Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe`) for immediate deletion via `graphqlDeleteMutation`.
162
+
163
+ ---
164
+
165
+ ## Part C — Integrating Armiger with Torznab Index Consumers & `qBittorrent`
166
+
167
+ Armiger integrates with standard Torznab XML/RSS index consumers (`Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`) and BitTorrent clients (`qBittorrent`, `Transmission`) at **two stages**:
168
+ 1. **Stage 1 — Pre-Grab Torznab Filtering Proxy (`port 8097`):** Blocks malware masquerades, fake-seeder honeypots, dead swarms, and unrequested foreign-language audio releases *before* any `*Arr` app grabs them.
169
+ 2. **Stage 2 — Post-Import Webhook & `ffprobe` Ground-Truth Receiver (`port 8098`):** Verifies completed downloads and `ffprobe` media streams to calibrate local Mondrian thresholds and optionally contribute differential-privacy blinded telemetry to the `BEP-44`/`BEP-46` RSI network.
170
+
171
+ ### 1. Stage 1: Wiring the Torznab Filtering Proxy (`Prowlarr` / `Sonarr` / `Radarr` / `Lidarr` / `Readarr`)
172
+
173
+ Start [`examples/arr-stack/docker-compose.yml`](../examples/arr-stack/docker-compose.yml):
174
+
175
+ ```bash
176
+ export UPSTREAM_TORZNAB_URL="http://dh-waypoint:3333/torznab"
177
+ docker compose -f examples/arr-stack/docker-compose.yml up -d
178
+ ```
179
+
180
+ #### Configuring in `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, or `Readarr`:
181
+ 1. Open **Settings -> Indexers -> + -> Torznab (Custom)**.
182
+ 2. Set **URL** to `http://armiger-torznab-proxy:8097` (instead of pointing directly at `http://bitmagnet:3333/torznab` or `http://dh-waypoint:3333/torznab`).
183
+ 3. Every Torznab search query (`?t=search`, `?t=tvsearch`, `?t=movie`, `?t=music`, `?t=book`) is transparently forwarded upstream, batch-scored by `ArmigerUltraPipeline`, stripped of unsafe/honeypot/dead/wrong-language items, and annotated with custom `<torznab:attr>` tags:
184
+ - `armiger_category` (`movie`, `tv_show`, `music`, `audiobook`, `ebook`, `comic`, `software`, `game`, `xxx`)
185
+ - `armiger_lang` (`english`, `non_english`, `multi_or_unknown`)
186
+ - `armiger_viability` (`alive_strong`, `alive_viable`, `marginal`, `dead`)
187
+ - `armiger_survival_prob` (`0.0000`–`1.0000`)
188
+
189
+ #### Torznab Proxy Environment Controls
190
+
191
+ | Variable | Default | Description |
192
+ | :--- | :---: | :--- |
193
+ | `UPSTREAM_TORZNAB_URL` | `http://dh-waypoint:3333/torznab` | Upstream Torznab base URL (`DH-Waypoint`, `Bitmagnet`, or `Prowlarr`). |
194
+ | `ARMIGER_VARIANT` | `int8` | `int8` (`8.3 MB` CPU fast-path, `2.44 ms` p50) or `fp32` (`32.3 MB` GPU/CPU). |
195
+ | `ARMIGER_REJECT_DEAD_SWARMS` | `true` | Drops Torznab items whose `SwarmSurvivalNet` viability tier is `dead`. |
196
+ | `ARMIGER_REQUIRE_ENGLISH_AUDIO` | `false` | When `true`, blocks releases classified as `non_english` audio by `AudioLanguageGuard`. |
197
+ | `ARMIGER_ALLOW_ENGLISH_SUBS` | `true` | When `ARMIGER_REQUIRE_ENGLISH_AUDIO=true`, permits foreign-audio anime/films that carry verified English subtitles (`has_eng_subs`). |
198
+
199
+ ### 2. Stage 2: Wiring Post-Import Webhooks (`Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `qBittorrent`)
200
+
201
+ `armiger-arr-webhook-receiver` (`port 8098`) exposes dedicated endpoints backed by the adapters in [`armiger_dht_rsi.py`](../armiger_dht_rsi.py):
202
+
203
+ | Application | Where to Configure in UI | Webhook URL | Underlying Armiger Adapter |
204
+ | :--- | :--- | :--- | :--- |
205
+ | **Sonarr** | `Settings -> Connect -> + -> Webhook` (`On Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/sonarr` | `SeriesWebhookAdapter` |
206
+ | **Radarr** | `Settings -> Connect -> + -> Webhook` (`On Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/radarr` | `MovieWebhookAdapter` |
207
+ | **Lidarr** | `Settings -> Connect -> + -> Webhook` (`On Release Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/lidarr` | `AudioWebhookAdapter` |
208
+ | **Readarr** | `Settings -> Connect -> + -> Webhook` (`On Release Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/readarr` | `PublicationWebhookAdapter` |
209
+ | **qBittorrent** | `Options -> Downloads -> Run external program` | `POST http://armiger-arr-webhook-receiver:8098/webhooks/qbittorrent` | `QBittorrentAdapter` |
210
+
211
+ #### `qBittorrent` Completion Command
212
+
213
+ In `qBittorrent` (`Tools -> Options -> Downloads -> Run external program on torrent completion`), configure:
214
+
215
+ ```bash
216
+ curl -sS -X POST http://armiger-arr-webhook-receiver:8098/webhooks/qbittorrent -H "Content-Type: application/json" -d "{\"name\":\"%N\",\"hash\":\"%I\",\"content_path\":\"%F\",\"size\":%Z,\"progress\":1.0,\"state\":\"uploading\"}"
217
+ ```
218
+
219
+ #### Programmatic Python Usage (`SeriesWebhookAdapter`, `MovieWebhookAdapter`, `QBittorrentAdapter`, `PhysicalGroundTruthVerifier`)
220
+
221
+ You can also invoke the webhook adapters and `ffprobe` verifier directly in Python via `ArmigerUltraPipeline.ingest_rsi_feedback`:
222
+
223
+ ```python
224
+ from armiger_ultra_pipeline import ArmigerUltraPipeline
225
+
226
+ pipeline = ArmigerUltraPipeline.from_directory(".", variant="int8")
227
+
228
+ # 1. Ingest a native Sonarr "Download" (Import) webhook + ffprobe output
229
+ ok, reason, record = pipeline.ingest_rsi_feedback(
230
+ {
231
+ "eventType": "Download",
232
+ "series": {"title": "Cosmos Laundromat", "tvdbId": 310123},
233
+ "episodes": [{"seasonNumber": 1, "episodeNumber": 1}],
234
+ "release": {
235
+ "releaseTitle": "Cosmos.Laundromat.S01E01.1080p.WEB-DL.DDP5.1.H.264-OPEN",
236
+ "size": 1_450_000_000,
237
+ },
238
+ "episodeFile": {
239
+ "relativePath": "Season 01/Cosmos.Laundromat.S01E01.mkv",
240
+ "size": 1_450_000_000,
241
+ "mediaInfo": {
242
+ "videoCodec": "h264",
243
+ "audioFormat": "eac3",
244
+ "audioChannels": 5.1,
245
+ "audioLanguages": "eng",
246
+ "subtitles": "eng",
247
+ },
248
+ },
249
+ },
250
+ adapter="series_webhook",
251
+ )
252
+ assert ok and reason == "accepted_and_queued_with_poisson_jitter"
253
+ print(record.blinded_stem.blinded_hash, record.category, record.lang)
254
+
255
+ # 2. Ingest a native Radarr "Download" (Import) webhook
256
+ ok, reason, record = pipeline.ingest_rsi_feedback(
257
+ {
258
+ "eventType": "Download",
259
+ "movie": {"title": "Sintel", "year": 2010, "tmdbId": 45745},
260
+ "release": {
261
+ "releaseTitle": "Sintel.2010.2160p.UHD.Master.HEVC.FLAC.5.1-OPEN",
262
+ "size": 24_800_000_000,
263
+ },
264
+ "movieFile": {
265
+ "relativePath": "Sintel (2010)/Sintel.2010.2160p.mkv",
266
+ "size": 24_800_000_000,
267
+ },
268
+ },
269
+ adapter="movie_webhook",
270
+ )
271
+ assert ok and record.category == "movie"
272
+ ```
273
+
274
+ ### 3. `BEP-44` / `BEP-46` Privacy & RSI Participation Modes
275
+
276
+ All adapters enforce `PhysicalGroundTruthVerifier` (never trusting unverified model predictions) and `TelemetryAnonymizer` (stripping tracker passkeys, local paths, and IP addresses; blinding title stems via post-quantum `HMAC-SHA256` + `SHAKE-256`; applying Laplace differential privacy $\epsilon = 1.0$; and scheduling outbound packets with Poisson timing jitter):
277
+
278
+ | `ARMIGER_DHT_RSI_MODE` | Behavior |
279
+ | :--- | :--- |
280
+ | `off` | Disables all `BEP-44`/`BEP-46` DHT traffic; calibration and ground-truth verification stay strictly local. |
281
+ | `pull_only` *(Default)* | Verifies and applies `M`-of-`N` hybrid post-quantum (`Ed25519` + `WOTS+-SHAKE256`) signed calibration manifests from the DHT after passing the 500-item golden canary gate; never transmits telemetry. |
282
+ | `anonymous_contribute` | Opts in to transmitting differential-privacy blinded, hybrid post-quantum (`X25519` + `ML-KEM-768` + `ChaCha20-Poly1305`) encrypted ground-truth observations. |