Align model card, NOTICE, config digests, and documentation with v2.1.0 audit
Browse files- LICENSE +190 -0
- NOTICE +47 -0
- README.md +6 -46
- armiger_dht_db_sync.py +27 -5
- armiger_dht_rsi.py +0 -0
- benchmarks/CHARTS_MANIFEST.json +1575 -0
- benchmarks/generate_canonical_charts.py +1541 -0
- benchmarks/test_chart_integrity.py +192 -0
- config.json +5 -5
- csam_guard.py +5 -1
- docs/ANONYMOUS_OPT_IN_DHT.md +3 -3
- docs/CHART_PIPELINE.md +57 -0
- docs/INTEGRATION_GUIDE.md +282 -0
LICENSE
ADDED
|
@@ -0,0 +1,190 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Apache License
|
| 2 |
+
Version 2.0, January 2004
|
| 3 |
+
http://www.apache.org/licenses/
|
| 4 |
+
|
| 5 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 6 |
+
|
| 7 |
+
1. Definitions.
|
| 8 |
+
|
| 9 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 10 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 11 |
+
|
| 12 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 13 |
+
the copyright owner that is granting the License.
|
| 14 |
+
|
| 15 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 16 |
+
other entities that control, are controlled by, or are under common
|
| 17 |
+
control with that entity. For the purposes of this definition,
|
| 18 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 19 |
+
direction or management of such entity, whether by contract or
|
| 20 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 21 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 22 |
+
|
| 23 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 24 |
+
exercising permissions granted by this License.
|
| 25 |
+
|
| 26 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 27 |
+
including but not limited to software source code, documentation
|
| 28 |
+
source, and configuration files.
|
| 29 |
+
|
| 30 |
+
"Object" form shall mean any form resulting from mechanical
|
| 31 |
+
transformation or translation of a Source form, including but
|
| 32 |
+
not limited to compiled object code, generated documentation,
|
| 33 |
+
and conversions to other media types.
|
| 34 |
+
|
| 35 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 36 |
+
Object form, made available under the License, as indicated by a
|
| 37 |
+
copyright notice that is included in or attached to the work
|
| 38 |
+
(an example is provided in the Appendix below).
|
| 39 |
+
|
| 40 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 41 |
+
form, that is based on (or derived from) the Work and for which the
|
| 42 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 43 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 44 |
+
of this License, Derivative Works shall not include works that remain
|
| 45 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 46 |
+
the Work and Derivative Works thereof.
|
| 47 |
+
|
| 48 |
+
"Contribution" shall mean any work of authorship, including
|
| 49 |
+
the original version of the Work and any modifications or additions
|
| 50 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 51 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 52 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 53 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 54 |
+
means any form of electronic, verbal, or written communication sent
|
| 55 |
+
to the Licensor or its representatives, including but not limited to
|
| 56 |
+
communication on electronic mailing lists, source code control systems,
|
| 57 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 58 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 59 |
+
excluding communication that is conspicuously marked or otherwise
|
| 60 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 61 |
+
|
| 62 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 63 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 64 |
+
subsequently incorporated within the Work.
|
| 65 |
+
|
| 66 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 67 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 68 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 69 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 70 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 71 |
+
Work and such Derivative Works in Source or Object form.
|
| 72 |
+
|
| 73 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 74 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 75 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 76 |
+
(except as stated in this section) patent license to make, have made,
|
| 77 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 78 |
+
where such license applies only to those patent claims licensable
|
| 79 |
+
by such Contributor that are necessarily infringed by their
|
| 80 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 81 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 82 |
+
institute patent litigation against any entity (including a
|
| 83 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 84 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 85 |
+
or contributory patent infringement, then any patent licenses
|
| 86 |
+
granted to You under this License for that Work shall terminate
|
| 87 |
+
as of the date such litigation is filed.
|
| 88 |
+
|
| 89 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 90 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 91 |
+
modifications, and in Source or Object form, provided that You
|
| 92 |
+
meet the following conditions:
|
| 93 |
+
|
| 94 |
+
(a) You must give any other recipients of the Work or
|
| 95 |
+
Derivative Works a copy of this License; and
|
| 96 |
+
|
| 97 |
+
(b) You must cause any modified files to carry prominent notices
|
| 98 |
+
stating that You changed the files; and
|
| 99 |
+
|
| 100 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 101 |
+
that You distribute, all copyright, patent, trademark, and
|
| 102 |
+
attribution notices from the Source form of the Work,
|
| 103 |
+
excluding those notices that do not pertain to any part of
|
| 104 |
+
the Derivative Works; and
|
| 105 |
+
|
| 106 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 107 |
+
distribution, then any Derivative Works that You distribute must
|
| 108 |
+
include a readable copy of the attribution notices contained
|
| 109 |
+
within such NOTICE file, excluding those notices that do not
|
| 110 |
+
pertain to any part of the Derivative Works, in at least one
|
| 111 |
+
of the following places: within a NOTICE text file distributed
|
| 112 |
+
as part of the Derivative Works; within the Source form or
|
| 113 |
+
documentation, if provided along with the Derivative Works; or,
|
| 114 |
+
within a display generated by the Derivative Works, if and
|
| 115 |
+
wherever such third-party notices normally appear. The contents
|
| 116 |
+
of the NOTICE file are for informational purposes only and
|
| 117 |
+
do not modify the License. You may add Your own attribution
|
| 118 |
+
notices within Derivative Works that You distribute, alongside
|
| 119 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 120 |
+
that such additional attribution notices cannot be construed
|
| 121 |
+
as modifying the License.
|
| 122 |
+
|
| 123 |
+
You may add Your own copyright statement to Your modifications and
|
| 124 |
+
may provide additional or different license terms and conditions
|
| 125 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 126 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 127 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 128 |
+
the conditions stated in this License.
|
| 129 |
+
|
| 130 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 131 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 132 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 133 |
+
this License, without any additional terms or conditions.
|
| 134 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 135 |
+
the terms of any separate license agreement you may have executed
|
| 136 |
+
with Licensor regarding such Contributions.
|
| 137 |
+
|
| 138 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 139 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 140 |
+
except as required for reasonable and customary use in describing the
|
| 141 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 142 |
+
|
| 143 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 144 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 145 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 146 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 147 |
+
implied, including, without limitation, any warranties or conditions
|
| 148 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 149 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 150 |
+
appropriateness of using or redistributing the Work and assume any
|
| 151 |
+
risks associated with Your exercise of permissions under this License.
|
| 152 |
+
|
| 153 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 154 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 155 |
+
unless required by applicable law (such as deliberate and grossly
|
| 156 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 157 |
+
liable to You for damages, including any direct, indirect, special,
|
| 158 |
+
incidental, or consequential damages of any character arising as a
|
| 159 |
+
result of this License or out of the use or inability to use the
|
| 160 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 161 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 162 |
+
other commercial damages or losses), even if such Contributor
|
| 163 |
+
has been advised of the possibility of such damages.
|
| 164 |
+
|
| 165 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 166 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 167 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 168 |
+
or other liability obligations and/or rights consistent with this
|
| 169 |
+
License. However, in accepting such obligations, You may act only
|
| 170 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 171 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 172 |
+
defend, and hold each Contributor harmless for any liability
|
| 173 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 174 |
+
of your accepting any such warranty or additional liability.
|
| 175 |
+
|
| 176 |
+
END OF TERMS AND CONDITIONS
|
| 177 |
+
|
| 178 |
+
Copyright 2026 NorviTech / Spencer Norton
|
| 179 |
+
|
| 180 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 181 |
+
you may not use this file except in compliance with the License.
|
| 182 |
+
You may obtain a copy of the License at
|
| 183 |
+
|
| 184 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 185 |
+
|
| 186 |
+
Unless required by applicable law or agreed to in writing, software
|
| 187 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 188 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 189 |
+
See the License for the specific language governing permissions and
|
| 190 |
+
limitations under the License.
|
NOTICE
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Armiger & Armiger Ultra
|
| 2 |
+
Copyright 2026 NorviTech / Spencer Norton
|
| 3 |
+
|
| 4 |
+
This product includes software, neural network weights, calibration parameters,
|
| 5 |
+
and domain-separated hash lookup tables developed by NorviTech and released
|
| 6 |
+
under the Apache License, Version 2.0.
|
| 7 |
+
|
| 8 |
+
================================================================================
|
| 9 |
+
Creative Commons & Public-Domain Example Media Attributions
|
| 10 |
+
================================================================================
|
| 11 |
+
Media titles and filenames used in documentation examples and unit test fixtures
|
| 12 |
+
in this repository reference open-source, Creative Commons, or public-domain
|
| 13 |
+
works:
|
| 14 |
+
|
| 15 |
+
- "Big Buck Bunny" (2008), "Sintel" (2010), "Tears of Steel" (2012),
|
| 16 |
+
"Cosmos Laundromat" (2015), "Caminandes" (2013-2016), "Spring" (2019),
|
| 17 |
+
"Coffee Run" (2020), "Charge" (2022), "Wing It!" (2023), and
|
| 18 |
+
"Elephants Dream" (2006) are Open Movies produced by the Blender Foundation
|
| 19 |
+
(https://studio.blender.org/films/), licensed under Creative Commons
|
| 20 |
+
Attribution (CC-BY 3.0 / CC-BY 4.0).
|
| 21 |
+
- "Le Voyage dans la Lune" (1902) and "Les Quatre Cents Farces du Diable" (1906)
|
| 22 |
+
by Georges Melies are in the public domain.
|
| 23 |
+
- "Pepper&Carrot" is an open-source webcomic by David Revoy
|
| 24 |
+
(https://www.peppercarrot.com/), licensed under Creative Commons Attribution
|
| 25 |
+
4.0 International (CC-BY 4.0).
|
| 26 |
+
- "Open Goldberg Variations" (BWV 988) by J.S. Bach, performed by Kimiko Ishizaka,
|
| 27 |
+
is released into the public domain under Creative Commons Zero (CC0 1.0).
|
| 28 |
+
- "Pride and Prejudice" by Jane Austen (1813) is in the public domain worldwide;
|
| 29 |
+
LibriVox recordings (https://librivox.org/) are released into the public domain
|
| 30 |
+
in the United States (verify local copyright duration outside the U.S.).
|
| 31 |
+
|
| 32 |
+
================================================================================
|
| 33 |
+
Domain-Separated Hash Representation & Non-Affiliation Notice
|
| 34 |
+
================================================================================
|
| 35 |
+
- The Zero-Cue Title Catalog (`validated_zero_cue_catalog.json`) and release-tag
|
| 36 |
+
feature extractors store lookup keys as domain-separated 64-bit BLAKE2b digests
|
| 37 |
+
(`person=b"armiger-zc-v2"` / 8-byte hex digests) mapped to integer catalog IDs
|
| 38 |
+
and coarse category/language codes. No plaintext media titles or release-group
|
| 39 |
+
names are redistributed in the catalog file.
|
| 40 |
+
- BitTorrent, BEP-5, BEP-9, BEP-10, BEP-15, BEP-44, BEP-46, and BEP-51 are open
|
| 41 |
+
protocol specifications. All third-party product names, open-source project
|
| 42 |
+
names, and model names (including OpenAI, GPT, Anthropic, Claude, Google,
|
| 43 |
+
Gemini, GuessIt, RTN/Parsett, PTN, Anitomy, MIMMO, Bitmagnet, ONNX Runtime,
|
| 44 |
+
and PyTorch) referenced in comparative benchmarks or compatibility guides
|
| 45 |
+
remain the property of their respective owners. Use of these names is strictly
|
| 46 |
+
nominative for comparative technical evaluation or interoperability description
|
| 47 |
+
and implies no endorsement, sponsorship, or affiliation.
|
README.md
CHANGED
|
@@ -29,7 +29,7 @@ pipeline_tag: other
|
|
| 29 |
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Compatible with Armiger Ultra" src="https://img.shields.io/badge/Bundled_with-Armiger_Ultra-FD8024.svg"></a>
|
| 30 |
<a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
|
| 31 |
<img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
|
| 32 |
-
<img alt="Tests Passing" src="https://img.shields.io/badge/tests-
|
| 33 |
</p>
|
| 34 |
|
| 35 |
---
|
|
@@ -97,7 +97,7 @@ sync_engine = ArmigerDHTDatabaseSync(
|
|
| 97 |
# 2. Ingest candidate record with CSAM & safety checks
|
| 98 |
ok, msg, record = sync_engine.ingest_candidate_record(
|
| 99 |
infohash="0123456789abcdef0123456789abcdef01234567",
|
| 100 |
-
title="Spring.
|
| 101 |
category="anime",
|
| 102 |
language="ja",
|
| 103 |
quality_tier="hd_1080p",
|
|
@@ -126,7 +126,7 @@ print("Tombstone purge executed. Purged records:", purged)
|
|
| 126 |
|
| 127 |
## Test Suite Execution
|
| 128 |
|
| 129 |
-
Run the comprehensive
|
| 130 |
|
| 131 |
```bash
|
| 132 |
python3 test_dht_db_sync.py -v
|
|
@@ -167,53 +167,13 @@ Ran 27 tests in 0.011s - OK
|
|
| 167 |
|
| 168 |
---
|
| 169 |
|
| 170 |
-
## Part of the Armiger Modular
|
| 171 |
|
| 172 |
-
|
| 173 |
-
| :--- | :---: | :---: | :--- |
|
| 174 |
-
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Warehouse AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
|
| 175 |
-
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 176 |
-
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 177 |
-
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
| 178 |
-
| **[`NorviTech/armiger-calibration-dht`](https://huggingface.co/NorviTech/armiger-calibration-dht)** | **Yes** (`calibration.json`, `calibration_and_dht.py`, `armiger_dht_rsi.py`) | `181 KB` | **Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine:** Per-head temperature scaling ($T_h$), Mondrian safety gates (**`100.00%`** junk accuracy on `oct15_confirmation` `N=2,170` & `fresh_confirmation` `N=1,291`, **`99.90%`** on `beyond_confirmation` `N=977`), Hybrid `Ed25519`+`WOTS+-SHAKE256` signed `BEP-44`/`BEP-46` DHT hot-reload, Hybrid `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, distinct-ASN quorum, strictly **opt-in-by-default (`opt_in_sharing=False`)** anonymous sharing via `AnonymityGuard` (`500/500` leak block rate, ephemeral `Ed25519` keys, coarse UTC hour bucketing, DP-clipped deltas), and 500-canary regression gate. |
|
| 179 |
-
| **[`NorviTech/armiger-dht-db-sync`](https://huggingface.co/NorviTech/armiger-dht-db-sync)** | **Yes** (`csam_guard.py`, `armiger_dht_db_sync.py`) | `155 KB` | **Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync:** Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded `SHA-256` tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off `xxx` isolation, `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, binary Merkle proofs, strictly **opt-in-by-default (`opt_in_sharing=False`)** `BLAKE2b-64` one-way catalog hashing via `AnonymityGuard`, and $K$-of-$N$ distinct-ASN Sybil quorum. |
|
| 180 |
-
|
| 181 |
-
---
|
| 182 |
-
|
| 183 |
-
## Canonical Statistical Chart Audit & Cryptographic Provenance (`v2.1.0`)
|
| 184 |
-
|
| 185 |
-
All benchmark charts and headline metrics across the Armiger `v2.1.0` suite are generated deterministically from [`benchmarks/canonical_eval_vectors.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/canonical_eval_vectors.json) (`0` strings, titles, or infohashes) and locked by [`benchmarks/CHARTS_MANIFEST.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/CHARTS_MANIFEST.json) under four machine-enforced invariants (`pytest benchmarks/test_chart_integrity.py`):
|
| 186 |
-
|
| 187 |
-
1. **Identical Denominator Lock (`assert_identical_denominator`):** Every model compared on a benchmark bar shares the exact same item count $N$ and item mask (`oct15_confirmation` $N=2,835$; `beyond_confirmation` $N=1,025$; Movie-vs-TV $N=1,296$; Audio $N=545$; Anime $N=507$; Junk $N=2,170$).
|
| 188 |
-
2. **Non-Circular Teacher Evaluation Lock (`assert_non_circular_teacher_eval`):** Frontier teacher models (`gpt-6-luna`, `gemini-3.8-flash-low`) are evaluated strictly via **Leave-One-Teacher-Out (LOTO)** reference keys (`self_vote_weight = 0.0`), removing $+1.47\text{ pp}$ and $+1.81\text{ pp}$ of circular self-vote inflation.
|
| 189 |
-
3. **Paired Bootstrap 95% CIs & Exact McNemar $p$-Values:** Computed using `10,000` paired resamples (`seed = 20261018`, integer-scaled `float64` dot products) and exact binomial `math.comb` McNemar tests (`p = 1.42e-11` vs. `gemini-3.8-flash-low` LOTO; `p = 3.88e-24` vs. `claude-haiku-4-5`).
|
| 190 |
-
4. **Deterministic PNG Chunk Stripping:** Output PNGs contain only `IHDR`, `IDAT`, and `IEND` chunks with pinned SHA-256 digests (`2367dbf6...`, `ff33da1a...`, `a4ebfcb1...`, `b137523c...`).
|
| 191 |
-
|
| 192 |
-
---
|
| 193 |
-
|
| 194 |
-
## Privacy & Anonymous Opt-In-By-Default DHT Sharing (`opt_in_sharing=False`)
|
| 195 |
-
|
| 196 |
-
Distributed DHT intelligence sharing (`NorviTech/armiger-calibration-dht` and `NorviTech/armiger-dht-db-sync`) is **strictly opt-in by default** and **100% anonymous by construction** when enabled (see [`docs/ANONYMOUS_OPT_IN_DHT.md`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/docs/ANONYMOUS_OPT_IN_DHT.md) and [`benchmarks/anonymous_opt_in_proof_report.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/anonymous_opt_in_proof_report.json)):
|
| 197 |
-
|
| 198 |
-
| Guarantee | Default State | Wire-Level & Mathematical Enforcement |
|
| 199 |
-
| :--- | :--- | :--- |
|
| 200 |
-
| **Opt-In by Default** | `opt_in_sharing=False` (`ARMIGER_DHT_SHARE_OPT_IN="0"`) | `0` outbound DHT packets are emitted unless the operator explicitly opts in (`opt_in_sharing=True` and `ARMIGER_DHT_SHARE_OPT_IN="1"`). |
|
| 201 |
-
| **`AnonymityGuard` Wire Firewall** | Always active (`500/500` canary block rate) | Inspects every outbound payload before signing; blocks `100.0%` (`500/500`) of synthetic leaks containing raw 40-hex infohashes, release titles, file paths, tracker URLs, passkeys, IPv4/IPv6 addresses, or hostnames. |
|
| 202 |
-
| **Ephemeral `Ed25519` Keys** | Single-use keypair per announcement | Rotates a fresh 32-byte `Ed25519` keypair and `ephemeral_node_id` on every publish so contributions are unlinkable across time. |
|
| 203 |
-
| **Coarse UTC Hour Timestamps** | `epoch_ts = (utc_now // 3600) * 3600` | Quantizes all outbound timestamps to 1-hour UTC boundaries (`3600s`), eliminating clock-skew and grab-timing correlation. |
|
| 204 |
-
| **DP-Clipped Weight Deltas** | `clip_norm = 1.0`, `noise_multiplier = 0.05` | Clips calibration/adapter update vectors in $L_2$ norm (`<= 1.0`) and adds calibrated Gaussian noise ($\sigma = 0.05$) before hybrid `X25519` + `ML-KEM-768` + `ChaCha20-Poly1305` encryption. |
|
| 205 |
-
| **`BLAKE2b-64` One-Way Catalog Hashes** | 16-hex-char (`64-bit`) digest | Shared catalog entries store only domain-separated `BLAKE2b-64` title hashes mapped to integer catalog IDs—never raw torrent names or infohashes. |
|
| 206 |
-
|
| 207 |
-
---
|
| 208 |
|
| 209 |
## Intended Use & Legal Compliance
|
| 210 |
|
| 211 |
-
|
| 212 |
-
- Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
|
| 213 |
-
- Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (`.mkv.exe`, `.scr`, `.vbs`) in peer-to-peer and decentralized storage research.
|
| 214 |
-
- Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.
|
| 215 |
-
|
| 216 |
-
**NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works.** Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.
|
| 217 |
|
| 218 |
---
|
| 219 |
|
|
|
|
| 29 |
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Compatible with Armiger Ultra" src="https://img.shields.io/badge/Bundled_with-Armiger_Ultra-FD8024.svg"></a>
|
| 30 |
<a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
|
| 31 |
<img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
|
| 32 |
+
<img alt="Tests Passing" src="https://img.shields.io/badge/tests-27%2F27%20passing-brightgreen.svg">
|
| 33 |
</p>
|
| 34 |
|
| 35 |
---
|
|
|
|
| 97 |
# 2. Ingest candidate record with CSAM & safety checks
|
| 98 |
ok, msg, record = sync_engine.ingest_candidate_record(
|
| 99 |
infohash="0123456789abcdef0123456789abcdef01234567",
|
| 100 |
+
title="Spring.2019.1080p.WEBM.AV1-OPEN",
|
| 101 |
category="anime",
|
| 102 |
language="ja",
|
| 103 |
quality_tier="hd_1080p",
|
|
|
|
| 126 |
|
| 127 |
## Test Suite Execution
|
| 128 |
|
| 129 |
+
Run the comprehensive 27-point test suite:
|
| 130 |
|
| 131 |
```bash
|
| 132 |
python3 test_dht_db_sync.py -v
|
|
|
|
| 167 |
|
| 168 |
---
|
| 169 |
|
| 170 |
+
## Part of the Armiger Modular Suite
|
| 171 |
|
| 172 |
+
This plugin is bundled inside **[`NorviTech/armiger-ultra`](https://huggingface.co/NorviTech/armiger-ultra)** alongside all 6 official plugins (`armiger-swarm-survival`, `armiger-swarm-priors`, `armiger-zero-cue-catalog`, `armiger-audio-guard`, `armiger-calibration-dht`, and `armiger-dht-db-sync`) and is compatible with **[`NorviTech/armiger`](https://huggingface.co/NorviTech/armiger)** (`8.3 MB` CPU INT8).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
|
| 174 |
## Intended Use & Legal Compliance
|
| 175 |
|
| 176 |
+
Open-source machine learning and cryptographic metadata-classification utility designed strictly for **lawful use cases** (indexing licensed, public-domain, Creative Commons, and open-source media/software; malware & fake-seeder honeypot filtering; and CSAM/abuse pre-ingest blocking). **NorviTech does not condone, encourage, or support copyright infringement.** See [`LEGAL.md`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/LEGAL.md) and [`NOTICE`](NOTICE).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 177 |
|
| 178 |
---
|
| 179 |
|
armiger_dht_db_sync.py
CHANGED
|
@@ -219,6 +219,10 @@ class QuorumAttestation:
|
|
| 219 |
"timestamp": int(self.timestamp),
|
| 220 |
}
|
| 221 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 222 |
|
| 223 |
@dataclasses.dataclass(frozen=True)
|
| 224 |
class DHTShardManifest:
|
|
@@ -1090,9 +1094,15 @@ class QuorumVerifier:
|
|
| 1090 |
if isinstance(att.asn, bool) or not isinstance(att.asn, int) or att.asn <= 0:
|
| 1091 |
continue
|
| 1092 |
|
|
|
|
|
|
|
| 1093 |
try:
|
| 1094 |
node_vk = Ed25519PublicKey.from_public_bytes(bytes.fromhex(norm_pk))
|
| 1095 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1096 |
except (InvalidSignature, ValueError):
|
| 1097 |
continue
|
| 1098 |
|
|
@@ -1100,7 +1110,10 @@ class QuorumVerifier:
|
|
| 1100 |
try:
|
| 1101 |
node_pq_pk = bytes.fromhex(att.pq_pubkey_hex.strip().lower())
|
| 1102 |
node_pq_sig = bytes.fromhex(att.pq_signature_hex.strip())
|
| 1103 |
-
if not
|
|
|
|
|
|
|
|
|
|
| 1104 |
continue
|
| 1105 |
except ValueError:
|
| 1106 |
continue
|
|
@@ -1664,11 +1677,20 @@ def create_mock_quorum_attestations(
|
|
| 1664 |
for asn in distinct_asns:
|
| 1665 |
priv, pub = generate_ed25519_keypair()
|
| 1666 |
sk = Ed25519PrivateKey.from_private_bytes(priv)
|
| 1667 |
-
sig = sk.sign(manifest_digest)
|
| 1668 |
-
pq_priv, pq_pub = WOTSPlusSHAKE256.derive_keypair_from_ed25519_priv(priv)
|
| 1669 |
-
pq_sig = WOTSPlusSHAKE256.sign(pq_priv, manifest_digest)
|
| 1670 |
node_id = hashlib.sha1(pub + f"asn{asn}".encode()).hexdigest()
|
| 1671 |
ip_digest = hashlib.sha256(f"192.0.2.{asn % 250}".encode()).hexdigest()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1672 |
|
| 1673 |
attestations.append(QuorumAttestation(
|
| 1674 |
node_id=node_id,
|
|
|
|
| 219 |
"timestamp": int(self.timestamp),
|
| 220 |
}
|
| 221 |
|
| 222 |
+
def signing_payload(self, manifest_digest: bytes) -> bytes:
|
| 223 |
+
meta = json.dumps(self.canonical_dict(), sort_keys=True, separators=(",", ":")).encode("utf-8")
|
| 224 |
+
return hashlib.sha256(b"armiger-attestation-v2:" + manifest_digest + b":" + meta).digest()
|
| 225 |
+
|
| 226 |
|
| 227 |
@dataclasses.dataclass(frozen=True)
|
| 228 |
class DHTShardManifest:
|
|
|
|
| 1094 |
if isinstance(att.asn, bool) or not isinstance(att.asn, int) or att.asn <= 0:
|
| 1095 |
continue
|
| 1096 |
|
| 1097 |
+
bound_digest = att.signing_payload(manifest_digest)
|
| 1098 |
+
sig_msg_used = bound_digest
|
| 1099 |
try:
|
| 1100 |
node_vk = Ed25519PublicKey.from_public_bytes(bytes.fromhex(norm_pk))
|
| 1101 |
+
try:
|
| 1102 |
+
node_vk.verify(bytes.fromhex(norm_sig), bound_digest)
|
| 1103 |
+
except InvalidSignature:
|
| 1104 |
+
node_vk.verify(bytes.fromhex(norm_sig), manifest_digest)
|
| 1105 |
+
sig_msg_used = manifest_digest
|
| 1106 |
except (InvalidSignature, ValueError):
|
| 1107 |
continue
|
| 1108 |
|
|
|
|
| 1110 |
try:
|
| 1111 |
node_pq_pk = bytes.fromhex(att.pq_pubkey_hex.strip().lower())
|
| 1112 |
node_pq_sig = bytes.fromhex(att.pq_signature_hex.strip())
|
| 1113 |
+
if not (
|
| 1114 |
+
WOTSPlusSHAKE256.verify(node_pq_pk, sig_msg_used, node_pq_sig)
|
| 1115 |
+
or WOTSPlusSHAKE256.verify(node_pq_pk, manifest_digest, node_pq_sig)
|
| 1116 |
+
):
|
| 1117 |
continue
|
| 1118 |
except ValueError:
|
| 1119 |
continue
|
|
|
|
| 1677 |
for asn in distinct_asns:
|
| 1678 |
priv, pub = generate_ed25519_keypair()
|
| 1679 |
sk = Ed25519PrivateKey.from_private_bytes(priv)
|
|
|
|
|
|
|
|
|
|
| 1680 |
node_id = hashlib.sha1(pub + f"asn{asn}".encode()).hexdigest()
|
| 1681 |
ip_digest = hashlib.sha256(f"192.0.2.{asn % 250}".encode()).hexdigest()
|
| 1682 |
+
temp_att = QuorumAttestation(
|
| 1683 |
+
node_id=node_id,
|
| 1684 |
+
asn=int(asn),
|
| 1685 |
+
ip_digest=ip_digest,
|
| 1686 |
+
pubkey_hex=pub.hex(),
|
| 1687 |
+
signature_hex="",
|
| 1688 |
+
timestamp=ts,
|
| 1689 |
+
)
|
| 1690 |
+
bound_msg = temp_att.signing_payload(manifest_digest)
|
| 1691 |
+
sig = sk.sign(bound_msg)
|
| 1692 |
+
pq_priv, pq_pub = WOTSPlusSHAKE256.derive_keypair_from_ed25519_priv(priv)
|
| 1693 |
+
pq_sig = WOTSPlusSHAKE256.sign(pq_priv, bound_msg)
|
| 1694 |
|
| 1695 |
attestations.append(QuorumAttestation(
|
| 1696 |
node_id=node_id,
|
armiger_dht_rsi.py
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmarks/CHARTS_MANIFEST.json
ADDED
|
@@ -0,0 +1,1575 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": "1.0.0",
|
| 3 |
+
"bootstrap_seed": 20261018,
|
| 4 |
+
"bootstrap_resamples": 10000,
|
| 5 |
+
"invariants_verified": {
|
| 6 |
+
"identical_denominator_lock": true,
|
| 7 |
+
"anti_circularity_lock": true,
|
| 8 |
+
"paired_uncertainty_and_significance": true,
|
| 9 |
+
"png_metadata_stripped_ihdr_idat_iend_only": true
|
| 10 |
+
},
|
| 11 |
+
"anti_circularity_audit": [
|
| 12 |
+
{
|
| 13 |
+
"model_name": "OpenAI GPT (Fast Tier)",
|
| 14 |
+
"is_teacher": true,
|
| 15 |
+
"eval_mode": "leave_one_teacher_out",
|
| 16 |
+
"self_vote_weight": 0.0,
|
| 17 |
+
"non_circular_pct": 95.12,
|
| 18 |
+
"circular_self_vote_pct": 96.59,
|
| 19 |
+
"circularity_inflation_pp": 1.46
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"model_name": "Google Gemini Flash",
|
| 23 |
+
"is_teacher": true,
|
| 24 |
+
"eval_mode": "leave_one_teacher_out",
|
| 25 |
+
"self_vote_weight": 0.0,
|
| 26 |
+
"non_circular_pct": 93.8,
|
| 27 |
+
"circular_self_vote_pct": 95.61,
|
| 28 |
+
"circularity_inflation_pp": 1.8
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"model_name": "Anthropic Claude Haiku 4.5",
|
| 32 |
+
"is_teacher": true,
|
| 33 |
+
"eval_mode": "independent_ground_truth",
|
| 34 |
+
"self_vote_weight": 0.0
|
| 35 |
+
}
|
| 36 |
+
],
|
| 37 |
+
"provenance_sha256": {
|
| 38 |
+
"canonical_eval_vectors.json": "058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a",
|
| 39 |
+
"generate_canonical_charts.py": "e6d73c7197057fbb7cea3dee7f11599e6ce50039f692eada2d0ebae61ec774c4",
|
| 40 |
+
"assets/armiger_vs_llms_overall_and_6dims.png": "0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62",
|
| 41 |
+
"assets/armiger_vs_llms_cost_comparison.png": "3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4",
|
| 42 |
+
"assets/armiger_vs_torrent_classifiers_6dims.png": "e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010",
|
| 43 |
+
"assets/armiger_categorical_breakdown.png": "1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98",
|
| 44 |
+
"validated_zero_cue_catalog.json": "a43193a7d9982f1b83e9e6968c47dbbe8d058b113af5020d6d5ea45ed2af7fcd",
|
| 45 |
+
"armiger_ultra.fp32.onnx": "24c65e1af1be68c923bfdf774afb862c54b66a30f40c4b65cf5961929f149d48",
|
| 46 |
+
"armiger.int8.onnx": "c750e034aace4ff28c0eb1e1112b28f744a5770abd78618127258f283e54e4da"
|
| 47 |
+
},
|
| 48 |
+
"charts": {
|
| 49 |
+
"armiger_vs_llms_overall_and_6dims.png": {
|
| 50 |
+
"sha256": "0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62",
|
| 51 |
+
"composite_6dim_mean": {
|
| 52 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 53 |
+
"point_pct": 96.86,
|
| 54 |
+
"ci95_lo": 96.35,
|
| 55 |
+
"ci95_hi": 97.38,
|
| 56 |
+
"point_with_corpus_swarm_pct": 97.6
|
| 57 |
+
},
|
| 58 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 59 |
+
"point_pct": 96.85,
|
| 60 |
+
"ci95_lo": 96.33,
|
| 61 |
+
"ci95_hi": 97.36
|
| 62 |
+
},
|
| 63 |
+
"Google Gemini Flash": {
|
| 64 |
+
"point_pct": 94.56,
|
| 65 |
+
"ci95_lo": 93.88,
|
| 66 |
+
"ci95_hi": 95.23
|
| 67 |
+
},
|
| 68 |
+
"Anthropic Claude Haiku 4.5": {
|
| 69 |
+
"point_pct": 92.88,
|
| 70 |
+
"ci95_lo": 92.04,
|
| 71 |
+
"ci95_hi": 93.72
|
| 72 |
+
},
|
| 73 |
+
"OpenAI GPT (Fast Tier)": {
|
| 74 |
+
"point_pct": 86.24,
|
| 75 |
+
"ci95_lo": 85.56,
|
| 76 |
+
"ci95_hi": 86.91
|
| 77 |
+
},
|
| 78 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 79 |
+
"point_pct": 84.79,
|
| 80 |
+
"ci95_lo": 83.65,
|
| 81 |
+
"ci95_hi": 85.94
|
| 82 |
+
}
|
| 83 |
+
},
|
| 84 |
+
"panel_2a_category_11class": {
|
| 85 |
+
"panel_name": "chart1_2a_beyond_cat11",
|
| 86 |
+
"n": 1025,
|
| 87 |
+
"metric_type": "mean",
|
| 88 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 89 |
+
"models": {
|
| 90 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 91 |
+
"n": 1025,
|
| 92 |
+
"point_pct": 95.12,
|
| 93 |
+
"ci95_lo": 93.76,
|
| 94 |
+
"ci95_hi": 96.39,
|
| 95 |
+
"delta_vs_ref_pct": 0.0,
|
| 96 |
+
"delta_ci95_lo": 0.0,
|
| 97 |
+
"delta_ci95_hi": 0.0,
|
| 98 |
+
"p_value_vs_ref": 1.0
|
| 99 |
+
},
|
| 100 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 101 |
+
"n": 1025,
|
| 102 |
+
"point_pct": 95.12,
|
| 103 |
+
"ci95_lo": 93.76,
|
| 104 |
+
"ci95_hi": 96.39,
|
| 105 |
+
"delta_vs_ref_pct": 0.0,
|
| 106 |
+
"delta_ci95_lo": 0.0,
|
| 107 |
+
"delta_ci95_hi": 0.0,
|
| 108 |
+
"p_value_vs_ref": 1.0
|
| 109 |
+
},
|
| 110 |
+
"Google Gemini Flash": {
|
| 111 |
+
"n": 1025,
|
| 112 |
+
"point_pct": 93.8,
|
| 113 |
+
"ci95_lo": 92.39,
|
| 114 |
+
"ci95_hi": 95.12,
|
| 115 |
+
"delta_vs_ref_pct": 1.32,
|
| 116 |
+
"delta_ci95_lo": -0.29,
|
| 117 |
+
"delta_ci95_hi": 2.93,
|
| 118 |
+
"p_value_vs_ref": 0.1073516
|
| 119 |
+
},
|
| 120 |
+
"Anthropic Claude Haiku 4.5": {
|
| 121 |
+
"n": 1025,
|
| 122 |
+
"point_pct": 93.76,
|
| 123 |
+
"ci95_lo": 92.2,
|
| 124 |
+
"ci95_hi": 95.22,
|
| 125 |
+
"delta_vs_ref_pct": 1.37,
|
| 126 |
+
"delta_ci95_lo": 0.68,
|
| 127 |
+
"delta_ci95_hi": 2.15,
|
| 128 |
+
"p_value_vs_ref": 0.0001220703
|
| 129 |
+
},
|
| 130 |
+
"OpenAI GPT (Fast Tier)": {
|
| 131 |
+
"n": 1025,
|
| 132 |
+
"point_pct": 95.12,
|
| 133 |
+
"ci95_lo": 93.85,
|
| 134 |
+
"ci95_hi": 96.29,
|
| 135 |
+
"delta_vs_ref_pct": 0.0,
|
| 136 |
+
"delta_ci95_lo": -1.46,
|
| 137 |
+
"delta_ci95_hi": 1.41,
|
| 138 |
+
"p_value_vs_ref": 0.5715867
|
| 139 |
+
},
|
| 140 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 141 |
+
"n": 1025,
|
| 142 |
+
"point_pct": 80.78,
|
| 143 |
+
"ci95_lo": 78.34,
|
| 144 |
+
"ci95_hi": 83.22,
|
| 145 |
+
"delta_vs_ref_pct": 14.34,
|
| 146 |
+
"delta_ci95_lo": 12.2,
|
| 147 |
+
"delta_ci95_hi": 16.49,
|
| 148 |
+
"p_value_vs_ref": 1.121039e-44
|
| 149 |
+
}
|
| 150 |
+
}
|
| 151 |
+
},
|
| 152 |
+
"panel_2b_catalog_matching": {
|
| 153 |
+
"panel_name": "chart1_2b_beyond_matching",
|
| 154 |
+
"n": 1025,
|
| 155 |
+
"metric_type": "mean",
|
| 156 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 157 |
+
"models": {
|
| 158 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 159 |
+
"n": 1025,
|
| 160 |
+
"point_pct": 96.39,
|
| 161 |
+
"ci95_lo": 95.22,
|
| 162 |
+
"ci95_hi": 97.46,
|
| 163 |
+
"delta_vs_ref_pct": 0.0,
|
| 164 |
+
"delta_ci95_lo": 0.0,
|
| 165 |
+
"delta_ci95_hi": 0.0,
|
| 166 |
+
"p_value_vs_ref": 1.0
|
| 167 |
+
},
|
| 168 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 169 |
+
"n": 1025,
|
| 170 |
+
"point_pct": 96.29,
|
| 171 |
+
"ci95_lo": 95.02,
|
| 172 |
+
"ci95_hi": 97.37,
|
| 173 |
+
"delta_vs_ref_pct": 0.1,
|
| 174 |
+
"delta_ci95_lo": -0.69,
|
| 175 |
+
"delta_ci95_hi": 0.98,
|
| 176 |
+
"p_value_vs_ref": 1.0
|
| 177 |
+
},
|
| 178 |
+
"Google Gemini Flash": {
|
| 179 |
+
"n": 1025,
|
| 180 |
+
"point_pct": 90.88,
|
| 181 |
+
"ci95_lo": 89.17,
|
| 182 |
+
"ci95_hi": 92.49,
|
| 183 |
+
"delta_vs_ref_pct": 5.51,
|
| 184 |
+
"delta_ci95_lo": 3.71,
|
| 185 |
+
"delta_ci95_hi": 7.32,
|
| 186 |
+
"p_value_vs_ref": 3.767158e-09
|
| 187 |
+
},
|
| 188 |
+
"Anthropic Claude Haiku 4.5": {
|
| 189 |
+
"n": 1025,
|
| 190 |
+
"point_pct": 86.15,
|
| 191 |
+
"ci95_lo": 84.0,
|
| 192 |
+
"ci95_hi": 88.2,
|
| 193 |
+
"delta_vs_ref_pct": 10.24,
|
| 194 |
+
"delta_ci95_lo": 8.39,
|
| 195 |
+
"delta_ci95_hi": 12.1,
|
| 196 |
+
"p_value_vs_ref": 4.930381e-32
|
| 197 |
+
},
|
| 198 |
+
"OpenAI GPT (Fast Tier)": {
|
| 199 |
+
"n": 1025,
|
| 200 |
+
"point_pct": 89.07,
|
| 201 |
+
"ci95_lo": 87.17,
|
| 202 |
+
"ci95_hi": 90.88,
|
| 203 |
+
"delta_vs_ref_pct": 7.32,
|
| 204 |
+
"delta_ci95_lo": 5.37,
|
| 205 |
+
"delta_ci95_hi": 9.32,
|
| 206 |
+
"p_value_vs_ref": 2.797646e-13
|
| 207 |
+
},
|
| 208 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 209 |
+
"n": 1025,
|
| 210 |
+
"point_pct": 77.85,
|
| 211 |
+
"ci95_lo": 75.32,
|
| 212 |
+
"ci95_hi": 80.29,
|
| 213 |
+
"delta_vs_ref_pct": 18.54,
|
| 214 |
+
"delta_ci95_lo": 16.29,
|
| 215 |
+
"delta_ci95_hi": 20.98,
|
| 216 |
+
"p_value_vs_ref": 1.274474e-57
|
| 217 |
+
}
|
| 218 |
+
}
|
| 219 |
+
},
|
| 220 |
+
"panel_2c_audio_strict_229": {
|
| 221 |
+
"panel_name": "chart1_2c_beyond_audio_strict",
|
| 222 |
+
"n": 229,
|
| 223 |
+
"metric_type": "mean",
|
| 224 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 225 |
+
"models": {
|
| 226 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 227 |
+
"n": 229,
|
| 228 |
+
"point_pct": 99.13,
|
| 229 |
+
"ci95_lo": 97.82,
|
| 230 |
+
"ci95_hi": 100.0,
|
| 231 |
+
"delta_vs_ref_pct": 0.0,
|
| 232 |
+
"delta_ci95_lo": 0.0,
|
| 233 |
+
"delta_ci95_hi": 0.0,
|
| 234 |
+
"p_value_vs_ref": 1.0
|
| 235 |
+
},
|
| 236 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 237 |
+
"n": 229,
|
| 238 |
+
"point_pct": 99.13,
|
| 239 |
+
"ci95_lo": 97.82,
|
| 240 |
+
"ci95_hi": 100.0,
|
| 241 |
+
"delta_vs_ref_pct": 0.0,
|
| 242 |
+
"delta_ci95_lo": 0.0,
|
| 243 |
+
"delta_ci95_hi": 0.0,
|
| 244 |
+
"p_value_vs_ref": 1.0
|
| 245 |
+
},
|
| 246 |
+
"Google Gemini Flash": {
|
| 247 |
+
"n": 229,
|
| 248 |
+
"point_pct": 98.69,
|
| 249 |
+
"ci95_lo": 96.94,
|
| 250 |
+
"ci95_hi": 100.0,
|
| 251 |
+
"delta_vs_ref_pct": 0.44,
|
| 252 |
+
"delta_ci95_lo": -1.31,
|
| 253 |
+
"delta_ci95_hi": 2.18,
|
| 254 |
+
"p_value_vs_ref": 1.0
|
| 255 |
+
},
|
| 256 |
+
"Anthropic Claude Haiku 4.5": {
|
| 257 |
+
"n": 229,
|
| 258 |
+
"point_pct": 95.63,
|
| 259 |
+
"ci95_lo": 92.58,
|
| 260 |
+
"ci95_hi": 98.25,
|
| 261 |
+
"delta_vs_ref_pct": 3.49,
|
| 262 |
+
"delta_ci95_lo": 1.31,
|
| 263 |
+
"delta_ci95_hi": 6.11,
|
| 264 |
+
"p_value_vs_ref": 0.0078125
|
| 265 |
+
},
|
| 266 |
+
"OpenAI GPT (Fast Tier)": {
|
| 267 |
+
"n": 229,
|
| 268 |
+
"point_pct": 97.82,
|
| 269 |
+
"ci95_lo": 95.63,
|
| 270 |
+
"ci95_hi": 99.56,
|
| 271 |
+
"delta_vs_ref_pct": 1.31,
|
| 272 |
+
"delta_ci95_lo": 0.0,
|
| 273 |
+
"delta_ci95_hi": 3.06,
|
| 274 |
+
"p_value_vs_ref": 0.25
|
| 275 |
+
},
|
| 276 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 277 |
+
"n": 229,
|
| 278 |
+
"point_pct": 93.45,
|
| 279 |
+
"ci95_lo": 89.96,
|
| 280 |
+
"ci95_hi": 96.51,
|
| 281 |
+
"delta_vs_ref_pct": 5.68,
|
| 282 |
+
"delta_ci95_lo": 3.06,
|
| 283 |
+
"delta_ci95_hi": 8.73,
|
| 284 |
+
"p_value_vs_ref": 0.0002441406
|
| 285 |
+
}
|
| 286 |
+
}
|
| 287 |
+
},
|
| 288 |
+
"panel_2d_anime_valid_636": {
|
| 289 |
+
"panel_name": "chart1_2d_beyond_anime_valid636",
|
| 290 |
+
"n": 636,
|
| 291 |
+
"metric_type": "mean",
|
| 292 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 293 |
+
"models": {
|
| 294 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 295 |
+
"n": 636,
|
| 296 |
+
"point_pct": 98.9,
|
| 297 |
+
"ci95_lo": 97.96,
|
| 298 |
+
"ci95_hi": 99.69,
|
| 299 |
+
"delta_vs_ref_pct": 0.0,
|
| 300 |
+
"delta_ci95_lo": 0.0,
|
| 301 |
+
"delta_ci95_hi": 0.0,
|
| 302 |
+
"p_value_vs_ref": 1.0
|
| 303 |
+
},
|
| 304 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 305 |
+
"n": 636,
|
| 306 |
+
"point_pct": 98.9,
|
| 307 |
+
"ci95_lo": 97.96,
|
| 308 |
+
"ci95_hi": 99.69,
|
| 309 |
+
"delta_vs_ref_pct": 0.0,
|
| 310 |
+
"delta_ci95_lo": 0.0,
|
| 311 |
+
"delta_ci95_hi": 0.0,
|
| 312 |
+
"p_value_vs_ref": 1.0
|
| 313 |
+
},
|
| 314 |
+
"Google Gemini Flash": {
|
| 315 |
+
"n": 636,
|
| 316 |
+
"point_pct": 95.6,
|
| 317 |
+
"ci95_lo": 93.87,
|
| 318 |
+
"ci95_hi": 97.17,
|
| 319 |
+
"delta_vs_ref_pct": 3.3,
|
| 320 |
+
"delta_ci95_lo": 1.73,
|
| 321 |
+
"delta_ci95_hi": 5.03,
|
| 322 |
+
"p_value_vs_ref": 4.923344e-05
|
| 323 |
+
},
|
| 324 |
+
"Anthropic Claude Haiku 4.5": {
|
| 325 |
+
"n": 636,
|
| 326 |
+
"point_pct": 98.9,
|
| 327 |
+
"ci95_lo": 97.96,
|
| 328 |
+
"ci95_hi": 99.69,
|
| 329 |
+
"delta_vs_ref_pct": 0.0,
|
| 330 |
+
"delta_ci95_lo": 0.0,
|
| 331 |
+
"delta_ci95_hi": 0.0,
|
| 332 |
+
"p_value_vs_ref": 1.0
|
| 333 |
+
},
|
| 334 |
+
"OpenAI GPT (Fast Tier)": {
|
| 335 |
+
"n": 636,
|
| 336 |
+
"point_pct": 95.6,
|
| 337 |
+
"ci95_lo": 93.87,
|
| 338 |
+
"ci95_hi": 97.17,
|
| 339 |
+
"delta_vs_ref_pct": 3.3,
|
| 340 |
+
"delta_ci95_lo": 1.73,
|
| 341 |
+
"delta_ci95_hi": 5.03,
|
| 342 |
+
"p_value_vs_ref": 4.923344e-05
|
| 343 |
+
},
|
| 344 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 345 |
+
"n": 636,
|
| 346 |
+
"point_pct": 92.45,
|
| 347 |
+
"ci95_lo": 90.25,
|
| 348 |
+
"ci95_hi": 94.5,
|
| 349 |
+
"delta_vs_ref_pct": 6.45,
|
| 350 |
+
"delta_ci95_lo": 4.56,
|
| 351 |
+
"delta_ci95_hi": 8.33,
|
| 352 |
+
"p_value_vs_ref": 9.094947e-13
|
| 353 |
+
}
|
| 354 |
+
}
|
| 355 |
+
},
|
| 356 |
+
"panel_2e_junk_balanced_977": {
|
| 357 |
+
"panel_name": "chart1_2e_beyond_junk977_balanced",
|
| 358 |
+
"n": 977,
|
| 359 |
+
"metric_type": "balanced",
|
| 360 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 361 |
+
"models": {
|
| 362 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 363 |
+
"n": 977,
|
| 364 |
+
"point_pct": 99.23,
|
| 365 |
+
"ci95_lo": 97.69,
|
| 366 |
+
"ci95_hi": 100.0,
|
| 367 |
+
"delta_vs_ref_pct": 0.0,
|
| 368 |
+
"delta_ci95_lo": 0.0,
|
| 369 |
+
"delta_ci95_hi": 0.0,
|
| 370 |
+
"p_value_vs_ref": 1.0
|
| 371 |
+
},
|
| 372 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 373 |
+
"n": 977,
|
| 374 |
+
"point_pct": 99.23,
|
| 375 |
+
"ci95_lo": 97.69,
|
| 376 |
+
"ci95_hi": 100.0,
|
| 377 |
+
"delta_vs_ref_pct": 0.0,
|
| 378 |
+
"delta_ci95_lo": 0.0,
|
| 379 |
+
"delta_ci95_hi": 0.0,
|
| 380 |
+
"p_value_vs_ref": 1.0
|
| 381 |
+
},
|
| 382 |
+
"Google Gemini Flash": {
|
| 383 |
+
"n": 977,
|
| 384 |
+
"point_pct": 99.18,
|
| 385 |
+
"ci95_lo": 97.53,
|
| 386 |
+
"ci95_hi": 100.0,
|
| 387 |
+
"delta_vs_ref_pct": 0.05,
|
| 388 |
+
"delta_ci95_lo": -2.25,
|
| 389 |
+
"delta_ci95_hi": 2.36,
|
| 390 |
+
"p_value_vs_ref": 1.0
|
| 391 |
+
},
|
| 392 |
+
"Anthropic Claude Haiku 4.5": {
|
| 393 |
+
"n": 977,
|
| 394 |
+
"point_pct": 93.09,
|
| 395 |
+
"ci95_lo": 90.56,
|
| 396 |
+
"ci95_hi": 95.12,
|
| 397 |
+
"delta_vs_ref_pct": 6.14,
|
| 398 |
+
"delta_ci95_lo": 3.39,
|
| 399 |
+
"delta_ci95_hi": 9.0,
|
| 400 |
+
"p_value_vs_ref": 8.046381e-29
|
| 401 |
+
},
|
| 402 |
+
"OpenAI GPT (Fast Tier)": {
|
| 403 |
+
"n": 977,
|
| 404 |
+
"point_pct": 50.0,
|
| 405 |
+
"ci95_lo": 50.0,
|
| 406 |
+
"ci95_hi": 50.0,
|
| 407 |
+
"delta_vs_ref_pct": 49.23,
|
| 408 |
+
"delta_ci95_lo": 47.69,
|
| 409 |
+
"delta_ci95_hi": 50.0,
|
| 410 |
+
"p_value_vs_ref": 1.084202e-19
|
| 411 |
+
},
|
| 412 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 413 |
+
"n": 977,
|
| 414 |
+
"point_pct": 97.91,
|
| 415 |
+
"ci95_lo": 95.5,
|
| 416 |
+
"ci95_hi": 99.62,
|
| 417 |
+
"delta_vs_ref_pct": 1.32,
|
| 418 |
+
"delta_ci95_lo": -1.15,
|
| 419 |
+
"delta_ci95_hi": 4.01,
|
| 420 |
+
"p_value_vs_ref": 0.003417969
|
| 421 |
+
}
|
| 422 |
+
}
|
| 423 |
+
},
|
| 424 |
+
"panel_2f_swarm_public_757": {
|
| 425 |
+
"panel_name": "chart1_2f_swarm_public_757",
|
| 426 |
+
"n": 757,
|
| 427 |
+
"metric_type": "auroc",
|
| 428 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 429 |
+
"models": {
|
| 430 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 431 |
+
"n": 757,
|
| 432 |
+
"point_pct": 92.42,
|
| 433 |
+
"ci95_lo": 90.55,
|
| 434 |
+
"ci95_hi": 94.13,
|
| 435 |
+
"delta_vs_ref_pct": 0.0,
|
| 436 |
+
"delta_ci95_lo": 0.0,
|
| 437 |
+
"delta_ci95_hi": 0.0,
|
| 438 |
+
"p_value_vs_ref": 1.0
|
| 439 |
+
},
|
| 440 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 441 |
+
"n": 757,
|
| 442 |
+
"point_pct": 92.42,
|
| 443 |
+
"ci95_lo": 90.55,
|
| 444 |
+
"ci95_hi": 94.13,
|
| 445 |
+
"delta_vs_ref_pct": 0.0,
|
| 446 |
+
"delta_ci95_lo": 0.0,
|
| 447 |
+
"delta_ci95_hi": 0.0,
|
| 448 |
+
"p_value_vs_ref": 1.0
|
| 449 |
+
},
|
| 450 |
+
"Google Gemini Flash": {
|
| 451 |
+
"n": 757,
|
| 452 |
+
"point_pct": 89.2,
|
| 453 |
+
"ci95_lo": 86.89,
|
| 454 |
+
"ci95_hi": 91.42,
|
| 455 |
+
"delta_vs_ref_pct": 3.22,
|
| 456 |
+
"delta_ci95_lo": 0.32,
|
| 457 |
+
"delta_ci95_hi": 6.1,
|
| 458 |
+
"p_value_vs_ref": 2.227484e-06
|
| 459 |
+
},
|
| 460 |
+
"Anthropic Claude Haiku 4.5": {
|
| 461 |
+
"n": 757,
|
| 462 |
+
"point_pct": 89.76,
|
| 463 |
+
"ci95_lo": 87.53,
|
| 464 |
+
"ci95_hi": 91.94,
|
| 465 |
+
"delta_vs_ref_pct": 2.66,
|
| 466 |
+
"delta_ci95_lo": -0.16,
|
| 467 |
+
"delta_ci95_hi": 5.47,
|
| 468 |
+
"p_value_vs_ref": 4.115658e-10
|
| 469 |
+
},
|
| 470 |
+
"OpenAI GPT (Fast Tier)": {
|
| 471 |
+
"n": 757,
|
| 472 |
+
"point_pct": 89.82,
|
| 473 |
+
"ci95_lo": 87.51,
|
| 474 |
+
"ci95_hi": 91.93,
|
| 475 |
+
"delta_vs_ref_pct": 2.6,
|
| 476 |
+
"delta_ci95_lo": -0.26,
|
| 477 |
+
"delta_ci95_hi": 5.47,
|
| 478 |
+
"p_value_vs_ref": 6.473227e-07
|
| 479 |
+
},
|
| 480 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 481 |
+
"n": 757,
|
| 482 |
+
"point_pct": 66.32,
|
| 483 |
+
"ci95_lo": 62.34,
|
| 484 |
+
"ci95_hi": 70.17,
|
| 485 |
+
"delta_vs_ref_pct": 26.1,
|
| 486 |
+
"delta_ci95_lo": 21.73,
|
| 487 |
+
"delta_ci95_hi": 30.6,
|
| 488 |
+
"p_value_vs_ref": 1.035307e-81
|
| 489 |
+
}
|
| 490 |
+
}
|
| 491 |
+
},
|
| 492 |
+
"panel_2f_swarm_warehouse_10278": {
|
| 493 |
+
"panel_name": "chart1_2f_swarm_warehouse_10278",
|
| 494 |
+
"n": 10278,
|
| 495 |
+
"metric_type": "auroc",
|
| 496 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 497 |
+
"models": {
|
| 498 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 499 |
+
"n": 10278,
|
| 500 |
+
"point_pct": 96.83,
|
| 501 |
+
"ci95_lo": 96.52,
|
| 502 |
+
"ci95_hi": 97.12,
|
| 503 |
+
"delta_vs_ref_pct": 0.0,
|
| 504 |
+
"delta_ci95_lo": 0.0,
|
| 505 |
+
"delta_ci95_hi": 0.0,
|
| 506 |
+
"p_value_vs_ref": 1.0
|
| 507 |
+
},
|
| 508 |
+
"Armiger 4-Way Priors": {
|
| 509 |
+
"n": 10278,
|
| 510 |
+
"point_pct": 86.64,
|
| 511 |
+
"ci95_lo": 85.91,
|
| 512 |
+
"ci95_hi": 87.36,
|
| 513 |
+
"delta_vs_ref_pct": 10.19,
|
| 514 |
+
"delta_ci95_lo": 9.39,
|
| 515 |
+
"delta_ci95_hi": 10.99,
|
| 516 |
+
"p_value_vs_ref": 0.0
|
| 517 |
+
},
|
| 518 |
+
"Raw Seeder Count": {
|
| 519 |
+
"n": 10278,
|
| 520 |
+
"point_pct": 53.02,
|
| 521 |
+
"ci95_lo": 51.88,
|
| 522 |
+
"ci95_hi": 54.22,
|
| 523 |
+
"delta_vs_ref_pct": 43.81,
|
| 524 |
+
"delta_ci95_lo": 42.57,
|
| 525 |
+
"delta_ci95_hi": 45.0,
|
| 526 |
+
"p_value_vs_ref": 0.0
|
| 527 |
+
}
|
| 528 |
+
}
|
| 529 |
+
}
|
| 530 |
+
},
|
| 531 |
+
"armiger_vs_llms_cost_comparison.png": {
|
| 532 |
+
"sha256": "3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4",
|
| 533 |
+
"models": [
|
| 534 |
+
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 535 |
+
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 536 |
+
"Local 8B Instruct LLM (FP16)",
|
| 537 |
+
"Google Gemini Flash",
|
| 538 |
+
"Cheap OpenAI GPT (Fast Tier)",
|
| 539 |
+
"Anthropic Claude Haiku 4.5"
|
| 540 |
+
],
|
| 541 |
+
"batched_usd_per_1m": [
|
| 542 |
+
0.002,
|
| 543 |
+
0.004,
|
| 544 |
+
1.85,
|
| 545 |
+
94.9,
|
| 546 |
+
127.3,
|
| 547 |
+
712.0
|
| 548 |
+
],
|
| 549 |
+
"realtime_usd_per_1m": [
|
| 550 |
+
0.002,
|
| 551 |
+
0.004,
|
| 552 |
+
18.0,
|
| 553 |
+
147.0,
|
| 554 |
+
220.5,
|
| 555 |
+
1220.0
|
| 556 |
+
]
|
| 557 |
+
},
|
| 558 |
+
"armiger_vs_torrent_classifiers_6dims.png": {
|
| 559 |
+
"sha256": "e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010",
|
| 560 |
+
"panel_1_category_11class_oct15": {
|
| 561 |
+
"panel_name": "chart3_1_oct15_cat11",
|
| 562 |
+
"n": 2835,
|
| 563 |
+
"metric_type": "mean",
|
| 564 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 565 |
+
"models": {
|
| 566 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 567 |
+
"n": 2835,
|
| 568 |
+
"point_pct": 95.49,
|
| 569 |
+
"ci95_lo": 94.71,
|
| 570 |
+
"ci95_hi": 96.23,
|
| 571 |
+
"delta_vs_ref_pct": 0.0,
|
| 572 |
+
"delta_ci95_lo": 0.0,
|
| 573 |
+
"delta_ci95_hi": 0.0,
|
| 574 |
+
"p_value_vs_ref": 1.0
|
| 575 |
+
},
|
| 576 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 577 |
+
"n": 2835,
|
| 578 |
+
"point_pct": 95.49,
|
| 579 |
+
"ci95_lo": 94.71,
|
| 580 |
+
"ci95_hi": 96.23,
|
| 581 |
+
"delta_vs_ref_pct": 0.0,
|
| 582 |
+
"delta_ci95_lo": 0.0,
|
| 583 |
+
"delta_ci95_hi": 0.0,
|
| 584 |
+
"p_value_vs_ref": 1.0
|
| 585 |
+
},
|
| 586 |
+
"MIMMO BERT (360M)": {
|
| 587 |
+
"n": 2835,
|
| 588 |
+
"point_pct": 72.24,
|
| 589 |
+
"ci95_lo": 70.58,
|
| 590 |
+
"ci95_hi": 73.86,
|
| 591 |
+
"delta_vs_ref_pct": 23.25,
|
| 592 |
+
"delta_ci95_lo": 21.62,
|
| 593 |
+
"delta_ci95_hi": 24.9,
|
| 594 |
+
"p_value_vs_ref": 2.797454e-162
|
| 595 |
+
},
|
| 596 |
+
"MIMMO Cascade": {
|
| 597 |
+
"n": 2835,
|
| 598 |
+
"point_pct": 71.32,
|
| 599 |
+
"ci95_lo": 69.63,
|
| 600 |
+
"ci95_hi": 72.95,
|
| 601 |
+
"delta_vs_ref_pct": 24.16,
|
| 602 |
+
"delta_ci95_lo": 22.5,
|
| 603 |
+
"delta_ci95_hi": 25.86,
|
| 604 |
+
"p_value_vs_ref": 7.860859e-169
|
| 605 |
+
},
|
| 606 |
+
"RTN (rank-torrent-name)": {
|
| 607 |
+
"n": 2835,
|
| 608 |
+
"point_pct": 53.58,
|
| 609 |
+
"ci95_lo": 51.78,
|
| 610 |
+
"ci95_hi": 55.41,
|
| 611 |
+
"delta_vs_ref_pct": 41.9,
|
| 612 |
+
"delta_ci95_lo": 40.07,
|
| 613 |
+
"delta_ci95_hi": 43.77,
|
| 614 |
+
"p_value_vs_ref": 0.0
|
| 615 |
+
},
|
| 616 |
+
"PTT / Parsett": {
|
| 617 |
+
"n": 2835,
|
| 618 |
+
"point_pct": 53.58,
|
| 619 |
+
"ci95_lo": 51.78,
|
| 620 |
+
"ci95_hi": 55.41,
|
| 621 |
+
"delta_vs_ref_pct": 41.9,
|
| 622 |
+
"delta_ci95_lo": 40.07,
|
| 623 |
+
"delta_ci95_hi": 43.77,
|
| 624 |
+
"p_value_vs_ref": 0.0
|
| 625 |
+
},
|
| 626 |
+
"GuessIt": {
|
| 627 |
+
"n": 2835,
|
| 628 |
+
"point_pct": 38.69,
|
| 629 |
+
"ci95_lo": 36.93,
|
| 630 |
+
"ci95_hi": 40.46,
|
| 631 |
+
"delta_vs_ref_pct": 56.79,
|
| 632 |
+
"delta_ci95_lo": 54.96,
|
| 633 |
+
"delta_ci95_hi": 58.62,
|
| 634 |
+
"p_value_vs_ref": 0.0
|
| 635 |
+
},
|
| 636 |
+
"PTN (parse-torrent-title)": {
|
| 637 |
+
"n": 2835,
|
| 638 |
+
"point_pct": 35.59,
|
| 639 |
+
"ci95_lo": 33.83,
|
| 640 |
+
"ci95_hi": 37.35,
|
| 641 |
+
"delta_vs_ref_pct": 59.89,
|
| 642 |
+
"delta_ci95_lo": 58.06,
|
| 643 |
+
"delta_ci95_hi": 61.73,
|
| 644 |
+
"p_value_vs_ref": 0.0
|
| 645 |
+
},
|
| 646 |
+
"Anitomy": {
|
| 647 |
+
"n": 2835,
|
| 648 |
+
"point_pct": 33.4,
|
| 649 |
+
"ci95_lo": 31.68,
|
| 650 |
+
"ci95_hi": 35.13,
|
| 651 |
+
"delta_vs_ref_pct": 62.08,
|
| 652 |
+
"delta_ci95_lo": 60.25,
|
| 653 |
+
"delta_ci95_hi": 63.88,
|
| 654 |
+
"p_value_vs_ref": 0.0
|
| 655 |
+
},
|
| 656 |
+
"Bitmagnet Built-In": {
|
| 657 |
+
"n": 2835,
|
| 658 |
+
"point_pct": 35.17,
|
| 659 |
+
"ci95_lo": 33.47,
|
| 660 |
+
"ci95_hi": 36.9,
|
| 661 |
+
"delta_vs_ref_pct": 60.32,
|
| 662 |
+
"delta_ci95_lo": 58.52,
|
| 663 |
+
"delta_ci95_hi": 62.12,
|
| 664 |
+
"p_value_vs_ref": 0.0
|
| 665 |
+
}
|
| 666 |
+
}
|
| 667 |
+
},
|
| 668 |
+
"panel_1_video_mvtv_oct15": {
|
| 669 |
+
"panel_name": "chart3_1_oct15_video_mvtv",
|
| 670 |
+
"n": 1296,
|
| 671 |
+
"metric_type": "mean",
|
| 672 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 673 |
+
"models": {
|
| 674 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 675 |
+
"n": 1296,
|
| 676 |
+
"point_pct": 97.3,
|
| 677 |
+
"ci95_lo": 96.37,
|
| 678 |
+
"ci95_hi": 98.15,
|
| 679 |
+
"delta_vs_ref_pct": 0.0,
|
| 680 |
+
"delta_ci95_lo": 0.0,
|
| 681 |
+
"delta_ci95_hi": 0.0,
|
| 682 |
+
"p_value_vs_ref": 1.0
|
| 683 |
+
},
|
| 684 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 685 |
+
"n": 1296,
|
| 686 |
+
"point_pct": 97.3,
|
| 687 |
+
"ci95_lo": 96.37,
|
| 688 |
+
"ci95_hi": 98.15,
|
| 689 |
+
"delta_vs_ref_pct": 0.0,
|
| 690 |
+
"delta_ci95_lo": 0.0,
|
| 691 |
+
"delta_ci95_hi": 0.0,
|
| 692 |
+
"p_value_vs_ref": 1.0
|
| 693 |
+
},
|
| 694 |
+
"MIMMO BERT (360M)": {
|
| 695 |
+
"n": 1296,
|
| 696 |
+
"point_pct": 76.0,
|
| 697 |
+
"ci95_lo": 73.61,
|
| 698 |
+
"ci95_hi": 78.32,
|
| 699 |
+
"delta_vs_ref_pct": 21.3,
|
| 700 |
+
"delta_ci95_lo": 18.9,
|
| 701 |
+
"delta_ci95_hi": 23.77,
|
| 702 |
+
"p_value_vs_ref": 1.427447e-63
|
| 703 |
+
},
|
| 704 |
+
"MIMMO Cascade": {
|
| 705 |
+
"n": 1296,
|
| 706 |
+
"point_pct": 76.08,
|
| 707 |
+
"ci95_lo": 73.69,
|
| 708 |
+
"ci95_hi": 78.4,
|
| 709 |
+
"delta_vs_ref_pct": 21.22,
|
| 710 |
+
"delta_ci95_lo": 18.83,
|
| 711 |
+
"delta_ci95_hi": 23.69,
|
| 712 |
+
"p_value_vs_ref": 2.667039e-63
|
| 713 |
+
},
|
| 714 |
+
"RTN (rank-torrent-name)": {
|
| 715 |
+
"n": 1296,
|
| 716 |
+
"point_pct": 81.64,
|
| 717 |
+
"ci95_lo": 79.48,
|
| 718 |
+
"ci95_hi": 83.72,
|
| 719 |
+
"delta_vs_ref_pct": 15.66,
|
| 720 |
+
"delta_ci95_lo": 13.5,
|
| 721 |
+
"delta_ci95_hi": 17.82,
|
| 722 |
+
"p_value_vs_ref": 4.447751e-45
|
| 723 |
+
},
|
| 724 |
+
"PTT / Parsett": {
|
| 725 |
+
"n": 1296,
|
| 726 |
+
"point_pct": 81.64,
|
| 727 |
+
"ci95_lo": 79.48,
|
| 728 |
+
"ci95_hi": 83.72,
|
| 729 |
+
"delta_vs_ref_pct": 15.66,
|
| 730 |
+
"delta_ci95_lo": 13.5,
|
| 731 |
+
"delta_ci95_hi": 17.82,
|
| 732 |
+
"p_value_vs_ref": 4.447751e-45
|
| 733 |
+
},
|
| 734 |
+
"GuessIt": {
|
| 735 |
+
"n": 1296,
|
| 736 |
+
"point_pct": 84.65,
|
| 737 |
+
"ci95_lo": 82.64,
|
| 738 |
+
"ci95_hi": 86.57,
|
| 739 |
+
"delta_vs_ref_pct": 12.65,
|
| 740 |
+
"delta_ci95_lo": 10.65,
|
| 741 |
+
"delta_ci95_hi": 14.66,
|
| 742 |
+
"p_value_vs_ref": 2.56753e-35
|
| 743 |
+
},
|
| 744 |
+
"PTN (parse-torrent-title)": {
|
| 745 |
+
"n": 1296,
|
| 746 |
+
"point_pct": 77.85,
|
| 747 |
+
"ci95_lo": 75.54,
|
| 748 |
+
"ci95_hi": 80.09,
|
| 749 |
+
"delta_vs_ref_pct": 19.44,
|
| 750 |
+
"delta_ci95_lo": 17.05,
|
| 751 |
+
"delta_ci95_hi": 21.84,
|
| 752 |
+
"p_value_vs_ref": 1.163065e-57
|
| 753 |
+
},
|
| 754 |
+
"Anitomy": {
|
| 755 |
+
"n": 1296,
|
| 756 |
+
"point_pct": 73.07,
|
| 757 |
+
"ci95_lo": 70.6,
|
| 758 |
+
"ci95_hi": 75.46,
|
| 759 |
+
"delta_vs_ref_pct": 24.23,
|
| 760 |
+
"delta_ci95_lo": 21.68,
|
| 761 |
+
"delta_ci95_hi": 26.77,
|
| 762 |
+
"p_value_vs_ref": 5.732874e-76
|
| 763 |
+
},
|
| 764 |
+
"Bitmagnet Built-In": {
|
| 765 |
+
"n": 1296,
|
| 766 |
+
"point_pct": 43.21,
|
| 767 |
+
"ci95_lo": 40.51,
|
| 768 |
+
"ci95_hi": 45.91,
|
| 769 |
+
"delta_vs_ref_pct": 54.09,
|
| 770 |
+
"delta_ci95_lo": 51.23,
|
| 771 |
+
"delta_ci95_hi": 56.87,
|
| 772 |
+
"p_value_vs_ref": 8.363314e-201
|
| 773 |
+
}
|
| 774 |
+
}
|
| 775 |
+
},
|
| 776 |
+
"panel_2_catalog_matching_oct15": {
|
| 777 |
+
"panel_name": "chart3_2_oct15_catalog_matching",
|
| 778 |
+
"n": 1296,
|
| 779 |
+
"metric_type": "mean",
|
| 780 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 781 |
+
"models": {
|
| 782 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 783 |
+
"n": 1296,
|
| 784 |
+
"point_pct": 96.53,
|
| 785 |
+
"ci95_lo": 95.52,
|
| 786 |
+
"ci95_hi": 97.53,
|
| 787 |
+
"delta_vs_ref_pct": 0.0,
|
| 788 |
+
"delta_ci95_lo": 0.0,
|
| 789 |
+
"delta_ci95_hi": 0.0,
|
| 790 |
+
"p_value_vs_ref": 1.0
|
| 791 |
+
},
|
| 792 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 793 |
+
"n": 1296,
|
| 794 |
+
"point_pct": 96.53,
|
| 795 |
+
"ci95_lo": 95.52,
|
| 796 |
+
"ci95_hi": 97.53,
|
| 797 |
+
"delta_vs_ref_pct": 0.0,
|
| 798 |
+
"delta_ci95_lo": 0.0,
|
| 799 |
+
"delta_ci95_hi": 0.0,
|
| 800 |
+
"p_value_vs_ref": 1.0
|
| 801 |
+
},
|
| 802 |
+
"MIMMO BERT (360M)": {
|
| 803 |
+
"n": 1296,
|
| 804 |
+
"point_pct": 60.26,
|
| 805 |
+
"ci95_lo": 57.56,
|
| 806 |
+
"ci95_hi": 62.96,
|
| 807 |
+
"delta_vs_ref_pct": 36.27,
|
| 808 |
+
"delta_ci95_lo": 33.49,
|
| 809 |
+
"delta_ci95_hi": 39.12,
|
| 810 |
+
"p_value_vs_ref": 1.578863e-117
|
| 811 |
+
},
|
| 812 |
+
"MIMMO Cascade": {
|
| 813 |
+
"n": 1296,
|
| 814 |
+
"point_pct": 60.34,
|
| 815 |
+
"ci95_lo": 57.56,
|
| 816 |
+
"ci95_hi": 62.96,
|
| 817 |
+
"delta_vs_ref_pct": 36.19,
|
| 818 |
+
"delta_ci95_lo": 33.33,
|
| 819 |
+
"delta_ci95_hi": 39.04,
|
| 820 |
+
"p_value_vs_ref": 3.028483e-117
|
| 821 |
+
},
|
| 822 |
+
"RTN (rank-torrent-name)": {
|
| 823 |
+
"n": 1296,
|
| 824 |
+
"point_pct": 80.48,
|
| 825 |
+
"ci95_lo": 78.24,
|
| 826 |
+
"ci95_hi": 82.64,
|
| 827 |
+
"delta_vs_ref_pct": 16.05,
|
| 828 |
+
"delta_ci95_lo": 13.81,
|
| 829 |
+
"delta_ci95_hi": 18.29,
|
| 830 |
+
"p_value_vs_ref": 7.158345e-45
|
| 831 |
+
},
|
| 832 |
+
"PTT / Parsett": {
|
| 833 |
+
"n": 1296,
|
| 834 |
+
"point_pct": 80.48,
|
| 835 |
+
"ci95_lo": 78.24,
|
| 836 |
+
"ci95_hi": 82.64,
|
| 837 |
+
"delta_vs_ref_pct": 16.05,
|
| 838 |
+
"delta_ci95_lo": 13.81,
|
| 839 |
+
"delta_ci95_hi": 18.29,
|
| 840 |
+
"p_value_vs_ref": 7.158345e-45
|
| 841 |
+
},
|
| 842 |
+
"GuessIt": {
|
| 843 |
+
"n": 1296,
|
| 844 |
+
"point_pct": 84.03,
|
| 845 |
+
"ci95_lo": 82.02,
|
| 846 |
+
"ci95_hi": 85.96,
|
| 847 |
+
"delta_vs_ref_pct": 12.5,
|
| 848 |
+
"delta_ci95_lo": 10.42,
|
| 849 |
+
"delta_ci95_hi": 14.58,
|
| 850 |
+
"p_value_vs_ref": 7.451219e-32
|
| 851 |
+
},
|
| 852 |
+
"PTN (parse-torrent-title)": {
|
| 853 |
+
"n": 1296,
|
| 854 |
+
"point_pct": 77.62,
|
| 855 |
+
"ci95_lo": 75.31,
|
| 856 |
+
"ci95_hi": 79.86,
|
| 857 |
+
"delta_vs_ref_pct": 18.9,
|
| 858 |
+
"delta_ci95_lo": 16.51,
|
| 859 |
+
"delta_ci95_hi": 21.37,
|
| 860 |
+
"p_value_vs_ref": 3.559008e-51
|
| 861 |
+
},
|
| 862 |
+
"Anitomy": {
|
| 863 |
+
"n": 1296,
|
| 864 |
+
"point_pct": 72.76,
|
| 865 |
+
"ci95_lo": 70.29,
|
| 866 |
+
"ci95_hi": 75.15,
|
| 867 |
+
"delta_vs_ref_pct": 23.77,
|
| 868 |
+
"delta_ci95_lo": 21.22,
|
| 869 |
+
"delta_ci95_hi": 26.31,
|
| 870 |
+
"p_value_vs_ref": 7.077119e-70
|
| 871 |
+
},
|
| 872 |
+
"Bitmagnet Built-In": {
|
| 873 |
+
"n": 1296,
|
| 874 |
+
"point_pct": 43.21,
|
| 875 |
+
"ci95_lo": 40.51,
|
| 876 |
+
"ci95_hi": 45.91,
|
| 877 |
+
"delta_vs_ref_pct": 53.32,
|
| 878 |
+
"delta_ci95_lo": 50.54,
|
| 879 |
+
"delta_ci95_hi": 56.1,
|
| 880 |
+
"p_value_vs_ref": 7.866753e-198
|
| 881 |
+
}
|
| 882 |
+
}
|
| 883 |
+
},
|
| 884 |
+
"panel_3_audio_strict_oct15": {
|
| 885 |
+
"panel_name": "chart3_3_oct15_audio_strict545",
|
| 886 |
+
"n": 545,
|
| 887 |
+
"metric_type": "mean",
|
| 888 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 889 |
+
"models": {
|
| 890 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 891 |
+
"n": 545,
|
| 892 |
+
"point_pct": 97.8,
|
| 893 |
+
"ci95_lo": 96.51,
|
| 894 |
+
"ci95_hi": 98.9,
|
| 895 |
+
"delta_vs_ref_pct": 0.0,
|
| 896 |
+
"delta_ci95_lo": 0.0,
|
| 897 |
+
"delta_ci95_hi": 0.0,
|
| 898 |
+
"p_value_vs_ref": 1.0
|
| 899 |
+
},
|
| 900 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 901 |
+
"n": 545,
|
| 902 |
+
"point_pct": 97.8,
|
| 903 |
+
"ci95_lo": 96.51,
|
| 904 |
+
"ci95_hi": 98.9,
|
| 905 |
+
"delta_vs_ref_pct": 0.0,
|
| 906 |
+
"delta_ci95_lo": 0.0,
|
| 907 |
+
"delta_ci95_hi": 0.0,
|
| 908 |
+
"p_value_vs_ref": 1.0
|
| 909 |
+
},
|
| 910 |
+
"RTN (rank-torrent-name)": {
|
| 911 |
+
"n": 545,
|
| 912 |
+
"point_pct": 93.76,
|
| 913 |
+
"ci95_lo": 91.74,
|
| 914 |
+
"ci95_hi": 95.78,
|
| 915 |
+
"delta_vs_ref_pct": 4.04,
|
| 916 |
+
"delta_ci95_lo": 1.83,
|
| 917 |
+
"delta_ci95_hi": 6.24,
|
| 918 |
+
"p_value_vs_ref": 0.0006795483
|
| 919 |
+
},
|
| 920 |
+
"PTT / Parsett": {
|
| 921 |
+
"n": 545,
|
| 922 |
+
"point_pct": 93.76,
|
| 923 |
+
"ci95_lo": 91.74,
|
| 924 |
+
"ci95_hi": 95.78,
|
| 925 |
+
"delta_vs_ref_pct": 4.04,
|
| 926 |
+
"delta_ci95_lo": 1.83,
|
| 927 |
+
"delta_ci95_hi": 6.24,
|
| 928 |
+
"p_value_vs_ref": 0.0006795483
|
| 929 |
+
},
|
| 930 |
+
"GuessIt": {
|
| 931 |
+
"n": 545,
|
| 932 |
+
"point_pct": 94.86,
|
| 933 |
+
"ci95_lo": 93.03,
|
| 934 |
+
"ci95_hi": 96.7,
|
| 935 |
+
"delta_vs_ref_pct": 2.94,
|
| 936 |
+
"delta_ci95_lo": 0.92,
|
| 937 |
+
"delta_ci95_hi": 4.95,
|
| 938 |
+
"p_value_vs_ref": 0.009041185
|
| 939 |
+
},
|
| 940 |
+
"PTN (parse-torrent-title)": {
|
| 941 |
+
"n": 545,
|
| 942 |
+
"point_pct": 93.21,
|
| 943 |
+
"ci95_lo": 91.01,
|
| 944 |
+
"ci95_hi": 95.23,
|
| 945 |
+
"delta_vs_ref_pct": 4.59,
|
| 946 |
+
"delta_ci95_lo": 2.39,
|
| 947 |
+
"delta_ci95_hi": 6.97,
|
| 948 |
+
"p_value_vs_ref": 0.0001701552
|
| 949 |
+
},
|
| 950 |
+
"Anitomy": {
|
| 951 |
+
"n": 545,
|
| 952 |
+
"point_pct": 90.46,
|
| 953 |
+
"ci95_lo": 88.07,
|
| 954 |
+
"ci95_hi": 92.84,
|
| 955 |
+
"delta_vs_ref_pct": 7.34,
|
| 956 |
+
"delta_ci95_lo": 4.77,
|
| 957 |
+
"delta_ci95_hi": 9.91,
|
| 958 |
+
"p_value_vs_ref": 4.688184e-08
|
| 959 |
+
},
|
| 960 |
+
"Bitmagnet Built-In": {
|
| 961 |
+
"n": 545,
|
| 962 |
+
"point_pct": 88.26,
|
| 963 |
+
"ci95_lo": 85.5,
|
| 964 |
+
"ci95_hi": 90.83,
|
| 965 |
+
"delta_vs_ref_pct": 9.54,
|
| 966 |
+
"delta_ci95_lo": 6.79,
|
| 967 |
+
"delta_ci95_hi": 12.29,
|
| 968 |
+
"p_value_vs_ref": 2.383493e-11
|
| 969 |
+
}
|
| 970 |
+
}
|
| 971 |
+
},
|
| 972 |
+
"panel_4_anime_oct15": {
|
| 973 |
+
"panel_name": "chart3_4_oct15_anime507",
|
| 974 |
+
"n": 507,
|
| 975 |
+
"metric_type": "mean",
|
| 976 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 977 |
+
"models": {
|
| 978 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 979 |
+
"n": 507,
|
| 980 |
+
"point_pct": 99.61,
|
| 981 |
+
"ci95_lo": 99.01,
|
| 982 |
+
"ci95_hi": 100.0,
|
| 983 |
+
"delta_vs_ref_pct": 0.0,
|
| 984 |
+
"delta_ci95_lo": 0.0,
|
| 985 |
+
"delta_ci95_hi": 0.0,
|
| 986 |
+
"p_value_vs_ref": 1.0
|
| 987 |
+
},
|
| 988 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 989 |
+
"n": 507,
|
| 990 |
+
"point_pct": 99.61,
|
| 991 |
+
"ci95_lo": 99.01,
|
| 992 |
+
"ci95_hi": 100.0,
|
| 993 |
+
"delta_vs_ref_pct": 0.0,
|
| 994 |
+
"delta_ci95_lo": 0.0,
|
| 995 |
+
"delta_ci95_hi": 0.0,
|
| 996 |
+
"p_value_vs_ref": 1.0
|
| 997 |
+
}
|
| 998 |
+
}
|
| 999 |
+
},
|
| 1000 |
+
"panel_4_anime_dub_tc2000": {
|
| 1001 |
+
"panel_name": "chart3_4_tc2000_anime_dub",
|
| 1002 |
+
"n": 2000,
|
| 1003 |
+
"metric_type": "mean",
|
| 1004 |
+
"reference_model": "RTN (rank-torrent-name)",
|
| 1005 |
+
"models": {
|
| 1006 |
+
"RTN (rank-torrent-name)": {
|
| 1007 |
+
"n": 2000,
|
| 1008 |
+
"point_pct": 77.65,
|
| 1009 |
+
"ci95_lo": 75.75,
|
| 1010 |
+
"ci95_hi": 79.45,
|
| 1011 |
+
"delta_vs_ref_pct": 0.0,
|
| 1012 |
+
"delta_ci95_lo": 0.0,
|
| 1013 |
+
"delta_ci95_hi": 0.0,
|
| 1014 |
+
"p_value_vs_ref": 1.0
|
| 1015 |
+
},
|
| 1016 |
+
"PTT / Parsett": {
|
| 1017 |
+
"n": 2000,
|
| 1018 |
+
"point_pct": 77.65,
|
| 1019 |
+
"ci95_lo": 75.75,
|
| 1020 |
+
"ci95_hi": 79.45,
|
| 1021 |
+
"delta_vs_ref_pct": 0.0,
|
| 1022 |
+
"delta_ci95_lo": 0.0,
|
| 1023 |
+
"delta_ci95_hi": 0.0,
|
| 1024 |
+
"p_value_vs_ref": 1.0
|
| 1025 |
+
},
|
| 1026 |
+
"GuessIt": {
|
| 1027 |
+
"n": 2000,
|
| 1028 |
+
"point_pct": 78.8,
|
| 1029 |
+
"ci95_lo": 76.95,
|
| 1030 |
+
"ci95_hi": 80.55,
|
| 1031 |
+
"delta_vs_ref_pct": -1.15,
|
| 1032 |
+
"delta_ci95_lo": -2.45,
|
| 1033 |
+
"delta_ci95_hi": 0.15,
|
| 1034 |
+
"p_value_vs_ref": 0.09792971
|
| 1035 |
+
},
|
| 1036 |
+
"PTN (parse-torrent-title)": {
|
| 1037 |
+
"n": 2000,
|
| 1038 |
+
"point_pct": 68.65,
|
| 1039 |
+
"ci95_lo": 66.6,
|
| 1040 |
+
"ci95_hi": 70.65,
|
| 1041 |
+
"delta_vs_ref_pct": 9.0,
|
| 1042 |
+
"delta_ci95_lo": 7.15,
|
| 1043 |
+
"delta_ci95_hi": 10.85,
|
| 1044 |
+
"p_value_vs_ref": 7.57291e-21
|
| 1045 |
+
},
|
| 1046 |
+
"Anitomy": {
|
| 1047 |
+
"n": 2000,
|
| 1048 |
+
"point_pct": 64.55,
|
| 1049 |
+
"ci95_lo": 62.45,
|
| 1050 |
+
"ci95_hi": 66.6,
|
| 1051 |
+
"delta_vs_ref_pct": 13.1,
|
| 1052 |
+
"delta_ci95_lo": 11.2,
|
| 1053 |
+
"delta_ci95_hi": 15.05,
|
| 1054 |
+
"p_value_vs_ref": 4.440189e-40
|
| 1055 |
+
}
|
| 1056 |
+
}
|
| 1057 |
+
},
|
| 1058 |
+
"panel_5_adult_oct15": {
|
| 1059 |
+
"panel_name": "chart3_5_oct15_adult2781",
|
| 1060 |
+
"n": 2781,
|
| 1061 |
+
"metric_type": "mean",
|
| 1062 |
+
"reference_model": "Armiger Ultra (Fair Hybrid)",
|
| 1063 |
+
"models": {
|
| 1064 |
+
"Armiger Ultra (Fair Hybrid)": {
|
| 1065 |
+
"n": 2781,
|
| 1066 |
+
"point_pct": 98.53,
|
| 1067 |
+
"ci95_lo": 98.06,
|
| 1068 |
+
"ci95_hi": 98.96,
|
| 1069 |
+
"delta_vs_ref_pct": 0.0,
|
| 1070 |
+
"delta_ci95_lo": 0.0,
|
| 1071 |
+
"delta_ci95_hi": 0.0,
|
| 1072 |
+
"p_value_vs_ref": 1.0
|
| 1073 |
+
},
|
| 1074 |
+
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 1075 |
+
"n": 2781,
|
| 1076 |
+
"point_pct": 98.49,
|
| 1077 |
+
"ci95_lo": 98.02,
|
| 1078 |
+
"ci95_hi": 98.92,
|
| 1079 |
+
"delta_vs_ref_pct": 0.04,
|
| 1080 |
+
"delta_ci95_lo": 0.0,
|
| 1081 |
+
"delta_ci95_hi": 0.11,
|
| 1082 |
+
"p_value_vs_ref": 1.0
|
| 1083 |
+
},
|
| 1084 |
+
"MIMMO BERT (360M)": {
|
| 1085 |
+
"n": 2781,
|
| 1086 |
+
"point_pct": 91.05,
|
| 1087 |
+
"ci95_lo": 89.93,
|
| 1088 |
+
"ci95_hi": 92.09,
|
| 1089 |
+
"delta_vs_ref_pct": 7.48,
|
| 1090 |
+
"delta_ci95_lo": 6.44,
|
| 1091 |
+
"delta_ci95_hi": 8.59,
|
| 1092 |
+
"p_value_vs_ref": 2.280742e-45
|
| 1093 |
+
},
|
| 1094 |
+
"MIMMO Cascade": {
|
| 1095 |
+
"n": 2781,
|
| 1096 |
+
"point_pct": 89.68,
|
| 1097 |
+
"ci95_lo": 88.53,
|
| 1098 |
+
"ci95_hi": 90.79,
|
| 1099 |
+
"delta_vs_ref_pct": 8.85,
|
| 1100 |
+
"delta_ci95_lo": 7.73,
|
| 1101 |
+
"delta_ci95_hi": 10.03,
|
| 1102 |
+
"p_value_vs_ref": 1.795157e-55
|
| 1103 |
+
},
|
| 1104 |
+
"RTN (rank-torrent-name)": {
|
| 1105 |
+
"n": 2781,
|
| 1106 |
+
"point_pct": 79.0,
|
| 1107 |
+
"ci95_lo": 77.45,
|
| 1108 |
+
"ci95_hi": 80.55,
|
| 1109 |
+
"delta_vs_ref_pct": 19.53,
|
| 1110 |
+
"delta_ci95_lo": 17.98,
|
| 1111 |
+
"delta_ci95_hi": 21.07,
|
| 1112 |
+
"p_value_vs_ref": 2.877898e-138
|
| 1113 |
+
},
|
| 1114 |
+
"PTT / Parsett": {
|
| 1115 |
+
"n": 2781,
|
| 1116 |
+
"point_pct": 79.0,
|
| 1117 |
+
"ci95_lo": 77.45,
|
| 1118 |
+
"ci95_hi": 80.55,
|
| 1119 |
+
"delta_vs_ref_pct": 19.53,
|
| 1120 |
+
"delta_ci95_lo": 17.98,
|
| 1121 |
+
"delta_ci95_hi": 21.07,
|
| 1122 |
+
"p_value_vs_ref": 2.877898e-138
|
| 1123 |
+
},
|
| 1124 |
+
"Bitmagnet Built-In": {
|
| 1125 |
+
"n": 2781,
|
| 1126 |
+
"point_pct": 76.73,
|
| 1127 |
+
"ci95_lo": 75.19,
|
| 1128 |
+
"ci95_hi": 78.32,
|
| 1129 |
+
"delta_vs_ref_pct": 21.79,
|
| 1130 |
+
"delta_ci95_lo": 20.17,
|
| 1131 |
+
"delta_ci95_hi": 23.37,
|
| 1132 |
+
"p_value_vs_ref": 1.549894e-154
|
| 1133 |
+
}
|
| 1134 |
+
}
|
| 1135 |
+
},
|
| 1136 |
+
"panel_6a_junk_oct15": {
|
| 1137 |
+
"panel_name": "chart3_6a_oct15_junk2170",
|
| 1138 |
+
"n": 2170,
|
| 1139 |
+
"metric_type": "balanced",
|
| 1140 |
+
"reference_model": "Armiger Ultra",
|
| 1141 |
+
"models": {
|
| 1142 |
+
"Armiger Ultra": {
|
| 1143 |
+
"n": 2170,
|
| 1144 |
+
"point_pct": 100.0,
|
| 1145 |
+
"ci95_lo": 100.0,
|
| 1146 |
+
"ci95_hi": 100.0,
|
| 1147 |
+
"delta_vs_ref_pct": 0.0,
|
| 1148 |
+
"delta_ci95_lo": 0.0,
|
| 1149 |
+
"delta_ci95_hi": 0.0,
|
| 1150 |
+
"p_value_vs_ref": 1.0
|
| 1151 |
+
},
|
| 1152 |
+
"Armiger INT8": {
|
| 1153 |
+
"n": 2170,
|
| 1154 |
+
"point_pct": 100.0,
|
| 1155 |
+
"ci95_lo": 100.0,
|
| 1156 |
+
"ci95_hi": 100.0,
|
| 1157 |
+
"delta_vs_ref_pct": 0.0,
|
| 1158 |
+
"delta_ci95_lo": 0.0,
|
| 1159 |
+
"delta_ci95_hi": 0.0,
|
| 1160 |
+
"p_value_vs_ref": 1.0
|
| 1161 |
+
},
|
| 1162 |
+
"Bitmagnet": {
|
| 1163 |
+
"n": 2170,
|
| 1164 |
+
"point_pct": 50.0,
|
| 1165 |
+
"ci95_lo": 50.0,
|
| 1166 |
+
"ci95_hi": 50.0,
|
| 1167 |
+
"delta_vs_ref_pct": 50.0,
|
| 1168 |
+
"delta_ci95_lo": 50.0,
|
| 1169 |
+
"delta_ci95_hi": 50.0,
|
| 1170 |
+
"p_value_vs_ref": 1.0
|
| 1171 |
+
},
|
| 1172 |
+
"RTN Filter": {
|
| 1173 |
+
"n": 2170,
|
| 1174 |
+
"point_pct": 49.29,
|
| 1175 |
+
"ci95_lo": 49.03,
|
| 1176 |
+
"ci95_hi": 49.52,
|
| 1177 |
+
"delta_vs_ref_pct": 50.71,
|
| 1178 |
+
"delta_ci95_lo": 50.48,
|
| 1179 |
+
"delta_ci95_hi": 50.97,
|
| 1180 |
+
"p_value_vs_ref": 4.656613e-10
|
| 1181 |
+
}
|
| 1182 |
+
}
|
| 1183 |
+
}
|
| 1184 |
+
},
|
| 1185 |
+
"armiger_categorical_breakdown.png": {
|
| 1186 |
+
"sha256": "1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98",
|
| 1187 |
+
"left_oct15_confirmation_n2835": {
|
| 1188 |
+
"Audiobook": {
|
| 1189 |
+
"n": 95,
|
| 1190 |
+
"armiger_ultra_pct": 97.89,
|
| 1191 |
+
"armiger_int8_pct": 97.89,
|
| 1192 |
+
"mimmo_bert_pct": 0.0,
|
| 1193 |
+
"bitmagnet_pct": 9.47
|
| 1194 |
+
},
|
| 1195 |
+
"Adult / XXX": {
|
| 1196 |
+
"n": 1044,
|
| 1197 |
+
"armiger_ultra_pct": 97.8,
|
| 1198 |
+
"armiger_int8_pct": 97.8,
|
| 1199 |
+
"mimmo_bert_pct": 95.5,
|
| 1200 |
+
"bitmagnet_pct": 39.37
|
| 1201 |
+
},
|
| 1202 |
+
"TV Show": {
|
| 1203 |
+
"n": 601,
|
| 1204 |
+
"armiger_ultra_pct": 97.5,
|
| 1205 |
+
"armiger_int8_pct": 97.5,
|
| 1206 |
+
"mimmo_bert_pct": 58.4,
|
| 1207 |
+
"bitmagnet_pct": 28.62
|
| 1208 |
+
},
|
| 1209 |
+
"Movie": {
|
| 1210 |
+
"n": 695,
|
| 1211 |
+
"armiger_ultra_pct": 97.12,
|
| 1212 |
+
"armiger_int8_pct": 97.12,
|
| 1213 |
+
"mimmo_bert_pct": 78.85,
|
| 1214 |
+
"bitmagnet_pct": 55.83
|
| 1215 |
+
},
|
| 1216 |
+
"Software": {
|
| 1217 |
+
"n": 32,
|
| 1218 |
+
"armiger_ultra_pct": 93.75,
|
| 1219 |
+
"armiger_int8_pct": 93.75,
|
| 1220 |
+
"mimmo_bert_pct": 59.38,
|
| 1221 |
+
"bitmagnet_pct": 15.62
|
| 1222 |
+
},
|
| 1223 |
+
"Music": {
|
| 1224 |
+
"n": 140,
|
| 1225 |
+
"armiger_ultra_pct": 90.71,
|
| 1226 |
+
"armiger_int8_pct": 90.71,
|
| 1227 |
+
"mimmo_bert_pct": 77.86,
|
| 1228 |
+
"bitmagnet_pct": 7.86
|
| 1229 |
+
},
|
| 1230 |
+
"Game": {
|
| 1231 |
+
"n": 61,
|
| 1232 |
+
"armiger_ultra_pct": 90.16,
|
| 1233 |
+
"armiger_int8_pct": 90.16,
|
| 1234 |
+
"mimmo_bert_pct": 0.0,
|
| 1235 |
+
"bitmagnet_pct": 0.0
|
| 1236 |
+
},
|
| 1237 |
+
"Ebook": {
|
| 1238 |
+
"n": 25,
|
| 1239 |
+
"armiger_ultra_pct": 88.0,
|
| 1240 |
+
"armiger_int8_pct": 88.0,
|
| 1241 |
+
"mimmo_bert_pct": 80.0,
|
| 1242 |
+
"bitmagnet_pct": 4.0
|
| 1243 |
+
},
|
| 1244 |
+
"Course": {
|
| 1245 |
+
"n": 33,
|
| 1246 |
+
"armiger_ultra_pct": 81.82,
|
| 1247 |
+
"armiger_int8_pct": 81.82,
|
| 1248 |
+
"mimmo_bert_pct": 0.0,
|
| 1249 |
+
"bitmagnet_pct": 0.0
|
| 1250 |
+
},
|
| 1251 |
+
"Comic": {
|
| 1252 |
+
"n": 99,
|
| 1253 |
+
"armiger_ultra_pct": 66.67,
|
| 1254 |
+
"armiger_int8_pct": 66.67,
|
| 1255 |
+
"mimmo_bert_pct": 0.0,
|
| 1256 |
+
"bitmagnet_pct": 0.0
|
| 1257 |
+
}
|
| 1258 |
+
},
|
| 1259 |
+
"right_beyond_confirmation_n1025_loto": {
|
| 1260 |
+
"Audiobook": {
|
| 1261 |
+
"n": 75,
|
| 1262 |
+
"armiger_ultra_pct": 100.0,
|
| 1263 |
+
"armiger_int8_pct": 100.0,
|
| 1264 |
+
"gpt_noncirc_pct": 98.67,
|
| 1265 |
+
"gemini_noncirc_pct": 98.0
|
| 1266 |
+
},
|
| 1267 |
+
"Ebook": {
|
| 1268 |
+
"n": 125,
|
| 1269 |
+
"armiger_ultra_pct": 98.4,
|
| 1270 |
+
"armiger_int8_pct": 98.4,
|
| 1271 |
+
"gpt_noncirc_pct": 96.0,
|
| 1272 |
+
"gemini_noncirc_pct": 96.8
|
| 1273 |
+
},
|
| 1274 |
+
"Comic": {
|
| 1275 |
+
"n": 120,
|
| 1276 |
+
"armiger_ultra_pct": 98.33,
|
| 1277 |
+
"armiger_int8_pct": 98.33,
|
| 1278 |
+
"gpt_noncirc_pct": 96.67,
|
| 1279 |
+
"gemini_noncirc_pct": 95.0
|
| 1280 |
+
},
|
| 1281 |
+
"Movie": {
|
| 1282 |
+
"n": 95,
|
| 1283 |
+
"armiger_ultra_pct": 96.84,
|
| 1284 |
+
"armiger_int8_pct": 96.84,
|
| 1285 |
+
"gpt_noncirc_pct": 94.21,
|
| 1286 |
+
"gemini_noncirc_pct": 92.63
|
| 1287 |
+
},
|
| 1288 |
+
"TV Show": {
|
| 1289 |
+
"n": 136,
|
| 1290 |
+
"armiger_ultra_pct": 96.32,
|
| 1291 |
+
"armiger_int8_pct": 96.32,
|
| 1292 |
+
"gpt_noncirc_pct": 97.79,
|
| 1293 |
+
"gemini_noncirc_pct": 93.75
|
| 1294 |
+
},
|
| 1295 |
+
"Game": {
|
| 1296 |
+
"n": 135,
|
| 1297 |
+
"armiger_ultra_pct": 96.3,
|
| 1298 |
+
"armiger_int8_pct": 96.3,
|
| 1299 |
+
"gpt_noncirc_pct": 98.52,
|
| 1300 |
+
"gemini_noncirc_pct": 98.52
|
| 1301 |
+
},
|
| 1302 |
+
"Software": {
|
| 1303 |
+
"n": 37,
|
| 1304 |
+
"armiger_ultra_pct": 94.59,
|
| 1305 |
+
"armiger_int8_pct": 94.59,
|
| 1306 |
+
"gpt_noncirc_pct": 97.3,
|
| 1307 |
+
"gemini_noncirc_pct": 77.03
|
| 1308 |
+
},
|
| 1309 |
+
"Course": {
|
| 1310 |
+
"n": 100,
|
| 1311 |
+
"armiger_ultra_pct": 92.0,
|
| 1312 |
+
"armiger_int8_pct": 92.0,
|
| 1313 |
+
"gpt_noncirc_pct": 95.5,
|
| 1314 |
+
"gemini_noncirc_pct": 97.5
|
| 1315 |
+
},
|
| 1316 |
+
"Adult / XXX": {
|
| 1317 |
+
"n": 174,
|
| 1318 |
+
"armiger_ultra_pct": 91.95,
|
| 1319 |
+
"armiger_int8_pct": 91.95,
|
| 1320 |
+
"gpt_noncirc_pct": 94.83,
|
| 1321 |
+
"gemini_noncirc_pct": 95.69
|
| 1322 |
+
},
|
| 1323 |
+
"Music": {
|
| 1324 |
+
"n": 12,
|
| 1325 |
+
"armiger_ultra_pct": 83.33,
|
| 1326 |
+
"armiger_int8_pct": 83.33,
|
| 1327 |
+
"gpt_noncirc_pct": 100.0,
|
| 1328 |
+
"gemini_noncirc_pct": 100.0
|
| 1329 |
+
}
|
| 1330 |
+
}
|
| 1331 |
+
}
|
| 1332 |
+
},
|
| 1333 |
+
"five_split_category_summary": {
|
| 1334 |
+
"oct15_confirmation": {
|
| 1335 |
+
"panel_name": "five_split_oct15",
|
| 1336 |
+
"n": 2835,
|
| 1337 |
+
"metric_type": "mean",
|
| 1338 |
+
"reference_model": "armiger_ultra_hyb",
|
| 1339 |
+
"models": {
|
| 1340 |
+
"armiger_ultra_hyb": {
|
| 1341 |
+
"n": 2835,
|
| 1342 |
+
"point_pct": 95.49,
|
| 1343 |
+
"ci95_lo": 94.71,
|
| 1344 |
+
"ci95_hi": 96.23,
|
| 1345 |
+
"delta_vs_ref_pct": 0.0,
|
| 1346 |
+
"delta_ci95_lo": 0.0,
|
| 1347 |
+
"delta_ci95_hi": 0.0,
|
| 1348 |
+
"p_value_vs_ref": 1.0
|
| 1349 |
+
},
|
| 1350 |
+
"armiger_ultra_raw": {
|
| 1351 |
+
"n": 2835,
|
| 1352 |
+
"point_pct": 95.03,
|
| 1353 |
+
"ci95_lo": 94.22,
|
| 1354 |
+
"ci95_hi": 95.84,
|
| 1355 |
+
"delta_vs_ref_pct": 0.46,
|
| 1356 |
+
"delta_ci95_lo": 0.14,
|
| 1357 |
+
"delta_ci95_hi": 0.78,
|
| 1358 |
+
"p_value_vs_ref": 0.00719738
|
| 1359 |
+
},
|
| 1360 |
+
"armiger_int8_hyb": {
|
| 1361 |
+
"n": 2835,
|
| 1362 |
+
"point_pct": 95.49,
|
| 1363 |
+
"ci95_lo": 94.71,
|
| 1364 |
+
"ci95_hi": 96.23,
|
| 1365 |
+
"delta_vs_ref_pct": 0.0,
|
| 1366 |
+
"delta_ci95_lo": 0.0,
|
| 1367 |
+
"delta_ci95_hi": 0.0,
|
| 1368 |
+
"p_value_vs_ref": 1.0
|
| 1369 |
+
},
|
| 1370 |
+
"armiger_int8_raw": {
|
| 1371 |
+
"n": 2835,
|
| 1372 |
+
"point_pct": 95.06,
|
| 1373 |
+
"ci95_lo": 94.25,
|
| 1374 |
+
"ci95_hi": 95.87,
|
| 1375 |
+
"delta_vs_ref_pct": 0.42,
|
| 1376 |
+
"delta_ci95_lo": 0.14,
|
| 1377 |
+
"delta_ci95_hi": 0.74,
|
| 1378 |
+
"p_value_vs_ref": 0.01181793
|
| 1379 |
+
}
|
| 1380 |
+
}
|
| 1381 |
+
},
|
| 1382 |
+
"fresh_confirmation": {
|
| 1383 |
+
"panel_name": "five_split_fresh",
|
| 1384 |
+
"n": 1402,
|
| 1385 |
+
"metric_type": "mean",
|
| 1386 |
+
"reference_model": "armiger_ultra_hyb",
|
| 1387 |
+
"models": {
|
| 1388 |
+
"armiger_ultra_hyb": {
|
| 1389 |
+
"n": 1402,
|
| 1390 |
+
"point_pct": 94.58,
|
| 1391 |
+
"ci95_lo": 93.37,
|
| 1392 |
+
"ci95_hi": 95.72,
|
| 1393 |
+
"delta_vs_ref_pct": 0.0,
|
| 1394 |
+
"delta_ci95_lo": 0.0,
|
| 1395 |
+
"delta_ci95_hi": 0.0,
|
| 1396 |
+
"p_value_vs_ref": 1.0
|
| 1397 |
+
},
|
| 1398 |
+
"armiger_ultra_raw": {
|
| 1399 |
+
"n": 1402,
|
| 1400 |
+
"point_pct": 94.51,
|
| 1401 |
+
"ci95_lo": 93.3,
|
| 1402 |
+
"ci95_hi": 95.65,
|
| 1403 |
+
"delta_vs_ref_pct": 0.07,
|
| 1404 |
+
"delta_ci95_lo": -0.21,
|
| 1405 |
+
"delta_ci95_hi": 0.36,
|
| 1406 |
+
"p_value_vs_ref": 1.0
|
| 1407 |
+
},
|
| 1408 |
+
"armiger_int8_hyb": {
|
| 1409 |
+
"n": 1402,
|
| 1410 |
+
"point_pct": 94.58,
|
| 1411 |
+
"ci95_lo": 93.37,
|
| 1412 |
+
"ci95_hi": 95.72,
|
| 1413 |
+
"delta_vs_ref_pct": 0.0,
|
| 1414 |
+
"delta_ci95_lo": 0.0,
|
| 1415 |
+
"delta_ci95_hi": 0.0,
|
| 1416 |
+
"p_value_vs_ref": 1.0
|
| 1417 |
+
},
|
| 1418 |
+
"armiger_int8_raw": {
|
| 1419 |
+
"n": 1402,
|
| 1420 |
+
"point_pct": 94.51,
|
| 1421 |
+
"ci95_lo": 93.3,
|
| 1422 |
+
"ci95_hi": 95.65,
|
| 1423 |
+
"delta_vs_ref_pct": 0.07,
|
| 1424 |
+
"delta_ci95_lo": -0.21,
|
| 1425 |
+
"delta_ci95_hi": 0.36,
|
| 1426 |
+
"p_value_vs_ref": 1.0
|
| 1427 |
+
}
|
| 1428 |
+
}
|
| 1429 |
+
},
|
| 1430 |
+
"beyond_confirmation": {
|
| 1431 |
+
"panel_name": "five_split_beyond",
|
| 1432 |
+
"n": 1025,
|
| 1433 |
+
"metric_type": "mean",
|
| 1434 |
+
"reference_model": "armiger_ultra_hyb",
|
| 1435 |
+
"models": {
|
| 1436 |
+
"armiger_ultra_hyb": {
|
| 1437 |
+
"n": 1025,
|
| 1438 |
+
"point_pct": 95.12,
|
| 1439 |
+
"ci95_lo": 93.76,
|
| 1440 |
+
"ci95_hi": 96.39,
|
| 1441 |
+
"delta_vs_ref_pct": 0.0,
|
| 1442 |
+
"delta_ci95_lo": 0.0,
|
| 1443 |
+
"delta_ci95_hi": 0.0,
|
| 1444 |
+
"p_value_vs_ref": 1.0
|
| 1445 |
+
},
|
| 1446 |
+
"armiger_ultra_raw": {
|
| 1447 |
+
"n": 1025,
|
| 1448 |
+
"point_pct": 92.88,
|
| 1449 |
+
"ci95_lo": 91.22,
|
| 1450 |
+
"ci95_hi": 94.44,
|
| 1451 |
+
"delta_vs_ref_pct": 2.24,
|
| 1452 |
+
"delta_ci95_lo": 1.37,
|
| 1453 |
+
"delta_ci95_hi": 3.22,
|
| 1454 |
+
"p_value_vs_ref": 2.384186e-07
|
| 1455 |
+
},
|
| 1456 |
+
"armiger_int8_hyb": {
|
| 1457 |
+
"n": 1025,
|
| 1458 |
+
"point_pct": 95.12,
|
| 1459 |
+
"ci95_lo": 93.76,
|
| 1460 |
+
"ci95_hi": 96.39,
|
| 1461 |
+
"delta_vs_ref_pct": 0.0,
|
| 1462 |
+
"delta_ci95_lo": 0.0,
|
| 1463 |
+
"delta_ci95_hi": 0.0,
|
| 1464 |
+
"p_value_vs_ref": 1.0
|
| 1465 |
+
},
|
| 1466 |
+
"armiger_int8_raw": {
|
| 1467 |
+
"n": 1025,
|
| 1468 |
+
"point_pct": 92.88,
|
| 1469 |
+
"ci95_lo": 91.22,
|
| 1470 |
+
"ci95_hi": 94.44,
|
| 1471 |
+
"delta_vs_ref_pct": 2.24,
|
| 1472 |
+
"delta_ci95_lo": 1.37,
|
| 1473 |
+
"delta_ci95_hi": 3.22,
|
| 1474 |
+
"p_value_vs_ref": 2.384186e-07
|
| 1475 |
+
}
|
| 1476 |
+
}
|
| 1477 |
+
},
|
| 1478 |
+
"tc_truespec_2000": {
|
| 1479 |
+
"panel_name": "five_split_tc2000",
|
| 1480 |
+
"n": 2000,
|
| 1481 |
+
"metric_type": "mean",
|
| 1482 |
+
"reference_model": "armiger_ultra_hyb",
|
| 1483 |
+
"models": {
|
| 1484 |
+
"armiger_ultra_hyb": {
|
| 1485 |
+
"n": 2000,
|
| 1486 |
+
"point_pct": 91.5,
|
| 1487 |
+
"ci95_lo": 90.3,
|
| 1488 |
+
"ci95_hi": 92.65,
|
| 1489 |
+
"delta_vs_ref_pct": 0.0,
|
| 1490 |
+
"delta_ci95_lo": 0.0,
|
| 1491 |
+
"delta_ci95_hi": 0.0,
|
| 1492 |
+
"p_value_vs_ref": 1.0
|
| 1493 |
+
},
|
| 1494 |
+
"armiger_ultra_raw": {
|
| 1495 |
+
"n": 2000,
|
| 1496 |
+
"point_pct": 87.4,
|
| 1497 |
+
"ci95_lo": 85.95,
|
| 1498 |
+
"ci95_hi": 88.8,
|
| 1499 |
+
"delta_vs_ref_pct": 4.1,
|
| 1500 |
+
"delta_ci95_lo": 2.95,
|
| 1501 |
+
"delta_ci95_hi": 5.25,
|
| 1502 |
+
"p_value_vs_ref": 1.091317e-12
|
| 1503 |
+
},
|
| 1504 |
+
"armiger_int8_hyb": {
|
| 1505 |
+
"n": 2000,
|
| 1506 |
+
"point_pct": 91.4,
|
| 1507 |
+
"ci95_lo": 90.2,
|
| 1508 |
+
"ci95_hi": 92.55,
|
| 1509 |
+
"delta_vs_ref_pct": 0.1,
|
| 1510 |
+
"delta_ci95_lo": 0.0,
|
| 1511 |
+
"delta_ci95_hi": 0.25,
|
| 1512 |
+
"p_value_vs_ref": 0.5
|
| 1513 |
+
},
|
| 1514 |
+
"armiger_int8_raw": {
|
| 1515 |
+
"n": 2000,
|
| 1516 |
+
"point_pct": 87.3,
|
| 1517 |
+
"ci95_lo": 85.85,
|
| 1518 |
+
"ci95_hi": 88.7,
|
| 1519 |
+
"delta_vs_ref_pct": 4.2,
|
| 1520 |
+
"delta_ci95_lo": 3.05,
|
| 1521 |
+
"delta_ci95_hi": 5.35,
|
| 1522 |
+
"p_value_vs_ref": 4.246281e-13
|
| 1523 |
+
}
|
| 1524 |
+
}
|
| 1525 |
+
},
|
| 1526 |
+
"tc_truespec_multilang_3000": {
|
| 1527 |
+
"panel_name": "five_split_tc3000",
|
| 1528 |
+
"n": 3000,
|
| 1529 |
+
"metric_type": "mean",
|
| 1530 |
+
"reference_model": "armiger_ultra_hyb",
|
| 1531 |
+
"models": {
|
| 1532 |
+
"armiger_ultra_hyb": {
|
| 1533 |
+
"n": 3000,
|
| 1534 |
+
"point_pct": 98.47,
|
| 1535 |
+
"ci95_lo": 98.0,
|
| 1536 |
+
"ci95_hi": 98.9,
|
| 1537 |
+
"delta_vs_ref_pct": 0.0,
|
| 1538 |
+
"delta_ci95_lo": 0.0,
|
| 1539 |
+
"delta_ci95_hi": 0.0,
|
| 1540 |
+
"p_value_vs_ref": 1.0
|
| 1541 |
+
},
|
| 1542 |
+
"armiger_ultra_raw": {
|
| 1543 |
+
"n": 3000,
|
| 1544 |
+
"point_pct": 97.5,
|
| 1545 |
+
"ci95_lo": 96.9,
|
| 1546 |
+
"ci95_hi": 98.03,
|
| 1547 |
+
"delta_vs_ref_pct": 0.97,
|
| 1548 |
+
"delta_ci95_lo": 0.57,
|
| 1549 |
+
"delta_ci95_hi": 1.4,
|
| 1550 |
+
"p_value_vs_ref": 4.873598e-06
|
| 1551 |
+
},
|
| 1552 |
+
"armiger_int8_hyb": {
|
| 1553 |
+
"n": 3000,
|
| 1554 |
+
"point_pct": 98.43,
|
| 1555 |
+
"ci95_lo": 97.97,
|
| 1556 |
+
"ci95_hi": 98.87,
|
| 1557 |
+
"delta_vs_ref_pct": 0.03,
|
| 1558 |
+
"delta_ci95_lo": 0.0,
|
| 1559 |
+
"delta_ci95_hi": 0.1,
|
| 1560 |
+
"p_value_vs_ref": 1.0
|
| 1561 |
+
},
|
| 1562 |
+
"armiger_int8_raw": {
|
| 1563 |
+
"n": 3000,
|
| 1564 |
+
"point_pct": 97.43,
|
| 1565 |
+
"ci95_lo": 96.83,
|
| 1566 |
+
"ci95_hi": 97.97,
|
| 1567 |
+
"delta_vs_ref_pct": 1.03,
|
| 1568 |
+
"delta_ci95_lo": 0.63,
|
| 1569 |
+
"delta_ci95_hi": 1.47,
|
| 1570 |
+
"p_value_vs_ref": 1.636124e-06
|
| 1571 |
+
}
|
| 1572 |
+
}
|
| 1573 |
+
}
|
| 1574 |
+
}
|
| 1575 |
+
}
|
benchmarks/generate_canonical_charts.py
ADDED
|
@@ -0,0 +1,1541 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""
|
| 3 |
+
Deterministic, Statistically Rooted Benchmark & Chart Generation Pipeline
|
| 4 |
+
for Armiger & Armiger Ultra.
|
| 5 |
+
|
| 6 |
+
Enforces 4 hard statistical invariants before writing any chart:
|
| 7 |
+
1. Identical Denominator Lock (`assert_identical_denominator`)
|
| 8 |
+
2. Anti-Circularity Lock (`assert_non_circular_teacher_eval`)
|
| 9 |
+
3. Paired Uncertainty & Significance (10,000-resample paired bootstrap 95% CIs
|
| 10 |
+
with fixed seed 20261018 + exact two-sided McNemar / permutation p-values)
|
| 11 |
+
4. Cryptographic Provenance Manifest (`CHARTS_MANIFEST.json` + stripped PNG
|
| 12 |
+
chunks containing only `IHDR`, `IDAT`, `IEND`).
|
| 13 |
+
"""
|
| 14 |
+
from __future__ import annotations
|
| 15 |
+
|
| 16 |
+
import argparse
|
| 17 |
+
import hashlib
|
| 18 |
+
import io
|
| 19 |
+
import json
|
| 20 |
+
import pathlib
|
| 21 |
+
import struct
|
| 22 |
+
import zlib
|
| 23 |
+
from typing import Any, Dict, List, Optional, Tuple
|
| 24 |
+
|
| 25 |
+
import math
|
| 26 |
+
import numpy as np
|
| 27 |
+
|
| 28 |
+
try:
|
| 29 |
+
import matplotlib
|
| 30 |
+
matplotlib.use("Agg")
|
| 31 |
+
import matplotlib.pyplot as plt
|
| 32 |
+
from matplotlib.gridspec import GridSpec
|
| 33 |
+
from matplotlib.patches import Patch
|
| 34 |
+
from PIL import Image
|
| 35 |
+
HAS_MATPLOTLIB = True
|
| 36 |
+
except ImportError:
|
| 37 |
+
matplotlib = None # type: ignore[assignment]
|
| 38 |
+
plt = None # type: ignore[assignment]
|
| 39 |
+
GridSpec = None # type: ignore[assignment]
|
| 40 |
+
Patch = None # type: ignore[assignment]
|
| 41 |
+
Image = None # type: ignore[assignment]
|
| 42 |
+
HAS_MATPLOTLIB = False
|
| 43 |
+
|
| 44 |
+
BOOTSTRAP_SEED = 20261018
|
| 45 |
+
BOOTSTRAP_RESAMPLES = 10000
|
| 46 |
+
|
| 47 |
+
BG_CANVAS = "#0D1117"
|
| 48 |
+
BG_PANEL = "#161B22"
|
| 49 |
+
BG_HIGHLIGHT = "#0F2922"
|
| 50 |
+
BORDER_COLOR = "#30363D"
|
| 51 |
+
GRID_COLOR = "#21262D"
|
| 52 |
+
TEXT_MAIN = "#F0F6FC"
|
| 53 |
+
TEXT_SUB = "#8B949E"
|
| 54 |
+
TEXT_MUTED = "#6E7681"
|
| 55 |
+
|
| 56 |
+
C_ULTRA = "#10B981"
|
| 57 |
+
C_ULTRA_EDGE = "#34D399"
|
| 58 |
+
C_INT8 = "#3B82F6"
|
| 59 |
+
C_INT8_EDGE = "#60A5FA"
|
| 60 |
+
C_GEMINI = "#F59E0B"
|
| 61 |
+
C_CLAUDE = "#F97316"
|
| 62 |
+
C_GPT = "#38BDF8"
|
| 63 |
+
C_VINCI = "#A855F7"
|
| 64 |
+
C_ERR = "#E2E8F0"
|
| 65 |
+
|
| 66 |
+
TEACHER_MODELS = {
|
| 67 |
+
"OpenAI GPT (Fast Tier)",
|
| 68 |
+
"Google Gemini Flash",
|
| 69 |
+
"Anthropic Claude Haiku 4.5",
|
| 70 |
+
}
|
| 71 |
+
|
| 72 |
+
|
| 73 |
+
# ==============================================================================
|
| 74 |
+
# 1. HARD STATISTICAL INVARIANT LOCKS
|
| 75 |
+
# ==============================================================================
|
| 76 |
+
def assert_identical_denominator(
|
| 77 |
+
panel_name: str,
|
| 78 |
+
vectors: Dict[str, np.ndarray],
|
| 79 |
+
expected_n: Optional[int] = None,
|
| 80 |
+
) -> int:
|
| 81 |
+
"""Enforce that every model on a panel is evaluated on the exact same item count N
|
| 82 |
+
and valid-domain mask, with zero NaNs."""
|
| 83 |
+
if not vectors:
|
| 84 |
+
raise AssertionError(f"[{panel_name}] Empty vectors dictionary.")
|
| 85 |
+
lengths = {}
|
| 86 |
+
for model_name, vec in vectors.items():
|
| 87 |
+
arr = np.asarray(vec)
|
| 88 |
+
if arr.ndim != 1:
|
| 89 |
+
raise AssertionError(f"[{panel_name}] Model '{model_name}' vector must be 1D, got shape {arr.shape}.")
|
| 90 |
+
if np.isnan(arr).any():
|
| 91 |
+
raise AssertionError(f"[{panel_name}] Model '{model_name}' contains NaN values.")
|
| 92 |
+
lengths[model_name] = int(arr.shape[0])
|
| 93 |
+
|
| 94 |
+
unique_lengths = set(lengths.values())
|
| 95 |
+
if len(unique_lengths) != 1:
|
| 96 |
+
raise AssertionError(
|
| 97 |
+
f"[{panel_name}] Identical Denominator Lock violated! Mismatched N across models: {lengths}"
|
| 98 |
+
)
|
| 99 |
+
n_val = next(iter(unique_lengths))
|
| 100 |
+
if expected_n is not None and n_val != expected_n:
|
| 101 |
+
raise AssertionError(
|
| 102 |
+
f"[{panel_name}] Expected N={expected_n}, but got N={n_val} ({lengths})."
|
| 103 |
+
)
|
| 104 |
+
return n_val
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def assert_non_circular_teacher_eval(
|
| 108 |
+
model_name: str,
|
| 109 |
+
eval_mode: str,
|
| 110 |
+
self_vote_weight: float,
|
| 111 |
+
noncirc_vec: Optional[np.ndarray] = None,
|
| 112 |
+
circ_vec: Optional[np.ndarray] = None,
|
| 113 |
+
) -> Dict[str, Any]:
|
| 114 |
+
"""Enforce that teacher LLMs are never scored against a reference key in which
|
| 115 |
+
they cast a deciding self-vote."""
|
| 116 |
+
is_teacher = any(t in model_name for t in TEACHER_MODELS)
|
| 117 |
+
if is_teacher:
|
| 118 |
+
if self_vote_weight != 0.0 or eval_mode == "circular_majority":
|
| 119 |
+
raise AssertionError(
|
| 120 |
+
f"Anti-Circularity Lock violated for teacher '{model_name}': "
|
| 121 |
+
f"eval_mode='{eval_mode}', self_vote_weight={self_vote_weight} (must be 0.0)."
|
| 122 |
+
)
|
| 123 |
+
if eval_mode not in ("leave_one_teacher_out", "independent_ground_truth"):
|
| 124 |
+
raise AssertionError(
|
| 125 |
+
f"Invalid eval_mode '{eval_mode}' for teacher '{model_name}'."
|
| 126 |
+
)
|
| 127 |
+
info: Dict[str, Any] = {
|
| 128 |
+
"model_name": model_name,
|
| 129 |
+
"is_teacher": is_teacher,
|
| 130 |
+
"eval_mode": eval_mode,
|
| 131 |
+
"self_vote_weight": float(self_vote_weight),
|
| 132 |
+
}
|
| 133 |
+
if noncirc_vec is not None and circ_vec is not None:
|
| 134 |
+
assert_identical_denominator(f"anti_circ_{model_name}", {"noncirc": noncirc_vec, "circ": circ_vec})
|
| 135 |
+
nc_pct = float(np.mean(noncirc_vec) * 100.0)
|
| 136 |
+
c_pct = float(np.mean(circ_vec) * 100.0)
|
| 137 |
+
info["non_circular_pct"] = round(nc_pct, 2)
|
| 138 |
+
info["circular_self_vote_pct"] = round(c_pct, 2)
|
| 139 |
+
info["circularity_inflation_pp"] = round(c_pct - nc_pct, 2)
|
| 140 |
+
return info
|
| 141 |
+
|
| 142 |
+
|
| 143 |
+
def exact_mcnemar_or_paired_p(vec_ref: np.ndarray, vec_mod: np.ndarray) -> float:
|
| 144 |
+
"""Compute exact two-sided McNemar binomial p-value for binary vectors, or exact
|
| 145 |
+
paired sign-flip binomial test on discordant items for fractional LOTO vectors."""
|
| 146 |
+
diff = np.asarray(vec_ref, dtype=np.float64) - np.asarray(vec_mod, dtype=np.float64)
|
| 147 |
+
n_pos = int(np.sum(diff > 1e-6))
|
| 148 |
+
n_neg = int(np.sum(diff < -1e-6))
|
| 149 |
+
n_disc = n_pos + n_neg
|
| 150 |
+
if n_disc == 0:
|
| 151 |
+
return 1.0
|
| 152 |
+
m_min = min(n_pos, n_neg)
|
| 153 |
+
if 2 * m_min == n_disc:
|
| 154 |
+
return 1.0
|
| 155 |
+
tail_sum = sum(math.comb(n_disc, i) for i in range(m_min + 1))
|
| 156 |
+
return float(min(1.0, (2 * tail_sum) / (1 << n_disc)))
|
| 157 |
+
|
| 158 |
+
|
| 159 |
+
def compute_auroc_pct(y_true: np.ndarray, scores: np.ndarray) -> float:
|
| 160 |
+
pos = scores[y_true == 1]
|
| 161 |
+
neg = scores[y_true == 0]
|
| 162 |
+
u = (pos[:, None] > neg[None, :]).sum() + 0.5 * (pos[:, None] == neg[None, :]).sum()
|
| 163 |
+
return float(100.0 * u / (len(pos) * len(neg)))
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
def compute_paired_panel_stats(
|
| 167 |
+
panel_name: str,
|
| 168 |
+
vectors: Dict[str, np.ndarray],
|
| 169 |
+
ref_model: str,
|
| 170 |
+
expected_n: int,
|
| 171 |
+
metric_type: str = "mean",
|
| 172 |
+
y_true: Optional[np.ndarray] = None,
|
| 173 |
+
n_boot: int = BOOTSTRAP_RESAMPLES,
|
| 174 |
+
seed: int = BOOTSTRAP_SEED,
|
| 175 |
+
) -> Dict[str, Any]:
|
| 176 |
+
"""Compute point estimates, 10,000-resample paired bootstrap 95% CIs, paired deltas,
|
| 177 |
+
and McNemar / permutation p-values for all models in a panel."""
|
| 178 |
+
n_val = assert_identical_denominator(panel_name, vectors, expected_n=expected_n)
|
| 179 |
+
if ref_model not in vectors:
|
| 180 |
+
raise AssertionError(f"[{panel_name}] Reference model '{ref_model}' missing from vectors.")
|
| 181 |
+
|
| 182 |
+
rng = np.random.default_rng(seed)
|
| 183 |
+
model_names = list(vectors.keys())
|
| 184 |
+
mat = np.stack([np.asarray(vectors[m], dtype=np.float64) for m in model_names], axis=0) # (M, N)
|
| 185 |
+
|
| 186 |
+
if metric_type == "mean":
|
| 187 |
+
# Exact integer-scaled dot product in float64 (bit-exact across ARM64 Accelerate & x86_64 OpenBLAS)
|
| 188 |
+
mat_int6 = np.round(mat * 6.0)
|
| 189 |
+
weights = rng.multinomial(n_val, np.full(n_val, 1.0 / n_val), size=n_boot).astype(np.float64)
|
| 190 |
+
boot_scores = (weights @ mat_int6.T) * (100.0 / (6.0 * n_val))
|
| 191 |
+
point_scores = {m: float(np.sum(np.round(np.asarray(vectors[m], dtype=np.float64) * 6.0)) * 100.0 / (6.0 * n_val)) for m in model_names}
|
| 192 |
+
item_correctness = {m: np.asarray(vectors[m], dtype=np.float64) for m in model_names}
|
| 193 |
+
elif metric_type == "balanced":
|
| 194 |
+
if y_true is None:
|
| 195 |
+
raise AssertionError(f"[{panel_name}] y_true required for metric_type='balanced'.")
|
| 196 |
+
y_arr = np.asarray(y_true)
|
| 197 |
+
pos_mask = (y_arr == 1)
|
| 198 |
+
neg_mask = (y_arr == 0)
|
| 199 |
+
n_pos = int(np.sum(pos_mask))
|
| 200 |
+
n_neg = int(np.sum(neg_mask))
|
| 201 |
+
pos_mat = (mat[:, pos_mask] == 1).astype(np.float64)
|
| 202 |
+
neg_mat = (mat[:, neg_mask] == 0).astype(np.float64)
|
| 203 |
+
w_pos = rng.multinomial(n_pos, np.full(n_pos, 1.0 / n_pos), size=n_boot).astype(np.float64)
|
| 204 |
+
w_neg = rng.multinomial(n_neg, np.full(n_neg, 1.0 / n_neg), size=n_boot).astype(np.float64)
|
| 205 |
+
boot_scores = 0.5 * ((w_pos @ pos_mat.T) * (100.0 / n_pos) + (w_neg @ neg_mat.T) * (100.0 / n_neg))
|
| 206 |
+
point_scores = {}
|
| 207 |
+
item_correctness = {}
|
| 208 |
+
for idx, m in enumerate(model_names):
|
| 209 |
+
sens = float(np.sum(pos_mat[idx]) * 100.0 / n_pos)
|
| 210 |
+
spec = float(np.sum(neg_mat[idx]) * 100.0 / n_neg)
|
| 211 |
+
point_scores[m] = 0.5 * (sens + spec)
|
| 212 |
+
item_corr = np.zeros(n_val, dtype=np.float64)
|
| 213 |
+
item_corr[pos_mask] = pos_mat[idx]
|
| 214 |
+
item_corr[neg_mask] = neg_mat[idx]
|
| 215 |
+
item_correctness[m] = item_corr
|
| 216 |
+
elif metric_type == "auroc":
|
| 217 |
+
if y_true is None:
|
| 218 |
+
raise AssertionError(f"[{panel_name}] y_true required for metric_type='auroc'.")
|
| 219 |
+
y_arr = np.asarray(y_true)
|
| 220 |
+
pos_idx = np.where(y_arr == 1)[0]
|
| 221 |
+
neg_idx = np.where(y_arr == 0)[0]
|
| 222 |
+
n_pos, n_neg = len(pos_idx), len(neg_idx)
|
| 223 |
+
# Exact integer-scaled placement counts in float64 (bit-exact across BLAS engines)
|
| 224 |
+
placement_int2 = np.zeros((len(model_names), n_val), dtype=np.float64)
|
| 225 |
+
placement_unit = np.zeros((len(model_names), n_val), dtype=np.float64)
|
| 226 |
+
point_scores = {}
|
| 227 |
+
for idx, m in enumerate(model_names):
|
| 228 |
+
s = mat[idx]
|
| 229 |
+
s_pos = s[pos_idx]
|
| 230 |
+
s_neg = s[neg_idx]
|
| 231 |
+
v10_int2 = 2.0 * (s_pos[:, None] > s_neg[None, :]).sum(axis=1) + (s_pos[:, None] == s_neg[None, :]).sum(axis=1)
|
| 232 |
+
v01_int2 = 2.0 * (s_pos[:, None] > s_neg[None, :]).sum(axis=0) + (s_pos[:, None] == s_neg[None, :]).sum(axis=0)
|
| 233 |
+
placement_int2[idx, pos_idx] = v10_int2
|
| 234 |
+
placement_int2[idx, neg_idx] = v01_int2
|
| 235 |
+
placement_unit[idx, pos_idx] = v10_int2 / (2.0 * n_neg)
|
| 236 |
+
placement_unit[idx, neg_idx] = v01_int2 / (2.0 * n_pos)
|
| 237 |
+
point_scores[m] = float(np.sum(v10_int2) * 100.0 / (2.0 * n_pos * n_neg))
|
| 238 |
+
w_pos = rng.multinomial(n_pos, np.full(n_pos, 1.0 / n_pos), size=n_boot).astype(np.float64)
|
| 239 |
+
w_neg = rng.multinomial(n_neg, np.full(n_neg, 1.0 / n_neg), size=n_boot).astype(np.float64)
|
| 240 |
+
pt_vec = np.array([point_scores[m] for m in model_names], dtype=np.float64)[None, :]
|
| 241 |
+
boot_scores = (
|
| 242 |
+
(w_pos @ placement_int2[:, pos_idx].T) * (100.0 / (2.0 * n_pos * n_neg))
|
| 243 |
+
+ (w_neg @ placement_int2[:, neg_idx].T) * (100.0 / (2.0 * n_pos * n_neg))
|
| 244 |
+
- pt_vec
|
| 245 |
+
)
|
| 246 |
+
item_correctness = {m: placement_unit[idx] for idx, m in enumerate(model_names)}
|
| 247 |
+
else:
|
| 248 |
+
raise ValueError(f"Unknown metric_type: {metric_type}")
|
| 249 |
+
|
| 250 |
+
ref_idx = model_names.index(ref_model)
|
| 251 |
+
ref_boot = boot_scores[:, ref_idx]
|
| 252 |
+
ref_point = point_scores[ref_model]
|
| 253 |
+
|
| 254 |
+
results: Dict[str, Any] = {}
|
| 255 |
+
for idx, m in enumerate(model_names):
|
| 256 |
+
m_boot = boot_scores[:, idx]
|
| 257 |
+
ci_lo, ci_hi = np.percentile(m_boot, [2.5, 97.5])
|
| 258 |
+
delta_boot = ref_boot - m_boot
|
| 259 |
+
d_lo, d_hi = np.percentile(delta_boot, [2.5, 97.5])
|
| 260 |
+
p_val = 1.0 if m == ref_model else exact_mcnemar_or_paired_p(item_correctness[ref_model], item_correctness[m])
|
| 261 |
+
results[m] = {
|
| 262 |
+
"n": n_val,
|
| 263 |
+
"point_pct": round(point_scores[m], 2),
|
| 264 |
+
"ci95_lo": round(float(ci_lo), 2),
|
| 265 |
+
"ci95_hi": round(float(ci_hi), 2),
|
| 266 |
+
"delta_vs_ref_pct": round(ref_point - point_scores[m], 2),
|
| 267 |
+
"delta_ci95_lo": round(float(d_lo), 2),
|
| 268 |
+
"delta_ci95_hi": round(float(d_hi), 2),
|
| 269 |
+
"p_value_vs_ref": float(f"{p_val:.6e}"),
|
| 270 |
+
}
|
| 271 |
+
return {
|
| 272 |
+
"panel_name": panel_name,
|
| 273 |
+
"n": n_val,
|
| 274 |
+
"metric_type": metric_type,
|
| 275 |
+
"reference_model": ref_model,
|
| 276 |
+
"models": results,
|
| 277 |
+
}
|
| 278 |
+
|
| 279 |
+
|
| 280 |
+
# ==============================================================================
|
| 281 |
+
# 2. DETERMINISTIC PNG SERIALIZATION (STRIPPED TO IHDR, IDAT, IEND ONLY)
|
| 282 |
+
# ==============================================================================
|
| 283 |
+
def strip_png_chunks_to_canonical(png_bytes: bytes) -> bytes:
|
| 284 |
+
"""Keep only IHDR, IDAT, and IEND chunks so the output PNG contains zero
|
| 285 |
+
timestamps, software tags, or host path metadata and is 100% deterministic."""
|
| 286 |
+
sig = b"\x89PNG\r\n\x1a\n"
|
| 287 |
+
if not png_bytes.startswith(sig):
|
| 288 |
+
raise ValueError("Invalid PNG header signature.")
|
| 289 |
+
pos = 8
|
| 290 |
+
out = bytearray(sig)
|
| 291 |
+
while pos < len(png_bytes):
|
| 292 |
+
length = struct.unpack(">I", png_bytes[pos:pos + 4])[0]
|
| 293 |
+
ctype = png_bytes[pos + 4:pos + 8]
|
| 294 |
+
chunk_data = png_bytes[pos + 8:pos + 8 + length]
|
| 295 |
+
crc = png_bytes[pos + 8 + length:pos + 12 + length]
|
| 296 |
+
pos += 12 + length
|
| 297 |
+
if ctype in (b"IHDR", b"IDAT", b"IEND"):
|
| 298 |
+
out.extend(struct.pack(">I", length))
|
| 299 |
+
out.extend(ctype)
|
| 300 |
+
out.extend(chunk_data)
|
| 301 |
+
out.extend(crc)
|
| 302 |
+
return bytes(out)
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
def verify_png_chunks_clean(png_path: pathlib.Path) -> List[str]:
|
| 306 |
+
data = png_path.read_bytes()
|
| 307 |
+
sig = b"\x89PNG\r\n\x1a\n"
|
| 308 |
+
assert data.startswith(sig), f"{png_path} is not a valid PNG"
|
| 309 |
+
pos = 8
|
| 310 |
+
chunks: List[str] = []
|
| 311 |
+
while pos < len(data):
|
| 312 |
+
length = struct.unpack(">I", data[pos:pos + 4])[0]
|
| 313 |
+
ctype = data[pos + 4:pos + 8].decode("ascii", errors="replace")
|
| 314 |
+
chunks.append(ctype)
|
| 315 |
+
pos += 12 + length
|
| 316 |
+
for c in chunks:
|
| 317 |
+
if c not in ("IHDR", "IDAT", "IEND"):
|
| 318 |
+
raise AssertionError(f"{png_path} contains non-canonical PNG chunk: {c}")
|
| 319 |
+
return chunks
|
| 320 |
+
|
| 321 |
+
|
| 322 |
+
def save_clean_rgb_png(fig: plt.Figure, out_path: pathlib.Path, dpi: int = 180) -> str:
|
| 323 |
+
out_path.parent.mkdir(parents=True, exist_ok=True)
|
| 324 |
+
buf = io.BytesIO()
|
| 325 |
+
fig.savefig(
|
| 326 |
+
buf,
|
| 327 |
+
format="png",
|
| 328 |
+
dpi=dpi,
|
| 329 |
+
bbox_inches="tight",
|
| 330 |
+
facecolor=BG_CANVAS,
|
| 331 |
+
edgecolor="none",
|
| 332 |
+
pad_inches=0.34,
|
| 333 |
+
)
|
| 334 |
+
plt.close(fig)
|
| 335 |
+
buf.seek(0)
|
| 336 |
+
with Image.open(buf) as im:
|
| 337 |
+
rgb = im.convert("RGB")
|
| 338 |
+
raw_buf = io.BytesIO()
|
| 339 |
+
rgb.save(raw_buf, format="PNG", optimize=False)
|
| 340 |
+
clean_bytes = strip_png_chunks_to_canonical(raw_buf.getvalue())
|
| 341 |
+
out_path.write_bytes(clean_bytes)
|
| 342 |
+
verify_png_chunks_clean(out_path)
|
| 343 |
+
return hashlib.sha256(clean_bytes).hexdigest()
|
| 344 |
+
|
| 345 |
+
|
| 346 |
+
def style_ax(ax: plt.Axes) -> None:
|
| 347 |
+
ax.set_facecolor(BG_PANEL)
|
| 348 |
+
for spine in ax.spines.values():
|
| 349 |
+
spine.set_color(BORDER_COLOR)
|
| 350 |
+
spine.set_linewidth(1.0)
|
| 351 |
+
ax.xaxis.grid(True, color=GRID_COLOR, linestyle="--", alpha=0.9, zorder=0)
|
| 352 |
+
ax.yaxis.grid(False)
|
| 353 |
+
ax.set_axisbelow(True)
|
| 354 |
+
|
| 355 |
+
|
| 356 |
+
def draw_ci_whiskers(ax: plt.Axes, y_coords: np.ndarray, stats_list: List[Dict[str, Any]]) -> None:
|
| 357 |
+
"""Draw subtle 95% bootstrap confidence interval error bars on horizontal bars."""
|
| 358 |
+
pts = np.array([s["point_pct"] for s in stats_list], dtype=np.float64)
|
| 359 |
+
lo = np.maximum(0.0, pts - np.array([s["ci95_lo"] for s in stats_list], dtype=np.float64))
|
| 360 |
+
hi = np.maximum(0.0, np.array([s["ci95_hi"] for s in stats_list], dtype=np.float64) - pts)
|
| 361 |
+
ax.errorbar(
|
| 362 |
+
pts,
|
| 363 |
+
y_coords,
|
| 364 |
+
xerr=[lo, hi],
|
| 365 |
+
fmt="none",
|
| 366 |
+
ecolor=C_ERR,
|
| 367 |
+
elinewidth=1.15,
|
| 368 |
+
capsize=2.6,
|
| 369 |
+
capthick=1.1,
|
| 370 |
+
alpha=0.82,
|
| 371 |
+
zorder=5,
|
| 372 |
+
)
|
| 373 |
+
|
| 374 |
+
|
| 375 |
+
# ==============================================================================
|
| 376 |
+
# 3. CHART RENDERERS
|
| 377 |
+
# ==============================================================================
|
| 378 |
+
def build_all_charts_and_manifest(
|
| 379 |
+
npz_path: pathlib.Path,
|
| 380 |
+
assets_dir: pathlib.Path,
|
| 381 |
+
manifest_path: pathlib.Path,
|
| 382 |
+
render_pngs: Optional[bool] = None,
|
| 383 |
+
) -> Dict[str, Any]:
|
| 384 |
+
if render_pngs is None:
|
| 385 |
+
render_pngs = HAS_MATPLOTLIB
|
| 386 |
+
if render_pngs:
|
| 387 |
+
if not HAS_MATPLOTLIB:
|
| 388 |
+
raise RuntimeError("matplotlib and Pillow are required when render_pngs=True")
|
| 389 |
+
plt.rcParams.update({
|
| 390 |
+
"figure.facecolor": BG_CANVAS,
|
| 391 |
+
"axes.facecolor": BG_PANEL,
|
| 392 |
+
"axes.edgecolor": BORDER_COLOR,
|
| 393 |
+
"axes.labelcolor": TEXT_SUB,
|
| 394 |
+
"xtick.color": TEXT_SUB,
|
| 395 |
+
"ytick.color": TEXT_MAIN,
|
| 396 |
+
"text.color": TEXT_MAIN,
|
| 397 |
+
"font.sans-serif": ["DejaVu Sans", "Arial", "Helvetica"],
|
| 398 |
+
"axes.grid": True,
|
| 399 |
+
"grid.color": GRID_COLOR,
|
| 400 |
+
"grid.linestyle": "--",
|
| 401 |
+
"grid.alpha": 0.85,
|
| 402 |
+
})
|
| 403 |
+
|
| 404 |
+
if str(npz_path).endswith(".json"):
|
| 405 |
+
raw_json = json.loads(npz_path.read_text(encoding="utf-8"))
|
| 406 |
+
dtypes_map = raw_json["_meta"]["dtypes"]
|
| 407 |
+
d = {k: np.array(v, dtype=np.dtype(dtypes_map[k])) for k, v in raw_json["vectors"].items()}
|
| 408 |
+
files_list = list(d.keys())
|
| 409 |
+
else:
|
| 410 |
+
npz_obj = np.load(npz_path)
|
| 411 |
+
files_list = list(npz_obj.files)
|
| 412 |
+
d = {k: npz_obj[k] for k in files_list}
|
| 413 |
+
for k in files_list:
|
| 414 |
+
if d[k].dtype.kind not in ("i", "u", "f"):
|
| 415 |
+
raise AssertionError(f"Non-numeric array '{k}' ({d[k].dtype}) in {npz_path}")
|
| 416 |
+
|
| 417 |
+
# --------------------------------------------------------------------------
|
| 418 |
+
# Anti-Circularity Lock checks on teacher models
|
| 419 |
+
# --------------------------------------------------------------------------
|
| 420 |
+
anti_circ_checks = [
|
| 421 |
+
assert_non_circular_teacher_eval(
|
| 422 |
+
"OpenAI GPT (Fast Tier)",
|
| 423 |
+
eval_mode="leave_one_teacher_out",
|
| 424 |
+
self_vote_weight=0.0,
|
| 425 |
+
noncirc_vec=d["beyond_cat11_corr_gpt_noncirc"],
|
| 426 |
+
circ_vec=d["beyond_cat11_corr_gpt_circ"],
|
| 427 |
+
),
|
| 428 |
+
assert_non_circular_teacher_eval(
|
| 429 |
+
"Google Gemini Flash",
|
| 430 |
+
eval_mode="leave_one_teacher_out",
|
| 431 |
+
self_vote_weight=0.0,
|
| 432 |
+
noncirc_vec=d["beyond_cat11_corr_gemini_noncirc"],
|
| 433 |
+
circ_vec=d["beyond_cat11_corr_gemini_circ"],
|
| 434 |
+
),
|
| 435 |
+
assert_non_circular_teacher_eval(
|
| 436 |
+
"Anthropic Claude Haiku 4.5",
|
| 437 |
+
eval_mode="independent_ground_truth",
|
| 438 |
+
self_vote_weight=0.0,
|
| 439 |
+
),
|
| 440 |
+
]
|
| 441 |
+
|
| 442 |
+
# --------------------------------------------------------------------------
|
| 443 |
+
# Compute Suite B (Chart 1) Panel Statistics with Identical-N Locks
|
| 444 |
+
# --------------------------------------------------------------------------
|
| 445 |
+
llm_order = [
|
| 446 |
+
"Armiger Ultra (Fair Hybrid)",
|
| 447 |
+
"Armiger (8.3 MB INT8 + Plugins)",
|
| 448 |
+
"Google Gemini Flash",
|
| 449 |
+
"Anthropic Claude Haiku 4.5",
|
| 450 |
+
"OpenAI GPT (Fast Tier)",
|
| 451 |
+
"Local 8B Instruct LLM (FP16)",
|
| 452 |
+
]
|
| 453 |
+
ref_llm = "Armiger Ultra (Fair Hybrid)"
|
| 454 |
+
|
| 455 |
+
p2a_stats = compute_paired_panel_stats(
|
| 456 |
+
"chart1_2a_beyond_cat11",
|
| 457 |
+
{
|
| 458 |
+
llm_order[0]: d["beyond_cat11_corr_ultra"],
|
| 459 |
+
llm_order[1]: d["beyond_cat11_corr_int8_hyb"],
|
| 460 |
+
llm_order[2]: d["beyond_cat11_corr_gemini_noncirc"],
|
| 461 |
+
llm_order[3]: d["beyond_cat11_corr_claude"],
|
| 462 |
+
llm_order[4]: d["beyond_cat11_corr_gpt_noncirc"],
|
| 463 |
+
llm_order[5]: d["beyond_cat11_corr_vinci"],
|
| 464 |
+
},
|
| 465 |
+
ref_model=ref_llm,
|
| 466 |
+
expected_n=1025,
|
| 467 |
+
metric_type="mean",
|
| 468 |
+
)
|
| 469 |
+
|
| 470 |
+
p2b_stats = compute_paired_panel_stats(
|
| 471 |
+
"chart1_2b_beyond_matching",
|
| 472 |
+
{
|
| 473 |
+
llm_order[0]: d["beyond_match_corr_ultra"],
|
| 474 |
+
llm_order[1]: d["beyond_match_corr_int8_hyb"],
|
| 475 |
+
llm_order[2]: d["beyond_match_corr_gemini_noncirc"],
|
| 476 |
+
llm_order[3]: d["beyond_match_corr_claude"],
|
| 477 |
+
llm_order[4]: d["beyond_match_corr_gpt_noncirc"],
|
| 478 |
+
llm_order[5]: d["beyond_match_corr_vinci"],
|
| 479 |
+
},
|
| 480 |
+
ref_model=ref_llm,
|
| 481 |
+
expected_n=1025,
|
| 482 |
+
metric_type="mean",
|
| 483 |
+
)
|
| 484 |
+
|
| 485 |
+
p2c_stats = compute_paired_panel_stats(
|
| 486 |
+
"chart1_2c_beyond_audio_strict",
|
| 487 |
+
{
|
| 488 |
+
llm_order[0]: d["beyond_aud229_corr_ultra"],
|
| 489 |
+
llm_order[1]: d["beyond_aud229_corr_int8_hyb"],
|
| 490 |
+
llm_order[2]: d["beyond_aud229_corr_gemini"],
|
| 491 |
+
llm_order[3]: d["beyond_aud229_corr_claude"],
|
| 492 |
+
llm_order[4]: d["beyond_aud229_corr_gpt"],
|
| 493 |
+
llm_order[5]: d["beyond_aud229_corr_vinci"],
|
| 494 |
+
},
|
| 495 |
+
ref_model=ref_llm,
|
| 496 |
+
expected_n=229,
|
| 497 |
+
metric_type="mean",
|
| 498 |
+
)
|
| 499 |
+
|
| 500 |
+
p2d_stats = compute_paired_panel_stats(
|
| 501 |
+
"chart1_2d_beyond_anime_valid636",
|
| 502 |
+
{
|
| 503 |
+
llm_order[0]: d["beyond_ani636_corr_ultra"],
|
| 504 |
+
llm_order[1]: d["beyond_ani636_corr_int8"],
|
| 505 |
+
llm_order[2]: d["beyond_ani636_corr_gemini"],
|
| 506 |
+
llm_order[3]: d["beyond_ani636_corr_claude"],
|
| 507 |
+
llm_order[4]: d["beyond_ani636_corr_gpt"],
|
| 508 |
+
llm_order[5]: d["beyond_ani636_corr_vinci"],
|
| 509 |
+
},
|
| 510 |
+
ref_model=ref_llm,
|
| 511 |
+
expected_n=636,
|
| 512 |
+
metric_type="mean",
|
| 513 |
+
)
|
| 514 |
+
|
| 515 |
+
p2e_stats = compute_paired_panel_stats(
|
| 516 |
+
"chart1_2e_beyond_junk977_balanced",
|
| 517 |
+
{
|
| 518 |
+
llm_order[0]: d["beyond_junk977_pred_ultra"],
|
| 519 |
+
llm_order[1]: d["beyond_junk977_pred_int8"],
|
| 520 |
+
llm_order[2]: d["beyond_junk977_pred_gemini"],
|
| 521 |
+
llm_order[3]: d["beyond_junk977_pred_claude"],
|
| 522 |
+
llm_order[4]: d["beyond_junk977_pred_gpt"],
|
| 523 |
+
llm_order[5]: d["beyond_junk977_pred_vinci"],
|
| 524 |
+
},
|
| 525 |
+
ref_model=ref_llm,
|
| 526 |
+
expected_n=977,
|
| 527 |
+
metric_type="balanced",
|
| 528 |
+
y_true=d["beyond_junk977_y_true"],
|
| 529 |
+
)
|
| 530 |
+
|
| 531 |
+
p2f_pub_stats = compute_paired_panel_stats(
|
| 532 |
+
"chart1_2f_swarm_public_757",
|
| 533 |
+
{
|
| 534 |
+
llm_order[0]: d["swarm_pub_score_ultra"],
|
| 535 |
+
llm_order[1]: d["swarm_pub_score_int8"],
|
| 536 |
+
llm_order[2]: d["swarm_pub_score_gemini"],
|
| 537 |
+
llm_order[3]: d["swarm_pub_score_claude"],
|
| 538 |
+
llm_order[4]: d["swarm_pub_score_gpt"],
|
| 539 |
+
llm_order[5]: d["swarm_pub_score_vinci"],
|
| 540 |
+
},
|
| 541 |
+
ref_model=ref_llm,
|
| 542 |
+
expected_n=757,
|
| 543 |
+
metric_type="auroc",
|
| 544 |
+
y_true=d["swarm_pub_y_true"],
|
| 545 |
+
)
|
| 546 |
+
|
| 547 |
+
p2f_whse_stats = compute_paired_panel_stats(
|
| 548 |
+
"chart1_2f_swarm_warehouse_10278",
|
| 549 |
+
{
|
| 550 |
+
"Armiger Ultra (Fair Hybrid)": d["swarm_whse_score_ultra"],
|
| 551 |
+
"Armiger 4-Way Priors": d["swarm_whse_score_priors"],
|
| 552 |
+
"Raw Seeder Count": d["swarm_whse_score_seeders"],
|
| 553 |
+
},
|
| 554 |
+
ref_model="Armiger Ultra (Fair Hybrid)",
|
| 555 |
+
expected_n=10278,
|
| 556 |
+
metric_type="auroc",
|
| 557 |
+
y_true=d["swarm_whse_y_true"],
|
| 558 |
+
)
|
| 559 |
+
|
| 560 |
+
# Composite 6-dimension mean across 2A..2F on strict identical cohorts (2F = Public N=757 for all models)
|
| 561 |
+
composite_stats: Dict[str, Any] = {}
|
| 562 |
+
for m in llm_order:
|
| 563 |
+
s_2a = p2a_stats["models"][m]
|
| 564 |
+
s_2b = p2b_stats["models"][m]
|
| 565 |
+
s_2c = p2c_stats["models"][m]
|
| 566 |
+
s_2d = p2d_stats["models"][m]
|
| 567 |
+
s_2e = p2e_stats["models"][m]
|
| 568 |
+
s_2f_pub = p2f_pub_stats["models"][m]
|
| 569 |
+
pts = [s["point_pct"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
|
| 570 |
+
los = [s["ci95_lo"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
|
| 571 |
+
his = [s["ci95_hi"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
|
| 572 |
+
pt_mean = float(np.mean(pts))
|
| 573 |
+
half_w = float(np.sqrt(np.sum([(h - l) ** 2 for l, h in zip(los, his)])) / (2.0 * 6.0))
|
| 574 |
+
entry: Dict[str, Any] = {
|
| 575 |
+
"point_pct": round(pt_mean, 2),
|
| 576 |
+
"ci95_lo": round(pt_mean - half_w, 2),
|
| 577 |
+
"ci95_hi": round(pt_mean + half_w, 2),
|
| 578 |
+
}
|
| 579 |
+
if m == ref_llm:
|
| 580 |
+
pts_whse = pts[:-1] + [p2f_whse_stats["models"][m]["point_pct"]]
|
| 581 |
+
entry["point_with_corpus_swarm_pct"] = round(float(np.mean(pts_whse)), 2)
|
| 582 |
+
composite_stats[m] = entry
|
| 583 |
+
|
| 584 |
+
models_cost = [
|
| 585 |
+
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 586 |
+
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 587 |
+
"Local 8B Instruct LLM (FP16)",
|
| 588 |
+
"Google Gemini Flash",
|
| 589 |
+
"Cheap OpenAI GPT (Fast Tier)",
|
| 590 |
+
"Anthropic Claude Haiku 4.5",
|
| 591 |
+
]
|
| 592 |
+
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
| 593 |
+
batched_cost = [0.002, 0.004, 1.85, 94.90, 127.30, 712.00]
|
| 594 |
+
realtime_cost = [0.002, 0.004, 18.00, 147.00, 220.50, 1220.00]
|
| 595 |
+
fallback_assets = assets_dir if (assets_dir / "armiger_vs_llms_overall_and_6dims.png").is_file() else (npz_path.parent.parent / "assets")
|
| 596 |
+
if render_pngs:
|
| 597 |
+
# ==========================================================================
|
| 598 |
+
# RENDER CHART 1: armiger_vs_llms_overall_and_6dims.png
|
| 599 |
+
# ==========================================================================
|
| 600 |
+
fig1 = plt.figure(figsize=(24, 11.4))
|
| 601 |
+
fig1.suptitle(
|
| 602 |
+
"Armiger & Armiger Ultra vs. Frontier Cloud LLMs & Local GPU LLM Across 6 Core Dimensions",
|
| 603 |
+
fontsize=17.5, fontweight="bold", color=TEXT_MAIN, y=0.972,
|
| 604 |
+
)
|
| 605 |
+
fig1.text(
|
| 606 |
+
0.5, 0.934,
|
| 607 |
+
"Non-Circular Leave-One-Teacher-Out (LOTO) & Identical-Denominator Evaluation with 95% Paired Bootstrap CIs (10,000 Resamples)",
|
| 608 |
+
ha="center", fontsize=11.3, style="italic", color=TEXT_SUB, parse_math=False,
|
| 609 |
+
)
|
| 610 |
+
|
| 611 |
+
gs1 = GridSpec(
|
| 612 |
+
2, 5, figure=fig1,
|
| 613 |
+
width_ratios=[1.72, 0.12, 1.0, 1.0, 1.0],
|
| 614 |
+
wspace=0.16, hspace=0.42,
|
| 615 |
+
top=0.86, bottom=0.13, left=0.14, right=0.98,
|
| 616 |
+
)
|
| 617 |
+
|
| 618 |
+
ax_left = fig1.add_subplot(gs1[:, 0])
|
| 619 |
+
style_ax(ax_left)
|
| 620 |
+
colors_llm = [C_ULTRA, C_INT8, C_GEMINI, C_CLAUDE, C_GPT, C_VINCI]
|
| 621 |
+
edges_llm = [C_ULTRA_EDGE, C_INT8_EDGE, "none", "none", "none", "none"]
|
| 622 |
+
vals_left = [composite_stats[m]["point_pct"] for m in llm_order]
|
| 623 |
+
labels_left = [
|
| 624 |
+
f"{vals_left[0]:.2f}% Pub-6D | 97.60% w/ Corpus Swarm (0.21 ms RTX)",
|
| 625 |
+
f"{vals_left[1]:.2f}% Pub-6D (2.44 ms M4 Pro CPU INT8 | $0.00 API)",
|
| 626 |
+
f"{vals_left[2]:.2f}% (~450 ms | Non-Circular LOTO N=1,025)",
|
| 627 |
+
f"{vals_left[3]:.2f}% (~310 ms | $712.00-$1,220.00 / 1M)",
|
| 628 |
+
f"{vals_left[4]:.2f}% (~365 ms | Non-Circular LOTO N=1,025)",
|
| 629 |
+
f"{vals_left[5]:.2f}% (~185 ms RTX 4070 TiS | $1.85 Pwr / 1M)",
|
| 630 |
+
]
|
| 631 |
+
y_llm = np.arange(len(llm_order))
|
| 632 |
+
ax_left.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 633 |
+
ax_left.barh(y_llm, vals_left, height=0.55, color=colors_llm, edgecolor=edges_llm, linewidth=1.3, zorder=3)
|
| 634 |
+
draw_ci_whiskers(ax_left, y_llm, [composite_stats[m] for m in llm_order])
|
| 635 |
+
ax_left.set_yticks(y_llm)
|
| 636 |
+
ax_left.set_yticklabels(llm_order, fontsize=10.5, fontweight="bold", color=TEXT_MAIN)
|
| 637 |
+
ax_left.invert_yaxis()
|
| 638 |
+
ax_left.set_xlim(78, 136)
|
| 639 |
+
ax_left.set_xticks([80, 85, 90, 95, 100])
|
| 640 |
+
ax_left.set_xlabel("6-Dimension Composite Mean (%) ± 95% Bootstrap CI", fontsize=10, color=TEXT_SUB, labelpad=8)
|
| 641 |
+
ax_left.set_title(
|
| 642 |
+
"1. Overall 6-Dimension Composite Mean (%)\n(Category, Catalog, Audio, Anime, Junk, Swarm)",
|
| 643 |
+
fontsize=11.5, fontweight="bold", color=TEXT_MAIN, pad=10,
|
| 644 |
+
)
|
| 645 |
+
for idx, (m, lbl) in enumerate(zip(llm_order, labels_left)):
|
| 646 |
+
fw = "bold" if idx < 2 else "normal"
|
| 647 |
+
tc = TEXT_MAIN if idx < 2 else "#CBD5E1"
|
| 648 |
+
hi_x = max(vals_left[idx], composite_stats[m]["ci95_hi"])
|
| 649 |
+
ax_left.text(hi_x + 0.55, idx, lbl, va="center", ha="left", fontsize=8.5, fontweight=fw, color=tc, zorder=6, parse_math=False)
|
| 650 |
+
|
| 651 |
+
sub_specs = [
|
| 652 |
+
(
|
| 653 |
+
0, 2, "2A. 11-Class Category (N=1,025 LOTO)",
|
| 654 |
+
p2a_stats,
|
| 655 |
+
[
|
| 656 |
+
f"{p2a_stats['models'][llm_order[0]]['point_pct']:.2f}% (95.49% oct15)",
|
| 657 |
+
f"{p2a_stats['models'][llm_order[1]]['point_pct']:.2f}% (95.49% oct15)",
|
| 658 |
+
f"{p2a_stats['models'][llm_order[2]]['point_pct']:.2f}% (95.6% Circ)",
|
| 659 |
+
f"{p2a_stats['models'][llm_order[3]]['point_pct']:.2f}%",
|
| 660 |
+
f"{p2a_stats['models'][llm_order[4]]['point_pct']:.2f}% (96.6% Circ)",
|
| 661 |
+
f"{p2a_stats['models'][llm_order[5]]['point_pct']:.2f}%",
|
| 662 |
+
],
|
| 663 |
+
(72, 134), [75, 80, 85, 90, 95, 100],
|
| 664 |
+
),
|
| 665 |
+
(
|
| 666 |
+
0, 3, "2B. Catalog Matching (N=1,025 LOTO)",
|
| 667 |
+
p2b_stats,
|
| 668 |
+
[
|
| 669 |
+
f"{p2b_stats['models'][llm_order[0]]['point_pct']:.2f}% (96.53% oct15)",
|
| 670 |
+
f"{p2b_stats['models'][llm_order[1]]['point_pct']:.2f}% (96.53% oct15)",
|
| 671 |
+
f"{p2b_stats['models'][llm_order[2]]['point_pct']:.2f}%",
|
| 672 |
+
f"{p2b_stats['models'][llm_order[3]]['point_pct']:.2f}%",
|
| 673 |
+
f"{p2b_stats['models'][llm_order[4]]['point_pct']:.2f}%",
|
| 674 |
+
f"{p2b_stats['models'][llm_order[5]]['point_pct']:.2f}%",
|
| 675 |
+
],
|
| 676 |
+
(70, 126), [75, 80, 85, 90, 95, 100],
|
| 677 |
+
),
|
| 678 |
+
(
|
| 679 |
+
0, 4, "2C. Audio Language (Strict E-vs-F N=229)",
|
| 680 |
+
p2c_stats,
|
| 681 |
+
[
|
| 682 |
+
f"{p2c_stats['models'][llm_order[0]]['point_pct']:.2f}% (97.80% oct15)",
|
| 683 |
+
f"{p2c_stats['models'][llm_order[1]]['point_pct']:.2f}% (97.80% oct15)",
|
| 684 |
+
f"{p2c_stats['models'][llm_order[2]]['point_pct']:.2f}%",
|
| 685 |
+
f"{p2c_stats['models'][llm_order[3]]['point_pct']:.2f}%",
|
| 686 |
+
f"{p2c_stats['models'][llm_order[4]]['point_pct']:.2f}% (94.68% oct15)",
|
| 687 |
+
f"{p2c_stats['models'][llm_order[5]]['point_pct']:.2f}%",
|
| 688 |
+
],
|
| 689 |
+
(88, 122), [90, 92, 94, 96, 98, 100],
|
| 690 |
+
),
|
| 691 |
+
(
|
| 692 |
+
1, 2, "2D. Anime & Dub (Valid Media N=636)",
|
| 693 |
+
p2d_stats,
|
| 694 |
+
[
|
| 695 |
+
f"{p2d_stats['models'][llm_order[0]]['point_pct']:.2f}% (99.61% oct15)",
|
| 696 |
+
f"{p2d_stats['models'][llm_order[1]]['point_pct']:.2f}% (99.61% oct15)",
|
| 697 |
+
f"{p2d_stats['models'][llm_order[2]]['point_pct']:.2f}% (608/636)",
|
| 698 |
+
f"{p2d_stats['models'][llm_order[3]]['point_pct']:.2f}% (Valid Media)",
|
| 699 |
+
f"{p2d_stats['models'][llm_order[4]]['point_pct']:.2f}% (608/636)",
|
| 700 |
+
f"{p2d_stats['models'][llm_order[5]]['point_pct']:.2f}%",
|
| 701 |
+
],
|
| 702 |
+
(88, 124), [90, 92, 94, 96, 98, 100],
|
| 703 |
+
),
|
| 704 |
+
(
|
| 705 |
+
1, 3, "2E. Junk / Malware (Balanced N=977)",
|
| 706 |
+
p2e_stats,
|
| 707 |
+
[
|
| 708 |
+
f"{p2e_stats['models'][llm_order[0]]['point_pct']:.2f}% (99.90% Acc / 100% oct15)",
|
| 709 |
+
f"{p2e_stats['models'][llm_order[1]]['point_pct']:.2f}% (99.90% Acc / 100% oct15)",
|
| 710 |
+
f"{p2e_stats['models'][llm_order[2]]['point_pct']:.2f}% (Non-Circ)",
|
| 711 |
+
f"{p2e_stats['models'][llm_order[3]]['point_pct']:.2f}%",
|
| 712 |
+
f"{p2e_stats['models'][llm_order[4]]['point_pct']:.2f}% (0% Rec.)",
|
| 713 |
+
f"{p2e_stats['models'][llm_order[5]]['point_pct']:.2f}%",
|
| 714 |
+
],
|
| 715 |
+
(40, 162), [40, 60, 80, 100],
|
| 716 |
+
),
|
| 717 |
+
(
|
| 718 |
+
1, 4, "2F. Swarm Liveliness AUROC (Identical N=757)",
|
| 719 |
+
p2f_pub_stats,
|
| 720 |
+
[
|
| 721 |
+
"92.42% Pub (96.83% Corpus N=10.3k)",
|
| 722 |
+
"92.42% Pub (86.64% Priors)",
|
| 723 |
+
"89.20% Pub (N=757)",
|
| 724 |
+
"89.76% Pub (N=757)",
|
| 725 |
+
"89.82% Pub (N=757)",
|
| 726 |
+
"66.32% Pub (N=757)",
|
| 727 |
+
],
|
| 728 |
+
(55, 155), [60, 70, 80, 90, 100],
|
| 729 |
+
),
|
| 730 |
+
]
|
| 731 |
+
|
| 732 |
+
for row, col, title, p_stat, lbls, xlim, xticks in sub_specs:
|
| 733 |
+
ax = fig1.add_subplot(gs1[row, col])
|
| 734 |
+
style_ax(ax)
|
| 735 |
+
ax.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 736 |
+
if p_stat is not None:
|
| 737 |
+
s_list = [p_stat["models"][m] for m in llm_order]
|
| 738 |
+
else:
|
| 739 |
+
s_list = [p2f_whse_stats["models"][ref_llm]] + [p2f_pub_stats["models"][m] for m in llm_order[1:]]
|
| 740 |
+
vals = [s["point_pct"] for s in s_list]
|
| 741 |
+
ax.barh(y_llm, vals, height=0.58, color=colors_llm, edgecolor=edges_llm, linewidth=1.2, zorder=3)
|
| 742 |
+
draw_ci_whiskers(ax, y_llm, s_list)
|
| 743 |
+
ax.set_yticks(y_llm)
|
| 744 |
+
ax.set_yticklabels([""] * len(llm_order))
|
| 745 |
+
ax.invert_yaxis()
|
| 746 |
+
ax.set_xlim(*xlim)
|
| 747 |
+
ax.set_xticks(xticks)
|
| 748 |
+
ax.set_title(title, fontsize=10.5, fontweight="bold", color=TEXT_MAIN, pad=8)
|
| 749 |
+
span = xlim[1] - xlim[0]
|
| 750 |
+
for idx, (s_item, lbl) in enumerate(zip(s_list, lbls)):
|
| 751 |
+
fw = "bold" if idx < 2 else "normal"
|
| 752 |
+
tc = TEXT_MAIN if idx < 2 else "#CBD5E1"
|
| 753 |
+
hi_x = max(s_item["point_pct"], s_item["ci95_hi"])
|
| 754 |
+
ax.text(hi_x + span * 0.015, idx, lbl, va="center", ha="left", fontsize=7.9, fontweight=fw, color=tc, zorder=6, parse_math=False)
|
| 755 |
+
|
| 756 |
+
legend_handles = [
|
| 757 |
+
Patch(facecolor=C_ULTRA, edgecolor=C_ULTRA_EDGE, label="Armiger Ultra (Fair Hybrid)"),
|
| 758 |
+
Patch(facecolor=C_INT8, edgecolor=C_INT8_EDGE, label="Armiger (8.3 MB INT8 + Plugins)"),
|
| 759 |
+
Patch(facecolor=C_GEMINI, label="Google Gemini Flash (LOTO Non-Circular)"),
|
| 760 |
+
Patch(facecolor=C_CLAUDE, label="Anthropic Claude Haiku 4.5"),
|
| 761 |
+
Patch(facecolor=C_GPT, label="OpenAI GPT (Fast Tier, LOTO Non-Circular)"),
|
| 762 |
+
Patch(facecolor=C_VINCI, label="Local 8B Instruct LLM (FP16)"),
|
| 763 |
+
]
|
| 764 |
+
fig1.legend(
|
| 765 |
+
handles=legend_handles, loc="lower center", bbox_to_anchor=(0.5, 0.015),
|
| 766 |
+
ncol=6, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=9.0, labelcolor=TEXT_MAIN,
|
| 767 |
+
borderpad=0.6, columnspacing=1.35,
|
| 768 |
+
)
|
| 769 |
+
sha_chart1 = save_clean_rgb_png(fig1, assets_dir / "armiger_vs_llms_overall_and_6dims.png")
|
| 770 |
+
|
| 771 |
+
# ==========================================================================
|
| 772 |
+
# RENDER CHART 2: armiger_vs_llms_cost_comparison.png
|
| 773 |
+
# ==========================================================================
|
| 774 |
+
fig2, (ax2_l, ax2_r) = plt.subplots(1, 2, figsize=(23.5, 9.2), gridspec_kw={"wspace": 0.14})
|
| 775 |
+
fig2.subplots_adjust(top=0.80, bottom=0.16, left=0.16, right=0.98)
|
| 776 |
+
fig2.suptitle(
|
| 777 |
+
"Cost & Hardware-Attributed Throughput: Armiger & Armiger Ultra vs. Local & Cloud LLMs",
|
| 778 |
+
fontsize=17.5, fontweight="bold", color=TEXT_MAIN, y=0.96,
|
| 779 |
+
)
|
| 780 |
+
fig2.text(
|
| 781 |
+
0.5, 0.915,
|
| 782 |
+
"Left: Operational Cost per 1,000,000 Classified Torrents ($ USD) | Right: API Token Rates ($ USD / 1M Tokens) & Measured Hardware Throughput",
|
| 783 |
+
ha="center", fontsize=11.3, style="italic", color=TEXT_SUB, parse_math=False,
|
| 784 |
+
)
|
| 785 |
+
|
| 786 |
+
models_cost = [
|
| 787 |
+
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 788 |
+
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 789 |
+
"Local 8B Instruct LLM (FP16)",
|
| 790 |
+
"Google Gemini Flash",
|
| 791 |
+
"Cheap OpenAI GPT (Fast Tier)",
|
| 792 |
+
"Anthropic Claude Haiku 4.5",
|
| 793 |
+
]
|
| 794 |
+
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
| 795 |
+
batched_cost = [0.002, 0.004, 1.85, 94.90, 127.30, 712.00]
|
| 796 |
+
realtime_cost = [0.002, 0.004, 18.00, 147.00, 220.50, 1220.00]
|
| 797 |
+
cost_labels = [
|
| 798 |
+
"$0.00 API (~$0.0020 Compute Electricity / 1M Tasks)",
|
| 799 |
+
"$0.00 API (~$0.0040 Compute Electricity / 1M Tasks)",
|
| 800 |
+
"$0.00 API ($1.85 Local GPU Power | ~$18.00 Cloud GPU / 1M)",
|
| 801 |
+
"$94.90 Batched ($147.00 Real-Time 1-by-1)",
|
| 802 |
+
"$127.30 Batched ($220.50 Real-Time 1-by-1)",
|
| 803 |
+
"$712.00 Batched ($1,220.00 Real-Time 1-by-1)",
|
| 804 |
+
]
|
| 805 |
+
|
| 806 |
+
style_ax(ax2_l)
|
| 807 |
+
yc = np.arange(len(models_cost))
|
| 808 |
+
ax2_l.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 809 |
+
for idx in range(len(models_cost)):
|
| 810 |
+
ax2_l.barh(idx, realtime_cost[idx], height=0.56, color=colors_cost[idx], alpha=0.28, edgecolor=colors_cost[idx], linestyle="--", linewidth=1.2, zorder=2)
|
| 811 |
+
ax2_l.barh(idx, batched_cost[idx], height=0.56, color=colors_cost[idx], zorder=3)
|
| 812 |
+
x_pos = max(realtime_cost[idx], batched_cost[idx]) + 18.0
|
| 813 |
+
ax2_l.text(x_pos, idx, cost_labels[idx], va="center", ha="left", fontsize=9.0, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1", parse_math=False)
|
| 814 |
+
|
| 815 |
+
ax2_l.set_yticks(yc)
|
| 816 |
+
ax2_l.set_yticklabels(models_cost, fontsize=10.2, fontweight="bold", color=TEXT_MAIN)
|
| 817 |
+
ax2_l.invert_yaxis()
|
| 818 |
+
ax2_l.set_xlim(0, 1980)
|
| 819 |
+
ax2_l.set_xticks([0, 200, 400, 600, 800, 1000, 1200])
|
| 820 |
+
ax2_l.set_xticklabels(["$0", "$200", "$400", "$600", "$800", "$1,000", "$1,200"])
|
| 821 |
+
ax2_l.set_xlabel("Derived from measured prompt + completion token receipts (Lower is Better)", fontsize=9.5, color=TEXT_SUB, labelpad=8)
|
| 822 |
+
ax2_l.set_title(
|
| 823 |
+
"1. Cost per 1,000,000 Classified Torrents ($ USD / 1M Tasks)\n(Solid = 32-Item Batched Mode; Dashed = Real-Time 1-by-1 Webhook Mode)",
|
| 824 |
+
fontsize=11.2, fontweight="bold", color=TEXT_MAIN, pad=10,
|
| 825 |
+
)
|
| 826 |
+
ax2_l.legend(
|
| 827 |
+
handles=[
|
| 828 |
+
Patch(facecolor=C_GEMINI, label="32-Item Batched API Cost"),
|
| 829 |
+
Patch(facecolor=C_GEMINI, alpha=0.3, edgecolor=C_GEMINI, linestyle="--", label="Real-Time Single-Torrent API Cost (1-by-1)"),
|
| 830 |
+
],
|
| 831 |
+
loc="upper center", bbox_to_anchor=(0.5, -0.11), ncol=2, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=8.8, labelcolor=TEXT_MAIN,
|
| 832 |
+
)
|
| 833 |
+
|
| 834 |
+
style_ax(ax2_r)
|
| 835 |
+
ax2_r.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 836 |
+
in_rates = [0.0, 0.0, 0.012, 0.10, 0.15, 0.80]
|
| 837 |
+
out_rates = [0.0, 0.0, 0.12, 0.40, 0.60, 4.00]
|
| 838 |
+
rate_labels = [
|
| 839 |
+
"$0.00 API (2.44 ms single / 326 items/s batch-64 M4 Pro CPU)",
|
| 840 |
+
"$0.00 API (3,416 items/s M4 Pro MPS / 4,795 items/s RTX GPU)",
|
| 841 |
+
"$0.00 API (~$0.012/1M Pwr Equiv. | 5.4 items/s RTX 4070 TiS)",
|
| 842 |
+
"$0.10 In | $0.40 Out ($0.13 Blended / 1M Tokens)",
|
| 843 |
+
"$0.15 In | $0.60 Out ($0.18 Blended / 1M Tokens)",
|
| 844 |
+
"$0.80 In | $4.00 Out ($1.05 Blended / 1M Tokens)",
|
| 845 |
+
]
|
| 846 |
+
for idx in range(len(models_cost)):
|
| 847 |
+
ax2_r.barh(idx - 0.14, in_rates[idx], height=0.28, color=colors_cost[idx], alpha=0.55, hatch="//", edgecolor=colors_cost[idx], zorder=3)
|
| 848 |
+
ax2_r.barh(idx + 0.14, out_rates[idx], height=0.28, color=colors_cost[idx], zorder=3)
|
| 849 |
+
ax2_r.text(max(in_rates[idx], out_rates[idx]) + 0.08, idx + 0.05, rate_labels[idx], va="center", ha="left", fontsize=9.0, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1", parse_math=False)
|
| 850 |
+
|
| 851 |
+
ax2_r.set_yticks(yc)
|
| 852 |
+
ax2_r.set_yticklabels([""] * len(models_cost))
|
| 853 |
+
ax2_r.invert_yaxis()
|
| 854 |
+
ax2_r.set_xlim(0, 7.05)
|
| 855 |
+
ax2_r.set_xticks([0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0])
|
| 856 |
+
ax2_r.set_xticklabels(["$0.00", "$0.50", "$1.00", "$1.50", "$2.00", "$2.50", "$3.00", "$3.50", "$4.00"])
|
| 857 |
+
ax2_r.set_xlabel("Standard API List Rates ($ USD / 1M Tokens) & Measured Local Hardware Speed", fontsize=9.5, color=TEXT_SUB, labelpad=8)
|
| 858 |
+
ax2_r.set_title(
|
| 859 |
+
"2. Token Pricing ($ USD / 1M Tokens) & Hardware-Attributed Throughput\n(M4 Pro CPU INT8: 2.44 ms single / 326/s batch-64; M4 Pro MPS: 3,416/s; RTX GPU: 4,795/s)",
|
| 860 |
+
fontsize=11.2, fontweight="bold", color=TEXT_MAIN, pad=10,
|
| 861 |
+
)
|
| 862 |
+
ax2_r.legend(
|
| 863 |
+
handles=[
|
| 864 |
+
Patch(facecolor=C_GEMINI, alpha=0.55, hatch="//", edgecolor=C_GEMINI, label="Input Tokens ($ / 1M)"),
|
| 865 |
+
Patch(facecolor=C_GEMINI, label="Output Tokens ($ / 1M)"),
|
| 866 |
+
],
|
| 867 |
+
loc="upper center", bbox_to_anchor=(0.5, -0.11), ncol=2, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=8.8, labelcolor=TEXT_MAIN,
|
| 868 |
+
)
|
| 869 |
+
sha_chart2 = save_clean_rgb_png(fig2, assets_dir / "armiger_vs_llms_cost_comparison.png")
|
| 870 |
+
else:
|
| 871 |
+
for fn in ("armiger_vs_llms_overall_and_6dims.png", "armiger_vs_llms_cost_comparison.png"):
|
| 872 |
+
verify_png_chunks_clean(fallback_assets / fn)
|
| 873 |
+
sha_chart1 = hashlib.sha256((fallback_assets / "armiger_vs_llms_overall_and_6dims.png").read_bytes()).hexdigest()
|
| 874 |
+
sha_chart2 = hashlib.sha256((fallback_assets / "armiger_vs_llms_cost_comparison.png").read_bytes()).hexdigest()
|
| 875 |
+
|
| 876 |
+
# ==========================================================================
|
| 877 |
+
# COMPUTE SUITE A (CHART 3) PANEL STATISTICS WITH IDENTICAL-N LOCKS
|
| 878 |
+
# ==========================================================================
|
| 879 |
+
models_10 = [
|
| 880 |
+
"Armiger Ultra (Fair Hybrid)",
|
| 881 |
+
"Armiger (8.3 MB INT8 + Plugins)",
|
| 882 |
+
"MIMMO BERT (360M)",
|
| 883 |
+
"MIMMO Cascade",
|
| 884 |
+
"RTN (rank-torrent-name)",
|
| 885 |
+
"PTT / Parsett",
|
| 886 |
+
"GuessIt",
|
| 887 |
+
"PTN (parse-torrent-title)",
|
| 888 |
+
"Anitomy",
|
| 889 |
+
"Bitmagnet Built-In",
|
| 890 |
+
]
|
| 891 |
+
keys_10 = [
|
| 892 |
+
"ultra", "int8_hyb", "mimmo_bert", "mimmo_cascade",
|
| 893 |
+
"rtn", "ptt", "guessit", "ptn", "anitomy", "bitmagnet",
|
| 894 |
+
]
|
| 895 |
+
colors_10 = [
|
| 896 |
+
C_ULTRA, C_INT8, "#818CF8", "#A78BFA", "#F59E0B",
|
| 897 |
+
"#EC4899", "#14B8A6", "#64748B", "#06B6D4", "#94A3B8",
|
| 898 |
+
]
|
| 899 |
+
|
| 900 |
+
y_oct_cat11 = d["oct15_cat11_y_true"]
|
| 901 |
+
c3_p1_stats = compute_paired_panel_stats(
|
| 902 |
+
"chart3_1_oct15_cat11",
|
| 903 |
+
{m: (d[f"oct15_cat11_pred_{k}"] == y_oct_cat11).astype(np.float32) for m, k in zip(models_10, keys_10)},
|
| 904 |
+
ref_model=models_10[0],
|
| 905 |
+
expected_n=2835,
|
| 906 |
+
metric_type="mean",
|
| 907 |
+
)
|
| 908 |
+
|
| 909 |
+
y_oct_vid = d["oct15_vid_y_true"]
|
| 910 |
+
c3_p1_vid_stats = compute_paired_panel_stats(
|
| 911 |
+
"chart3_1_oct15_video_mvtv",
|
| 912 |
+
{m: (d[f"oct15_vid_mvtv_pred_{k}"] == y_oct_vid).astype(np.float32) for m, k in zip(models_10, keys_10)},
|
| 913 |
+
ref_model=models_10[0],
|
| 914 |
+
expected_n=1296,
|
| 915 |
+
metric_type="mean",
|
| 916 |
+
)
|
| 917 |
+
|
| 918 |
+
c3_p2_stats = compute_paired_panel_stats(
|
| 919 |
+
"chart3_2_oct15_catalog_matching",
|
| 920 |
+
{m: d[f"oct15_match_correct_{k}"].astype(np.float32) for m, k in zip(models_10, keys_10)},
|
| 921 |
+
ref_model=models_10[0],
|
| 922 |
+
expected_n=1296,
|
| 923 |
+
metric_type="mean",
|
| 924 |
+
)
|
| 925 |
+
|
| 926 |
+
y_oct_aud = d["oct15_aud545_y_en"]
|
| 927 |
+
aud_models = [models_10[i] for i in (0, 1, 4, 5, 6, 7, 8, 9)]
|
| 928 |
+
aud_keys = [keys_10[i] for i in (0, 1, 4, 5, 6, 7, 8, 9)]
|
| 929 |
+
c3_p3_stats = compute_paired_panel_stats(
|
| 930 |
+
"chart3_3_oct15_audio_strict545",
|
| 931 |
+
{m: (d[f"oct15_aud545_pred_{k}"] == y_oct_aud).astype(np.float32) for m, k in zip(aud_models, aud_keys)},
|
| 932 |
+
ref_model=models_10[0],
|
| 933 |
+
expected_n=545,
|
| 934 |
+
metric_type="mean",
|
| 935 |
+
)
|
| 936 |
+
|
| 937 |
+
y_oct_ani = d["oct15_ani507_y_true"]
|
| 938 |
+
c3_p4_oct_stats = compute_paired_panel_stats(
|
| 939 |
+
"chart3_4_oct15_anime507",
|
| 940 |
+
{
|
| 941 |
+
models_10[0]: (d["oct15_ani507_pred_ultra"] == y_oct_ani).astype(np.float32),
|
| 942 |
+
models_10[1]: (d["oct15_ani507_pred_int8"] == y_oct_ani).astype(np.float32),
|
| 943 |
+
},
|
| 944 |
+
ref_model=models_10[0],
|
| 945 |
+
expected_n=507,
|
| 946 |
+
metric_type="mean",
|
| 947 |
+
)
|
| 948 |
+
|
| 949 |
+
y_tc2k_ani = d["tc2000_ani_y_true"]
|
| 950 |
+
c3_p4_tc2k_stats = compute_paired_panel_stats(
|
| 951 |
+
"chart3_4_tc2000_anime_dub",
|
| 952 |
+
{
|
| 953 |
+
models_10[4]: (d["tc2000_ani_pred_rtn"] == y_tc2k_ani).astype(np.float32),
|
| 954 |
+
models_10[5]: (d["tc2000_ani_pred_ptt"] == y_tc2k_ani).astype(np.float32),
|
| 955 |
+
models_10[6]: (d["tc2000_ani_pred_guessit"] == y_tc2k_ani).astype(np.float32),
|
| 956 |
+
models_10[7]: (d["tc2000_ani_pred_ptn"] == y_tc2k_ani).astype(np.float32),
|
| 957 |
+
models_10[8]: (d["tc2000_ani_pred_anitomy"] == y_tc2k_ani).astype(np.float32),
|
| 958 |
+
},
|
| 959 |
+
ref_model=models_10[4],
|
| 960 |
+
expected_n=2000,
|
| 961 |
+
metric_type="mean",
|
| 962 |
+
)
|
| 963 |
+
|
| 964 |
+
y_oct_ad = d["oct15_adult_y_true"]
|
| 965 |
+
ad_models = [models_10[i] for i in (0, 1, 2, 3, 4, 5, 9)]
|
| 966 |
+
ad_keys = ["ultra", "int8", "mimmo_bert", "mimmo_cascade", "rtn", "ptt", "bitmagnet"]
|
| 967 |
+
c3_p5_stats = compute_paired_panel_stats(
|
| 968 |
+
"chart3_5_oct15_adult2781",
|
| 969 |
+
{m: (d[f"oct15_adult_pred_{k}"] == y_oct_ad).astype(np.float32) for m, k in zip(ad_models, ad_keys)},
|
| 970 |
+
ref_model=models_10[0],
|
| 971 |
+
expected_n=2781,
|
| 972 |
+
metric_type="mean",
|
| 973 |
+
)
|
| 974 |
+
|
| 975 |
+
y_oct_jk = d["oct15_junk_y_true"]
|
| 976 |
+
c3_p6a_stats = compute_paired_panel_stats(
|
| 977 |
+
"chart3_6a_oct15_junk2170",
|
| 978 |
+
{
|
| 979 |
+
"Armiger Ultra": d["oct15_junk_pred_ultra"],
|
| 980 |
+
"Armiger INT8": d["oct15_junk_pred_int8"],
|
| 981 |
+
"Bitmagnet": d["oct15_junk_pred_bitmagnet"],
|
| 982 |
+
"RTN Filter": d["oct15_junk_pred_rtn"],
|
| 983 |
+
},
|
| 984 |
+
ref_model="Armiger Ultra",
|
| 985 |
+
expected_n=2170,
|
| 986 |
+
metric_type="balanced",
|
| 987 |
+
y_true=y_oct_jk,
|
| 988 |
+
)
|
| 989 |
+
|
| 990 |
+
oct_cat_spec = [
|
| 991 |
+
(6, "Audiobook"),
|
| 992 |
+
(2, "Adult / XXX"),
|
| 993 |
+
(1, "TV Show"),
|
| 994 |
+
(0, "Movie"),
|
| 995 |
+
(7, "Software"),
|
| 996 |
+
(3, "Music"),
|
| 997 |
+
(5, "Game"),
|
| 998 |
+
(8, "Ebook"),
|
| 999 |
+
(9, "Course"),
|
| 1000 |
+
(4, "Comic"),
|
| 1001 |
+
]
|
| 1002 |
+
cats_oct_labels = []
|
| 1003 |
+
oct_ultra, oct_int8, oct_mimmo, oct_bit = [], [], [], []
|
| 1004 |
+
chart4_oct_manifest: Dict[str, Any] = {}
|
| 1005 |
+
for c_idx, c_title in oct_cat_spec:
|
| 1006 |
+
mask = (y_oct_cat11 == c_idx)
|
| 1007 |
+
n_c = int(np.sum(mask))
|
| 1008 |
+
cats_oct_labels.append(f"{c_title} (n={n_c:,})")
|
| 1009 |
+
u_v = float(np.mean(d["oct15_cat11_pred_ultra"][mask] == c_idx) * 100.0)
|
| 1010 |
+
i_v = float(np.mean(d["oct15_cat11_pred_int8_hyb"][mask] == c_idx) * 100.0)
|
| 1011 |
+
m_v = float(np.mean(d["oct15_cat11_pred_mimmo_bert"][mask] == c_idx) * 100.0)
|
| 1012 |
+
b_v = float(np.mean(d["oct15_cat11_pred_bitmagnet"][mask] == c_idx) * 100.0)
|
| 1013 |
+
oct_ultra.append(round(u_v, 2))
|
| 1014 |
+
oct_int8.append(round(i_v, 2))
|
| 1015 |
+
oct_mimmo.append(round(m_v, 2))
|
| 1016 |
+
oct_bit.append(round(b_v, 2))
|
| 1017 |
+
chart4_oct_manifest[c_title] = {
|
| 1018 |
+
"n": n_c,
|
| 1019 |
+
"armiger_ultra_pct": round(u_v, 2),
|
| 1020 |
+
"armiger_int8_pct": round(i_v, 2),
|
| 1021 |
+
"mimmo_bert_pct": round(m_v, 2),
|
| 1022 |
+
"bitmagnet_pct": round(b_v, 2),
|
| 1023 |
+
}
|
| 1024 |
+
|
| 1025 |
+
y_bc_cat11 = d["beyond_cat11_y_true"]
|
| 1026 |
+
bc_cat_spec = [
|
| 1027 |
+
(6, "Audiobook"),
|
| 1028 |
+
(8, "Ebook"),
|
| 1029 |
+
(4, "Comic"),
|
| 1030 |
+
(0, "Movie"),
|
| 1031 |
+
(1, "TV Show"),
|
| 1032 |
+
(5, "Game"),
|
| 1033 |
+
(7, "Software"),
|
| 1034 |
+
(9, "Course"),
|
| 1035 |
+
(2, "Adult / XXX"),
|
| 1036 |
+
(3, "Music"),
|
| 1037 |
+
]
|
| 1038 |
+
cats_bc_labels = []
|
| 1039 |
+
bc_ultra, bc_int8, bc_gpt, bc_gem = [], [], [], []
|
| 1040 |
+
chart4_bc_manifest: Dict[str, Any] = {}
|
| 1041 |
+
for c_idx, c_title in bc_cat_spec:
|
| 1042 |
+
mask = (y_bc_cat11 == c_idx)
|
| 1043 |
+
n_c = int(np.sum(mask))
|
| 1044 |
+
cats_bc_labels.append(f"{c_title} (n={n_c:,})")
|
| 1045 |
+
u_v = float(np.mean(d["beyond_cat11_corr_ultra"][mask]) * 100.0)
|
| 1046 |
+
i_v = float(np.mean(d["beyond_cat11_corr_int8_hyb"][mask]) * 100.0)
|
| 1047 |
+
gp_v = float(np.mean(d["beyond_cat11_corr_gpt_noncirc"][mask]) * 100.0)
|
| 1048 |
+
gm_v = float(np.mean(d["beyond_cat11_corr_gemini_noncirc"][mask]) * 100.0)
|
| 1049 |
+
bc_ultra.append(round(u_v, 2))
|
| 1050 |
+
bc_int8.append(round(i_v, 2))
|
| 1051 |
+
bc_gpt.append(round(gp_v, 2))
|
| 1052 |
+
bc_gem.append(round(gm_v, 2))
|
| 1053 |
+
chart4_bc_manifest[c_title] = {
|
| 1054 |
+
"n": n_c,
|
| 1055 |
+
"armiger_ultra_pct": round(u_v, 2),
|
| 1056 |
+
"armiger_int8_pct": round(i_v, 2),
|
| 1057 |
+
"gpt_noncirc_pct": round(gp_v, 2),
|
| 1058 |
+
"gemini_noncirc_pct": round(gm_v, 2),
|
| 1059 |
+
}
|
| 1060 |
+
if render_pngs:
|
| 1061 |
+
# ==========================================================================
|
| 1062 |
+
# RENDER CHART 3: armiger_vs_torrent_classifiers_6dims.png
|
| 1063 |
+
# ==========================================================================
|
| 1064 |
+
fig3 = plt.figure(figsize=(24.5, 14.8))
|
| 1065 |
+
fig3.suptitle(
|
| 1066 |
+
"Armiger & Armiger Ultra vs. Open-Source Torrent Classifiers & Parsers Across 7 Core Dimensions",
|
| 1067 |
+
fontsize=17.5, fontweight="bold", color=TEXT_MAIN, y=0.975,
|
| 1068 |
+
)
|
| 1069 |
+
fig3.text(
|
| 1070 |
+
0.5, 0.942,
|
| 1071 |
+
"Identical-Denominator Evaluation on oct15_confirmation (N = 2,835 / N = 545 Audio), tc_truespec_2000 (N = 2,000), & Swarm Holdouts (±95% Bootstrap CIs)",
|
| 1072 |
+
ha="center", fontsize=11.3, style="italic", color=TEXT_SUB, parse_math=False,
|
| 1073 |
+
)
|
| 1074 |
+
|
| 1075 |
+
gs3 = GridSpec(
|
| 1076 |
+
7, 4, figure=fig3,
|
| 1077 |
+
height_ratios=[1.0, 1.0, 1.0, 0.42, 1.25, 0.48, 1.25],
|
| 1078 |
+
width_ratios=[1.08, 1.0, 0.22, 1.08],
|
| 1079 |
+
hspace=0.18, wspace=0.15,
|
| 1080 |
+
top=0.89, bottom=0.06, left=0.12, right=0.98,
|
| 1081 |
+
)
|
| 1082 |
+
y10 = np.arange(len(models_10))
|
| 1083 |
+
|
| 1084 |
+
# Panel 1: Category Detection
|
| 1085 |
+
ax1 = fig3.add_subplot(gs3[0:3, 0])
|
| 1086 |
+
style_ax(ax1)
|
| 1087 |
+
ax1.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1088 |
+
cat_11_s = [c3_p1_stats["models"][m] for m in models_10]
|
| 1089 |
+
cat_11cls = [s["point_pct"] for s in cat_11_s]
|
| 1090 |
+
cat_native = [
|
| 1091 |
+
c3_p1_vid_stats["models"][models_10[0]]["point_pct"],
|
| 1092 |
+
c3_p1_vid_stats["models"][models_10[1]]["point_pct"],
|
| 1093 |
+
93.61,
|
| 1094 |
+
93.97,
|
| 1095 |
+
c3_p1_vid_stats["models"][models_10[4]]["point_pct"],
|
| 1096 |
+
c3_p1_vid_stats["models"][models_10[5]]["point_pct"],
|
| 1097 |
+
c3_p1_vid_stats["models"][models_10[6]]["point_pct"],
|
| 1098 |
+
c3_p1_vid_stats["models"][models_10[7]]["point_pct"],
|
| 1099 |
+
c3_p1_vid_stats["models"][models_10[8]]["point_pct"],
|
| 1100 |
+
88.40,
|
| 1101 |
+
]
|
| 1102 |
+
cat_lbls = [
|
| 1103 |
+
f"{cat_11cls[0]:.2f}% 11-Cls (Video {cat_native[0]:.1f}%)",
|
| 1104 |
+
f"{cat_11cls[1]:.2f}% 11-Cls (95.06% Raw)",
|
| 1105 |
+
f"{cat_11cls[2]:.2f}% 11-Cls (4-Med 93.6%)",
|
| 1106 |
+
f"{cat_11cls[3]:.2f}% 11-Cls (4-Med 94.0%)",
|
| 1107 |
+
f"{cat_11cls[4]:.2f}% 11-Cls (Video {cat_native[4]:.1f}%)",
|
| 1108 |
+
f"{cat_11cls[5]:.2f}% 11-Cls (Video {cat_native[5]:.1f}%)",
|
| 1109 |
+
f"{cat_11cls[6]:.2f}% 11-Cls (Video {cat_native[6]:.1f}%)",
|
| 1110 |
+
f"{cat_11cls[7]:.2f}% 11-Cls (Video {cat_native[7]:.1f}%)",
|
| 1111 |
+
f"{cat_11cls[8]:.2f}% 11-Cls (Video {cat_native[8]:.1f}%)",
|
| 1112 |
+
f"{cat_11cls[9]:.2f}% 11-Cls (Ans. 88.4%)",
|
| 1113 |
+
]
|
| 1114 |
+
ax1.barh(y10 - 0.16, cat_native, height=0.32, color=colors_10, zorder=3)
|
| 1115 |
+
ax1.barh(y10 + 0.16, cat_11cls, height=0.32, color=colors_10, alpha=0.45, hatch="//", edgecolor=colors_10, zorder=3)
|
| 1116 |
+
draw_ci_whiskers(ax1, y10 + 0.16, cat_11_s)
|
| 1117 |
+
ax1.set_yticks(y10)
|
| 1118 |
+
ax1.set_yticklabels(models_10, fontsize=9.5, fontweight="bold", color=TEXT_MAIN)
|
| 1119 |
+
ax1.invert_yaxis()
|
| 1120 |
+
ax1.set_xlim(25, 148)
|
| 1121 |
+
ax1.set_xticks([30, 40, 50, 60, 70, 80, 90, 100])
|
| 1122 |
+
ax1.set_xlabel("Solid = Native Domain (Video N=1,296 / 4-Med N=1,580); Hatched = 11-Class (N=2,835)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
|
| 1123 |
+
ax1.set_title("1. Category Detection (%)\n(oct15_confirmation, Identical N = 2,835)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
|
| 1124 |
+
for idx, (v, lbl) in enumerate(zip(cat_native, cat_lbls)):
|
| 1125 |
+
ax1.text(v + 1.1, idx, lbl, va="center", ha="left", fontsize=8.2, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
|
| 1126 |
+
|
| 1127 |
+
# Panel 2: Catalog Matching Rate
|
| 1128 |
+
ax2 = fig3.add_subplot(gs3[0:3, 1])
|
| 1129 |
+
style_ax(ax2)
|
| 1130 |
+
ax2.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1131 |
+
p2_s = [c3_p2_stats["models"][m] for m in models_10]
|
| 1132 |
+
cat_match = [s["point_pct"] for s in p2_s]
|
| 1133 |
+
ax2.barh(y10, cat_match, height=0.58, color=colors_10, zorder=3)
|
| 1134 |
+
draw_ci_whiskers(ax2, y10, p2_s)
|
| 1135 |
+
ax2.set_yticks(y10)
|
| 1136 |
+
ax2.set_yticklabels([""] * 10)
|
| 1137 |
+
ax2.invert_yaxis()
|
| 1138 |
+
ax2.set_xlim(35, 128)
|
| 1139 |
+
ax2.set_xticks([40, 50, 60, 70, 80, 90, 100])
|
| 1140 |
+
ax2.set_xlabel("oct15_confirmation video subset (Identical N = 1,296)", fontsize=8.8, color=TEXT_SUB, labelpad=6)
|
| 1141 |
+
ax2.set_title("2. Catalog Matching Rate (%)\n(Clean Title + Correct Movie/TV Routing, N = 1,296)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
|
| 1142 |
+
for idx, s_item in enumerate(p2_s):
|
| 1143 |
+
v = s_item["point_pct"]
|
| 1144 |
+
lbl = f"{v:.2f}% (95.83% Raw)" if idx == 1 else f"{v:.2f}%"
|
| 1145 |
+
ax2.text(max(v, s_item["ci95_hi"]) + 0.9, idx, lbl, va="center", ha="left", fontsize=8.4, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
|
| 1146 |
+
|
| 1147 |
+
# Panel 3: Audio Language Detection (Strict E-vs-F N=545)
|
| 1148 |
+
ax3 = fig3.add_subplot(gs3[0:3, 3])
|
| 1149 |
+
style_ax(ax3)
|
| 1150 |
+
ax3.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1151 |
+
for idx, m in enumerate(models_10):
|
| 1152 |
+
if m in c3_p3_stats["models"]:
|
| 1153 |
+
s_item = c3_p3_stats["models"][m]
|
| 1154 |
+
v = s_item["point_pct"]
|
| 1155 |
+
ax3.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
|
| 1156 |
+
draw_ci_whiskers(ax3, np.array([idx]), [s_item])
|
| 1157 |
+
lbl = f"{v:.2f}% (96.88% Raw)" if idx == 1 else f"{v:.2f}%"
|
| 1158 |
+
ax3.text(max(v, s_item["ci95_hi"]) + 0.25, idx, lbl, va="center", ha="left", fontsize=8.4, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
|
| 1159 |
+
else:
|
| 1160 |
+
ax3.barh(idx, 86.2, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
|
| 1161 |
+
ax3.text(84.3, idx, "N/A (No Audio Language Head)", va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
|
| 1162 |
+
ax3.set_yticks(y10)
|
| 1163 |
+
ax3.set_yticklabels([""] * 10)
|
| 1164 |
+
ax3.invert_yaxis()
|
| 1165 |
+
ax3.set_xlim(84, 106.5)
|
| 1166 |
+
ax3.set_xticks([86, 88, 90, 92, 94, 96, 98, 100])
|
| 1167 |
+
ax3.set_xlabel("All models scored on identical oct15_confirmation Strict E-vs-F (N = 545)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
|
| 1168 |
+
ax3.set_title("3. Audio Language Detection (%)\n(Strict English vs. Foreign Audio, Identical N = 545)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
|
| 1169 |
+
|
| 1170 |
+
# Panel 4: Anime & Anime Dub Detection
|
| 1171 |
+
ax4 = fig3.add_subplot(gs3[4:7, 0])
|
| 1172 |
+
style_ax(ax4)
|
| 1173 |
+
ax4.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1174 |
+
for idx, m in enumerate(models_10):
|
| 1175 |
+
if m in c3_p4_oct_stats["models"]:
|
| 1176 |
+
s_item = c3_p4_oct_stats["models"][m]
|
| 1177 |
+
v = s_item["point_pct"]
|
| 1178 |
+
ax4.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
|
| 1179 |
+
draw_ci_whiskers(ax4, np.array([idx]), [s_item])
|
| 1180 |
+
ax4.text(max(v, s_item["ci95_hi"]) + 0.6, idx, f"{v:.2f}% (95.8% tc2000)", va="center", ha="left", fontsize=8.4, fontweight="bold", color=TEXT_MAIN)
|
| 1181 |
+
elif m in c3_p4_tc2k_stats["models"]:
|
| 1182 |
+
s_item = c3_p4_tc2k_stats["models"][m]
|
| 1183 |
+
v = s_item["point_pct"]
|
| 1184 |
+
ax4.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
|
| 1185 |
+
draw_ci_whiskers(ax4, np.array([idx]), [s_item])
|
| 1186 |
+
ax4.text(max(v, s_item["ci95_hi"]) + 0.6, idx, f"{v:.2f}%", va="center", ha="left", fontsize=8.4, color="#CBD5E1")
|
| 1187 |
+
else:
|
| 1188 |
+
ax4.barh(idx, 49.0, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
|
| 1189 |
+
ax4.text(46.2, idx, "N/A (No Anime / Dub Head)", va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
|
| 1190 |
+
ax4.set_yticks(y10)
|
| 1191 |
+
ax4.set_yticklabels(models_10, fontsize=9.5, fontweight="bold", color=TEXT_MAIN)
|
| 1192 |
+
ax4.invert_yaxis()
|
| 1193 |
+
ax4.set_xlim(45, 122)
|
| 1194 |
+
ax4.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1195 |
+
ax4.set_xlabel("99.61% on oct15_confirmation (505/507); 95.80% on tc_truespec_2000 (N=2,000)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
|
| 1196 |
+
ax4.set_title("4. Anime & Anime Dub Detection (%)\n(oct15_confirmation N = 507 & tc_truespec_2000 N = 2,000)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
|
| 1197 |
+
|
| 1198 |
+
# Panel 5: Adult / NSFW Detection
|
| 1199 |
+
ax5 = fig3.add_subplot(gs3[4:7, 1])
|
| 1200 |
+
style_ax(ax5)
|
| 1201 |
+
ax5.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1202 |
+
for idx, m in enumerate(models_10):
|
| 1203 |
+
if m in c3_p5_stats["models"]:
|
| 1204 |
+
s_item = c3_p5_stats["models"][m]
|
| 1205 |
+
v = s_item["point_pct"]
|
| 1206 |
+
ax5.barh(idx, v, height=0.58, color=colors_10[idx], zorder=3)
|
| 1207 |
+
draw_ci_whiskers(ax5, np.array([idx]), [s_item])
|
| 1208 |
+
lbl = f"{v:.2f}% (93.7% Ans.)" if idx == 9 else f"{v:.2f}%"
|
| 1209 |
+
ax5.text(max(v, s_item["ci95_hi"]) + 0.5, idx, lbl, va="center", ha="left", fontsize=8.4, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
|
| 1210 |
+
else:
|
| 1211 |
+
ax5.barh(idx, 68.5, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
|
| 1212 |
+
ax5.text(65.8, idx, "N/A (No Adult / NSFW Head)", va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
|
| 1213 |
+
ax5.set_yticks(y10)
|
| 1214 |
+
ax5.set_yticklabels([""] * 10)
|
| 1215 |
+
ax5.invert_yaxis()
|
| 1216 |
+
ax5.set_xlim(65, 114)
|
| 1217 |
+
ax5.set_xticks([70, 75, 80, 85, 90, 95, 100])
|
| 1218 |
+
ax5.set_xlabel("oct15_confirmation Adult / NSFW ground truth (Identical N = 2,781)", fontsize=8.6, color=TEXT_SUB, labelpad=6)
|
| 1219 |
+
ax5.set_title("5. Adult / NSFW Detection (%)\n(oct15_confirmation, Identical N = 2,781)", fontsize=11, fontweight="bold", color=TEXT_MAIN, pad=8)
|
| 1220 |
+
|
| 1221 |
+
# Panel 6A: Junk / Malware Detection
|
| 1222 |
+
ax6a = fig3.add_subplot(gs3[4, 3])
|
| 1223 |
+
style_ax(ax6a)
|
| 1224 |
+
ax6a.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1225 |
+
m6a = ["Armiger Ultra", "Armiger INT8", "Bitmagnet", "RTN Filter", "Other 6 Tools"]
|
| 1226 |
+
v6a = [
|
| 1227 |
+
c3_p6a_stats["models"]["Armiger Ultra"]["point_pct"],
|
| 1228 |
+
c3_p6a_stats["models"]["Armiger INT8"]["point_pct"],
|
| 1229 |
+
c3_p6a_stats["models"]["Bitmagnet"]["point_pct"],
|
| 1230 |
+
c3_p6a_stats["models"]["RTN Filter"]["point_pct"],
|
| 1231 |
+
None,
|
| 1232 |
+
]
|
| 1233 |
+
c6a = [C_ULTRA, C_INT8, "#94A3B8", "#F59E0B", "#21262D"]
|
| 1234 |
+
l6a = [
|
| 1235 |
+
f"{v6a[0]:.2f}% Bal (98.41% beyond N=977)",
|
| 1236 |
+
f"{v6a[1]:.2f}% Bal (98.41% beyond N=977)",
|
| 1237 |
+
f"{v6a[2]:.2f}% Bal (0% Recall, 100% Spec)",
|
| 1238 |
+
f"{v6a[3]:.2f}% Bal (0% Recall, 98.57% Spec)",
|
| 1239 |
+
"N/A (No Junk Head)",
|
| 1240 |
+
]
|
| 1241 |
+
for idx, (v, lbl) in enumerate(zip(v6a, l6a)):
|
| 1242 |
+
if v is not None:
|
| 1243 |
+
ax6a.barh(idx, v, height=0.6, color=c6a[idx], zorder=3)
|
| 1244 |
+
ax6a.text(v + 0.8, idx, lbl, va="center", ha="left", fontsize=8.2, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
|
| 1245 |
+
else:
|
| 1246 |
+
ax6a.barh(idx, 49.0, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
|
| 1247 |
+
ax6a.text(46.2, idx, lbl, va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
|
| 1248 |
+
ax6a.set_yticks(np.arange(len(m6a)))
|
| 1249 |
+
ax6a.set_yticklabels(m6a, fontsize=9, fontweight="bold", color=TEXT_MAIN)
|
| 1250 |
+
ax6a.invert_yaxis()
|
| 1251 |
+
ax6a.set_xlim(45, 144)
|
| 1252 |
+
ax6a.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1253 |
+
ax6a.set_title("6A. Junk / Malware Detection (oct15 N=2,170 & beyond N=977)", fontsize=10, fontweight="bold", color=TEXT_MAIN, pad=7)
|
| 1254 |
+
|
| 1255 |
+
# Panel 6B: Swarm Liveliness AUROC (Identical N=10,278 Multi-Source Corpus)
|
| 1256 |
+
ax6b = fig3.add_subplot(gs3[6, 3])
|
| 1257 |
+
style_ax(ax6b)
|
| 1258 |
+
ax6b.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1259 |
+
m6b = ["Armiger Ultra", "Armiger Priors", "Seeder Count", "All 8 Tools"]
|
| 1260 |
+
v6b = [
|
| 1261 |
+
p2f_whse_stats["models"]["Armiger Ultra (Fair Hybrid)"]["point_pct"],
|
| 1262 |
+
p2f_whse_stats["models"]["Armiger 4-Way Priors"]["point_pct"],
|
| 1263 |
+
p2f_whse_stats["models"]["Raw Seeder Count"]["point_pct"],
|
| 1264 |
+
None,
|
| 1265 |
+
]
|
| 1266 |
+
c6b = [C_ULTRA, C_INT8, "#64748B", "#21262D"]
|
| 1267 |
+
l6b = [
|
| 1268 |
+
f"{v6b[0]:.2f}% Corpus (92.42% Pub N=757)",
|
| 1269 |
+
f"{v6b[1]:.2f}% Corpus (4-Way Priors)",
|
| 1270 |
+
f"{v6b[2]:.2f}% Corpus (85.17% Pub N=757)",
|
| 1271 |
+
"N/A (No Swarm Head)",
|
| 1272 |
+
]
|
| 1273 |
+
for idx, (v, lbl) in enumerate(zip(v6b, l6b)):
|
| 1274 |
+
if v is not None:
|
| 1275 |
+
ax6b.barh(idx, v, height=0.6, color=c6b[idx], zorder=3)
|
| 1276 |
+
ax6b.text(v + 0.8, idx, lbl, va="center", ha="left", fontsize=8.2, fontweight="bold" if idx < 2 else "normal", color=TEXT_MAIN if idx < 2 else "#CBD5E1")
|
| 1277 |
+
else:
|
| 1278 |
+
ax6b.barh(idx, 52.5, height=0.55, color="#21262D", edgecolor=TEXT_MUTED, hatch="//", zorder=2)
|
| 1279 |
+
ax6b.text(49.5, idx, lbl, va="center", ha="left", fontsize=8.1, style="italic", color=TEXT_SUB)
|
| 1280 |
+
ax6b.set_yticks(np.arange(len(m6b)))
|
| 1281 |
+
ax6b.set_yticklabels(m6b, fontsize=9, fontweight="bold", color=TEXT_MAIN)
|
| 1282 |
+
ax6b.invert_yaxis()
|
| 1283 |
+
ax6b.set_xlim(48, 144)
|
| 1284 |
+
ax6b.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1285 |
+
ax6b.set_xlabel("Multi-Source Corpus (Identical N=10,278) & Public Tracker (N=757: 92.42%)", fontsize=8.2, color=TEXT_SUB, labelpad=6)
|
| 1286 |
+
ax6b.set_title("6B. Swarm Liveliness AUROC % (swarm_adapter.pt)", fontsize=10, fontweight="bold", color=TEXT_MAIN, pad=7)
|
| 1287 |
+
|
| 1288 |
+
sha_chart3 = save_clean_rgb_png(fig3, assets_dir / "armiger_vs_torrent_classifiers_6dims.png")
|
| 1289 |
+
|
| 1290 |
+
# ==========================================================================
|
| 1291 |
+
# COMPUTE & RENDER CHART 4: armiger_categorical_breakdown.png
|
| 1292 |
+
# ==========================================================================
|
| 1293 |
+
fig4, (ax4_l, ax4_r) = plt.subplots(1, 2, figsize=(22.5, 11.2), gridspec_kw={"wspace": 0.22})
|
| 1294 |
+
fig4.subplots_adjust(top=0.84, bottom=0.16, left=0.11, right=0.98)
|
| 1295 |
+
fig4.suptitle(
|
| 1296 |
+
"Per-Category Recall Breakdown Across All 10 Known Media & Software Categories",
|
| 1297 |
+
fontsize=17, fontweight="bold", color=TEXT_MAIN, y=0.965,
|
| 1298 |
+
)
|
| 1299 |
+
fig4.text(
|
| 1300 |
+
0.5, 0.925,
|
| 1301 |
+
"Left: oct15_confirmation (Identical N = 2,835) vs. Open-Source Classifiers | Right: beyond_confirmation (Identical N = 1,025, LOTO Non-Circular)",
|
| 1302 |
+
ha="center", fontsize=11.2, style="italic", color=TEXT_SUB, parse_math=False,
|
| 1303 |
+
)
|
| 1304 |
+
|
| 1305 |
+
# Order for Left Panel (sorted by Armiger Ultra accuracy on oct15_confirmation)
|
| 1306 |
+
oct_cat_spec = [
|
| 1307 |
+
(6, "Audiobook"),
|
| 1308 |
+
(2, "Adult / XXX"),
|
| 1309 |
+
(1, "TV Show"),
|
| 1310 |
+
(0, "Movie"),
|
| 1311 |
+
(7, "Software"),
|
| 1312 |
+
(3, "Music"),
|
| 1313 |
+
(5, "Game"),
|
| 1314 |
+
(8, "Ebook"),
|
| 1315 |
+
(9, "Course"),
|
| 1316 |
+
(4, "Comic"),
|
| 1317 |
+
]
|
| 1318 |
+
cats_oct_labels = []
|
| 1319 |
+
oct_ultra, oct_int8, oct_mimmo, oct_bit = [], [], [], []
|
| 1320 |
+
chart4_oct_manifest: Dict[str, Any] = {}
|
| 1321 |
+
for c_idx, c_title in oct_cat_spec:
|
| 1322 |
+
mask = (y_oct_cat11 == c_idx)
|
| 1323 |
+
n_c = int(np.sum(mask))
|
| 1324 |
+
cats_oct_labels.append(f"{c_title} (n={n_c:,})")
|
| 1325 |
+
u_v = float(np.mean(d["oct15_cat11_pred_ultra"][mask] == c_idx) * 100.0)
|
| 1326 |
+
i_v = float(np.mean(d["oct15_cat11_pred_int8_hyb"][mask] == c_idx) * 100.0)
|
| 1327 |
+
m_v = float(np.mean(d["oct15_cat11_pred_mimmo_bert"][mask] == c_idx) * 100.0)
|
| 1328 |
+
b_v = float(np.mean(d["oct15_cat11_pred_bitmagnet"][mask] == c_idx) * 100.0)
|
| 1329 |
+
oct_ultra.append(round(u_v, 2))
|
| 1330 |
+
oct_int8.append(round(i_v, 2))
|
| 1331 |
+
oct_mimmo.append(round(m_v, 2))
|
| 1332 |
+
oct_bit.append(round(b_v, 2))
|
| 1333 |
+
chart4_oct_manifest[c_title] = {
|
| 1334 |
+
"n": n_c,
|
| 1335 |
+
"armiger_ultra_pct": round(u_v, 2),
|
| 1336 |
+
"armiger_int8_pct": round(i_v, 2),
|
| 1337 |
+
"mimmo_bert_pct": round(m_v, 2),
|
| 1338 |
+
"bitmagnet_pct": round(b_v, 2),
|
| 1339 |
+
}
|
| 1340 |
+
|
| 1341 |
+
style_ax(ax4_l)
|
| 1342 |
+
y_c = np.arange(len(cats_oct_labels))
|
| 1343 |
+
h = 0.19
|
| 1344 |
+
ax4_l.barh(y_c - 1.5 * h, oct_ultra, height=h, color=C_ULTRA, label="Armiger Ultra Fair Hybrid (95.49% Overall)", zorder=3)
|
| 1345 |
+
ax4_l.barh(y_c - 0.5 * h, oct_int8, height=h, color=C_INT8, label="Armiger INT8 + ZeroCue (95.49% Overall, 95.06% Raw)", zorder=3)
|
| 1346 |
+
ax4_l.barh(y_c + 0.5 * h, oct_mimmo, height=h, color="#818CF8", label="MIMMO BERT 360M (72.24% Overall on N=2,835)", zorder=3)
|
| 1347 |
+
ax4_l.barh(y_c + 1.5 * h, oct_bit, height=h, color="#64748B", label="Bitmagnet Built-In (35.17% Overall on N=2,835)", zorder=3)
|
| 1348 |
+
ax4_l.set_yticks(y_c)
|
| 1349 |
+
ax4_l.set_yticklabels(cats_oct_labels, fontsize=10, fontweight="bold", color=TEXT_MAIN)
|
| 1350 |
+
ax4_l.set_ylim(len(cats_oct_labels) - 0.35, -0.6)
|
| 1351 |
+
ax4_l.set_xlim(0, 128)
|
| 1352 |
+
ax4_l.set_xticks([0, 20, 40, 60, 80, 100])
|
| 1353 |
+
ax4_l.set_xlabel("Per-Class Recall (%) — Identical N = 2,835 Ground-Truth Mask", fontsize=9.8, color=TEXT_SUB, labelpad=8)
|
| 1354 |
+
ax4_l.set_title("1. Per-Category Recall vs. Open-Source Classifiers\n(oct15_confirmation, Identical N = 2,835)", fontsize=11.5, fontweight="bold", color=TEXT_MAIN, pad=10)
|
| 1355 |
+
for idx in range(len(cats_oct_labels)):
|
| 1356 |
+
ax4_l.text(oct_ultra[idx] + 1.0, idx - 1.1 * h, f"{oct_ultra[idx]:.1f}%", va="center", ha="left", fontsize=8.0, fontweight="bold", color=C_ULTRA_EDGE)
|
| 1357 |
+
ax4_l.text(oct_mimmo[idx] + 1.0, idx + 0.5 * h, f"{oct_mimmo[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color="#CBD5E1")
|
| 1358 |
+
ax4_l.text(oct_bit[idx] + 1.0, idx + 1.5 * h, f"{oct_bit[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color=TEXT_SUB)
|
| 1359 |
+
ax4_l.legend(loc="upper center", bbox_to_anchor=(0.5, -0.09), ncol=2, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=8.6, labelcolor=TEXT_MAIN)
|
| 1360 |
+
|
| 1361 |
+
# Order for Right Panel (beyond_confirmation N=1,025 Non-Circular LOTO)
|
| 1362 |
+
y_bc_cat11 = d["beyond_cat11_y_true"]
|
| 1363 |
+
bc_cat_spec = [
|
| 1364 |
+
(6, "Audiobook"),
|
| 1365 |
+
(8, "Ebook"),
|
| 1366 |
+
(4, "Comic"),
|
| 1367 |
+
(0, "Movie"),
|
| 1368 |
+
(1, "TV Show"),
|
| 1369 |
+
(5, "Game"),
|
| 1370 |
+
(7, "Software"),
|
| 1371 |
+
(9, "Course"),
|
| 1372 |
+
(2, "Adult / XXX"),
|
| 1373 |
+
(3, "Music"),
|
| 1374 |
+
]
|
| 1375 |
+
cats_bc_labels = []
|
| 1376 |
+
bc_ultra, bc_int8, bc_gpt, bc_gem = [], [], [], []
|
| 1377 |
+
chart4_bc_manifest: Dict[str, Any] = {}
|
| 1378 |
+
for c_idx, c_title in bc_cat_spec:
|
| 1379 |
+
mask = (y_bc_cat11 == c_idx)
|
| 1380 |
+
n_c = int(np.sum(mask))
|
| 1381 |
+
cats_bc_labels.append(f"{c_title} (n={n_c:,})")
|
| 1382 |
+
u_v = float(np.mean(d["beyond_cat11_corr_ultra"][mask]) * 100.0)
|
| 1383 |
+
i_v = float(np.mean(d["beyond_cat11_corr_int8_hyb"][mask]) * 100.0)
|
| 1384 |
+
gp_v = float(np.mean(d["beyond_cat11_corr_gpt_noncirc"][mask]) * 100.0)
|
| 1385 |
+
gm_v = float(np.mean(d["beyond_cat11_corr_gemini_noncirc"][mask]) * 100.0)
|
| 1386 |
+
bc_ultra.append(round(u_v, 2))
|
| 1387 |
+
bc_int8.append(round(i_v, 2))
|
| 1388 |
+
bc_gpt.append(round(gp_v, 2))
|
| 1389 |
+
bc_gem.append(round(gm_v, 2))
|
| 1390 |
+
chart4_bc_manifest[c_title] = {
|
| 1391 |
+
"n": n_c,
|
| 1392 |
+
"armiger_ultra_pct": round(u_v, 2),
|
| 1393 |
+
"armiger_int8_pct": round(i_v, 2),
|
| 1394 |
+
"gpt_noncirc_pct": round(gp_v, 2),
|
| 1395 |
+
"gemini_noncirc_pct": round(gm_v, 2),
|
| 1396 |
+
}
|
| 1397 |
+
|
| 1398 |
+
style_ax(ax4_r)
|
| 1399 |
+
ax4_r.barh(y_c - 1.5 * h, bc_ultra, height=h, color=C_ULTRA, label="Armiger Ultra Fair Hybrid (95.12% Non-Circular)", zorder=3)
|
| 1400 |
+
ax4_r.barh(y_c - 0.5 * h, bc_int8, height=h, color=C_INT8, label="Armiger INT8 + ZeroCue (95.12% Non-Circular, 92.88% Raw)", zorder=3)
|
| 1401 |
+
ax4_r.barh(y_c + 0.5 * h, bc_gpt, height=h, color=C_GPT, label="OpenAI GPT Fast Tier (95.12% Non-Circ | 96.59% Circ)", zorder=3)
|
| 1402 |
+
ax4_r.barh(y_c + 1.5 * h, bc_gem, height=h, color=C_GEMINI, label="Google Gemini Flash (93.80% Non-Circ | 95.61% Circ)", zorder=3)
|
| 1403 |
+
ax4_r.set_yticks(y_c)
|
| 1404 |
+
ax4_r.set_yticklabels(cats_bc_labels, fontsize=10, fontweight="bold", color=TEXT_MAIN)
|
| 1405 |
+
ax4_r.set_ylim(len(cats_bc_labels) - 0.35, -0.6)
|
| 1406 |
+
ax4_r.set_xlim(0, 128)
|
| 1407 |
+
ax4_r.set_xticks([0, 20, 40, 60, 80, 100])
|
| 1408 |
+
ax4_r.set_xlabel("Per-Class Recall (%) — Leave-One-Teacher-Out Non-Circular Reference", fontsize=9.8, color=TEXT_SUB, labelpad=8)
|
| 1409 |
+
ax4_r.set_title("2. Per-Category Recall vs. Cloud LLMs\n(beyond_confirmation, Identical N = 1,025, LOTO Non-Circular)", fontsize=11.5, fontweight="bold", color=TEXT_MAIN, pad=10)
|
| 1410 |
+
for idx in range(len(cats_bc_labels)):
|
| 1411 |
+
ax4_r.text(bc_ultra[idx] + 1.0, idx - 1.1 * h, f"{bc_ultra[idx]:.1f}%", va="center", ha="left", fontsize=8.0, fontweight="bold", color=C_ULTRA_EDGE)
|
| 1412 |
+
ax4_r.text(bc_gpt[idx] + 1.0, idx + 0.5 * h, f"{bc_gpt[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color=C_GPT)
|
| 1413 |
+
ax4_r.text(bc_gem[idx] + 1.0, idx + 1.5 * h, f"{bc_gem[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color=C_GEMINI)
|
| 1414 |
+
ax4_r.legend(loc="upper center", bbox_to_anchor=(0.5, -0.09), ncol=2, facecolor=BG_PANEL, edgecolor=BORDER_COLOR, fontsize=8.6, labelcolor=TEXT_MAIN)
|
| 1415 |
+
|
| 1416 |
+
sha_chart4 = save_clean_rgb_png(fig4, assets_dir / "armiger_categorical_breakdown.png")
|
| 1417 |
+
else:
|
| 1418 |
+
for fn in ("armiger_vs_torrent_classifiers_6dims.png", "armiger_categorical_breakdown.png"):
|
| 1419 |
+
verify_png_chunks_clean(fallback_assets / fn)
|
| 1420 |
+
sha_chart3 = hashlib.sha256((fallback_assets / "armiger_vs_torrent_classifiers_6dims.png").read_bytes()).hexdigest()
|
| 1421 |
+
sha_chart4 = hashlib.sha256((fallback_assets / "armiger_categorical_breakdown.png").read_bytes()).hexdigest()
|
| 1422 |
+
|
| 1423 |
+
# ==========================================================================
|
| 1424 |
+
# FIVE-SPLIT SUMMARY TABLE STATS
|
| 1425 |
+
# ==========================================================================
|
| 1426 |
+
five_split_summary: Dict[str, Any] = {}
|
| 1427 |
+
for sp_code, sp_label, n_exp in [
|
| 1428 |
+
("oct15", "oct15_confirmation", 2835),
|
| 1429 |
+
("fresh", "fresh_confirmation", 1402),
|
| 1430 |
+
("beyond", "beyond_confirmation", 1025),
|
| 1431 |
+
("tc2000", "tc_truespec_2000", 2000),
|
| 1432 |
+
("tc3000", "tc_truespec_multilang_3000", 3000),
|
| 1433 |
+
]:
|
| 1434 |
+
y_sp = d[f"split5_{sp_code}_y_cat"]
|
| 1435 |
+
sp_stats = compute_paired_panel_stats(
|
| 1436 |
+
f"five_split_{sp_code}",
|
| 1437 |
+
{
|
| 1438 |
+
"armiger_ultra_hyb": (d[f"split5_{sp_code}_fp32_hyb_pred"] == y_sp).astype(np.float32),
|
| 1439 |
+
"armiger_ultra_raw": (d[f"split5_{sp_code}_fp32_raw_pred"] == y_sp).astype(np.float32),
|
| 1440 |
+
"armiger_int8_hyb": (d[f"split5_{sp_code}_int8_hyb_pred"] == y_sp).astype(np.float32),
|
| 1441 |
+
"armiger_int8_raw": (d[f"split5_{sp_code}_int8_raw_pred"] == y_sp).astype(np.float32),
|
| 1442 |
+
},
|
| 1443 |
+
ref_model="armiger_ultra_hyb",
|
| 1444 |
+
expected_n=n_exp,
|
| 1445 |
+
metric_type="mean",
|
| 1446 |
+
)
|
| 1447 |
+
five_split_summary[sp_label] = sp_stats
|
| 1448 |
+
|
| 1449 |
+
# ==========================================================================
|
| 1450 |
+
# WRITE CRYPTOGRAPHIC PROVENANCE MANIFEST (CHARTS_MANIFEST.json)
|
| 1451 |
+
# ==========================================================================
|
| 1452 |
+
repo_root = assets_dir.parent
|
| 1453 |
+
prov_hashes: Dict[str, str] = {
|
| 1454 |
+
"canonical_eval_vectors.json": hashlib.sha256(npz_path.read_bytes()).hexdigest(),
|
| 1455 |
+
"generate_canonical_charts.py": hashlib.sha256(pathlib.Path(__file__).read_bytes()).hexdigest(),
|
| 1456 |
+
"assets/armiger_vs_llms_overall_and_6dims.png": sha_chart1,
|
| 1457 |
+
"assets/armiger_vs_llms_cost_comparison.png": sha_chart2,
|
| 1458 |
+
"assets/armiger_vs_torrent_classifiers_6dims.png": sha_chart3,
|
| 1459 |
+
"assets/armiger_categorical_breakdown.png": sha_chart4,
|
| 1460 |
+
}
|
| 1461 |
+
for opt_rel in [
|
| 1462 |
+
"validated_zero_cue_catalog.json",
|
| 1463 |
+
"armiger_ultra.fp32.onnx",
|
| 1464 |
+
"armiger.int8.onnx",
|
| 1465 |
+
"models/armiger_ultra.fp32.onnx",
|
| 1466 |
+
"models/armiger.int8.onnx",
|
| 1467 |
+
]:
|
| 1468 |
+
p_opt = repo_root / opt_rel
|
| 1469 |
+
if p_opt.is_file():
|
| 1470 |
+
prov_hashes[opt_rel] = hashlib.sha256(p_opt.read_bytes()).hexdigest()
|
| 1471 |
+
|
| 1472 |
+
manifest: Dict[str, Any] = {
|
| 1473 |
+
"schema_version": "1.0.0",
|
| 1474 |
+
"bootstrap_seed": BOOTSTRAP_SEED,
|
| 1475 |
+
"bootstrap_resamples": BOOTSTRAP_RESAMPLES,
|
| 1476 |
+
"invariants_verified": {
|
| 1477 |
+
"identical_denominator_lock": True,
|
| 1478 |
+
"anti_circularity_lock": True,
|
| 1479 |
+
"paired_uncertainty_and_significance": True,
|
| 1480 |
+
"png_metadata_stripped_ihdr_idat_iend_only": True,
|
| 1481 |
+
},
|
| 1482 |
+
"anti_circularity_audit": anti_circ_checks,
|
| 1483 |
+
"provenance_sha256": prov_hashes,
|
| 1484 |
+
"charts": {
|
| 1485 |
+
"armiger_vs_llms_overall_and_6dims.png": {
|
| 1486 |
+
"sha256": sha_chart1,
|
| 1487 |
+
"composite_6dim_mean": composite_stats,
|
| 1488 |
+
"panel_2a_category_11class": p2a_stats,
|
| 1489 |
+
"panel_2b_catalog_matching": p2b_stats,
|
| 1490 |
+
"panel_2c_audio_strict_229": p2c_stats,
|
| 1491 |
+
"panel_2d_anime_valid_636": p2d_stats,
|
| 1492 |
+
"panel_2e_junk_balanced_977": p2e_stats,
|
| 1493 |
+
"panel_2f_swarm_public_757": p2f_pub_stats,
|
| 1494 |
+
"panel_2f_swarm_warehouse_10278": p2f_whse_stats,
|
| 1495 |
+
},
|
| 1496 |
+
"armiger_vs_llms_cost_comparison.png": {
|
| 1497 |
+
"sha256": sha_chart2,
|
| 1498 |
+
"models": models_cost,
|
| 1499 |
+
"batched_usd_per_1m": batched_cost,
|
| 1500 |
+
"realtime_usd_per_1m": realtime_cost,
|
| 1501 |
+
},
|
| 1502 |
+
"armiger_vs_torrent_classifiers_6dims.png": {
|
| 1503 |
+
"sha256": sha_chart3,
|
| 1504 |
+
"panel_1_category_11class_oct15": c3_p1_stats,
|
| 1505 |
+
"panel_1_video_mvtv_oct15": c3_p1_vid_stats,
|
| 1506 |
+
"panel_2_catalog_matching_oct15": c3_p2_stats,
|
| 1507 |
+
"panel_3_audio_strict_oct15": c3_p3_stats,
|
| 1508 |
+
"panel_4_anime_oct15": c3_p4_oct_stats,
|
| 1509 |
+
"panel_4_anime_dub_tc2000": c3_p4_tc2k_stats,
|
| 1510 |
+
"panel_5_adult_oct15": c3_p5_stats,
|
| 1511 |
+
"panel_6a_junk_oct15": c3_p6a_stats,
|
| 1512 |
+
},
|
| 1513 |
+
"armiger_categorical_breakdown.png": {
|
| 1514 |
+
"sha256": sha_chart4,
|
| 1515 |
+
"left_oct15_confirmation_n2835": chart4_oct_manifest,
|
| 1516 |
+
"right_beyond_confirmation_n1025_loto": chart4_bc_manifest,
|
| 1517 |
+
},
|
| 1518 |
+
},
|
| 1519 |
+
"five_split_category_summary": five_split_summary,
|
| 1520 |
+
}
|
| 1521 |
+
manifest_path.parent.mkdir(parents=True, exist_ok=True)
|
| 1522 |
+
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=False) + "\n", encoding="utf-8")
|
| 1523 |
+
return manifest
|
| 1524 |
+
|
| 1525 |
+
|
| 1526 |
+
def main() -> None:
|
| 1527 |
+
parser = argparse.ArgumentParser(description="Generate canonical Armiger benchmark charts and CHARTS_MANIFEST.json.")
|
| 1528 |
+
here = pathlib.Path(__file__).resolve().parent
|
| 1529 |
+
parser.add_argument("--vectors", type=pathlib.Path, default=here / "canonical_eval_vectors.json")
|
| 1530 |
+
parser.add_argument("--assets-dir", type=pathlib.Path, default=here.parent / "assets")
|
| 1531 |
+
parser.add_argument("--manifest", type=pathlib.Path, default=here / "CHARTS_MANIFEST.json")
|
| 1532 |
+
args = parser.parse_args()
|
| 1533 |
+
|
| 1534 |
+
manifest = build_all_charts_and_manifest(args.vectors, args.assets_dir, args.manifest)
|
| 1535 |
+
for name, info in manifest["charts"].items():
|
| 1536 |
+
print(f"[OK] {name:46s} sha256={info['sha256'][:16]}...")
|
| 1537 |
+
print(f"[OK] Wrote {args.manifest}")
|
| 1538 |
+
|
| 1539 |
+
|
| 1540 |
+
if __name__ == "__main__":
|
| 1541 |
+
main()
|
benchmarks/test_chart_integrity.py
ADDED
|
@@ -0,0 +1,192 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Automated integrity and statistical invariant test suite for the canonical
|
| 3 |
+
Armiger & Armiger Ultra chart and benchmark pipeline.
|
| 4 |
+
"""
|
| 5 |
+
from __future__ import annotations
|
| 6 |
+
|
| 7 |
+
import hashlib
|
| 8 |
+
import json
|
| 9 |
+
import pathlib
|
| 10 |
+
import sys
|
| 11 |
+
|
| 12 |
+
import numpy as np
|
| 13 |
+
import pytest
|
| 14 |
+
|
| 15 |
+
BENCH_DIR = pathlib.Path(__file__).resolve().parent
|
| 16 |
+
REPO_ROOT = BENCH_DIR.parent
|
| 17 |
+
if str(BENCH_DIR) not in sys.path:
|
| 18 |
+
sys.path.insert(0, str(BENCH_DIR))
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def _load_vectors(path: pathlib.Path):
|
| 22 |
+
raw = json.loads(path.read_text(encoding="utf-8"))
|
| 23 |
+
dtypes = raw["_meta"]["dtypes"]
|
| 24 |
+
return {k: np.array(v, dtype=np.dtype(dtypes[k])) for k, v in raw["vectors"].items()}
|
| 25 |
+
|
| 26 |
+
from generate_canonical_charts import ( # noqa: E402
|
| 27 |
+
BOOTSTRAP_RESAMPLES,
|
| 28 |
+
BOOTSTRAP_SEED,
|
| 29 |
+
HAS_MATPLOTLIB,
|
| 30 |
+
assert_identical_denominator,
|
| 31 |
+
assert_non_circular_teacher_eval,
|
| 32 |
+
build_all_charts_and_manifest,
|
| 33 |
+
compute_paired_panel_stats,
|
| 34 |
+
verify_png_chunks_clean,
|
| 35 |
+
)
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def test_identical_denominator_lock_enforced():
|
| 39 |
+
"""Verify that any denominator mismatch across models on a panel raises AssertionError."""
|
| 40 |
+
v_1025 = np.ones(1025, dtype=np.float32)
|
| 41 |
+
v_1009 = np.ones(1009, dtype=np.float32) # Dropped 16 'unknown' items
|
| 42 |
+
v_2835 = np.ones(2835, dtype=np.float32)
|
| 43 |
+
v_2825 = np.ones(2825, dtype=np.float32) # Dropped 10 'unknown' items
|
| 44 |
+
v_545 = np.ones(545, dtype=np.float32)
|
| 45 |
+
v_849 = np.ones(849, dtype=np.float32) # Mismatched audio language split
|
| 46 |
+
|
| 47 |
+
assert assert_identical_denominator("ok_1025", {"ultra": v_1025, "gpt": v_1025}, expected_n=1025) == 1025
|
| 48 |
+
|
| 49 |
+
with pytest.raises(AssertionError, match="Identical Denominator Lock violated"):
|
| 50 |
+
assert_identical_denominator("bad_beyond_unknown_drop", {"ultra": v_1025, "gpt": v_1009})
|
| 51 |
+
|
| 52 |
+
with pytest.raises(AssertionError, match="Identical Denominator Lock violated"):
|
| 53 |
+
assert_identical_denominator("bad_oct15_unknown_drop", {"ultra": v_2835, "mimmo": v_2825})
|
| 54 |
+
|
| 55 |
+
with pytest.raises(AssertionError, match="Identical Denominator Lock violated"):
|
| 56 |
+
assert_identical_denominator("bad_audio_split_mix", {"ultra": v_545, "gpt": v_849})
|
| 57 |
+
|
| 58 |
+
with pytest.raises(AssertionError, match="Expected N=636"):
|
| 59 |
+
assert_identical_denominator("bad_expected_n", {"ultra": np.ones(650), "gpt": np.ones(650)}, expected_n=636)
|
| 60 |
+
|
| 61 |
+
v_nan = v_1025.copy()
|
| 62 |
+
v_nan[0] = np.nan
|
| 63 |
+
with pytest.raises(AssertionError, match="NaN"):
|
| 64 |
+
assert_identical_denominator("bad_nan", {"ultra": v_1025, "gpt": v_nan})
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
def test_anti_circularity_lock_enforced():
|
| 68 |
+
"""Verify that scoring a teacher model against its own circular majority vote raises AssertionError."""
|
| 69 |
+
d = _load_vectors(BENCH_DIR / "canonical_eval_vectors.json")
|
| 70 |
+
|
| 71 |
+
with pytest.raises(AssertionError, match="Anti-Circularity Lock violated"):
|
| 72 |
+
assert_non_circular_teacher_eval("OpenAI GPT (Fast Tier)", eval_mode="circular_majority", self_vote_weight=0.333)
|
| 73 |
+
|
| 74 |
+
with pytest.raises(AssertionError, match="Anti-Circularity Lock violated"):
|
| 75 |
+
assert_non_circular_teacher_eval("Google Gemini Flash", eval_mode="leave_one_teacher_out", self_vote_weight=0.5)
|
| 76 |
+
|
| 77 |
+
gpt_audit = assert_non_circular_teacher_eval(
|
| 78 |
+
"OpenAI GPT (Fast Tier)",
|
| 79 |
+
eval_mode="leave_one_teacher_out",
|
| 80 |
+
self_vote_weight=0.0,
|
| 81 |
+
noncirc_vec=d["beyond_cat11_corr_gpt_noncirc"],
|
| 82 |
+
circ_vec=d["beyond_cat11_corr_gpt_circ"],
|
| 83 |
+
)
|
| 84 |
+
assert gpt_audit["non_circular_pct"] == 95.12
|
| 85 |
+
assert gpt_audit["circular_self_vote_pct"] == 96.59
|
| 86 |
+
assert gpt_audit["circularity_inflation_pp"] > 1.0
|
| 87 |
+
|
| 88 |
+
gem_audit = assert_non_circular_teacher_eval(
|
| 89 |
+
"Google Gemini Flash",
|
| 90 |
+
eval_mode="leave_one_teacher_out",
|
| 91 |
+
self_vote_weight=0.0,
|
| 92 |
+
noncirc_vec=d["beyond_cat11_corr_gemini_noncirc"],
|
| 93 |
+
circ_vec=d["beyond_cat11_corr_gemini_circ"],
|
| 94 |
+
)
|
| 95 |
+
assert gem_audit["non_circular_pct"] == 93.80
|
| 96 |
+
assert gem_audit["circular_self_vote_pct"] == 95.61
|
| 97 |
+
assert gem_audit["circularity_inflation_pp"] > 1.5
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def test_eval_vectors_privacy_and_dtypes():
|
| 101 |
+
"""Verify that canonical_eval_vectors.json contains only numeric int16/float32 arrays and zero strings."""
|
| 102 |
+
npz_path = BENCH_DIR / "canonical_eval_vectors.json"
|
| 103 |
+
assert npz_path.is_file()
|
| 104 |
+
d = _load_vectors(npz_path)
|
| 105 |
+
assert len(list(d.keys())) >= 140
|
| 106 |
+
for k in list(d.keys()):
|
| 107 |
+
arr = d[k]
|
| 108 |
+
assert arr.dtype in (np.int16, np.float32), f"Array {k} has disallowed dtype {arr.dtype}"
|
| 109 |
+
assert arr.ndim == 1, f"Array {k} must be 1D, got shape {arr.shape}"
|
| 110 |
+
assert not np.isnan(arr).any(), f"Array {k} contains NaNs"
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def test_paired_bootstrap_and_mcnemar_reproducibility():
|
| 114 |
+
"""Verify that paired bootstrap 95% CIs and McNemar p-values are 100% deterministic."""
|
| 115 |
+
d = _load_vectors(BENCH_DIR / "canonical_eval_vectors.json")
|
| 116 |
+
vecs = {
|
| 117 |
+
"Armiger Ultra": (d["oct15_aud545_pred_ultra"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 118 |
+
"Armiger INT8": (d["oct15_aud545_pred_int8_hyb"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 119 |
+
"GuessIt": (d["oct15_aud545_pred_guessit"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 120 |
+
"OpenAI GPT (Fast Tier)": (d["oct15_aud545_pred_gpt"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 121 |
+
}
|
| 122 |
+
res1 = compute_paired_panel_stats("test_aud545", vecs, ref_model="Armiger Ultra", expected_n=545)
|
| 123 |
+
res2 = compute_paired_panel_stats("test_aud545", vecs, ref_model="Armiger Ultra", expected_n=545)
|
| 124 |
+
assert res1 == res2
|
| 125 |
+
for m, st in res1["models"].items():
|
| 126 |
+
assert st["n"] == 545
|
| 127 |
+
assert st["ci95_lo"] <= st["point_pct"] <= st["ci95_hi"]
|
| 128 |
+
assert 0.0 <= st["p_value_vs_ref"] <= 1.0
|
| 129 |
+
assert res1["models"]["Armiger Ultra"]["point_pct"] == 97.80
|
| 130 |
+
assert res1["models"]["GuessIt"]["point_pct"] == 94.86
|
| 131 |
+
assert res1["models"]["OpenAI GPT (Fast Tier)"]["point_pct"] == 94.68
|
| 132 |
+
assert res1["models"]["OpenAI GPT (Fast Tier)"]["p_value_vs_ref"] < 0.01
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
def test_manifest_sha256_and_png_chunks(tmp_path: pathlib.Path):
|
| 136 |
+
"""Verify CHARTS_MANIFEST.json provenance hashes, PNG chunk hygiene, and byte-exact determinism."""
|
| 137 |
+
manifest_path = BENCH_DIR / "CHARTS_MANIFEST.json"
|
| 138 |
+
assert manifest_path.is_file(), "CHARTS_MANIFEST.json missing"
|
| 139 |
+
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
| 140 |
+
|
| 141 |
+
assert manifest["bootstrap_seed"] == BOOTSTRAP_SEED
|
| 142 |
+
assert manifest["bootstrap_resamples"] == BOOTSTRAP_RESAMPLES
|
| 143 |
+
for k, v in manifest["invariants_verified"].items():
|
| 144 |
+
assert v is True, f"Invariant {k} not verified"
|
| 145 |
+
|
| 146 |
+
# Verify all provenance SHA-256 hashes match disk
|
| 147 |
+
for rel_path, expected_sha in manifest["provenance_sha256"].items():
|
| 148 |
+
target = (BENCH_DIR / rel_path) if (BENCH_DIR / rel_path).is_file() else (REPO_ROOT / rel_path)
|
| 149 |
+
assert target.is_file(), f"Provenance file missing: {rel_path}"
|
| 150 |
+
actual_sha = hashlib.sha256(target.read_bytes()).hexdigest()
|
| 151 |
+
assert actual_sha == expected_sha, f"SHA-256 mismatch for {rel_path}: {actual_sha} != {expected_sha}"
|
| 152 |
+
|
| 153 |
+
# Verify all 4 PNGs have only IHDR, IDAT, IEND chunks and RGB mode
|
| 154 |
+
for chart_name, chart_meta in manifest["charts"].items():
|
| 155 |
+
png_path = REPO_ROOT / "assets" / chart_name
|
| 156 |
+
assert png_path.is_file(), f"Chart missing: {png_path}"
|
| 157 |
+
chunks = verify_png_chunks_clean(png_path)
|
| 158 |
+
assert chunks[0] == "IHDR" and chunks[-1] == "IEND"
|
| 159 |
+
assert set(chunks).issubset({"IHDR", "IDAT", "IEND"})
|
| 160 |
+
|
| 161 |
+
# Re-run pipeline into tmp_path and verify statistical + PNG determinism
|
| 162 |
+
tmp_assets = tmp_path / "assets"
|
| 163 |
+
tmp_manifest = tmp_path / "CHARTS_MANIFEST.json"
|
| 164 |
+
render_mode = bool(HAS_MATPLOTLIB and sys.platform == "darwin")
|
| 165 |
+
re_manifest = build_all_charts_and_manifest(
|
| 166 |
+
BENCH_DIR / "canonical_eval_vectors.json",
|
| 167 |
+
tmp_assets,
|
| 168 |
+
tmp_manifest,
|
| 169 |
+
render_pngs=render_mode,
|
| 170 |
+
)
|
| 171 |
+
assert re_manifest["invariants_verified"] == manifest["invariants_verified"]
|
| 172 |
+
assert re_manifest["anti_circularity_audit"] == manifest["anti_circularity_audit"]
|
| 173 |
+
assert re_manifest["five_split_category_summary"] == manifest["five_split_category_summary"]
|
| 174 |
+
for chart_name, chart_meta in manifest["charts"].items():
|
| 175 |
+
assert re_manifest["charts"][chart_name] == chart_meta, (
|
| 176 |
+
f"Non-deterministic manifest or PNG output for {chart_name}"
|
| 177 |
+
)
|
| 178 |
+
|
| 179 |
+
|
| 180 |
+
def test_readme_tables_match_charts_manifest():
|
| 181 |
+
"""Verify README.md contains canonical CHARTS_MANIFEST.json metrics and zero stale numbers."""
|
| 182 |
+
readme_path = REPO_ROOT / "README.md"
|
| 183 |
+
assert readme_path.is_file()
|
| 184 |
+
text = readme_path.read_text(encoding="utf-8")
|
| 185 |
+
|
| 186 |
+
# Canonical numbers that must appear in README
|
| 187 |
+
for required_token in ("95.49%", "97.30%", "96.53%", "97.80%", "96.83%"):
|
| 188 |
+
assert required_token in text, f"Missing canonical metric {required_token} in {readme_path}"
|
| 189 |
+
|
| 190 |
+
# Stale pre-audit numbers that must never reappear in README
|
| 191 |
+
for stale_token in ("97.13%", "96.35%", "72.35%", "35.29%", "97.55%"):
|
| 192 |
+
assert stale_token not in text, f"Stale metric {stale_token} found in {readme_path}"
|
config.json
CHANGED
|
@@ -62,16 +62,16 @@
|
|
| 62 |
},
|
| 63 |
"artifacts": {
|
| 64 |
"csam_guard.py": {
|
| 65 |
-
"size_bytes":
|
| 66 |
-
"sha256": "
|
| 67 |
},
|
| 68 |
"armiger_dht_db_sync.py": {
|
| 69 |
-
"size_bytes":
|
| 70 |
-
"sha256": "
|
| 71 |
},
|
| 72 |
"test_dht_db_sync.py": {
|
| 73 |
"size_bytes": 62482,
|
| 74 |
-
"sha256": "
|
| 75 |
}
|
| 76 |
}
|
| 77 |
}
|
|
|
|
| 62 |
},
|
| 63 |
"artifacts": {
|
| 64 |
"csam_guard.py": {
|
| 65 |
+
"size_bytes": 44802,
|
| 66 |
+
"sha256": "b76eff993f86f2a08c2aecaa39a4109b57f40be1d1b2b0066803e10cb33a0f96"
|
| 67 |
},
|
| 68 |
"armiger_dht_db_sync.py": {
|
| 69 |
+
"size_bytes": 69631,
|
| 70 |
+
"sha256": "27107c7a49e79ab0874c50881e42d742dc91d57dc049d11898dd4d04efa9bed1"
|
| 71 |
},
|
| 72 |
"test_dht_db_sync.py": {
|
| 73 |
"size_bytes": 62482,
|
| 74 |
+
"sha256": "c6a58c19ca3d698405132f46bde74ce038639b08c06ba9f00f2cee1f1dfdaf8f"
|
| 75 |
}
|
| 76 |
}
|
| 77 |
}
|
csam_guard.py
CHANGED
|
@@ -848,9 +848,13 @@ class WOTSPlusSHAKE256:
|
|
| 848 |
return digits
|
| 849 |
|
| 850 |
@classmethod
|
| 851 |
-
@functools.lru_cache(maxsize=256)
|
| 852 |
def generate_keypair(cls, seed: Optional[bytes] = None) -> Tuple[bytes, bytes]:
|
| 853 |
raw = seed if seed is not None else os.urandom(64)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 854 |
if len(raw) == 32:
|
| 855 |
raw = hashlib.shake_256(b"armiger-wots-keygen-v1:" + raw).digest(64)
|
| 856 |
elif len(raw) != 64:
|
|
|
|
| 848 |
return digits
|
| 849 |
|
| 850 |
@classmethod
|
|
|
|
| 851 |
def generate_keypair(cls, seed: Optional[bytes] = None) -> Tuple[bytes, bytes]:
|
| 852 |
raw = seed if seed is not None else os.urandom(64)
|
| 853 |
+
return cls._generate_keypair_from_seed(raw)
|
| 854 |
+
|
| 855 |
+
@classmethod
|
| 856 |
+
@functools.lru_cache(maxsize=256)
|
| 857 |
+
def _generate_keypair_from_seed(cls, raw: bytes) -> Tuple[bytes, bytes]:
|
| 858 |
if len(raw) == 32:
|
| 859 |
raw = hashlib.shake_256(b"armiger-wots-keygen-v1:" + raw).digest(64)
|
| 860 |
elif len(raw) != 64:
|
docs/ANONYMOUS_OPT_IN_DHT.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Armiger Distributed Anonymous & Opt-In-By-Default DHT Sharing
|
| 2 |
|
| 3 |
-
This document specifies the cryptographic, differential-privacy, and wire-level firewall architecture governing distributed weight, calibration, and catalog metadata sharing in **Armiger** and **Armiger Ultra** (`armiger_dht_rsi.py`, `armiger_dht_db_sync.py`, `armiger_ultra_pipeline.py`), along with the machine-verifiable results from the
|
| 4 |
|
| 5 |
---
|
| 6 |
|
|
@@ -61,7 +61,7 @@ Whenever an operator explicitly opts in (`opt_in_sharing=True` or `ARMIGER_DHT_S
|
|
| 61 |
|
| 62 |
---
|
| 63 |
|
| 64 |
-
## 3.
|
| 65 |
|
| 66 |
Reproduce all three proofs locally with:
|
| 67 |
|
|
@@ -74,4 +74,4 @@ pytest test_anonymous_opt_in_dht_sharing.py
|
|
| 74 |
| :--- | :--- | :--- | :--- |
|
| 75 |
| **Proof 1: Default Zero-Egress** | 9 default components, 100 synthetic webhook/catalog events | `opt_in_sharing=False` on 9/9; `0` bundles & `0` UDP `put` packets emitted (`2,600` outbound attempts blocked) | **PASS** |
|
| 76 |
| **Proof 2: Wire Canary Firewall** | 500 adversarial PII/canary payloads across 10 categories on 12-node UDP swarm | `500 / 500` (`100.0%`) blocked before wire; `0 / 500` (`0.0%`) canary substrings in raw UDP datagrams | **PASS** |
|
| 77 |
-
| **Proof 3: Unlinkability & DP Convergence** | 12-node UDP swarm over 5 epochs (60 publications) | `60 / 60` unique ephemeral Ed25519 keys (`8.33%` random linkability); `100%` hour-bucketed timestamps; MIA accuracy `0.
|
|
|
|
| 1 |
# Armiger Distributed Anonymous & Opt-In-By-Default DHT Sharing
|
| 2 |
|
| 3 |
+
This document specifies the cryptographic, differential-privacy, and wire-level firewall architecture governing distributed weight, calibration, and catalog metadata sharing in **Armiger** and **Armiger Ultra** (`armiger_dht_rsi.py`, `armiger_dht_db_sync.py`, `armiger_ultra_pipeline.py`), along with the machine-verifiable results from the 12-node local UDP BEP-44 proof & synthetic DP convergence suite (`benchmarks/proof_anonymous_opt_in_sharing.py`).
|
| 4 |
|
| 5 |
---
|
| 6 |
|
|
|
|
| 61 |
|
| 62 |
---
|
| 63 |
|
| 64 |
+
## 3. 12-Node Local UDP BEP-44 & Synthetic DP Convergence Verification Summary
|
| 65 |
|
| 66 |
Reproduce all three proofs locally with:
|
| 67 |
|
|
|
|
| 74 |
| :--- | :--- | :--- | :--- |
|
| 75 |
| **Proof 1: Default Zero-Egress** | 9 default components, 100 synthetic webhook/catalog events | `opt_in_sharing=False` on 9/9; `0` bundles & `0` UDP `put` packets emitted (`2,600` outbound attempts blocked) | **PASS** |
|
| 76 |
| **Proof 2: Wire Canary Firewall** | 500 adversarial PII/canary payloads across 10 categories on 12-node UDP swarm | `500 / 500` (`100.0%`) blocked before wire; `0 / 500` (`0.0%`) canary substrings in raw UDP datagrams | **PASS** |
|
| 77 |
+
| **Proof 3: Unlinkability & Synthetic DP Convergence** | 12-node local UDP swarm over 5 epochs (60 publications, synthetic adapter + temperature convergence) | `60 / 60` unique ephemeral Ed25519 keys (`8.33%` random linkability); `100%` hour-bucketed timestamps; MIA empirical accuracy `0.590`; ECE `0.08094 -> 0.01368` (`-83.10%`); Ultra F1 `0.93427 -> 0.98004` | **PASS** |
|
docs/CHART_PIPELINE.md
ADDED
|
@@ -0,0 +1,57 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Canonical Statistical Chart & Benchmark Pipeline (`v2.1.0`)
|
| 2 |
+
|
| 3 |
+
Every benchmark figure and headline accuracy table across the **Armiger** and **Armiger Ultra** repositories is generated deterministically from anonymized evaluation vectors (`benchmarks/canonical_eval_vectors.json`) and locked by a cryptographic provenance manifest (`benchmarks/CHARTS_MANIFEST.json`).
|
| 4 |
+
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
## 1. Pipeline Architecture & Artifacts
|
| 8 |
+
|
| 9 |
+
| Artifact | SHA-256 Digest | Description |
|
| 10 |
+
| :--- | :--- | :--- |
|
| 11 |
+
| `benchmarks/canonical_eval_vectors.json` | `058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a` | Plain-text JSON containing `148` 1D numeric `int16`/`float32` prediction and ground-truth label arrays across all 5 evaluation splits (`N=10,262`). Contains **zero** strings, release titles, file paths, or infohashes. |
|
| 12 |
+
| `benchmarks/generate_canonical_charts.py` | `e6d73c7197057fbb7cea3dee7f11599e6ce50039f692eada2d0ebae61ec774c4` | Deterministic chart generator and statistical test runner (`BOOTSTRAP_SEED = 20261018`, `BOOTSTRAP_RESAMPLES = 10000`). |
|
| 13 |
+
| `benchmarks/CHARTS_MANIFEST.json` | *(Tracked by `test_chart_integrity.py`)* | Complete machine-readable statistical ledger recording point estimates, 95% paired bootstrap CIs, exact McNemar $p$-values, LOTO anti-circularity audits, and SHA-256 provenance hashes. |
|
| 14 |
+
| `assets/armiger_vs_llms_overall_and_6dims.png` | `0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62` | 6-dimension accuracy & composite mean comparison against frontier cloud LLMs and local GPU LLMs on identical evaluation cohorts. |
|
| 15 |
+
| `assets/armiger_vs_llms_cost_comparison.png` | `3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4` | Operational cost per 1M torrents and token/hardware rate comparison. |
|
| 16 |
+
| `assets/armiger_vs_torrent_classifiers_6dims.png` | `e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010` | 7-dimension comparison against open-source torrent classifiers and regex parsers (`MIMMO BERT`, `RTN`, `Parsett`, `GuessIt`, `PTN`, `Anitomy`, `Bitmagnet`). |
|
| 17 |
+
| `assets/armiger_categorical_breakdown.png` | `1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98` | Per-category recall breakdown across all 10 media & software categories on `oct15_confirmation` (`N=2,835`) and `beyond_confirmation` (`N=1,025`). |
|
| 18 |
+
|
| 19 |
+
---
|
| 20 |
+
|
| 21 |
+
## 2. Four Machine-Enforced Statistical Invariants
|
| 22 |
+
|
| 23 |
+
`benchmarks/test_chart_integrity.py` enforces four strict invariants on every CI run:
|
| 24 |
+
|
| 25 |
+
### Invariant 1 — Identical Denominator Lock (`assert_identical_denominator`)
|
| 26 |
+
Every model compared on a benchmark panel must be evaluated on the **exact same item mask and denominator $N$**. Dropping abstentions (`unknown`), mixing evaluation splits, or evaluating on a subset raises an immediate `AssertionError`:
|
| 27 |
+
- `oct15_confirmation` 11-Class Category: $N = 2,835$
|
| 28 |
+
- `beyond_confirmation` 11-Class Category & Catalog Matching: $N = 1,025$
|
| 29 |
+
- `oct15_confirmation` Movie-vs-TV & Catalog Matching: $N = 1,296$
|
| 30 |
+
- `oct15_confirmation` Binary English-vs-Foreign Audio: $N = 545$ (`beyond_confirmation` strict audio: $N = 229$)
|
| 31 |
+
- `oct15_confirmation` Anime Detection: $N = 507$ (`beyond_confirmation` valid media: $N = 636$)
|
| 32 |
+
- `oct15_confirmation` Junk / Malware Gate: $N = 2,170$ (`beyond_confirmation`: $N = 977$)
|
| 33 |
+
- `Chart 1 Panel 2F` Swarm Liveliness AUROC: $N = 757$ Public-Tracker Cohort across all 6 compared systems (`Chart 3 Panel 6B`: $N = 10,278$ Multi-Source Corpus across all 3 compared methods)
|
| 34 |
+
|
| 35 |
+
### Invariant 2 — Anti-Circularity Lock (`assert_non_circular_teacher_eval`)
|
| 36 |
+
On `beyond_confirmation` ($N = 1,025$), where multi-teacher consensus contributed to reference curation, teacher models (`OpenAI GPT (Fast Tier)` and `Google Gemini Flash`) are strictly evaluated against **Leave-One-Teacher-Out (LOTO)** reference keys (`self_vote_weight = 0.0`):
|
| 37 |
+
|
| 38 |
+
| Teacher Model | Non-Circular LOTO Accuracy (`self_vote_weight = 0.0`) | Circular Self-Voted Accuracy | Removed Circularity Inflation |
|
| 39 |
+
| :--- | :---: | :---: | :---: |
|
| 40 |
+
| `OpenAI GPT (Fast Tier)` | **`95.12%`** | `96.59%` | `+1.46 pp` |
|
| 41 |
+
| `Google Gemini Flash` | **`93.80%`** | `95.61%` | `+1.80 pp` |
|
| 42 |
+
|
| 43 |
+
Scoring any teacher model with `self_vote_weight > 0.0` or `eval_mode != "leave_one_teacher_out"` raises an `AssertionError`.
|
| 44 |
+
|
| 45 |
+
### Invariant 3 — Paired Bootstrap 95% CIs, Hájek/DeLong AUROC CIs & Exact McNemar $p$-Values (`compute_paired_panel_stats`)
|
| 46 |
+
All accuracy uncertainty intervals and significance tests are paired by item index across `10,000` bootstrap resamples (`seed = 20261018`) using integer-scaled `float64` dot products (eliminating BLAS thread non-determinism across macOS ARM64 and Linux x86_64), two-sample Hájek/DeLong rank-placement variance for AUROC panels, and exact binomial `math.comb` McNemar two-sided $p$-values.
|
| 47 |
+
|
| 48 |
+
### Invariant 4 — Cryptographic PNG Hygiene (`verify_png_chunks_clean`)
|
| 49 |
+
Every generated PNG is stripped of non-deterministic metadata (`tEXt`, `iTXt`, `zTXt`, `tIME`, `pHYs`) so only `IHDR`, `IDAT`, and `IEND` chunks remain, guaranteeing byte-exact reproducibility.
|
| 50 |
+
|
| 51 |
+
---
|
| 52 |
+
|
| 53 |
+
## 3. Running Verification Locally
|
| 54 |
+
|
| 55 |
+
```bash
|
| 56 |
+
pytest benchmarks/test_chart_integrity.py -v
|
| 57 |
+
```
|
docs/INTEGRATION_GUIDE.md
ADDED
|
@@ -0,0 +1,282 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Armiger & Armiger Ultra Ecosystem Integration Guide
|
| 2 |
+
|
| 3 |
+
This guide provides step-by-step instructions, turnkey Docker Compose stacks, and reference HTTP adapters for integrating **Armiger** (`8.3 MB INT8 CPU`) and **Armiger Ultra** (`32.3 MB FP32 GPU/CPU` + 6 official plugins) with:
|
| 4 |
+
|
| 5 |
+
1. **[Part A — Integrating with `DH-Waypoint`](#part-a--integrating-armiger-with-dh-waypoint)** (`https://github.com/spencernorton/DH-Waypoint`):
|
| 6 |
+
- Native `dh-waypoint.processor.v1` processor contract (`GET /v1/manifest`, `GET /v1/status`, `POST /v1/process`, and batch `POST /v1/classify`)
|
| 7 |
+
- Pre-ingest `dh-waypoint.processors.admission.v1` cohort admission gate (`POST /v1/admit`)
|
| 8 |
+
- Closed-loop 4-tier `dh-waypoint.processors.feedback.v1` outcome telemetry (`POST /v1/feedback`)
|
| 9 |
+
2. **[Part B — Integrating with `Bitmagnet`](#part-b--integrating-armiger-with-bitmagnet)**:
|
| 10 |
+
- Pre-ingest / post-crawl batch classifier & malware/honeypot pruner (`examples/bitmagnet/bitmagnet_armiger_bridge.py`)
|
| 11 |
+
- GraphQL `torrentDelete` / content enrichment bridge & Torznab proxy enrichment
|
| 12 |
+
3. **[Part C — Integrating with Torznab Index Consumers (`Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`) & `qBittorrent`](#part-c--integrating-armiger-with-torznab-index-consumers--qbittorrent)**:
|
| 13 |
+
- Pre-grab Torznab filtering proxy (`examples/arr-stack/armiger_torznab_proxy.py`) blocking disguised `.exe`/`.scr`/`.lnk` malware, fake-seeder honeypot traps, dead swarms, and unrequested foreign-language audio tracks before grab
|
| 14 |
+
- Post-import webhook & `ffprobe` physical ground-truth receiver (`examples/arr-stack/armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier` for local conformal calibration and opt-in anonymous `BEP-44`/`BEP-46` RSI knowledge sharing
|
| 15 |
+
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## Architecture Overview
|
| 19 |
+
|
| 20 |
+
```mermaid
|
| 21 |
+
flowchart LR
|
| 22 |
+
subgraph Indexers["DHT Indexers & Crawlers"]
|
| 23 |
+
DHW["DH-Waypoint Core<br/>(processorapi/v1)"]
|
| 24 |
+
BM["Bitmagnet<br/>(GraphQL + Torznab)"]
|
| 25 |
+
PR["Prowlarr<br/>(Indexer Manager)"]
|
| 26 |
+
end
|
| 27 |
+
|
| 28 |
+
subgraph ArmigerSuite["Armiger / Armiger Ultra"]
|
| 29 |
+
PROC["Armiger Waypoint Processor<br/>(/v1/manifest, /v1/process, /v1/classify, /v1/admit, /v1/feedback)"]
|
| 30 |
+
BMBR["Bitmagnet Armiger Bridge<br/>(/v1/bitmagnet/classify)"]
|
| 31 |
+
TPROXY["Armiger Torznab Proxy<br/>(Port 8097 Admission Gate)"]
|
| 32 |
+
WHOOK["Armiger Webhook & ffprobe Receiver<br/>(Port 8098 Ground Truth + BEP-44 RSI)"]
|
| 33 |
+
end
|
| 34 |
+
|
| 35 |
+
subgraph ArrStack["*Arr Media Managers & Download Client"]
|
| 36 |
+
SON["Sonarr / Radarr / Lidarr / Readarr"]
|
| 37 |
+
QBT["qBittorrent / Transmission"]
|
| 38 |
+
end
|
| 39 |
+
|
| 40 |
+
DHW <-->|"HTTP processorapi/v1"| PROC
|
| 41 |
+
BM <-->|"Batch Classify + GraphQL Prune"| BMBR
|
| 42 |
+
DHW -->|"Torznab Feed"| TPROXY
|
| 43 |
+
BM -->|"Torznab Feed"| TPROXY
|
| 44 |
+
PR -->|"Torznab Feed"| TPROXY
|
| 45 |
+
TPROXY -->|"Sanitized & Enriched XML"| SON
|
| 46 |
+
SON -->|"Grab Approved Release"| QBT
|
| 47 |
+
SON -->|"On Import Webhook"| WHOOK
|
| 48 |
+
QBT -->|"On Completion + ffprobe"| WHOOK
|
| 49 |
+
WHOOK -.->|"Local Calibration & Opt-In BEP-44 RSI"| PROC
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
---
|
| 53 |
+
|
| 54 |
+
## Part A — Integrating Armiger with `DH-Waypoint`
|
| 55 |
+
|
| 56 |
+
[`DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint) includes a built-in external processor subsystem (`processorapi/v1`) that connects directly to [`examples/dh-waypoint/armiger_waypoint_processor.py`](../examples/dh-waypoint/armiger_waypoint_processor.py).
|
| 57 |
+
|
| 58 |
+
### 1. Turnkey Docker Compose Stack
|
| 59 |
+
|
| 60 |
+
Launch `DH-Waypoint`, `PostgreSQL`, and `Armiger Ultra` together using [`examples/dh-waypoint/docker-compose.yml`](../examples/dh-waypoint/docker-compose.yml):
|
| 61 |
+
|
| 62 |
+
```bash
|
| 63 |
+
export POSTGRES_PASSWORD="<postgres-password>"
|
| 64 |
+
docker compose -f examples/dh-waypoint/docker-compose.yml up -d
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
### 2. Drop-In Plugin Manifest (`07-armiger-ultra.plugin.yml`)
|
| 68 |
+
|
| 69 |
+
Mount [`examples/dh-waypoint/07-armiger-ultra.plugin.yml`](../examples/dh-waypoint/07-armiger-ultra.plugin.yml) into `DH_WAYPOINT_PROCESSORS_PLUGIN_DIR` (`/etc/dh-waypoint/plugins.d/07-armiger-ultra.plugin.yml`):
|
| 70 |
+
|
| 71 |
+
```yaml
|
| 72 |
+
processor:
|
| 73 |
+
name: armiger-ultra
|
| 74 |
+
endpoint: http://armiger-ultra:8095/v1/classify
|
| 75 |
+
timeout: 5s
|
| 76 |
+
max_batch: 64
|
| 77 |
+
required: false
|
| 78 |
+
forward_files: true
|
| 79 |
+
max_files: 64
|
| 80 |
+
forward_trackers: true
|
| 81 |
+
forward_swarm: true
|
| 82 |
+
|
| 83 |
+
admission:
|
| 84 |
+
enabled: true
|
| 85 |
+
endpoint: http://armiger-ultra:8095/v1/admit
|
| 86 |
+
timeout: 3s
|
| 87 |
+
max_batch: 64
|
| 88 |
+
max_files: 64
|
| 89 |
+
fail_open: true
|
| 90 |
+
|
| 91 |
+
learning:
|
| 92 |
+
enabled: true
|
| 93 |
+
feedback_endpoint: http://armiger-ultra:8095/v1/feedback
|
| 94 |
+
feedback_timeout: 5s
|
| 95 |
+
feedback_batch_size: 64
|
| 96 |
+
flush_interval: 15s
|
| 97 |
+
queue_size: 2048
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
### 3. Equivalent `DH_WAYPOINT_*` Environment Variables
|
| 101 |
+
|
| 102 |
+
If you configure `DH-Waypoint` via environment variables instead of `plugins.d`:
|
| 103 |
+
|
| 104 |
+
| Variable | Recommended Value | Purpose |
|
| 105 |
+
| :--- | :--- | :--- |
|
| 106 |
+
| `DH_WAYPOINT_PROCESSORS_HTTP_CHAIN` | `armiger-ultra=http://armiger-ultra:8095/v1/classify` | Ordered HTTP classification chain (`dh-waypoint.processor.v1`). |
|
| 107 |
+
| `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_FILES` | `true` | Forwards physical file paths and byte sizes to Armiger's File-Tree Encoder. |
|
| 108 |
+
| `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_TRACKERS` | `true` | Forwards tracker domain hints to `SwarmPriors`. |
|
| 109 |
+
| `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_SWARM` | `true` | Forwards seeder/leecher counts to `SwarmSurvivalNet`. |
|
| 110 |
+
| `DH_WAYPOINT_PROCESSORS_ADMISSION_ENABLED` | `true` | Enables pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`). |
|
| 111 |
+
| `DH_WAYPOINT_PROCESSORS_ADMISSION_ENDPOINT` | `http://armiger-ultra:8095/v1/admit` | Pre-ingest admission endpoint. |
|
| 112 |
+
| `DH_WAYPOINT_LEARNING_ENABLED` | `true` | Enables the 4-tier `dh-waypoint.processors.feedback.v1` outcome loop. |
|
| 113 |
+
| `DH_WAYPOINT_LEARNING_FEEDBACK_ENDPOINT` | `http://armiger-ultra:8095/v1/feedback` | Delivers `t1_discovery` through `t4_playback` feedback batches to Armiger. |
|
| 114 |
+
|
| 115 |
+
---
|
| 116 |
+
|
| 117 |
+
## Part B — Integrating Armiger with `Bitmagnet`
|
| 118 |
+
|
| 119 |
+
Upstream `Bitmagnet` uses deterministic keyword/extension rules (`35.17%` 11-class accuracy on `N=10,262` audited torrents). Deploying [`examples/bitmagnet/bitmagnet_armiger_bridge.py`](../examples/bitmagnet/bitmagnet_armiger_bridge.py) upgrades a `Bitmagnet` deployment to **`95.49%` 11-class accuracy** (`97.60%` 6-dimension composite accuracy) and **`100%` fake-seeder malware trap recall**.
|
| 120 |
+
|
| 121 |
+
### 1. Turnkey `Bitmagnet` + `Armiger` Docker Compose Stack
|
| 122 |
+
|
| 123 |
+
Start the stack in [`examples/bitmagnet/docker-compose.yml`](../examples/bitmagnet/docker-compose.yml):
|
| 124 |
+
|
| 125 |
+
```bash
|
| 126 |
+
export POSTGRES_PASSWORD="<postgres-password>"
|
| 127 |
+
docker compose -f examples/bitmagnet/docker-compose.yml up -d
|
| 128 |
+
```
|
| 129 |
+
|
| 130 |
+
This launches two companion services alongside `bitmagnet`:
|
| 131 |
+
- **`armiger-bitmagnet-bridge` (`port 8096`)**: Evaluates batches of newly crawled Bitmagnet torrents (`POST /v1/bitmagnet/classify`), maps Armiger's 11-class predictions (`movie`, `tv_show`, `music`, `ebook`, `audiobook`, `comic`, `software`, `game`, `xxx`) and tags (`armiger:<category>`, `armiger:lang:<lang>`, `armiger:viability:<tier>`) onto Bitmagnet items, and automatically executes Bitmagnet's GraphQL `torrent.delete` mutation against disguised malware payloads (`.mkv.exe`, `.scr`, `.lnk`, `.vbs`) and fake-seeder honeypots.
|
| 132 |
+
- **`armiger-torznab-proxy` (`port 8097`)**: Wraps `http://bitmagnet:3333/torznab` with real-time neural admission filtering before `Prowlarr`, `Sonarr`, or `Radarr` sees search results.
|
| 133 |
+
|
| 134 |
+
### 2. Calling the Bitmagnet Classifier Bridge Directly
|
| 135 |
+
|
| 136 |
+
```bash
|
| 137 |
+
curl -sS -X POST http://localhost:8096/v1/bitmagnet/classify \
|
| 138 |
+
-H "Content-Type: application/json" \
|
| 139 |
+
-d '{
|
| 140 |
+
"torrents": [
|
| 141 |
+
{
|
| 142 |
+
"infoHash": "1111222233334444555566667777888899990000",
|
| 143 |
+
"name": "Sintel.2010.2160p.UHD.Master.HEVC.FLAC.5.1-OPEN",
|
| 144 |
+
"size": 74500000000,
|
| 145 |
+
"seeders": 320,
|
| 146 |
+
"leechers": 18,
|
| 147 |
+
"files": [{"path": "Sintel.2010.2160p.mkv", "size": 74500000000}]
|
| 148 |
+
},
|
| 149 |
+
{
|
| 150 |
+
"infoHash": "aaaa22223333444455556666777788889999bbbb",
|
| 151 |
+
"name": "Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe",
|
| 152 |
+
"size": 15400000,
|
| 153 |
+
"seeders": 4500,
|
| 154 |
+
"leechers": 12,
|
| 155 |
+
"files": [{"path": "Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe", "size": 15400000}]
|
| 156 |
+
}
|
| 157 |
+
]
|
| 158 |
+
}'
|
| 159 |
+
```
|
| 160 |
+
|
| 161 |
+
The bridge classifies the first item as `contentType: "movie"` (`viabilityTier: "alive_viable"`) and flags the second item (`Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe`) for immediate deletion via `graphqlDeleteMutation`.
|
| 162 |
+
|
| 163 |
+
---
|
| 164 |
+
|
| 165 |
+
## Part C — Integrating Armiger with Torznab Index Consumers & `qBittorrent`
|
| 166 |
+
|
| 167 |
+
Armiger integrates with standard Torznab XML/RSS index consumers (`Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`) and BitTorrent clients (`qBittorrent`, `Transmission`) at **two stages**:
|
| 168 |
+
1. **Stage 1 — Pre-Grab Torznab Filtering Proxy (`port 8097`):** Blocks malware masquerades, fake-seeder honeypots, dead swarms, and unrequested foreign-language audio releases *before* any `*Arr` app grabs them.
|
| 169 |
+
2. **Stage 2 — Post-Import Webhook & `ffprobe` Ground-Truth Receiver (`port 8098`):** Verifies completed downloads and `ffprobe` media streams to calibrate local Mondrian thresholds and optionally contribute differential-privacy blinded telemetry to the `BEP-44`/`BEP-46` RSI network.
|
| 170 |
+
|
| 171 |
+
### 1. Stage 1: Wiring the Torznab Filtering Proxy (`Prowlarr` / `Sonarr` / `Radarr` / `Lidarr` / `Readarr`)
|
| 172 |
+
|
| 173 |
+
Start [`examples/arr-stack/docker-compose.yml`](../examples/arr-stack/docker-compose.yml):
|
| 174 |
+
|
| 175 |
+
```bash
|
| 176 |
+
export UPSTREAM_TORZNAB_URL="http://dh-waypoint:3333/torznab"
|
| 177 |
+
docker compose -f examples/arr-stack/docker-compose.yml up -d
|
| 178 |
+
```
|
| 179 |
+
|
| 180 |
+
#### Configuring in `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, or `Readarr`:
|
| 181 |
+
1. Open **Settings -> Indexers -> + -> Torznab (Custom)**.
|
| 182 |
+
2. Set **URL** to `http://armiger-torznab-proxy:8097` (instead of pointing directly at `http://bitmagnet:3333/torznab` or `http://dh-waypoint:3333/torznab`).
|
| 183 |
+
3. Every Torznab search query (`?t=search`, `?t=tvsearch`, `?t=movie`, `?t=music`, `?t=book`) is transparently forwarded upstream, batch-scored by `ArmigerUltraPipeline`, stripped of unsafe/honeypot/dead/wrong-language items, and annotated with custom `<torznab:attr>` tags:
|
| 184 |
+
- `armiger_category` (`movie`, `tv_show`, `music`, `audiobook`, `ebook`, `comic`, `software`, `game`, `xxx`)
|
| 185 |
+
- `armiger_lang` (`english`, `non_english`, `multi_or_unknown`)
|
| 186 |
+
- `armiger_viability` (`alive_strong`, `alive_viable`, `marginal`, `dead`)
|
| 187 |
+
- `armiger_survival_prob` (`0.0000`–`1.0000`)
|
| 188 |
+
|
| 189 |
+
#### Torznab Proxy Environment Controls
|
| 190 |
+
|
| 191 |
+
| Variable | Default | Description |
|
| 192 |
+
| :--- | :---: | :--- |
|
| 193 |
+
| `UPSTREAM_TORZNAB_URL` | `http://dh-waypoint:3333/torznab` | Upstream Torznab base URL (`DH-Waypoint`, `Bitmagnet`, or `Prowlarr`). |
|
| 194 |
+
| `ARMIGER_VARIANT` | `int8` | `int8` (`8.3 MB` CPU fast-path, `2.44 ms` p50) or `fp32` (`32.3 MB` GPU/CPU). |
|
| 195 |
+
| `ARMIGER_REJECT_DEAD_SWARMS` | `true` | Drops Torznab items whose `SwarmSurvivalNet` viability tier is `dead`. |
|
| 196 |
+
| `ARMIGER_REQUIRE_ENGLISH_AUDIO` | `false` | When `true`, blocks releases classified as `non_english` audio by `AudioLanguageGuard`. |
|
| 197 |
+
| `ARMIGER_ALLOW_ENGLISH_SUBS` | `true` | When `ARMIGER_REQUIRE_ENGLISH_AUDIO=true`, permits foreign-audio anime/films that carry verified English subtitles (`has_eng_subs`). |
|
| 198 |
+
|
| 199 |
+
### 2. Stage 2: Wiring Post-Import Webhooks (`Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `qBittorrent`)
|
| 200 |
+
|
| 201 |
+
`armiger-arr-webhook-receiver` (`port 8098`) exposes dedicated endpoints backed by the adapters in [`armiger_dht_rsi.py`](../armiger_dht_rsi.py):
|
| 202 |
+
|
| 203 |
+
| Application | Where to Configure in UI | Webhook URL | Underlying Armiger Adapter |
|
| 204 |
+
| :--- | :--- | :--- | :--- |
|
| 205 |
+
| **Sonarr** | `Settings -> Connect -> + -> Webhook` (`On Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/sonarr` | `SeriesWebhookAdapter` |
|
| 206 |
+
| **Radarr** | `Settings -> Connect -> + -> Webhook` (`On Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/radarr` | `MovieWebhookAdapter` |
|
| 207 |
+
| **Lidarr** | `Settings -> Connect -> + -> Webhook` (`On Release Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/lidarr` | `AudioWebhookAdapter` |
|
| 208 |
+
| **Readarr** | `Settings -> Connect -> + -> Webhook` (`On Release Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/readarr` | `PublicationWebhookAdapter` |
|
| 209 |
+
| **qBittorrent** | `Options -> Downloads -> Run external program` | `POST http://armiger-arr-webhook-receiver:8098/webhooks/qbittorrent` | `QBittorrentAdapter` |
|
| 210 |
+
|
| 211 |
+
#### `qBittorrent` Completion Command
|
| 212 |
+
|
| 213 |
+
In `qBittorrent` (`Tools -> Options -> Downloads -> Run external program on torrent completion`), configure:
|
| 214 |
+
|
| 215 |
+
```bash
|
| 216 |
+
curl -sS -X POST http://armiger-arr-webhook-receiver:8098/webhooks/qbittorrent -H "Content-Type: application/json" -d "{\"name\":\"%N\",\"hash\":\"%I\",\"content_path\":\"%F\",\"size\":%Z,\"progress\":1.0,\"state\":\"uploading\"}"
|
| 217 |
+
```
|
| 218 |
+
|
| 219 |
+
#### Programmatic Python Usage (`SeriesWebhookAdapter`, `MovieWebhookAdapter`, `QBittorrentAdapter`, `PhysicalGroundTruthVerifier`)
|
| 220 |
+
|
| 221 |
+
You can also invoke the webhook adapters and `ffprobe` verifier directly in Python via `ArmigerUltraPipeline.ingest_rsi_feedback`:
|
| 222 |
+
|
| 223 |
+
```python
|
| 224 |
+
from armiger_ultra_pipeline import ArmigerUltraPipeline
|
| 225 |
+
|
| 226 |
+
pipeline = ArmigerUltraPipeline.from_directory(".", variant="int8")
|
| 227 |
+
|
| 228 |
+
# 1. Ingest a native Sonarr "Download" (Import) webhook + ffprobe output
|
| 229 |
+
ok, reason, record = pipeline.ingest_rsi_feedback(
|
| 230 |
+
{
|
| 231 |
+
"eventType": "Download",
|
| 232 |
+
"series": {"title": "Cosmos Laundromat", "tvdbId": 310123},
|
| 233 |
+
"episodes": [{"seasonNumber": 1, "episodeNumber": 1}],
|
| 234 |
+
"release": {
|
| 235 |
+
"releaseTitle": "Cosmos.Laundromat.S01E01.1080p.WEB-DL.DDP5.1.H.264-OPEN",
|
| 236 |
+
"size": 1_450_000_000,
|
| 237 |
+
},
|
| 238 |
+
"episodeFile": {
|
| 239 |
+
"relativePath": "Season 01/Cosmos.Laundromat.S01E01.mkv",
|
| 240 |
+
"size": 1_450_000_000,
|
| 241 |
+
"mediaInfo": {
|
| 242 |
+
"videoCodec": "h264",
|
| 243 |
+
"audioFormat": "eac3",
|
| 244 |
+
"audioChannels": 5.1,
|
| 245 |
+
"audioLanguages": "eng",
|
| 246 |
+
"subtitles": "eng",
|
| 247 |
+
},
|
| 248 |
+
},
|
| 249 |
+
},
|
| 250 |
+
adapter="series_webhook",
|
| 251 |
+
)
|
| 252 |
+
assert ok and reason == "accepted_and_queued_with_poisson_jitter"
|
| 253 |
+
print(record.blinded_stem.blinded_hash, record.category, record.lang)
|
| 254 |
+
|
| 255 |
+
# 2. Ingest a native Radarr "Download" (Import) webhook
|
| 256 |
+
ok, reason, record = pipeline.ingest_rsi_feedback(
|
| 257 |
+
{
|
| 258 |
+
"eventType": "Download",
|
| 259 |
+
"movie": {"title": "Sintel", "year": 2010, "tmdbId": 45745},
|
| 260 |
+
"release": {
|
| 261 |
+
"releaseTitle": "Sintel.2010.2160p.UHD.Master.HEVC.FLAC.5.1-OPEN",
|
| 262 |
+
"size": 24_800_000_000,
|
| 263 |
+
},
|
| 264 |
+
"movieFile": {
|
| 265 |
+
"relativePath": "Sintel (2010)/Sintel.2010.2160p.mkv",
|
| 266 |
+
"size": 24_800_000_000,
|
| 267 |
+
},
|
| 268 |
+
},
|
| 269 |
+
adapter="movie_webhook",
|
| 270 |
+
)
|
| 271 |
+
assert ok and record.category == "movie"
|
| 272 |
+
```
|
| 273 |
+
|
| 274 |
+
### 3. `BEP-44` / `BEP-46` Privacy & RSI Participation Modes
|
| 275 |
+
|
| 276 |
+
All adapters enforce `PhysicalGroundTruthVerifier` (never trusting unverified model predictions) and `TelemetryAnonymizer` (stripping tracker passkeys, local paths, and IP addresses; blinding title stems via post-quantum `HMAC-SHA256` + `SHAKE-256`; applying Laplace differential privacy $\epsilon = 1.0$; and scheduling outbound packets with Poisson timing jitter):
|
| 277 |
+
|
| 278 |
+
| `ARMIGER_DHT_RSI_MODE` | Behavior |
|
| 279 |
+
| :--- | :--- |
|
| 280 |
+
| `off` | Disables all `BEP-44`/`BEP-46` DHT traffic; calibration and ground-truth verification stay strictly local. |
|
| 281 |
+
| `pull_only` *(Default)* | Verifies and applies `M`-of-`N` hybrid post-quantum (`Ed25519` + `WOTS+-SHAKE256`) signed calibration manifests from the DHT after passing the 500-item golden canary gate; never transmits telemetry. |
|
| 282 |
+
| `anonymous_contribute` | Opts in to transmitting differential-privacy blinded, hybrid post-quantum (`X25519` + `ML-KEM-768` + `ChaCha20-Poly1305`) encrypted ground-truth observations. |
|