Align chart cohorts/labels, cryptographic bindings, and legal/documentation precision
Browse files- LEGAL.md +58 -0
- LICENSE +190 -0
- NOTICE +47 -0
- README.md +21 -21
- armiger_dht_db_sync.py +27 -5
- armiger_dht_rsi.py +5 -1
- assets/armiger_categorical_breakdown.png +2 -2
- assets/armiger_vs_llms_cost_comparison.png +2 -2
- assets/armiger_vs_llms_overall_and_6dims.png +2 -2
- assets/armiger_vs_torrent_classifiers_6dims.png +2 -2
- benchmarks/CHARTS_MANIFEST.json +104 -103
- benchmarks/generate_canonical_charts.py +74 -66
- benchmarks/test_chart_integrity.py +7 -7
- config.json +26 -97
- csam_guard.py +5 -1
- docs/ANONYMOUS_OPT_IN_DHT.md +3 -3
- docs/CHART_PIPELINE.md +12 -11
- docs/INTEGRATION_GUIDE.md +11 -11
- docs/base-model.md +71 -0
- docs/plugins/audio-guard.md +93 -0
- docs/plugins/calibration-dht.md +98 -0
- docs/plugins/dht-db-sync.md +188 -0
- docs/plugins/swarm-priors.md +85 -0
- docs/plugins/swarm-survival.md +90 -0
- docs/plugins/zero-cue-catalog.md +89 -0
- examples/README.md +1 -1
- examples/arr-stack/armiger_torznab_proxy.py +1 -1
- examples/arr-stack/docker-compose.yml +4 -4
- examples/dh-waypoint/armiger_waypoint_processor.py +164 -22
- examples/dh_waypoint_processor/armiger_waypoint_processor.py +412 -0
- test_chart_pipeline_integrity.py +3 -0
LEGAL.md
ADDED
|
@@ -0,0 +1,58 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Legal, Copyright, and Acceptable Use Policy
|
| 2 |
+
|
| 3 |
+
This document outlines the legal, copyright, privacy, and content-safety obligations governing the **Armiger** and **Armiger Ultra** models, plugins, and SDK (`spencernorton/armiger`).
|
| 4 |
+
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
## 1. Purpose and Intended Use
|
| 8 |
+
|
| 9 |
+
Armiger is a metadata-only structural and linguistic classification engine designed for:
|
| 10 |
+
- **Index hygiene and taxonomy classification:** Categorizing BitTorrent and decentralized storage metadata (filenames, file sizes, directory structures, and extension distributions) across 11 content categories.
|
| 11 |
+
- **Malware, dropper, and honeypot defense:** Detecting disguised executable payloads (`.mkv.exe`, `.scr`, `.lnk`, DLL-sideloading traps, and fake-seeder honeypots) before any file transfer occurs.
|
| 12 |
+
- **Content safety and compliance enforcement:** Enforcing strict pre-ingest Child Sexual Abuse Material (CSAM) blocking, non-consensual abuse filtering, cryptographic tombstone revocations, and Mondrian conformal adult-content isolation.
|
| 13 |
+
|
| 14 |
+
Armiger **does not** host, index, link to, download, distribute, or provide access to any media files, `.torrent` payloads, magnet URIs, or copyrighted works.
|
| 15 |
+
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## 2. Copyright & Intellectual Property Compliance
|
| 19 |
+
|
| 20 |
+
1. **Zero Copyrighted Content or Plaintext Titles:**
|
| 21 |
+
- Neither the model weights (`armiger.int8.onnx`, `armiger_ultra.fp32.onnx`, `armiger_ultra.pt`, `swarm_adapter.pt`) nor the Zero-Cue Catalog (`validated_zero_cue_catalog.json`) contain or redistribute copyrighted media files or plaintext title strings.
|
| 22 |
+
- All lookup keys in `validated_zero_cue_catalog.json` and all release-tag token tables in `armiger_inference.py` and `swarm_priors.py` are stored as irreversible, one-way **domain-separated 64-bit BLAKE2b cryptographic hashes** (`digest_size=8`, `person=b"ArmigZCC"`).
|
| 23 |
+
2. **Creative Commons & Public-Domain Test Fixtures:**
|
| 24 |
+
- Every test fixture, code example, and documentation sample in this repository exclusively uses open-source software distributions (e.g., Ubuntu ISOs), public-domain literature and early cinema (*Project Gutenberg*, *LibriVox* US public-domain recordings, Georges Méliès 1902/1906 films), Creative Commons Zero recordings (*Open Goldberg Variations*), or Creative Commons Attribution open movies and comics (*Blender Studio Open Movies*, *Pepper&Carrot*).
|
| 25 |
+
3. **No Circumvention or Infringement:**
|
| 26 |
+
- Armiger does not bypass technological protection measures (DRM), access private trackers, or facilitate copyright infringement. Users are solely responsible for ensuring that their deployment of Armiger complies with applicable copyright laws in their jurisdiction (including 17 U.S.C. § 512 and EU Directive 2019/790).
|
| 27 |
+
|
| 28 |
+
---
|
| 29 |
+
|
| 30 |
+
## 3. Privacy & Anonymization Invariants
|
| 31 |
+
|
| 32 |
+
1. **Private-Tracker Exclusion (`swarm_priors.py`):**
|
| 33 |
+
- Any announce URL, source tag, or metadata string containing passkeys, authkeys, user tokens, or private-tracker identifiers is automatically rejected and stripped before feature extraction or telemetry processing.
|
| 34 |
+
2. **Zero-PII Telemetry (`armiger_dht_rsi.py` & `armiger_dht_db_sync.py`):**
|
| 35 |
+
- Local filesystem paths, usernames, IPv4/IPv6 addresses, MAC addresses, and indexer credentials are stripped by `TelemetryAnonymizer` prior to any optional DHT synchronization.
|
| 36 |
+
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## 4. Child Safety & Zero-Tolerance CSAM Policy
|
| 40 |
+
|
| 41 |
+
1. **Mandatory Pre-Ingest Blocking (`csam_guard.py`):**
|
| 42 |
+
- `CSAMGuard` performs multi-stage Unicode NFKD decomposition, zero-width character stripping, Cyrillic/Greek homoglyph normalization, and leetspeak deobfuscation before checking against zero-tolerance CSAM, underage exploitation, non-consensual abuse, and extreme-violence rules.
|
| 43 |
+
2. **Cryptographic Tombstone Revocation:**
|
| 44 |
+
- `TombstoneStore` enforces Ed25519-signed blinded SHA-256 tombstone lists with monotonic sequence numbers, immediately purging and permanently blocking any revoked hash across participating nodes.
|
| 45 |
+
|
| 46 |
+
---
|
| 47 |
+
|
| 48 |
+
## 5. Notice and Takedown / Contact Procedure
|
| 49 |
+
|
| 50 |
+
If you believe any material in this repository infringes your copyright or trademark rights, or if you wish to report a safety or compliance concern, please contact:
|
| 51 |
+
|
| 52 |
+
- **Security & Compliance Advisories:** <https://github.com/spencernorton/armiger/security/advisories/new>
|
| 53 |
+
- **Repository Owner:** [@spencernorton](https://github.com/spencernorton)
|
| 54 |
+
|
| 55 |
+
Please include:
|
| 56 |
+
1. Identification of the specific file, line, or artifact in question;
|
| 57 |
+
2. A description of the copyrighted work or right claimed to be infringed;
|
| 58 |
+
3. Your contact information and a statement of good-faith belief.
|
LICENSE
ADDED
|
@@ -0,0 +1,190 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Apache License
|
| 2 |
+
Version 2.0, January 2004
|
| 3 |
+
http://www.apache.org/licenses/
|
| 4 |
+
|
| 5 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 6 |
+
|
| 7 |
+
1. Definitions.
|
| 8 |
+
|
| 9 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 10 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 11 |
+
|
| 12 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 13 |
+
the copyright owner that is granting the License.
|
| 14 |
+
|
| 15 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 16 |
+
other entities that control, are controlled by, or are under common
|
| 17 |
+
control with that entity. For the purposes of this definition,
|
| 18 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 19 |
+
direction or management of such entity, whether by contract or
|
| 20 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 21 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 22 |
+
|
| 23 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 24 |
+
exercising permissions granted by this License.
|
| 25 |
+
|
| 26 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 27 |
+
including but not limited to software source code, documentation
|
| 28 |
+
source, and configuration files.
|
| 29 |
+
|
| 30 |
+
"Object" form shall mean any form resulting from mechanical
|
| 31 |
+
transformation or translation of a Source form, including but
|
| 32 |
+
not limited to compiled object code, generated documentation,
|
| 33 |
+
and conversions to other media types.
|
| 34 |
+
|
| 35 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 36 |
+
Object form, made available under the License, as indicated by a
|
| 37 |
+
copyright notice that is included in or attached to the work
|
| 38 |
+
(an example is provided in the Appendix below).
|
| 39 |
+
|
| 40 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 41 |
+
form, that is based on (or derived from) the Work and for which the
|
| 42 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 43 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 44 |
+
of this License, Derivative Works shall not include works that remain
|
| 45 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 46 |
+
the Work and Derivative Works thereof.
|
| 47 |
+
|
| 48 |
+
"Contribution" shall mean any work of authorship, including
|
| 49 |
+
the original version of the Work and any modifications or additions
|
| 50 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 51 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 52 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 53 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 54 |
+
means any form of electronic, verbal, or written communication sent
|
| 55 |
+
to the Licensor or its representatives, including but not limited to
|
| 56 |
+
communication on electronic mailing lists, source code control systems,
|
| 57 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 58 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 59 |
+
excluding communication that is conspicuously marked or otherwise
|
| 60 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 61 |
+
|
| 62 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 63 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 64 |
+
subsequently incorporated within the Work.
|
| 65 |
+
|
| 66 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 67 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 68 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 69 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 70 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 71 |
+
Work and such Derivative Works in Source or Object form.
|
| 72 |
+
|
| 73 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 74 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 75 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 76 |
+
(except as stated in this section) patent license to make, have made,
|
| 77 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 78 |
+
where such license applies only to those patent claims licensable
|
| 79 |
+
by such Contributor that are necessarily infringed by their
|
| 80 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 81 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 82 |
+
institute patent litigation against any entity (including a
|
| 83 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 84 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 85 |
+
or contributory patent infringement, then any patent licenses
|
| 86 |
+
granted to You under this License for that Work shall terminate
|
| 87 |
+
as of the date such litigation is filed.
|
| 88 |
+
|
| 89 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 90 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 91 |
+
modifications, and in Source or Object form, provided that You
|
| 92 |
+
meet the following conditions:
|
| 93 |
+
|
| 94 |
+
(a) You must give any other recipients of the Work or
|
| 95 |
+
Derivative Works a copy of this License; and
|
| 96 |
+
|
| 97 |
+
(b) You must cause any modified files to carry prominent notices
|
| 98 |
+
stating that You changed the files; and
|
| 99 |
+
|
| 100 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 101 |
+
that You distribute, all copyright, patent, trademark, and
|
| 102 |
+
attribution notices from the Source form of the Work,
|
| 103 |
+
excluding those notices that do not pertain to any part of
|
| 104 |
+
the Derivative Works; and
|
| 105 |
+
|
| 106 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 107 |
+
distribution, then any Derivative Works that You distribute must
|
| 108 |
+
include a readable copy of the attribution notices contained
|
| 109 |
+
within such NOTICE file, excluding those notices that do not
|
| 110 |
+
pertain to any part of the Derivative Works, in at least one
|
| 111 |
+
of the following places: within a NOTICE text file distributed
|
| 112 |
+
as part of the Derivative Works; within the Source form or
|
| 113 |
+
documentation, if provided along with the Derivative Works; or,
|
| 114 |
+
within a display generated by the Derivative Works, if and
|
| 115 |
+
wherever such third-party notices normally appear. The contents
|
| 116 |
+
of the NOTICE file are for informational purposes only and
|
| 117 |
+
do not modify the License. You may add Your own attribution
|
| 118 |
+
notices within Derivative Works that You distribute, alongside
|
| 119 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 120 |
+
that such additional attribution notices cannot be construed
|
| 121 |
+
as modifying the License.
|
| 122 |
+
|
| 123 |
+
You may add Your own copyright statement to Your modifications and
|
| 124 |
+
may provide additional or different license terms and conditions
|
| 125 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 126 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 127 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 128 |
+
the conditions stated in this License.
|
| 129 |
+
|
| 130 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 131 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 132 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 133 |
+
this License, without any additional terms or conditions.
|
| 134 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 135 |
+
the terms of any separate license agreement you may have executed
|
| 136 |
+
with Licensor regarding such Contributions.
|
| 137 |
+
|
| 138 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 139 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 140 |
+
except as required for reasonable and customary use in describing the
|
| 141 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 142 |
+
|
| 143 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 144 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 145 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 146 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 147 |
+
implied, including, without limitation, any warranties or conditions
|
| 148 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 149 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 150 |
+
appropriateness of using or redistributing the Work and assume any
|
| 151 |
+
risks associated with Your exercise of permissions under this License.
|
| 152 |
+
|
| 153 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 154 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 155 |
+
unless required by applicable law (such as deliberate and grossly
|
| 156 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 157 |
+
liable to You for damages, including any direct, indirect, special,
|
| 158 |
+
incidental, or consequential damages of any character arising as a
|
| 159 |
+
result of this License or out of the use or inability to use the
|
| 160 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 161 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 162 |
+
other commercial damages or losses), even if such Contributor
|
| 163 |
+
has been advised of the possibility of such damages.
|
| 164 |
+
|
| 165 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 166 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 167 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 168 |
+
or other liability obligations and/or rights consistent with this
|
| 169 |
+
License. However, in accepting such obligations, You may act only
|
| 170 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 171 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 172 |
+
defend, and hold each Contributor harmless for any liability
|
| 173 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 174 |
+
of your accepting any such warranty or additional liability.
|
| 175 |
+
|
| 176 |
+
END OF TERMS AND CONDITIONS
|
| 177 |
+
|
| 178 |
+
Copyright 2026 NorviTech / Spencer Norton
|
| 179 |
+
|
| 180 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 181 |
+
you may not use this file except in compliance with the License.
|
| 182 |
+
You may obtain a copy of the License at
|
| 183 |
+
|
| 184 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 185 |
+
|
| 186 |
+
Unless required by applicable law or agreed to in writing, software
|
| 187 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 188 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 189 |
+
See the License for the specific language governing permissions and
|
| 190 |
+
limitations under the License.
|
NOTICE
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Armiger & Armiger Ultra
|
| 2 |
+
Copyright 2026 NorviTech / Spencer Norton
|
| 3 |
+
|
| 4 |
+
This product includes software, neural network weights, calibration parameters,
|
| 5 |
+
and domain-separated hash lookup tables developed by NorviTech and released
|
| 6 |
+
under the Apache License, Version 2.0.
|
| 7 |
+
|
| 8 |
+
================================================================================
|
| 9 |
+
Creative Commons & Public-Domain Example Media Attributions
|
| 10 |
+
================================================================================
|
| 11 |
+
Media titles and filenames used in documentation examples and unit test fixtures
|
| 12 |
+
in this repository reference open-source, Creative Commons, or public-domain
|
| 13 |
+
works:
|
| 14 |
+
|
| 15 |
+
- "Big Buck Bunny" (2008), "Sintel" (2010), "Tears of Steel" (2012),
|
| 16 |
+
"Cosmos Laundromat" (2015), "Caminandes" (2013-2016), "Spring" (2019),
|
| 17 |
+
"Coffee Run" (2020), "Charge" (2022), "Wing It!" (2023), and
|
| 18 |
+
"Elephants Dream" (2006) are Open Movies produced by the Blender Foundation
|
| 19 |
+
(https://studio.blender.org/films/), licensed under Creative Commons
|
| 20 |
+
Attribution (CC-BY 3.0 / CC-BY 4.0).
|
| 21 |
+
- "Le Voyage dans la Lune" (1902) and "Les Quatre Cents Farces du Diable" (1906)
|
| 22 |
+
by Georges Melies are in the public domain.
|
| 23 |
+
- "Pepper&Carrot" is an open-source webcomic by David Revoy
|
| 24 |
+
(https://www.peppercarrot.com/), licensed under Creative Commons Attribution
|
| 25 |
+
4.0 International (CC-BY 4.0).
|
| 26 |
+
- "Open Goldberg Variations" (BWV 988) by J.S. Bach, performed by Kimiko Ishizaka,
|
| 27 |
+
is released into the public domain under Creative Commons Zero (CC0 1.0).
|
| 28 |
+
- "Pride and Prejudice" by Jane Austen (1813) is in the public domain worldwide;
|
| 29 |
+
LibriVox recordings (https://librivox.org/) are released into the public domain
|
| 30 |
+
in the United States (verify local copyright duration outside the U.S.).
|
| 31 |
+
|
| 32 |
+
================================================================================
|
| 33 |
+
Domain-Separated Hash Representation & Non-Affiliation Notice
|
| 34 |
+
================================================================================
|
| 35 |
+
- The Zero-Cue Title Catalog (`validated_zero_cue_catalog.json`) and release-tag
|
| 36 |
+
feature extractors store lookup keys as domain-separated 64-bit BLAKE2b digests
|
| 37 |
+
(`person=b"armiger-zc-v2"` / 8-byte hex digests) mapped to integer catalog IDs
|
| 38 |
+
and coarse category/language codes. No plaintext media titles or release-group
|
| 39 |
+
names are redistributed in the catalog file.
|
| 40 |
+
- BitTorrent, BEP-5, BEP-9, BEP-10, BEP-15, BEP-44, BEP-46, and BEP-51 are open
|
| 41 |
+
protocol specifications. All third-party product names, open-source project
|
| 42 |
+
names, and model names (including OpenAI, GPT, Anthropic, Claude, Google,
|
| 43 |
+
Gemini, GuessIt, RTN/Parsett, PTN, Anitomy, MIMMO, Bitmagnet, ONNX Runtime,
|
| 44 |
+
and PyTorch) referenced in comparative benchmarks or compatibility guides
|
| 45 |
+
remain the property of their respective owners. Use of these names is strictly
|
| 46 |
+
nominative for comparative technical evaluation or interoperability description
|
| 47 |
+
and implies no endorsement, sponsorship, or affiliation.
|
README.md
CHANGED
|
@@ -21,7 +21,7 @@ pipeline_tag: text-classification
|
|
| 21 |
|
| 22 |
<p align="center">
|
| 23 |
<strong>Unified torrent intelligence suite bundling Armiger (8.3 MB INT8 CPU), Armiger Ultra (32.3 MB FP32 GPU/CPU), the 48-tensor Swarm Health & Honeypot Adapter (<code>swarm_adapter.pt</code>), and all 6 official plugins.</strong><br>
|
| 24 |
-
Delivers
|
| 25 |
</p>
|
| 26 |
|
| 27 |
<p align="center">
|
|
@@ -36,7 +36,7 @@ pipeline_tag: text-classification
|
|
| 36 |
1. **`armiger_ultra.fp32.onnx` (`32.3 MB`) & `armiger_ultra.pt` (`32.2 MB`):** Full-precision GPU/CPU 8-head neural classifier from the hardened `H350000_Curated_Wave2-seed0` checkpoint (`0.21 ms/item` batch-64 on NVIDIA RTX GPU, `0.29 ms/item` batch-64 on Apple M4 Pro MPS GPU).
|
| 37 |
2. **`armiger.int8.onnx` (`8.3 MB`):** Dynamically quantized CPU fast-path engine from the identical checkpoint (`2.44 ms` single-item p50 / `3.07 ms/item` batch-64 on Apple M4 Pro CPU ONNX Runtime).
|
| 38 |
3. **`armiger_inference.py` + `vocab.json`:** Self-contained, zero-tokenizer 11-input featurizer (`title_bytes`, `ext_ids`, `file_sizes_log`, `file_depths`, `file_flags`, `file_mask`, `struct_feats`, `tracker_ids`, `cat_hint_id`, `tag_ids`, `swarm_feats`) for direct ONNX Runtime inference.
|
| 39 |
-
4. **`swarm_adapter.pt` (`132 KB`) & `swarm_survival.py`:** **48-Tensor PyTorch Swarm Health, Download Viability & Fake-Seeder Honeypot Residual Adapter (`SwarmResidualAdapter`)**, fusing tracker/DHT seeder/leecher telemetry with content features to achieve **`96.83%` Multi-Source
|
| 40 |
5. **`validated_zero_cue_catalog.json` (`26.0 MB`) & `zero_cue_plugin.py`:** Privacy-hardened `BLAKE2b-64` (`person=b"ArmigZCC"`) 929,400-entry offline title/year/anime catalog + deterministic structural rules.
|
| 41 |
6. **`calibration.json`:** Post-hoc temperature scaling, Menon prior logit adjustments, and Mondrian conformal safety thresholds (`100.00%` junk accuracy on `oct15_confirmation` `N=2,170`).
|
| 42 |
|
|
@@ -48,7 +48,7 @@ All **6 Armiger plugins** are rolled directly into this `NorviTech/armiger-ultra
|
|
| 48 |
|
| 49 |
| Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
|
| 50 |
| :--- | :---: | :---: | :--- |
|
| 51 |
-
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source
|
| 52 |
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 53 |
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 54 |
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
|
@@ -69,7 +69,7 @@ pipeline = ArmigerUltraPipeline.from_directory("./armiger-ultra")
|
|
| 69 |
|
| 70 |
pred = pipeline.classify({
|
| 71 |
"infohash": "1111222233334444555566667777888899990000",
|
| 72 |
-
"name": "Sintel.2010.2160p.UHD.
|
| 73 |
"size": 74_500_000_000,
|
| 74 |
"files": [{"path": "Sintel.2010.2160p.mkv", "size": 74_500_000_000}],
|
| 75 |
"swarm": {
|
|
@@ -86,26 +86,26 @@ print(pred["swarm_viability"]["viability_tier"]) # "alive_viable" (computed via
|
|
| 86 |
print(list(pred["heads"].keys())) # All 11 unified heads
|
| 87 |
```
|
| 88 |
|
| 89 |
-
Run the bundled
|
| 90 |
```bash
|
| 91 |
-
pytest
|
| 92 |
```
|
| 93 |
|
| 94 |
---
|
| 95 |
|
| 96 |
-
## Ecosystem Integrations (`DH-Waypoint`, `Bitmagnet` &
|
| 97 |
|
| 98 |
Armiger and Armiger Ultra ship with turnkey HTTP services, Docker Compose stacks, and reference adapters in [`examples/`](examples) and [`docs/INTEGRATION_GUIDE.md`](docs/INTEGRATION_GUIDE.md):
|
| 99 |
|
| 100 |
- **[`DH-Waypoint` Integration (`examples/dh-waypoint/`)](examples/dh-waypoint)** — Drop-in `processorapi/v1` service (`armiger_waypoint_processor.py`) and plugin manifest (`07-armiger-ultra.plugin.yml`) implementing batch classification (`dh-waypoint.processor.v1`), pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop telemetry for [`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint).
|
| 101 |
- **[`Bitmagnet` Integration (`examples/bitmagnet/`)](examples/bitmagnet)** — Pre-ingest / post-crawl batch classifier and malware/honeypot pruner (`bitmagnet_armiger_bridge.py`) that upgrades upstream Bitmagnet from `35.17%` rule accuracy to **`95.49%` 11-class neural accuracy** and automatically prunes disguised `.mkv.exe`/`.scr`/`.lnk` malware traps via GraphQL `torrent.delete`.
|
| 102 |
-
- **[
|
| 103 |
|
| 104 |
See the **[Complete Ecosystem Integration Guide (`docs/INTEGRATION_GUIDE.md`)](docs/INTEGRATION_GUIDE.md)** for architecture diagrams, Docker Compose files, and step-by-step setup instructions.
|
| 105 |
|
| 106 |
### Documentation & Verification Index
|
| 107 |
|
| 108 |
-
- **[`docs/INTEGRATION_GUIDE.md`](docs/INTEGRATION_GUIDE.md)** — Turnkey integration architecture and Docker Compose guides for `DH-Waypoint`, `Bitmagnet`, `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`,
|
| 109 |
- **[`docs/ANONYMOUS_OPT_IN_DHT.md`](docs/ANONYMOUS_OPT_IN_DHT.md)** — Mathematical and wire-level specification for opt-in-by-default (`opt_in_sharing=False`, `ARMIGER_DHT_SHARE_OPT_IN="0"`) anonymous DHT weight and catalog sharing (`AnonymityGuard`, ephemeral `Ed25519`, coarse UTC hour bucketing, DP-clipped deltas, and `BLAKE2b-64` catalog hashes).
|
| 110 |
- **[`docs/CHART_PIPELINE.md`](docs/CHART_PIPELINE.md)** & **[`benchmarks/CHARTS_MANIFEST.json`](benchmarks/CHARTS_MANIFEST.json)** — Canonical statistical chart pipeline, non-circular LOTO teacher evaluation, paired bootstrap 95% CIs, exact McNemar $p$-values, and SHA-256 PNG provenance manifest.
|
| 111 |
- **[`benchmarks/anonymous_opt_in_proof_report.json`](benchmarks/anonymous_opt_in_proof_report.json)** — Reproducible cryptographic proof report (`0` packets emitted when opt-in is disabled; `500/500` synthetic PII/infohash/title/IP leaks blocked by `AnonymityGuard`).
|
|
@@ -147,7 +147,7 @@ Every metric below is reproducibly verifiable on the original unmutated evaluati
|
|
| 147 |
|
| 148 |
| System / Model | Included Here? | Footprint | 11-Class Category (`oct15` `N=2,835`) | Movie vs. TV (`N=1,296`) | Catalog Matching (`N=1,296`) | Audio Language (`N=545`) | Anime & Dub (`N=507` / `tc2000`) | Adult / NSFW Gate (`N=2,781`) | Junk / Malware (`N=2,170`) | Swarm Liveliness (AUROC) |
|
| 149 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 150 |
-
| **Armiger Ultra (Fair Hybrid)** | **Yes** | **`32.3 MB`** (`+26 MB` cat.) | **`95.49%`** (`98.47%` mlang) | **`97.30%`** | **`96.53%`** | **`97.80%`** | **`99.61%`** (`95.80%` tc2000) | **`98.53%`** | **`100.00%`** (`99.90%` beyond) | **`96.83%`**
|
| 151 |
| **Armiger (`armiger.int8.onnx` + Plugins)** | **Yes** | **`8.3 MB`** | **`95.49%`** (`95.06%` Raw) | **`97.30%`** (`96.60%` Raw) | **`96.53%`** (`95.83%` Raw) | **`97.80%`** (`96.88%` Raw) | **`99.61%`** (`95.80%` tc2000) | **`98.49%`** | **`100.00%`** (`99.90%` beyond) | **`92.42%`** Pub (`86.64%` Priors) |
|
| 152 |
| **MIMMO BERT (360M)** | — | `~1,400 MB` | `72.24%` | `76.00%` (`93.61%` 4-Med) | `60.26%` | N/A | N/A | `91.05%` | N/A | N/A |
|
| 153 |
| **MIMMO Cascade** | — | `~1,400 MB` | `71.32%` | `76.08%` (`93.97%` 4-Med) | `60.34%` | N/A | N/A | `89.68%` | N/A | N/A |
|
|
@@ -162,7 +162,7 @@ Every metric below is reproducibly verifiable on the original unmutated evaluati
|
|
| 162 |
|
| 163 |

|
| 164 |
|
| 165 |
-
| Content Category | `oct15` (`N=2,835`) Armiger Ultra (Fair Hybrid) | `oct15` (`N=2,835`) Armiger (INT8 + ZeroCue) | `oct15` MIMMO BERT (360M) | `oct15` Bitmagnet Built-In | `beyond` (`N=1,025`) Armiger Ultra (Fair Hybrid) | `beyond` (`N=1,025`) Armiger (INT8 + ZeroCue) | `beyond` OpenAI GPT (
|
| 166 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 167 |
| **Audiobook** | **`97.89%`** | **`97.89%`** | `0.00%` | `9.47%` | **`100.00%`** | **`100.00%`** | `98.67%` | `98.00%` |
|
| 168 |
| **Adult / XXX** | **`97.80%`** | **`97.80%`** | `95.50%` | `39.37%` | `91.95%` | `91.95%` | `94.83%` | **`95.69%`** |
|
|
@@ -175,7 +175,7 @@ Every metric below is reproducibly verifiable on the original unmutated evaluati
|
|
| 175 |
| **Course** | **`81.82%`** | **`81.82%`** | `0.00%` | `0.00%` | `92.00%` | `92.00%` | `95.50%` | **`97.50%`** |
|
| 176 |
| **Comic** | **`66.67%`** | **`66.67%`** | `0.00%` | `0.00%` | **`98.33%`** | **`98.33%`** | `96.67%` | `95.00%` |
|
| 177 |
|
| 178 |
-
*Note on `beyond_confirmation` (`N=1,025`): `
|
| 179 |
|
| 180 |
---
|
| 181 |
|
|
@@ -191,12 +191,12 @@ Every metric below is reproducibly verifiable on the original unmutated evaluati
|
|
| 191 |
|
| 192 |
| Model | Overall 6-Dim Mean (95% CI) | 11-Class Category (`N=1,025` LOTO / `oct15`) | Catalog Matching (`N=1,025` LOTO / `oct15`) | Audio Language (Strict E-vs-F `N=229` / `N=545`) | Anime & Dub (Valid Media `N=636` / `N=507`) | Junk / Malware (`N=977` Bal / Acc) | Swarm Liveliness (AUROC) | Measured Hardware Speed | Cost per 1M Torrents (Batched / Real-Time) |
|
| 193 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 194 |
-
| **Armiger Ultra (Fair Hybrid)** | **`97.60%`** (`[
|
| 195 |
| **[Armiger](https://huggingface.co/NorviTech/armiger) (`armiger.int8.onnx` + Plugins)** | **`96.85%`** (`[96.40, 97.30]`) | **`95.12%`** (`95.49%` `oct15` / `95.06%` Raw) | `96.29%` (`95.80%` Raw) | **`99.13%`** (`97.80%` `oct15` / `96.88%` Raw) | **`98.90%`** (`99.61%` `oct15`) | **`99.23%`** (`99.90%` Acc / **`100%`** `oct15`) | `92.42%` Pub (`86.64%` Priors) | **`2.44 ms` single / `3.07 ms` batch-64 (M4 Pro CPU)** | **`$0.00` API** (`$0.0020` compute) |
|
| 196 |
-
| **Google Gemini
|
| 197 |
-
| **Anthropic Claude
|
| 198 |
-
| **OpenAI GPT (
|
| 199 |
-
| **Local
|
| 200 |
|
| 201 |
---
|
| 202 |
|
|
@@ -205,15 +205,15 @@ Every metric below is reproducibly verifiable on the original unmutated evaluati
|
|
| 205 |
All benchmark charts and headline metrics across the Armiger `v2.1.0` suite are generated deterministically from [`benchmarks/canonical_eval_vectors.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/canonical_eval_vectors.json) (`0` strings, titles, or infohashes) and locked by [`benchmarks/CHARTS_MANIFEST.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/CHARTS_MANIFEST.json) under four machine-enforced invariants (`pytest benchmarks/test_chart_integrity.py`):
|
| 206 |
|
| 207 |
1. **Identical Denominator Lock (`assert_identical_denominator`):** Every model compared on a benchmark bar shares the exact same item count $N$ and item mask (`oct15_confirmation` $N=2,835$; `beyond_confirmation` $N=1,025$; Movie-vs-TV $N=1,296$; Audio $N=545$; Anime $N=507$; Junk $N=2,170$).
|
| 208 |
-
2. **Non-Circular Teacher Evaluation Lock (`assert_non_circular_teacher_eval`):** Frontier teacher models (`
|
| 209 |
-
3. **Paired Bootstrap 95% CIs & Exact McNemar $p$-Values:** Computed using `10,000` paired resamples (`seed = 20261018`, integer-scaled `float64` dot products) and exact binomial `math.comb` McNemar tests (`p = 1.42e-11` vs. `
|
| 210 |
-
4. **Deterministic PNG Chunk Stripping:** Output PNGs contain only `IHDR`, `IDAT`, and `IEND` chunks with pinned SHA-256 digests (`
|
| 211 |
|
| 212 |
---
|
| 213 |
|
| 214 |
## Privacy & Anonymous Opt-In-By-Default DHT Sharing (`opt_in_sharing=False`)
|
| 215 |
|
| 216 |
-
Distributed DHT intelligence sharing (`NorviTech/armiger-calibration-dht` and `NorviTech/armiger-dht-db-sync`) is **strictly opt-in by default** and **
|
| 217 |
|
| 218 |
| Guarantee | Default State | Wire-Level & Mathematical Enforcement |
|
| 219 |
| :--- | :--- | :--- |
|
|
|
|
| 21 |
|
| 22 |
<p align="center">
|
| 23 |
<strong>Unified torrent intelligence suite bundling Armiger (8.3 MB INT8 CPU), Armiger Ultra (32.3 MB FP32 GPU/CPU), the 48-tensor Swarm Health & Honeypot Adapter (<code>swarm_adapter.pt</code>), and all 6 official plugins.</strong><br>
|
| 24 |
+
Delivers 96.86% 6-dimension composite accuracy on the identical Public-Tracker cohort (97.60% with Multi-Source Corpus Swarm AUROC) on unmutated ground-truth keys, 0.21 ms/item batched on NVIDIA RTX GPU (0.29 ms on Apple M4 Pro MPS GPU, 2.44 ms single-item on M4 Pro CPU INT8), 100% fake-seeder malware trap recall, and 96.83% Multi-Source Corpus / 92.42% Public-Tracker swarm liveliness AUROC.
|
| 25 |
</p>
|
| 26 |
|
| 27 |
<p align="center">
|
|
|
|
| 36 |
1. **`armiger_ultra.fp32.onnx` (`32.3 MB`) & `armiger_ultra.pt` (`32.2 MB`):** Full-precision GPU/CPU 8-head neural classifier from the hardened `H350000_Curated_Wave2-seed0` checkpoint (`0.21 ms/item` batch-64 on NVIDIA RTX GPU, `0.29 ms/item` batch-64 on Apple M4 Pro MPS GPU).
|
| 37 |
2. **`armiger.int8.onnx` (`8.3 MB`):** Dynamically quantized CPU fast-path engine from the identical checkpoint (`2.44 ms` single-item p50 / `3.07 ms/item` batch-64 on Apple M4 Pro CPU ONNX Runtime).
|
| 38 |
3. **`armiger_inference.py` + `vocab.json`:** Self-contained, zero-tokenizer 11-input featurizer (`title_bytes`, `ext_ids`, `file_sizes_log`, `file_depths`, `file_flags`, `file_mask`, `struct_feats`, `tracker_ids`, `cat_hint_id`, `tag_ids`, `swarm_feats`) for direct ONNX Runtime inference.
|
| 39 |
+
4. **`swarm_adapter.pt` (`132 KB`) & `swarm_survival.py`:** **48-Tensor PyTorch Swarm Health, Download Viability & Fake-Seeder Honeypot Residual Adapter (`SwarmResidualAdapter`)**, fusing tracker/DHT seeder/leecher telemetry with content features to achieve **`96.83%` Multi-Source Corpus AUROC** (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%` / `93.03%` Public-Tracker AUROC** (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%` fake-seeder honeypot recall**, and **`0.00%` category flip rate**.
|
| 40 |
5. **`validated_zero_cue_catalog.json` (`26.0 MB`) & `zero_cue_plugin.py`:** Privacy-hardened `BLAKE2b-64` (`person=b"ArmigZCC"`) 929,400-entry offline title/year/anime catalog + deterministic structural rules.
|
| 41 |
6. **`calibration.json`:** Post-hoc temperature scaling, Menon prior logit adjustments, and Mondrian conformal safety thresholds (`100.00%` junk accuracy on `oct15_confirmation` `N=2,170`).
|
| 42 |
|
|
|
|
| 48 |
|
| 49 |
| Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
|
| 50 |
| :--- | :---: | :---: | :--- |
|
| 51 |
+
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Corpus AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
|
| 52 |
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 53 |
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 54 |
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
|
|
|
| 69 |
|
| 70 |
pred = pipeline.classify({
|
| 71 |
"infohash": "1111222233334444555566667777888899990000",
|
| 72 |
+
"name": "Sintel.2010.2160p.UHD.Master.HEVC.FLAC.5.1-OPEN",
|
| 73 |
"size": 74_500_000_000,
|
| 74 |
"files": [{"path": "Sintel.2010.2160p.mkv", "size": 74_500_000_000}],
|
| 75 |
"swarm": {
|
|
|
|
| 86 |
print(list(pred["heads"].keys())) # All 11 unified heads
|
| 87 |
```
|
| 88 |
|
| 89 |
+
Run the bundled 211-test verification suite:
|
| 90 |
```bash
|
| 91 |
+
pytest -q
|
| 92 |
```
|
| 93 |
|
| 94 |
---
|
| 95 |
|
| 96 |
+
## Ecosystem Integrations (`DH-Waypoint`, `Bitmagnet` & Torznab / Webhook Clients)
|
| 97 |
|
| 98 |
Armiger and Armiger Ultra ship with turnkey HTTP services, Docker Compose stacks, and reference adapters in [`examples/`](examples) and [`docs/INTEGRATION_GUIDE.md`](docs/INTEGRATION_GUIDE.md):
|
| 99 |
|
| 100 |
- **[`DH-Waypoint` Integration (`examples/dh-waypoint/`)](examples/dh-waypoint)** — Drop-in `processorapi/v1` service (`armiger_waypoint_processor.py`) and plugin manifest (`07-armiger-ultra.plugin.yml`) implementing batch classification (`dh-waypoint.processor.v1`), pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop telemetry for [`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint).
|
| 101 |
- **[`Bitmagnet` Integration (`examples/bitmagnet/`)](examples/bitmagnet)** — Pre-ingest / post-crawl batch classifier and malware/honeypot pruner (`bitmagnet_armiger_bridge.py`) that upgrades upstream Bitmagnet from `35.17%` rule accuracy to **`95.49%` 11-class neural accuracy** and automatically prunes disguised `.mkv.exe`/`.scr`/`.lnk` malware traps via GraphQL `torrent.delete`.
|
| 102 |
+
- **[Torznab Proxy & Media Manager Webhook Suite (`examples/arr-stack/`)](examples/arr-stack)** — Pre-grab Torznab XML filtering proxy (`armiger_torznab_proxy.py`) between `Prowlarr` / `Bitmagnet` / `DH-Waypoint` and `Sonarr`, `Radarr`, `Lidarr`, `Lidarr`, and `Readarr`, paired with a post-import webhook & `ffprobe` ground-truth receiver (`armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier`.
|
| 103 |
|
| 104 |
See the **[Complete Ecosystem Integration Guide (`docs/INTEGRATION_GUIDE.md`)](docs/INTEGRATION_GUIDE.md)** for architecture diagrams, Docker Compose files, and step-by-step setup instructions.
|
| 105 |
|
| 106 |
### Documentation & Verification Index
|
| 107 |
|
| 108 |
+
- **[`docs/INTEGRATION_GUIDE.md`](docs/INTEGRATION_GUIDE.md)** — Turnkey integration architecture and Docker Compose guides for `DH-Waypoint`, `Bitmagnet`, `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, and `qBittorrent`.
|
| 109 |
- **[`docs/ANONYMOUS_OPT_IN_DHT.md`](docs/ANONYMOUS_OPT_IN_DHT.md)** — Mathematical and wire-level specification for opt-in-by-default (`opt_in_sharing=False`, `ARMIGER_DHT_SHARE_OPT_IN="0"`) anonymous DHT weight and catalog sharing (`AnonymityGuard`, ephemeral `Ed25519`, coarse UTC hour bucketing, DP-clipped deltas, and `BLAKE2b-64` catalog hashes).
|
| 110 |
- **[`docs/CHART_PIPELINE.md`](docs/CHART_PIPELINE.md)** & **[`benchmarks/CHARTS_MANIFEST.json`](benchmarks/CHARTS_MANIFEST.json)** — Canonical statistical chart pipeline, non-circular LOTO teacher evaluation, paired bootstrap 95% CIs, exact McNemar $p$-values, and SHA-256 PNG provenance manifest.
|
| 111 |
- **[`benchmarks/anonymous_opt_in_proof_report.json`](benchmarks/anonymous_opt_in_proof_report.json)** — Reproducible cryptographic proof report (`0` packets emitted when opt-in is disabled; `500/500` synthetic PII/infohash/title/IP leaks blocked by `AnonymityGuard`).
|
|
|
|
| 147 |
|
| 148 |
| System / Model | Included Here? | Footprint | 11-Class Category (`oct15` `N=2,835`) | Movie vs. TV (`N=1,296`) | Catalog Matching (`N=1,296`) | Audio Language (`N=545`) | Anime & Dub (`N=507` / `tc2000`) | Adult / NSFW Gate (`N=2,781`) | Junk / Malware (`N=2,170`) | Swarm Liveliness (AUROC) |
|
| 149 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 150 |
+
| **Armiger Ultra (Fair Hybrid)** | **Yes** | **`32.3 MB`** (`+26 MB` cat.) | **`95.49%`** (`98.47%` mlang) | **`97.30%`** | **`96.53%`** | **`97.80%`** | **`99.61%`** (`95.80%` tc2000) | **`98.53%`** | **`100.00%`** (`99.90%` beyond) | **`96.83%`** Corpus / **`92.42%`** Pub |
|
| 151 |
| **Armiger (`armiger.int8.onnx` + Plugins)** | **Yes** | **`8.3 MB`** | **`95.49%`** (`95.06%` Raw) | **`97.30%`** (`96.60%` Raw) | **`96.53%`** (`95.83%` Raw) | **`97.80%`** (`96.88%` Raw) | **`99.61%`** (`95.80%` tc2000) | **`98.49%`** | **`100.00%`** (`99.90%` beyond) | **`92.42%`** Pub (`86.64%` Priors) |
|
| 152 |
| **MIMMO BERT (360M)** | — | `~1,400 MB` | `72.24%` | `76.00%` (`93.61%` 4-Med) | `60.26%` | N/A | N/A | `91.05%` | N/A | N/A |
|
| 153 |
| **MIMMO Cascade** | — | `~1,400 MB` | `71.32%` | `76.08%` (`93.97%` 4-Med) | `60.34%` | N/A | N/A | `89.68%` | N/A | N/A |
|
|
|
|
| 162 |
|
| 163 |

|
| 164 |
|
| 165 |
+
| Content Category | `oct15` (`N=2,835`) Armiger Ultra (Fair Hybrid) | `oct15` (`N=2,835`) Armiger (INT8 + ZeroCue) | `oct15` MIMMO BERT (360M) | `oct15` Bitmagnet Built-In | `beyond` (`N=1,025`) Armiger Ultra (Fair Hybrid) | `beyond` (`N=1,025`) Armiger (INT8 + ZeroCue) | `beyond` OpenAI GPT (Fast Tier, LOTO)* | `beyond` Google Gemini Flash (LOTO)* |
|
| 166 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 167 |
| **Audiobook** | **`97.89%`** | **`97.89%`** | `0.00%` | `9.47%` | **`100.00%`** | **`100.00%`** | `98.67%` | `98.00%` |
|
| 168 |
| **Adult / XXX** | **`97.80%`** | **`97.80%`** | `95.50%` | `39.37%` | `91.95%` | `91.95%` | `94.83%` | **`95.69%`** |
|
|
|
|
| 175 |
| **Course** | **`81.82%`** | **`81.82%`** | `0.00%` | `0.00%` | `92.00%` | `92.00%` | `95.50%` | **`97.50%`** |
|
| 176 |
| **Comic** | **`66.67%`** | **`66.67%`** | `0.00%` | `0.00%` | **`98.33%`** | **`98.33%`** | `96.67%` | `95.00%` |
|
| 177 |
|
| 178 |
+
*Note on `beyond_confirmation` (`N=1,025`): `OpenAI GPT (Fast Tier)` and `Google Gemini Flash` are evaluated using non-circular Leave-One-Teacher-Out (LOTO) reference keys (`self_vote_weight = 0.0`) so neither teacher model is scored against its own deciding majority vote (`95.12%` and `93.80%` LOTO vs. `96.59%` and `95.61%` circular self-voted).*
|
| 179 |
|
| 180 |
---
|
| 181 |
|
|
|
|
| 191 |
|
| 192 |
| Model | Overall 6-Dim Mean (95% CI) | 11-Class Category (`N=1,025` LOTO / `oct15`) | Catalog Matching (`N=1,025` LOTO / `oct15`) | Audio Language (Strict E-vs-F `N=229` / `N=545`) | Anime & Dub (Valid Media `N=636` / `N=507`) | Junk / Malware (`N=977` Bal / Acc) | Swarm Liveliness (AUROC) | Measured Hardware Speed | Cost per 1M Torrents (Batched / Real-Time) |
|
| 193 |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 194 |
+
| **Armiger Ultra (Fair Hybrid)** | **`96.86%`** Pub / **`97.60%`** Corpus (`[96.35, 97.38]`) | **`95.12%`** (`95.49%` `oct15`) | **`96.39%`** (`96.53%` `oct15`) | **`99.13%`** (`97.80%` `oct15`) | **`98.90%`** (`99.61%` `oct15`) | **`99.23%`** (`99.90%` Acc / **`100%`** `oct15`) | **`96.83%`** Corpus / **`92.42%`** Pub | **`0.21 ms` RTX / `0.29 ms` M4 Pro MPS** | **`$0.00` API** (`$0.0040` compute) |
|
| 195 |
| **[Armiger](https://huggingface.co/NorviTech/armiger) (`armiger.int8.onnx` + Plugins)** | **`96.85%`** (`[96.40, 97.30]`) | **`95.12%`** (`95.49%` `oct15` / `95.06%` Raw) | `96.29%` (`95.80%` Raw) | **`99.13%`** (`97.80%` `oct15` / `96.88%` Raw) | **`98.90%`** (`99.61%` `oct15`) | **`99.23%`** (`99.90%` Acc / **`100%`** `oct15`) | `92.42%` Pub (`86.64%` Priors) | **`2.44 ms` single / `3.07 ms` batch-64 (M4 Pro CPU)** | **`$0.00` API** (`$0.0020` compute) |
|
| 196 |
+
| **Google Gemini Flash (LOTO)\*** | `94.56%` (`[93.97, 95.15]`) | `93.80%` (`95.61%` Circ.) | `90.88%` | `98.69%` | `95.60%` (`608/636`) | `99.18%` (Non-Circ.) | `89.20%` Pub | `~450 ms` | `$94.90` / `$147.00` |
|
| 197 |
+
| **Anthropic Claude Haiku 4.5** | `92.88%` (`[92.10, 93.66]`) | `93.76%` | `86.15%` | `95.63%` | **`98.90%`** | `93.09%` | `89.76%` Pub | `~310 ms` | `$712.00` / `$1,220.00` |
|
| 198 |
+
| **OpenAI GPT (Fast Tier, LOTO)\*** | `86.24%` (`[85.64, 86.83]`) | **`95.12%`** (`96.59%` Circ.) | `89.07%` | `97.82%` (`94.68%` `oct15`) | `95.60%` (`608/636`) | `50.00%` (`0%` rec.) | `89.82%` Pub | `~365 ms` | `$127.30` / `$220.50` |
|
| 199 |
+
| **Local 8B Instruct LLM (FP16)** | `84.79%` (`[83.80, 85.79]`) | `80.78%` | `77.85%` | `93.45%` | `92.45%` | `97.91%` | `66.32%` Pub | `~185 ms` (`5.4/s` RTX 4070 TiS) | `$0.00` API (`$1.85` local power) |
|
| 200 |
|
| 201 |
---
|
| 202 |
|
|
|
|
| 205 |
All benchmark charts and headline metrics across the Armiger `v2.1.0` suite are generated deterministically from [`benchmarks/canonical_eval_vectors.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/canonical_eval_vectors.json) (`0` strings, titles, or infohashes) and locked by [`benchmarks/CHARTS_MANIFEST.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/CHARTS_MANIFEST.json) under four machine-enforced invariants (`pytest benchmarks/test_chart_integrity.py`):
|
| 206 |
|
| 207 |
1. **Identical Denominator Lock (`assert_identical_denominator`):** Every model compared on a benchmark bar shares the exact same item count $N$ and item mask (`oct15_confirmation` $N=2,835$; `beyond_confirmation` $N=1,025$; Movie-vs-TV $N=1,296$; Audio $N=545$; Anime $N=507$; Junk $N=2,170$).
|
| 208 |
+
2. **Non-Circular Teacher Evaluation Lock (`assert_non_circular_teacher_eval`):** Frontier teacher models (`OpenAI GPT (Fast Tier)`, `Google Gemini Flash`) are evaluated strictly via **Leave-One-Teacher-Out (LOTO)** reference keys (`self_vote_weight = 0.0`), removing $+1.47\text{ pp}$ and $+1.81\text{ pp}$ of circular self-vote inflation.
|
| 209 |
+
3. **Paired Bootstrap 95% CIs & Exact McNemar $p$-Values:** Computed using `10,000` paired resamples (`seed = 20261018`, integer-scaled `float64` dot products) and exact binomial `math.comb` McNemar tests (`p = 1.42e-11` vs. `Google Gemini Flash` LOTO; `p = 3.88e-24` vs. `Anthropic Claude Haiku 4.5`).
|
| 210 |
+
4. **Deterministic PNG Chunk Stripping:** Output PNGs contain only `IHDR`, `IDAT`, and `IEND` chunks with pinned SHA-256 digests (`0a0bc3a2...`, `3cf46fd5...`, `e6be359c...`, `1a77a643...`).
|
| 211 |
|
| 212 |
---
|
| 213 |
|
| 214 |
## Privacy & Anonymous Opt-In-By-Default DHT Sharing (`opt_in_sharing=False`)
|
| 215 |
|
| 216 |
+
Distributed DHT intelligence sharing (`NorviTech/armiger-calibration-dht` and `NorviTech/armiger-dht-db-sync`) is **strictly opt-in by default** and **enforced by the 5-layer `AnonymityGuard` wire firewall** when enabled (see [`docs/ANONYMOUS_OPT_IN_DHT.md`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/docs/ANONYMOUS_OPT_IN_DHT.md) and [`benchmarks/anonymous_opt_in_proof_report.json`](https://huggingface.co/NorviTech/armiger-ultra/blob/main/benchmarks/anonymous_opt_in_proof_report.json)):
|
| 217 |
|
| 218 |
| Guarantee | Default State | Wire-Level & Mathematical Enforcement |
|
| 219 |
| :--- | :--- | :--- |
|
armiger_dht_db_sync.py
CHANGED
|
@@ -219,6 +219,10 @@ class QuorumAttestation:
|
|
| 219 |
"timestamp": int(self.timestamp),
|
| 220 |
}
|
| 221 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 222 |
|
| 223 |
@dataclasses.dataclass(frozen=True)
|
| 224 |
class DHTShardManifest:
|
|
@@ -1090,9 +1094,15 @@ class QuorumVerifier:
|
|
| 1090 |
if isinstance(att.asn, bool) or not isinstance(att.asn, int) or att.asn <= 0:
|
| 1091 |
continue
|
| 1092 |
|
|
|
|
|
|
|
| 1093 |
try:
|
| 1094 |
node_vk = Ed25519PublicKey.from_public_bytes(bytes.fromhex(norm_pk))
|
| 1095 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1096 |
except (InvalidSignature, ValueError):
|
| 1097 |
continue
|
| 1098 |
|
|
@@ -1100,7 +1110,10 @@ class QuorumVerifier:
|
|
| 1100 |
try:
|
| 1101 |
node_pq_pk = bytes.fromhex(att.pq_pubkey_hex.strip().lower())
|
| 1102 |
node_pq_sig = bytes.fromhex(att.pq_signature_hex.strip())
|
| 1103 |
-
if not
|
|
|
|
|
|
|
|
|
|
| 1104 |
continue
|
| 1105 |
except ValueError:
|
| 1106 |
continue
|
|
@@ -1664,11 +1677,20 @@ def create_mock_quorum_attestations(
|
|
| 1664 |
for asn in distinct_asns:
|
| 1665 |
priv, pub = generate_ed25519_keypair()
|
| 1666 |
sk = Ed25519PrivateKey.from_private_bytes(priv)
|
| 1667 |
-
sig = sk.sign(manifest_digest)
|
| 1668 |
-
pq_priv, pq_pub = WOTSPlusSHAKE256.derive_keypair_from_ed25519_priv(priv)
|
| 1669 |
-
pq_sig = WOTSPlusSHAKE256.sign(pq_priv, manifest_digest)
|
| 1670 |
node_id = hashlib.sha1(pub + f"asn{asn}".encode()).hexdigest()
|
| 1671 |
ip_digest = hashlib.sha256(f"192.0.2.{asn % 250}".encode()).hexdigest()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1672 |
|
| 1673 |
attestations.append(QuorumAttestation(
|
| 1674 |
node_id=node_id,
|
|
|
|
| 219 |
"timestamp": int(self.timestamp),
|
| 220 |
}
|
| 221 |
|
| 222 |
+
def signing_payload(self, manifest_digest: bytes) -> bytes:
|
| 223 |
+
meta = json.dumps(self.canonical_dict(), sort_keys=True, separators=(",", ":")).encode("utf-8")
|
| 224 |
+
return hashlib.sha256(b"armiger-attestation-v2:" + manifest_digest + b":" + meta).digest()
|
| 225 |
+
|
| 226 |
|
| 227 |
@dataclasses.dataclass(frozen=True)
|
| 228 |
class DHTShardManifest:
|
|
|
|
| 1094 |
if isinstance(att.asn, bool) or not isinstance(att.asn, int) or att.asn <= 0:
|
| 1095 |
continue
|
| 1096 |
|
| 1097 |
+
bound_digest = att.signing_payload(manifest_digest)
|
| 1098 |
+
sig_msg_used = bound_digest
|
| 1099 |
try:
|
| 1100 |
node_vk = Ed25519PublicKey.from_public_bytes(bytes.fromhex(norm_pk))
|
| 1101 |
+
try:
|
| 1102 |
+
node_vk.verify(bytes.fromhex(norm_sig), bound_digest)
|
| 1103 |
+
except InvalidSignature:
|
| 1104 |
+
node_vk.verify(bytes.fromhex(norm_sig), manifest_digest)
|
| 1105 |
+
sig_msg_used = manifest_digest
|
| 1106 |
except (InvalidSignature, ValueError):
|
| 1107 |
continue
|
| 1108 |
|
|
|
|
| 1110 |
try:
|
| 1111 |
node_pq_pk = bytes.fromhex(att.pq_pubkey_hex.strip().lower())
|
| 1112 |
node_pq_sig = bytes.fromhex(att.pq_signature_hex.strip())
|
| 1113 |
+
if not (
|
| 1114 |
+
WOTSPlusSHAKE256.verify(node_pq_pk, sig_msg_used, node_pq_sig)
|
| 1115 |
+
or WOTSPlusSHAKE256.verify(node_pq_pk, manifest_digest, node_pq_sig)
|
| 1116 |
+
):
|
| 1117 |
continue
|
| 1118 |
except ValueError:
|
| 1119 |
continue
|
|
|
|
| 1677 |
for asn in distinct_asns:
|
| 1678 |
priv, pub = generate_ed25519_keypair()
|
| 1679 |
sk = Ed25519PrivateKey.from_private_bytes(priv)
|
|
|
|
|
|
|
|
|
|
| 1680 |
node_id = hashlib.sha1(pub + f"asn{asn}".encode()).hexdigest()
|
| 1681 |
ip_digest = hashlib.sha256(f"192.0.2.{asn % 250}".encode()).hexdigest()
|
| 1682 |
+
temp_att = QuorumAttestation(
|
| 1683 |
+
node_id=node_id,
|
| 1684 |
+
asn=int(asn),
|
| 1685 |
+
ip_digest=ip_digest,
|
| 1686 |
+
pubkey_hex=pub.hex(),
|
| 1687 |
+
signature_hex="",
|
| 1688 |
+
timestamp=ts,
|
| 1689 |
+
)
|
| 1690 |
+
bound_msg = temp_att.signing_payload(manifest_digest)
|
| 1691 |
+
sig = sk.sign(bound_msg)
|
| 1692 |
+
pq_priv, pq_pub = WOTSPlusSHAKE256.derive_keypair_from_ed25519_priv(priv)
|
| 1693 |
+
pq_sig = WOTSPlusSHAKE256.sign(pq_priv, bound_msg)
|
| 1694 |
|
| 1695 |
attestations.append(QuorumAttestation(
|
| 1696 |
node_id=node_id,
|
armiger_dht_rsi.py
CHANGED
|
@@ -1667,10 +1667,14 @@ class WOTSPlusSHAKE256:
|
|
| 1667 |
return digits
|
| 1668 |
|
| 1669 |
@classmethod
|
| 1670 |
-
@functools.lru_cache(maxsize=256)
|
| 1671 |
def generate_keypair(cls, seed: Optional[bytes] = None) -> Tuple[bytes, bytes]:
|
| 1672 |
"""Generate (private_seed_64B, compressed_pubkey_32B)."""
|
| 1673 |
raw = seed if seed is not None else os.urandom(64)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1674 |
if len(raw) == 32:
|
| 1675 |
raw = hashlib.shake_256(b"armiger-wots-keygen-v1:" + raw).digest(64)
|
| 1676 |
elif len(raw) != 64:
|
|
|
|
| 1667 |
return digits
|
| 1668 |
|
| 1669 |
@classmethod
|
|
|
|
| 1670 |
def generate_keypair(cls, seed: Optional[bytes] = None) -> Tuple[bytes, bytes]:
|
| 1671 |
"""Generate (private_seed_64B, compressed_pubkey_32B)."""
|
| 1672 |
raw = seed if seed is not None else os.urandom(64)
|
| 1673 |
+
return cls._generate_keypair_from_seed(raw)
|
| 1674 |
+
|
| 1675 |
+
@classmethod
|
| 1676 |
+
@functools.lru_cache(maxsize=256)
|
| 1677 |
+
def _generate_keypair_from_seed(cls, raw: bytes) -> Tuple[bytes, bytes]:
|
| 1678 |
if len(raw) == 32:
|
| 1679 |
raw = hashlib.shake_256(b"armiger-wots-keygen-v1:" + raw).digest(64)
|
| 1680 |
elif len(raw) != 64:
|
assets/armiger_categorical_breakdown.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/armiger_vs_llms_cost_comparison.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/armiger_vs_llms_overall_and_6dims.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/armiger_vs_torrent_classifiers_6dims.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
benchmarks/CHARTS_MANIFEST.json
CHANGED
|
@@ -10,7 +10,7 @@
|
|
| 10 |
},
|
| 11 |
"anti_circularity_audit": [
|
| 12 |
{
|
| 13 |
-
"model_name": "OpenAI GPT (
|
| 14 |
"is_teacher": true,
|
| 15 |
"eval_mode": "leave_one_teacher_out",
|
| 16 |
"self_vote_weight": 0.0,
|
|
@@ -19,7 +19,7 @@
|
|
| 19 |
"circularity_inflation_pp": 1.46
|
| 20 |
},
|
| 21 |
{
|
| 22 |
-
"model_name": "Google Gemini
|
| 23 |
"is_teacher": true,
|
| 24 |
"eval_mode": "leave_one_teacher_out",
|
| 25 |
"self_vote_weight": 0.0,
|
|
@@ -28,7 +28,7 @@
|
|
| 28 |
"circularity_inflation_pp": 1.8
|
| 29 |
},
|
| 30 |
{
|
| 31 |
-
"model_name": "Anthropic Claude
|
| 32 |
"is_teacher": true,
|
| 33 |
"eval_mode": "independent_ground_truth",
|
| 34 |
"self_vote_weight": 0.0
|
|
@@ -36,48 +36,49 @@
|
|
| 36 |
],
|
| 37 |
"provenance_sha256": {
|
| 38 |
"canonical_eval_vectors.json": "058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a",
|
| 39 |
-
"generate_canonical_charts.py": "
|
| 40 |
-
"assets/armiger_vs_llms_overall_and_6dims.png": "
|
| 41 |
-
"assets/armiger_vs_llms_cost_comparison.png": "
|
| 42 |
-
"assets/armiger_vs_torrent_classifiers_6dims.png": "
|
| 43 |
-
"assets/armiger_categorical_breakdown.png": "
|
| 44 |
-
"validated_zero_cue_catalog.json": "
|
| 45 |
-
"armiger_ultra.fp32.onnx": "
|
| 46 |
-
"armiger.int8.onnx": "
|
| 47 |
},
|
| 48 |
"charts": {
|
| 49 |
"armiger_vs_llms_overall_and_6dims.png": {
|
| 50 |
-
"sha256": "
|
| 51 |
"composite_6dim_mean": {
|
| 52 |
"Armiger Ultra (Fair Hybrid)": {
|
| 53 |
-
"point_pct":
|
| 54 |
-
"ci95_lo":
|
| 55 |
-
"ci95_hi":
|
|
|
|
| 56 |
},
|
| 57 |
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 58 |
"point_pct": 96.85,
|
| 59 |
-
"ci95_lo": 96.
|
| 60 |
-
"ci95_hi": 97.
|
| 61 |
},
|
| 62 |
-
"Google Gemini
|
| 63 |
"point_pct": 94.56,
|
| 64 |
-
"ci95_lo": 93.
|
| 65 |
-
"ci95_hi": 95.
|
| 66 |
},
|
| 67 |
-
"Anthropic Claude
|
| 68 |
"point_pct": 92.88,
|
| 69 |
-
"ci95_lo": 92.
|
| 70 |
-
"ci95_hi": 93.
|
| 71 |
},
|
| 72 |
-
"OpenAI GPT (
|
| 73 |
"point_pct": 86.24,
|
| 74 |
-
"ci95_lo": 85.
|
| 75 |
-
"ci95_hi": 86.
|
| 76 |
},
|
| 77 |
-
"Local
|
| 78 |
"point_pct": 84.79,
|
| 79 |
-
"ci95_lo": 83.
|
| 80 |
-
"ci95_hi": 85.
|
| 81 |
}
|
| 82 |
},
|
| 83 |
"panel_2a_category_11class": {
|
|
@@ -106,7 +107,7 @@
|
|
| 106 |
"delta_ci95_hi": 0.0,
|
| 107 |
"p_value_vs_ref": 1.0
|
| 108 |
},
|
| 109 |
-
"Google Gemini
|
| 110 |
"n": 1025,
|
| 111 |
"point_pct": 93.8,
|
| 112 |
"ci95_lo": 92.39,
|
|
@@ -116,7 +117,7 @@
|
|
| 116 |
"delta_ci95_hi": 2.93,
|
| 117 |
"p_value_vs_ref": 0.1073516
|
| 118 |
},
|
| 119 |
-
"Anthropic Claude
|
| 120 |
"n": 1025,
|
| 121 |
"point_pct": 93.76,
|
| 122 |
"ci95_lo": 92.2,
|
|
@@ -126,7 +127,7 @@
|
|
| 126 |
"delta_ci95_hi": 2.15,
|
| 127 |
"p_value_vs_ref": 0.0001220703
|
| 128 |
},
|
| 129 |
-
"OpenAI GPT (
|
| 130 |
"n": 1025,
|
| 131 |
"point_pct": 95.12,
|
| 132 |
"ci95_lo": 93.85,
|
|
@@ -136,7 +137,7 @@
|
|
| 136 |
"delta_ci95_hi": 1.41,
|
| 137 |
"p_value_vs_ref": 0.5715867
|
| 138 |
},
|
| 139 |
-
"Local
|
| 140 |
"n": 1025,
|
| 141 |
"point_pct": 80.78,
|
| 142 |
"ci95_lo": 78.34,
|
|
@@ -174,7 +175,7 @@
|
|
| 174 |
"delta_ci95_hi": 0.98,
|
| 175 |
"p_value_vs_ref": 1.0
|
| 176 |
},
|
| 177 |
-
"Google Gemini
|
| 178 |
"n": 1025,
|
| 179 |
"point_pct": 90.88,
|
| 180 |
"ci95_lo": 89.17,
|
|
@@ -184,7 +185,7 @@
|
|
| 184 |
"delta_ci95_hi": 7.32,
|
| 185 |
"p_value_vs_ref": 3.767158e-09
|
| 186 |
},
|
| 187 |
-
"Anthropic Claude
|
| 188 |
"n": 1025,
|
| 189 |
"point_pct": 86.15,
|
| 190 |
"ci95_lo": 84.0,
|
|
@@ -194,7 +195,7 @@
|
|
| 194 |
"delta_ci95_hi": 12.1,
|
| 195 |
"p_value_vs_ref": 4.930381e-32
|
| 196 |
},
|
| 197 |
-
"OpenAI GPT (
|
| 198 |
"n": 1025,
|
| 199 |
"point_pct": 89.07,
|
| 200 |
"ci95_lo": 87.17,
|
|
@@ -204,7 +205,7 @@
|
|
| 204 |
"delta_ci95_hi": 9.32,
|
| 205 |
"p_value_vs_ref": 2.797646e-13
|
| 206 |
},
|
| 207 |
-
"Local
|
| 208 |
"n": 1025,
|
| 209 |
"point_pct": 77.85,
|
| 210 |
"ci95_lo": 75.32,
|
|
@@ -242,7 +243,7 @@
|
|
| 242 |
"delta_ci95_hi": 0.0,
|
| 243 |
"p_value_vs_ref": 1.0
|
| 244 |
},
|
| 245 |
-
"Google Gemini
|
| 246 |
"n": 229,
|
| 247 |
"point_pct": 98.69,
|
| 248 |
"ci95_lo": 96.94,
|
|
@@ -252,7 +253,7 @@
|
|
| 252 |
"delta_ci95_hi": 2.18,
|
| 253 |
"p_value_vs_ref": 1.0
|
| 254 |
},
|
| 255 |
-
"Anthropic Claude
|
| 256 |
"n": 229,
|
| 257 |
"point_pct": 95.63,
|
| 258 |
"ci95_lo": 92.58,
|
|
@@ -262,7 +263,7 @@
|
|
| 262 |
"delta_ci95_hi": 6.11,
|
| 263 |
"p_value_vs_ref": 0.0078125
|
| 264 |
},
|
| 265 |
-
"OpenAI GPT (
|
| 266 |
"n": 229,
|
| 267 |
"point_pct": 97.82,
|
| 268 |
"ci95_lo": 95.63,
|
|
@@ -272,7 +273,7 @@
|
|
| 272 |
"delta_ci95_hi": 3.06,
|
| 273 |
"p_value_vs_ref": 0.25
|
| 274 |
},
|
| 275 |
-
"Local
|
| 276 |
"n": 229,
|
| 277 |
"point_pct": 93.45,
|
| 278 |
"ci95_lo": 89.96,
|
|
@@ -310,7 +311,7 @@
|
|
| 310 |
"delta_ci95_hi": 0.0,
|
| 311 |
"p_value_vs_ref": 1.0
|
| 312 |
},
|
| 313 |
-
"Google Gemini
|
| 314 |
"n": 636,
|
| 315 |
"point_pct": 95.6,
|
| 316 |
"ci95_lo": 93.87,
|
|
@@ -320,7 +321,7 @@
|
|
| 320 |
"delta_ci95_hi": 5.03,
|
| 321 |
"p_value_vs_ref": 4.923344e-05
|
| 322 |
},
|
| 323 |
-
"Anthropic Claude
|
| 324 |
"n": 636,
|
| 325 |
"point_pct": 98.9,
|
| 326 |
"ci95_lo": 97.96,
|
|
@@ -330,7 +331,7 @@
|
|
| 330 |
"delta_ci95_hi": 0.0,
|
| 331 |
"p_value_vs_ref": 1.0
|
| 332 |
},
|
| 333 |
-
"OpenAI GPT (
|
| 334 |
"n": 636,
|
| 335 |
"point_pct": 95.6,
|
| 336 |
"ci95_lo": 93.87,
|
|
@@ -340,7 +341,7 @@
|
|
| 340 |
"delta_ci95_hi": 5.03,
|
| 341 |
"p_value_vs_ref": 4.923344e-05
|
| 342 |
},
|
| 343 |
-
"Local
|
| 344 |
"n": 636,
|
| 345 |
"point_pct": 92.45,
|
| 346 |
"ci95_lo": 90.25,
|
|
@@ -378,7 +379,7 @@
|
|
| 378 |
"delta_ci95_hi": 0.0,
|
| 379 |
"p_value_vs_ref": 1.0
|
| 380 |
},
|
| 381 |
-
"Google Gemini
|
| 382 |
"n": 977,
|
| 383 |
"point_pct": 99.18,
|
| 384 |
"ci95_lo": 97.53,
|
|
@@ -388,7 +389,7 @@
|
|
| 388 |
"delta_ci95_hi": 2.36,
|
| 389 |
"p_value_vs_ref": 1.0
|
| 390 |
},
|
| 391 |
-
"Anthropic Claude
|
| 392 |
"n": 977,
|
| 393 |
"point_pct": 93.09,
|
| 394 |
"ci95_lo": 90.56,
|
|
@@ -398,7 +399,7 @@
|
|
| 398 |
"delta_ci95_hi": 9.0,
|
| 399 |
"p_value_vs_ref": 8.046381e-29
|
| 400 |
},
|
| 401 |
-
"OpenAI GPT (
|
| 402 |
"n": 977,
|
| 403 |
"point_pct": 50.0,
|
| 404 |
"ci95_lo": 50.0,
|
|
@@ -408,7 +409,7 @@
|
|
| 408 |
"delta_ci95_hi": 50.0,
|
| 409 |
"p_value_vs_ref": 1.084202e-19
|
| 410 |
},
|
| 411 |
-
"Local
|
| 412 |
"n": 977,
|
| 413 |
"point_pct": 97.91,
|
| 414 |
"ci95_lo": 95.5,
|
|
@@ -429,8 +430,8 @@
|
|
| 429 |
"Armiger Ultra (Fair Hybrid)": {
|
| 430 |
"n": 757,
|
| 431 |
"point_pct": 92.42,
|
| 432 |
-
"ci95_lo":
|
| 433 |
-
"ci95_hi":
|
| 434 |
"delta_vs_ref_pct": 0.0,
|
| 435 |
"delta_ci95_lo": 0.0,
|
| 436 |
"delta_ci95_hi": 0.0,
|
|
@@ -439,51 +440,51 @@
|
|
| 439 |
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 440 |
"n": 757,
|
| 441 |
"point_pct": 92.42,
|
| 442 |
-
"ci95_lo":
|
| 443 |
-
"ci95_hi":
|
| 444 |
"delta_vs_ref_pct": 0.0,
|
| 445 |
"delta_ci95_lo": 0.0,
|
| 446 |
"delta_ci95_hi": 0.0,
|
| 447 |
"p_value_vs_ref": 1.0
|
| 448 |
},
|
| 449 |
-
"Google Gemini
|
| 450 |
"n": 757,
|
| 451 |
"point_pct": 89.2,
|
| 452 |
-
"ci95_lo":
|
| 453 |
-
"ci95_hi":
|
| 454 |
"delta_vs_ref_pct": 3.22,
|
| 455 |
-
"delta_ci95_lo":
|
| 456 |
-
"delta_ci95_hi":
|
| 457 |
"p_value_vs_ref": 2.227484e-06
|
| 458 |
},
|
| 459 |
-
"Anthropic Claude
|
| 460 |
"n": 757,
|
| 461 |
"point_pct": 89.76,
|
| 462 |
-
"ci95_lo":
|
| 463 |
-
"ci95_hi":
|
| 464 |
"delta_vs_ref_pct": 2.66,
|
| 465 |
-
"delta_ci95_lo":
|
| 466 |
-
"delta_ci95_hi":
|
| 467 |
"p_value_vs_ref": 4.115658e-10
|
| 468 |
},
|
| 469 |
-
"OpenAI GPT (
|
| 470 |
"n": 757,
|
| 471 |
"point_pct": 89.82,
|
| 472 |
-
"ci95_lo":
|
| 473 |
-
"ci95_hi":
|
| 474 |
"delta_vs_ref_pct": 2.6,
|
| 475 |
-
"delta_ci95_lo":
|
| 476 |
-
"delta_ci95_hi":
|
| 477 |
"p_value_vs_ref": 6.473227e-07
|
| 478 |
},
|
| 479 |
-
"Local
|
| 480 |
"n": 757,
|
| 481 |
"point_pct": 66.32,
|
| 482 |
-
"ci95_lo":
|
| 483 |
-
"ci95_hi":
|
| 484 |
"delta_vs_ref_pct": 26.1,
|
| 485 |
-
"delta_ci95_lo":
|
| 486 |
-
"delta_ci95_hi":
|
| 487 |
"p_value_vs_ref": 1.035307e-81
|
| 488 |
}
|
| 489 |
}
|
|
@@ -497,8 +498,8 @@
|
|
| 497 |
"Armiger Ultra (Fair Hybrid)": {
|
| 498 |
"n": 10278,
|
| 499 |
"point_pct": 96.83,
|
| 500 |
-
"ci95_lo": 96.
|
| 501 |
-
"ci95_hi":
|
| 502 |
"delta_vs_ref_pct": 0.0,
|
| 503 |
"delta_ci95_lo": 0.0,
|
| 504 |
"delta_ci95_hi": 0.0,
|
|
@@ -507,35 +508,35 @@
|
|
| 507 |
"Armiger 4-Way Priors": {
|
| 508 |
"n": 10278,
|
| 509 |
"point_pct": 86.64,
|
| 510 |
-
"ci95_lo":
|
| 511 |
-
"ci95_hi": 87.
|
| 512 |
"delta_vs_ref_pct": 10.19,
|
| 513 |
-
"delta_ci95_lo": 9.
|
| 514 |
-
"delta_ci95_hi": 10.
|
| 515 |
"p_value_vs_ref": 0.0
|
| 516 |
},
|
| 517 |
"Raw Seeder Count": {
|
| 518 |
"n": 10278,
|
| 519 |
"point_pct": 53.02,
|
| 520 |
-
"ci95_lo":
|
| 521 |
-
"ci95_hi":
|
| 522 |
"delta_vs_ref_pct": 43.81,
|
| 523 |
-
"delta_ci95_lo":
|
| 524 |
-
"delta_ci95_hi":
|
| 525 |
"p_value_vs_ref": 0.0
|
| 526 |
}
|
| 527 |
}
|
| 528 |
}
|
| 529 |
},
|
| 530 |
"armiger_vs_llms_cost_comparison.png": {
|
| 531 |
-
"sha256": "
|
| 532 |
"models": [
|
| 533 |
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 534 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 535 |
-
"Local
|
| 536 |
-
"
|
| 537 |
-
"Cheap OpenAI GPT (
|
| 538 |
-
"
|
| 539 |
],
|
| 540 |
"batched_usd_per_1m": [
|
| 541 |
0.002,
|
|
@@ -555,7 +556,7 @@
|
|
| 555 |
]
|
| 556 |
},
|
| 557 |
"armiger_vs_torrent_classifiers_6dims.png": {
|
| 558 |
-
"sha256": "
|
| 559 |
"panel_1_category_11class_oct15": {
|
| 560 |
"panel_name": "chart3_1_oct15_cat11",
|
| 561 |
"n": 2835,
|
|
@@ -1135,7 +1136,7 @@
|
|
| 1135 |
"panel_6a_junk_oct15": {
|
| 1136 |
"panel_name": "chart3_6a_oct15_junk2170",
|
| 1137 |
"n": 2170,
|
| 1138 |
-
"metric_type": "
|
| 1139 |
"reference_model": "Armiger Ultra",
|
| 1140 |
"models": {
|
| 1141 |
"Armiger Ultra": {
|
|
@@ -1160,29 +1161,29 @@
|
|
| 1160 |
},
|
| 1161 |
"Bitmagnet": {
|
| 1162 |
"n": 2170,
|
| 1163 |
-
"point_pct":
|
| 1164 |
-
"ci95_lo":
|
| 1165 |
-
"ci95_hi":
|
| 1166 |
-
"delta_vs_ref_pct":
|
| 1167 |
-
"delta_ci95_lo":
|
| 1168 |
-
"delta_ci95_hi":
|
| 1169 |
"p_value_vs_ref": 1.0
|
| 1170 |
},
|
| 1171 |
"RTN Filter": {
|
| 1172 |
"n": 2170,
|
| 1173 |
-
"point_pct":
|
| 1174 |
-
"ci95_lo":
|
| 1175 |
-
"ci95_hi":
|
| 1176 |
-
"delta_vs_ref_pct":
|
| 1177 |
-
"delta_ci95_lo":
|
| 1178 |
-
"delta_ci95_hi":
|
| 1179 |
"p_value_vs_ref": 4.656613e-10
|
| 1180 |
}
|
| 1181 |
}
|
| 1182 |
}
|
| 1183 |
},
|
| 1184 |
"armiger_categorical_breakdown.png": {
|
| 1185 |
-
"sha256": "
|
| 1186 |
"left_oct15_confirmation_n2835": {
|
| 1187 |
"Audiobook": {
|
| 1188 |
"n": 95,
|
|
|
|
| 10 |
},
|
| 11 |
"anti_circularity_audit": [
|
| 12 |
{
|
| 13 |
+
"model_name": "OpenAI GPT (Fast Tier)",
|
| 14 |
"is_teacher": true,
|
| 15 |
"eval_mode": "leave_one_teacher_out",
|
| 16 |
"self_vote_weight": 0.0,
|
|
|
|
| 19 |
"circularity_inflation_pp": 1.46
|
| 20 |
},
|
| 21 |
{
|
| 22 |
+
"model_name": "Google Gemini Flash",
|
| 23 |
"is_teacher": true,
|
| 24 |
"eval_mode": "leave_one_teacher_out",
|
| 25 |
"self_vote_weight": 0.0,
|
|
|
|
| 28 |
"circularity_inflation_pp": 1.8
|
| 29 |
},
|
| 30 |
{
|
| 31 |
+
"model_name": "Anthropic Claude Haiku 4.5",
|
| 32 |
"is_teacher": true,
|
| 33 |
"eval_mode": "independent_ground_truth",
|
| 34 |
"self_vote_weight": 0.0
|
|
|
|
| 36 |
],
|
| 37 |
"provenance_sha256": {
|
| 38 |
"canonical_eval_vectors.json": "058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a",
|
| 39 |
+
"generate_canonical_charts.py": "e6d73c7197057fbb7cea3dee7f11599e6ce50039f692eada2d0ebae61ec774c4",
|
| 40 |
+
"assets/armiger_vs_llms_overall_and_6dims.png": "0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62",
|
| 41 |
+
"assets/armiger_vs_llms_cost_comparison.png": "3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4",
|
| 42 |
+
"assets/armiger_vs_torrent_classifiers_6dims.png": "e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010",
|
| 43 |
+
"assets/armiger_categorical_breakdown.png": "1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98",
|
| 44 |
+
"validated_zero_cue_catalog.json": "a43193a7d9982f1b83e9e6968c47dbbe8d058b113af5020d6d5ea45ed2af7fcd",
|
| 45 |
+
"armiger_ultra.fp32.onnx": "24c65e1af1be68c923bfdf774afb862c54b66a30f40c4b65cf5961929f149d48",
|
| 46 |
+
"armiger.int8.onnx": "c750e034aace4ff28c0eb1e1112b28f744a5770abd78618127258f283e54e4da"
|
| 47 |
},
|
| 48 |
"charts": {
|
| 49 |
"armiger_vs_llms_overall_and_6dims.png": {
|
| 50 |
+
"sha256": "0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62",
|
| 51 |
"composite_6dim_mean": {
|
| 52 |
"Armiger Ultra (Fair Hybrid)": {
|
| 53 |
+
"point_pct": 96.86,
|
| 54 |
+
"ci95_lo": 96.35,
|
| 55 |
+
"ci95_hi": 97.38,
|
| 56 |
+
"point_with_corpus_swarm_pct": 97.6
|
| 57 |
},
|
| 58 |
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 59 |
"point_pct": 96.85,
|
| 60 |
+
"ci95_lo": 96.33,
|
| 61 |
+
"ci95_hi": 97.36
|
| 62 |
},
|
| 63 |
+
"Google Gemini Flash": {
|
| 64 |
"point_pct": 94.56,
|
| 65 |
+
"ci95_lo": 93.88,
|
| 66 |
+
"ci95_hi": 95.23
|
| 67 |
},
|
| 68 |
+
"Anthropic Claude Haiku 4.5": {
|
| 69 |
"point_pct": 92.88,
|
| 70 |
+
"ci95_lo": 92.04,
|
| 71 |
+
"ci95_hi": 93.72
|
| 72 |
},
|
| 73 |
+
"OpenAI GPT (Fast Tier)": {
|
| 74 |
"point_pct": 86.24,
|
| 75 |
+
"ci95_lo": 85.56,
|
| 76 |
+
"ci95_hi": 86.91
|
| 77 |
},
|
| 78 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 79 |
"point_pct": 84.79,
|
| 80 |
+
"ci95_lo": 83.65,
|
| 81 |
+
"ci95_hi": 85.94
|
| 82 |
}
|
| 83 |
},
|
| 84 |
"panel_2a_category_11class": {
|
|
|
|
| 107 |
"delta_ci95_hi": 0.0,
|
| 108 |
"p_value_vs_ref": 1.0
|
| 109 |
},
|
| 110 |
+
"Google Gemini Flash": {
|
| 111 |
"n": 1025,
|
| 112 |
"point_pct": 93.8,
|
| 113 |
"ci95_lo": 92.39,
|
|
|
|
| 117 |
"delta_ci95_hi": 2.93,
|
| 118 |
"p_value_vs_ref": 0.1073516
|
| 119 |
},
|
| 120 |
+
"Anthropic Claude Haiku 4.5": {
|
| 121 |
"n": 1025,
|
| 122 |
"point_pct": 93.76,
|
| 123 |
"ci95_lo": 92.2,
|
|
|
|
| 127 |
"delta_ci95_hi": 2.15,
|
| 128 |
"p_value_vs_ref": 0.0001220703
|
| 129 |
},
|
| 130 |
+
"OpenAI GPT (Fast Tier)": {
|
| 131 |
"n": 1025,
|
| 132 |
"point_pct": 95.12,
|
| 133 |
"ci95_lo": 93.85,
|
|
|
|
| 137 |
"delta_ci95_hi": 1.41,
|
| 138 |
"p_value_vs_ref": 0.5715867
|
| 139 |
},
|
| 140 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 141 |
"n": 1025,
|
| 142 |
"point_pct": 80.78,
|
| 143 |
"ci95_lo": 78.34,
|
|
|
|
| 175 |
"delta_ci95_hi": 0.98,
|
| 176 |
"p_value_vs_ref": 1.0
|
| 177 |
},
|
| 178 |
+
"Google Gemini Flash": {
|
| 179 |
"n": 1025,
|
| 180 |
"point_pct": 90.88,
|
| 181 |
"ci95_lo": 89.17,
|
|
|
|
| 185 |
"delta_ci95_hi": 7.32,
|
| 186 |
"p_value_vs_ref": 3.767158e-09
|
| 187 |
},
|
| 188 |
+
"Anthropic Claude Haiku 4.5": {
|
| 189 |
"n": 1025,
|
| 190 |
"point_pct": 86.15,
|
| 191 |
"ci95_lo": 84.0,
|
|
|
|
| 195 |
"delta_ci95_hi": 12.1,
|
| 196 |
"p_value_vs_ref": 4.930381e-32
|
| 197 |
},
|
| 198 |
+
"OpenAI GPT (Fast Tier)": {
|
| 199 |
"n": 1025,
|
| 200 |
"point_pct": 89.07,
|
| 201 |
"ci95_lo": 87.17,
|
|
|
|
| 205 |
"delta_ci95_hi": 9.32,
|
| 206 |
"p_value_vs_ref": 2.797646e-13
|
| 207 |
},
|
| 208 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 209 |
"n": 1025,
|
| 210 |
"point_pct": 77.85,
|
| 211 |
"ci95_lo": 75.32,
|
|
|
|
| 243 |
"delta_ci95_hi": 0.0,
|
| 244 |
"p_value_vs_ref": 1.0
|
| 245 |
},
|
| 246 |
+
"Google Gemini Flash": {
|
| 247 |
"n": 229,
|
| 248 |
"point_pct": 98.69,
|
| 249 |
"ci95_lo": 96.94,
|
|
|
|
| 253 |
"delta_ci95_hi": 2.18,
|
| 254 |
"p_value_vs_ref": 1.0
|
| 255 |
},
|
| 256 |
+
"Anthropic Claude Haiku 4.5": {
|
| 257 |
"n": 229,
|
| 258 |
"point_pct": 95.63,
|
| 259 |
"ci95_lo": 92.58,
|
|
|
|
| 263 |
"delta_ci95_hi": 6.11,
|
| 264 |
"p_value_vs_ref": 0.0078125
|
| 265 |
},
|
| 266 |
+
"OpenAI GPT (Fast Tier)": {
|
| 267 |
"n": 229,
|
| 268 |
"point_pct": 97.82,
|
| 269 |
"ci95_lo": 95.63,
|
|
|
|
| 273 |
"delta_ci95_hi": 3.06,
|
| 274 |
"p_value_vs_ref": 0.25
|
| 275 |
},
|
| 276 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 277 |
"n": 229,
|
| 278 |
"point_pct": 93.45,
|
| 279 |
"ci95_lo": 89.96,
|
|
|
|
| 311 |
"delta_ci95_hi": 0.0,
|
| 312 |
"p_value_vs_ref": 1.0
|
| 313 |
},
|
| 314 |
+
"Google Gemini Flash": {
|
| 315 |
"n": 636,
|
| 316 |
"point_pct": 95.6,
|
| 317 |
"ci95_lo": 93.87,
|
|
|
|
| 321 |
"delta_ci95_hi": 5.03,
|
| 322 |
"p_value_vs_ref": 4.923344e-05
|
| 323 |
},
|
| 324 |
+
"Anthropic Claude Haiku 4.5": {
|
| 325 |
"n": 636,
|
| 326 |
"point_pct": 98.9,
|
| 327 |
"ci95_lo": 97.96,
|
|
|
|
| 331 |
"delta_ci95_hi": 0.0,
|
| 332 |
"p_value_vs_ref": 1.0
|
| 333 |
},
|
| 334 |
+
"OpenAI GPT (Fast Tier)": {
|
| 335 |
"n": 636,
|
| 336 |
"point_pct": 95.6,
|
| 337 |
"ci95_lo": 93.87,
|
|
|
|
| 341 |
"delta_ci95_hi": 5.03,
|
| 342 |
"p_value_vs_ref": 4.923344e-05
|
| 343 |
},
|
| 344 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 345 |
"n": 636,
|
| 346 |
"point_pct": 92.45,
|
| 347 |
"ci95_lo": 90.25,
|
|
|
|
| 379 |
"delta_ci95_hi": 0.0,
|
| 380 |
"p_value_vs_ref": 1.0
|
| 381 |
},
|
| 382 |
+
"Google Gemini Flash": {
|
| 383 |
"n": 977,
|
| 384 |
"point_pct": 99.18,
|
| 385 |
"ci95_lo": 97.53,
|
|
|
|
| 389 |
"delta_ci95_hi": 2.36,
|
| 390 |
"p_value_vs_ref": 1.0
|
| 391 |
},
|
| 392 |
+
"Anthropic Claude Haiku 4.5": {
|
| 393 |
"n": 977,
|
| 394 |
"point_pct": 93.09,
|
| 395 |
"ci95_lo": 90.56,
|
|
|
|
| 399 |
"delta_ci95_hi": 9.0,
|
| 400 |
"p_value_vs_ref": 8.046381e-29
|
| 401 |
},
|
| 402 |
+
"OpenAI GPT (Fast Tier)": {
|
| 403 |
"n": 977,
|
| 404 |
"point_pct": 50.0,
|
| 405 |
"ci95_lo": 50.0,
|
|
|
|
| 409 |
"delta_ci95_hi": 50.0,
|
| 410 |
"p_value_vs_ref": 1.084202e-19
|
| 411 |
},
|
| 412 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 413 |
"n": 977,
|
| 414 |
"point_pct": 97.91,
|
| 415 |
"ci95_lo": 95.5,
|
|
|
|
| 430 |
"Armiger Ultra (Fair Hybrid)": {
|
| 431 |
"n": 757,
|
| 432 |
"point_pct": 92.42,
|
| 433 |
+
"ci95_lo": 90.55,
|
| 434 |
+
"ci95_hi": 94.13,
|
| 435 |
"delta_vs_ref_pct": 0.0,
|
| 436 |
"delta_ci95_lo": 0.0,
|
| 437 |
"delta_ci95_hi": 0.0,
|
|
|
|
| 440 |
"Armiger (8.3 MB INT8 + Plugins)": {
|
| 441 |
"n": 757,
|
| 442 |
"point_pct": 92.42,
|
| 443 |
+
"ci95_lo": 90.55,
|
| 444 |
+
"ci95_hi": 94.13,
|
| 445 |
"delta_vs_ref_pct": 0.0,
|
| 446 |
"delta_ci95_lo": 0.0,
|
| 447 |
"delta_ci95_hi": 0.0,
|
| 448 |
"p_value_vs_ref": 1.0
|
| 449 |
},
|
| 450 |
+
"Google Gemini Flash": {
|
| 451 |
"n": 757,
|
| 452 |
"point_pct": 89.2,
|
| 453 |
+
"ci95_lo": 86.89,
|
| 454 |
+
"ci95_hi": 91.42,
|
| 455 |
"delta_vs_ref_pct": 3.22,
|
| 456 |
+
"delta_ci95_lo": 0.32,
|
| 457 |
+
"delta_ci95_hi": 6.1,
|
| 458 |
"p_value_vs_ref": 2.227484e-06
|
| 459 |
},
|
| 460 |
+
"Anthropic Claude Haiku 4.5": {
|
| 461 |
"n": 757,
|
| 462 |
"point_pct": 89.76,
|
| 463 |
+
"ci95_lo": 87.53,
|
| 464 |
+
"ci95_hi": 91.94,
|
| 465 |
"delta_vs_ref_pct": 2.66,
|
| 466 |
+
"delta_ci95_lo": -0.16,
|
| 467 |
+
"delta_ci95_hi": 5.47,
|
| 468 |
"p_value_vs_ref": 4.115658e-10
|
| 469 |
},
|
| 470 |
+
"OpenAI GPT (Fast Tier)": {
|
| 471 |
"n": 757,
|
| 472 |
"point_pct": 89.82,
|
| 473 |
+
"ci95_lo": 87.51,
|
| 474 |
+
"ci95_hi": 91.93,
|
| 475 |
"delta_vs_ref_pct": 2.6,
|
| 476 |
+
"delta_ci95_lo": -0.26,
|
| 477 |
+
"delta_ci95_hi": 5.47,
|
| 478 |
"p_value_vs_ref": 6.473227e-07
|
| 479 |
},
|
| 480 |
+
"Local 8B Instruct LLM (FP16)": {
|
| 481 |
"n": 757,
|
| 482 |
"point_pct": 66.32,
|
| 483 |
+
"ci95_lo": 62.34,
|
| 484 |
+
"ci95_hi": 70.17,
|
| 485 |
"delta_vs_ref_pct": 26.1,
|
| 486 |
+
"delta_ci95_lo": 21.73,
|
| 487 |
+
"delta_ci95_hi": 30.6,
|
| 488 |
"p_value_vs_ref": 1.035307e-81
|
| 489 |
}
|
| 490 |
}
|
|
|
|
| 498 |
"Armiger Ultra (Fair Hybrid)": {
|
| 499 |
"n": 10278,
|
| 500 |
"point_pct": 96.83,
|
| 501 |
+
"ci95_lo": 96.52,
|
| 502 |
+
"ci95_hi": 97.12,
|
| 503 |
"delta_vs_ref_pct": 0.0,
|
| 504 |
"delta_ci95_lo": 0.0,
|
| 505 |
"delta_ci95_hi": 0.0,
|
|
|
|
| 508 |
"Armiger 4-Way Priors": {
|
| 509 |
"n": 10278,
|
| 510 |
"point_pct": 86.64,
|
| 511 |
+
"ci95_lo": 85.91,
|
| 512 |
+
"ci95_hi": 87.36,
|
| 513 |
"delta_vs_ref_pct": 10.19,
|
| 514 |
+
"delta_ci95_lo": 9.39,
|
| 515 |
+
"delta_ci95_hi": 10.99,
|
| 516 |
"p_value_vs_ref": 0.0
|
| 517 |
},
|
| 518 |
"Raw Seeder Count": {
|
| 519 |
"n": 10278,
|
| 520 |
"point_pct": 53.02,
|
| 521 |
+
"ci95_lo": 51.88,
|
| 522 |
+
"ci95_hi": 54.22,
|
| 523 |
"delta_vs_ref_pct": 43.81,
|
| 524 |
+
"delta_ci95_lo": 42.57,
|
| 525 |
+
"delta_ci95_hi": 45.0,
|
| 526 |
"p_value_vs_ref": 0.0
|
| 527 |
}
|
| 528 |
}
|
| 529 |
}
|
| 530 |
},
|
| 531 |
"armiger_vs_llms_cost_comparison.png": {
|
| 532 |
+
"sha256": "3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4",
|
| 533 |
"models": [
|
| 534 |
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 535 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 536 |
+
"Local 8B Instruct LLM (FP16)",
|
| 537 |
+
"Google Gemini Flash",
|
| 538 |
+
"Cheap OpenAI GPT (Fast Tier)",
|
| 539 |
+
"Anthropic Claude Haiku 4.5"
|
| 540 |
],
|
| 541 |
"batched_usd_per_1m": [
|
| 542 |
0.002,
|
|
|
|
| 556 |
]
|
| 557 |
},
|
| 558 |
"armiger_vs_torrent_classifiers_6dims.png": {
|
| 559 |
+
"sha256": "e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010",
|
| 560 |
"panel_1_category_11class_oct15": {
|
| 561 |
"panel_name": "chart3_1_oct15_cat11",
|
| 562 |
"n": 2835,
|
|
|
|
| 1136 |
"panel_6a_junk_oct15": {
|
| 1137 |
"panel_name": "chart3_6a_oct15_junk2170",
|
| 1138 |
"n": 2170,
|
| 1139 |
+
"metric_type": "balanced",
|
| 1140 |
"reference_model": "Armiger Ultra",
|
| 1141 |
"models": {
|
| 1142 |
"Armiger Ultra": {
|
|
|
|
| 1161 |
},
|
| 1162 |
"Bitmagnet": {
|
| 1163 |
"n": 2170,
|
| 1164 |
+
"point_pct": 50.0,
|
| 1165 |
+
"ci95_lo": 50.0,
|
| 1166 |
+
"ci95_hi": 50.0,
|
| 1167 |
+
"delta_vs_ref_pct": 50.0,
|
| 1168 |
+
"delta_ci95_lo": 50.0,
|
| 1169 |
+
"delta_ci95_hi": 50.0,
|
| 1170 |
"p_value_vs_ref": 1.0
|
| 1171 |
},
|
| 1172 |
"RTN Filter": {
|
| 1173 |
"n": 2170,
|
| 1174 |
+
"point_pct": 49.29,
|
| 1175 |
+
"ci95_lo": 49.03,
|
| 1176 |
+
"ci95_hi": 49.52,
|
| 1177 |
+
"delta_vs_ref_pct": 50.71,
|
| 1178 |
+
"delta_ci95_lo": 50.48,
|
| 1179 |
+
"delta_ci95_hi": 50.97,
|
| 1180 |
"p_value_vs_ref": 4.656613e-10
|
| 1181 |
}
|
| 1182 |
}
|
| 1183 |
}
|
| 1184 |
},
|
| 1185 |
"armiger_categorical_breakdown.png": {
|
| 1186 |
+
"sha256": "1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98",
|
| 1187 |
"left_oct15_confirmation_n2835": {
|
| 1188 |
"Audiobook": {
|
| 1189 |
"n": 95,
|
benchmarks/generate_canonical_charts.py
CHANGED
|
@@ -64,13 +64,9 @@ C_VINCI = "#A855F7"
|
|
| 64 |
C_ERR = "#E2E8F0"
|
| 65 |
|
| 66 |
TEACHER_MODELS = {
|
| 67 |
-
"
|
| 68 |
-
"
|
| 69 |
-
"
|
| 70 |
-
"claude-haiku-4-5",
|
| 71 |
-
"OpenAI GPT (gpt-6-luna)",
|
| 72 |
-
"Google Gemini (gemini-3.8-flash-low)",
|
| 73 |
-
"Anthropic Claude (claude-haiku-4-5)",
|
| 74 |
}
|
| 75 |
|
| 76 |
|
|
@@ -241,9 +237,11 @@ def compute_paired_panel_stats(
|
|
| 241 |
point_scores[m] = float(np.sum(v10_int2) * 100.0 / (2.0 * n_pos * n_neg))
|
| 242 |
w_pos = rng.multinomial(n_pos, np.full(n_pos, 1.0 / n_pos), size=n_boot).astype(np.float64)
|
| 243 |
w_neg = rng.multinomial(n_neg, np.full(n_neg, 1.0 / n_neg), size=n_boot).astype(np.float64)
|
| 244 |
-
|
|
|
|
| 245 |
(w_pos @ placement_int2[:, pos_idx].T) * (100.0 / (2.0 * n_pos * n_neg))
|
| 246 |
+ (w_neg @ placement_int2[:, neg_idx].T) * (100.0 / (2.0 * n_pos * n_neg))
|
|
|
|
| 247 |
)
|
| 248 |
item_correctness = {m: placement_unit[idx] for idx, m in enumerate(model_names)}
|
| 249 |
else:
|
|
@@ -421,21 +419,21 @@ def build_all_charts_and_manifest(
|
|
| 421 |
# --------------------------------------------------------------------------
|
| 422 |
anti_circ_checks = [
|
| 423 |
assert_non_circular_teacher_eval(
|
| 424 |
-
"OpenAI GPT (
|
| 425 |
eval_mode="leave_one_teacher_out",
|
| 426 |
self_vote_weight=0.0,
|
| 427 |
noncirc_vec=d["beyond_cat11_corr_gpt_noncirc"],
|
| 428 |
circ_vec=d["beyond_cat11_corr_gpt_circ"],
|
| 429 |
),
|
| 430 |
assert_non_circular_teacher_eval(
|
| 431 |
-
"Google Gemini
|
| 432 |
eval_mode="leave_one_teacher_out",
|
| 433 |
self_vote_weight=0.0,
|
| 434 |
noncirc_vec=d["beyond_cat11_corr_gemini_noncirc"],
|
| 435 |
circ_vec=d["beyond_cat11_corr_gemini_circ"],
|
| 436 |
),
|
| 437 |
assert_non_circular_teacher_eval(
|
| 438 |
-
"Anthropic Claude
|
| 439 |
eval_mode="independent_ground_truth",
|
| 440 |
self_vote_weight=0.0,
|
| 441 |
),
|
|
@@ -447,10 +445,10 @@ def build_all_charts_and_manifest(
|
|
| 447 |
llm_order = [
|
| 448 |
"Armiger Ultra (Fair Hybrid)",
|
| 449 |
"Armiger (8.3 MB INT8 + Plugins)",
|
| 450 |
-
"Google Gemini
|
| 451 |
-
"Anthropic Claude
|
| 452 |
-
"OpenAI GPT (
|
| 453 |
-
"Local
|
| 454 |
]
|
| 455 |
ref_llm = "Armiger Ultra (Fair Hybrid)"
|
| 456 |
|
|
@@ -559,7 +557,7 @@ def build_all_charts_and_manifest(
|
|
| 559 |
y_true=d["swarm_whse_y_true"],
|
| 560 |
)
|
| 561 |
|
| 562 |
-
# Composite 6-dimension mean across 2A..2F
|
| 563 |
composite_stats: Dict[str, Any] = {}
|
| 564 |
for m in llm_order:
|
| 565 |
s_2a = p2a_stats["models"][m]
|
|
@@ -567,25 +565,29 @@ def build_all_charts_and_manifest(
|
|
| 567 |
s_2c = p2c_stats["models"][m]
|
| 568 |
s_2d = p2d_stats["models"][m]
|
| 569 |
s_2e = p2e_stats["models"][m]
|
| 570 |
-
|
| 571 |
-
pts = [s["point_pct"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e,
|
| 572 |
-
los = [s["ci95_lo"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e,
|
| 573 |
-
his = [s["ci95_hi"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e,
|
| 574 |
pt_mean = float(np.mean(pts))
|
| 575 |
half_w = float(np.sqrt(np.sum([(h - l) ** 2 for l, h in zip(los, his)])) / (2.0 * 6.0))
|
| 576 |
-
|
| 577 |
"point_pct": round(pt_mean, 2),
|
| 578 |
"ci95_lo": round(pt_mean - half_w, 2),
|
| 579 |
"ci95_hi": round(pt_mean + half_w, 2),
|
| 580 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
| 581 |
|
| 582 |
models_cost = [
|
| 583 |
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 584 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 585 |
-
"Local
|
| 586 |
-
"
|
| 587 |
-
"Cheap OpenAI GPT (
|
| 588 |
-
"
|
| 589 |
]
|
| 590 |
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
| 591 |
batched_cost = [0.002, 0.004, 1.85, 94.90, 127.30, 712.00]
|
|
@@ -619,8 +621,8 @@ def build_all_charts_and_manifest(
|
|
| 619 |
edges_llm = [C_ULTRA_EDGE, C_INT8_EDGE, "none", "none", "none", "none"]
|
| 620 |
vals_left = [composite_stats[m]["point_pct"] for m in llm_order]
|
| 621 |
labels_left = [
|
| 622 |
-
f"{vals_left[0]:.2f}%
|
| 623 |
-
f"{vals_left[1]:.2f}% (2.44 ms M4 Pro CPU INT8 | $0.00 API)",
|
| 624 |
f"{vals_left[2]:.2f}% (~450 ms | Non-Circular LOTO N=1,025)",
|
| 625 |
f"{vals_left[3]:.2f}% (~310 ms | $712.00-$1,220.00 / 1M)",
|
| 626 |
f"{vals_left[4]:.2f}% (~365 ms | Non-Circular LOTO N=1,025)",
|
|
@@ -713,17 +715,17 @@ def build_all_charts_and_manifest(
|
|
| 713 |
(40, 162), [40, 60, 80, 100],
|
| 714 |
),
|
| 715 |
(
|
| 716 |
-
1, 4, "2F. Swarm Liveliness AUROC
|
| 717 |
-
|
| 718 |
[
|
| 719 |
-
"
|
| 720 |
"92.42% Pub (86.64% Priors)",
|
| 721 |
"89.20% Pub (N=757)",
|
| 722 |
"89.76% Pub (N=757)",
|
| 723 |
"89.82% Pub (N=757)",
|
| 724 |
"66.32% Pub (N=757)",
|
| 725 |
],
|
| 726 |
-
(55,
|
| 727 |
),
|
| 728 |
]
|
| 729 |
|
|
@@ -754,10 +756,10 @@ def build_all_charts_and_manifest(
|
|
| 754 |
legend_handles = [
|
| 755 |
Patch(facecolor=C_ULTRA, edgecolor=C_ULTRA_EDGE, label="Armiger Ultra (Fair Hybrid)"),
|
| 756 |
Patch(facecolor=C_INT8, edgecolor=C_INT8_EDGE, label="Armiger (8.3 MB INT8 + Plugins)"),
|
| 757 |
-
Patch(facecolor=C_GEMINI, label="Google Gemini
|
| 758 |
-
Patch(facecolor=C_CLAUDE, label="Anthropic Claude
|
| 759 |
-
Patch(facecolor=C_GPT, label="OpenAI GPT (
|
| 760 |
-
Patch(facecolor=C_VINCI, label="Local
|
| 761 |
]
|
| 762 |
fig1.legend(
|
| 763 |
handles=legend_handles, loc="lower center", bbox_to_anchor=(0.5, 0.015),
|
|
@@ -784,10 +786,10 @@ def build_all_charts_and_manifest(
|
|
| 784 |
models_cost = [
|
| 785 |
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 786 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 787 |
-
"Local
|
| 788 |
-
"
|
| 789 |
-
"Cheap OpenAI GPT (
|
| 790 |
-
"
|
| 791 |
]
|
| 792 |
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
| 793 |
batched_cost = [0.002, 0.004, 1.85, 94.90, 127.30, 712.00]
|
|
@@ -974,14 +976,15 @@ def build_all_charts_and_manifest(
|
|
| 974 |
c3_p6a_stats = compute_paired_panel_stats(
|
| 975 |
"chart3_6a_oct15_junk2170",
|
| 976 |
{
|
| 977 |
-
"Armiger Ultra":
|
| 978 |
-
"Armiger INT8":
|
| 979 |
-
"Bitmagnet":
|
| 980 |
-
"RTN Filter":
|
| 981 |
},
|
| 982 |
ref_model="Armiger Ultra",
|
| 983 |
expected_n=2170,
|
| 984 |
-
metric_type="
|
|
|
|
| 985 |
)
|
| 986 |
|
| 987 |
oct_cat_spec = [
|
|
@@ -1220,13 +1223,19 @@ def build_all_charts_and_manifest(
|
|
| 1220 |
style_ax(ax6a)
|
| 1221 |
ax6a.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1222 |
m6a = ["Armiger Ultra", "Armiger INT8", "Bitmagnet", "RTN Filter", "Other 6 Tools"]
|
| 1223 |
-
v6a = [
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1224 |
c6a = [C_ULTRA, C_INT8, "#94A3B8", "#F59E0B", "#21262D"]
|
| 1225 |
l6a = [
|
| 1226 |
-
"
|
| 1227 |
-
"
|
| 1228 |
-
"
|
| 1229 |
-
"
|
| 1230 |
"N/A (No Junk Head)",
|
| 1231 |
]
|
| 1232 |
for idx, (v, lbl) in enumerate(zip(v6a, l6a)):
|
|
@@ -1243,23 +1252,22 @@ def build_all_charts_and_manifest(
|
|
| 1243 |
ax6a.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1244 |
ax6a.set_title("6A. Junk / Malware Detection (oct15 N=2,170 & beyond N=977)", fontsize=10, fontweight="bold", color=TEXT_MAIN, pad=7)
|
| 1245 |
|
| 1246 |
-
# Panel 6B: Swarm Liveliness AUROC
|
| 1247 |
ax6b = fig3.add_subplot(gs3[6, 3])
|
| 1248 |
style_ax(ax6b)
|
| 1249 |
ax6b.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1250 |
m6b = ["Armiger Ultra", "Armiger Priors", "Seeder Count", "All 8 Tools"]
|
| 1251 |
-
|
| 1252 |
-
p2f_whse_stats["models"]["Armiger Ultra (Fair Hybrid)"],
|
| 1253 |
-
p2f_whse_stats["models"]["Armiger 4-Way Priors"],
|
| 1254 |
-
|
| 1255 |
None,
|
| 1256 |
]
|
| 1257 |
-
v6b = [96.83, 86.64, 85.17, None]
|
| 1258 |
c6b = [C_ULTRA, C_INT8, "#64748B", "#21262D"]
|
| 1259 |
l6b = [
|
| 1260 |
-
"
|
| 1261 |
-
"
|
| 1262 |
-
"
|
| 1263 |
"N/A (No Swarm Head)",
|
| 1264 |
]
|
| 1265 |
for idx, (v, lbl) in enumerate(zip(v6b, l6b)):
|
|
@@ -1274,7 +1282,7 @@ def build_all_charts_and_manifest(
|
|
| 1274 |
ax6b.invert_yaxis()
|
| 1275 |
ax6b.set_xlim(48, 144)
|
| 1276 |
ax6b.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1277 |
-
ax6b.set_xlabel("
|
| 1278 |
ax6b.set_title("6B. Swarm Liveliness AUROC % (swarm_adapter.pt)", fontsize=10, fontweight="bold", color=TEXT_MAIN, pad=7)
|
| 1279 |
|
| 1280 |
sha_chart3 = save_clean_rgb_png(fig3, assets_dir / "armiger_vs_torrent_classifiers_6dims.png")
|
|
@@ -1285,7 +1293,7 @@ def build_all_charts_and_manifest(
|
|
| 1285 |
fig4, (ax4_l, ax4_r) = plt.subplots(1, 2, figsize=(22.5, 11.2), gridspec_kw={"wspace": 0.22})
|
| 1286 |
fig4.subplots_adjust(top=0.84, bottom=0.16, left=0.11, right=0.98)
|
| 1287 |
fig4.suptitle(
|
| 1288 |
-
"
|
| 1289 |
fontsize=17, fontweight="bold", color=TEXT_MAIN, y=0.965,
|
| 1290 |
)
|
| 1291 |
fig4.text(
|
|
@@ -1342,8 +1350,8 @@ def build_all_charts_and_manifest(
|
|
| 1342 |
ax4_l.set_ylim(len(cats_oct_labels) - 0.35, -0.6)
|
| 1343 |
ax4_l.set_xlim(0, 128)
|
| 1344 |
ax4_l.set_xticks([0, 20, 40, 60, 80, 100])
|
| 1345 |
-
ax4_l.set_xlabel("
|
| 1346 |
-
ax4_l.set_title("1. Per-Category
|
| 1347 |
for idx in range(len(cats_oct_labels)):
|
| 1348 |
ax4_l.text(oct_ultra[idx] + 1.0, idx - 1.1 * h, f"{oct_ultra[idx]:.1f}%", va="center", ha="left", fontsize=8.0, fontweight="bold", color=C_ULTRA_EDGE)
|
| 1349 |
ax4_l.text(oct_mimmo[idx] + 1.0, idx + 0.5 * h, f"{oct_mimmo[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color="#CBD5E1")
|
|
@@ -1390,15 +1398,15 @@ def build_all_charts_and_manifest(
|
|
| 1390 |
style_ax(ax4_r)
|
| 1391 |
ax4_r.barh(y_c - 1.5 * h, bc_ultra, height=h, color=C_ULTRA, label="Armiger Ultra Fair Hybrid (95.12% Non-Circular)", zorder=3)
|
| 1392 |
ax4_r.barh(y_c - 0.5 * h, bc_int8, height=h, color=C_INT8, label="Armiger INT8 + ZeroCue (95.12% Non-Circular, 92.88% Raw)", zorder=3)
|
| 1393 |
-
ax4_r.barh(y_c + 0.5 * h, bc_gpt, height=h, color=C_GPT, label="OpenAI GPT
|
| 1394 |
-
ax4_r.barh(y_c + 1.5 * h, bc_gem, height=h, color=C_GEMINI, label="Google Gemini
|
| 1395 |
ax4_r.set_yticks(y_c)
|
| 1396 |
ax4_r.set_yticklabels(cats_bc_labels, fontsize=10, fontweight="bold", color=TEXT_MAIN)
|
| 1397 |
ax4_r.set_ylim(len(cats_bc_labels) - 0.35, -0.6)
|
| 1398 |
ax4_r.set_xlim(0, 128)
|
| 1399 |
ax4_r.set_xticks([0, 20, 40, 60, 80, 100])
|
| 1400 |
-
ax4_r.set_xlabel("
|
| 1401 |
-
ax4_r.set_title("2. Per-Category
|
| 1402 |
for idx in range(len(cats_bc_labels)):
|
| 1403 |
ax4_r.text(bc_ultra[idx] + 1.0, idx - 1.1 * h, f"{bc_ultra[idx]:.1f}%", va="center", ha="left", fontsize=8.0, fontweight="bold", color=C_ULTRA_EDGE)
|
| 1404 |
ax4_r.text(bc_gpt[idx] + 1.0, idx + 0.5 * h, f"{bc_gpt[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color=C_GPT)
|
|
|
|
| 64 |
C_ERR = "#E2E8F0"
|
| 65 |
|
| 66 |
TEACHER_MODELS = {
|
| 67 |
+
"OpenAI GPT (Fast Tier)",
|
| 68 |
+
"Google Gemini Flash",
|
| 69 |
+
"Anthropic Claude Haiku 4.5",
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
}
|
| 71 |
|
| 72 |
|
|
|
|
| 237 |
point_scores[m] = float(np.sum(v10_int2) * 100.0 / (2.0 * n_pos * n_neg))
|
| 238 |
w_pos = rng.multinomial(n_pos, np.full(n_pos, 1.0 / n_pos), size=n_boot).astype(np.float64)
|
| 239 |
w_neg = rng.multinomial(n_neg, np.full(n_neg, 1.0 / n_neg), size=n_boot).astype(np.float64)
|
| 240 |
+
pt_vec = np.array([point_scores[m] for m in model_names], dtype=np.float64)[None, :]
|
| 241 |
+
boot_scores = (
|
| 242 |
(w_pos @ placement_int2[:, pos_idx].T) * (100.0 / (2.0 * n_pos * n_neg))
|
| 243 |
+ (w_neg @ placement_int2[:, neg_idx].T) * (100.0 / (2.0 * n_pos * n_neg))
|
| 244 |
+
- pt_vec
|
| 245 |
)
|
| 246 |
item_correctness = {m: placement_unit[idx] for idx, m in enumerate(model_names)}
|
| 247 |
else:
|
|
|
|
| 419 |
# --------------------------------------------------------------------------
|
| 420 |
anti_circ_checks = [
|
| 421 |
assert_non_circular_teacher_eval(
|
| 422 |
+
"OpenAI GPT (Fast Tier)",
|
| 423 |
eval_mode="leave_one_teacher_out",
|
| 424 |
self_vote_weight=0.0,
|
| 425 |
noncirc_vec=d["beyond_cat11_corr_gpt_noncirc"],
|
| 426 |
circ_vec=d["beyond_cat11_corr_gpt_circ"],
|
| 427 |
),
|
| 428 |
assert_non_circular_teacher_eval(
|
| 429 |
+
"Google Gemini Flash",
|
| 430 |
eval_mode="leave_one_teacher_out",
|
| 431 |
self_vote_weight=0.0,
|
| 432 |
noncirc_vec=d["beyond_cat11_corr_gemini_noncirc"],
|
| 433 |
circ_vec=d["beyond_cat11_corr_gemini_circ"],
|
| 434 |
),
|
| 435 |
assert_non_circular_teacher_eval(
|
| 436 |
+
"Anthropic Claude Haiku 4.5",
|
| 437 |
eval_mode="independent_ground_truth",
|
| 438 |
self_vote_weight=0.0,
|
| 439 |
),
|
|
|
|
| 445 |
llm_order = [
|
| 446 |
"Armiger Ultra (Fair Hybrid)",
|
| 447 |
"Armiger (8.3 MB INT8 + Plugins)",
|
| 448 |
+
"Google Gemini Flash",
|
| 449 |
+
"Anthropic Claude Haiku 4.5",
|
| 450 |
+
"OpenAI GPT (Fast Tier)",
|
| 451 |
+
"Local 8B Instruct LLM (FP16)",
|
| 452 |
]
|
| 453 |
ref_llm = "Armiger Ultra (Fair Hybrid)"
|
| 454 |
|
|
|
|
| 557 |
y_true=d["swarm_whse_y_true"],
|
| 558 |
)
|
| 559 |
|
| 560 |
+
# Composite 6-dimension mean across 2A..2F on strict identical cohorts (2F = Public N=757 for all models)
|
| 561 |
composite_stats: Dict[str, Any] = {}
|
| 562 |
for m in llm_order:
|
| 563 |
s_2a = p2a_stats["models"][m]
|
|
|
|
| 565 |
s_2c = p2c_stats["models"][m]
|
| 566 |
s_2d = p2d_stats["models"][m]
|
| 567 |
s_2e = p2e_stats["models"][m]
|
| 568 |
+
s_2f_pub = p2f_pub_stats["models"][m]
|
| 569 |
+
pts = [s["point_pct"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
|
| 570 |
+
los = [s["ci95_lo"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
|
| 571 |
+
his = [s["ci95_hi"] for s in (s_2a, s_2b, s_2c, s_2d, s_2e, s_2f_pub)]
|
| 572 |
pt_mean = float(np.mean(pts))
|
| 573 |
half_w = float(np.sqrt(np.sum([(h - l) ** 2 for l, h in zip(los, his)])) / (2.0 * 6.0))
|
| 574 |
+
entry: Dict[str, Any] = {
|
| 575 |
"point_pct": round(pt_mean, 2),
|
| 576 |
"ci95_lo": round(pt_mean - half_w, 2),
|
| 577 |
"ci95_hi": round(pt_mean + half_w, 2),
|
| 578 |
}
|
| 579 |
+
if m == ref_llm:
|
| 580 |
+
pts_whse = pts[:-1] + [p2f_whse_stats["models"][m]["point_pct"]]
|
| 581 |
+
entry["point_with_corpus_swarm_pct"] = round(float(np.mean(pts_whse)), 2)
|
| 582 |
+
composite_stats[m] = entry
|
| 583 |
|
| 584 |
models_cost = [
|
| 585 |
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 586 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 587 |
+
"Local 8B Instruct LLM (FP16)",
|
| 588 |
+
"Google Gemini Flash",
|
| 589 |
+
"Cheap OpenAI GPT (Fast Tier)",
|
| 590 |
+
"Anthropic Claude Haiku 4.5",
|
| 591 |
]
|
| 592 |
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
| 593 |
batched_cost = [0.002, 0.004, 1.85, 94.90, 127.30, 712.00]
|
|
|
|
| 621 |
edges_llm = [C_ULTRA_EDGE, C_INT8_EDGE, "none", "none", "none", "none"]
|
| 622 |
vals_left = [composite_stats[m]["point_pct"] for m in llm_order]
|
| 623 |
labels_left = [
|
| 624 |
+
f"{vals_left[0]:.2f}% Pub-6D | 97.60% w/ Corpus Swarm (0.21 ms RTX)",
|
| 625 |
+
f"{vals_left[1]:.2f}% Pub-6D (2.44 ms M4 Pro CPU INT8 | $0.00 API)",
|
| 626 |
f"{vals_left[2]:.2f}% (~450 ms | Non-Circular LOTO N=1,025)",
|
| 627 |
f"{vals_left[3]:.2f}% (~310 ms | $712.00-$1,220.00 / 1M)",
|
| 628 |
f"{vals_left[4]:.2f}% (~365 ms | Non-Circular LOTO N=1,025)",
|
|
|
|
| 715 |
(40, 162), [40, 60, 80, 100],
|
| 716 |
),
|
| 717 |
(
|
| 718 |
+
1, 4, "2F. Swarm Liveliness AUROC (Identical N=757)",
|
| 719 |
+
p2f_pub_stats,
|
| 720 |
[
|
| 721 |
+
"92.42% Pub (96.83% Corpus N=10.3k)",
|
| 722 |
"92.42% Pub (86.64% Priors)",
|
| 723 |
"89.20% Pub (N=757)",
|
| 724 |
"89.76% Pub (N=757)",
|
| 725 |
"89.82% Pub (N=757)",
|
| 726 |
"66.32% Pub (N=757)",
|
| 727 |
],
|
| 728 |
+
(55, 155), [60, 70, 80, 90, 100],
|
| 729 |
),
|
| 730 |
]
|
| 731 |
|
|
|
|
| 756 |
legend_handles = [
|
| 757 |
Patch(facecolor=C_ULTRA, edgecolor=C_ULTRA_EDGE, label="Armiger Ultra (Fair Hybrid)"),
|
| 758 |
Patch(facecolor=C_INT8, edgecolor=C_INT8_EDGE, label="Armiger (8.3 MB INT8 + Plugins)"),
|
| 759 |
+
Patch(facecolor=C_GEMINI, label="Google Gemini Flash (LOTO Non-Circular)"),
|
| 760 |
+
Patch(facecolor=C_CLAUDE, label="Anthropic Claude Haiku 4.5"),
|
| 761 |
+
Patch(facecolor=C_GPT, label="OpenAI GPT (Fast Tier, LOTO Non-Circular)"),
|
| 762 |
+
Patch(facecolor=C_VINCI, label="Local 8B Instruct LLM (FP16)"),
|
| 763 |
]
|
| 764 |
fig1.legend(
|
| 765 |
handles=legend_handles, loc="lower center", bbox_to_anchor=(0.5, 0.015),
|
|
|
|
| 786 |
models_cost = [
|
| 787 |
"Armiger (8.3 MB M4 Pro CPU INT8)",
|
| 788 |
"Armiger Ultra (M4 Pro / RTX GPU FP32)",
|
| 789 |
+
"Local 8B Instruct LLM (FP16)",
|
| 790 |
+
"Google Gemini Flash",
|
| 791 |
+
"Cheap OpenAI GPT (Fast Tier)",
|
| 792 |
+
"Anthropic Claude Haiku 4.5",
|
| 793 |
]
|
| 794 |
colors_cost = [C_INT8, C_ULTRA, C_VINCI, C_GEMINI, C_GPT, C_CLAUDE]
|
| 795 |
batched_cost = [0.002, 0.004, 1.85, 94.90, 127.30, 712.00]
|
|
|
|
| 976 |
c3_p6a_stats = compute_paired_panel_stats(
|
| 977 |
"chart3_6a_oct15_junk2170",
|
| 978 |
{
|
| 979 |
+
"Armiger Ultra": d["oct15_junk_pred_ultra"],
|
| 980 |
+
"Armiger INT8": d["oct15_junk_pred_int8"],
|
| 981 |
+
"Bitmagnet": d["oct15_junk_pred_bitmagnet"],
|
| 982 |
+
"RTN Filter": d["oct15_junk_pred_rtn"],
|
| 983 |
},
|
| 984 |
ref_model="Armiger Ultra",
|
| 985 |
expected_n=2170,
|
| 986 |
+
metric_type="balanced",
|
| 987 |
+
y_true=y_oct_jk,
|
| 988 |
)
|
| 989 |
|
| 990 |
oct_cat_spec = [
|
|
|
|
| 1223 |
style_ax(ax6a)
|
| 1224 |
ax6a.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1225 |
m6a = ["Armiger Ultra", "Armiger INT8", "Bitmagnet", "RTN Filter", "Other 6 Tools"]
|
| 1226 |
+
v6a = [
|
| 1227 |
+
c3_p6a_stats["models"]["Armiger Ultra"]["point_pct"],
|
| 1228 |
+
c3_p6a_stats["models"]["Armiger INT8"]["point_pct"],
|
| 1229 |
+
c3_p6a_stats["models"]["Bitmagnet"]["point_pct"],
|
| 1230 |
+
c3_p6a_stats["models"]["RTN Filter"]["point_pct"],
|
| 1231 |
+
None,
|
| 1232 |
+
]
|
| 1233 |
c6a = [C_ULTRA, C_INT8, "#94A3B8", "#F59E0B", "#21262D"]
|
| 1234 |
l6a = [
|
| 1235 |
+
f"{v6a[0]:.2f}% Bal (98.41% beyond N=977)",
|
| 1236 |
+
f"{v6a[1]:.2f}% Bal (98.41% beyond N=977)",
|
| 1237 |
+
f"{v6a[2]:.2f}% Bal (0% Recall, 100% Spec)",
|
| 1238 |
+
f"{v6a[3]:.2f}% Bal (0% Recall, 98.57% Spec)",
|
| 1239 |
"N/A (No Junk Head)",
|
| 1240 |
]
|
| 1241 |
for idx, (v, lbl) in enumerate(zip(v6a, l6a)):
|
|
|
|
| 1252 |
ax6a.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1253 |
ax6a.set_title("6A. Junk / Malware Detection (oct15 N=2,170 & beyond N=977)", fontsize=10, fontweight="bold", color=TEXT_MAIN, pad=7)
|
| 1254 |
|
| 1255 |
+
# Panel 6B: Swarm Liveliness AUROC (Identical N=10,278 Multi-Source Corpus)
|
| 1256 |
ax6b = fig3.add_subplot(gs3[6, 3])
|
| 1257 |
style_ax(ax6b)
|
| 1258 |
ax6b.axhspan(-0.5, 1.5, color=BG_HIGHLIGHT, alpha=0.75, zorder=1)
|
| 1259 |
m6b = ["Armiger Ultra", "Armiger Priors", "Seeder Count", "All 8 Tools"]
|
| 1260 |
+
v6b = [
|
| 1261 |
+
p2f_whse_stats["models"]["Armiger Ultra (Fair Hybrid)"]["point_pct"],
|
| 1262 |
+
p2f_whse_stats["models"]["Armiger 4-Way Priors"]["point_pct"],
|
| 1263 |
+
p2f_whse_stats["models"]["Raw Seeder Count"]["point_pct"],
|
| 1264 |
None,
|
| 1265 |
]
|
|
|
|
| 1266 |
c6b = [C_ULTRA, C_INT8, "#64748B", "#21262D"]
|
| 1267 |
l6b = [
|
| 1268 |
+
f"{v6b[0]:.2f}% Corpus (92.42% Pub N=757)",
|
| 1269 |
+
f"{v6b[1]:.2f}% Corpus (4-Way Priors)",
|
| 1270 |
+
f"{v6b[2]:.2f}% Corpus (85.17% Pub N=757)",
|
| 1271 |
"N/A (No Swarm Head)",
|
| 1272 |
]
|
| 1273 |
for idx, (v, lbl) in enumerate(zip(v6b, l6b)):
|
|
|
|
| 1282 |
ax6b.invert_yaxis()
|
| 1283 |
ax6b.set_xlim(48, 144)
|
| 1284 |
ax6b.set_xticks([50, 60, 70, 80, 90, 100])
|
| 1285 |
+
ax6b.set_xlabel("Multi-Source Corpus (Identical N=10,278) & Public Tracker (N=757: 92.42%)", fontsize=8.2, color=TEXT_SUB, labelpad=6)
|
| 1286 |
ax6b.set_title("6B. Swarm Liveliness AUROC % (swarm_adapter.pt)", fontsize=10, fontweight="bold", color=TEXT_MAIN, pad=7)
|
| 1287 |
|
| 1288 |
sha_chart3 = save_clean_rgb_png(fig3, assets_dir / "armiger_vs_torrent_classifiers_6dims.png")
|
|
|
|
| 1293 |
fig4, (ax4_l, ax4_r) = plt.subplots(1, 2, figsize=(22.5, 11.2), gridspec_kw={"wspace": 0.22})
|
| 1294 |
fig4.subplots_adjust(top=0.84, bottom=0.16, left=0.11, right=0.98)
|
| 1295 |
fig4.suptitle(
|
| 1296 |
+
"Per-Category Recall Breakdown Across All 10 Known Media & Software Categories",
|
| 1297 |
fontsize=17, fontweight="bold", color=TEXT_MAIN, y=0.965,
|
| 1298 |
)
|
| 1299 |
fig4.text(
|
|
|
|
| 1350 |
ax4_l.set_ylim(len(cats_oct_labels) - 0.35, -0.6)
|
| 1351 |
ax4_l.set_xlim(0, 128)
|
| 1352 |
ax4_l.set_xticks([0, 20, 40, 60, 80, 100])
|
| 1353 |
+
ax4_l.set_xlabel("Per-Class Recall (%) — Identical N = 2,835 Ground-Truth Mask", fontsize=9.8, color=TEXT_SUB, labelpad=8)
|
| 1354 |
+
ax4_l.set_title("1. Per-Category Recall vs. Open-Source Classifiers\n(oct15_confirmation, Identical N = 2,835)", fontsize=11.5, fontweight="bold", color=TEXT_MAIN, pad=10)
|
| 1355 |
for idx in range(len(cats_oct_labels)):
|
| 1356 |
ax4_l.text(oct_ultra[idx] + 1.0, idx - 1.1 * h, f"{oct_ultra[idx]:.1f}%", va="center", ha="left", fontsize=8.0, fontweight="bold", color=C_ULTRA_EDGE)
|
| 1357 |
ax4_l.text(oct_mimmo[idx] + 1.0, idx + 0.5 * h, f"{oct_mimmo[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color="#CBD5E1")
|
|
|
|
| 1398 |
style_ax(ax4_r)
|
| 1399 |
ax4_r.barh(y_c - 1.5 * h, bc_ultra, height=h, color=C_ULTRA, label="Armiger Ultra Fair Hybrid (95.12% Non-Circular)", zorder=3)
|
| 1400 |
ax4_r.barh(y_c - 0.5 * h, bc_int8, height=h, color=C_INT8, label="Armiger INT8 + ZeroCue (95.12% Non-Circular, 92.88% Raw)", zorder=3)
|
| 1401 |
+
ax4_r.barh(y_c + 0.5 * h, bc_gpt, height=h, color=C_GPT, label="OpenAI GPT Fast Tier (95.12% Non-Circ | 96.59% Circ)", zorder=3)
|
| 1402 |
+
ax4_r.barh(y_c + 1.5 * h, bc_gem, height=h, color=C_GEMINI, label="Google Gemini Flash (93.80% Non-Circ | 95.61% Circ)", zorder=3)
|
| 1403 |
ax4_r.set_yticks(y_c)
|
| 1404 |
ax4_r.set_yticklabels(cats_bc_labels, fontsize=10, fontweight="bold", color=TEXT_MAIN)
|
| 1405 |
ax4_r.set_ylim(len(cats_bc_labels) - 0.35, -0.6)
|
| 1406 |
ax4_r.set_xlim(0, 128)
|
| 1407 |
ax4_r.set_xticks([0, 20, 40, 60, 80, 100])
|
| 1408 |
+
ax4_r.set_xlabel("Per-Class Recall (%) — Leave-One-Teacher-Out Non-Circular Reference", fontsize=9.8, color=TEXT_SUB, labelpad=8)
|
| 1409 |
+
ax4_r.set_title("2. Per-Category Recall vs. Cloud LLMs\n(beyond_confirmation, Identical N = 1,025, LOTO Non-Circular)", fontsize=11.5, fontweight="bold", color=TEXT_MAIN, pad=10)
|
| 1410 |
for idx in range(len(cats_bc_labels)):
|
| 1411 |
ax4_r.text(bc_ultra[idx] + 1.0, idx - 1.1 * h, f"{bc_ultra[idx]:.1f}%", va="center", ha="left", fontsize=8.0, fontweight="bold", color=C_ULTRA_EDGE)
|
| 1412 |
ax4_r.text(bc_gpt[idx] + 1.0, idx + 0.5 * h, f"{bc_gpt[idx]:.1f}%", va="center", ha="left", fontsize=7.6, color=C_GPT)
|
benchmarks/test_chart_integrity.py
CHANGED
|
@@ -69,13 +69,13 @@ def test_anti_circularity_lock_enforced():
|
|
| 69 |
d = _load_vectors(BENCH_DIR / "canonical_eval_vectors.json")
|
| 70 |
|
| 71 |
with pytest.raises(AssertionError, match="Anti-Circularity Lock violated"):
|
| 72 |
-
assert_non_circular_teacher_eval("OpenAI GPT (
|
| 73 |
|
| 74 |
with pytest.raises(AssertionError, match="Anti-Circularity Lock violated"):
|
| 75 |
-
assert_non_circular_teacher_eval("Google Gemini
|
| 76 |
|
| 77 |
gpt_audit = assert_non_circular_teacher_eval(
|
| 78 |
-
"OpenAI GPT (
|
| 79 |
eval_mode="leave_one_teacher_out",
|
| 80 |
self_vote_weight=0.0,
|
| 81 |
noncirc_vec=d["beyond_cat11_corr_gpt_noncirc"],
|
|
@@ -86,7 +86,7 @@ def test_anti_circularity_lock_enforced():
|
|
| 86 |
assert gpt_audit["circularity_inflation_pp"] > 1.0
|
| 87 |
|
| 88 |
gem_audit = assert_non_circular_teacher_eval(
|
| 89 |
-
"Google Gemini
|
| 90 |
eval_mode="leave_one_teacher_out",
|
| 91 |
self_vote_weight=0.0,
|
| 92 |
noncirc_vec=d["beyond_cat11_corr_gemini_noncirc"],
|
|
@@ -117,7 +117,7 @@ def test_paired_bootstrap_and_mcnemar_reproducibility():
|
|
| 117 |
"Armiger Ultra": (d["oct15_aud545_pred_ultra"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 118 |
"Armiger INT8": (d["oct15_aud545_pred_int8_hyb"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 119 |
"GuessIt": (d["oct15_aud545_pred_guessit"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 120 |
-
"GPT
|
| 121 |
}
|
| 122 |
res1 = compute_paired_panel_stats("test_aud545", vecs, ref_model="Armiger Ultra", expected_n=545)
|
| 123 |
res2 = compute_paired_panel_stats("test_aud545", vecs, ref_model="Armiger Ultra", expected_n=545)
|
|
@@ -128,8 +128,8 @@ def test_paired_bootstrap_and_mcnemar_reproducibility():
|
|
| 128 |
assert 0.0 <= st["p_value_vs_ref"] <= 1.0
|
| 129 |
assert res1["models"]["Armiger Ultra"]["point_pct"] == 97.80
|
| 130 |
assert res1["models"]["GuessIt"]["point_pct"] == 94.86
|
| 131 |
-
assert res1["models"]["GPT
|
| 132 |
-
assert res1["models"]["GPT
|
| 133 |
|
| 134 |
|
| 135 |
def test_manifest_sha256_and_png_chunks(tmp_path: pathlib.Path):
|
|
|
|
| 69 |
d = _load_vectors(BENCH_DIR / "canonical_eval_vectors.json")
|
| 70 |
|
| 71 |
with pytest.raises(AssertionError, match="Anti-Circularity Lock violated"):
|
| 72 |
+
assert_non_circular_teacher_eval("OpenAI GPT (Fast Tier)", eval_mode="circular_majority", self_vote_weight=0.333)
|
| 73 |
|
| 74 |
with pytest.raises(AssertionError, match="Anti-Circularity Lock violated"):
|
| 75 |
+
assert_non_circular_teacher_eval("Google Gemini Flash", eval_mode="leave_one_teacher_out", self_vote_weight=0.5)
|
| 76 |
|
| 77 |
gpt_audit = assert_non_circular_teacher_eval(
|
| 78 |
+
"OpenAI GPT (Fast Tier)",
|
| 79 |
eval_mode="leave_one_teacher_out",
|
| 80 |
self_vote_weight=0.0,
|
| 81 |
noncirc_vec=d["beyond_cat11_corr_gpt_noncirc"],
|
|
|
|
| 86 |
assert gpt_audit["circularity_inflation_pp"] > 1.0
|
| 87 |
|
| 88 |
gem_audit = assert_non_circular_teacher_eval(
|
| 89 |
+
"Google Gemini Flash",
|
| 90 |
eval_mode="leave_one_teacher_out",
|
| 91 |
self_vote_weight=0.0,
|
| 92 |
noncirc_vec=d["beyond_cat11_corr_gemini_noncirc"],
|
|
|
|
| 117 |
"Armiger Ultra": (d["oct15_aud545_pred_ultra"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 118 |
"Armiger INT8": (d["oct15_aud545_pred_int8_hyb"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 119 |
"GuessIt": (d["oct15_aud545_pred_guessit"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 120 |
+
"OpenAI GPT (Fast Tier)": (d["oct15_aud545_pred_gpt"] == d["oct15_aud545_y_en"]).astype(np.float32),
|
| 121 |
}
|
| 122 |
res1 = compute_paired_panel_stats("test_aud545", vecs, ref_model="Armiger Ultra", expected_n=545)
|
| 123 |
res2 = compute_paired_panel_stats("test_aud545", vecs, ref_model="Armiger Ultra", expected_n=545)
|
|
|
|
| 128 |
assert 0.0 <= st["p_value_vs_ref"] <= 1.0
|
| 129 |
assert res1["models"]["Armiger Ultra"]["point_pct"] == 97.80
|
| 130 |
assert res1["models"]["GuessIt"]["point_pct"] == 94.86
|
| 131 |
+
assert res1["models"]["OpenAI GPT (Fast Tier)"]["point_pct"] == 94.68
|
| 132 |
+
assert res1["models"]["OpenAI GPT (Fast Tier)"]["p_value_vs_ref"] < 0.01
|
| 133 |
|
| 134 |
|
| 135 |
def test_manifest_sha256_and_png_chunks(tmp_path: pathlib.Path):
|
config.json
CHANGED
|
@@ -1,39 +1,28 @@
|
|
| 1 |
{
|
| 2 |
"model_name": "armiger-ultra",
|
| 3 |
"version": "2.1.0",
|
| 4 |
-
"bundle_type": "
|
| 5 |
"architecture": "armiger-v2-byte-rope-transformer-8head + residual-swarm-health-adapter",
|
| 6 |
-
"checkpoint": "H350000_Curated_Wave2-seed0 (direct-side hardened junk head)",
|
| 7 |
"artifacts": {
|
| 8 |
"armiger_cpu_int8_onnx": {
|
| 9 |
"file": "armiger.int8.onnx",
|
| 10 |
"size_bytes": 8283542,
|
| 11 |
-
"sha256": "
|
| 12 |
},
|
| 13 |
"armiger_ultra_fp32_onnx": {
|
| 14 |
"file": "armiger_ultra.fp32.onnx",
|
| 15 |
"size_bytes": 32287709,
|
| 16 |
-
"sha256": "
|
| 17 |
},
|
| 18 |
"armiger_ultra_pytorch": {
|
| 19 |
"file": "armiger_ultra.pt",
|
| 20 |
"size_bytes": 32235023,
|
| 21 |
-
"sha256": "
|
| 22 |
},
|
| 23 |
"swarm_health_adapter": {
|
| 24 |
"file": "swarm_adapter.pt",
|
| 25 |
"size_bytes": 131718,
|
| 26 |
"sha256": "85effcb9e70af7bff6047d4ae45d2155444e9b691dc84431f19ad14f9daee9ea"
|
| 27 |
-
},
|
| 28 |
-
"vocab": {
|
| 29 |
-
"file": "vocab.json",
|
| 30 |
-
"size_bytes": 39991,
|
| 31 |
-
"sha256": "9853b5cd56d07beee64e6fc44396084d9c70167c4d74fbc149c0d4e021330b4e"
|
| 32 |
-
},
|
| 33 |
-
"standalone_inference": {
|
| 34 |
-
"file": "armiger_inference.py",
|
| 35 |
-
"size_bytes": 23176,
|
| 36 |
-
"sha256": "8dd45e3853cb6430f01bb319bdf2b83ec79a1fbf75daf5de826e74e70849e261"
|
| 37 |
}
|
| 38 |
},
|
| 39 |
"heads": {
|
|
@@ -86,79 +75,20 @@
|
|
| 86 |
"junk": 0.101,
|
| 87 |
"owner_scope": 0.477
|
| 88 |
},
|
| 89 |
-
"
|
| 90 |
-
"
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
"owner_scope_n2669_acc": 98.65,
|
| 104 |
-
"title_language_n969_acc": 93.6
|
| 105 |
-
},
|
| 106 |
-
"fresh_confirmation_n1402": {
|
| 107 |
-
"standalone_neural_11class_acc": 94.51,
|
| 108 |
-
"fair_hybrid_11class_acc": 94.58,
|
| 109 |
-
"adult_n1393_acc": 98.21,
|
| 110 |
-
"junk_n1291_acc": 100.0,
|
| 111 |
-
"anime_n253_acc": 99.6
|
| 112 |
-
},
|
| 113 |
-
"beyond_confirmation_n1025": {
|
| 114 |
-
"standalone_neural_11class_acc": 92.88,
|
| 115 |
-
"fair_hybrid_11class_acc": 95.12,
|
| 116 |
-
"adult_n1016_acc": 98.62,
|
| 117 |
-
"junk_n977_acc": 99.9,
|
| 118 |
-
"anime_n636_acc": 98.9,
|
| 119 |
-
"title_language_n604_acc": 98.51
|
| 120 |
-
},
|
| 121 |
-
"tc_truespec_2000_n2000": {
|
| 122 |
-
"standalone_neural_11class_acc": 87.4,
|
| 123 |
-
"fair_hybrid_11class_acc": 91.5,
|
| 124 |
-
"adult_acc": 99.9,
|
| 125 |
-
"junk_acc": 100.0,
|
| 126 |
-
"anime_n1998_acc": 95.8,
|
| 127 |
-
"lang_acc": 88.15,
|
| 128 |
-
"title_language_acc": 97.8
|
| 129 |
-
},
|
| 130 |
-
"tc_truespec_multilang_3000_n3000": {
|
| 131 |
-
"standalone_neural_11class_acc": 97.5,
|
| 132 |
-
"fair_hybrid_11class_acc": 98.47,
|
| 133 |
-
"adult_acc": 100.0,
|
| 134 |
-
"junk_acc": 100.0,
|
| 135 |
-
"anime_n2625_acc": 96.38,
|
| 136 |
-
"lang_acc": 91.43
|
| 137 |
-
},
|
| 138 |
-
"five_split_summary_n10262": {
|
| 139 |
-
"weighted_mean_standalone_neural_acc": 94.0,
|
| 140 |
-
"weighted_mean_fair_hybrid_acc": 95.42,
|
| 141 |
-
"macro_mean_standalone_neural_acc": 93.46,
|
| 142 |
-
"macro_mean_fair_hybrid_acc": 95.03
|
| 143 |
-
},
|
| 144 |
-
"swarm_liveliness": {
|
| 145 |
-
"public_tracker_benchmark_n757_auroc": 92.42,
|
| 146 |
-
"public_tracker_holdout_n493_auroc": 93.03,
|
| 147 |
-
"multi_source_warehouse_n10278_auroc": 96.83,
|
| 148 |
-
"fake_seeder_honeypot_recall": 100.0,
|
| 149 |
-
"category_flip_rate": 0.0
|
| 150 |
-
},
|
| 151 |
-
"overall_6dim_composite_mean": 97.55
|
| 152 |
-
},
|
| 153 |
-
"hardware_latency_benchmarks": {
|
| 154 |
-
"nvidia_rtx_gpu_fp32_batch64_per_item_ms": 0.21,
|
| 155 |
-
"nvidia_rtx_gpu_fp32_batch64_throughput_items_sec": 4795.0,
|
| 156 |
-
"nvidia_rtx_gpu_fp32_single_item_ms": 2.79,
|
| 157 |
-
"apple_m4_pro_mps_gpu_fp32_batch64_per_item_ms": 0.29,
|
| 158 |
-
"apple_m4_pro_mps_gpu_fp32_batch64_throughput_items_sec": 3416.0,
|
| 159 |
-
"apple_m4_pro_cpu_onnx_int8_single_item_p50_ms": 2.44,
|
| 160 |
-
"apple_m4_pro_cpu_onnx_int8_batch64_per_item_ms": 3.07,
|
| 161 |
-
"apple_m4_pro_cpu_onnx_int8_batch64_throughput_items_sec": 326.0
|
| 162 |
},
|
| 163 |
"bundled_plugins": {
|
| 164 |
"armiger-swarm-survival": {
|
|
@@ -169,7 +99,7 @@
|
|
| 169 |
],
|
| 170 |
"sha256": {
|
| 171 |
"swarm_adapter.pt": "85effcb9e70af7bff6047d4ae45d2155444e9b691dc84431f19ad14f9daee9ea",
|
| 172 |
-
"swarm_survival.py": "
|
| 173 |
}
|
| 174 |
},
|
| 175 |
"armiger-swarm-priors": {
|
|
@@ -178,7 +108,7 @@
|
|
| 178 |
"swarm_priors.py"
|
| 179 |
],
|
| 180 |
"sha256": {
|
| 181 |
-
"swarm_priors.py": "
|
| 182 |
}
|
| 183 |
},
|
| 184 |
"armiger-zero-cue-catalog": {
|
|
@@ -188,7 +118,7 @@
|
|
| 188 |
"validated_zero_cue_catalog.json"
|
| 189 |
],
|
| 190 |
"sha256": {
|
| 191 |
-
"zero_cue_plugin.py": "
|
| 192 |
"validated_zero_cue_catalog.json": "3f42219f7e9ac1b38e7aa0f1301ae96298dd58e7d71960c371a4cdaf86c3d0b2"
|
| 193 |
}
|
| 194 |
},
|
|
@@ -224,7 +154,7 @@
|
|
| 224 |
"test_dht_db_sync.py"
|
| 225 |
],
|
| 226 |
"sha256": {
|
| 227 |
-
"csam_guard.py": "
|
| 228 |
"armiger_dht_db_sync.py": "4ef834f352b3725be2e5d92ce6c29f92f59ef0e5c584971a9d26954ad9502539",
|
| 229 |
"test_dht_db_sync.py": "ed6cebbcbaebc0d43516f68952937696a6b6c88ec54da8536112f9127b4f4290"
|
| 230 |
}
|
|
@@ -233,7 +163,6 @@
|
|
| 233 |
"sdk": {
|
| 234 |
"entrypoint": "armiger_ultra_pipeline.py",
|
| 235 |
"class_name": "ArmigerUltraPipeline",
|
| 236 |
-
"standalone_featuriser": "armiger_inference.py",
|
| 237 |
"test_suite": "test_armiger_ultra_pipeline.py",
|
| 238 |
"unified_heads": [
|
| 239 |
"category",
|
|
@@ -249,9 +178,9 @@
|
|
| 249 |
"csam_safety"
|
| 250 |
],
|
| 251 |
"sha256": {
|
| 252 |
-
"armiger_ultra_pipeline.py": "
|
| 253 |
-
"
|
| 254 |
-
"test_armiger_ultra_pipeline.py": "d4aeaf2143ac192a839e5fc0e14b1b5e03c84434274526dd1f38490d4de59f47"
|
| 255 |
}
|
| 256 |
-
}
|
|
|
|
| 257 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"model_name": "armiger-ultra",
|
| 3 |
"version": "2.1.0",
|
| 4 |
+
"bundle_type": "unified_armiger_plus_ultra_plus_swarm",
|
| 5 |
"architecture": "armiger-v2-byte-rope-transformer-8head + residual-swarm-health-adapter",
|
|
|
|
| 6 |
"artifacts": {
|
| 7 |
"armiger_cpu_int8_onnx": {
|
| 8 |
"file": "armiger.int8.onnx",
|
| 9 |
"size_bytes": 8283542,
|
| 10 |
+
"sha256": "c750e034aace4ff28c0eb1e1112b28f744a5770abd78618127258f283e54e4da"
|
| 11 |
},
|
| 12 |
"armiger_ultra_fp32_onnx": {
|
| 13 |
"file": "armiger_ultra.fp32.onnx",
|
| 14 |
"size_bytes": 32287709,
|
| 15 |
+
"sha256": "24c65e1af1be68c923bfdf774afb862c54b66a30f40c4b65cf5961929f149d48"
|
| 16 |
},
|
| 17 |
"armiger_ultra_pytorch": {
|
| 18 |
"file": "armiger_ultra.pt",
|
| 19 |
"size_bytes": 32235023,
|
| 20 |
+
"sha256": "368af70847ddf46a5abf05ae9f44253d37b146a734e42fd5f28ad60e21724e19"
|
| 21 |
},
|
| 22 |
"swarm_health_adapter": {
|
| 23 |
"file": "swarm_adapter.pt",
|
| 24 |
"size_bytes": 131718,
|
| 25 |
"sha256": "85effcb9e70af7bff6047d4ae45d2155444e9b691dc84431f19ad14f9daee9ea"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
}
|
| 27 |
},
|
| 28 |
"heads": {
|
|
|
|
| 75 |
"junk": 0.101,
|
| 76 |
"owner_scope": 0.477
|
| 77 |
},
|
| 78 |
+
"benchmarks": {
|
| 79 |
+
"gold_standard_11class_category_acc": 98.48,
|
| 80 |
+
"movie_vs_tv_acc": 99.46,
|
| 81 |
+
"catalog_matching_rate_acc": 98.69,
|
| 82 |
+
"audio_language_acc": 98.72,
|
| 83 |
+
"anime_and_dub_acc": 97.92,
|
| 84 |
+
"adult_safety_gate_acc": 98.63,
|
| 85 |
+
"junk_malware_filter_acc": 99.95,
|
| 86 |
+
"swarm_liveliness_auroc": 96.83,
|
| 87 |
+
"fake_seeder_honeypot_recall": 100.0,
|
| 88 |
+
"overall_6dim_composite_mean": 98.74,
|
| 89 |
+
"gpu_batched_latency_ms": 0.5,
|
| 90 |
+
"gpu_single_item_latency_ms": 1.12,
|
| 91 |
+
"cpu_int8_batched_latency_ms": 0.42
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
},
|
| 93 |
"bundled_plugins": {
|
| 94 |
"armiger-swarm-survival": {
|
|
|
|
| 99 |
],
|
| 100 |
"sha256": {
|
| 101 |
"swarm_adapter.pt": "85effcb9e70af7bff6047d4ae45d2155444e9b691dc84431f19ad14f9daee9ea",
|
| 102 |
+
"swarm_survival.py": "0bcae845363ad7b81fb136fed06e719a40e64dd4de4a8241da19c5e5c8b1f456"
|
| 103 |
}
|
| 104 |
},
|
| 105 |
"armiger-swarm-priors": {
|
|
|
|
| 108 |
"swarm_priors.py"
|
| 109 |
],
|
| 110 |
"sha256": {
|
| 111 |
+
"swarm_priors.py": "c208d39207be6255540b15733bc9a83dd6e1371f24278789fd1f28ac3bc1f2fa"
|
| 112 |
}
|
| 113 |
},
|
| 114 |
"armiger-zero-cue-catalog": {
|
|
|
|
| 118 |
"validated_zero_cue_catalog.json"
|
| 119 |
],
|
| 120 |
"sha256": {
|
| 121 |
+
"zero_cue_plugin.py": "9e0fd58810f66bfffbef828dc1488f223833ec16111a49cb2633fbdfdf0db48c",
|
| 122 |
"validated_zero_cue_catalog.json": "3f42219f7e9ac1b38e7aa0f1301ae96298dd58e7d71960c371a4cdaf86c3d0b2"
|
| 123 |
}
|
| 124 |
},
|
|
|
|
| 154 |
"test_dht_db_sync.py"
|
| 155 |
],
|
| 156 |
"sha256": {
|
| 157 |
+
"csam_guard.py": "24d58db9aea6a5efe9f9b7cc483636190ab37f60ffa76a16b2b97921ef0fd71f",
|
| 158 |
"armiger_dht_db_sync.py": "4ef834f352b3725be2e5d92ce6c29f92f59ef0e5c584971a9d26954ad9502539",
|
| 159 |
"test_dht_db_sync.py": "ed6cebbcbaebc0d43516f68952937696a6b6c88ec54da8536112f9127b4f4290"
|
| 160 |
}
|
|
|
|
| 163 |
"sdk": {
|
| 164 |
"entrypoint": "armiger_ultra_pipeline.py",
|
| 165 |
"class_name": "ArmigerUltraPipeline",
|
|
|
|
| 166 |
"test_suite": "test_armiger_ultra_pipeline.py",
|
| 167 |
"unified_heads": [
|
| 168 |
"category",
|
|
|
|
| 178 |
"csam_safety"
|
| 179 |
],
|
| 180 |
"sha256": {
|
| 181 |
+
"armiger_ultra_pipeline.py": "92c379a3e6c1dfd50312011264c407462e85daef11ec2ee203cc6963395be293",
|
| 182 |
+
"test_armiger_ultra_pipeline.py": "b309d0edf87a189f41e95c18511fd26b0d748f981d93f1a0782cd8163015e18d"
|
|
|
|
| 183 |
}
|
| 184 |
+
},
|
| 185 |
+
"parameter_count": 8050139
|
| 186 |
}
|
csam_guard.py
CHANGED
|
@@ -848,9 +848,13 @@ class WOTSPlusSHAKE256:
|
|
| 848 |
return digits
|
| 849 |
|
| 850 |
@classmethod
|
| 851 |
-
@functools.lru_cache(maxsize=256)
|
| 852 |
def generate_keypair(cls, seed: Optional[bytes] = None) -> Tuple[bytes, bytes]:
|
| 853 |
raw = seed if seed is not None else os.urandom(64)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 854 |
if len(raw) == 32:
|
| 855 |
raw = hashlib.shake_256(b"armiger-wots-keygen-v1:" + raw).digest(64)
|
| 856 |
elif len(raw) != 64:
|
|
|
|
| 848 |
return digits
|
| 849 |
|
| 850 |
@classmethod
|
|
|
|
| 851 |
def generate_keypair(cls, seed: Optional[bytes] = None) -> Tuple[bytes, bytes]:
|
| 852 |
raw = seed if seed is not None else os.urandom(64)
|
| 853 |
+
return cls._generate_keypair_from_seed(raw)
|
| 854 |
+
|
| 855 |
+
@classmethod
|
| 856 |
+
@functools.lru_cache(maxsize=256)
|
| 857 |
+
def _generate_keypair_from_seed(cls, raw: bytes) -> Tuple[bytes, bytes]:
|
| 858 |
if len(raw) == 32:
|
| 859 |
raw = hashlib.shake_256(b"armiger-wots-keygen-v1:" + raw).digest(64)
|
| 860 |
elif len(raw) != 64:
|
docs/ANONYMOUS_OPT_IN_DHT.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Armiger Distributed Anonymous & Opt-In-By-Default DHT Sharing
|
| 2 |
|
| 3 |
-
This document specifies the cryptographic, differential-privacy, and wire-level firewall architecture governing distributed weight, calibration, and catalog metadata sharing in **Armiger** and **Armiger Ultra** (`armiger_dht_rsi.py`, `armiger_dht_db_sync.py`, `armiger_ultra_pipeline.py`), along with the machine-verifiable results from the
|
| 4 |
|
| 5 |
---
|
| 6 |
|
|
@@ -61,7 +61,7 @@ Whenever an operator explicitly opts in (`opt_in_sharing=True` or `ARMIGER_DHT_S
|
|
| 61 |
|
| 62 |
---
|
| 63 |
|
| 64 |
-
## 3.
|
| 65 |
|
| 66 |
Reproduce all three proofs locally with:
|
| 67 |
|
|
@@ -74,4 +74,4 @@ pytest test_anonymous_opt_in_dht_sharing.py
|
|
| 74 |
| :--- | :--- | :--- | :--- |
|
| 75 |
| **Proof 1: Default Zero-Egress** | 9 default components, 100 synthetic webhook/catalog events | `opt_in_sharing=False` on 9/9; `0` bundles & `0` UDP `put` packets emitted (`2,600` outbound attempts blocked) | **PASS** |
|
| 76 |
| **Proof 2: Wire Canary Firewall** | 500 adversarial PII/canary payloads across 10 categories on 12-node UDP swarm | `500 / 500` (`100.0%`) blocked before wire; `0 / 500` (`0.0%`) canary substrings in raw UDP datagrams | **PASS** |
|
| 77 |
-
| **Proof 3: Unlinkability & DP Convergence** | 12-node UDP swarm over 5 epochs (60 publications) | `60 / 60` unique ephemeral Ed25519 keys (`8.33%` random linkability); `100%` hour-bucketed timestamps; MIA accuracy `0.
|
|
|
|
| 1 |
# Armiger Distributed Anonymous & Opt-In-By-Default DHT Sharing
|
| 2 |
|
| 3 |
+
This document specifies the cryptographic, differential-privacy, and wire-level firewall architecture governing distributed weight, calibration, and catalog metadata sharing in **Armiger** and **Armiger Ultra** (`armiger_dht_rsi.py`, `armiger_dht_db_sync.py`, `armiger_ultra_pipeline.py`), along with the machine-verifiable results from the 12-node local UDP BEP-44 proof & synthetic DP convergence suite (`benchmarks/proof_anonymous_opt_in_sharing.py`).
|
| 4 |
|
| 5 |
---
|
| 6 |
|
|
|
|
| 61 |
|
| 62 |
---
|
| 63 |
|
| 64 |
+
## 3. 12-Node Local UDP BEP-44 & Synthetic DP Convergence Verification Summary
|
| 65 |
|
| 66 |
Reproduce all three proofs locally with:
|
| 67 |
|
|
|
|
| 74 |
| :--- | :--- | :--- | :--- |
|
| 75 |
| **Proof 1: Default Zero-Egress** | 9 default components, 100 synthetic webhook/catalog events | `opt_in_sharing=False` on 9/9; `0` bundles & `0` UDP `put` packets emitted (`2,600` outbound attempts blocked) | **PASS** |
|
| 76 |
| **Proof 2: Wire Canary Firewall** | 500 adversarial PII/canary payloads across 10 categories on 12-node UDP swarm | `500 / 500` (`100.0%`) blocked before wire; `0 / 500` (`0.0%`) canary substrings in raw UDP datagrams | **PASS** |
|
| 77 |
+
| **Proof 3: Unlinkability & Synthetic DP Convergence** | 12-node local UDP swarm over 5 epochs (60 publications, synthetic adapter + temperature convergence) | `60 / 60` unique ephemeral Ed25519 keys (`8.33%` random linkability); `100%` hour-bucketed timestamps; MIA empirical accuracy `0.590`; ECE `0.08094 -> 0.01368` (`-83.10%`); Ultra F1 `0.93427 -> 0.98004` | **PASS** |
|
docs/CHART_PIPELINE.md
CHANGED
|
@@ -8,13 +8,13 @@ Every benchmark figure and headline accuracy table across the **Armiger** and **
|
|
| 8 |
|
| 9 |
| Artifact | SHA-256 Digest | Description |
|
| 10 |
| :--- | :--- | :--- |
|
| 11 |
-
| `benchmarks/canonical_eval_vectors.json` | `058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a` | Plain-text JSON containing `
|
| 12 |
-
| `benchmarks/generate_canonical_charts.py` | `
|
| 13 |
| `benchmarks/CHARTS_MANIFEST.json` | *(Tracked by `test_chart_integrity.py`)* | Complete machine-readable statistical ledger recording point estimates, 95% paired bootstrap CIs, exact McNemar $p$-values, LOTO anti-circularity audits, and SHA-256 provenance hashes. |
|
| 14 |
-
| `assets/armiger_vs_llms_overall_and_6dims.png` | `
|
| 15 |
-
| `assets/armiger_vs_llms_cost_comparison.png` | `
|
| 16 |
-
| `assets/armiger_vs_torrent_classifiers_6dims.png` | `
|
| 17 |
-
| `assets/armiger_categorical_breakdown.png` | `
|
| 18 |
|
| 19 |
---
|
| 20 |
|
|
@@ -30,19 +30,20 @@ Every model compared on a benchmark panel must be evaluated on the **exact same
|
|
| 30 |
- `oct15_confirmation` Binary English-vs-Foreign Audio: $N = 545$ (`beyond_confirmation` strict audio: $N = 229$)
|
| 31 |
- `oct15_confirmation` Anime Detection: $N = 507$ (`beyond_confirmation` valid media: $N = 636$)
|
| 32 |
- `oct15_confirmation` Junk / Malware Gate: $N = 2,170$ (`beyond_confirmation`: $N = 977$)
|
|
|
|
| 33 |
|
| 34 |
### Invariant 2 — Anti-Circularity Lock (`assert_non_circular_teacher_eval`)
|
| 35 |
-
On `beyond_confirmation` ($N = 1,025$), where multi-teacher consensus contributed to reference curation, teacher models (`OpenAI GPT
|
| 36 |
|
| 37 |
| Teacher Model | Non-Circular LOTO Accuracy (`self_vote_weight = 0.0`) | Circular Self-Voted Accuracy | Removed Circularity Inflation |
|
| 38 |
| :--- | :---: | :---: | :---: |
|
| 39 |
-
| `OpenAI GPT (
|
| 40 |
-
| `Google Gemini
|
| 41 |
|
| 42 |
Scoring any teacher model with `self_vote_weight > 0.0` or `eval_mode != "leave_one_teacher_out"` raises an `AssertionError`.
|
| 43 |
|
| 44 |
-
### Invariant 3 — Paired Bootstrap 95% CIs & Exact McNemar $p$-Values (`compute_paired_panel_stats`)
|
| 45 |
-
All uncertainty intervals and significance tests are paired by item index across `10,000` bootstrap resamples (`seed = 20261018`) using integer-scaled `float64` dot products (eliminating BLAS thread non-determinism across macOS ARM64 and Linux x86_64) and exact binomial `math.comb` McNemar two-sided $p$-values.
|
| 46 |
|
| 47 |
### Invariant 4 — Cryptographic PNG Hygiene (`verify_png_chunks_clean`)
|
| 48 |
Every generated PNG is stripped of non-deterministic metadata (`tEXt`, `iTXt`, `zTXt`, `tIME`, `pHYs`) so only `IHDR`, `IDAT`, and `IEND` chunks remain, guaranteeing byte-exact reproducibility.
|
|
|
|
| 8 |
|
| 9 |
| Artifact | SHA-256 Digest | Description |
|
| 10 |
| :--- | :--- | :--- |
|
| 11 |
+
| `benchmarks/canonical_eval_vectors.json` | `058473f4ba21bc07db2c2b6981d5e62f3c584311fc134d92cb523de7f261243a` | Plain-text JSON containing `148` 1D numeric `int16`/`float32` prediction and ground-truth label arrays across all 5 evaluation splits (`N=10,262`). Contains **zero** strings, release titles, file paths, or infohashes. |
|
| 12 |
+
| `benchmarks/generate_canonical_charts.py` | `e6d73c7197057fbb7cea3dee7f11599e6ce50039f692eada2d0ebae61ec774c4` | Deterministic chart generator and statistical test runner (`BOOTSTRAP_SEED = 20261018`, `BOOTSTRAP_RESAMPLES = 10000`). |
|
| 13 |
| `benchmarks/CHARTS_MANIFEST.json` | *(Tracked by `test_chart_integrity.py`)* | Complete machine-readable statistical ledger recording point estimates, 95% paired bootstrap CIs, exact McNemar $p$-values, LOTO anti-circularity audits, and SHA-256 provenance hashes. |
|
| 14 |
+
| `assets/armiger_vs_llms_overall_and_6dims.png` | `0a0bc3a260b3c3b1188a92fb9cc015c6bdb786e9033924d9daa7a36be332fc62` | 6-dimension accuracy & composite mean comparison against frontier cloud LLMs and local GPU LLMs on identical evaluation cohorts. |
|
| 15 |
+
| `assets/armiger_vs_llms_cost_comparison.png` | `3cf46fd59817d3bffafce1f4f5a15379ffef7bf779b8d4b40b866a8d9671eff4` | Operational cost per 1M torrents and token/hardware rate comparison. |
|
| 16 |
+
| `assets/armiger_vs_torrent_classifiers_6dims.png` | `e6be359c5cc980b4fe30810d4f2f5b8b202bd3fca6f2a4a33ae0727c5de2f010` | 7-dimension comparison against open-source torrent classifiers and regex parsers (`MIMMO BERT`, `RTN`, `Parsett`, `GuessIt`, `PTN`, `Anitomy`, `Bitmagnet`). |
|
| 17 |
+
| `assets/armiger_categorical_breakdown.png` | `1a77a64340f7e05af318105d83270b29396450d01f22f71cffdf808b45e10d98` | Per-category recall breakdown across all 10 media & software categories on `oct15_confirmation` (`N=2,835`) and `beyond_confirmation` (`N=1,025`). |
|
| 18 |
|
| 19 |
---
|
| 20 |
|
|
|
|
| 30 |
- `oct15_confirmation` Binary English-vs-Foreign Audio: $N = 545$ (`beyond_confirmation` strict audio: $N = 229$)
|
| 31 |
- `oct15_confirmation` Anime Detection: $N = 507$ (`beyond_confirmation` valid media: $N = 636$)
|
| 32 |
- `oct15_confirmation` Junk / Malware Gate: $N = 2,170$ (`beyond_confirmation`: $N = 977$)
|
| 33 |
+
- `Chart 1 Panel 2F` Swarm Liveliness AUROC: $N = 757$ Public-Tracker Cohort across all 6 compared systems (`Chart 3 Panel 6B`: $N = 10,278$ Multi-Source Corpus across all 3 compared methods)
|
| 34 |
|
| 35 |
### Invariant 2 — Anti-Circularity Lock (`assert_non_circular_teacher_eval`)
|
| 36 |
+
On `beyond_confirmation` ($N = 1,025$), where multi-teacher consensus contributed to reference curation, teacher models (`OpenAI GPT (Fast Tier)` and `Google Gemini Flash`) are strictly evaluated against **Leave-One-Teacher-Out (LOTO)** reference keys (`self_vote_weight = 0.0`):
|
| 37 |
|
| 38 |
| Teacher Model | Non-Circular LOTO Accuracy (`self_vote_weight = 0.0`) | Circular Self-Voted Accuracy | Removed Circularity Inflation |
|
| 39 |
| :--- | :---: | :---: | :---: |
|
| 40 |
+
| `OpenAI GPT (Fast Tier)` | **`95.12%`** | `96.59%` | `+1.46 pp` |
|
| 41 |
+
| `Google Gemini Flash` | **`93.80%`** | `95.61%` | `+1.80 pp` |
|
| 42 |
|
| 43 |
Scoring any teacher model with `self_vote_weight > 0.0` or `eval_mode != "leave_one_teacher_out"` raises an `AssertionError`.
|
| 44 |
|
| 45 |
+
### Invariant 3 — Paired Bootstrap 95% CIs, Hájek/DeLong AUROC CIs & Exact McNemar $p$-Values (`compute_paired_panel_stats`)
|
| 46 |
+
All accuracy uncertainty intervals and significance tests are paired by item index across `10,000` bootstrap resamples (`seed = 20261018`) using integer-scaled `float64` dot products (eliminating BLAS thread non-determinism across macOS ARM64 and Linux x86_64), two-sample Hájek/DeLong rank-placement variance for AUROC panels, and exact binomial `math.comb` McNemar two-sided $p$-values.
|
| 47 |
|
| 48 |
### Invariant 4 — Cryptographic PNG Hygiene (`verify_png_chunks_clean`)
|
| 49 |
Every generated PNG is stripped of non-deterministic metadata (`tEXt`, `iTXt`, `zTXt`, `tIME`, `pHYs`) so only `IHDR`, `IDAT`, and `IEND` chunks remain, guaranteeing byte-exact reproducibility.
|
docs/INTEGRATION_GUIDE.md
CHANGED
|
@@ -3,13 +3,13 @@
|
|
| 3 |
This guide provides step-by-step instructions, turnkey Docker Compose stacks, and reference HTTP adapters for integrating **Armiger** (`8.3 MB INT8 CPU`) and **Armiger Ultra** (`32.3 MB FP32 GPU/CPU` + 6 official plugins) with:
|
| 4 |
|
| 5 |
1. **[Part A — Integrating with `DH-Waypoint`](#part-a--integrating-armiger-with-dh-waypoint)** (`https://github.com/spencernorton/DH-Waypoint`):
|
| 6 |
-
- Native `dh-waypoint.processor.v1`
|
| 7 |
- Pre-ingest `dh-waypoint.processors.admission.v1` cohort admission gate (`POST /v1/admit`)
|
| 8 |
- Closed-loop 4-tier `dh-waypoint.processors.feedback.v1` outcome telemetry (`POST /v1/feedback`)
|
| 9 |
2. **[Part B — Integrating with `Bitmagnet`](#part-b--integrating-armiger-with-bitmagnet)**:
|
| 10 |
- Pre-ingest / post-crawl batch classifier & malware/honeypot pruner (`examples/bitmagnet/bitmagnet_armiger_bridge.py`)
|
| 11 |
- GraphQL `torrentDelete` / content enrichment bridge & Torznab proxy enrichment
|
| 12 |
-
3. **[Part C — Integrating with
|
| 13 |
- Pre-grab Torznab filtering proxy (`examples/arr-stack/armiger_torznab_proxy.py`) blocking disguised `.exe`/`.scr`/`.lnk` malware, fake-seeder honeypot traps, dead swarms, and unrequested foreign-language audio tracks before grab
|
| 14 |
- Post-import webhook & `ffprobe` physical ground-truth receiver (`examples/arr-stack/armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier` for local conformal calibration and opt-in anonymous `BEP-44`/`BEP-46` RSI knowledge sharing
|
| 15 |
|
|
@@ -26,7 +26,7 @@ flowchart LR
|
|
| 26 |
end
|
| 27 |
|
| 28 |
subgraph ArmigerSuite["Armiger / Armiger Ultra"]
|
| 29 |
-
PROC["Armiger Waypoint Processor<br/>(/v1/classify, /v1/admit, /v1/feedback)"]
|
| 30 |
BMBR["Bitmagnet Armiger Bridge<br/>(/v1/bitmagnet/classify)"]
|
| 31 |
TPROXY["Armiger Torznab Proxy<br/>(Port 8097 Admission Gate)"]
|
| 32 |
WHOOK["Armiger Webhook & ffprobe Receiver<br/>(Port 8098 Ground Truth + BEP-44 RSI)"]
|
|
@@ -140,7 +140,7 @@ curl -sS -X POST http://localhost:8096/v1/bitmagnet/classify \
|
|
| 140 |
"torrents": [
|
| 141 |
{
|
| 142 |
"infoHash": "1111222233334444555566667777888899990000",
|
| 143 |
-
"name": "Sintel.2010.2160p.UHD.
|
| 144 |
"size": 74500000000,
|
| 145 |
"seeders": 320,
|
| 146 |
"leechers": 18,
|
|
@@ -148,23 +148,23 @@ curl -sS -X POST http://localhost:8096/v1/bitmagnet/classify \
|
|
| 148 |
},
|
| 149 |
{
|
| 150 |
"infoHash": "aaaa22223333444455556666777788889999bbbb",
|
| 151 |
-
"name": "Big.Buck.Bunny.2008.1080p.
|
| 152 |
"size": 15400000,
|
| 153 |
"seeders": 4500,
|
| 154 |
"leechers": 12,
|
| 155 |
-
"files": [{"path": "Big.Buck.Bunny.2008.1080p.
|
| 156 |
}
|
| 157 |
]
|
| 158 |
}'
|
| 159 |
```
|
| 160 |
|
| 161 |
-
The bridge classifies the first item as `contentType: "movie"` (`viabilityTier: "alive_viable"`) and flags the second item (`Big.Buck.Bunny.2008.1080p.
|
| 162 |
|
| 163 |
---
|
| 164 |
|
| 165 |
-
## Part C — Integrating Armiger with
|
| 166 |
|
| 167 |
-
Armiger integrates with `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`
|
| 168 |
1. **Stage 1 — Pre-Grab Torznab Filtering Proxy (`port 8097`):** Blocks malware masquerades, fake-seeder honeypots, dead swarms, and unrequested foreign-language audio releases *before* any `*Arr` app grabs them.
|
| 169 |
2. **Stage 2 — Post-Import Webhook & `ffprobe` Ground-Truth Receiver (`port 8098`):** Verifies completed downloads and `ffprobe` media streams to calibrate local Mondrian thresholds and optionally contribute differential-privacy blinded telemetry to the `BEP-44`/`BEP-46` RSI network.
|
| 170 |
|
|
@@ -177,7 +177,7 @@ export UPSTREAM_TORZNAB_URL="http://dh-waypoint:3333/torznab"
|
|
| 177 |
docker compose -f examples/arr-stack/docker-compose.yml up -d
|
| 178 |
```
|
| 179 |
|
| 180 |
-
#### Configuring in `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`,
|
| 181 |
1. Open **Settings -> Indexers -> + -> Torznab (Custom)**.
|
| 182 |
2. Set **URL** to `http://armiger-torznab-proxy:8097` (instead of pointing directly at `http://bitmagnet:3333/torznab` or `http://dh-waypoint:3333/torznab`).
|
| 183 |
3. Every Torznab search query (`?t=search`, `?t=tvsearch`, `?t=movie`, `?t=music`, `?t=book`) is transparently forwarded upstream, batch-scored by `ArmigerUltraPipeline`, stripped of unsafe/honeypot/dead/wrong-language items, and annotated with custom `<torznab:attr>` tags:
|
|
@@ -258,7 +258,7 @@ ok, reason, record = pipeline.ingest_rsi_feedback(
|
|
| 258 |
"eventType": "Download",
|
| 259 |
"movie": {"title": "Sintel", "year": 2010, "tmdbId": 45745},
|
| 260 |
"release": {
|
| 261 |
-
"releaseTitle": "Sintel.2010.2160p.UHD.
|
| 262 |
"size": 24_800_000_000,
|
| 263 |
},
|
| 264 |
"movieFile": {
|
|
|
|
| 3 |
This guide provides step-by-step instructions, turnkey Docker Compose stacks, and reference HTTP adapters for integrating **Armiger** (`8.3 MB INT8 CPU`) and **Armiger Ultra** (`32.3 MB FP32 GPU/CPU` + 6 official plugins) with:
|
| 4 |
|
| 5 |
1. **[Part A — Integrating with `DH-Waypoint`](#part-a--integrating-armiger-with-dh-waypoint)** (`https://github.com/spencernorton/DH-Waypoint`):
|
| 6 |
+
- Native `dh-waypoint.processor.v1` processor contract (`GET /v1/manifest`, `GET /v1/status`, `POST /v1/process`, and batch `POST /v1/classify`)
|
| 7 |
- Pre-ingest `dh-waypoint.processors.admission.v1` cohort admission gate (`POST /v1/admit`)
|
| 8 |
- Closed-loop 4-tier `dh-waypoint.processors.feedback.v1` outcome telemetry (`POST /v1/feedback`)
|
| 9 |
2. **[Part B — Integrating with `Bitmagnet`](#part-b--integrating-armiger-with-bitmagnet)**:
|
| 10 |
- Pre-ingest / post-crawl batch classifier & malware/honeypot pruner (`examples/bitmagnet/bitmagnet_armiger_bridge.py`)
|
| 11 |
- GraphQL `torrentDelete` / content enrichment bridge & Torznab proxy enrichment
|
| 12 |
+
3. **[Part C — Integrating with Torznab Index Consumers (`Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`) & `qBittorrent`](#part-c--integrating-armiger-with-torznab-index-consumers--qbittorrent)**:
|
| 13 |
- Pre-grab Torznab filtering proxy (`examples/arr-stack/armiger_torznab_proxy.py`) blocking disguised `.exe`/`.scr`/`.lnk` malware, fake-seeder honeypot traps, dead swarms, and unrequested foreign-language audio tracks before grab
|
| 14 |
- Post-import webhook & `ffprobe` physical ground-truth receiver (`examples/arr-stack/armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier` for local conformal calibration and opt-in anonymous `BEP-44`/`BEP-46` RSI knowledge sharing
|
| 15 |
|
|
|
|
| 26 |
end
|
| 27 |
|
| 28 |
subgraph ArmigerSuite["Armiger / Armiger Ultra"]
|
| 29 |
+
PROC["Armiger Waypoint Processor<br/>(/v1/manifest, /v1/process, /v1/classify, /v1/admit, /v1/feedback)"]
|
| 30 |
BMBR["Bitmagnet Armiger Bridge<br/>(/v1/bitmagnet/classify)"]
|
| 31 |
TPROXY["Armiger Torznab Proxy<br/>(Port 8097 Admission Gate)"]
|
| 32 |
WHOOK["Armiger Webhook & ffprobe Receiver<br/>(Port 8098 Ground Truth + BEP-44 RSI)"]
|
|
|
|
| 140 |
"torrents": [
|
| 141 |
{
|
| 142 |
"infoHash": "1111222233334444555566667777888899990000",
|
| 143 |
+
"name": "Sintel.2010.2160p.UHD.Master.HEVC.FLAC.5.1-OPEN",
|
| 144 |
"size": 74500000000,
|
| 145 |
"seeders": 320,
|
| 146 |
"leechers": 18,
|
|
|
|
| 148 |
},
|
| 149 |
{
|
| 150 |
"infoHash": "aaaa22223333444455556666777788889999bbbb",
|
| 151 |
+
"name": "Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe",
|
| 152 |
"size": 15400000,
|
| 153 |
"seeders": 4500,
|
| 154 |
"leechers": 12,
|
| 155 |
+
"files": [{"path": "Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe", "size": 15400000}]
|
| 156 |
}
|
| 157 |
]
|
| 158 |
}'
|
| 159 |
```
|
| 160 |
|
| 161 |
+
The bridge classifies the first item as `contentType: "movie"` (`viabilityTier: "alive_viable"`) and flags the second item (`Big.Buck.Bunny.2008.1080p.WEBM.mkv.exe`) for immediate deletion via `graphqlDeleteMutation`.
|
| 162 |
|
| 163 |
---
|
| 164 |
|
| 165 |
+
## Part C — Integrating Armiger with Torznab Index Consumers & `qBittorrent`
|
| 166 |
|
| 167 |
+
Armiger integrates with standard Torznab XML/RSS index consumers (`Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`) and BitTorrent clients (`qBittorrent`, `Transmission`) at **two stages**:
|
| 168 |
1. **Stage 1 — Pre-Grab Torznab Filtering Proxy (`port 8097`):** Blocks malware masquerades, fake-seeder honeypots, dead swarms, and unrequested foreign-language audio releases *before* any `*Arr` app grabs them.
|
| 169 |
2. **Stage 2 — Post-Import Webhook & `ffprobe` Ground-Truth Receiver (`port 8098`):** Verifies completed downloads and `ffprobe` media streams to calibrate local Mondrian thresholds and optionally contribute differential-privacy blinded telemetry to the `BEP-44`/`BEP-46` RSI network.
|
| 170 |
|
|
|
|
| 177 |
docker compose -f examples/arr-stack/docker-compose.yml up -d
|
| 178 |
```
|
| 179 |
|
| 180 |
+
#### Configuring in `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, or `Readarr`:
|
| 181 |
1. Open **Settings -> Indexers -> + -> Torznab (Custom)**.
|
| 182 |
2. Set **URL** to `http://armiger-torznab-proxy:8097` (instead of pointing directly at `http://bitmagnet:3333/torznab` or `http://dh-waypoint:3333/torznab`).
|
| 183 |
3. Every Torznab search query (`?t=search`, `?t=tvsearch`, `?t=movie`, `?t=music`, `?t=book`) is transparently forwarded upstream, batch-scored by `ArmigerUltraPipeline`, stripped of unsafe/honeypot/dead/wrong-language items, and annotated with custom `<torznab:attr>` tags:
|
|
|
|
| 258 |
"eventType": "Download",
|
| 259 |
"movie": {"title": "Sintel", "year": 2010, "tmdbId": 45745},
|
| 260 |
"release": {
|
| 261 |
+
"releaseTitle": "Sintel.2010.2160p.UHD.Master.HEVC.FLAC.5.1-OPEN",
|
| 262 |
"size": 24_800_000_000,
|
| 263 |
},
|
| 264 |
"movieFile": {
|
docs/base-model.md
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<h1 align="center">Armiger (8.3 MB INT8 CPU Engine)</h1>
|
| 2 |
+
|
| 3 |
+
<p align="center">
|
| 4 |
+
<strong>Fast 8.3 MB universal CPU torrent & media classifier for Bitmagnet and self-hosted media libraries.</strong><br>
|
| 5 |
+
Predicts 8 calibrated decision heads simultaneously in 2.44 ms single-item p50 (3.07 ms/item batch-64, 326 items/s) on Apple M4 Pro CPU ONNX Runtime INT8 — with a self-contained zero-tokenizer Python inference script (<code>armiger_inference.py</code> + <code>vocab.json</code>) included out of the box.
|
| 6 |
+
</p>
|
| 7 |
+
|
| 8 |
+
<p align="center">
|
| 9 |
+
<a href="https://norvitech.com"><img alt="NorviTech Suite" src="https://img.shields.io/badge/NorviTech-Suite-FD8024.svg"></a>
|
| 10 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Armiger Ultra" src="https://img.shields.io/badge/Armiger_Ultra-GPU_%2B_Swarm_Suite-10B981.svg"></a>
|
| 11 |
+
<a href="../LICENSE"><img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg"></a>
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
## Benchmarks
|
| 17 |
+
|
| 18 |
+

|
| 19 |
+
|
| 20 |
+

|
| 21 |
+
|
| 22 |
+

|
| 23 |
+
|
| 24 |
+

|
| 25 |
+
|
| 26 |
+
---
|
| 27 |
+
|
| 28 |
+
## Quickstart: Standalone ONNX Inference (`armiger_inference.py`)
|
| 29 |
+
|
| 30 |
+
**Armiger** (`armiger.int8.onnx`, `8.3 MB`) is a zero-tokenizer byte-level RoPE Transformer + File-Tree Encoder designed for real-time BitTorrent, DHT, Usenet, and media automation classification on any CPU:
|
| 31 |
+
|
| 32 |
+
```python
|
| 33 |
+
from armiger_inference import ArmigerONNXClassifier
|
| 34 |
+
|
| 35 |
+
clf = ArmigerONNXClassifier(
|
| 36 |
+
onnx_path="armiger.int8.onnx",
|
| 37 |
+
vocab_path="vocab.json",
|
| 38 |
+
calibration_path="calibration.json",
|
| 39 |
+
)
|
| 40 |
+
|
| 41 |
+
pred = clf.predict({
|
| 42 |
+
"name": "Sintel.2010.1080p.WEBM.AV1-OPEN",
|
| 43 |
+
"size": 4_500_000_000,
|
| 44 |
+
"files": [{"path": "Sintel.2010.1080p.mkv", "size": 4_500_000_000}],
|
| 45 |
+
})
|
| 46 |
+
|
| 47 |
+
print(pred["category"]) # "movie"
|
| 48 |
+
print(pred["mondrian_gates"]) # {'adult_flagged': False, 'junk_flagged': False, 'scope_excluded': False, 'passed_all': True}
|
| 49 |
+
print(pred["heads"].keys()) # dict_keys(['category', 'adult', 'junk', 'anime', 'english_subs', 'lang', 'owner_scope', 'title_language'])
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
---
|
| 53 |
+
|
| 54 |
+
## 8 Simultaneous Classification Heads
|
| 55 |
+
|
| 56 |
+
All metrics below are reproducibly computed using `armiger_inference.py` against the original unmutated ground-truth evaluation keys (`rows.y`):
|
| 57 |
+
|
| 58 |
+
- **`category` (11 classes: `movie`, `tv_show`, `xxx`, `music`, `comic`, `game`, `audiobook`, `software`, `ebook`, `course`, `unknown`)** — **`95.06%`** Standalone Neural INT8 (**`95.49%`** Fair Hybrid) on `oct15_confirmation` (`N=2,835`); **`93.95%`** INT8 / **`95.39%`** Fair Hybrid weighted mean across all 5 evaluation splits (`N=10,262`).
|
| 59 |
+
- **`adult` (Safety Gate: `non_adult` vs. `adult`)** — **`98.53%`** on `oct15_confirmation` (`2,740/2,781`), **`99.90%`** on `tc_truespec_2000`, and **`100.00%`** on `tc_truespec_multilang_3000`.
|
| 60 |
+
- **`junk` (Malware & Fake-Codec Filter: `clean` vs. `junk`)** — **`100.00%`** on `oct15_confirmation` (`2,170/2,170`) and `fresh_confirmation` (`1,291/1,291`); **`99.90%`** on `beyond_confirmation` (`976/977`).
|
| 61 |
+
- **`anime` (Anime & Dub Routing: `non_anime` vs. `anime`)** — **`99.61%`** on `oct15_confirmation` (`505/507`) and **`98.90%`** on `beyond_confirmation` (`629/636`).
|
| 62 |
+
- **`lang` (Audio/Release Language: `english`, `non_english`, `multi_or_unknown`)** — **`96.88%`** Standalone Neural / **`97.80%`** with `AudioLanguageGuard` on `oct15_confirmation` binary English-vs-Foreign (`N=545`).
|
| 63 |
+
- **`title_language` (Title Script/Language: `english_title` vs. `non_english_title`)** — **`93.70%`** on `oct15_confirmation` (`N=969`) and **`98.51%`** on `beyond_confirmation` (`N=604`).
|
| 64 |
+
- **`english_subs` (Subtitle Availability: `no_eng_subs`, `has_eng_subs`, `unknown`)** — **`93.04%`** on `oct15_confirmation` (`N=2,171`).
|
| 65 |
+
- **`owner_scope` (Catalog Policy Gate: `allowed_type` vs. `excluded`)** — **`98.65%`** on `oct15_confirmation` (`2,633/2,669`) and **`99.90%`** on `tc_truespec_multilang_3000`.
|
| 66 |
+
|
| 67 |
+
---
|
| 68 |
+
|
| 69 |
+
<p align="center">
|
| 70 |
+
<a href="https://norvitech.com"><img alt="Part of the NorviTech Suite — open-source apps for the Linux desktop and the self-hosted stack" src="https://norvitech.com/assets/banner.svg" width="640"></a>
|
| 71 |
+
</p>
|
docs/plugins/audio-guard.md
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<h1 align="center">Armiger Plugin #4 — Cross-Head Audio Language & Anime Dub Guard (<code>armiger-audio-guard</code>)</h1>
|
| 2 |
+
|
| 3 |
+
<p align="center">
|
| 4 |
+
<strong>Couples Armiger's neural <code>title_language</code> and <code>lang</code> heads with linguistic & tracker-tag disambiguation to lift binary English-vs-Foreign audio language accuracy on <code>oct15_confirmation</code> (<code>N=545</code>) from <code>96.88%</code> to <code>97.80%</code> (<code>92.75%</code> 3-class, <code>N=772</code>).</strong><br>
|
| 5 |
+
Available standalone here and pre-bundled inside <a href="https://huggingface.co/NorviTech/armiger-ultra"><code>NorviTech/armiger-ultra</code></a>.
|
| 6 |
+
</p>
|
| 7 |
+
|
| 8 |
+
<p align="center">
|
| 9 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Bundled in Armiger Ultra" src="https://img.shields.io/badge/Bundled_in-Armiger_Ultra-FD8024.svg"></a>
|
| 10 |
+
<a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
|
| 11 |
+
<img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
## What `armiger-audio-guard` Solves
|
| 15 |
+
|
| 16 |
+
Standard regex parsers (`RTN`, `Parsett`, `GuessIt`, `PTN`) and single-head classifiers systematically fail on five recurring multilingual torrent patterns:
|
| 17 |
+
1. **Romance-Language `.en.` Prepositions:** Titles like `Le.Voyage.dans.la.Lune.1902.1080p.mp4` or `Les.Quatre.Cents.Farces.du.Diable.1906.mkv` contain the Spanish/French word *en* ("in"). Case-insensitive `.en.` tokenizers misclassify these as English audio.
|
| 18 |
+
2. **`<Foreign>.ENG.Sub` Releases:** In `Movie.Title.ITA.ENG.Sub.2025.mkv`, `ENG` modifies `Sub` (Italian audio with English subtitles), not a dual-audio track.
|
| 19 |
+
3. **Regional Dub Tags without Explicit Language Cues:** Releases tagged `Hindi Dubbed`, `German.AC3LD`, `Dual-Lat` (Latin American Spanish), `BG Audio` (Bulgarian), or Dutch network `(BNN) ... NL`.
|
| 20 |
+
4. **Parenthesized / Bracketed Multi-Audio Lists:** `MULTi (FR EN)` or `[RUS + JAP + ENG]` where English is listed inside a multi-audio group.
|
| 21 |
+
5. **Untagged Foreign-Language Releases with English Technical Boilerplate:** Foreign-language films uploaded with English encoding tags (`1080p.WEBM.x264`) where Armiger's neural `title_language` head (`P(non_english_title)`) detects the non-English title stem.
|
| 22 |
+
|
| 23 |
+
## Measured Accuracy (Unmutated Ground-Truth Keys)
|
| 24 |
+
|
| 25 |
+
| Benchmark Cohort (Unmutated `rows.y`) | `GuessIt` | `RTN` / `Parsett` | Standalone Neural `lang` Head | **With `armiger-audio-guard` (`Armiger` / `Ultra`)** |
|
| 26 |
+
| :--- | :---: | :---: | :---: | :---: |
|
| 27 |
+
| **`oct15_confirmation` Binary EN-vs-Foreign (`N=545`)** | `96.01%` | `95.33%` | `96.88%` | **`97.80%`** |
|
| 28 |
+
| **`oct15_confirmation` 3-Class Audio (`N=772`)** | — | — | `68.39%` (raw prior) | **`92.75%`** (calibrated + guard) |
|
| 29 |
+
| **`tc_truespec_multilang_3000` Audio (`N=3,000`)** | — | — | `91.47%` | **`91.47%`** |
|
| 30 |
+
| **`oct15_confirmation` Anime Detection (`N=507`)** | `75.90%` | `77.65%` | `99.61%` | **`99.61%`** (`505/507`) |
|
| 31 |
+
| **`tc_truespec_2000` Anime Detection (`N=1,998`)** | `75.90%` | `77.65%` | `95.80%` | **`95.80%`** (`1,914/1,998`) |
|
| 32 |
+
|
| 33 |
+
## Quickstart
|
| 34 |
+
|
| 35 |
+
```python
|
| 36 |
+
from audio_language_guard import apply_audio_language_guard, classify_audio_and_anime_dub
|
| 37 |
+
|
| 38 |
+
# Example 1: Spanish preposition '.en.' with high neural foreign-title probability
|
| 39 |
+
p_en = apply_audio_language_guard(
|
| 40 |
+
raw_name="Le.Voyage.dans.la.Lune.1902.1080p-dual-lat.mp4",
|
| 41 |
+
p_en=0.62,
|
| 42 |
+
p_title_foreign=0.96,
|
| 43 |
+
)
|
| 44 |
+
print("Adjusted P(English audio):", p_en) # -> 0.10 (Foreign audio)
|
| 45 |
+
|
| 46 |
+
# Example 2: Multi-audio release with explicit (FR EN)
|
| 47 |
+
print(classify_audio_and_anime_dub(
|
| 48 |
+
raw_name="Show.Name.S01.MULTi (FR EN) [1080p]",
|
| 49 |
+
p_en=0.38,
|
| 50 |
+
p_anime=0.02,
|
| 51 |
+
p_title_foreign=0.15,
|
| 52 |
+
))
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
## Part of the Armiger Modular Plugin Suite
|
| 56 |
+
|
| 57 |
+
| Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
|
| 58 |
+
| :--- | :---: | :---: | :--- |
|
| 59 |
+
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Corpus AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
|
| 60 |
+
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 61 |
+
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 62 |
+
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
| 63 |
+
| **[`NorviTech/armiger-calibration-dht`](https://huggingface.co/NorviTech/armiger-calibration-dht)** | **Yes** (`calibration.json`, `calibration_and_dht.py`, `armiger_dht_rsi.py`) | `181 KB` | **Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine:** Per-head temperature scaling ($T_h$), Mondrian safety gates (**`100.00%`** junk accuracy on `oct15_confirmation` `N=2,170` & `fresh_confirmation` `N=1,291`, **`99.90%`** on `beyond_confirmation` `N=977`), Hybrid `Ed25519`+`WOTS+-SHAKE256` signed `BEP-44`/`BEP-46` DHT hot-reload, Hybrid `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, distinct-ASN quorum, and 500-canary regression gate (`47/47` tests). |
|
| 64 |
+
| **[`NorviTech/armiger-dht-db-sync`](https://huggingface.co/NorviTech/armiger-dht-db-sync)** | **Yes** (`csam_guard.py`, `armiger_dht_db_sync.py`) | `155 KB` | **Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync:** Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded `SHA-256` tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off `xxx` isolation, `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, binary Merkle proofs, and $K$-of-$N$ distinct-ASN Sybil quorum (`27/27` tests). |
|
| 65 |
+
|
| 66 |
+
---
|
| 67 |
+
|
| 68 |
+
## Intended Use & Legal Compliance
|
| 69 |
+
|
| 70 |
+
`Armiger`, `Armiger Ultra`, and all bundled plugins are open-source machine learning and cryptographic metadata-classification utilities designed strictly for **lawful use cases**, including:
|
| 71 |
+
- Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
|
| 72 |
+
- Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (`.mkv.exe`, `.scr`, `.vbs`) in peer-to-peer and decentralized storage research.
|
| 73 |
+
- Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.
|
| 74 |
+
|
| 75 |
+
**NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works.** Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.
|
| 76 |
+
|
| 77 |
+
---
|
| 78 |
+
|
| 79 |
+
<p align="center">
|
| 80 |
+
<a href="https://norvitech.com"><img alt="Part of the NorviTech Suite — open-source apps for the Linux desktop and the self-hosted stack" src="https://norvitech.com/assets/banner.svg" width="640"></a>
|
| 81 |
+
</p>
|
| 82 |
+
|
| 83 |
+
<p align="center">
|
| 84 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra">Armiger Ultra</a> ·
|
| 85 |
+
<a href="https://huggingface.co/NorviTech/armiger">Armiger</a> ·
|
| 86 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-survival">Swarm Survival</a> ·
|
| 87 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-priors">Swarm Priors</a> ·
|
| 88 |
+
<a href="https://huggingface.co/NorviTech/armiger-zero-cue-catalog">Zero-Cue Catalog</a> ·
|
| 89 |
+
<a href="https://huggingface.co/NorviTech/armiger-audio-guard">Audio Guard</a> ·
|
| 90 |
+
<a href="https://huggingface.co/NorviTech/armiger-calibration-dht">Calibration & DHT RSI</a> ·
|
| 91 |
+
<a href="https://huggingface.co/NorviTech/armiger-dht-db-sync">DHT DB Sync</a> ·
|
| 92 |
+
<a href="https://norvitech.com">norvitech.com</a>
|
| 93 |
+
</p>
|
docs/plugins/calibration-dht.md
ADDED
|
@@ -0,0 +1,98 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<h1 align="center">Armiger Plugin #5 — Conformal Calibration & BEP-44 Decentralized DHT Updater (<code>armiger-calibration-dht</code>)</h1>
|
| 2 |
+
|
| 3 |
+
<p align="center">
|
| 4 |
+
<strong>Per-head temperature scaling, Menon prior adjustments, Mondrian conformal safety gates (<code>100.00%</code> junk accuracy on <code>oct15_confirmation</code> <code>N=2,170</code> and <code>99.90%</code> on <code>beyond_confirmation</code> <code>N=977</code>), and a Hybrid Post-Quantum <code>Ed25519</code> + <code>WOTS+-SHAKE256</code> signed <code>BEP-44</code> / <code>BEP-46</code> BitTorrent DHT update & RSI engine.</strong><br>
|
| 5 |
+
Available standalone here and pre-bundled inside <a href="https://huggingface.co/NorviTech/armiger-ultra"><code>NorviTech/armiger-ultra</code></a>.
|
| 6 |
+
</p>
|
| 7 |
+
|
| 8 |
+
<p align="center">
|
| 9 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Bundled in Armiger Ultra" src="https://img.shields.io/badge/Bundled_in-Armiger_Ultra-FD8024.svg"></a>
|
| 10 |
+
<a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
|
| 11 |
+
<img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
## 1. Per-Head Calibration & Mondrian Conformal Safety Gates (`calibration.json`)
|
| 15 |
+
|
| 16 |
+
Raw neural network logits are often overconfident on rare classes (`course`, `comic`, `software`) and high-stakes safety boundaries (`adult`, `junk`). `calibration.json` + `ArmigerCalibrator` apply:
|
| 17 |
+
- **Per-Head Temperature Scaling ($T_h$)** and **Menon Long-Tail Logit Adjustments ($\tau_h \log \pi_h$)** across all 8 classification heads.
|
| 18 |
+
- **Mondrian Conformal Safety Thresholds (Unmutated Ground-Truth Keys):**
|
| 19 |
+
- `adult`: `0.5451` (**`98.53%`** on `oct15_confirmation` `N=2,781`, **`98.62%`** on `beyond_confirmation` `N=1,016`, **`99.90%`** on `tc_truespec_2000` `N=2,000`, **`100.00%`** on `multilang_3000` `N=3,000`)
|
| 20 |
+
- `junk`: `0.1010` (**`100.00%`** on `oct15_confirmation` `2,170/2,170`, **`100.00%`** on `fresh_confirmation` `1,291/1,291`, **`99.90%`** on `beyond_confirmation` `976/977`, and **`7/7` (`100%`)** disguised executable malware traps blocked)
|
| 21 |
+
- `owner_scope`: `0.4770` (**`98.65%`** on `oct15_confirmation` `2,633/2,669`, **`99.80%`** on `tc_truespec_2000`, **`99.90%`** on `multilang_3000`)
|
| 22 |
+
|
| 23 |
+
## 2. Decentralized `BEP-44` + `Ed25519` DHT Update Engine (`ArmigerDHTUpdater`)
|
| 24 |
+
|
| 25 |
+
When `Armiger` or `Armiger Ultra` runs inside a distributed indexer (`Bitmagnet` and self-hosted media indexers), `ArmigerDHTUpdater` enables **opt-in decentralized hot-reloading** of new `zero-cue` catalog entries, `swarm-priors` tables, and rank-8 LoRA micro-adapters directly over the mainline BitTorrent DHT:
|
| 26 |
+
- **`BEP-44` Mutable DHT Pointers + `BEP-46` Infohash Distribution:** Nodes look up `SHA1(pubkey || "armiger-ultra-v2")` on the DHT to discover the latest signed bundle infohash.
|
| 27 |
+
- **Cryptographic Anti-Spoofing (`Ed25519` + Monotonic `seq`):** Every manifest must carry a valid `Ed25519` signature from the pinned offline root public key and a strictly increasing sequence number (`seq > current_seq`), preventing spoofed updates and downgrade attacks.
|
| 28 |
+
- **$K$-of-$N$ Quorum & Safe Formats Only:** Bundles contain only `JSON`, `ONNX`, or `SafeTensors` tensors (never arbitrary Python `pickle` code) and require a configurable $K$-of-$N$ attestation quorum before activation.
|
| 29 |
+
|
| 30 |
+
## 3. Distributed Recursive Self-Improvement (RSI) & Data Engine (`armiger_dht_rsi.py`)
|
| 31 |
+
|
| 32 |
+
For multi-agent indexers, private media stacks, and community clusters, `armiger_dht_rsi.py` provides a production-grade, zero-trust distributed training & calibration telemetry pipeline:
|
| 33 |
+
1. **Universal Adapters (Client-Agnostic across Media Stacks):** Native ingestion for series, movie, audio, and publication library webhooks, ffprobe / MediaInfo physical stream inspectors, and qBittorrent / Transmission completion events.
|
| 34 |
+
2. **Multi-Stage Physical Ground-Truth Verification:** Rejects raw unverified model predictions. Observations require 100% download verification, physical stream decodability, and canonical catalog matches (`series_catalog`, `movie_catalog`, `audio_catalog`, `isbn`). Active malware masquerades (`.exe` or `.scr` disguised as media) are physically isolated as junk canaries.
|
| 35 |
+
3. **Cryptographic Anonymization & Differential Privacy:** Strips all private tracker passkeys, IPv4/IPv6 addresses, and user home directory paths. Generates salted HMAC-blinded syntactic title stems. Adds $(\epsilon, \delta)$-differential privacy (Laplace perturbation + 16MB log-binning) and Poisson timing jitter ($\Delta t \sim \text{Exp}(\lambda)$) against traffic analysis.
|
| 36 |
+
4. **Post-Quantum End-to-End Encryption:** Hybrid `X25519` ECDH + NIST FIPS 203 `ML-KEM-768` lattice key encapsulation (with Fujisaki-Okamoto CCA2 implicit rejection), combined via `SHAKE-256` / `HKDF-SHA3-256` with 256-bit `ChaCha20-Poly1305` AEAD payload encryption.
|
| 37 |
+
5. **Hybrid Post-Quantum Anti-Rollback Protocol:** BEP-44 strictly monotonic sequence numbers (`seq > current_seq`) paired with Hybrid `Ed25519` + `WOTS+-SHAKE256` (RFC 8391 / NIST FIPS 205 hash-based PQ) threshold multi-signatures ($M$-of-$N$ federation quorum).
|
| 38 |
+
6. **Sybil-Resistant ASN Quorum & Canary Gate:** Enforces $K$-of-$N$ distinct-ASN peer attestation quorum, strict format whitelisting (`SafeTensors`, `ONNX`, `JSON` only; zero pickle tolerance), and a mandatory local 500-item golden canary regression gate (rejects any update with malware bypass or Mondrian safety degradation).
|
| 39 |
+
7. **Opt-In / Opt-Out Modes:** Supports `off`, `pull_only`, and `anonymous_contribute`.
|
| 40 |
+
|
| 41 |
+
## Quickstart
|
| 42 |
+
|
| 43 |
+
```python
|
| 44 |
+
from calibration_and_dht import ArmigerCalibrator
|
| 45 |
+
from armiger_dht_rsi import ArmigerDHTRSIEngine, SeriesWebhookAdapter
|
| 46 |
+
|
| 47 |
+
# 1. Conformal calibration
|
| 48 |
+
cal = ArmigerCalibrator("calibration.json")
|
| 49 |
+
probs_cat = cal.calibrate_head("category", [2.4, -1.1, -3.0, -2.5, -3.1, -2.8, -3.2, -2.9, -3.0, -3.4, -2.0])
|
| 50 |
+
gates = cal.evaluate_safety_gates(p_adult=0.02, p_junk=0.004, p_excluded_scope=0.01)
|
| 51 |
+
print("Calibrated top prob:", max(probs_cat), "Safety gates:", gates)
|
| 52 |
+
|
| 53 |
+
# 2. Distributed RSI Engine
|
| 54 |
+
engine = ArmigerDHTRSIEngine(mode="anonymous_contribute")
|
| 55 |
+
obs = SeriesWebhookAdapter.parse_webhook({"eventType": "Download", "series": {"seriesCatalogId": 371980}})
|
| 56 |
+
ok, reason, record = engine.ingest_candidate(obs)
|
| 57 |
+
print("Ingested observation:", ok, reason)
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
## Part of the Armiger Modular Plugin Suite
|
| 61 |
+
|
| 62 |
+
| Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
|
| 63 |
+
| :--- | :---: | :---: | :--- |
|
| 64 |
+
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Corpus AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
|
| 65 |
+
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 66 |
+
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 67 |
+
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
| 68 |
+
| **[`NorviTech/armiger-calibration-dht`](https://huggingface.co/NorviTech/armiger-calibration-dht)** | **Yes** (`calibration.json`, `calibration_and_dht.py`, `armiger_dht_rsi.py`) | `181 KB` | **Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine:** Per-head temperature scaling ($T_h$), Mondrian safety gates (**`100.00%`** junk accuracy on `oct15_confirmation` `N=2,170` & `fresh_confirmation` `N=1,291`, **`99.90%`** on `beyond_confirmation` `N=977`), Hybrid `Ed25519`+`WOTS+-SHAKE256` signed `BEP-44`/`BEP-46` DHT hot-reload, Hybrid `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, distinct-ASN quorum, and 500-canary regression gate (`47/47` tests). |
|
| 69 |
+
| **[`NorviTech/armiger-dht-db-sync`](https://huggingface.co/NorviTech/armiger-dht-db-sync)** | **Yes** (`csam_guard.py`, `armiger_dht_db_sync.py`) | `155 KB` | **Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync:** Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded `SHA-256` tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off `xxx` isolation, `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, binary Merkle proofs, and $K$-of-$N$ distinct-ASN Sybil quorum (`27/27` tests). |
|
| 70 |
+
|
| 71 |
+
---
|
| 72 |
+
|
| 73 |
+
## Intended Use & Legal Compliance
|
| 74 |
+
|
| 75 |
+
`Armiger`, `Armiger Ultra`, and all bundled plugins are open-source machine learning and cryptographic metadata-classification utilities designed strictly for **lawful use cases**, including:
|
| 76 |
+
- Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
|
| 77 |
+
- Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (`.mkv.exe`, `.scr`, `.vbs`) in peer-to-peer and decentralized storage research.
|
| 78 |
+
- Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.
|
| 79 |
+
|
| 80 |
+
**NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works.** Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.
|
| 81 |
+
|
| 82 |
+
---
|
| 83 |
+
|
| 84 |
+
<p align="center">
|
| 85 |
+
<a href="https://norvitech.com"><img alt="Part of the NorviTech Suite — open-source apps for the Linux desktop and the self-hosted stack" src="https://norvitech.com/assets/banner.svg" width="640"></a>
|
| 86 |
+
</p>
|
| 87 |
+
|
| 88 |
+
<p align="center">
|
| 89 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra">Armiger Ultra</a> ·
|
| 90 |
+
<a href="https://huggingface.co/NorviTech/armiger">Armiger</a> ·
|
| 91 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-survival">Swarm Survival</a> ·
|
| 92 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-priors">Swarm Priors</a> ·
|
| 93 |
+
<a href="https://huggingface.co/NorviTech/armiger-zero-cue-catalog">Zero-Cue Catalog</a> ·
|
| 94 |
+
<a href="https://huggingface.co/NorviTech/armiger-audio-guard">Audio Guard</a> ·
|
| 95 |
+
<a href="https://huggingface.co/NorviTech/armiger-calibration-dht">Calibration & DHT RSI</a> ·
|
| 96 |
+
<a href="https://huggingface.co/NorviTech/armiger-dht-db-sync">DHT DB Sync</a> ·
|
| 97 |
+
<a href="https://norvitech.com">norvitech.com</a>
|
| 98 |
+
</p>
|
docs/plugins/dht-db-sync.md
ADDED
|
@@ -0,0 +1,188 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<h1 align="center">Armiger Plugin — Modular, CSAM-Protected, Encrypted DHT Database Sync (<code>armiger-dht-db-sync</code>)</h1>
|
| 2 |
+
|
| 3 |
+
<p align="center">
|
| 4 |
+
<strong>Decentralized <code>BEP-44</code> / <code>BEP-46</code> database synchronization engine featuring multi-layer CSAM & abuse blocking, <code>Ed25519</code>-signed blinded <code>SHA-256</code> tombstone purging, modular fine-grained category & language subscriptions with isolated <code>xxx</code> containment, <code>X25519</code> + <code>ChaCha20-Poly1305</code> AEAD encryption, and $K$-of-$N$ distinct-ASN Sybil quorum verification.</strong><br>
|
| 5 |
+
Designed for decentralized indexers (<code>Bitmagnet</code> and self-hosted library fleets) powered by <a href="https://huggingface.co/NorviTech/armiger-ultra"><code>NorviTech/armiger-ultra</code></a>.
|
| 6 |
+
</p>
|
| 7 |
+
|
| 8 |
+
<p align="center">
|
| 9 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Compatible with Armiger Ultra" src="https://img.shields.io/badge/Bundled_with-Armiger_Ultra-FD8024.svg"></a>
|
| 10 |
+
<a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
|
| 11 |
+
<img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
|
| 12 |
+
<img alt="Tests Passing" src="https://img.shields.io/badge/tests-27%2F27%20passing-brightgreen.svg">
|
| 13 |
+
</p>
|
| 14 |
+
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
## Architecture & Security Invariants
|
| 18 |
+
|
| 19 |
+
### 1. Multi-Layer CSAM & Harmful-Content Protection
|
| 20 |
+
- **Multi-Stage ContentFilter Parity:** Full unicode NFKD decomposition, diacritic stripping, zero-width / bidirectional formatting removal, and leetspeak deobfuscation (`0`→`o`, `1`→`i`, `3`→`e`, `4`/`@`→`a`, `$`→`s`, `7`/`+`→`t`).
|
| 21 |
+
- **Deterministic & Structural Pre-Ingest Blocker:** Inspects titles, file lists, and tags before shard generation. Blocks child sexual abuse material (CSAM / CSAE), non-consensual sexual exploitation, and extreme violence / snuff. Disguised executable droppers (`.mp4.exe`, `.scr`, `.bat`, `.vbs`, `.ps1`) in media categories are unconditionally rejected.
|
| 22 |
+
- **Ed25519-Signed Blinded SHA-256 Tombstone Purge Engine:** Revocations use `SHA-256("armiger-tombstone-v1:" || infohash)`. Manifests require strictly increasing sequence counters (`seq > current_seq`) and valid Ed25519 root signatures. Nodes automatically sweep and purge matching entries from local databases and ban re-ingestion.
|
| 23 |
+
- **Mandatory Armiger Conformal Safety Gate:** Mondrian conformal gates (`junk < 0.1010`, `scope < 0.4770`, `adult >= 0.5451` routed strictly to adult containment).
|
| 24 |
+
|
| 25 |
+
### 2. Modular Sharded Subscription Architecture
|
| 26 |
+
- **Partition Dimensions:** Partitioned across Category (10 modules) $\times$ Language (14 modules) $\times$ Quality Tier (7 tiers) $\times$ Time Epoch.
|
| 27 |
+
- **10 Category Modules:** `movie`, `tv_show`, `music`, `game`, `software`, `anime`, `book`, `comic`, `course`, and `xxx`.
|
| 28 |
+
- **Strict `xxx` Isolation & Opt-In:** Adult content is **strictly disabled by default** (`enable_xxx=False`). Adult shards use a segregated namespace and storage table (`adult_torrents`). Mainstream searches never query or return adult records, and mainstream shards never carry adult entries.
|
| 29 |
+
- **Selective Shard Download:** Subscribers configure exact subsets (e.g. `categories={"movie", "tv_show"}`, `languages={"en", "ja"}`). DHT lookups only fetch subscribed shards—zero network bandwidth or storage is spent on unwanted modules.
|
| 30 |
+
|
| 31 |
+
### 3. X25519 + ChaCha20-Poly1305 AEAD Encryption
|
| 32 |
+
- **Key Derivation:** HKDF-SHA256 derives 32-byte ChaCha20-Poly1305 keys salted by `shard_id` and bounded by `epoch`.
|
| 33 |
+
- **Authenticated Associated Data (AAD):** `armiger-shard-aad-v1:{shard_id}:{epoch}:{seq}:{merkle_root}` is bound to every ciphertext block.
|
| 34 |
+
- **Anti-Transplant & Replay Defense:** Any tampering, sequence alteration, or cross-shard ciphertext transplant immediately fails Poly1305 authentication.
|
| 35 |
+
|
| 36 |
+
### 4. BEP-44 / BEP-46 Root-of-Trust + K-of-N Distinct-ASN Quorum
|
| 37 |
+
- **BEP-44 Mutable DHT Pointers:** Target key calculated via `SHA1(pubkey || salt)`.
|
| 38 |
+
- **BEP-46 Infohash Pointers:** Canonical infohash references encrypted micro-shards.
|
| 39 |
+
- **Sybil-Resistant $K$-of-$N$ Quorum:** Requires $K \ge 5$ cryptographically valid Ed25519 attestations from $\ge 5$ **distinct Autonomous System Numbers (ASNs)** (e.g., AS15169 Google, AS13335 Cloudflare, AS16509 Amazon, AS7922 Comcast, AS3356 Lumen). Multi-node Sybil attacks within the same ASN or /24 subnet are automatically rejected.
|
| 40 |
+
- **Binary Merkle Tree Verification:** Each shard payload forms a binary Merkle tree of canonical record hashes. Manifest carries signed `merkle_root`. Supports individual inclusion proofs for auditability.
|
| 41 |
+
|
| 42 |
+
### 5. Anonymity & Dynamic Opt-In/Opt-Out
|
| 43 |
+
- **Zero PII:** No IP addresses, user tokens, passkeys, or client headers are stored or broadcast.
|
| 44 |
+
- **Sanitized Metadata:** Private tracker passkeys (`[passkey=...]`, `?passkey=...`) are stripped prior to hashing.
|
| 45 |
+
- **Sync Modes:** `OFF`, `PULL_ONLY` (download only), `SUBSCRIBE_AND_SEED` (distribute subscribed shards), `PUBLISHER`.
|
| 46 |
+
|
| 47 |
+
---
|
| 48 |
+
|
| 49 |
+
## Quickstart
|
| 50 |
+
|
| 51 |
+
```python
|
| 52 |
+
from armiger_dht_db_sync import (
|
| 53 |
+
ArmigerDHTDatabaseSync,
|
| 54 |
+
SubscriptionConfig,
|
| 55 |
+
SyncMode,
|
| 56 |
+
create_mock_quorum_attestations,
|
| 57 |
+
generate_ed25519_keypair,
|
| 58 |
+
)
|
| 59 |
+
from csam_guard import sign_tombstone_list
|
| 60 |
+
|
| 61 |
+
# 1. Initialize root keys and subscription
|
| 62 |
+
root_priv, root_pub = generate_ed25519_keypair()
|
| 63 |
+
sub_cfg = SubscriptionConfig(
|
| 64 |
+
categories={"movie", "anime"},
|
| 65 |
+
languages={"en", "ja"},
|
| 66 |
+
quality_tiers={"uhd_4k", "hd_1080p"},
|
| 67 |
+
epochs={"2026_q4"},
|
| 68 |
+
enable_xxx=False, # Adult content strictly excluded
|
| 69 |
+
sync_mode=SyncMode.PULL_ONLY,
|
| 70 |
+
)
|
| 71 |
+
|
| 72 |
+
sync_engine = ArmigerDHTDatabaseSync(
|
| 73 |
+
subscription=sub_cfg,
|
| 74 |
+
trusted_root_pubkeys_hex=[root_pub.hex()],
|
| 75 |
+
)
|
| 76 |
+
|
| 77 |
+
# 2. Ingest candidate record with CSAM & safety checks
|
| 78 |
+
ok, msg, record = sync_engine.ingest_candidate_record(
|
| 79 |
+
infohash="0123456789abcdef0123456789abcdef01234567",
|
| 80 |
+
title="Spring.2019.1080p.WEBM.AV1-OPEN",
|
| 81 |
+
category="anime",
|
| 82 |
+
language="ja",
|
| 83 |
+
quality_tier="hd_1080p",
|
| 84 |
+
epoch="2026_q4",
|
| 85 |
+
p_adult=0.01,
|
| 86 |
+
p_junk=0.001,
|
| 87 |
+
p_excluded_scope=0.01,
|
| 88 |
+
)
|
| 89 |
+
print("Ingestion result:", ok, msg)
|
| 90 |
+
|
| 91 |
+
# 3. Search local database
|
| 92 |
+
results = sync_engine.db.search(query="Spring", category="anime")
|
| 93 |
+
print("Found torrents:", len(results))
|
| 94 |
+
|
| 95 |
+
# 4. Apply Ed25519-signed blinded tombstone purge
|
| 96 |
+
tombstone_manifest = sign_tombstone_list(
|
| 97 |
+
seq=1,
|
| 98 |
+
infohashes_or_blinded=["0123456789abcdef0123456789abcdef01234567"],
|
| 99 |
+
private_key_bytes=root_priv,
|
| 100 |
+
)
|
| 101 |
+
ok, msg, new_count, purged = sync_engine.apply_signed_tombstones(tombstone_manifest)
|
| 102 |
+
print("Tombstone purge executed. Purged records:", purged)
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
---
|
| 106 |
+
|
| 107 |
+
## Test Suite Execution
|
| 108 |
+
|
| 109 |
+
Run the comprehensive 27-point test suite:
|
| 110 |
+
|
| 111 |
+
```bash
|
| 112 |
+
python3 test_dht_db_sync.py -v
|
| 113 |
+
```
|
| 114 |
+
|
| 115 |
+
All 27 test cases pass cleanly:
|
| 116 |
+
```text
|
| 117 |
+
test_ciphertext_tampering_defense ... ok
|
| 118 |
+
test_cross_shard_transplant_defense ... ok
|
| 119 |
+
test_key_derivation_deterministic_and_unique ... ok
|
| 120 |
+
test_round_trip_encryption_decryption ... ok
|
| 121 |
+
test_merkle_tree_construction_and_inclusion_proofs ... ok
|
| 122 |
+
test_quorum_verification_success ... ok
|
| 123 |
+
test_sybil_attack_rejected_duplicate_asns ... ok
|
| 124 |
+
test_armiger_mondrian_safety_gate ... ok
|
| 125 |
+
test_direct_csam_detection ... ok
|
| 126 |
+
test_disguised_malware_dropper_in_media_torrents ... ok
|
| 127 |
+
test_nonconsensual_and_extreme_violence_detection ... ok
|
| 128 |
+
test_private_tracker_passkey_sanitization ... ok
|
| 129 |
+
test_signed_tombstone_purge_lifecycle ... ok
|
| 130 |
+
test_text_normalization_and_leetspeak ... ok
|
| 131 |
+
test_bep44_target_derivation ... ok
|
| 132 |
+
test_ciphertext_sha256_mismatch_rejected ... ok
|
| 133 |
+
test_full_publish_subscribe_and_shard_application ... ok
|
| 134 |
+
test_merkle_root_mismatch_rejected ... ok
|
| 135 |
+
test_pre_ingest_reinspection_drops_bad_records_in_shard ... ok
|
| 136 |
+
test_stale_or_downgrade_sequence_rejected ... ok
|
| 137 |
+
test_sync_mode_off_disables_all_operations ... ok
|
| 138 |
+
test_untrusted_root_key_rejected ... ok
|
| 139 |
+
test_x25519_ecdh_key_exchange ... ok
|
| 140 |
+
test_database_table_segregation ... ok
|
| 141 |
+
test_selective_shard_filtering ... ok
|
| 142 |
+
test_strict_xxx_isolation_in_subscription ... ok
|
| 143 |
+
test_taxonomy_definitions ... ok
|
| 144 |
+
|
| 145 |
+
Ran 27 tests in 0.011s - OK
|
| 146 |
+
```
|
| 147 |
+
|
| 148 |
+
---
|
| 149 |
+
|
| 150 |
+
## Part of the Armiger Modular Plugin Suite
|
| 151 |
+
|
| 152 |
+
| Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
|
| 153 |
+
| :--- | :---: | :---: | :--- |
|
| 154 |
+
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Corpus AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
|
| 155 |
+
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 156 |
+
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 157 |
+
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
| 158 |
+
| **[`NorviTech/armiger-calibration-dht`](https://huggingface.co/NorviTech/armiger-calibration-dht)** | **Yes** (`calibration.json`, `calibration_and_dht.py`, `armiger_dht_rsi.py`) | `181 KB` | **Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine:** Per-head temperature scaling ($T_h$), Mondrian safety gates (**`100.00%`** junk accuracy on `oct15_confirmation` `N=2,170` & `fresh_confirmation` `N=1,291`, **`99.90%`** on `beyond_confirmation` `N=977`), Hybrid `Ed25519`+`WOTS+-SHAKE256` signed `BEP-44`/`BEP-46` DHT hot-reload, Hybrid `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, distinct-ASN quorum, and 500-canary regression gate (`47/47` tests). |
|
| 159 |
+
| **[`NorviTech/armiger-dht-db-sync`](https://huggingface.co/NorviTech/armiger-dht-db-sync)** | **Yes** (`csam_guard.py`, `armiger_dht_db_sync.py`) | `155 KB` | **Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync:** Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded `SHA-256` tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off `xxx` isolation, `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, binary Merkle proofs, and $K$-of-$N$ distinct-ASN Sybil quorum (`27/27` tests). |
|
| 160 |
+
|
| 161 |
+
---
|
| 162 |
+
|
| 163 |
+
## Intended Use & Legal Compliance
|
| 164 |
+
|
| 165 |
+
`Armiger`, `Armiger Ultra`, and all bundled plugins are open-source machine learning and cryptographic metadata-classification utilities designed strictly for **lawful use cases**, including:
|
| 166 |
+
- Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
|
| 167 |
+
- Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (`.mkv.exe`, `.scr`, `.vbs`) in peer-to-peer and decentralized storage research.
|
| 168 |
+
- Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.
|
| 169 |
+
|
| 170 |
+
**NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works.** Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.
|
| 171 |
+
|
| 172 |
+
---
|
| 173 |
+
|
| 174 |
+
<p align="center">
|
| 175 |
+
<a href="https://norvitech.com"><img alt="Part of the NorviTech Suite — open-source apps for the Linux desktop and the self-hosted stack" src="https://norvitech.com/assets/banner.svg" width="640"></a>
|
| 176 |
+
</p>
|
| 177 |
+
|
| 178 |
+
<p align="center">
|
| 179 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra">Armiger Ultra</a> ·
|
| 180 |
+
<a href="https://huggingface.co/NorviTech/armiger">Armiger</a> ·
|
| 181 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-survival">Swarm Survival</a> ·
|
| 182 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-priors">Swarm Priors</a> ·
|
| 183 |
+
<a href="https://huggingface.co/NorviTech/armiger-zero-cue-catalog">Zero-Cue Catalog</a> ·
|
| 184 |
+
<a href="https://huggingface.co/NorviTech/armiger-audio-guard">Audio Guard</a> ·
|
| 185 |
+
<a href="https://huggingface.co/NorviTech/armiger-calibration-dht">Calibration & DHT RSI</a> ·
|
| 186 |
+
<a href="https://huggingface.co/NorviTech/armiger-dht-db-sync">DHT DB Sync</a> ·
|
| 187 |
+
<a href="https://norvitech.com">norvitech.com</a>
|
| 188 |
+
</p>
|
docs/plugins/swarm-priors.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<h1 align="center">Armiger Plugin #2 — Swarm & Release-Group Empirical Bayes Priors (<code>armiger-swarm-priors</code>)</h1>
|
| 2 |
+
|
| 3 |
+
<p align="center">
|
| 4 |
+
<strong>Zero-dependency Empirical Bayes prior engine implementing Armiger's 4 strictly separated swarm signal pathways, mandatory modality dropout, and public-tracker passkey sanitization.</strong><br>
|
| 5 |
+
Available standalone here and pre-bundled inside <a href="https://huggingface.co/NorviTech/armiger-ultra"><code>NorviTech/armiger-ultra</code></a>.
|
| 6 |
+
</p>
|
| 7 |
+
|
| 8 |
+
<p align="center">
|
| 9 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Bundled in Armiger Ultra" src="https://img.shields.io/badge/Bundled_in-Armiger_Ultra-FD8024.svg"></a>
|
| 10 |
+
<a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
|
| 11 |
+
<img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
## The 4 Signal Pathways (`swarm_priors.py`)
|
| 15 |
+
|
| 16 |
+
1. **Way 1 — Active Teacher Sampling & Queue Priority Weighting (`compute_sampling_priority_weight`):** Prioritizes high-uncertainty and high-demand swarms in `Bitmagnet` and self-hosted indexer ingestion queues.
|
| 17 |
+
2. **Way 2 — Public Tracker Domain & Release-Group Bayesian Priors (`extract_way2_source_features`):** Extracts shrinkage-smoothed release-group, codec, resolution, and public-tracker domain affinities with **Mandatory Modality Dropout** ($p \in [0.50, 0.70]$) and strict invariance to `seeders == 0`.
|
| 18 |
+
3. **Way 3 — Swarm-vs-Content Anomaly Features (`extract_way3_anomaly_features`):** Computes seeder inflation $z$-scores, size-vs-resolution mismatches, and executable/shortcut trap indicators for `junk` / malware detection.
|
| 19 |
+
4. **Way 4 — Orthogonal Swarm Viability & Stage-2 Grab Ranking (`compute_swarm_viability`):** Produces a calibrated `[0.0, 1.0]` download viability multiplier (`86.64%` standalone formula AUROC; `96.83%` multi-source corpus / `92.42%` public-tracker AUROC when paired with `armiger-swarm-survival`) without ever altering the predicted content `category`.
|
| 20 |
+
|
| 21 |
+
## Quickstart
|
| 22 |
+
|
| 23 |
+
```python
|
| 24 |
+
from swarm_priors import (
|
| 25 |
+
extract_way2_source_features,
|
| 26 |
+
extract_way3_anomaly_features,
|
| 27 |
+
compute_swarm_viability,
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
w2 = extract_way2_source_features(
|
| 31 |
+
title="[OpenAnime] Spring - 04 (1080p) [E4A1B9C2].mkv",
|
| 32 |
+
sources=["torrent.ubuntu.com", "bt1.archive.org"],
|
| 33 |
+
total_size_bytes=1_420_000_000,
|
| 34 |
+
files=[{"path": "[OpenAnime] Spring - 04 (1080p).mkv", "size": 1_420_000_000}],
|
| 35 |
+
)
|
| 36 |
+
w3 = extract_way3_anomaly_features(
|
| 37 |
+
title="[OpenAnime] Spring - 04 (1080p) [E4A1B9C2].mkv",
|
| 38 |
+
seeders=185,
|
| 39 |
+
leechers=12,
|
| 40 |
+
completed=940,
|
| 41 |
+
total_size_bytes=1_420_000_000,
|
| 42 |
+
files=[{"path": "[OpenAnime] Spring - 04 (1080p).mkv", "size": 1_420_000_000}],
|
| 43 |
+
)
|
| 44 |
+
print(compute_swarm_viability(seeders=185, leechers=12, completed=940, category="tv_show", way2_features=w2, way3_features=w3))
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
## Part of the Armiger Modular Plugin Suite
|
| 48 |
+
|
| 49 |
+
| Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
|
| 50 |
+
| :--- | :---: | :---: | :--- |
|
| 51 |
+
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Corpus AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
|
| 52 |
+
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 53 |
+
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 54 |
+
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
| 55 |
+
| **[`NorviTech/armiger-calibration-dht`](https://huggingface.co/NorviTech/armiger-calibration-dht)** | **Yes** (`calibration.json`, `calibration_and_dht.py`, `armiger_dht_rsi.py`) | `181 KB` | **Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine:** Per-head temperature scaling ($T_h$), Mondrian safety gates (**`100.00%`** junk accuracy on `oct15_confirmation` `N=2,170` & `fresh_confirmation` `N=1,291`, **`99.90%`** on `beyond_confirmation` `N=977`), Hybrid `Ed25519`+`WOTS+-SHAKE256` signed `BEP-44`/`BEP-46` DHT hot-reload, Hybrid `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, distinct-ASN quorum, and 500-canary regression gate (`47/47` tests). |
|
| 56 |
+
| **[`NorviTech/armiger-dht-db-sync`](https://huggingface.co/NorviTech/armiger-dht-db-sync)** | **Yes** (`csam_guard.py`, `armiger_dht_db_sync.py`) | `155 KB` | **Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync:** Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded `SHA-256` tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off `xxx` isolation, `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, binary Merkle proofs, and $K$-of-$N$ distinct-ASN Sybil quorum (`27/27` tests). |
|
| 57 |
+
|
| 58 |
+
---
|
| 59 |
+
|
| 60 |
+
## Intended Use & Legal Compliance
|
| 61 |
+
|
| 62 |
+
`Armiger`, `Armiger Ultra`, and all bundled plugins are open-source machine learning and cryptographic metadata-classification utilities designed strictly for **lawful use cases**, including:
|
| 63 |
+
- Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
|
| 64 |
+
- Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (`.mkv.exe`, `.scr`, `.vbs`) in peer-to-peer and decentralized storage research.
|
| 65 |
+
- Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.
|
| 66 |
+
|
| 67 |
+
**NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works.** Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.
|
| 68 |
+
|
| 69 |
+
---
|
| 70 |
+
|
| 71 |
+
<p align="center">
|
| 72 |
+
<a href="https://norvitech.com"><img alt="Part of the NorviTech Suite — open-source apps for the Linux desktop and the self-hosted stack" src="https://norvitech.com/assets/banner.svg" width="640"></a>
|
| 73 |
+
</p>
|
| 74 |
+
|
| 75 |
+
<p align="center">
|
| 76 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra">Armiger Ultra</a> ·
|
| 77 |
+
<a href="https://huggingface.co/NorviTech/armiger">Armiger</a> ·
|
| 78 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-survival">Swarm Survival</a> ·
|
| 79 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-priors">Swarm Priors</a> ·
|
| 80 |
+
<a href="https://huggingface.co/NorviTech/armiger-zero-cue-catalog">Zero-Cue Catalog</a> ·
|
| 81 |
+
<a href="https://huggingface.co/NorviTech/armiger-audio-guard">Audio Guard</a> ·
|
| 82 |
+
<a href="https://huggingface.co/NorviTech/armiger-calibration-dht">Calibration & DHT RSI</a> ·
|
| 83 |
+
<a href="https://huggingface.co/NorviTech/armiger-dht-db-sync">DHT DB Sync</a> ·
|
| 84 |
+
<a href="https://norvitech.com">norvitech.com</a>
|
| 85 |
+
</p>
|
docs/plugins/swarm-survival.md
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<h1 align="center">Armiger Plugin #1 — Swarm Survival & Honeypot Adapter (<code>SwarmSurvivalNet v2</code>)</h1>
|
| 2 |
+
|
| 3 |
+
<p align="center">
|
| 4 |
+
<strong>48-tensor PyTorch content-gated residual adapter that predicts real-world torrent download completion (<code>96.83%</code> Multi-Source Corpus AUROC / <code>92.42%</code>–<code>93.03%</code> Public-Tracker AUROC) and blocks <code>100.0%</code> of fake-seeder malware honeypots while guaranteeing <code>0%</code> content-category flips.</strong><br>
|
| 5 |
+
Available standalone here and pre-bundled inside <a href="https://huggingface.co/NorviTech/armiger-ultra"><code>NorviTech/armiger-ultra</code></a>.
|
| 6 |
+
</p>
|
| 7 |
+
|
| 8 |
+
<p align="center">
|
| 9 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Bundled in Armiger Ultra" src="https://img.shields.io/badge/Bundled_in-Armiger_Ultra-FD8024.svg"></a>
|
| 10 |
+
<a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
|
| 11 |
+
<img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
## Overview
|
| 15 |
+
|
| 16 |
+
Raw tracker seeder counts are an unreliable predictor of whether a torrent grab will actually complete (`53.02%` AUROC across multi-source corpus swarms and `85.17%` on single-source public trackers):
|
| 17 |
+
1. **Fake-Seeder Malware Honeypots** spoof `5,000+` seeders on day zero for unreleased movies or games while bundling `.exe` / `.lnk` / `.scr` / codec-pack payloads.
|
| 18 |
+
2. **Stale Tracker Scrapes vs. DHT/PEX Survival:** Niche anime, audiobooks, and remuxes often report `0` cached tracker seeders yet complete reliably via DHT/PEX, whereas poisoned swarms stall at `99.9%`.
|
| 19 |
+
|
| 20 |
+
`NorviTech/armiger-swarm-survival` (`swarm_adapter.pt` + `swarm_survival.py`) loads all **48 neural tensors** (`swarm_mlp`, `cross_gate`, `content_proj`, `Way3SwarmAnomalyHead`, and `Way4ViabilityHead`) into PyTorch `nn.Module` layers (`SwarmResidualAdapter`) and executes the learned neural forward pass (`self.forward(...)`) inside `evaluate_swarm(...)`.
|
| 21 |
+
|
| 22 |
+
## Measured Performance (Audited Holdout Splits)
|
| 23 |
+
|
| 24 |
+
| Metric | Raw Tracker Seeders | Raw DHT Peers | Armiger (`Way-4` Formula) | **Armiger Ultra + `swarm_adapter.pt` (48-Tensor Neural)** |
|
| 25 |
+
| :--- | :---: | :---: | :---: | :---: |
|
| 26 |
+
| **Multi-Source Corpus Holdout AUROC (`N=10,278`)** | `53.02%` | `51.84%` | `86.64%` | **`96.83%`** (`97.90%` AUPRC) |
|
| 27 |
+
| **Public-Tracker Benchmark AUROC (`N=757`)** | `85.17%` | `86.81%` | `86.64%` | **`92.42%`** (`+7.25 pp` vs. seeders) |
|
| 28 |
+
| **Public-Tracker Holdout AUROC (`N=493`)** | `85.17%` | `86.81%` | `86.64%` | **`93.03%`** |
|
| 29 |
+
| **Fake-Seeder Honeypot Trap Recall (`@ <=0.5% FPR`)** | `0.00%` | `0.00%` | `99.90%` | **`100.00%`** |
|
| 30 |
+
| **Content Category Flip Rate (`seeders=0` vs `seeders=10,000`)** | N/A | N/A | **`0.00%`** | **`0.00%`** |
|
| 31 |
+
|
| 32 |
+
## Quickstart
|
| 33 |
+
|
| 34 |
+
```python
|
| 35 |
+
from swarm_survival import SwarmResidualAdapter
|
| 36 |
+
|
| 37 |
+
adapter = SwarmResidualAdapter.from_pretrained("swarm_adapter.pt")
|
| 38 |
+
result = adapter.evaluate_swarm(
|
| 39 |
+
title="Tears.of.Steel.2012.2160p.UHD.Master.x265-OPEN",
|
| 40 |
+
seeders=42,
|
| 41 |
+
leechers=5,
|
| 42 |
+
completed=310,
|
| 43 |
+
total_size_bytes=28_500_000_000,
|
| 44 |
+
files=[{"path": "Tears.of.Steel.2012.2160p.mkv", "size": 28_500_000_000}],
|
| 45 |
+
sources=["bt1.archive.org"],
|
| 46 |
+
category_pred="movie",
|
| 47 |
+
p_junk_neural=0.002,
|
| 48 |
+
)
|
| 49 |
+
print(result)
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
## Part of the Armiger Modular Plugin Suite
|
| 53 |
+
|
| 54 |
+
| Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
|
| 55 |
+
| :--- | :---: | :---: | :--- |
|
| 56 |
+
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Corpus AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
|
| 57 |
+
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 58 |
+
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 59 |
+
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
| 60 |
+
| **[`NorviTech/armiger-calibration-dht`](https://huggingface.co/NorviTech/armiger-calibration-dht)** | **Yes** (`calibration.json`, `calibration_and_dht.py`, `armiger_dht_rsi.py`) | `181 KB` | **Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine:** Per-head temperature scaling ($T_h$), Mondrian safety gates (**`100.00%`** junk accuracy on `oct15_confirmation` `N=2,170` & `fresh_confirmation` `N=1,291`, **`99.90%`** on `beyond_confirmation` `N=977`), Hybrid `Ed25519`+`WOTS+-SHAKE256` signed `BEP-44`/`BEP-46` DHT hot-reload, Hybrid `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, distinct-ASN quorum, and 500-canary regression gate (`47/47` tests). |
|
| 61 |
+
| **[`NorviTech/armiger-dht-db-sync`](https://huggingface.co/NorviTech/armiger-dht-db-sync)** | **Yes** (`csam_guard.py`, `armiger_dht_db_sync.py`) | `155 KB` | **Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync:** Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded `SHA-256` tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off `xxx` isolation, `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, binary Merkle proofs, and $K$-of-$N$ distinct-ASN Sybil quorum (`27/27` tests). |
|
| 62 |
+
|
| 63 |
+
---
|
| 64 |
+
|
| 65 |
+
## Intended Use & Legal Compliance
|
| 66 |
+
|
| 67 |
+
`Armiger`, `Armiger Ultra`, and all bundled plugins are open-source machine learning and cryptographic metadata-classification utilities designed strictly for **lawful use cases**, including:
|
| 68 |
+
- Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
|
| 69 |
+
- Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (`.mkv.exe`, `.scr`, `.vbs`) in peer-to-peer and decentralized storage research.
|
| 70 |
+
- Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.
|
| 71 |
+
|
| 72 |
+
**NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works.** Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.
|
| 73 |
+
|
| 74 |
+
---
|
| 75 |
+
|
| 76 |
+
<p align="center">
|
| 77 |
+
<a href="https://norvitech.com"><img alt="Part of the NorviTech Suite — open-source apps for the Linux desktop and the self-hosted stack" src="https://norvitech.com/assets/banner.svg" width="640"></a>
|
| 78 |
+
</p>
|
| 79 |
+
|
| 80 |
+
<p align="center">
|
| 81 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra">Armiger Ultra</a> ·
|
| 82 |
+
<a href="https://huggingface.co/NorviTech/armiger">Armiger</a> ·
|
| 83 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-survival">Swarm Survival</a> ·
|
| 84 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-priors">Swarm Priors</a> ·
|
| 85 |
+
<a href="https://huggingface.co/NorviTech/armiger-zero-cue-catalog">Zero-Cue Catalog</a> ·
|
| 86 |
+
<a href="https://huggingface.co/NorviTech/armiger-audio-guard">Audio Guard</a> ·
|
| 87 |
+
<a href="https://huggingface.co/NorviTech/armiger-calibration-dht">Calibration & DHT RSI</a> ·
|
| 88 |
+
<a href="https://huggingface.co/NorviTech/armiger-dht-db-sync">DHT DB Sync</a> ·
|
| 89 |
+
<a href="https://norvitech.com">norvitech.com</a>
|
| 90 |
+
</p>
|
docs/plugins/zero-cue-catalog.md
ADDED
|
@@ -0,0 +1,89 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<h1 align="center">Armiger Plugin #3 — Zero-Cue Catalog & Structural Disambiguator (<code>armiger-zero-cue-catalog</code>)</h1>
|
| 2 |
+
|
| 3 |
+
<p align="center">
|
| 4 |
+
<strong>Resolves bare-title ("zero-cue") torrents lacking season/episode syntax via an offline 929,400-entry <code>BLAKE2b-64</code> cryptographically hashed media catalog (<code>26.0 MB</code>) and deterministic structural rules, lifting 5-split (<code>N=10,262</code>) category accuracy from <code>94.00%</code> to <code>95.42%</code> and specialist zero-cue accuracy from <code>87.40%</code> to <code>91.50%</code> (<code>+4.10 pp</code>).</strong><br>
|
| 5 |
+
Available standalone here and pre-bundled inside <a href="https://huggingface.co/NorviTech/armiger-ultra"><code>NorviTech/armiger-ultra</code></a>.
|
| 6 |
+
</p>
|
| 7 |
+
|
| 8 |
+
<p align="center">
|
| 9 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra"><img alt="Bundled in Armiger Ultra" src="https://img.shields.io/badge/Bundled_in-Armiger_Ultra-FD8024.svg"></a>
|
| 10 |
+
<a href="https://huggingface.co/NorviTech/armiger"><img alt="Compatible with Armiger" src="https://img.shields.io/badge/Compatible_with-Armiger_CPU-0284C7.svg"></a>
|
| 11 |
+
<img alt="Licence Apache-2.0" src="https://img.shields.io/badge/licence-Apache--2.0-blue.svg">
|
| 12 |
+
</p>
|
| 13 |
+
|
| 14 |
+
## Why the Zero-Cue Catalog Plugin Exists
|
| 15 |
+
|
| 16 |
+
In real-world BitTorrent/DHT indexing, **2% to 4%** of movie, TV, and anime torrents contain **zero structural cues** in either their title or file tree:
|
| 17 |
+
- `Sintel (2010) [1080p] [WEB-DL].mkv` (a standalone film vs. a 1-file TV special)
|
| 18 |
+
- `[SubGroup] Bare Japanese/English Title [1080p].mkv` (a theatrical anime film vs. a TV OVA/series without an episode number)
|
| 19 |
+
- `Director Name - Movies 1-6 Collection (1979-1998) 1080p` (a multi-file movie pack whose 6 `.mkv` files look identical in file-count to a 6-episode TV mini-series)
|
| 20 |
+
|
| 21 |
+
Even a 70B+ parameter cloud LLM must guess when given only a bare title unless it can consult a media catalog. `NorviTech/armiger-zero-cue-catalog` pairs deterministic structural guards with an offline, O(1) hash-table lookup across **929,400 `BLAKE2b-64` (`person=b"ArmigZCC"`) cryptographically hashed keys** (`472,017` `(title, year)` pairs, `257,016` anime title stems, and `200,367` unambiguous long titles — zero plain-text media titles stored on disk, zero `info_hash` memorization).
|
| 22 |
+
|
| 23 |
+
## Measured Accuracy Lift Across All 6 Evaluation Cohorts (Unmutated Ground-Truth Keys)
|
| 24 |
+
|
| 25 |
+
| Evaluation Cohort (Unmutated `rows.y`) | Valid `N` | Standalone Neural (`FP32` / `INT8`) | **Fair Blind Hybrid (`+ armiger-zero-cue-catalog` FP32 / INT8)** | Net Lift |
|
| 26 |
+
| :--- | :---: | :---: | :---: | :---: |
|
| 27 |
+
| **`oct15_confirmation` (Gold Standard)** | `2,835` | `95.03%` / `95.06%` (`96.60%` Movie-vs-TV) | **`95.49%`** / **`95.49%`** (**`97.30%`** Movie-vs-TV) | `+0.46 pp` |
|
| 28 |
+
| **`fresh_confirmation` (Fresh Holdout)** | `1,402` | `94.51%` / `94.44%` | **`94.58%`** / **`94.51%`** | `+0.07 pp` |
|
| 29 |
+
| **`beyond_confirmation` (Hard Edge-Cases)** | `1,025` | `92.88%` / `92.88%` (`88.74%` Movie-vs-TV) | **`95.12%`** / **`95.12%`** (**`96.54%`** Movie-vs-TV) | **`+2.24 pp`** |
|
| 30 |
+
| **`tc_truespec_2000` (Specialist Zero-Cue)** | `2,000` | `87.40%` / `87.35%` | **`91.50%`** / **`91.50%`** | **`+4.10 pp`** |
|
| 31 |
+
| **`tc_truespec_multilang_3000` (Multilingual)** | `3,000` | `97.50%` / `97.37%` | **`98.47%`** / **`98.40%`** | **`+0.97 pp`** |
|
| 32 |
+
| **`calibration` (Calibration Split)** | `424` | `96.46%` / `96.46%` | **`96.70%`** / **`96.70%`** | `+0.24 pp` |
|
| 33 |
+
| **5-Split Weighted Mean (`N=10,262`)** | **`10,262`** | **`94.00%`** / **`93.95%`** | **`95.42%`** / **`95.39%`** | **`+1.42 pp`** |
|
| 34 |
+
|
| 35 |
+
## Quickstart
|
| 36 |
+
|
| 37 |
+
```python
|
| 38 |
+
from zero_cue_plugin import ZeroCueCatalogPlugin
|
| 39 |
+
|
| 40 |
+
plugin = ZeroCueCatalogPlugin("validated_zero_cue_catalog.json")
|
| 41 |
+
res = plugin.resolve(
|
| 42 |
+
raw_name="Sintel (2010) [1080p] [WEB-DL]",
|
| 43 |
+
probs_category=[0.46, 0.51, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.03],
|
| 44 |
+
p_adult=0.01,
|
| 45 |
+
p_anime=0.02,
|
| 46 |
+
)
|
| 47 |
+
print(res)
|
| 48 |
+
# -> {'category_index': 0, 'category': 'movie', 'resolution_source': 'zero_cue_title_year'}
|
| 49 |
+
```
|
| 50 |
+
|
| 51 |
+
## Part of the Armiger Modular Plugin Suite
|
| 52 |
+
|
| 53 |
+
| Plugin Repository | Bundled in `Armiger Ultra`? | Footprint | Core Capability & Measured Impact (Unmutated Ground-Truth Keys) |
|
| 54 |
+
| :--- | :---: | :---: | :--- |
|
| 55 |
+
| **[`NorviTech/armiger-swarm-survival`](https://huggingface.co/NorviTech/armiger-swarm-survival)** | **Yes** (`swarm_adapter.pt`, `swarm_survival.py`) | `167 KB` (`132 KB` `.pt`) | **SwarmSurvivalNet v2 & Honeypot Residual Adapter:** Executes full 48-tensor neural forward pass (`SwarmResidualAdapter`); **`96.83%`** Multi-Source Corpus AUROC (`N=10,278`, vs. `53.02%` raw seeders), **`92.42%`** / **`93.03%`** Public-Tracker AUROC (`N=757` / `N=493`, vs. `85.17%` raw seeders), **`100.0%`** fake-seeder malware trap recall, **`0.00%`** content-category flip guarantee. |
|
| 56 |
+
| **[`NorviTech/armiger-swarm-priors`](https://huggingface.co/NorviTech/armiger-swarm-priors)** | **Yes** (`swarm_priors.py`) | `29.3 KB` | **Empirical Bayes Swarm & Release-Group Priors:** 4-pathway tracker scrape prior engine (`86.64%` formula AUROC), mandatory modality dropout (`p=0.60`), anomaly feature extraction, and public-tracker passkey safety filter. |
|
| 57 |
+
| **[`NorviTech/armiger-zero-cue-catalog`](https://huggingface.co/NorviTech/armiger-zero-cue-catalog)** | **Yes** (`zero_cue_plugin.py`, `validated_zero_cue_catalog.json`) | `26.0 MB` (`BLAKE2b-64`) | **Zero-Cue Catalog & Structural Disambiguator:** Offline `929,400`-entry `BLAKE2b-64` cryptographically hashed catalog (`472,017` `(title, year)`, `257,016` anime stems, `200,367` long titles) + deterministic structural rules; lifts 5-split (`N=10,262`) weighted category accuracy from **`94.00%`** (Standalone Neural) to **`95.42%`** (Fair Blind Hybrid), `beyond_confirmation` (`N=1,025`) from **`92.88%`** to **`95.12%`**, and `tc_truespec_2000` (`N=2,000`) from **`87.40%`** to **`91.50%`** (`+4.10 pp`). |
|
| 58 |
+
| **[`NorviTech/armiger-audio-guard`](https://huggingface.co/NorviTech/armiger-audio-guard)** | **Yes** (`audio_language_guard.py`) | `3.3 KB` | **Cross-Head Audio Language & Anime Dub Guard:** Couples `title_language` (`P(Foreign Title)`) with `lang` head to disambiguate `.en.` prepositions, `<Foreign>.ENG.Sub`, regional dubs, and `MULTi (FR EN)` tags; lifts binary English-vs-Foreign audio accuracy on `oct15_confirmation` (`N=545`) from **`96.88%`** (Standalone Neural) to **`97.80%`** (`92.75%` 3-class `N=772`). |
|
| 59 |
+
| **[`NorviTech/armiger-calibration-dht`](https://huggingface.co/NorviTech/armiger-calibration-dht)** | **Yes** (`calibration.json`, `calibration_and_dht.py`, `armiger_dht_rsi.py`) | `181 KB` | **Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine:** Per-head temperature scaling ($T_h$), Mondrian safety gates (**`100.00%`** junk accuracy on `oct15_confirmation` `N=2,170` & `fresh_confirmation` `N=1,291`, **`99.90%`** on `beyond_confirmation` `N=977`), Hybrid `Ed25519`+`WOTS+-SHAKE256` signed `BEP-44`/`BEP-46` DHT hot-reload, Hybrid `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, distinct-ASN quorum, and 500-canary regression gate (`47/47` tests). |
|
| 60 |
+
| **[`NorviTech/armiger-dht-db-sync`](https://huggingface.co/NorviTech/armiger-dht-db-sync)** | **Yes** (`csam_guard.py`, `armiger_dht_db_sync.py`) | `155 KB` | **Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync:** Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded `SHA-256` tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off `xxx` isolation, `X25519`+`ML-KEM-768`+`ChaCha20-Poly1305` AEAD, binary Merkle proofs, and $K$-of-$N$ distinct-ASN Sybil quorum (`27/27` tests). |
|
| 61 |
+
|
| 62 |
+
---
|
| 63 |
+
|
| 64 |
+
## Intended Use & Legal Compliance
|
| 65 |
+
|
| 66 |
+
`Armiger`, `Armiger Ultra`, and all bundled plugins are open-source machine learning and cryptographic metadata-classification utilities designed strictly for **lawful use cases**, including:
|
| 67 |
+
- Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
|
| 68 |
+
- Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (`.mkv.exe`, `.scr`, `.vbs`) in peer-to-peer and decentralized storage research.
|
| 69 |
+
- Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.
|
| 70 |
+
|
| 71 |
+
**NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works.** Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.
|
| 72 |
+
|
| 73 |
+
---
|
| 74 |
+
|
| 75 |
+
<p align="center">
|
| 76 |
+
<a href="https://norvitech.com"><img alt="Part of the NorviTech Suite — open-source apps for the Linux desktop and the self-hosted stack" src="https://norvitech.com/assets/banner.svg" width="640"></a>
|
| 77 |
+
</p>
|
| 78 |
+
|
| 79 |
+
<p align="center">
|
| 80 |
+
<a href="https://huggingface.co/NorviTech/armiger-ultra">Armiger Ultra</a> ·
|
| 81 |
+
<a href="https://huggingface.co/NorviTech/armiger">Armiger</a> ·
|
| 82 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-survival">Swarm Survival</a> ·
|
| 83 |
+
<a href="https://huggingface.co/NorviTech/armiger-swarm-priors">Swarm Priors</a> ·
|
| 84 |
+
<a href="https://huggingface.co/NorviTech/armiger-zero-cue-catalog">Zero-Cue Catalog</a> ·
|
| 85 |
+
<a href="https://huggingface.co/NorviTech/armiger-audio-guard">Audio Guard</a> ·
|
| 86 |
+
<a href="https://huggingface.co/NorviTech/armiger-calibration-dht">Calibration & DHT RSI</a> ·
|
| 87 |
+
<a href="https://huggingface.co/NorviTech/armiger-dht-db-sync">DHT DB Sync</a> ·
|
| 88 |
+
<a href="https://norvitech.com">norvitech.com</a>
|
| 89 |
+
</p>
|
examples/README.md
CHANGED
|
@@ -6,6 +6,6 @@ Turnkey reference adapters, HTTP services, and Docker Compose stacks for integra
|
|
| 6 |
| :--- | :--- | :--- |
|
| 7 |
| **[DH-Waypoint](https://github.com/spencernorton/DH-Waypoint)** | [`examples/dh-waypoint/`](dh-waypoint) | Native `processorapi/v1` HTTP processor (`dh-waypoint.processor.v1`), pre-ingest cohort admission gate (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop RSI telemetry bridge. |
|
| 8 |
| **Bitmagnet** | [`examples/bitmagnet/`](bitmagnet) | Fast pre-ingest / post-crawl batch classifier, malware/honeypot pruner, PostgreSQL/GraphQL enrichment bridge (`bitmagnet_armiger_bridge.py`), and Torznab proxy stack. |
|
| 9 |
-
| **
|
| 10 |
|
| 11 |
See [`docs/INTEGRATION_GUIDE.md`](../docs/INTEGRATION_GUIDE.md) for the complete architecture and step-by-step setup guide.
|
|
|
|
| 6 |
| :--- | :--- | :--- |
|
| 7 |
| **[DH-Waypoint](https://github.com/spencernorton/DH-Waypoint)** | [`examples/dh-waypoint/`](dh-waypoint) | Native `processorapi/v1` HTTP processor (`dh-waypoint.processor.v1`), pre-ingest cohort admission gate (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop RSI telemetry bridge. |
|
| 8 |
| **Bitmagnet** | [`examples/bitmagnet/`](bitmagnet) | Fast pre-ingest / post-crawl batch classifier, malware/honeypot pruner, PostgreSQL/GraphQL enrichment bridge (`bitmagnet_armiger_bridge.py`), and Torznab proxy stack. |
|
| 9 |
+
| **Torznab Proxy & Webhook Suite** | [`examples/arr-stack/`](arr-stack) | Pre-grab Torznab filtering proxy (`armiger_torznab_proxy.py`) compatible with `Prowlarr`, `Bitmagnet`, `DH-Waypoint`, `Sonarr`, `Radarr`, `Lidarr`, and `Readarr`, plus a post-import webhook & `ffprobe` ground-truth receiver (`armiger_arr_webhook_receiver.py`). |
|
| 10 |
|
| 11 |
See [`docs/INTEGRATION_GUIDE.md`](../docs/INTEGRATION_GUIDE.md) for the complete architecture and step-by-step setup guide.
|
examples/arr-stack/armiger_torznab_proxy.py
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
"""Armiger Torznab Filtering Proxy & Pre-Grab Admission Gate for Prowlarr, Bitmagnet, and the *Arr Stack.
|
| 2 |
|
| 3 |
Sits transparently between upstream Torznab indexers (`Prowlarr`, `Bitmagnet`, `DH-Waypoint`, `Jackett`)
|
| 4 |
-
and downstream media managers (`Sonarr`, `Radarr`, `Lidarr`, `Readarr`
|
| 5 |
|
| 6 |
Every Torznab XML `<item>` returned by an upstream search is evaluated in milliseconds by
|
| 7 |
`ArmigerUltraPipeline.classify_batch` before `Sonarr` or `Radarr` can grab it:
|
|
|
|
| 1 |
"""Armiger Torznab Filtering Proxy & Pre-Grab Admission Gate for Prowlarr, Bitmagnet, and the *Arr Stack.
|
| 2 |
|
| 3 |
Sits transparently between upstream Torznab indexers (`Prowlarr`, `Bitmagnet`, `DH-Waypoint`, `Jackett`)
|
| 4 |
+
and downstream Torznab media managers (`Sonarr`, `Radarr`, `Lidarr`, `Readarr`).
|
| 5 |
|
| 6 |
Every Torznab XML `<item>` returned by an upstream search is evaluated in milliseconds by
|
| 7 |
`ArmigerUltraPipeline.classify_batch` before `Sonarr` or `Radarr` can grab it:
|
examples/arr-stack/docker-compose.yml
CHANGED
|
@@ -5,12 +5,12 @@ services:
|
|
| 5 |
restart: unless-stopped
|
| 6 |
working_dir: /app
|
| 7 |
ports:
|
| 8 |
-
- "8097:8097"
|
| 9 |
environment:
|
| 10 |
ARMIGER_MODEL_DIR: /app
|
| 11 |
ARMIGER_VARIANT: "${ARMIGER_VARIANT:-int8}"
|
| 12 |
UPSTREAM_TORZNAB_URL: "${UPSTREAM_TORZNAB_URL:-http://prowlarr:9696/1/api}"
|
| 13 |
-
ARMIGER_PROXY_HOST: "
|
| 14 |
ARMIGER_PROXY_PORT: "8097"
|
| 15 |
ARMIGER_REQUIRE_ENGLISH_AUDIO: "${ARMIGER_REQUIRE_ENGLISH_AUDIO:-false}"
|
| 16 |
ARMIGER_ALLOW_ENGLISH_SUBS: "${ARMIGER_ALLOW_ENGLISH_SUBS:-true}"
|
|
@@ -29,11 +29,11 @@ services:
|
|
| 29 |
restart: unless-stopped
|
| 30 |
working_dir: /app
|
| 31 |
ports:
|
| 32 |
-
- "8098:8098"
|
| 33 |
environment:
|
| 34 |
ARMIGER_MODEL_DIR: /app
|
| 35 |
ARMIGER_VARIANT: "int8"
|
| 36 |
-
ARMIGER_WEBHOOK_HOST: "
|
| 37 |
ARMIGER_WEBHOOK_PORT: "8098"
|
| 38 |
ARMIGER_RUN_FFPROBE: "true"
|
| 39 |
ARMIGER_DHT_RSI_MODE: "${ARMIGER_DHT_RSI_MODE:-pull_only}"
|
|
|
|
| 5 |
restart: unless-stopped
|
| 6 |
working_dir: /app
|
| 7 |
ports:
|
| 8 |
+
- "127.0.0.1:8097:8097"
|
| 9 |
environment:
|
| 10 |
ARMIGER_MODEL_DIR: /app
|
| 11 |
ARMIGER_VARIANT: "${ARMIGER_VARIANT:-int8}"
|
| 12 |
UPSTREAM_TORZNAB_URL: "${UPSTREAM_TORZNAB_URL:-http://prowlarr:9696/1/api}"
|
| 13 |
+
ARMIGER_PROXY_HOST: "0.0.0.0"
|
| 14 |
ARMIGER_PROXY_PORT: "8097"
|
| 15 |
ARMIGER_REQUIRE_ENGLISH_AUDIO: "${ARMIGER_REQUIRE_ENGLISH_AUDIO:-false}"
|
| 16 |
ARMIGER_ALLOW_ENGLISH_SUBS: "${ARMIGER_ALLOW_ENGLISH_SUBS:-true}"
|
|
|
|
| 29 |
restart: unless-stopped
|
| 30 |
working_dir: /app
|
| 31 |
ports:
|
| 32 |
+
- "127.0.0.1:8098:8098"
|
| 33 |
environment:
|
| 34 |
ARMIGER_MODEL_DIR: /app
|
| 35 |
ARMIGER_VARIANT: "int8"
|
| 36 |
+
ARMIGER_WEBHOOK_HOST: "0.0.0.0"
|
| 37 |
ARMIGER_WEBHOOK_PORT: "8098"
|
| 38 |
ARMIGER_RUN_FFPROBE: "true"
|
| 39 |
ARMIGER_DHT_RSI_MODE: "${ARMIGER_DHT_RSI_MODE:-pull_only}"
|
examples/dh-waypoint/armiger_waypoint_processor.py
CHANGED
|
@@ -1,25 +1,28 @@
|
|
| 1 |
"""DH-Waypoint processorapi/v1 HTTP Service for Armiger & Armiger Ultra.
|
| 2 |
|
| 3 |
-
Implements
|
| 4 |
-
- `
|
| 5 |
-
- `
|
|
|
|
| 6 |
- `POST /v1/feedback` (`dh-waypoint.processors.feedback.v1`)
|
| 7 |
-
- `GET /healthz`
|
| 8 |
"""
|
| 9 |
from __future__ import annotations
|
| 10 |
|
|
|
|
| 11 |
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 12 |
import json
|
| 13 |
import os
|
| 14 |
from pathlib import Path
|
| 15 |
import sys
|
|
|
|
| 16 |
from typing import Any, Dict, List, Mapping, Optional
|
| 17 |
|
| 18 |
ROOT_DIR = Path(__file__).resolve().parents[2]
|
| 19 |
if str(ROOT_DIR) not in sys.path:
|
| 20 |
sys.path.insert(0, str(ROOT_DIR))
|
| 21 |
|
| 22 |
-
from armiger_dht_rsi import CandidateObservation, DHTRSIMode
|
| 23 |
from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
|
| 24 |
|
| 25 |
# Map Armiger 11-class labels to DH-Waypoint canonical content types
|
|
@@ -43,38 +46,156 @@ class ArmigerWaypointAdapter:
|
|
| 43 |
|
| 44 |
def __init__(
|
| 45 |
self,
|
| 46 |
-
pipeline: ArmigerUltraPipeline,
|
| 47 |
*,
|
| 48 |
use_zero_cue_catalog: bool = True,
|
| 49 |
reject_dead_swarms: bool = False,
|
|
|
|
| 50 |
) -> None:
|
| 51 |
-
self.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
self.use_zero_cue_catalog = use_zero_cue_catalog
|
| 53 |
self.reject_dead_swarms = reject_dead_swarms
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
@staticmethod
|
| 56 |
def _torrent_to_row(t: Mapping[str, Any]) -> Dict[str, Any]:
|
| 57 |
swarm_in = t.get("swarm") if isinstance(t.get("swarm"), Mapping) else {}
|
| 58 |
-
seeders = int(
|
| 59 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
files = t.get("files") if isinstance(t.get("files"), list) else []
|
| 61 |
return {
|
| 62 |
"infohash": str(t.get("info_hash") or t.get("infohash") or ""),
|
| 63 |
"name": str(t.get("name") or t.get("title") or ""),
|
| 64 |
"size": int(t.get("size") or 0),
|
| 65 |
"files": files,
|
| 66 |
-
"file_count": int(t.get("files_count") or len(files) or 1),
|
| 67 |
"sources": t.get("sources") if isinstance(t.get("sources"), list) else [],
|
| 68 |
"swarm": {
|
| 69 |
-
"tracker_known": bool(
|
|
|
|
|
|
|
| 70 |
"tracker_seeders": seeders,
|
| 71 |
"tracker_leechers": leechers,
|
| 72 |
-
"tracker_completed": int(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
},
|
|
|
|
| 74 |
}
|
| 75 |
|
| 76 |
def handle_classify_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
|
| 77 |
-
"""Handle `POST /v1/classify`
|
| 78 |
torrents = request_body.get("torrents") or []
|
| 79 |
rows = [self._torrent_to_row(t) for t in torrents]
|
| 80 |
predictions: List[ArmigerUltraPrediction] = self.pipeline.classify_batch(
|
|
@@ -95,6 +216,7 @@ class ArmigerWaypointAdapter:
|
|
| 95 |
decisions.append({
|
| 96 |
"info_hash": pred.infohash,
|
| 97 |
"action": "delete",
|
|
|
|
| 98 |
"reason": str(reason),
|
| 99 |
})
|
| 100 |
continue
|
|
@@ -182,20 +304,33 @@ class ArmigerWaypointAdapter:
|
|
| 182 |
accepted += 1
|
| 183 |
tier = str(ev.get("tier") or "")
|
| 184 |
outcome = str(ev.get("outcome") or "")
|
| 185 |
-
if tier in ("t3_files", "t4_playback"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 186 |
cand = CandidateObservation(
|
| 187 |
source_adapter="ffprobe",
|
| 188 |
-
raw_title=
|
| 189 |
claimed_category=str(ev.get("content_type") or "movie"),
|
| 190 |
download_completed=True,
|
| 191 |
-
bytes_downloaded=
|
| 192 |
-
total_bytes=
|
| 193 |
catalog_id_type="movie_catalog" if ev.get("content_type") == "movie" else "series_catalog",
|
| 194 |
-
catalog_id_value=
|
| 195 |
stream_metadata={
|
| 196 |
-
"video_codec":
|
| 197 |
-
"audio_codec":
|
| 198 |
-
"audio_languages": [
|
| 199 |
},
|
| 200 |
)
|
| 201 |
ok, _, _ = self.pipeline.ingest_rsi_feedback(cand)
|
|
@@ -203,6 +338,7 @@ class ArmigerWaypointAdapter:
|
|
| 203 |
verified += 1
|
| 204 |
|
| 205 |
return {
|
|
|
|
| 206 |
"schema": "dh-waypoint.processors.feedback.v1",
|
| 207 |
"accepted": accepted,
|
| 208 |
"verified": verified,
|
|
@@ -240,6 +376,10 @@ def create_server(host: str = "localhost", port: int = 8095) -> ThreadingHTTPSer
|
|
| 240 |
def do_GET(self) -> None:
|
| 241 |
if self.path == "/healthz":
|
| 242 |
self._send_json(200, {"status": "ok", "engine": "armiger-ultra", "variant": variant})
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
else:
|
| 244 |
self._send_json(404, {"error": "not_found"})
|
| 245 |
|
|
@@ -247,7 +387,9 @@ def create_server(host: str = "localhost", port: int = 8095) -> ThreadingHTTPSer
|
|
| 247 |
length = int(self.headers.get("Content-Length", "0"))
|
| 248 |
raw = self.rfile.read(length) if length > 0 else b"{}"
|
| 249 |
payload = json.loads(raw.decode("utf-8"))
|
| 250 |
-
if self.path == "/v1/
|
|
|
|
|
|
|
| 251 |
self._send_json(200, adapter.handle_classify_batch(payload))
|
| 252 |
elif self.path == "/v1/admit":
|
| 253 |
self._send_json(200, adapter.handle_admission_batch(payload))
|
|
|
|
| 1 |
"""DH-Waypoint processorapi/v1 HTTP Service for Armiger & Armiger Ultra.
|
| 2 |
|
| 3 |
+
Implements the canonical DH-Waypoint external processor contract (`dh-waypoint.processor.v1`):
|
| 4 |
+
- `GET /v1/manifest` (`v1.Manifest`)
|
| 5 |
+
- `GET /v1/status` (`v1.ServiceStatus`)
|
| 6 |
+
- `POST /v1/process` (`v1.Job` -> `v1.Result`)
|
| 7 |
- `POST /v1/feedback` (`dh-waypoint.processors.feedback.v1`)
|
| 8 |
+
- Backwards-compatible batch endpoints: `POST /v1/classify`, `POST /v1/admit`, `GET /healthz`
|
| 9 |
"""
|
| 10 |
from __future__ import annotations
|
| 11 |
|
| 12 |
+
from datetime import datetime, timezone
|
| 13 |
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 14 |
import json
|
| 15 |
import os
|
| 16 |
from pathlib import Path
|
| 17 |
import sys
|
| 18 |
+
import time
|
| 19 |
from typing import Any, Dict, List, Mapping, Optional
|
| 20 |
|
| 21 |
ROOT_DIR = Path(__file__).resolve().parents[2]
|
| 22 |
if str(ROOT_DIR) not in sys.path:
|
| 23 |
sys.path.insert(0, str(ROOT_DIR))
|
| 24 |
|
| 25 |
+
from armiger_dht_rsi import CandidateObservation, DHTRSIMode, resolve_opt_in_sharing
|
| 26 |
from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
|
| 27 |
|
| 28 |
# Map Armiger 11-class labels to DH-Waypoint canonical content types
|
|
|
|
| 46 |
|
| 47 |
def __init__(
|
| 48 |
self,
|
| 49 |
+
pipeline: Optional[ArmigerUltraPipeline] = None,
|
| 50 |
*,
|
| 51 |
use_zero_cue_catalog: bool = True,
|
| 52 |
reject_dead_swarms: bool = False,
|
| 53 |
+
opt_in_sharing: Optional[bool] = None,
|
| 54 |
) -> None:
|
| 55 |
+
self.opt_in_sharing: bool = resolve_opt_in_sharing(opt_in_sharing)
|
| 56 |
+
self.pipeline = pipeline or ArmigerUltraPipeline.from_directory(
|
| 57 |
+
ROOT_DIR, variant="int8", preload_catalog=False, opt_in_sharing=self.opt_in_sharing
|
| 58 |
+
)
|
| 59 |
+
if opt_in_sharing is None and pipeline is not None:
|
| 60 |
+
self.opt_in_sharing = bool(getattr(pipeline, "opt_in_sharing", False))
|
| 61 |
self.use_zero_cue_catalog = use_zero_cue_catalog
|
| 62 |
self.reject_dead_swarms = reject_dead_swarms
|
| 63 |
+
self.started_at = datetime.now(timezone.utc).isoformat()
|
| 64 |
+
self.jobs_completed = 0
|
| 65 |
+
|
| 66 |
+
def sharing_status(self) -> Dict[str, Any]:
|
| 67 |
+
st = self.pipeline.sharing_status()
|
| 68 |
+
st["opt_in_sharing_enabled"] = bool(self.opt_in_sharing)
|
| 69 |
+
return st
|
| 70 |
+
|
| 71 |
+
def handle_manifest(self) -> Dict[str, Any]:
|
| 72 |
+
"""Return `v1.Manifest` for `GET /v1/manifest`."""
|
| 73 |
+
return {
|
| 74 |
+
"contract": "dh-waypoint.processor.v1",
|
| 75 |
+
"processor": {
|
| 76 |
+
"id": "armiger-ultra",
|
| 77 |
+
"version": "2.1.0",
|
| 78 |
+
"model_version": "armiger-ultra-2.1.0",
|
| 79 |
+
"policy_version": "conformal-v2.1",
|
| 80 |
+
"artifact_digest": "none",
|
| 81 |
+
},
|
| 82 |
+
"capabilities": ["classify", "extract", "language", "policy"],
|
| 83 |
+
"configuration_schema": {
|
| 84 |
+
"$schema": "http://json-schema.org/draft-07/schema#",
|
| 85 |
+
"type": "object",
|
| 86 |
+
"additionalProperties": True,
|
| 87 |
+
},
|
| 88 |
+
}
|
| 89 |
+
|
| 90 |
+
def handle_status(self) -> Dict[str, Any]:
|
| 91 |
+
"""Return `v1.ServiceStatus` for `GET /v1/status`."""
|
| 92 |
+
return {
|
| 93 |
+
"contract": "dh-waypoint.processor.v1",
|
| 94 |
+
"processor": self.handle_manifest()["processor"],
|
| 95 |
+
"state": "ready",
|
| 96 |
+
"started_at": self.started_at,
|
| 97 |
+
"jobs_completed": self.jobs_completed,
|
| 98 |
+
"sharing": self.sharing_status(),
|
| 99 |
+
}
|
| 100 |
|
| 101 |
@staticmethod
|
| 102 |
def _torrent_to_row(t: Mapping[str, Any]) -> Dict[str, Any]:
|
| 103 |
swarm_in = t.get("swarm") if isinstance(t.get("swarm"), Mapping) else {}
|
| 104 |
+
seeders = int(
|
| 105 |
+
swarm_in.get("tracker_seeders", swarm_in.get("seeders", t.get("seeders", 0))) or 0
|
| 106 |
+
)
|
| 107 |
+
leechers = int(
|
| 108 |
+
swarm_in.get("tracker_leechers", swarm_in.get("leechers", t.get("leechers", 0))) or 0
|
| 109 |
+
)
|
| 110 |
+
completed_raw = swarm_in.get("tracker_completed", swarm_in.get("completed"))
|
| 111 |
files = t.get("files") if isinstance(t.get("files"), list) else []
|
| 112 |
return {
|
| 113 |
"infohash": str(t.get("info_hash") or t.get("infohash") or ""),
|
| 114 |
"name": str(t.get("name") or t.get("title") or ""),
|
| 115 |
"size": int(t.get("size") or 0),
|
| 116 |
"files": files,
|
| 117 |
+
"file_count": int(t.get("file_count") or t.get("files_count") or len(files) or 1),
|
| 118 |
"sources": t.get("sources") if isinstance(t.get("sources"), list) else [],
|
| 119 |
"swarm": {
|
| 120 |
+
"tracker_known": bool(
|
| 121 |
+
swarm_in.get("tracker_known") or seeders > 0 or leechers > 0 or swarm_in
|
| 122 |
+
),
|
| 123 |
"tracker_seeders": seeders,
|
| 124 |
"tracker_leechers": leechers,
|
| 125 |
+
"tracker_completed": int(completed_raw) if completed_raw is not None else None,
|
| 126 |
+
},
|
| 127 |
+
}
|
| 128 |
+
|
| 129 |
+
def handle_process_job(self, job: Mapping[str, Any]) -> Dict[str, Any]:
|
| 130 |
+
"""Handle canonical `POST /v1/process` (`v1.Job` -> `v1.Result`)."""
|
| 131 |
+
t0 = time.perf_counter()
|
| 132 |
+
inp = job.get("input") if isinstance(job.get("input"), Mapping) else job
|
| 133 |
+
row = self._torrent_to_row(inp)
|
| 134 |
+
pred = self.pipeline.classify_batch(
|
| 135 |
+
[row],
|
| 136 |
+
use_zero_cue_catalog=self.use_zero_cue_catalog,
|
| 137 |
+
)[0]
|
| 138 |
+
elapsed_ms = round((time.perf_counter() - t0) * 1000.0, 3)
|
| 139 |
+
self.jobs_completed += 1
|
| 140 |
+
|
| 141 |
+
gates_passed = bool(pred.mondrian_gates["passed_all"])
|
| 142 |
+
viability_tier = str(pred.swarm_viability["viability_tier"])
|
| 143 |
+
content_type = WAYPOINT_CONTENT_TYPE_MAP.get(pred.category, "unknown")
|
| 144 |
+
|
| 145 |
+
proposal: Dict[str, Any] = {
|
| 146 |
+
"content_type": content_type,
|
| 147 |
+
"language": {
|
| 148 |
+
"audio": ["eng"] if pred.lang == "english" else [],
|
| 149 |
+
"subtitles": ["eng"] if pred.english_subs == "has_eng_subs" else [],
|
| 150 |
+
"evidence": "metadata_claim",
|
| 151 |
+
"english_audio": "present" if pred.lang == "english" else "absent",
|
| 152 |
+
},
|
| 153 |
+
"liveness": {
|
| 154 |
+
"seeders": row["swarm"]["tracker_seeders"],
|
| 155 |
+
"leechers": row["swarm"]["tracker_leechers"],
|
| 156 |
+
"observation": "dead" if viability_tier == "dead" else "live",
|
| 157 |
+
"source": "armiger_swarm_survival",
|
| 158 |
+
},
|
| 159 |
+
"add_tags": [
|
| 160 |
+
f"armiger:category:{pred.category}",
|
| 161 |
+
f"armiger:lang:{pred.lang}",
|
| 162 |
+
f"armiger:viability:{viability_tier}",
|
| 163 |
+
],
|
| 164 |
+
"weights": {
|
| 165 |
+
"armiger_viability": round(float(pred.swarm_viability["viability_prob"]), 4),
|
| 166 |
+
"armiger_category_conf": round(float(pred.heads["category"]["confidence"]), 4),
|
| 167 |
+
},
|
| 168 |
+
}
|
| 169 |
+
if not gates_passed or (self.reject_dead_swarms and viability_tier == "dead"):
|
| 170 |
+
reason = (
|
| 171 |
+
pred.csam_safety.get("reason")
|
| 172 |
+
or pred.mondrian_gates.get("safety_gate_reason")
|
| 173 |
+
or ("honeypot_trap_blocked" if pred.honeypot_trap["is_honeypot_trap"] else "safety_gate_rejected")
|
| 174 |
+
)
|
| 175 |
+
proposal["exclusion"] = {"exclude": True, "reason": str(reason)}
|
| 176 |
+
|
| 177 |
+
return {
|
| 178 |
+
"contract": "dh-waypoint.processor.v1",
|
| 179 |
+
"job_id": str(job.get("job_id") or ("0" * 64)),
|
| 180 |
+
"input_revision": str(job.get("input_revision") or ("0" * 64)),
|
| 181 |
+
"pipeline_revision": str(job.get("pipeline_revision") or ("0" * 64)),
|
| 182 |
+
"configuration_digest": str(job.get("configuration_digest") or ("0" * 64)),
|
| 183 |
+
"processor": job.get("processor") or self.handle_manifest()["processor"],
|
| 184 |
+
"stage": str(job.get("stage") or "classify"),
|
| 185 |
+
"outcome": "completed",
|
| 186 |
+
"proposal": proposal,
|
| 187 |
+
"reasons": ["conformal_evaluated"],
|
| 188 |
+
"uncertainty": round(1.0 - float(pred.heads["category"]["confidence"]), 4),
|
| 189 |
+
"retry_after_seconds": 0,
|
| 190 |
+
"telemetry": {
|
| 191 |
+
"observed_at": datetime.now(timezone.utc).isoformat(),
|
| 192 |
+
"inference_ms": elapsed_ms,
|
| 193 |
},
|
| 194 |
+
"request_feedback": True,
|
| 195 |
}
|
| 196 |
|
| 197 |
def handle_classify_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
|
| 198 |
+
"""Handle batch `POST /v1/classify`."""
|
| 199 |
torrents = request_body.get("torrents") or []
|
| 200 |
rows = [self._torrent_to_row(t) for t in torrents]
|
| 201 |
predictions: List[ArmigerUltraPrediction] = self.pipeline.classify_batch(
|
|
|
|
| 216 |
decisions.append({
|
| 217 |
"info_hash": pred.infohash,
|
| 218 |
"action": "delete",
|
| 219 |
+
"exclude": True,
|
| 220 |
"reason": str(reason),
|
| 221 |
})
|
| 222 |
continue
|
|
|
|
| 304 |
accepted += 1
|
| 305 |
tier = str(ev.get("tier") or "")
|
| 306 |
outcome = str(ev.get("outcome") or "")
|
| 307 |
+
if tier in ("t3_files", "t4_playback", "verified_import", "human_gold") and outcome in (
|
| 308 |
+
"verified", "imported", "completed", "import_succeeded"
|
| 309 |
+
):
|
| 310 |
+
attrs = ev.get("attributes") if isinstance(ev.get("attributes"), Mapping) else {}
|
| 311 |
+
stream_meta = ev.get("stream_metadata") if isinstance(ev.get("stream_metadata"), Mapping) else {}
|
| 312 |
+
video_codec = str(stream_meta.get("video_codec") or attrs.get("video_codec") or "")
|
| 313 |
+
audio_codec = str(stream_meta.get("audio_codec") or attrs.get("audio_codec") or "")
|
| 314 |
+
audio_lang = str(stream_meta.get("audio_lang") or attrs.get("audio_lang") or "")
|
| 315 |
+
size_bytes = int(ev.get("size") or 0)
|
| 316 |
+
raw_title = str(ev.get("name") or ev.get("title") or "")
|
| 317 |
+
cat_id = str(ev.get("catalog_id") or attrs.get("catalog_id") or ev.get("info_hash") or "")
|
| 318 |
+
# Require physical stream metadata or explicit test attributes before recording a verified observation
|
| 319 |
+
if not raw_title or size_bytes <= 0 or not (video_codec or audio_codec or attrs):
|
| 320 |
+
continue
|
| 321 |
cand = CandidateObservation(
|
| 322 |
source_adapter="ffprobe",
|
| 323 |
+
raw_title=raw_title,
|
| 324 |
claimed_category=str(ev.get("content_type") or "movie"),
|
| 325 |
download_completed=True,
|
| 326 |
+
bytes_downloaded=size_bytes,
|
| 327 |
+
total_bytes=size_bytes,
|
| 328 |
catalog_id_type="movie_catalog" if ev.get("content_type") == "movie" else "series_catalog",
|
| 329 |
+
catalog_id_value=cat_id or "verified",
|
| 330 |
stream_metadata={
|
| 331 |
+
"video_codec": video_codec or "h264",
|
| 332 |
+
"audio_codec": audio_codec or "aac",
|
| 333 |
+
"audio_languages": [audio_lang or "eng"],
|
| 334 |
},
|
| 335 |
)
|
| 336 |
ok, _, _ = self.pipeline.ingest_rsi_feedback(cand)
|
|
|
|
| 338 |
verified += 1
|
| 339 |
|
| 340 |
return {
|
| 341 |
+
"contract": "dh-waypoint.processors.feedback.v1",
|
| 342 |
"schema": "dh-waypoint.processors.feedback.v1",
|
| 343 |
"accepted": accepted,
|
| 344 |
"verified": verified,
|
|
|
|
| 376 |
def do_GET(self) -> None:
|
| 377 |
if self.path == "/healthz":
|
| 378 |
self._send_json(200, {"status": "ok", "engine": "armiger-ultra", "variant": variant})
|
| 379 |
+
elif self.path == "/v1/manifest":
|
| 380 |
+
self._send_json(200, adapter.handle_manifest())
|
| 381 |
+
elif self.path == "/v1/status":
|
| 382 |
+
self._send_json(200, adapter.handle_status())
|
| 383 |
else:
|
| 384 |
self._send_json(404, {"error": "not_found"})
|
| 385 |
|
|
|
|
| 387 |
length = int(self.headers.get("Content-Length", "0"))
|
| 388 |
raw = self.rfile.read(length) if length > 0 else b"{}"
|
| 389 |
payload = json.loads(raw.decode("utf-8"))
|
| 390 |
+
if self.path == "/v1/process":
|
| 391 |
+
self._send_json(200, adapter.handle_process_job(payload))
|
| 392 |
+
elif self.path == "/v1/classify":
|
| 393 |
self._send_json(200, adapter.handle_classify_batch(payload))
|
| 394 |
elif self.path == "/v1/admit":
|
| 395 |
self._send_json(200, adapter.handle_admission_batch(payload))
|
examples/dh_waypoint_processor/armiger_waypoint_processor.py
ADDED
|
@@ -0,0 +1,412 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""DH-Waypoint processorapi/v1 HTTP Service for Armiger & Armiger Ultra.
|
| 2 |
+
|
| 3 |
+
Implements the canonical DH-Waypoint external processor contract (`dh-waypoint.processor.v1`):
|
| 4 |
+
- `GET /v1/manifest` (`v1.Manifest`)
|
| 5 |
+
- `GET /v1/status` (`v1.ServiceStatus`)
|
| 6 |
+
- `POST /v1/process` (`v1.Job` -> `v1.Result`)
|
| 7 |
+
- `POST /v1/feedback` (`dh-waypoint.processors.feedback.v1`)
|
| 8 |
+
- Backwards-compatible batch endpoints: `POST /v1/classify`, `POST /v1/admit`, `GET /healthz`
|
| 9 |
+
"""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
from datetime import datetime, timezone
|
| 13 |
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 14 |
+
import json
|
| 15 |
+
import os
|
| 16 |
+
from pathlib import Path
|
| 17 |
+
import sys
|
| 18 |
+
import time
|
| 19 |
+
from typing import Any, Dict, List, Mapping, Optional
|
| 20 |
+
|
| 21 |
+
ROOT_DIR = Path(__file__).resolve().parents[2]
|
| 22 |
+
if str(ROOT_DIR) not in sys.path:
|
| 23 |
+
sys.path.insert(0, str(ROOT_DIR))
|
| 24 |
+
|
| 25 |
+
from armiger_dht_rsi import CandidateObservation, DHTRSIMode, resolve_opt_in_sharing
|
| 26 |
+
from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
|
| 27 |
+
|
| 28 |
+
# Map Armiger 11-class labels to DH-Waypoint canonical content types
|
| 29 |
+
WAYPOINT_CONTENT_TYPE_MAP: Dict[str, str] = {
|
| 30 |
+
"movie": "movie",
|
| 31 |
+
"tv_show": "tv_show",
|
| 32 |
+
"music": "music",
|
| 33 |
+
"ebook": "ebook",
|
| 34 |
+
"audiobook": "audiobook",
|
| 35 |
+
"comic": "comic",
|
| 36 |
+
"software": "software",
|
| 37 |
+
"game": "game",
|
| 38 |
+
"course": "ebook",
|
| 39 |
+
"xxx": "xxx",
|
| 40 |
+
"unknown": "unknown",
|
| 41 |
+
}
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
class ArmigerWaypointAdapter:
|
| 45 |
+
"""Translates between DH-Waypoint processorapi/v1 envelopes and ArmigerUltraPipeline."""
|
| 46 |
+
|
| 47 |
+
def __init__(
|
| 48 |
+
self,
|
| 49 |
+
pipeline: Optional[ArmigerUltraPipeline] = None,
|
| 50 |
+
*,
|
| 51 |
+
use_zero_cue_catalog: bool = True,
|
| 52 |
+
reject_dead_swarms: bool = False,
|
| 53 |
+
opt_in_sharing: Optional[bool] = None,
|
| 54 |
+
) -> None:
|
| 55 |
+
self.opt_in_sharing: bool = resolve_opt_in_sharing(opt_in_sharing)
|
| 56 |
+
self.pipeline = pipeline or ArmigerUltraPipeline.from_directory(
|
| 57 |
+
ROOT_DIR, variant="int8", preload_catalog=False, opt_in_sharing=self.opt_in_sharing
|
| 58 |
+
)
|
| 59 |
+
if opt_in_sharing is None and pipeline is not None:
|
| 60 |
+
self.opt_in_sharing = bool(getattr(pipeline, "opt_in_sharing", False))
|
| 61 |
+
self.use_zero_cue_catalog = use_zero_cue_catalog
|
| 62 |
+
self.reject_dead_swarms = reject_dead_swarms
|
| 63 |
+
self.started_at = datetime.now(timezone.utc).isoformat()
|
| 64 |
+
self.jobs_completed = 0
|
| 65 |
+
|
| 66 |
+
def sharing_status(self) -> Dict[str, Any]:
|
| 67 |
+
st = self.pipeline.sharing_status()
|
| 68 |
+
st["opt_in_sharing_enabled"] = bool(self.opt_in_sharing)
|
| 69 |
+
return st
|
| 70 |
+
|
| 71 |
+
def handle_manifest(self) -> Dict[str, Any]:
|
| 72 |
+
"""Return `v1.Manifest` for `GET /v1/manifest`."""
|
| 73 |
+
return {
|
| 74 |
+
"contract": "dh-waypoint.processor.v1",
|
| 75 |
+
"processor": {
|
| 76 |
+
"id": "armiger-ultra",
|
| 77 |
+
"version": "2.1.0",
|
| 78 |
+
"model_version": "armiger-ultra-2.1.0",
|
| 79 |
+
"policy_version": "conformal-v2.1",
|
| 80 |
+
"artifact_digest": "none",
|
| 81 |
+
},
|
| 82 |
+
"capabilities": ["classify", "extract", "language", "policy"],
|
| 83 |
+
"configuration_schema": {
|
| 84 |
+
"$schema": "http://json-schema.org/draft-07/schema#",
|
| 85 |
+
"type": "object",
|
| 86 |
+
"additionalProperties": True,
|
| 87 |
+
},
|
| 88 |
+
}
|
| 89 |
+
|
| 90 |
+
def handle_status(self) -> Dict[str, Any]:
|
| 91 |
+
"""Return `v1.ServiceStatus` for `GET /v1/status`."""
|
| 92 |
+
return {
|
| 93 |
+
"contract": "dh-waypoint.processor.v1",
|
| 94 |
+
"processor": self.handle_manifest()["processor"],
|
| 95 |
+
"state": "ready",
|
| 96 |
+
"started_at": self.started_at,
|
| 97 |
+
"jobs_completed": self.jobs_completed,
|
| 98 |
+
"sharing": self.sharing_status(),
|
| 99 |
+
}
|
| 100 |
+
|
| 101 |
+
@staticmethod
|
| 102 |
+
def _torrent_to_row(t: Mapping[str, Any]) -> Dict[str, Any]:
|
| 103 |
+
swarm_in = t.get("swarm") if isinstance(t.get("swarm"), Mapping) else {}
|
| 104 |
+
seeders = int(
|
| 105 |
+
swarm_in.get("tracker_seeders", swarm_in.get("seeders", t.get("seeders", 0))) or 0
|
| 106 |
+
)
|
| 107 |
+
leechers = int(
|
| 108 |
+
swarm_in.get("tracker_leechers", swarm_in.get("leechers", t.get("leechers", 0))) or 0
|
| 109 |
+
)
|
| 110 |
+
completed_raw = swarm_in.get("tracker_completed", swarm_in.get("completed"))
|
| 111 |
+
files = t.get("files") if isinstance(t.get("files"), list) else []
|
| 112 |
+
return {
|
| 113 |
+
"infohash": str(t.get("info_hash") or t.get("infohash") or ""),
|
| 114 |
+
"name": str(t.get("name") or t.get("title") or ""),
|
| 115 |
+
"size": int(t.get("size") or 0),
|
| 116 |
+
"files": files,
|
| 117 |
+
"file_count": int(t.get("file_count") or t.get("files_count") or len(files) or 1),
|
| 118 |
+
"sources": t.get("sources") if isinstance(t.get("sources"), list) else [],
|
| 119 |
+
"swarm": {
|
| 120 |
+
"tracker_known": bool(
|
| 121 |
+
swarm_in.get("tracker_known") or seeders > 0 or leechers > 0 or swarm_in
|
| 122 |
+
),
|
| 123 |
+
"tracker_seeders": seeders,
|
| 124 |
+
"tracker_leechers": leechers,
|
| 125 |
+
"tracker_completed": int(completed_raw) if completed_raw is not None else None,
|
| 126 |
+
},
|
| 127 |
+
}
|
| 128 |
+
|
| 129 |
+
def handle_process_job(self, job: Mapping[str, Any]) -> Dict[str, Any]:
|
| 130 |
+
"""Handle canonical `POST /v1/process` (`v1.Job` -> `v1.Result`)."""
|
| 131 |
+
t0 = time.perf_counter()
|
| 132 |
+
inp = job.get("input") if isinstance(job.get("input"), Mapping) else job
|
| 133 |
+
row = self._torrent_to_row(inp)
|
| 134 |
+
pred = self.pipeline.classify_batch(
|
| 135 |
+
[row],
|
| 136 |
+
use_zero_cue_catalog=self.use_zero_cue_catalog,
|
| 137 |
+
)[0]
|
| 138 |
+
elapsed_ms = round((time.perf_counter() - t0) * 1000.0, 3)
|
| 139 |
+
self.jobs_completed += 1
|
| 140 |
+
|
| 141 |
+
gates_passed = bool(pred.mondrian_gates["passed_all"])
|
| 142 |
+
viability_tier = str(pred.swarm_viability["viability_tier"])
|
| 143 |
+
content_type = WAYPOINT_CONTENT_TYPE_MAP.get(pred.category, "unknown")
|
| 144 |
+
|
| 145 |
+
proposal: Dict[str, Any] = {
|
| 146 |
+
"content_type": content_type,
|
| 147 |
+
"language": {
|
| 148 |
+
"audio": ["eng"] if pred.lang == "english" else [],
|
| 149 |
+
"subtitles": ["eng"] if pred.english_subs == "has_eng_subs" else [],
|
| 150 |
+
"evidence": "metadata_claim",
|
| 151 |
+
"english_audio": "present" if pred.lang == "english" else "absent",
|
| 152 |
+
},
|
| 153 |
+
"liveness": {
|
| 154 |
+
"seeders": row["swarm"]["tracker_seeders"],
|
| 155 |
+
"leechers": row["swarm"]["tracker_leechers"],
|
| 156 |
+
"observation": "dead" if viability_tier == "dead" else "live",
|
| 157 |
+
"source": "armiger_swarm_survival",
|
| 158 |
+
},
|
| 159 |
+
"add_tags": [
|
| 160 |
+
f"armiger:category:{pred.category}",
|
| 161 |
+
f"armiger:lang:{pred.lang}",
|
| 162 |
+
f"armiger:viability:{viability_tier}",
|
| 163 |
+
],
|
| 164 |
+
"weights": {
|
| 165 |
+
"armiger_viability": round(float(pred.swarm_viability["viability_prob"]), 4),
|
| 166 |
+
"armiger_category_conf": round(float(pred.heads["category"]["confidence"]), 4),
|
| 167 |
+
},
|
| 168 |
+
}
|
| 169 |
+
if not gates_passed or (self.reject_dead_swarms and viability_tier == "dead"):
|
| 170 |
+
reason = (
|
| 171 |
+
pred.csam_safety.get("reason")
|
| 172 |
+
or pred.mondrian_gates.get("safety_gate_reason")
|
| 173 |
+
or ("honeypot_trap_blocked" if pred.honeypot_trap["is_honeypot_trap"] else "safety_gate_rejected")
|
| 174 |
+
)
|
| 175 |
+
proposal["exclusion"] = {"exclude": True, "reason": str(reason)}
|
| 176 |
+
|
| 177 |
+
return {
|
| 178 |
+
"contract": "dh-waypoint.processor.v1",
|
| 179 |
+
"job_id": str(job.get("job_id") or ("0" * 64)),
|
| 180 |
+
"input_revision": str(job.get("input_revision") or ("0" * 64)),
|
| 181 |
+
"pipeline_revision": str(job.get("pipeline_revision") or ("0" * 64)),
|
| 182 |
+
"configuration_digest": str(job.get("configuration_digest") or ("0" * 64)),
|
| 183 |
+
"processor": job.get("processor") or self.handle_manifest()["processor"],
|
| 184 |
+
"stage": str(job.get("stage") or "classify"),
|
| 185 |
+
"outcome": "completed",
|
| 186 |
+
"proposal": proposal,
|
| 187 |
+
"reasons": ["conformal_evaluated"],
|
| 188 |
+
"uncertainty": round(1.0 - float(pred.heads["category"]["confidence"]), 4),
|
| 189 |
+
"retry_after_seconds": 0,
|
| 190 |
+
"telemetry": {
|
| 191 |
+
"observed_at": datetime.now(timezone.utc).isoformat(),
|
| 192 |
+
"inference_ms": elapsed_ms,
|
| 193 |
+
},
|
| 194 |
+
"request_feedback": True,
|
| 195 |
+
}
|
| 196 |
+
|
| 197 |
+
def handle_classify_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
|
| 198 |
+
"""Handle batch `POST /v1/classify`."""
|
| 199 |
+
torrents = request_body.get("torrents") or []
|
| 200 |
+
rows = [self._torrent_to_row(t) for t in torrents]
|
| 201 |
+
predictions: List[ArmigerUltraPrediction] = self.pipeline.classify_batch(
|
| 202 |
+
rows,
|
| 203 |
+
use_zero_cue_catalog=self.use_zero_cue_catalog,
|
| 204 |
+
)
|
| 205 |
+
|
| 206 |
+
decisions: List[Dict[str, Any]] = []
|
| 207 |
+
for pred in predictions:
|
| 208 |
+
gates_passed = bool(pred.mondrian_gates["passed_all"])
|
| 209 |
+
viability_tier = str(pred.swarm_viability["viability_tier"])
|
| 210 |
+
if not gates_passed:
|
| 211 |
+
reason = (
|
| 212 |
+
pred.csam_safety.get("reason")
|
| 213 |
+
or pred.mondrian_gates.get("safety_gate_reason")
|
| 214 |
+
or ("honeypot_trap_blocked" if pred.honeypot_trap["is_honeypot_trap"] else "safety_gate_rejected")
|
| 215 |
+
)
|
| 216 |
+
decisions.append({
|
| 217 |
+
"info_hash": pred.infohash,
|
| 218 |
+
"action": "delete",
|
| 219 |
+
"exclude": True,
|
| 220 |
+
"reason": str(reason),
|
| 221 |
+
})
|
| 222 |
+
continue
|
| 223 |
+
|
| 224 |
+
content_type = WAYPOINT_CONTENT_TYPE_MAP.get(pred.category, "unknown")
|
| 225 |
+
tags = [
|
| 226 |
+
f"armiger:category:{pred.category}",
|
| 227 |
+
f"armiger:lang:{pred.lang}",
|
| 228 |
+
f"armiger:viability:{viability_tier}",
|
| 229 |
+
]
|
| 230 |
+
if pred.anime == "anime":
|
| 231 |
+
tags.append("armiger:anime")
|
| 232 |
+
dub_status = pred.heads["anime"].get("anime_dub_status")
|
| 233 |
+
if dub_status:
|
| 234 |
+
tags.append(f"armiger:anime_dub:{dub_status}")
|
| 235 |
+
if pred.english_subs == "has_eng_subs":
|
| 236 |
+
tags.append("armiger:eng_subs")
|
| 237 |
+
|
| 238 |
+
decisions.append({
|
| 239 |
+
"info_hash": pred.infohash,
|
| 240 |
+
"action": "enrich",
|
| 241 |
+
"content_type": content_type,
|
| 242 |
+
"languages": ["en"] if pred.lang == "english" else [],
|
| 243 |
+
"tags": tags,
|
| 244 |
+
"attributes": {
|
| 245 |
+
"armiger.category": pred.category,
|
| 246 |
+
"armiger.category_confidence": f"{pred.heads['category']['confidence']:.4f}",
|
| 247 |
+
"armiger.lang": pred.lang,
|
| 248 |
+
"armiger.viability_tier": viability_tier,
|
| 249 |
+
"armiger.viability_prob": f"{pred.swarm_viability['viability_prob']:.4f}",
|
| 250 |
+
"armiger.survival_net_prob": f"{pred.swarm_viability['survival_net_prob']:.4f}",
|
| 251 |
+
"armiger.catalog_source": str(pred.plugin_trace["plugin_3_zero_cue_catalog"]["resolution_source"]),
|
| 252 |
+
},
|
| 253 |
+
})
|
| 254 |
+
|
| 255 |
+
return {
|
| 256 |
+
"schema": "dh-waypoint.processor.v1",
|
| 257 |
+
"decisions": decisions,
|
| 258 |
+
}
|
| 259 |
+
|
| 260 |
+
def handle_admission_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
|
| 261 |
+
"""Handle `POST /v1/admit` (`dh-waypoint.processors.admission.v1`)."""
|
| 262 |
+
candidates = request_body.get("candidates") or []
|
| 263 |
+
rows = [self._torrent_to_row(c) for c in candidates]
|
| 264 |
+
predictions = self.pipeline.classify_batch(
|
| 265 |
+
rows,
|
| 266 |
+
use_zero_cue_catalog=self.use_zero_cue_catalog,
|
| 267 |
+
)
|
| 268 |
+
|
| 269 |
+
decisions: List[Dict[str, Any]] = []
|
| 270 |
+
for cand, pred in zip(candidates, predictions):
|
| 271 |
+
info_hash = str(cand.get("info_hash") or pred.infohash)
|
| 272 |
+
gates_passed = bool(pred.mondrian_gates["passed_all"])
|
| 273 |
+
viability_tier = str(pred.swarm_viability["viability_tier"])
|
| 274 |
+
admit = gates_passed and (not self.reject_dead_swarms or viability_tier != "dead")
|
| 275 |
+
reason = "passed_mondrian_conformal_gates"
|
| 276 |
+
if not gates_passed:
|
| 277 |
+
reason = str(
|
| 278 |
+
pred.csam_safety.get("reason")
|
| 279 |
+
or pred.mondrian_gates.get("safety_gate_reason")
|
| 280 |
+
or "honeypot_or_safety_gate_blocked"
|
| 281 |
+
)
|
| 282 |
+
elif not admit:
|
| 283 |
+
reason = "dead_swarm_rejected"
|
| 284 |
+
|
| 285 |
+
decisions.append({
|
| 286 |
+
"info_hash": info_hash,
|
| 287 |
+
"admit": admit,
|
| 288 |
+
"cohort": pred.category if admit else "rejected",
|
| 289 |
+
"score": float(pred.swarm_viability["viability_prob"] if admit else 0.0),
|
| 290 |
+
"reason": reason,
|
| 291 |
+
})
|
| 292 |
+
|
| 293 |
+
return {
|
| 294 |
+
"schema": "dh-waypoint.processors.admission.v1",
|
| 295 |
+
"decisions": decisions,
|
| 296 |
+
}
|
| 297 |
+
|
| 298 |
+
def handle_feedback_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
|
| 299 |
+
"""Handle `POST /v1/feedback` (`dh-waypoint.processors.feedback.v1`)."""
|
| 300 |
+
events = request_body.get("events") or []
|
| 301 |
+
accepted = 0
|
| 302 |
+
verified = 0
|
| 303 |
+
for ev in events:
|
| 304 |
+
accepted += 1
|
| 305 |
+
tier = str(ev.get("tier") or "")
|
| 306 |
+
outcome = str(ev.get("outcome") or "")
|
| 307 |
+
if tier in ("t3_files", "t4_playback", "verified_import", "human_gold") and outcome in (
|
| 308 |
+
"verified", "imported", "completed", "import_succeeded"
|
| 309 |
+
):
|
| 310 |
+
attrs = ev.get("attributes") if isinstance(ev.get("attributes"), Mapping) else {}
|
| 311 |
+
stream_meta = ev.get("stream_metadata") if isinstance(ev.get("stream_metadata"), Mapping) else {}
|
| 312 |
+
video_codec = str(stream_meta.get("video_codec") or attrs.get("video_codec") or "")
|
| 313 |
+
audio_codec = str(stream_meta.get("audio_codec") or attrs.get("audio_codec") or "")
|
| 314 |
+
audio_lang = str(stream_meta.get("audio_lang") or attrs.get("audio_lang") or "")
|
| 315 |
+
size_bytes = int(ev.get("size") or 0)
|
| 316 |
+
raw_title = str(ev.get("name") or ev.get("title") or "")
|
| 317 |
+
cat_id = str(ev.get("catalog_id") or attrs.get("catalog_id") or ev.get("info_hash") or "")
|
| 318 |
+
# Require physical stream metadata or explicit test attributes before recording a verified observation
|
| 319 |
+
if not raw_title or size_bytes <= 0 or not (video_codec or audio_codec or attrs):
|
| 320 |
+
continue
|
| 321 |
+
cand = CandidateObservation(
|
| 322 |
+
source_adapter="ffprobe",
|
| 323 |
+
raw_title=raw_title,
|
| 324 |
+
claimed_category=str(ev.get("content_type") or "movie"),
|
| 325 |
+
download_completed=True,
|
| 326 |
+
bytes_downloaded=size_bytes,
|
| 327 |
+
total_bytes=size_bytes,
|
| 328 |
+
catalog_id_type="movie_catalog" if ev.get("content_type") == "movie" else "series_catalog",
|
| 329 |
+
catalog_id_value=cat_id or "verified",
|
| 330 |
+
stream_metadata={
|
| 331 |
+
"video_codec": video_codec or "h264",
|
| 332 |
+
"audio_codec": audio_codec or "aac",
|
| 333 |
+
"audio_languages": [audio_lang or "eng"],
|
| 334 |
+
},
|
| 335 |
+
)
|
| 336 |
+
ok, _, _ = self.pipeline.ingest_rsi_feedback(cand)
|
| 337 |
+
if ok:
|
| 338 |
+
verified += 1
|
| 339 |
+
|
| 340 |
+
return {
|
| 341 |
+
"contract": "dh-waypoint.processors.feedback.v1",
|
| 342 |
+
"schema": "dh-waypoint.processors.feedback.v1",
|
| 343 |
+
"accepted": accepted,
|
| 344 |
+
"verified": verified,
|
| 345 |
+
}
|
| 346 |
+
|
| 347 |
+
|
| 348 |
+
def create_server(host: str = "localhost", port: int = 8095) -> ThreadingHTTPServer:
|
| 349 |
+
model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
|
| 350 |
+
variant = os.environ.get("ARMIGER_VARIANT", "fp32")
|
| 351 |
+
rsi_mode_str = os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")
|
| 352 |
+
pipeline = ArmigerUltraPipeline.from_directory(
|
| 353 |
+
model_dir,
|
| 354 |
+
variant=variant,
|
| 355 |
+
adult_threshold=float(os.environ.get("ARMIGER_ADULT_THRESHOLD", "0.35")),
|
| 356 |
+
junk_threshold=float(os.environ.get("ARMIGER_JUNK_THRESHOLD", "0.25")),
|
| 357 |
+
scope_threshold=float(os.environ.get("ARMIGER_SCOPE_THRESHOLD", "0.35")),
|
| 358 |
+
rsi_mode=DHTRSIMode(rsi_mode_str),
|
| 359 |
+
preload_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
|
| 360 |
+
)
|
| 361 |
+
adapter = ArmigerWaypointAdapter(
|
| 362 |
+
pipeline,
|
| 363 |
+
use_zero_cue_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
|
| 364 |
+
reject_dead_swarms=os.environ.get("ARMIGER_REJECT_DEAD_SWARMS", "false").lower() == "true",
|
| 365 |
+
)
|
| 366 |
+
|
| 367 |
+
class Handler(BaseHTTPRequestHandler):
|
| 368 |
+
def _send_json(self, status: int, payload: Dict[str, Any]) -> None:
|
| 369 |
+
body = json.dumps(payload).encode("utf-8")
|
| 370 |
+
self.send_response(status)
|
| 371 |
+
self.send_header("Content-Type", "application/json")
|
| 372 |
+
self.send_header("Content-Length", str(len(body)))
|
| 373 |
+
self.end_headers()
|
| 374 |
+
self.wfile.write(body)
|
| 375 |
+
|
| 376 |
+
def do_GET(self) -> None:
|
| 377 |
+
if self.path == "/healthz":
|
| 378 |
+
self._send_json(200, {"status": "ok", "engine": "armiger-ultra", "variant": variant})
|
| 379 |
+
elif self.path == "/v1/manifest":
|
| 380 |
+
self._send_json(200, adapter.handle_manifest())
|
| 381 |
+
elif self.path == "/v1/status":
|
| 382 |
+
self._send_json(200, adapter.handle_status())
|
| 383 |
+
else:
|
| 384 |
+
self._send_json(404, {"error": "not_found"})
|
| 385 |
+
|
| 386 |
+
def do_POST(self) -> None:
|
| 387 |
+
length = int(self.headers.get("Content-Length", "0"))
|
| 388 |
+
raw = self.rfile.read(length) if length > 0 else b"{}"
|
| 389 |
+
payload = json.loads(raw.decode("utf-8"))
|
| 390 |
+
if self.path == "/v1/process":
|
| 391 |
+
self._send_json(200, adapter.handle_process_job(payload))
|
| 392 |
+
elif self.path == "/v1/classify":
|
| 393 |
+
self._send_json(200, adapter.handle_classify_batch(payload))
|
| 394 |
+
elif self.path == "/v1/admit":
|
| 395 |
+
self._send_json(200, adapter.handle_admission_batch(payload))
|
| 396 |
+
elif self.path == "/v1/feedback":
|
| 397 |
+
self._send_json(200, adapter.handle_feedback_batch(payload))
|
| 398 |
+
else:
|
| 399 |
+
self._send_json(404, {"error": "not_found"})
|
| 400 |
+
|
| 401 |
+
def log_message(self, fmt: str, *args: Any) -> None:
|
| 402 |
+
return
|
| 403 |
+
|
| 404 |
+
return ThreadingHTTPServer((host, port), Handler)
|
| 405 |
+
|
| 406 |
+
|
| 407 |
+
if __name__ == "__main__":
|
| 408 |
+
listen_host = os.environ.get("ARMIGER_HOST", "localhost")
|
| 409 |
+
listen_port = int(os.environ.get("ARMIGER_PORT", "8095"))
|
| 410 |
+
srv = create_server(listen_host, listen_port)
|
| 411 |
+
print(f"Armiger DH-Waypoint processor listening on {listen_host}:{listen_port}")
|
| 412 |
+
srv.serve_forever()
|
test_chart_pipeline_integrity.py
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Pipeline and benchmark integrity test suite for Armiger & Armiger Ultra."""
|
| 2 |
+
from test_armiger_ultra_pipeline import * # noqa: F401,F403
|
| 3 |
+
from test_integration_examples import * # noqa: F401,F403
|