spencernorton commited on
Commit
9af29a6
·
verified ·
1 Parent(s): 21778af

docs: add DH-Waypoint, Bitmagnet, and *Arr ecosystem integration guide and examples

Browse files
README.md CHANGED
@@ -62,6 +62,18 @@ print(pred["heads"].keys()) # dict_keys(['category', 'adult', 'junk', 'anime
62
 
63
  ---
64
 
 
 
 
 
 
 
 
 
 
 
 
 
65
  ## 8 Simultaneous Classification Heads (Audited Unmutated Ground-Truth Accuracy)
66
 
67
  All metrics below are reproducibly computed using `armiger_inference.py` against the **original, unmutated ground-truth evaluation keys** (`rows.y`), separating **Standalone Neural ONNX (`armiger.onnx`)** from **Fair Blind Hybrid (`+ ZeroCueCatalogPlugin` & `AudioLanguageGuard`)**:
 
62
 
63
  ---
64
 
65
+ ## Ecosystem Integrations (`DH-Waypoint`, `Bitmagnet` & `*Arr` Stack)
66
+
67
+ Armiger and Armiger Ultra ship with turnkey HTTP services, Docker Compose stacks, and reference adapters in [`examples/`](examples) and [`docs/INTEGRATION_GUIDE.md`](docs/INTEGRATION_GUIDE.md):
68
+
69
+ - **[`DH-Waypoint` Integration (`examples/dh-waypoint/`)](examples/dh-waypoint)** — Drop-in `processorapi/v1` service (`armiger_waypoint_processor.py`) and plugin manifest (`07-armiger-ultra.plugin.yml`) implementing batch classification (`dh-waypoint.processor.v1`), pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop telemetry for [`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint).
70
+ - **[`Bitmagnet` Integration (`examples/bitmagnet/`)](examples/bitmagnet)** — Pre-ingest / post-crawl batch classifier and malware/honeypot pruner (`bitmagnet_armiger_bridge.py`) that upgrades upstream Bitmagnet from `35.29%` rule accuracy to **`95.49%` 11-class neural accuracy** and automatically prunes disguised `.mkv.exe`/`.scr`/`.lnk` malware traps via GraphQL `torrent.delete`.
71
+ - **[`*Arr` Stack & `qBittorrent` Integration (`examples/arr-stack/`)](examples/arr-stack)** — Pre-grab Torznab XML filtering proxy (`armiger_torznab_proxy.py`) between `Prowlarr` / `Bitmagnet` / `DH-Waypoint` and `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, and `Whisparr`, paired with a post-import webhook & `ffprobe` ground-truth receiver (`armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier`.
72
+
73
+ See the **[Complete Ecosystem Integration Guide (`docs/INTEGRATION_GUIDE.md`)](docs/INTEGRATION_GUIDE.md)** for architecture diagrams, Docker Compose files, and step-by-step setup instructions.
74
+
75
+ ---
76
+
77
  ## 8 Simultaneous Classification Heads (Audited Unmutated Ground-Truth Accuracy)
78
 
79
  All metrics below are reproducibly computed using `armiger_inference.py` against the **original, unmutated ground-truth evaluation keys** (`rows.y`), separating **Standalone Neural ONNX (`armiger.onnx`)** from **Fair Blind Hybrid (`+ ZeroCueCatalogPlugin` & `AudioLanguageGuard`)**:
docs/INTEGRATION_GUIDE.md ADDED
@@ -0,0 +1,282 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Armiger & Armiger Ultra Ecosystem Integration Guide
2
+
3
+ This guide provides step-by-step instructions, turnkey Docker Compose stacks, and reference HTTP adapters for integrating **Armiger** (`8.3 MB INT8 CPU`) and **Armiger Ultra** (`32.3 MB FP32 GPU/CPU` + 6 official plugins) with:
4
+
5
+ 1. **[Part A — Integrating with `DH-Waypoint`](#part-a--integrating-armiger-with-dh-waypoint)** (`https://github.com/spencernorton/DH-Waypoint`):
6
+ - Native `dh-waypoint.processor.v1` batch classification (`POST /v1/classify`)
7
+ - Pre-ingest `dh-waypoint.processors.admission.v1` cohort admission gate (`POST /v1/admit`)
8
+ - Closed-loop 4-tier `dh-waypoint.processors.feedback.v1` outcome telemetry (`POST /v1/feedback`)
9
+ 2. **[Part B — Integrating with `Bitmagnet`](#part-b--integrating-armiger-with-bitmagnet)**:
10
+ - Pre-ingest / post-crawl batch classifier & malware/honeypot pruner (`examples/bitmagnet/bitmagnet_armiger_bridge.py`)
11
+ - GraphQL `torrentDelete` / content enrichment bridge & Torznab proxy enrichment
12
+ 3. **[Part C — Integrating with the `*Arr` Stack (`Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `Whisparr`) & `qBittorrent`](#part-c--integrating-armiger-with-the-arr-stack--qbittorrent)**:
13
+ - Pre-grab Torznab filtering proxy (`examples/arr-stack/armiger_torznab_proxy.py`) blocking disguised `.exe`/`.scr`/`.lnk` malware, fake-seeder honeypot traps, dead swarms, and unrequested foreign-language audio tracks before grab
14
+ - Post-import webhook & `ffprobe` physical ground-truth receiver (`examples/arr-stack/armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier` for local conformal calibration and opt-in anonymous `BEP-44`/`BEP-46` RSI knowledge sharing
15
+
16
+ ---
17
+
18
+ ## Architecture Overview
19
+
20
+ ```mermaid
21
+ flowchart LR
22
+ subgraph Indexers["DHT Indexers & Crawlers"]
23
+ DHW["DH-Waypoint Core<br/>(processorapi/v1)"]
24
+ BM["Bitmagnet<br/>(GraphQL + Torznab)"]
25
+ PR["Prowlarr<br/>(Indexer Manager)"]
26
+ end
27
+
28
+ subgraph ArmigerSuite["Armiger / Armiger Ultra"]
29
+ PROC["Armiger Waypoint Processor<br/>(/v1/classify, /v1/admit, /v1/feedback)"]
30
+ BMBR["Bitmagnet Armiger Bridge<br/>(/v1/bitmagnet/classify)"]
31
+ TPROXY["Armiger Torznab Proxy<br/>(Port 8097 Admission Gate)"]
32
+ WHOOK["Armiger Webhook & ffprobe Receiver<br/>(Port 8098 Ground Truth + BEP-44 RSI)"]
33
+ end
34
+
35
+ subgraph ArrStack["*Arr Media Managers & Download Client"]
36
+ SON["Sonarr / Radarr / Lidarr / Readarr"]
37
+ QBT["qBittorrent / Transmission"]
38
+ end
39
+
40
+ DHW <-->|"HTTP processorapi/v1"| PROC
41
+ BM <-->|"Batch Classify + GraphQL Prune"| BMBR
42
+ DHW -->|"Torznab Feed"| TPROXY
43
+ BM -->|"Torznab Feed"| TPROXY
44
+ PR -->|"Torznab Feed"| TPROXY
45
+ TPROXY -->|"Sanitized & Enriched XML"| SON
46
+ SON -->|"Grab Approved Release"| QBT
47
+ SON -->|"On Import Webhook"| WHOOK
48
+ QBT -->|"On Completion + ffprobe"| WHOOK
49
+ WHOOK -.->|"Local Calibration & Opt-In BEP-44 RSI"| PROC
50
+ ```
51
+
52
+ ---
53
+
54
+ ## Part A — Integrating Armiger with `DH-Waypoint`
55
+
56
+ [`DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint) includes a built-in external processor subsystem (`processorapi/v1`) that connects directly to [`examples/dh-waypoint/armiger_waypoint_processor.py`](../examples/dh-waypoint/armiger_waypoint_processor.py).
57
+
58
+ ### 1. Turnkey Docker Compose Stack
59
+
60
+ Launch `DH-Waypoint`, `PostgreSQL`, and `Armiger Ultra` together using [`examples/dh-waypoint/docker-compose.yml`](../examples/dh-waypoint/docker-compose.yml):
61
+
62
+ ```bash
63
+ export POSTGRES_PASSWORD="<postgres-password>"
64
+ docker compose -f examples/dh-waypoint/docker-compose.yml up -d
65
+ ```
66
+
67
+ ### 2. Drop-In Plugin Manifest (`07-armiger-ultra.plugin.yml`)
68
+
69
+ Mount [`examples/dh-waypoint/07-armiger-ultra.plugin.yml`](../examples/dh-waypoint/07-armiger-ultra.plugin.yml) into `DH_WAYPOINT_PROCESSORS_PLUGIN_DIR` (`/etc/dh-waypoint/plugins.d/07-armiger-ultra.plugin.yml`):
70
+
71
+ ```yaml
72
+ processor:
73
+ name: armiger-ultra
74
+ endpoint: http://armiger-ultra:8095/v1/classify
75
+ timeout: 5s
76
+ max_batch: 64
77
+ required: false
78
+ forward_files: true
79
+ max_files: 64
80
+ forward_trackers: true
81
+ forward_swarm: true
82
+
83
+ admission:
84
+ enabled: true
85
+ endpoint: http://armiger-ultra:8095/v1/admit
86
+ timeout: 3s
87
+ max_batch: 64
88
+ max_files: 64
89
+ fail_open: true
90
+
91
+ learning:
92
+ enabled: true
93
+ feedback_endpoint: http://armiger-ultra:8095/v1/feedback
94
+ feedback_timeout: 5s
95
+ feedback_batch_size: 64
96
+ flush_interval: 15s
97
+ queue_size: 2048
98
+ ```
99
+
100
+ ### 3. Equivalent `DH_WAYPOINT_*` Environment Variables
101
+
102
+ If you configure `DH-Waypoint` via environment variables instead of `plugins.d`:
103
+
104
+ | Variable | Recommended Value | Purpose |
105
+ | :--- | :--- | :--- |
106
+ | `DH_WAYPOINT_PROCESSORS_HTTP_CHAIN` | `armiger-ultra=http://armiger-ultra:8095/v1/classify` | Ordered HTTP classification chain (`dh-waypoint.processor.v1`). |
107
+ | `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_FILES` | `true` | Forwards physical file paths and byte sizes to Armiger's File-Tree Encoder. |
108
+ | `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_TRACKERS` | `true` | Forwards tracker domain hints to `SwarmPriors`. |
109
+ | `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_SWARM` | `true` | Forwards seeder/leecher counts to `SwarmSurvivalNet`. |
110
+ | `DH_WAYPOINT_PROCESSORS_ADMISSION_ENABLED` | `true` | Enables pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`). |
111
+ | `DH_WAYPOINT_PROCESSORS_ADMISSION_ENDPOINT` | `http://armiger-ultra:8095/v1/admit` | Pre-ingest admission endpoint. |
112
+ | `DH_WAYPOINT_LEARNING_ENABLED` | `true` | Enables the 4-tier `dh-waypoint.processors.feedback.v1` outcome loop. |
113
+ | `DH_WAYPOINT_LEARNING_FEEDBACK_ENDPOINT` | `http://armiger-ultra:8095/v1/feedback` | Delivers `t1_discovery` through `t4_playback` feedback batches to Armiger. |
114
+
115
+ ---
116
+
117
+ ## Part B — Integrating Armiger with `Bitmagnet`
118
+
119
+ Upstream `Bitmagnet` uses deterministic keyword/extension rules (`35.29%` 11-class accuracy on `N=10,262` audited torrents). Deploying [`examples/bitmagnet/bitmagnet_armiger_bridge.py`](../examples/bitmagnet/bitmagnet_armiger_bridge.py) upgrades a `Bitmagnet` deployment to **`95.49%` 11-class accuracy** (`97.55%` 6-dimension composite accuracy) and **`100%` fake-seeder malware trap recall**.
120
+
121
+ ### 1. Turnkey `Bitmagnet` + `Armiger` Docker Compose Stack
122
+
123
+ Start the stack in [`examples/bitmagnet/docker-compose.yml`](../examples/bitmagnet/docker-compose.yml):
124
+
125
+ ```bash
126
+ export POSTGRES_PASSWORD="<postgres-password>"
127
+ docker compose -f examples/bitmagnet/docker-compose.yml up -d
128
+ ```
129
+
130
+ This launches two companion services alongside `bitmagnet`:
131
+ - **`armiger-bitmagnet-bridge` (`port 8096`)**: Evaluates batches of newly crawled Bitmagnet torrents (`POST /v1/bitmagnet/classify`), maps Armiger's 11-class predictions (`movie`, `tv_show`, `music`, `ebook`, `audiobook`, `comic`, `software`, `game`, `xxx`) and tags (`armiger:<category>`, `armiger:lang:<lang>`, `armiger:viability:<tier>`) onto Bitmagnet items, and automatically executes Bitmagnet's GraphQL `torrent.delete` mutation against disguised malware payloads (`.mkv.exe`, `.scr`, `.lnk`, `.vbs`) and fake-seeder honeypots.
132
+ - **`armiger-torznab-proxy` (`port 8097`)**: Wraps `http://bitmagnet:3333/torznab` with real-time neural admission filtering before `Prowlarr`, `Sonarr`, or `Radarr` sees search results.
133
+
134
+ ### 2. Calling the Bitmagnet Classifier Bridge Directly
135
+
136
+ ```bash
137
+ curl -sS -X POST http://localhost:8096/v1/bitmagnet/classify \
138
+ -H "Content-Type: application/json" \
139
+ -d '{
140
+ "torrents": [
141
+ {
142
+ "infoHash": "1111222233334444555566667777888899990000",
143
+ "name": "Sintel.2010.2160p.UHD.BluRay.REMUX.HEVC.TrueHD.7.1.Atmos-OPEN",
144
+ "size": 74500000000,
145
+ "seeders": 320,
146
+ "leechers": 18,
147
+ "files": [{"path": "Sintel.2010.2160p.mkv", "size": 74500000000}]
148
+ },
149
+ {
150
+ "infoHash": "aaaa22223333444455556666777788889999bbbb",
151
+ "name": "Big.Buck.Bunny.2008.1080p.BluRay.mkv.exe",
152
+ "size": 15400000,
153
+ "seeders": 4500,
154
+ "leechers": 12,
155
+ "files": [{"path": "Big.Buck.Bunny.2008.1080p.BluRay.mkv.exe", "size": 15400000}]
156
+ }
157
+ ]
158
+ }'
159
+ ```
160
+
161
+ The bridge classifies the first item as `contentType: "movie"` (`viabilityTier: "alive_viable"`) and flags the second item (`Big.Buck.Bunny.2008.1080p.BluRay.mkv.exe`) for immediate deletion via `graphqlDeleteMutation`.
162
+
163
+ ---
164
+
165
+ ## Part C — Integrating Armiger with the `*Arr` Stack & `qBittorrent`
166
+
167
+ Armiger integrates with `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `Whisparr`, and `qBittorrent` at **two stages**:
168
+ 1. **Stage 1 — Pre-Grab Torznab Filtering Proxy (`port 8097`):** Blocks malware masquerades, fake-seeder honeypots, dead swarms, and unrequested foreign-language audio releases *before* any `*Arr` app grabs them.
169
+ 2. **Stage 2 — Post-Import Webhook & `ffprobe` Ground-Truth Receiver (`port 8098`):** Verifies completed downloads and `ffprobe` media streams to calibrate local Mondrian thresholds and optionally contribute differential-privacy blinded telemetry to the `BEP-44`/`BEP-46` RSI network.
170
+
171
+ ### 1. Stage 1: Wiring the Torznab Filtering Proxy (`Prowlarr` / `Sonarr` / `Radarr` / `Lidarr` / `Readarr`)
172
+
173
+ Start [`examples/arr-stack/docker-compose.yml`](../examples/arr-stack/docker-compose.yml):
174
+
175
+ ```bash
176
+ export UPSTREAM_TORZNAB_URL="http://dh-waypoint:3333/torznab"
177
+ docker compose -f examples/arr-stack/docker-compose.yml up -d
178
+ ```
179
+
180
+ #### Configuring in `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, or `Whisparr`:
181
+ 1. Open **Settings -> Indexers -> + -> Torznab (Custom)**.
182
+ 2. Set **URL** to `http://armiger-torznab-proxy:8097` (instead of pointing directly at `http://bitmagnet:3333/torznab` or `http://dh-waypoint:3333/torznab`).
183
+ 3. Every Torznab search query (`?t=search`, `?t=tvsearch`, `?t=movie`, `?t=music`, `?t=book`) is transparently forwarded upstream, batch-scored by `ArmigerUltraPipeline`, stripped of unsafe/honeypot/dead/wrong-language items, and annotated with custom `<torznab:attr>` tags:
184
+ - `armiger_category` (`movie`, `tv_show`, `music`, `audiobook`, `ebook`, `comic`, `software`, `game`, `xxx`)
185
+ - `armiger_lang` (`english`, `non_english`, `multi_or_unknown`)
186
+ - `armiger_viability` (`alive_strong`, `alive_viable`, `marginal`, `dead`)
187
+ - `armiger_survival_prob` (`0.0000`–`1.0000`)
188
+
189
+ #### Torznab Proxy Environment Controls
190
+
191
+ | Variable | Default | Description |
192
+ | :--- | :---: | :--- |
193
+ | `UPSTREAM_TORZNAB_URL` | `http://dh-waypoint:3333/torznab` | Upstream Torznab base URL (`DH-Waypoint`, `Bitmagnet`, or `Prowlarr`). |
194
+ | `ARMIGER_VARIANT` | `int8` | `int8` (`8.3 MB` CPU fast-path, `2.44 ms` p50) or `fp32` (`32.3 MB` GPU/CPU). |
195
+ | `ARMIGER_REJECT_DEAD_SWARMS` | `true` | Drops Torznab items whose `SwarmSurvivalNet` viability tier is `dead`. |
196
+ | `ARMIGER_REQUIRE_ENGLISH_AUDIO` | `false` | When `true`, blocks releases classified as `non_english` audio by `AudioLanguageGuard`. |
197
+ | `ARMIGER_ALLOW_ENGLISH_SUBS` | `true` | When `ARMIGER_REQUIRE_ENGLISH_AUDIO=true`, permits foreign-audio anime/films that carry verified English subtitles (`has_eng_subs`). |
198
+
199
+ ### 2. Stage 2: Wiring Post-Import Webhooks (`Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `qBittorrent`)
200
+
201
+ `armiger-arr-webhook-receiver` (`port 8098`) exposes dedicated endpoints backed by the adapters in [`armiger_dht_rsi.py`](../armiger_dht_rsi.py):
202
+
203
+ | Application | Where to Configure in UI | Webhook URL | Underlying Armiger Adapter |
204
+ | :--- | :--- | :--- | :--- |
205
+ | **Sonarr** | `Settings -> Connect -> + -> Webhook` (`On Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/sonarr` | `SeriesWebhookAdapter` |
206
+ | **Radarr** | `Settings -> Connect -> + -> Webhook` (`On Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/radarr` | `MovieWebhookAdapter` |
207
+ | **Lidarr** | `Settings -> Connect -> + -> Webhook` (`On Release Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/lidarr` | `AudioWebhookAdapter` |
208
+ | **Readarr** | `Settings -> Connect -> + -> Webhook` (`On Release Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/readarr` | `PublicationWebhookAdapter` |
209
+ | **qBittorrent** | `Options -> Downloads -> Run external program` | `POST http://armiger-arr-webhook-receiver:8098/webhooks/qbittorrent` | `QBittorrentAdapter` |
210
+
211
+ #### `qBittorrent` Completion Command
212
+
213
+ In `qBittorrent` (`Tools -> Options -> Downloads -> Run external program on torrent completion`), configure:
214
+
215
+ ```bash
216
+ curl -sS -X POST http://armiger-arr-webhook-receiver:8098/webhooks/qbittorrent -H "Content-Type: application/json" -d "{\"name\":\"%N\",\"hash\":\"%I\",\"content_path\":\"%F\",\"size\":%Z,\"progress\":1.0,\"state\":\"uploading\"}"
217
+ ```
218
+
219
+ #### Programmatic Python Usage (`SeriesWebhookAdapter`, `MovieWebhookAdapter`, `QBittorrentAdapter`, `PhysicalGroundTruthVerifier`)
220
+
221
+ You can also invoke the webhook adapters and `ffprobe` verifier directly in Python via `ArmigerUltraPipeline.ingest_rsi_feedback`:
222
+
223
+ ```python
224
+ from armiger_ultra_pipeline import ArmigerUltraPipeline
225
+
226
+ pipeline = ArmigerUltraPipeline.from_directory(".", variant="int8")
227
+
228
+ # 1. Ingest a native Sonarr "Download" (Import) webhook + ffprobe output
229
+ ok, reason, record = pipeline.ingest_rsi_feedback(
230
+ {
231
+ "eventType": "Download",
232
+ "series": {"title": "Cosmos Laundromat", "tvdbId": 310123},
233
+ "episodes": [{"seasonNumber": 1, "episodeNumber": 1}],
234
+ "release": {
235
+ "releaseTitle": "Cosmos.Laundromat.S01E01.1080p.WEB-DL.DDP5.1.H.264-OPEN",
236
+ "size": 1_450_000_000,
237
+ },
238
+ "episodeFile": {
239
+ "relativePath": "Season 01/Cosmos.Laundromat.S01E01.mkv",
240
+ "size": 1_450_000_000,
241
+ "mediaInfo": {
242
+ "videoCodec": "h264",
243
+ "audioFormat": "eac3",
244
+ "audioChannels": 5.1,
245
+ "audioLanguages": "eng",
246
+ "subtitles": "eng",
247
+ },
248
+ },
249
+ },
250
+ adapter="series_webhook",
251
+ )
252
+ assert ok and reason == "accepted_and_queued_with_poisson_jitter"
253
+ print(record.blinded_stem.blinded_hash, record.category, record.lang)
254
+
255
+ # 2. Ingest a native Radarr "Download" (Import) webhook
256
+ ok, reason, record = pipeline.ingest_rsi_feedback(
257
+ {
258
+ "eventType": "Download",
259
+ "movie": {"title": "Sintel", "year": 2010, "tmdbId": 45745},
260
+ "release": {
261
+ "releaseTitle": "Sintel.2010.2160p.UHD.BluRay.REMUX.HEVC.TrueHD.7.1.Atmos-OPEN",
262
+ "size": 24_800_000_000,
263
+ },
264
+ "movieFile": {
265
+ "relativePath": "Sintel (2010)/Sintel.2010.2160p.mkv",
266
+ "size": 24_800_000_000,
267
+ },
268
+ },
269
+ adapter="movie_webhook",
270
+ )
271
+ assert ok and record.category == "movie"
272
+ ```
273
+
274
+ ### 3. `BEP-44` / `BEP-46` Privacy & RSI Participation Modes
275
+
276
+ All adapters enforce `PhysicalGroundTruthVerifier` (never trusting unverified model predictions) and `TelemetryAnonymizer` (stripping tracker passkeys, local paths, and IP addresses; blinding title stems via post-quantum `HMAC-SHA256` + `SHAKE-256`; applying Laplace differential privacy $\epsilon = 1.0$; and scheduling outbound packets with Poisson timing jitter):
277
+
278
+ | `ARMIGER_DHT_RSI_MODE` | Behavior |
279
+ | :--- | :--- |
280
+ | `off` | Disables all `BEP-44`/`BEP-46` DHT traffic; calibration and ground-truth verification stay strictly local. |
281
+ | `pull_only` *(Default)* | Verifies and applies `M`-of-`N` hybrid post-quantum (`Ed25519` + `WOTS+-SHAKE256`) signed calibration manifests from the DHT after passing the 500-item golden canary gate; never transmits telemetry. |
282
+ | `anonymous_contribute` | Opts in to transmitting differential-privacy blinded, hybrid post-quantum (`X25519` + `ML-KEM-768` + `ChaCha20-Poly1305`) encrypted ground-truth observations. |
examples/README.md ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Armiger & Armiger Ultra Integration Examples
2
+
3
+ Turnkey reference adapters, HTTP services, and Docker Compose stacks for integrating **Armiger** (`8.3 MB INT8`) and **Armiger Ultra** (`32.3 MB FP32` + 6 plugins) across the self-hosted indexer and media automation ecosystem:
4
+
5
+ | Stack | Directory | Description |
6
+ | :--- | :--- | :--- |
7
+ | **[DH-Waypoint](https://github.com/spencernorton/DH-Waypoint)** | [`examples/dh-waypoint/`](dh-waypoint) | Native `processorapi/v1` HTTP processor (`dh-waypoint.processor.v1`), pre-ingest cohort admission gate (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop RSI telemetry bridge. |
8
+ | **Bitmagnet** | [`examples/bitmagnet/`](bitmagnet) | Fast pre-ingest / post-crawl batch classifier, malware/honeypot pruner, PostgreSQL/GraphQL enrichment bridge (`bitmagnet_armiger_bridge.py`), and Torznab proxy stack. |
9
+ | **\*Arr Stack & qBittorrent** | [`examples/arr-stack/`](arr-stack) | Pre-grab Torznab filtering proxy (`armiger_torznab_proxy.py`) between `Prowlarr` / `Bitmagnet` and `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, and `Whisparr`, plus a post-import webhook & `ffprobe` ground-truth receiver (`armiger_arr_webhook_receiver.py`). |
10
+
11
+ See [`docs/INTEGRATION_GUIDE.md`](../docs/INTEGRATION_GUIDE.md) for the complete architecture and step-by-step setup guide.
examples/arr-stack/armiger_arr_webhook_receiver.py ADDED
@@ -0,0 +1,197 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Armiger Post-Import Webhook & ffprobe Ground-Truth Receiver for Sonarr, Radarr, Lidarr, Readarr & qBittorrent.
2
+
3
+ Wires completion and import events from the *Arr media automation stack and qBittorrent directly into
4
+ Armiger's `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`,
5
+ `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier`:
6
+ - `POST /webhooks/sonarr` -> `SeriesWebhookAdapter` (`adapter="series_webhook"`)
7
+ - `POST /webhooks/radarr` -> `MovieWebhookAdapter` (`adapter="movie_webhook"`)
8
+ - `POST /webhooks/lidarr` -> `AudioWebhookAdapter` (`adapter="audio_webhook"`)
9
+ - `POST /webhooks/readarr` -> `PublicationWebhookAdapter` (`adapter="publication_webhook"`)
10
+ - `POST /webhooks/qbittorrent` -> `QBittorrentAdapter` (`adapter="qbittorrent"`)
11
+
12
+ When `ARMIGER_RUN_FFPROBE=true` and the imported media path is accessible on disk, the receiver runs
13
+ `ffprobe -v quiet -print_format json -show_format -show_streams <path>` to physically verify video/audio
14
+ codecs, channel counts, audio languages, and subtitle tracks before passing the observation to
15
+ `ArmigerUltraPipeline.ingest_rsi_feedback(...)`.
16
+ """
17
+ from __future__ import annotations
18
+
19
+ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
20
+ import json
21
+ import os
22
+ from pathlib import Path
23
+ import subprocess
24
+ import sys
25
+ from typing import Any, Callable, Dict, Mapping, Optional
26
+
27
+ ROOT_DIR = Path(__file__).resolve().parents[2]
28
+ if str(ROOT_DIR) not in sys.path:
29
+ sys.path.insert(0, str(ROOT_DIR))
30
+
31
+ from armiger_dht_rsi import DHTRSIMode
32
+ from armiger_ultra_pipeline import ArmigerUltraPipeline
33
+
34
+ ROUTE_TO_ADAPTER: Dict[str, str] = {
35
+ "/webhooks/sonarr": "series_webhook",
36
+ "/webhooks/radarr": "movie_webhook",
37
+ "/webhooks/lidarr": "audio_webhook",
38
+ "/webhooks/readarr": "publication_webhook",
39
+ "/webhooks/qbittorrent": "qbittorrent",
40
+ }
41
+
42
+
43
+ def run_ffprobe_json(media_path: str, timeout_sec: float = 10.0) -> Optional[Dict[str, Any]]:
44
+ """Run ffprobe on a local media file and return parsed JSON stream/format metadata."""
45
+ p = Path(media_path)
46
+ if not p.is_file():
47
+ return None
48
+ try:
49
+ proc = subprocess.run(
50
+ [
51
+ "ffprobe",
52
+ "-v",
53
+ "quiet",
54
+ "-print_format",
55
+ "json",
56
+ "-show_format",
57
+ "-show_streams",
58
+ str(p),
59
+ ],
60
+ capture_output=True,
61
+ text=True,
62
+ timeout=timeout_sec,
63
+ check=True,
64
+ )
65
+ return json.loads(proc.stdout)
66
+ except Exception:
67
+ return None
68
+
69
+
70
+ class ArmigerArrWebhookService:
71
+ """Processes *Arr and qBittorrent completion webhooks into Armiger RSI ground-truth observations."""
72
+
73
+ def __init__(
74
+ self,
75
+ pipeline: ArmigerUltraPipeline,
76
+ *,
77
+ enable_ffprobe: bool = True,
78
+ ffprobe_runner: Callable[[str], Optional[Dict[str, Any]]] = run_ffprobe_json,
79
+ ) -> None:
80
+ self.pipeline = pipeline
81
+ self.enable_ffprobe = enable_ffprobe
82
+ self.ffprobe_runner = ffprobe_runner
83
+
84
+ @staticmethod
85
+ def _extract_media_path(payload: Mapping[str, Any]) -> Optional[str]:
86
+ for section_key in ("movieFile", "episodeFile"):
87
+ sec = payload.get(section_key)
88
+ if isinstance(sec, Mapping):
89
+ full_path = sec.get("path") or sec.get("relativePath")
90
+ if full_path:
91
+ return str(full_path)
92
+ for list_key in ("trackFiles", "bookFiles"):
93
+ items = payload.get(list_key)
94
+ if isinstance(items, list) and items and isinstance(items[0], Mapping):
95
+ full_path = items[0].get("path")
96
+ if full_path:
97
+ return str(full_path)
98
+ if payload.get("content_path") or payload.get("save_path"):
99
+ return str(payload.get("content_path") or payload.get("save_path"))
100
+ return None
101
+
102
+ def process_webhook(
103
+ self,
104
+ route_path: str,
105
+ payload: Mapping[str, Any],
106
+ *,
107
+ explicit_ffprobe: Optional[Dict[str, Any]] = None,
108
+ ) -> Dict[str, Any]:
109
+ """Process a webhook payload from Sonarr, Radarr, Lidarr, Readarr, or qBittorrent."""
110
+ adapter_name = ROUTE_TO_ADAPTER.get(route_path)
111
+ if not adapter_name:
112
+ return {"accepted": False, "reason": f"unknown_route:{route_path}"}
113
+
114
+ event_type = str(payload.get("eventType") or "")
115
+ if event_type.lower() == "test":
116
+ return {
117
+ "accepted": True,
118
+ "verified": False,
119
+ "adapter": adapter_name,
120
+ "reason": "test_webhook_acknowledged",
121
+ }
122
+
123
+ ffprobe_data = explicit_ffprobe
124
+ if ffprobe_data is None and self.enable_ffprobe:
125
+ media_path = self._extract_media_path(payload)
126
+ if media_path:
127
+ ffprobe_data = self.ffprobe_runner(media_path)
128
+
129
+ ok, reason, record = self.pipeline.ingest_rsi_feedback(
130
+ payload,
131
+ adapter=adapter_name,
132
+ ffprobe_output=ffprobe_data,
133
+ )
134
+ return {
135
+ "accepted": True,
136
+ "verified": bool(ok),
137
+ "adapter": adapter_name,
138
+ "reason": reason,
139
+ "blinded_hash": record.blinded_stem.blinded_hash if record is not None else None,
140
+ "category": record.category if record is not None else None,
141
+ "lang": record.lang if record is not None else None,
142
+ "junk": record.junk if record is not None else None,
143
+ }
144
+
145
+
146
+ def create_webhook_server(host: str = "localhost", port: int = 8098) -> ThreadingHTTPServer:
147
+ model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
148
+ variant = os.environ.get("ARMIGER_VARIANT", "int8")
149
+ rsi_mode_str = os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")
150
+ pipeline = ArmigerUltraPipeline.from_directory(
151
+ model_dir,
152
+ variant=variant,
153
+ rsi_mode=DHTRSIMode(rsi_mode_str),
154
+ preload_catalog=False,
155
+ )
156
+ service = ArmigerArrWebhookService(
157
+ pipeline,
158
+ enable_ffprobe=os.environ.get("ARMIGER_RUN_FFPROBE", "true").lower() == "true",
159
+ )
160
+
161
+ class WebhookHandler(BaseHTTPRequestHandler):
162
+ def _send_json(self, status: int, data: Dict[str, Any]) -> None:
163
+ body = json.dumps(data).encode("utf-8")
164
+ self.send_response(status)
165
+ self.send_header("Content-Type", "application/json")
166
+ self.send_header("Content-Length", str(len(body)))
167
+ self.end_headers()
168
+ self.wfile.write(body)
169
+
170
+ def do_GET(self) -> None:
171
+ if self.path == "/healthz":
172
+ self._send_json(200, {"status": "ok", "service": "armiger-arr-webhook-receiver", "rsi_mode": rsi_mode_str})
173
+ else:
174
+ self._send_json(404, {"error": "not_found"})
175
+
176
+ def do_POST(self) -> None:
177
+ length = int(self.headers.get("Content-Length", "0"))
178
+ raw = self.rfile.read(length) if length > 0 else b"{}"
179
+ payload = json.loads(raw.decode("utf-8"))
180
+ if self.path in ROUTE_TO_ADAPTER:
181
+ res = service.process_webhook(self.path, payload)
182
+ self._send_json(200, res)
183
+ else:
184
+ self._send_json(404, {"error": "not_found"})
185
+
186
+ def log_message(self, fmt: str, *args: Any) -> None:
187
+ return
188
+
189
+ return ThreadingHTTPServer((host, port), WebhookHandler)
190
+
191
+
192
+ if __name__ == "__main__":
193
+ listen_host = os.environ.get("ARMIGER_WEBHOOK_HOST", "localhost")
194
+ listen_port = int(os.environ.get("ARMIGER_WEBHOOK_PORT", "8098"))
195
+ srv = create_webhook_server(listen_host, listen_port)
196
+ print(f"Armiger *Arr & qBittorrent webhook receiver listening on {listen_host}:{listen_port}")
197
+ srv.serve_forever()
examples/arr-stack/armiger_torznab_proxy.py ADDED
@@ -0,0 +1,254 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Armiger Torznab Filtering Proxy & Pre-Grab Admission Gate for Prowlarr, Bitmagnet, and the *Arr Stack.
2
+
3
+ Sits transparently between upstream Torznab indexers (`Prowlarr`, `Bitmagnet`, `DH-Waypoint`, `Jackett`)
4
+ and downstream media managers (`Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `Whisparr`).
5
+
6
+ Every Torznab XML `<item>` returned by an upstream search is evaluated in milliseconds by
7
+ `ArmigerUltraPipeline.classify_batch` before `Sonarr` or `Radarr` can grab it:
8
+ 1. Blocks disguised malware executables (`.mkv.exe`, `.scr`, `.vbs`, `.lnk`, `.ps1`, passworded archives).
9
+ 2. Blocks fake-seeder malware honeypots (`SwarmSurvivalNet` `is_honeypot_trap == True`).
10
+ 3. Blocks dead swarms (`viability_tier == "dead"` when `ARMIGER_REJECT_DEAD_SWARMS=true`).
11
+ 4. Blocks unrequested foreign-language audio traps (`AudioLanguageGuard` cross-head check when
12
+ `ARMIGER_REQUIRE_ENGLISH_AUDIO=true`, with optional anime/subtitle allowance via `ARMIGER_ALLOW_ENGLISH_SUBS=true`).
13
+ 5. Enriches surviving `<item>` nodes with `<torznab:attr name="armiger_category" .../>` and
14
+ `<torznab:attr name="armiger_viability" .../>` attributes.
15
+ """
16
+ from __future__ import annotations
17
+
18
+ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
19
+ import json
20
+ import os
21
+ from pathlib import Path
22
+ import sys
23
+ from typing import Any, Dict, List, Optional, Tuple
24
+ import urllib.parse
25
+ import urllib.request
26
+ import xml.etree.ElementTree as ET
27
+
28
+ ROOT_DIR = Path(__file__).resolve().parents[2]
29
+ if str(ROOT_DIR) not in sys.path:
30
+ sys.path.insert(0, str(ROOT_DIR))
31
+
32
+ from armiger_dht_rsi import DHTRSIMode
33
+ from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
34
+
35
+ TORZNAB_NS = "http://torznab.com/schemas/2015/feed"
36
+ ET.register_namespace("torznab", TORZNAB_NS)
37
+
38
+
39
+ class ArmigerTorznabFilter:
40
+ """Filters and enriches Torznab RSS/XML feeds using ArmigerUltraPipeline."""
41
+
42
+ def __init__(
43
+ self,
44
+ pipeline: ArmigerUltraPipeline,
45
+ *,
46
+ require_english_audio: bool = False,
47
+ allow_english_subs: bool = True,
48
+ reject_dead_swarms: bool = True,
49
+ allowed_categories: Optional[List[str]] = None,
50
+ use_zero_cue_catalog: bool = True,
51
+ ) -> None:
52
+ self.pipeline = pipeline
53
+ self.require_english_audio = require_english_audio
54
+ self.allow_english_subs = allow_english_subs
55
+ self.reject_dead_swarms = reject_dead_swarms
56
+ self.allowed_categories = set(allowed_categories) if allowed_categories else None
57
+ self.use_zero_cue_catalog = use_zero_cue_catalog
58
+
59
+ @staticmethod
60
+ def _extract_item_row(item_elem: ET.Element) -> Dict[str, Any]:
61
+ title = (item_elem.findtext("title") or "").strip()
62
+ size_text = item_elem.findtext("size") or "0"
63
+ try:
64
+ size_val = int(size_text)
65
+ except ValueError:
66
+ size_val = 0
67
+
68
+ attrs: Dict[str, str] = {}
69
+ for attr in item_elem.findall(f"{{{TORZNAB_NS}}}attr"):
70
+ name = (attr.attrib.get("name") or "").lower().strip()
71
+ val = attr.attrib.get("value") or ""
72
+ if name:
73
+ attrs[name] = val
74
+
75
+ seeders = int(attrs.get("seeders", "0") or 0)
76
+ peers = int(attrs.get("peers", "0") or 0)
77
+ leechers = max(peers - seeders, int(attrs.get("leechers", "0") or 0))
78
+ grabs = int(attrs["grabs"]) if "grabs" in attrs and attrs["grabs"].strip() else None
79
+ infohash = attrs.get("infohash") or ""
80
+
81
+ files_attr = attrs.get("files") or ""
82
+ if files_attr:
83
+ files = [{"path": p.strip(), "size": size_val} for p in files_attr.split(";") if p.strip()]
84
+ else:
85
+ tail = title.rsplit(".", 1)[-1].lower() if "." in title else ""
86
+ known_exts = {"mkv", "mp4", "avi", "flac", "mp3", "m4b", "epub", "pdf", "cbz", "cbr", "iso", "exe", "scr", "vbs", "bat", "cmd", "lnk", "ps1", "msi"}
87
+ if tail in known_exts:
88
+ files = [{"path": title, "size": size_val}]
89
+ else:
90
+ low = title.lower()
91
+ if any(k in low for k in ("flac", "24bit", "lossless")):
92
+ synth_ext = ".flac"
93
+ elif any(k in low for k in ("m4b", "audiobook", "unabridged")):
94
+ synth_ext = ".m4b"
95
+ elif any(k in low for k in ("epub", "ebook")):
96
+ synth_ext = ".epub"
97
+ elif any(k in low for k in ("cbz", "cbr")):
98
+ synth_ext = ".cbz"
99
+ else:
100
+ synth_ext = ".mkv"
101
+ files = [{"path": f"{title}{synth_ext}", "size": size_val}]
102
+
103
+ return {
104
+ "infohash": infohash,
105
+ "name": title,
106
+ "size": size_val,
107
+ "files": files,
108
+ "swarm": {
109
+ "tracker_known": bool(seeders > 0 or leechers > 0 or "seeders" in attrs),
110
+ "tracker_seeders": seeders,
111
+ "tracker_leechers": leechers,
112
+ "tracker_completed": grabs,
113
+ },
114
+ }
115
+
116
+ def should_admit(self, pred: ArmigerUltraPrediction) -> Tuple[bool, str]:
117
+ """Evaluate whether a Torznab release should be admitted to Sonarr/Radarr/Lidarr/Readarr."""
118
+ if not pred.csam_safety["is_safe"]:
119
+ return False, str(pred.csam_safety.get("reason") or "csam_or_malware_blocked")
120
+ if pred.honeypot_trap["is_honeypot_trap"]:
121
+ return False, "honeypot_trap_blocked"
122
+ if pred.junk == "junk" or not pred.mondrian_gates["passed_all"]:
123
+ return False, str(pred.mondrian_gates.get("safety_gate_reason") or "mondrian_gate_blocked")
124
+
125
+ if self.reject_dead_swarms and pred.swarm_viability["viability_tier"] == "dead":
126
+ return False, "dead_swarm_blocked"
127
+
128
+ if self.allowed_categories and pred.category not in self.allowed_categories:
129
+ return False, f"category_mismatch:{pred.category}"
130
+
131
+ if self.require_english_audio and pred.lang == "non_english":
132
+ has_subs = pred.english_subs == "has_eng_subs"
133
+ if not (self.allow_english_subs and has_subs):
134
+ return False, "non_english_audio_blocked"
135
+
136
+ return True, "admitted"
137
+
138
+ def filter_torznab_xml(self, xml_bytes: bytes) -> Tuple[bytes, Dict[str, Any]]:
139
+ """Parse upstream Torznab XML, remove blocked `<item>` nodes, and enrich surviving items."""
140
+ root = ET.fromstring(xml_bytes)
141
+ channel = root.find("channel")
142
+ if channel is None:
143
+ return xml_bytes, {"total": 0, "admitted": 0, "blocked": 0, "blocked_reasons": {}}
144
+
145
+ items = channel.findall("item")
146
+ if not items:
147
+ return xml_bytes, {"total": 0, "admitted": 0, "blocked": 0, "blocked_reasons": {}}
148
+
149
+ rows = [self._extract_item_row(it) for it in items]
150
+ predictions = self.pipeline.classify_batch(
151
+ rows,
152
+ use_zero_cue_catalog=self.use_zero_cue_catalog,
153
+ )
154
+
155
+ admitted_count = 0
156
+ blocked_count = 0
157
+ blocked_reasons: Dict[str, int] = {}
158
+
159
+ for item_elem, pred in zip(items, predictions):
160
+ admit, reason = self.should_admit(pred)
161
+ if not admit:
162
+ channel.remove(item_elem)
163
+ blocked_count += 1
164
+ blocked_reasons[reason] = blocked_reasons.get(reason, 0) + 1
165
+ continue
166
+
167
+ admitted_count += 1
168
+ for attr_name, attr_val in (
169
+ ("armiger_category", pred.category),
170
+ ("armiger_lang", pred.lang),
171
+ ("armiger_viability", str(pred.swarm_viability["viability_tier"])),
172
+ ("armiger_survival_prob", f"{pred.swarm_viability['survival_net_prob']:.4f}"),
173
+ ):
174
+ attr_el = ET.SubElement(item_elem, f"{{{TORZNAB_NS}}}attr")
175
+ attr_el.set("name", attr_name)
176
+ attr_el.set("value", attr_val)
177
+
178
+ filtered_bytes = ET.tostring(root, encoding="utf-8", xml_declaration=True)
179
+ stats = {
180
+ "total": len(items),
181
+ "admitted": admitted_count,
182
+ "blocked": blocked_count,
183
+ "blocked_reasons": blocked_reasons,
184
+ }
185
+ return filtered_bytes, stats
186
+
187
+
188
+ def create_proxy_server(host: str = "localhost", port: int = 8097) -> ThreadingHTTPServer:
189
+ model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
190
+ variant = os.environ.get("ARMIGER_VARIANT", "int8")
191
+ upstream_url = os.environ.get("UPSTREAM_TORZNAB_URL", "http://dh-waypoint:3333/torznab").rstrip("/")
192
+ pipeline = ArmigerUltraPipeline.from_directory(
193
+ model_dir,
194
+ variant=variant,
195
+ rsi_mode=DHTRSIMode(os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")),
196
+ preload_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
197
+ )
198
+ torznab_filter = ArmigerTorznabFilter(
199
+ pipeline,
200
+ require_english_audio=os.environ.get("ARMIGER_REQUIRE_ENGLISH_AUDIO", "false").lower() == "true",
201
+ allow_english_subs=os.environ.get("ARMIGER_ALLOW_ENGLISH_SUBS", "true").lower() == "true",
202
+ reject_dead_swarms=os.environ.get("ARMIGER_REJECT_DEAD_SWARMS", "true").lower() == "true",
203
+ )
204
+
205
+ class ProxyHandler(BaseHTTPRequestHandler):
206
+ def do_GET(self) -> None:
207
+ if self.path == "/healthz":
208
+ body = json.dumps({"status": "ok", "service": "armiger-torznab-proxy", "variant": variant}).encode("utf-8")
209
+ self.send_response(200)
210
+ self.send_header("Content-Type", "application/json")
211
+ self.send_header("Content-Length", str(len(body)))
212
+ self.end_headers()
213
+ self.wfile.write(body)
214
+ return
215
+
216
+ parsed = urllib.parse.urlsplit(self.path)
217
+ params = urllib.parse.parse_qs(parsed.query)
218
+ t_param = (params.get("t") or [""])[0].lower()
219
+ target = f"{upstream_url}{parsed.path}"
220
+ if parsed.query:
221
+ target = f"{target}?{parsed.query}"
222
+
223
+ req = urllib.request.Request(target, method="GET")
224
+ with urllib.request.urlopen(req, timeout=15.0) as resp:
225
+ raw_xml = resp.read()
226
+ content_type = resp.headers.get("Content-Type", "application/rss+xml; charset=utf-8")
227
+
228
+ if t_param in ("caps", ""):
229
+ out_bytes = raw_xml
230
+ stats = {"total": 0, "admitted": 0, "blocked": 0}
231
+ else:
232
+ out_bytes, stats = torznab_filter.filter_torznab_xml(raw_xml)
233
+
234
+ self.send_response(200)
235
+ self.send_header("Content-Type", content_type)
236
+ self.send_header("X-Armiger-Total", str(stats["total"]))
237
+ self.send_header("X-Armiger-Admitted", str(stats["admitted"]))
238
+ self.send_header("X-Armiger-Blocked", str(stats["blocked"]))
239
+ self.send_header("Content-Length", str(len(out_bytes)))
240
+ self.end_headers()
241
+ self.wfile.write(out_bytes)
242
+
243
+ def log_message(self, fmt: str, *args: Any) -> None:
244
+ return
245
+
246
+ return ThreadingHTTPServer((host, port), ProxyHandler)
247
+
248
+
249
+ if __name__ == "__main__":
250
+ listen_host = os.environ.get("ARMIGER_PROXY_HOST", "localhost")
251
+ listen_port = int(os.environ.get("ARMIGER_PROXY_PORT", "8097"))
252
+ srv = create_proxy_server(listen_host, listen_port)
253
+ print(f"Armiger Torznab proxy listening on {listen_host}:{listen_port}")
254
+ srv.serve_forever()
examples/arr-stack/docker-compose.yml ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ services:
2
+ armiger-torznab-proxy:
3
+ image: python:3.12-slim
4
+ container_name: armiger-torznab-proxy
5
+ restart: unless-stopped
6
+ working_dir: /app
7
+ ports:
8
+ - "8097:8097"
9
+ environment:
10
+ ARMIGER_MODEL_DIR: /app
11
+ ARMIGER_VARIANT: "${ARMIGER_VARIANT:-int8}"
12
+ UPSTREAM_TORZNAB_URL: "${UPSTREAM_TORZNAB_URL:-http://prowlarr:9696/1/api}"
13
+ ARMIGER_PROXY_HOST: "localhost"
14
+ ARMIGER_PROXY_PORT: "8097"
15
+ ARMIGER_REQUIRE_ENGLISH_AUDIO: "${ARMIGER_REQUIRE_ENGLISH_AUDIO:-false}"
16
+ ARMIGER_ALLOW_ENGLISH_SUBS: "${ARMIGER_ALLOW_ENGLISH_SUBS:-true}"
17
+ ARMIGER_REJECT_DEAD_SWARMS: "${ARMIGER_REJECT_DEAD_SWARMS:-true}"
18
+ ARMIGER_USE_ZERO_CUE_CATALOG: "true"
19
+ volumes:
20
+ - ../..:/app:ro
21
+ command:
22
+ - sh
23
+ - -c
24
+ - "pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/arr-stack/armiger_torznab_proxy.py"
25
+
26
+ armiger-arr-webhook-receiver:
27
+ image: python:3.12-slim
28
+ container_name: armiger-arr-webhook-receiver
29
+ restart: unless-stopped
30
+ working_dir: /app
31
+ ports:
32
+ - "8098:8098"
33
+ environment:
34
+ ARMIGER_MODEL_DIR: /app
35
+ ARMIGER_VARIANT: "int8"
36
+ ARMIGER_WEBHOOK_HOST: "localhost"
37
+ ARMIGER_WEBHOOK_PORT: "8098"
38
+ ARMIGER_RUN_FFPROBE: "true"
39
+ ARMIGER_DHT_RSI_MODE: "${ARMIGER_DHT_RSI_MODE:-pull_only}"
40
+ volumes:
41
+ - ../..:/app:ro
42
+ - media_library:/data/media:ro
43
+ command:
44
+ - sh
45
+ - -c
46
+ - "apt-get update && apt-get install -y --no-install-recommends ffmpeg && pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/arr-stack/armiger_arr_webhook_receiver.py"
47
+
48
+ volumes:
49
+ media_library:
examples/bitmagnet/bitmagnet_armiger_bridge.py ADDED
@@ -0,0 +1,266 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Bitmagnet <-> Armiger Ultra Pre-Ingest / Post-Crawl Classifier & Admission Bridge.
2
+
3
+ Provides two operational modes for upstream Bitmagnet deployments:
4
+ 1. HTTP Classifier & Admission Webhook Bridge (`POST /v1/bitmagnet/classify`):
5
+ Accepts batches of newly discovered Bitmagnet torrents, runs `ArmigerUltraPipeline.classify_batch`,
6
+ and returns Bitmagnet-compatible content types (`movie`, `tv_show`, `music`, `ebook`, `audiobook`,
7
+ `comic`, `software`, `game`, `xxx`), structured tags (`armiger:category:*`, `armiger:lang:*`,
8
+ `armiger:viability:*`), and `delete` actions for disguised malware (`.mkv.exe`, `.scr`, `.lnk`),
9
+ fake-seeder honeypot traps, or CSAM/safety gate violations.
10
+ 2. GraphQL Mutation & PostgreSQL Reconciliation Helper (`BitmagnetGraphQLSynchronizer`):
11
+ Generates Bitmagnet GraphQL ` torrentContentUpdate ` / ` torrentDelete ` mutations or SQL updates
12
+ so upstream Bitmagnet instances gain Armiger's 95.49% 11-class accuracy (vs. 35.29% built-in rules)
13
+ and 100% fake-seeder malware trap recall.
14
+ """
15
+ from __future__ import annotations
16
+
17
+ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
18
+ import json
19
+ import os
20
+ from pathlib import Path
21
+ import sys
22
+ from typing import Any, Dict, List, Mapping, Optional, Sequence
23
+ import urllib.request
24
+
25
+ ROOT_DIR = Path(__file__).resolve().parents[2]
26
+ if str(ROOT_DIR) not in sys.path:
27
+ sys.path.insert(0, str(ROOT_DIR))
28
+
29
+ from armiger_dht_rsi import DHTRSIMode
30
+ from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
31
+
32
+ # Map Armiger 11-class labels to Bitmagnet ContentType enum values
33
+ BITMAGNET_CONTENT_TYPE_MAP: Dict[str, Optional[str]] = {
34
+ "movie": "movie",
35
+ "tv_show": "tv_show",
36
+ "music": "music",
37
+ "ebook": "ebook",
38
+ "audiobook": "audiobook",
39
+ "comic": "comic",
40
+ "software": "software",
41
+ "game": "game",
42
+ "course": "ebook",
43
+ "xxx": "xxx",
44
+ "unknown": None,
45
+ }
46
+
47
+
48
+ class BitmagnetArmigerBridge:
49
+ """Classifies, filters, and enriches Bitmagnet torrent items with Armiger Ultra."""
50
+
51
+ def __init__(
52
+ self,
53
+ pipeline: ArmigerUltraPipeline,
54
+ *,
55
+ use_zero_cue_catalog: bool = True,
56
+ prune_honeypots_and_malware: bool = True,
57
+ prune_dead_swarms: bool = False,
58
+ isolate_adult_namespace: bool = True,
59
+ ) -> None:
60
+ self.pipeline = pipeline
61
+ self.use_zero_cue_catalog = use_zero_cue_catalog
62
+ self.prune_honeypots_and_malware = prune_honeypots_and_malware
63
+ self.prune_dead_swarms = prune_dead_swarms
64
+ self.isolate_adult_namespace = isolate_adult_namespace
65
+
66
+ @staticmethod
67
+ def _bitmagnet_item_to_row(item: Mapping[str, Any]) -> Dict[str, Any]:
68
+ files_in = item.get("files") if isinstance(item.get("files"), list) else []
69
+ seeders = int(item.get("seeders", item.get("tracker_seeders", 0)) or 0)
70
+ leechers = int(item.get("leechers", item.get("peers", item.get("tracker_leechers", 0))) or 0)
71
+ return {
72
+ "infohash": str(item.get("infoHash") or item.get("info_hash") or item.get("infohash") or ""),
73
+ "name": str(item.get("name") or item.get("title") or ""),
74
+ "size": int(item.get("size") or 0),
75
+ "files": [
76
+ {"path": str(f.get("path") or f.get("name") or ""), "size": int(f.get("size") or 0)}
77
+ if isinstance(f, Mapping)
78
+ else {"path": str(f), "size": 0}
79
+ for f in files_in
80
+ ],
81
+ "file_count": int(item.get("filesCount") or item.get("file_count") or len(files_in) or 1),
82
+ "swarm": {
83
+ "tracker_known": bool(seeders > 0 or leechers > 0),
84
+ "tracker_seeders": seeders,
85
+ "tracker_leechers": leechers,
86
+ "tracker_completed": int(item["completed"]) if "completed" in item and item["completed"] is not None else None,
87
+ },
88
+ }
89
+
90
+ def evaluate_batch(self, items: Sequence[Mapping[str, Any]]) -> Dict[str, Any]:
91
+ """Run Armiger Ultra on a batch of Bitmagnet torrents and return enrichment/prune actions."""
92
+ rows = [self._bitmagnet_item_to_row(it) for it in items]
93
+ predictions: List[ArmigerUltraPrediction] = self.pipeline.classify_batch(
94
+ rows,
95
+ use_zero_cue_catalog=self.use_zero_cue_catalog,
96
+ )
97
+
98
+ actions: List[Dict[str, Any]] = []
99
+ delete_infohashes: List[str] = []
100
+
101
+ for pred in predictions:
102
+ is_malware_or_unsafe = not pred.csam_safety["is_safe"] or pred.junk == "junk"
103
+ is_honeypot = bool(pred.honeypot_trap["is_honeypot_trap"])
104
+ viability_tier = str(pred.swarm_viability["viability_tier"])
105
+ is_dead = viability_tier == "dead"
106
+
107
+ if self.prune_honeypots_and_malware and (is_malware_or_unsafe or is_honeypot):
108
+ reason = (
109
+ pred.csam_safety.get("reason")
110
+ or ("honeypot_trap" if is_honeypot else "junk_or_malware_gate")
111
+ )
112
+ delete_infohashes.append(pred.infohash)
113
+ actions.append({
114
+ "infoHash": pred.infohash,
115
+ "action": "delete",
116
+ "reason": str(reason),
117
+ "category": pred.category,
118
+ "honeypot_trap": is_honeypot,
119
+ "junk": pred.junk,
120
+ })
121
+ continue
122
+
123
+ if self.prune_dead_swarms and is_dead:
124
+ delete_infohashes.append(pred.infohash)
125
+ actions.append({
126
+ "infoHash": pred.infohash,
127
+ "action": "delete",
128
+ "reason": "dead_swarm",
129
+ "category": pred.category,
130
+ "viability_tier": viability_tier,
131
+ })
132
+ continue
133
+
134
+ if self.isolate_adult_namespace and pred.adult == "adult" and pred.category != "xxx":
135
+ bitmagnet_type: Optional[str] = "xxx"
136
+ else:
137
+ bitmagnet_type = BITMAGNET_CONTENT_TYPE_MAP.get(pred.category)
138
+
139
+ tags = [
140
+ f"armiger:{pred.category}",
141
+ f"armiger:lang:{pred.lang}",
142
+ f"armiger:viability:{viability_tier}",
143
+ ]
144
+ if pred.anime == "anime":
145
+ tags.append("armiger:anime")
146
+ dub_status = pred.heads["anime"].get("anime_dub_status")
147
+ if dub_status:
148
+ tags.append(f"armiger:dub:{dub_status}")
149
+ if pred.english_subs == "has_eng_subs":
150
+ tags.append("armiger:eng_subs")
151
+
152
+ actions.append({
153
+ "infoHash": pred.infohash,
154
+ "action": "update",
155
+ "contentType": bitmagnet_type,
156
+ "armigerCategory": pred.category,
157
+ "languages": ["en"] if pred.lang == "english" else [],
158
+ "tags": tags,
159
+ "viabilityTier": viability_tier,
160
+ "viabilityProb": float(pred.swarm_viability["viability_prob"]),
161
+ "mondrianPassed": bool(pred.mondrian_gates["passed_all"]),
162
+ })
163
+
164
+ return {
165
+ "total": len(actions),
166
+ "deletedCount": len(delete_infohashes),
167
+ "deleteInfoHashes": delete_infohashes,
168
+ "actions": actions,
169
+ "graphqlDeleteMutation": self.build_graphql_delete_mutation(delete_infohashes),
170
+ }
171
+
172
+ @staticmethod
173
+ def build_graphql_delete_mutation(infohashes: Sequence[str]) -> Optional[Dict[str, Any]]:
174
+ """Build a Bitmagnet GraphQL `torrentDelete` mutation payload for blocked infohashes."""
175
+ clean = [h.strip().lower() for h in infohashes if h and h.strip()]
176
+ if not clean:
177
+ return None
178
+ return {
179
+ "query": (
180
+ "mutation ArmigerPruneMalwareAndHoneypots($infoHashes: [Hash20!]!) { "
181
+ "torrent { delete(input: { infoHashes: $infoHashes }) { infoHashes } } "
182
+ "}"
183
+ ),
184
+ "variables": {"infoHashes": clean},
185
+ }
186
+
187
+ @staticmethod
188
+ def push_graphql_delete(
189
+ graphql_endpoint: str,
190
+ infohashes: Sequence[str],
191
+ timeout: float = 5.0,
192
+ ) -> Optional[Dict[str, Any]]:
193
+ """Execute the `torrentDelete` GraphQL mutation against a running Bitmagnet endpoint."""
194
+ mutation = BitmagnetArmigerBridge.build_graphql_delete_mutation(infohashes)
195
+ if mutation is None:
196
+ return None
197
+ req = urllib.request.Request(
198
+ graphql_endpoint,
199
+ data=json.dumps(mutation).encode("utf-8"),
200
+ headers={"Content-Type": "application/json"},
201
+ method="POST",
202
+ )
203
+ with urllib.request.urlopen(req, timeout=timeout) as resp:
204
+ return json.loads(resp.read().decode("utf-8"))
205
+
206
+
207
+ def create_server(host: str = "localhost", port: int = 8096) -> ThreadingHTTPServer:
208
+ model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
209
+ variant = os.environ.get("ARMIGER_VARIANT", "fp32")
210
+ pipeline = ArmigerUltraPipeline.from_directory(
211
+ model_dir,
212
+ variant=variant,
213
+ rsi_mode=DHTRSIMode(os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")),
214
+ preload_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
215
+ )
216
+ bridge = BitmagnetArmigerBridge(
217
+ pipeline,
218
+ use_zero_cue_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
219
+ prune_honeypots_and_malware=os.environ.get("ARMIGER_PRUNE_MALWARE", "true").lower() == "true",
220
+ prune_dead_swarms=os.environ.get("ARMIGER_PRUNE_DEAD_SWARMS", "false").lower() == "true",
221
+ )
222
+
223
+ class Handler(BaseHTTPRequestHandler):
224
+ def _send_json(self, status: int, payload: Dict[str, Any]) -> None:
225
+ body = json.dumps(payload).encode("utf-8")
226
+ self.send_response(status)
227
+ self.send_header("Content-Type", "application/json")
228
+ self.send_header("Content-Length", str(len(body)))
229
+ self.end_headers()
230
+ self.wfile.write(body)
231
+
232
+ def do_GET(self) -> None:
233
+ if self.path == "/healthz":
234
+ self._send_json(200, {"status": "ok", "service": "bitmagnet-armiger-bridge", "variant": variant})
235
+ else:
236
+ self._send_json(404, {"error": "not_found"})
237
+
238
+ def do_POST(self) -> None:
239
+ length = int(self.headers.get("Content-Length", "0"))
240
+ raw = self.rfile.read(length) if length > 0 else b"{}"
241
+ payload = json.loads(raw.decode("utf-8"))
242
+ if self.path == "/v1/bitmagnet/classify":
243
+ items = payload.get("torrents") or payload.get("items") or []
244
+ result = bridge.evaluate_batch(items)
245
+ gql_url = os.environ.get("BITMAGNET_GRAPHQL_URL", "").strip()
246
+ if gql_url and result["deleteInfoHashes"]:
247
+ try:
248
+ bridge.push_graphql_delete(gql_url, result["deleteInfoHashes"])
249
+ except Exception as exc:
250
+ result["graphqlPushError"] = str(exc)
251
+ self._send_json(200, result)
252
+ else:
253
+ self._send_json(404, {"error": "not_found"})
254
+
255
+ def log_message(self, fmt: str, *args: Any) -> None:
256
+ return
257
+
258
+ return ThreadingHTTPServer((host, port), Handler)
259
+
260
+
261
+ if __name__ == "__main__":
262
+ listen_host = os.environ.get("ARMIGER_BRIDGE_HOST", "localhost")
263
+ listen_port = int(os.environ.get("ARMIGER_BRIDGE_PORT", "8096"))
264
+ srv = create_server(listen_host, listen_port)
265
+ print(f"Bitmagnet <-> Armiger Ultra bridge listening on {listen_host}:{listen_port}")
266
+ srv.serve_forever()
examples/bitmagnet/docker-compose.yml ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ services:
2
+ bitmagnet:
3
+ image: ghcr.io/bitmagnet-io/bitmagnet:latest
4
+ container_name: bitmagnet
5
+ restart: unless-stopped
6
+ depends_on:
7
+ postgres:
8
+ condition: service_healthy
9
+ ports:
10
+ - "3333:3333"
11
+ - "3334:3334/tcp"
12
+ - "3334:3334/udp"
13
+ environment:
14
+ POSTGRES_HOST: postgres
15
+ POSTGRES_NAME: bitmagnet
16
+ POSTGRES_USER: postgres
17
+ POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
18
+ command:
19
+ - worker
20
+ - run
21
+ - --keys=http_server
22
+ - --keys=queue_server
23
+ - --keys=dht_crawler
24
+
25
+ armiger-bitmagnet-bridge:
26
+ image: python:3.12-slim
27
+ container_name: armiger-bitmagnet-bridge
28
+ restart: unless-stopped
29
+ working_dir: /app
30
+ depends_on:
31
+ - bitmagnet
32
+ ports:
33
+ - "8096:8096"
34
+ environment:
35
+ ARMIGER_MODEL_DIR: /app
36
+ ARMIGER_VARIANT: "${ARMIGER_VARIANT:-fp32}"
37
+ ARMIGER_BRIDGE_HOST: "localhost"
38
+ ARMIGER_BRIDGE_PORT: "8096"
39
+ BITMAGNET_GRAPHQL_URL: "http://bitmagnet:3333/graphql"
40
+ ARMIGER_PRUNE_MALWARE: "true"
41
+ ARMIGER_PRUNE_DEAD_SWARMS: "false"
42
+ volumes:
43
+ - ../..:/app:ro
44
+ command:
45
+ - sh
46
+ - -c
47
+ - "pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/bitmagnet/bitmagnet_armiger_bridge.py"
48
+
49
+ armiger-torznab-proxy:
50
+ image: python:3.12-slim
51
+ container_name: armiger-torznab-proxy
52
+ restart: unless-stopped
53
+ working_dir: /app
54
+ depends_on:
55
+ - bitmagnet
56
+ ports:
57
+ - "8097:8097"
58
+ environment:
59
+ ARMIGER_MODEL_DIR: /app
60
+ ARMIGER_VARIANT: "${ARMIGER_VARIANT:-int8}"
61
+ UPSTREAM_TORZNAB_URL: "http://bitmagnet:3333/torznab"
62
+ ARMIGER_PROXY_PORT: "8097"
63
+ ARMIGER_REQUIRE_ENGLISH_AUDIO: "false"
64
+ ARMIGER_ALLOW_ENGLISH_SUBS: "true"
65
+ ARMIGER_REJECT_DEAD_SWARMS: "true"
66
+ volumes:
67
+ - ../..:/app:ro
68
+ command:
69
+ - sh
70
+ - -c
71
+ - "pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/arr-stack/armiger_torznab_proxy.py"
72
+
73
+ postgres:
74
+ image: postgres:16-alpine
75
+ container_name: bitmagnet-postgres
76
+ restart: unless-stopped
77
+ environment:
78
+ POSTGRES_DB: bitmagnet
79
+ POSTGRES_USER: postgres
80
+ POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
81
+ volumes:
82
+ - bitmagnet_pgdata:/var/lib/postgresql/data
83
+ healthcheck:
84
+ test: ["CMD-SHELL", "pg_isready -d bitmagnet -U postgres"]
85
+ interval: 10s
86
+ timeout: 5s
87
+ retries: 5
88
+
89
+ volumes:
90
+ bitmagnet_pgdata:
examples/dh-waypoint/07-armiger-ultra.plugin.yml ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Drop this file into DH_WAYPOINT_PROCESSORS_PLUGIN_DIR (default /etc/dh-waypoint/plugins.d).
2
+ #
3
+ # Full Armiger Ultra integration wiring:
4
+ # 1. Batch classification (`POST /v1/classify`, `dh-waypoint.processor.v1`)
5
+ # 2. Cohort admission gate (`POST /v1/admit`, `dh-waypoint.processors.admission.v1`)
6
+ # 3. 4-tier closed-loop outcome feedback (`POST /v1/feedback`, `dh-waypoint.processors.feedback.v1`)
7
+
8
+ processor:
9
+ name: armiger-ultra
10
+ endpoint: http://armiger-ultra:8095/v1/classify
11
+ timeout: 5s
12
+ max_batch: 64
13
+ required: false
14
+ forward_files: true
15
+ max_files: 64
16
+ forward_trackers: true
17
+ forward_swarm: true
18
+
19
+ admission:
20
+ enabled: true
21
+ endpoint: http://armiger-ultra:8095/v1/admit
22
+ timeout: 3s
23
+ max_batch: 64
24
+ max_files: 64
25
+ fail_open: true
26
+
27
+ learning:
28
+ enabled: true
29
+ feedback_endpoint: http://armiger-ultra:8095/v1/feedback
30
+ feedback_timeout: 5s
31
+ feedback_batch_size: 64
32
+ flush_interval: 15s
33
+ queue_size: 2048
examples/dh-waypoint/armiger_waypoint_processor.py ADDED
@@ -0,0 +1,270 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """DH-Waypoint processorapi/v1 HTTP Service for Armiger & Armiger Ultra.
2
+
3
+ Implements all three DH-Waypoint external processor contracts:
4
+ - `POST /v1/classify` (`dh-waypoint.processor.v1`)
5
+ - `POST /v1/admit` (`dh-waypoint.processors.admission.v1`)
6
+ - `POST /v1/feedback` (`dh-waypoint.processors.feedback.v1`)
7
+ - `GET /healthz`
8
+ """
9
+ from __future__ import annotations
10
+
11
+ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
12
+ import json
13
+ import os
14
+ from pathlib import Path
15
+ import sys
16
+ from typing import Any, Dict, List, Mapping, Optional
17
+
18
+ ROOT_DIR = Path(__file__).resolve().parents[2]
19
+ if str(ROOT_DIR) not in sys.path:
20
+ sys.path.insert(0, str(ROOT_DIR))
21
+
22
+ from armiger_dht_rsi import CandidateObservation, DHTRSIMode
23
+ from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
24
+
25
+ # Map Armiger 11-class labels to DH-Waypoint canonical content types
26
+ WAYPOINT_CONTENT_TYPE_MAP: Dict[str, str] = {
27
+ "movie": "movie",
28
+ "tv_show": "tv_show",
29
+ "music": "music",
30
+ "ebook": "ebook",
31
+ "audiobook": "audiobook",
32
+ "comic": "comic",
33
+ "software": "software",
34
+ "game": "game",
35
+ "course": "ebook",
36
+ "xxx": "xxx",
37
+ "unknown": "unknown",
38
+ }
39
+
40
+
41
+ class ArmigerWaypointAdapter:
42
+ """Translates between DH-Waypoint processorapi/v1 envelopes and ArmigerUltraPipeline."""
43
+
44
+ def __init__(
45
+ self,
46
+ pipeline: ArmigerUltraPipeline,
47
+ *,
48
+ use_zero_cue_catalog: bool = True,
49
+ reject_dead_swarms: bool = False,
50
+ ) -> None:
51
+ self.pipeline = pipeline
52
+ self.use_zero_cue_catalog = use_zero_cue_catalog
53
+ self.reject_dead_swarms = reject_dead_swarms
54
+
55
+ @staticmethod
56
+ def _torrent_to_row(t: Mapping[str, Any]) -> Dict[str, Any]:
57
+ swarm_in = t.get("swarm") if isinstance(t.get("swarm"), Mapping) else {}
58
+ seeders = int(swarm_in.get("seeders", t.get("seeders", 0)) or 0)
59
+ leechers = int(swarm_in.get("leechers", t.get("leechers", 0)) or 0)
60
+ files = t.get("files") if isinstance(t.get("files"), list) else []
61
+ return {
62
+ "infohash": str(t.get("info_hash") or t.get("infohash") or ""),
63
+ "name": str(t.get("name") or t.get("title") or ""),
64
+ "size": int(t.get("size") or 0),
65
+ "files": files,
66
+ "file_count": int(t.get("files_count") or len(files) or 1),
67
+ "sources": t.get("sources") if isinstance(t.get("sources"), list) else [],
68
+ "swarm": {
69
+ "tracker_known": bool(seeders > 0 or leechers > 0 or swarm_in),
70
+ "tracker_seeders": seeders,
71
+ "tracker_leechers": leechers,
72
+ "tracker_completed": int(swarm_in["completed"]) if "completed" in swarm_in and swarm_in["completed"] is not None else None,
73
+ },
74
+ }
75
+
76
+ def handle_classify_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
77
+ """Handle `POST /v1/classify` (`dh-waypoint.processor.v1`)."""
78
+ torrents = request_body.get("torrents") or []
79
+ rows = [self._torrent_to_row(t) for t in torrents]
80
+ predictions: List[ArmigerUltraPrediction] = self.pipeline.classify_batch(
81
+ rows,
82
+ use_zero_cue_catalog=self.use_zero_cue_catalog,
83
+ )
84
+
85
+ decisions: List[Dict[str, Any]] = []
86
+ for pred in predictions:
87
+ gates_passed = bool(pred.mondrian_gates["passed_all"])
88
+ viability_tier = str(pred.swarm_viability["viability_tier"])
89
+ if not gates_passed:
90
+ reason = (
91
+ pred.csam_safety.get("reason")
92
+ or pred.mondrian_gates.get("safety_gate_reason")
93
+ or ("honeypot_trap_blocked" if pred.honeypot_trap["is_honeypot_trap"] else "safety_gate_rejected")
94
+ )
95
+ decisions.append({
96
+ "info_hash": pred.infohash,
97
+ "action": "delete",
98
+ "reason": str(reason),
99
+ })
100
+ continue
101
+
102
+ content_type = WAYPOINT_CONTENT_TYPE_MAP.get(pred.category, "unknown")
103
+ tags = [
104
+ f"armiger:category:{pred.category}",
105
+ f"armiger:lang:{pred.lang}",
106
+ f"armiger:viability:{viability_tier}",
107
+ ]
108
+ if pred.anime == "anime":
109
+ tags.append("armiger:anime")
110
+ dub_status = pred.heads["anime"].get("anime_dub_status")
111
+ if dub_status:
112
+ tags.append(f"armiger:anime_dub:{dub_status}")
113
+ if pred.english_subs == "has_eng_subs":
114
+ tags.append("armiger:eng_subs")
115
+
116
+ decisions.append({
117
+ "info_hash": pred.infohash,
118
+ "action": "enrich",
119
+ "content_type": content_type,
120
+ "languages": ["en"] if pred.lang == "english" else [],
121
+ "tags": tags,
122
+ "attributes": {
123
+ "armiger.category": pred.category,
124
+ "armiger.category_confidence": f"{pred.heads['category']['confidence']:.4f}",
125
+ "armiger.lang": pred.lang,
126
+ "armiger.viability_tier": viability_tier,
127
+ "armiger.viability_prob": f"{pred.swarm_viability['viability_prob']:.4f}",
128
+ "armiger.survival_net_prob": f"{pred.swarm_viability['survival_net_prob']:.4f}",
129
+ "armiger.catalog_source": str(pred.plugin_trace["plugin_3_zero_cue_catalog"]["resolution_source"]),
130
+ },
131
+ })
132
+
133
+ return {
134
+ "schema": "dh-waypoint.processor.v1",
135
+ "decisions": decisions,
136
+ }
137
+
138
+ def handle_admission_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
139
+ """Handle `POST /v1/admit` (`dh-waypoint.processors.admission.v1`)."""
140
+ candidates = request_body.get("candidates") or []
141
+ rows = [self._torrent_to_row(c) for c in candidates]
142
+ predictions = self.pipeline.classify_batch(
143
+ rows,
144
+ use_zero_cue_catalog=self.use_zero_cue_catalog,
145
+ )
146
+
147
+ decisions: List[Dict[str, Any]] = []
148
+ for cand, pred in zip(candidates, predictions):
149
+ info_hash = str(cand.get("info_hash") or pred.infohash)
150
+ gates_passed = bool(pred.mondrian_gates["passed_all"])
151
+ viability_tier = str(pred.swarm_viability["viability_tier"])
152
+ admit = gates_passed and (not self.reject_dead_swarms or viability_tier != "dead")
153
+ reason = "passed_mondrian_conformal_gates"
154
+ if not gates_passed:
155
+ reason = str(
156
+ pred.csam_safety.get("reason")
157
+ or pred.mondrian_gates.get("safety_gate_reason")
158
+ or "honeypot_or_safety_gate_blocked"
159
+ )
160
+ elif not admit:
161
+ reason = "dead_swarm_rejected"
162
+
163
+ decisions.append({
164
+ "info_hash": info_hash,
165
+ "admit": admit,
166
+ "cohort": pred.category if admit else "rejected",
167
+ "score": float(pred.swarm_viability["viability_prob"] if admit else 0.0),
168
+ "reason": reason,
169
+ })
170
+
171
+ return {
172
+ "schema": "dh-waypoint.processors.admission.v1",
173
+ "decisions": decisions,
174
+ }
175
+
176
+ def handle_feedback_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
177
+ """Handle `POST /v1/feedback` (`dh-waypoint.processors.feedback.v1`)."""
178
+ events = request_body.get("events") or []
179
+ accepted = 0
180
+ verified = 0
181
+ for ev in events:
182
+ accepted += 1
183
+ tier = str(ev.get("tier") or "")
184
+ outcome = str(ev.get("outcome") or "")
185
+ if tier in ("t3_files", "t4_playback") and outcome in ("verified", "imported", "completed"):
186
+ cand = CandidateObservation(
187
+ source_adapter="ffprobe",
188
+ raw_title=str(ev.get("name") or ev.get("info_hash") or "verified_item"),
189
+ claimed_category=str(ev.get("content_type") or "movie"),
190
+ download_completed=True,
191
+ bytes_downloaded=int(ev.get("size") or 100_000_000),
192
+ total_bytes=int(ev.get("size") or 100_000_000),
193
+ catalog_id_type="movie_catalog" if ev.get("content_type") == "movie" else "series_catalog",
194
+ catalog_id_value=str(ev.get("info_hash") or "verified"),
195
+ stream_metadata={
196
+ "video_codec": str((ev.get("attributes") or {}).get("video_codec", "hevc")),
197
+ "audio_codec": str((ev.get("attributes") or {}).get("audio_codec", "eac3")),
198
+ "audio_languages": [str((ev.get("attributes") or {}).get("audio_lang", "eng"))],
199
+ },
200
+ )
201
+ ok, _, _ = self.pipeline.ingest_rsi_feedback(cand)
202
+ if ok:
203
+ verified += 1
204
+
205
+ return {
206
+ "schema": "dh-waypoint.processors.feedback.v1",
207
+ "accepted": accepted,
208
+ "verified": verified,
209
+ }
210
+
211
+
212
+ def create_server(host: str = "localhost", port: int = 8095) -> ThreadingHTTPServer:
213
+ model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
214
+ variant = os.environ.get("ARMIGER_VARIANT", "fp32")
215
+ rsi_mode_str = os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")
216
+ pipeline = ArmigerUltraPipeline.from_directory(
217
+ model_dir,
218
+ variant=variant,
219
+ adult_threshold=float(os.environ.get("ARMIGER_ADULT_THRESHOLD", "0.35")),
220
+ junk_threshold=float(os.environ.get("ARMIGER_JUNK_THRESHOLD", "0.25")),
221
+ scope_threshold=float(os.environ.get("ARMIGER_SCOPE_THRESHOLD", "0.35")),
222
+ rsi_mode=DHTRSIMode(rsi_mode_str),
223
+ preload_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
224
+ )
225
+ adapter = ArmigerWaypointAdapter(
226
+ pipeline,
227
+ use_zero_cue_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
228
+ reject_dead_swarms=os.environ.get("ARMIGER_REJECT_DEAD_SWARMS", "false").lower() == "true",
229
+ )
230
+
231
+ class Handler(BaseHTTPRequestHandler):
232
+ def _send_json(self, status: int, payload: Dict[str, Any]) -> None:
233
+ body = json.dumps(payload).encode("utf-8")
234
+ self.send_response(status)
235
+ self.send_header("Content-Type", "application/json")
236
+ self.send_header("Content-Length", str(len(body)))
237
+ self.end_headers()
238
+ self.wfile.write(body)
239
+
240
+ def do_GET(self) -> None:
241
+ if self.path == "/healthz":
242
+ self._send_json(200, {"status": "ok", "engine": "armiger-ultra", "variant": variant})
243
+ else:
244
+ self._send_json(404, {"error": "not_found"})
245
+
246
+ def do_POST(self) -> None:
247
+ length = int(self.headers.get("Content-Length", "0"))
248
+ raw = self.rfile.read(length) if length > 0 else b"{}"
249
+ payload = json.loads(raw.decode("utf-8"))
250
+ if self.path == "/v1/classify":
251
+ self._send_json(200, adapter.handle_classify_batch(payload))
252
+ elif self.path == "/v1/admit":
253
+ self._send_json(200, adapter.handle_admission_batch(payload))
254
+ elif self.path == "/v1/feedback":
255
+ self._send_json(200, adapter.handle_feedback_batch(payload))
256
+ else:
257
+ self._send_json(404, {"error": "not_found"})
258
+
259
+ def log_message(self, fmt: str, *args: Any) -> None:
260
+ return
261
+
262
+ return ThreadingHTTPServer((host, port), Handler)
263
+
264
+
265
+ if __name__ == "__main__":
266
+ listen_host = os.environ.get("ARMIGER_HOST", "localhost")
267
+ listen_port = int(os.environ.get("ARMIGER_PORT", "8095"))
268
+ srv = create_server(listen_host, listen_port)
269
+ print(f"Armiger DH-Waypoint processor listening on {listen_host}:{listen_port}")
270
+ srv.serve_forever()
examples/dh-waypoint/docker-compose.yml ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ services:
2
+ dh-waypoint:
3
+ image: ghcr.io/spencernorton/dh-waypoint:v5.6.0
4
+ container_name: dh-waypoint
5
+ restart: unless-stopped
6
+ depends_on:
7
+ postgres:
8
+ condition: service_healthy
9
+ armiger-ultra:
10
+ condition: service_healthy
11
+ ports:
12
+ - "3333:3333"
13
+ - "3334:3334/tcp"
14
+ - "3334:3334/udp"
15
+ environment:
16
+ POSTGRES_HOST: postgres
17
+ POSTGRES_PORT: "5432"
18
+ POSTGRES_DB: dh_waypoint
19
+ POSTGRES_USER: postgres
20
+ POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
21
+ DH_WAYPOINT_PROCESSORS_PLUGIN_DIR: /etc/dh-waypoint/plugins.d
22
+ volumes:
23
+ - ./07-armiger-ultra.plugin.yml:/etc/dh-waypoint/plugins.d/07-armiger-ultra.plugin.yml:ro
24
+ command:
25
+ - worker
26
+ - run
27
+ - --keys=http_server
28
+ - --keys=queue_server
29
+ - --keys=dht_crawler
30
+
31
+ armiger-ultra:
32
+ image: python:3.12-slim
33
+ container_name: armiger-ultra
34
+ restart: unless-stopped
35
+ working_dir: /app
36
+ ports:
37
+ - "8095:8095"
38
+ environment:
39
+ ARMIGER_MODEL_DIR: /app
40
+ ARMIGER_VARIANT: "${ARMIGER_VARIANT:-fp32}"
41
+ ARMIGER_PORT: "8095"
42
+ ARMIGER_ADULT_THRESHOLD: "0.35"
43
+ ARMIGER_JUNK_THRESHOLD: "0.25"
44
+ ARMIGER_SCOPE_THRESHOLD: "0.35"
45
+ ARMIGER_USE_ZERO_CUE_CATALOG: "true"
46
+ ARMIGER_DHT_RSI_MODE: "${ARMIGER_DHT_RSI_MODE:-pull_only}"
47
+ volumes:
48
+ - ../..:/app:ro
49
+ - armiger_state:/var/lib/armiger
50
+ command:
51
+ - sh
52
+ - -c
53
+ - "pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/dh-waypoint/armiger_waypoint_processor.py"
54
+ healthcheck:
55
+ test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8095/healthz')"]
56
+ interval: 15s
57
+ timeout: 5s
58
+ retries: 5
59
+
60
+ postgres:
61
+ image: postgres:16-alpine
62
+ container_name: dh-waypoint-postgres
63
+ restart: unless-stopped
64
+ environment:
65
+ POSTGRES_DB: dh_waypoint
66
+ POSTGRES_USER: postgres
67
+ POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
68
+ volumes:
69
+ - dh_waypoint_pgdata:/var/lib/postgresql/data
70
+ healthcheck:
71
+ test: ["CMD-SHELL", "pg_isready -d dh_waypoint -U postgres"]
72
+ interval: 10s
73
+ timeout: 5s
74
+ retries: 5
75
+
76
+ volumes:
77
+ armiger_state:
78
+ dh_waypoint_pgdata: