docs: add DH-Waypoint, Bitmagnet, and *Arr ecosystem integration guide and examples
Browse files- README.md +12 -0
- docs/INTEGRATION_GUIDE.md +282 -0
- examples/README.md +11 -0
- examples/arr-stack/armiger_arr_webhook_receiver.py +197 -0
- examples/arr-stack/armiger_torznab_proxy.py +254 -0
- examples/arr-stack/docker-compose.yml +49 -0
- examples/bitmagnet/bitmagnet_armiger_bridge.py +266 -0
- examples/bitmagnet/docker-compose.yml +90 -0
- examples/dh-waypoint/07-armiger-ultra.plugin.yml +33 -0
- examples/dh-waypoint/armiger_waypoint_processor.py +270 -0
- examples/dh-waypoint/docker-compose.yml +78 -0
README.md
CHANGED
|
@@ -62,6 +62,18 @@ print(pred["heads"].keys()) # dict_keys(['category', 'adult', 'junk', 'anime
|
|
| 62 |
|
| 63 |
---
|
| 64 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
## 8 Simultaneous Classification Heads (Audited Unmutated Ground-Truth Accuracy)
|
| 66 |
|
| 67 |
All metrics below are reproducibly computed using `armiger_inference.py` against the **original, unmutated ground-truth evaluation keys** (`rows.y`), separating **Standalone Neural ONNX (`armiger.onnx`)** from **Fair Blind Hybrid (`+ ZeroCueCatalogPlugin` & `AudioLanguageGuard`)**:
|
|
|
|
| 62 |
|
| 63 |
---
|
| 64 |
|
| 65 |
+
## Ecosystem Integrations (`DH-Waypoint`, `Bitmagnet` & `*Arr` Stack)
|
| 66 |
+
|
| 67 |
+
Armiger and Armiger Ultra ship with turnkey HTTP services, Docker Compose stacks, and reference adapters in [`examples/`](examples) and [`docs/INTEGRATION_GUIDE.md`](docs/INTEGRATION_GUIDE.md):
|
| 68 |
+
|
| 69 |
+
- **[`DH-Waypoint` Integration (`examples/dh-waypoint/`)](examples/dh-waypoint)** — Drop-in `processorapi/v1` service (`armiger_waypoint_processor.py`) and plugin manifest (`07-armiger-ultra.plugin.yml`) implementing batch classification (`dh-waypoint.processor.v1`), pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop telemetry for [`spencernorton/DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint).
|
| 70 |
+
- **[`Bitmagnet` Integration (`examples/bitmagnet/`)](examples/bitmagnet)** — Pre-ingest / post-crawl batch classifier and malware/honeypot pruner (`bitmagnet_armiger_bridge.py`) that upgrades upstream Bitmagnet from `35.29%` rule accuracy to **`95.49%` 11-class neural accuracy** and automatically prunes disguised `.mkv.exe`/`.scr`/`.lnk` malware traps via GraphQL `torrent.delete`.
|
| 71 |
+
- **[`*Arr` Stack & `qBittorrent` Integration (`examples/arr-stack/`)](examples/arr-stack)** — Pre-grab Torznab XML filtering proxy (`armiger_torznab_proxy.py`) between `Prowlarr` / `Bitmagnet` / `DH-Waypoint` and `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, and `Whisparr`, paired with a post-import webhook & `ffprobe` ground-truth receiver (`armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier`.
|
| 72 |
+
|
| 73 |
+
See the **[Complete Ecosystem Integration Guide (`docs/INTEGRATION_GUIDE.md`)](docs/INTEGRATION_GUIDE.md)** for architecture diagrams, Docker Compose files, and step-by-step setup instructions.
|
| 74 |
+
|
| 75 |
+
---
|
| 76 |
+
|
| 77 |
## 8 Simultaneous Classification Heads (Audited Unmutated Ground-Truth Accuracy)
|
| 78 |
|
| 79 |
All metrics below are reproducibly computed using `armiger_inference.py` against the **original, unmutated ground-truth evaluation keys** (`rows.y`), separating **Standalone Neural ONNX (`armiger.onnx`)** from **Fair Blind Hybrid (`+ ZeroCueCatalogPlugin` & `AudioLanguageGuard`)**:
|
docs/INTEGRATION_GUIDE.md
ADDED
|
@@ -0,0 +1,282 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Armiger & Armiger Ultra Ecosystem Integration Guide
|
| 2 |
+
|
| 3 |
+
This guide provides step-by-step instructions, turnkey Docker Compose stacks, and reference HTTP adapters for integrating **Armiger** (`8.3 MB INT8 CPU`) and **Armiger Ultra** (`32.3 MB FP32 GPU/CPU` + 6 official plugins) with:
|
| 4 |
+
|
| 5 |
+
1. **[Part A — Integrating with `DH-Waypoint`](#part-a--integrating-armiger-with-dh-waypoint)** (`https://github.com/spencernorton/DH-Waypoint`):
|
| 6 |
+
- Native `dh-waypoint.processor.v1` batch classification (`POST /v1/classify`)
|
| 7 |
+
- Pre-ingest `dh-waypoint.processors.admission.v1` cohort admission gate (`POST /v1/admit`)
|
| 8 |
+
- Closed-loop 4-tier `dh-waypoint.processors.feedback.v1` outcome telemetry (`POST /v1/feedback`)
|
| 9 |
+
2. **[Part B — Integrating with `Bitmagnet`](#part-b--integrating-armiger-with-bitmagnet)**:
|
| 10 |
+
- Pre-ingest / post-crawl batch classifier & malware/honeypot pruner (`examples/bitmagnet/bitmagnet_armiger_bridge.py`)
|
| 11 |
+
- GraphQL `torrentDelete` / content enrichment bridge & Torznab proxy enrichment
|
| 12 |
+
3. **[Part C — Integrating with the `*Arr` Stack (`Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `Whisparr`) & `qBittorrent`](#part-c--integrating-armiger-with-the-arr-stack--qbittorrent)**:
|
| 13 |
+
- Pre-grab Torznab filtering proxy (`examples/arr-stack/armiger_torznab_proxy.py`) blocking disguised `.exe`/`.scr`/`.lnk` malware, fake-seeder honeypot traps, dead swarms, and unrequested foreign-language audio tracks before grab
|
| 14 |
+
- Post-import webhook & `ffprobe` physical ground-truth receiver (`examples/arr-stack/armiger_arr_webhook_receiver.py`) wired into `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`, `QBittorrentAdapter`, and `PhysicalGroundTruthVerifier` for local conformal calibration and opt-in anonymous `BEP-44`/`BEP-46` RSI knowledge sharing
|
| 15 |
+
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## Architecture Overview
|
| 19 |
+
|
| 20 |
+
```mermaid
|
| 21 |
+
flowchart LR
|
| 22 |
+
subgraph Indexers["DHT Indexers & Crawlers"]
|
| 23 |
+
DHW["DH-Waypoint Core<br/>(processorapi/v1)"]
|
| 24 |
+
BM["Bitmagnet<br/>(GraphQL + Torznab)"]
|
| 25 |
+
PR["Prowlarr<br/>(Indexer Manager)"]
|
| 26 |
+
end
|
| 27 |
+
|
| 28 |
+
subgraph ArmigerSuite["Armiger / Armiger Ultra"]
|
| 29 |
+
PROC["Armiger Waypoint Processor<br/>(/v1/classify, /v1/admit, /v1/feedback)"]
|
| 30 |
+
BMBR["Bitmagnet Armiger Bridge<br/>(/v1/bitmagnet/classify)"]
|
| 31 |
+
TPROXY["Armiger Torznab Proxy<br/>(Port 8097 Admission Gate)"]
|
| 32 |
+
WHOOK["Armiger Webhook & ffprobe Receiver<br/>(Port 8098 Ground Truth + BEP-44 RSI)"]
|
| 33 |
+
end
|
| 34 |
+
|
| 35 |
+
subgraph ArrStack["*Arr Media Managers & Download Client"]
|
| 36 |
+
SON["Sonarr / Radarr / Lidarr / Readarr"]
|
| 37 |
+
QBT["qBittorrent / Transmission"]
|
| 38 |
+
end
|
| 39 |
+
|
| 40 |
+
DHW <-->|"HTTP processorapi/v1"| PROC
|
| 41 |
+
BM <-->|"Batch Classify + GraphQL Prune"| BMBR
|
| 42 |
+
DHW -->|"Torznab Feed"| TPROXY
|
| 43 |
+
BM -->|"Torznab Feed"| TPROXY
|
| 44 |
+
PR -->|"Torznab Feed"| TPROXY
|
| 45 |
+
TPROXY -->|"Sanitized & Enriched XML"| SON
|
| 46 |
+
SON -->|"Grab Approved Release"| QBT
|
| 47 |
+
SON -->|"On Import Webhook"| WHOOK
|
| 48 |
+
QBT -->|"On Completion + ffprobe"| WHOOK
|
| 49 |
+
WHOOK -.->|"Local Calibration & Opt-In BEP-44 RSI"| PROC
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
---
|
| 53 |
+
|
| 54 |
+
## Part A — Integrating Armiger with `DH-Waypoint`
|
| 55 |
+
|
| 56 |
+
[`DH-Waypoint`](https://github.com/spencernorton/DH-Waypoint) includes a built-in external processor subsystem (`processorapi/v1`) that connects directly to [`examples/dh-waypoint/armiger_waypoint_processor.py`](../examples/dh-waypoint/armiger_waypoint_processor.py).
|
| 57 |
+
|
| 58 |
+
### 1. Turnkey Docker Compose Stack
|
| 59 |
+
|
| 60 |
+
Launch `DH-Waypoint`, `PostgreSQL`, and `Armiger Ultra` together using [`examples/dh-waypoint/docker-compose.yml`](../examples/dh-waypoint/docker-compose.yml):
|
| 61 |
+
|
| 62 |
+
```bash
|
| 63 |
+
export POSTGRES_PASSWORD="<postgres-password>"
|
| 64 |
+
docker compose -f examples/dh-waypoint/docker-compose.yml up -d
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
### 2. Drop-In Plugin Manifest (`07-armiger-ultra.plugin.yml`)
|
| 68 |
+
|
| 69 |
+
Mount [`examples/dh-waypoint/07-armiger-ultra.plugin.yml`](../examples/dh-waypoint/07-armiger-ultra.plugin.yml) into `DH_WAYPOINT_PROCESSORS_PLUGIN_DIR` (`/etc/dh-waypoint/plugins.d/07-armiger-ultra.plugin.yml`):
|
| 70 |
+
|
| 71 |
+
```yaml
|
| 72 |
+
processor:
|
| 73 |
+
name: armiger-ultra
|
| 74 |
+
endpoint: http://armiger-ultra:8095/v1/classify
|
| 75 |
+
timeout: 5s
|
| 76 |
+
max_batch: 64
|
| 77 |
+
required: false
|
| 78 |
+
forward_files: true
|
| 79 |
+
max_files: 64
|
| 80 |
+
forward_trackers: true
|
| 81 |
+
forward_swarm: true
|
| 82 |
+
|
| 83 |
+
admission:
|
| 84 |
+
enabled: true
|
| 85 |
+
endpoint: http://armiger-ultra:8095/v1/admit
|
| 86 |
+
timeout: 3s
|
| 87 |
+
max_batch: 64
|
| 88 |
+
max_files: 64
|
| 89 |
+
fail_open: true
|
| 90 |
+
|
| 91 |
+
learning:
|
| 92 |
+
enabled: true
|
| 93 |
+
feedback_endpoint: http://armiger-ultra:8095/v1/feedback
|
| 94 |
+
feedback_timeout: 5s
|
| 95 |
+
feedback_batch_size: 64
|
| 96 |
+
flush_interval: 15s
|
| 97 |
+
queue_size: 2048
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
### 3. Equivalent `DH_WAYPOINT_*` Environment Variables
|
| 101 |
+
|
| 102 |
+
If you configure `DH-Waypoint` via environment variables instead of `plugins.d`:
|
| 103 |
+
|
| 104 |
+
| Variable | Recommended Value | Purpose |
|
| 105 |
+
| :--- | :--- | :--- |
|
| 106 |
+
| `DH_WAYPOINT_PROCESSORS_HTTP_CHAIN` | `armiger-ultra=http://armiger-ultra:8095/v1/classify` | Ordered HTTP classification chain (`dh-waypoint.processor.v1`). |
|
| 107 |
+
| `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_FILES` | `true` | Forwards physical file paths and byte sizes to Armiger's File-Tree Encoder. |
|
| 108 |
+
| `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_TRACKERS` | `true` | Forwards tracker domain hints to `SwarmPriors`. |
|
| 109 |
+
| `DH_WAYPOINT_PROCESSORS_HTTP_FORWARD_SWARM` | `true` | Forwards seeder/leecher counts to `SwarmSurvivalNet`. |
|
| 110 |
+
| `DH_WAYPOINT_PROCESSORS_ADMISSION_ENABLED` | `true` | Enables pre-ingest cohort admission (`dh-waypoint.processors.admission.v1`). |
|
| 111 |
+
| `DH_WAYPOINT_PROCESSORS_ADMISSION_ENDPOINT` | `http://armiger-ultra:8095/v1/admit` | Pre-ingest admission endpoint. |
|
| 112 |
+
| `DH_WAYPOINT_LEARNING_ENABLED` | `true` | Enables the 4-tier `dh-waypoint.processors.feedback.v1` outcome loop. |
|
| 113 |
+
| `DH_WAYPOINT_LEARNING_FEEDBACK_ENDPOINT` | `http://armiger-ultra:8095/v1/feedback` | Delivers `t1_discovery` through `t4_playback` feedback batches to Armiger. |
|
| 114 |
+
|
| 115 |
+
---
|
| 116 |
+
|
| 117 |
+
## Part B — Integrating Armiger with `Bitmagnet`
|
| 118 |
+
|
| 119 |
+
Upstream `Bitmagnet` uses deterministic keyword/extension rules (`35.29%` 11-class accuracy on `N=10,262` audited torrents). Deploying [`examples/bitmagnet/bitmagnet_armiger_bridge.py`](../examples/bitmagnet/bitmagnet_armiger_bridge.py) upgrades a `Bitmagnet` deployment to **`95.49%` 11-class accuracy** (`97.55%` 6-dimension composite accuracy) and **`100%` fake-seeder malware trap recall**.
|
| 120 |
+
|
| 121 |
+
### 1. Turnkey `Bitmagnet` + `Armiger` Docker Compose Stack
|
| 122 |
+
|
| 123 |
+
Start the stack in [`examples/bitmagnet/docker-compose.yml`](../examples/bitmagnet/docker-compose.yml):
|
| 124 |
+
|
| 125 |
+
```bash
|
| 126 |
+
export POSTGRES_PASSWORD="<postgres-password>"
|
| 127 |
+
docker compose -f examples/bitmagnet/docker-compose.yml up -d
|
| 128 |
+
```
|
| 129 |
+
|
| 130 |
+
This launches two companion services alongside `bitmagnet`:
|
| 131 |
+
- **`armiger-bitmagnet-bridge` (`port 8096`)**: Evaluates batches of newly crawled Bitmagnet torrents (`POST /v1/bitmagnet/classify`), maps Armiger's 11-class predictions (`movie`, `tv_show`, `music`, `ebook`, `audiobook`, `comic`, `software`, `game`, `xxx`) and tags (`armiger:<category>`, `armiger:lang:<lang>`, `armiger:viability:<tier>`) onto Bitmagnet items, and automatically executes Bitmagnet's GraphQL `torrent.delete` mutation against disguised malware payloads (`.mkv.exe`, `.scr`, `.lnk`, `.vbs`) and fake-seeder honeypots.
|
| 132 |
+
- **`armiger-torznab-proxy` (`port 8097`)**: Wraps `http://bitmagnet:3333/torznab` with real-time neural admission filtering before `Prowlarr`, `Sonarr`, or `Radarr` sees search results.
|
| 133 |
+
|
| 134 |
+
### 2. Calling the Bitmagnet Classifier Bridge Directly
|
| 135 |
+
|
| 136 |
+
```bash
|
| 137 |
+
curl -sS -X POST http://localhost:8096/v1/bitmagnet/classify \
|
| 138 |
+
-H "Content-Type: application/json" \
|
| 139 |
+
-d '{
|
| 140 |
+
"torrents": [
|
| 141 |
+
{
|
| 142 |
+
"infoHash": "1111222233334444555566667777888899990000",
|
| 143 |
+
"name": "Sintel.2010.2160p.UHD.BluRay.REMUX.HEVC.TrueHD.7.1.Atmos-OPEN",
|
| 144 |
+
"size": 74500000000,
|
| 145 |
+
"seeders": 320,
|
| 146 |
+
"leechers": 18,
|
| 147 |
+
"files": [{"path": "Sintel.2010.2160p.mkv", "size": 74500000000}]
|
| 148 |
+
},
|
| 149 |
+
{
|
| 150 |
+
"infoHash": "aaaa22223333444455556666777788889999bbbb",
|
| 151 |
+
"name": "Big.Buck.Bunny.2008.1080p.BluRay.mkv.exe",
|
| 152 |
+
"size": 15400000,
|
| 153 |
+
"seeders": 4500,
|
| 154 |
+
"leechers": 12,
|
| 155 |
+
"files": [{"path": "Big.Buck.Bunny.2008.1080p.BluRay.mkv.exe", "size": 15400000}]
|
| 156 |
+
}
|
| 157 |
+
]
|
| 158 |
+
}'
|
| 159 |
+
```
|
| 160 |
+
|
| 161 |
+
The bridge classifies the first item as `contentType: "movie"` (`viabilityTier: "alive_viable"`) and flags the second item (`Big.Buck.Bunny.2008.1080p.BluRay.mkv.exe`) for immediate deletion via `graphqlDeleteMutation`.
|
| 162 |
+
|
| 163 |
+
---
|
| 164 |
+
|
| 165 |
+
## Part C — Integrating Armiger with the `*Arr` Stack & `qBittorrent`
|
| 166 |
+
|
| 167 |
+
Armiger integrates with `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `Whisparr`, and `qBittorrent` at **two stages**:
|
| 168 |
+
1. **Stage 1 — Pre-Grab Torznab Filtering Proxy (`port 8097`):** Blocks malware masquerades, fake-seeder honeypots, dead swarms, and unrequested foreign-language audio releases *before* any `*Arr` app grabs them.
|
| 169 |
+
2. **Stage 2 — Post-Import Webhook & `ffprobe` Ground-Truth Receiver (`port 8098`):** Verifies completed downloads and `ffprobe` media streams to calibrate local Mondrian thresholds and optionally contribute differential-privacy blinded telemetry to the `BEP-44`/`BEP-46` RSI network.
|
| 170 |
+
|
| 171 |
+
### 1. Stage 1: Wiring the Torznab Filtering Proxy (`Prowlarr` / `Sonarr` / `Radarr` / `Lidarr` / `Readarr`)
|
| 172 |
+
|
| 173 |
+
Start [`examples/arr-stack/docker-compose.yml`](../examples/arr-stack/docker-compose.yml):
|
| 174 |
+
|
| 175 |
+
```bash
|
| 176 |
+
export UPSTREAM_TORZNAB_URL="http://dh-waypoint:3333/torznab"
|
| 177 |
+
docker compose -f examples/arr-stack/docker-compose.yml up -d
|
| 178 |
+
```
|
| 179 |
+
|
| 180 |
+
#### Configuring in `Prowlarr`, `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, or `Whisparr`:
|
| 181 |
+
1. Open **Settings -> Indexers -> + -> Torznab (Custom)**.
|
| 182 |
+
2. Set **URL** to `http://armiger-torznab-proxy:8097` (instead of pointing directly at `http://bitmagnet:3333/torznab` or `http://dh-waypoint:3333/torznab`).
|
| 183 |
+
3. Every Torznab search query (`?t=search`, `?t=tvsearch`, `?t=movie`, `?t=music`, `?t=book`) is transparently forwarded upstream, batch-scored by `ArmigerUltraPipeline`, stripped of unsafe/honeypot/dead/wrong-language items, and annotated with custom `<torznab:attr>` tags:
|
| 184 |
+
- `armiger_category` (`movie`, `tv_show`, `music`, `audiobook`, `ebook`, `comic`, `software`, `game`, `xxx`)
|
| 185 |
+
- `armiger_lang` (`english`, `non_english`, `multi_or_unknown`)
|
| 186 |
+
- `armiger_viability` (`alive_strong`, `alive_viable`, `marginal`, `dead`)
|
| 187 |
+
- `armiger_survival_prob` (`0.0000`–`1.0000`)
|
| 188 |
+
|
| 189 |
+
#### Torznab Proxy Environment Controls
|
| 190 |
+
|
| 191 |
+
| Variable | Default | Description |
|
| 192 |
+
| :--- | :---: | :--- |
|
| 193 |
+
| `UPSTREAM_TORZNAB_URL` | `http://dh-waypoint:3333/torznab` | Upstream Torznab base URL (`DH-Waypoint`, `Bitmagnet`, or `Prowlarr`). |
|
| 194 |
+
| `ARMIGER_VARIANT` | `int8` | `int8` (`8.3 MB` CPU fast-path, `2.44 ms` p50) or `fp32` (`32.3 MB` GPU/CPU). |
|
| 195 |
+
| `ARMIGER_REJECT_DEAD_SWARMS` | `true` | Drops Torznab items whose `SwarmSurvivalNet` viability tier is `dead`. |
|
| 196 |
+
| `ARMIGER_REQUIRE_ENGLISH_AUDIO` | `false` | When `true`, blocks releases classified as `non_english` audio by `AudioLanguageGuard`. |
|
| 197 |
+
| `ARMIGER_ALLOW_ENGLISH_SUBS` | `true` | When `ARMIGER_REQUIRE_ENGLISH_AUDIO=true`, permits foreign-audio anime/films that carry verified English subtitles (`has_eng_subs`). |
|
| 198 |
+
|
| 199 |
+
### 2. Stage 2: Wiring Post-Import Webhooks (`Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `qBittorrent`)
|
| 200 |
+
|
| 201 |
+
`armiger-arr-webhook-receiver` (`port 8098`) exposes dedicated endpoints backed by the adapters in [`armiger_dht_rsi.py`](../armiger_dht_rsi.py):
|
| 202 |
+
|
| 203 |
+
| Application | Where to Configure in UI | Webhook URL | Underlying Armiger Adapter |
|
| 204 |
+
| :--- | :--- | :--- | :--- |
|
| 205 |
+
| **Sonarr** | `Settings -> Connect -> + -> Webhook` (`On Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/sonarr` | `SeriesWebhookAdapter` |
|
| 206 |
+
| **Radarr** | `Settings -> Connect -> + -> Webhook` (`On Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/radarr` | `MovieWebhookAdapter` |
|
| 207 |
+
| **Lidarr** | `Settings -> Connect -> + -> Webhook` (`On Release Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/lidarr` | `AudioWebhookAdapter` |
|
| 208 |
+
| **Readarr** | `Settings -> Connect -> + -> Webhook` (`On Release Import`, `On Upgrade`) | `http://armiger-arr-webhook-receiver:8098/webhooks/readarr` | `PublicationWebhookAdapter` |
|
| 209 |
+
| **qBittorrent** | `Options -> Downloads -> Run external program` | `POST http://armiger-arr-webhook-receiver:8098/webhooks/qbittorrent` | `QBittorrentAdapter` |
|
| 210 |
+
|
| 211 |
+
#### `qBittorrent` Completion Command
|
| 212 |
+
|
| 213 |
+
In `qBittorrent` (`Tools -> Options -> Downloads -> Run external program on torrent completion`), configure:
|
| 214 |
+
|
| 215 |
+
```bash
|
| 216 |
+
curl -sS -X POST http://armiger-arr-webhook-receiver:8098/webhooks/qbittorrent -H "Content-Type: application/json" -d "{\"name\":\"%N\",\"hash\":\"%I\",\"content_path\":\"%F\",\"size\":%Z,\"progress\":1.0,\"state\":\"uploading\"}"
|
| 217 |
+
```
|
| 218 |
+
|
| 219 |
+
#### Programmatic Python Usage (`SeriesWebhookAdapter`, `MovieWebhookAdapter`, `QBittorrentAdapter`, `PhysicalGroundTruthVerifier`)
|
| 220 |
+
|
| 221 |
+
You can also invoke the webhook adapters and `ffprobe` verifier directly in Python via `ArmigerUltraPipeline.ingest_rsi_feedback`:
|
| 222 |
+
|
| 223 |
+
```python
|
| 224 |
+
from armiger_ultra_pipeline import ArmigerUltraPipeline
|
| 225 |
+
|
| 226 |
+
pipeline = ArmigerUltraPipeline.from_directory(".", variant="int8")
|
| 227 |
+
|
| 228 |
+
# 1. Ingest a native Sonarr "Download" (Import) webhook + ffprobe output
|
| 229 |
+
ok, reason, record = pipeline.ingest_rsi_feedback(
|
| 230 |
+
{
|
| 231 |
+
"eventType": "Download",
|
| 232 |
+
"series": {"title": "Cosmos Laundromat", "tvdbId": 310123},
|
| 233 |
+
"episodes": [{"seasonNumber": 1, "episodeNumber": 1}],
|
| 234 |
+
"release": {
|
| 235 |
+
"releaseTitle": "Cosmos.Laundromat.S01E01.1080p.WEB-DL.DDP5.1.H.264-OPEN",
|
| 236 |
+
"size": 1_450_000_000,
|
| 237 |
+
},
|
| 238 |
+
"episodeFile": {
|
| 239 |
+
"relativePath": "Season 01/Cosmos.Laundromat.S01E01.mkv",
|
| 240 |
+
"size": 1_450_000_000,
|
| 241 |
+
"mediaInfo": {
|
| 242 |
+
"videoCodec": "h264",
|
| 243 |
+
"audioFormat": "eac3",
|
| 244 |
+
"audioChannels": 5.1,
|
| 245 |
+
"audioLanguages": "eng",
|
| 246 |
+
"subtitles": "eng",
|
| 247 |
+
},
|
| 248 |
+
},
|
| 249 |
+
},
|
| 250 |
+
adapter="series_webhook",
|
| 251 |
+
)
|
| 252 |
+
assert ok and reason == "accepted_and_queued_with_poisson_jitter"
|
| 253 |
+
print(record.blinded_stem.blinded_hash, record.category, record.lang)
|
| 254 |
+
|
| 255 |
+
# 2. Ingest a native Radarr "Download" (Import) webhook
|
| 256 |
+
ok, reason, record = pipeline.ingest_rsi_feedback(
|
| 257 |
+
{
|
| 258 |
+
"eventType": "Download",
|
| 259 |
+
"movie": {"title": "Sintel", "year": 2010, "tmdbId": 45745},
|
| 260 |
+
"release": {
|
| 261 |
+
"releaseTitle": "Sintel.2010.2160p.UHD.BluRay.REMUX.HEVC.TrueHD.7.1.Atmos-OPEN",
|
| 262 |
+
"size": 24_800_000_000,
|
| 263 |
+
},
|
| 264 |
+
"movieFile": {
|
| 265 |
+
"relativePath": "Sintel (2010)/Sintel.2010.2160p.mkv",
|
| 266 |
+
"size": 24_800_000_000,
|
| 267 |
+
},
|
| 268 |
+
},
|
| 269 |
+
adapter="movie_webhook",
|
| 270 |
+
)
|
| 271 |
+
assert ok and record.category == "movie"
|
| 272 |
+
```
|
| 273 |
+
|
| 274 |
+
### 3. `BEP-44` / `BEP-46` Privacy & RSI Participation Modes
|
| 275 |
+
|
| 276 |
+
All adapters enforce `PhysicalGroundTruthVerifier` (never trusting unverified model predictions) and `TelemetryAnonymizer` (stripping tracker passkeys, local paths, and IP addresses; blinding title stems via post-quantum `HMAC-SHA256` + `SHAKE-256`; applying Laplace differential privacy $\epsilon = 1.0$; and scheduling outbound packets with Poisson timing jitter):
|
| 277 |
+
|
| 278 |
+
| `ARMIGER_DHT_RSI_MODE` | Behavior |
|
| 279 |
+
| :--- | :--- |
|
| 280 |
+
| `off` | Disables all `BEP-44`/`BEP-46` DHT traffic; calibration and ground-truth verification stay strictly local. |
|
| 281 |
+
| `pull_only` *(Default)* | Verifies and applies `M`-of-`N` hybrid post-quantum (`Ed25519` + `WOTS+-SHAKE256`) signed calibration manifests from the DHT after passing the 500-item golden canary gate; never transmits telemetry. |
|
| 282 |
+
| `anonymous_contribute` | Opts in to transmitting differential-privacy blinded, hybrid post-quantum (`X25519` + `ML-KEM-768` + `ChaCha20-Poly1305`) encrypted ground-truth observations. |
|
examples/README.md
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Armiger & Armiger Ultra Integration Examples
|
| 2 |
+
|
| 3 |
+
Turnkey reference adapters, HTTP services, and Docker Compose stacks for integrating **Armiger** (`8.3 MB INT8`) and **Armiger Ultra** (`32.3 MB FP32` + 6 plugins) across the self-hosted indexer and media automation ecosystem:
|
| 4 |
+
|
| 5 |
+
| Stack | Directory | Description |
|
| 6 |
+
| :--- | :--- | :--- |
|
| 7 |
+
| **[DH-Waypoint](https://github.com/spencernorton/DH-Waypoint)** | [`examples/dh-waypoint/`](dh-waypoint) | Native `processorapi/v1` HTTP processor (`dh-waypoint.processor.v1`), pre-ingest cohort admission gate (`dh-waypoint.processors.admission.v1`), and 4-tier `dh-waypoint.processors.feedback.v1` closed-loop RSI telemetry bridge. |
|
| 8 |
+
| **Bitmagnet** | [`examples/bitmagnet/`](bitmagnet) | Fast pre-ingest / post-crawl batch classifier, malware/honeypot pruner, PostgreSQL/GraphQL enrichment bridge (`bitmagnet_armiger_bridge.py`), and Torznab proxy stack. |
|
| 9 |
+
| **\*Arr Stack & qBittorrent** | [`examples/arr-stack/`](arr-stack) | Pre-grab Torznab filtering proxy (`armiger_torznab_proxy.py`) between `Prowlarr` / `Bitmagnet` and `Sonarr`, `Radarr`, `Lidarr`, `Readarr`, and `Whisparr`, plus a post-import webhook & `ffprobe` ground-truth receiver (`armiger_arr_webhook_receiver.py`). |
|
| 10 |
+
|
| 11 |
+
See [`docs/INTEGRATION_GUIDE.md`](../docs/INTEGRATION_GUIDE.md) for the complete architecture and step-by-step setup guide.
|
examples/arr-stack/armiger_arr_webhook_receiver.py
ADDED
|
@@ -0,0 +1,197 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Armiger Post-Import Webhook & ffprobe Ground-Truth Receiver for Sonarr, Radarr, Lidarr, Readarr & qBittorrent.
|
| 2 |
+
|
| 3 |
+
Wires completion and import events from the *Arr media automation stack and qBittorrent directly into
|
| 4 |
+
Armiger's `SeriesWebhookAdapter`, `MovieWebhookAdapter`, `AudioWebhookAdapter`, `PublicationWebhookAdapter`,
|
| 5 |
+
`QBittorrentAdapter`, and `PhysicalGroundTruthVerifier`:
|
| 6 |
+
- `POST /webhooks/sonarr` -> `SeriesWebhookAdapter` (`adapter="series_webhook"`)
|
| 7 |
+
- `POST /webhooks/radarr` -> `MovieWebhookAdapter` (`adapter="movie_webhook"`)
|
| 8 |
+
- `POST /webhooks/lidarr` -> `AudioWebhookAdapter` (`adapter="audio_webhook"`)
|
| 9 |
+
- `POST /webhooks/readarr` -> `PublicationWebhookAdapter` (`adapter="publication_webhook"`)
|
| 10 |
+
- `POST /webhooks/qbittorrent` -> `QBittorrentAdapter` (`adapter="qbittorrent"`)
|
| 11 |
+
|
| 12 |
+
When `ARMIGER_RUN_FFPROBE=true` and the imported media path is accessible on disk, the receiver runs
|
| 13 |
+
`ffprobe -v quiet -print_format json -show_format -show_streams <path>` to physically verify video/audio
|
| 14 |
+
codecs, channel counts, audio languages, and subtitle tracks before passing the observation to
|
| 15 |
+
`ArmigerUltraPipeline.ingest_rsi_feedback(...)`.
|
| 16 |
+
"""
|
| 17 |
+
from __future__ import annotations
|
| 18 |
+
|
| 19 |
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 20 |
+
import json
|
| 21 |
+
import os
|
| 22 |
+
from pathlib import Path
|
| 23 |
+
import subprocess
|
| 24 |
+
import sys
|
| 25 |
+
from typing import Any, Callable, Dict, Mapping, Optional
|
| 26 |
+
|
| 27 |
+
ROOT_DIR = Path(__file__).resolve().parents[2]
|
| 28 |
+
if str(ROOT_DIR) not in sys.path:
|
| 29 |
+
sys.path.insert(0, str(ROOT_DIR))
|
| 30 |
+
|
| 31 |
+
from armiger_dht_rsi import DHTRSIMode
|
| 32 |
+
from armiger_ultra_pipeline import ArmigerUltraPipeline
|
| 33 |
+
|
| 34 |
+
ROUTE_TO_ADAPTER: Dict[str, str] = {
|
| 35 |
+
"/webhooks/sonarr": "series_webhook",
|
| 36 |
+
"/webhooks/radarr": "movie_webhook",
|
| 37 |
+
"/webhooks/lidarr": "audio_webhook",
|
| 38 |
+
"/webhooks/readarr": "publication_webhook",
|
| 39 |
+
"/webhooks/qbittorrent": "qbittorrent",
|
| 40 |
+
}
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def run_ffprobe_json(media_path: str, timeout_sec: float = 10.0) -> Optional[Dict[str, Any]]:
|
| 44 |
+
"""Run ffprobe on a local media file and return parsed JSON stream/format metadata."""
|
| 45 |
+
p = Path(media_path)
|
| 46 |
+
if not p.is_file():
|
| 47 |
+
return None
|
| 48 |
+
try:
|
| 49 |
+
proc = subprocess.run(
|
| 50 |
+
[
|
| 51 |
+
"ffprobe",
|
| 52 |
+
"-v",
|
| 53 |
+
"quiet",
|
| 54 |
+
"-print_format",
|
| 55 |
+
"json",
|
| 56 |
+
"-show_format",
|
| 57 |
+
"-show_streams",
|
| 58 |
+
str(p),
|
| 59 |
+
],
|
| 60 |
+
capture_output=True,
|
| 61 |
+
text=True,
|
| 62 |
+
timeout=timeout_sec,
|
| 63 |
+
check=True,
|
| 64 |
+
)
|
| 65 |
+
return json.loads(proc.stdout)
|
| 66 |
+
except Exception:
|
| 67 |
+
return None
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
class ArmigerArrWebhookService:
|
| 71 |
+
"""Processes *Arr and qBittorrent completion webhooks into Armiger RSI ground-truth observations."""
|
| 72 |
+
|
| 73 |
+
def __init__(
|
| 74 |
+
self,
|
| 75 |
+
pipeline: ArmigerUltraPipeline,
|
| 76 |
+
*,
|
| 77 |
+
enable_ffprobe: bool = True,
|
| 78 |
+
ffprobe_runner: Callable[[str], Optional[Dict[str, Any]]] = run_ffprobe_json,
|
| 79 |
+
) -> None:
|
| 80 |
+
self.pipeline = pipeline
|
| 81 |
+
self.enable_ffprobe = enable_ffprobe
|
| 82 |
+
self.ffprobe_runner = ffprobe_runner
|
| 83 |
+
|
| 84 |
+
@staticmethod
|
| 85 |
+
def _extract_media_path(payload: Mapping[str, Any]) -> Optional[str]:
|
| 86 |
+
for section_key in ("movieFile", "episodeFile"):
|
| 87 |
+
sec = payload.get(section_key)
|
| 88 |
+
if isinstance(sec, Mapping):
|
| 89 |
+
full_path = sec.get("path") or sec.get("relativePath")
|
| 90 |
+
if full_path:
|
| 91 |
+
return str(full_path)
|
| 92 |
+
for list_key in ("trackFiles", "bookFiles"):
|
| 93 |
+
items = payload.get(list_key)
|
| 94 |
+
if isinstance(items, list) and items and isinstance(items[0], Mapping):
|
| 95 |
+
full_path = items[0].get("path")
|
| 96 |
+
if full_path:
|
| 97 |
+
return str(full_path)
|
| 98 |
+
if payload.get("content_path") or payload.get("save_path"):
|
| 99 |
+
return str(payload.get("content_path") or payload.get("save_path"))
|
| 100 |
+
return None
|
| 101 |
+
|
| 102 |
+
def process_webhook(
|
| 103 |
+
self,
|
| 104 |
+
route_path: str,
|
| 105 |
+
payload: Mapping[str, Any],
|
| 106 |
+
*,
|
| 107 |
+
explicit_ffprobe: Optional[Dict[str, Any]] = None,
|
| 108 |
+
) -> Dict[str, Any]:
|
| 109 |
+
"""Process a webhook payload from Sonarr, Radarr, Lidarr, Readarr, or qBittorrent."""
|
| 110 |
+
adapter_name = ROUTE_TO_ADAPTER.get(route_path)
|
| 111 |
+
if not adapter_name:
|
| 112 |
+
return {"accepted": False, "reason": f"unknown_route:{route_path}"}
|
| 113 |
+
|
| 114 |
+
event_type = str(payload.get("eventType") or "")
|
| 115 |
+
if event_type.lower() == "test":
|
| 116 |
+
return {
|
| 117 |
+
"accepted": True,
|
| 118 |
+
"verified": False,
|
| 119 |
+
"adapter": adapter_name,
|
| 120 |
+
"reason": "test_webhook_acknowledged",
|
| 121 |
+
}
|
| 122 |
+
|
| 123 |
+
ffprobe_data = explicit_ffprobe
|
| 124 |
+
if ffprobe_data is None and self.enable_ffprobe:
|
| 125 |
+
media_path = self._extract_media_path(payload)
|
| 126 |
+
if media_path:
|
| 127 |
+
ffprobe_data = self.ffprobe_runner(media_path)
|
| 128 |
+
|
| 129 |
+
ok, reason, record = self.pipeline.ingest_rsi_feedback(
|
| 130 |
+
payload,
|
| 131 |
+
adapter=adapter_name,
|
| 132 |
+
ffprobe_output=ffprobe_data,
|
| 133 |
+
)
|
| 134 |
+
return {
|
| 135 |
+
"accepted": True,
|
| 136 |
+
"verified": bool(ok),
|
| 137 |
+
"adapter": adapter_name,
|
| 138 |
+
"reason": reason,
|
| 139 |
+
"blinded_hash": record.blinded_stem.blinded_hash if record is not None else None,
|
| 140 |
+
"category": record.category if record is not None else None,
|
| 141 |
+
"lang": record.lang if record is not None else None,
|
| 142 |
+
"junk": record.junk if record is not None else None,
|
| 143 |
+
}
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
def create_webhook_server(host: str = "localhost", port: int = 8098) -> ThreadingHTTPServer:
|
| 147 |
+
model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
|
| 148 |
+
variant = os.environ.get("ARMIGER_VARIANT", "int8")
|
| 149 |
+
rsi_mode_str = os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")
|
| 150 |
+
pipeline = ArmigerUltraPipeline.from_directory(
|
| 151 |
+
model_dir,
|
| 152 |
+
variant=variant,
|
| 153 |
+
rsi_mode=DHTRSIMode(rsi_mode_str),
|
| 154 |
+
preload_catalog=False,
|
| 155 |
+
)
|
| 156 |
+
service = ArmigerArrWebhookService(
|
| 157 |
+
pipeline,
|
| 158 |
+
enable_ffprobe=os.environ.get("ARMIGER_RUN_FFPROBE", "true").lower() == "true",
|
| 159 |
+
)
|
| 160 |
+
|
| 161 |
+
class WebhookHandler(BaseHTTPRequestHandler):
|
| 162 |
+
def _send_json(self, status: int, data: Dict[str, Any]) -> None:
|
| 163 |
+
body = json.dumps(data).encode("utf-8")
|
| 164 |
+
self.send_response(status)
|
| 165 |
+
self.send_header("Content-Type", "application/json")
|
| 166 |
+
self.send_header("Content-Length", str(len(body)))
|
| 167 |
+
self.end_headers()
|
| 168 |
+
self.wfile.write(body)
|
| 169 |
+
|
| 170 |
+
def do_GET(self) -> None:
|
| 171 |
+
if self.path == "/healthz":
|
| 172 |
+
self._send_json(200, {"status": "ok", "service": "armiger-arr-webhook-receiver", "rsi_mode": rsi_mode_str})
|
| 173 |
+
else:
|
| 174 |
+
self._send_json(404, {"error": "not_found"})
|
| 175 |
+
|
| 176 |
+
def do_POST(self) -> None:
|
| 177 |
+
length = int(self.headers.get("Content-Length", "0"))
|
| 178 |
+
raw = self.rfile.read(length) if length > 0 else b"{}"
|
| 179 |
+
payload = json.loads(raw.decode("utf-8"))
|
| 180 |
+
if self.path in ROUTE_TO_ADAPTER:
|
| 181 |
+
res = service.process_webhook(self.path, payload)
|
| 182 |
+
self._send_json(200, res)
|
| 183 |
+
else:
|
| 184 |
+
self._send_json(404, {"error": "not_found"})
|
| 185 |
+
|
| 186 |
+
def log_message(self, fmt: str, *args: Any) -> None:
|
| 187 |
+
return
|
| 188 |
+
|
| 189 |
+
return ThreadingHTTPServer((host, port), WebhookHandler)
|
| 190 |
+
|
| 191 |
+
|
| 192 |
+
if __name__ == "__main__":
|
| 193 |
+
listen_host = os.environ.get("ARMIGER_WEBHOOK_HOST", "localhost")
|
| 194 |
+
listen_port = int(os.environ.get("ARMIGER_WEBHOOK_PORT", "8098"))
|
| 195 |
+
srv = create_webhook_server(listen_host, listen_port)
|
| 196 |
+
print(f"Armiger *Arr & qBittorrent webhook receiver listening on {listen_host}:{listen_port}")
|
| 197 |
+
srv.serve_forever()
|
examples/arr-stack/armiger_torznab_proxy.py
ADDED
|
@@ -0,0 +1,254 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Armiger Torznab Filtering Proxy & Pre-Grab Admission Gate for Prowlarr, Bitmagnet, and the *Arr Stack.
|
| 2 |
+
|
| 3 |
+
Sits transparently between upstream Torznab indexers (`Prowlarr`, `Bitmagnet`, `DH-Waypoint`, `Jackett`)
|
| 4 |
+
and downstream media managers (`Sonarr`, `Radarr`, `Lidarr`, `Readarr`, `Whisparr`).
|
| 5 |
+
|
| 6 |
+
Every Torznab XML `<item>` returned by an upstream search is evaluated in milliseconds by
|
| 7 |
+
`ArmigerUltraPipeline.classify_batch` before `Sonarr` or `Radarr` can grab it:
|
| 8 |
+
1. Blocks disguised malware executables (`.mkv.exe`, `.scr`, `.vbs`, `.lnk`, `.ps1`, passworded archives).
|
| 9 |
+
2. Blocks fake-seeder malware honeypots (`SwarmSurvivalNet` `is_honeypot_trap == True`).
|
| 10 |
+
3. Blocks dead swarms (`viability_tier == "dead"` when `ARMIGER_REJECT_DEAD_SWARMS=true`).
|
| 11 |
+
4. Blocks unrequested foreign-language audio traps (`AudioLanguageGuard` cross-head check when
|
| 12 |
+
`ARMIGER_REQUIRE_ENGLISH_AUDIO=true`, with optional anime/subtitle allowance via `ARMIGER_ALLOW_ENGLISH_SUBS=true`).
|
| 13 |
+
5. Enriches surviving `<item>` nodes with `<torznab:attr name="armiger_category" .../>` and
|
| 14 |
+
`<torznab:attr name="armiger_viability" .../>` attributes.
|
| 15 |
+
"""
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 19 |
+
import json
|
| 20 |
+
import os
|
| 21 |
+
from pathlib import Path
|
| 22 |
+
import sys
|
| 23 |
+
from typing import Any, Dict, List, Optional, Tuple
|
| 24 |
+
import urllib.parse
|
| 25 |
+
import urllib.request
|
| 26 |
+
import xml.etree.ElementTree as ET
|
| 27 |
+
|
| 28 |
+
ROOT_DIR = Path(__file__).resolve().parents[2]
|
| 29 |
+
if str(ROOT_DIR) not in sys.path:
|
| 30 |
+
sys.path.insert(0, str(ROOT_DIR))
|
| 31 |
+
|
| 32 |
+
from armiger_dht_rsi import DHTRSIMode
|
| 33 |
+
from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
|
| 34 |
+
|
| 35 |
+
TORZNAB_NS = "http://torznab.com/schemas/2015/feed"
|
| 36 |
+
ET.register_namespace("torznab", TORZNAB_NS)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
class ArmigerTorznabFilter:
|
| 40 |
+
"""Filters and enriches Torznab RSS/XML feeds using ArmigerUltraPipeline."""
|
| 41 |
+
|
| 42 |
+
def __init__(
|
| 43 |
+
self,
|
| 44 |
+
pipeline: ArmigerUltraPipeline,
|
| 45 |
+
*,
|
| 46 |
+
require_english_audio: bool = False,
|
| 47 |
+
allow_english_subs: bool = True,
|
| 48 |
+
reject_dead_swarms: bool = True,
|
| 49 |
+
allowed_categories: Optional[List[str]] = None,
|
| 50 |
+
use_zero_cue_catalog: bool = True,
|
| 51 |
+
) -> None:
|
| 52 |
+
self.pipeline = pipeline
|
| 53 |
+
self.require_english_audio = require_english_audio
|
| 54 |
+
self.allow_english_subs = allow_english_subs
|
| 55 |
+
self.reject_dead_swarms = reject_dead_swarms
|
| 56 |
+
self.allowed_categories = set(allowed_categories) if allowed_categories else None
|
| 57 |
+
self.use_zero_cue_catalog = use_zero_cue_catalog
|
| 58 |
+
|
| 59 |
+
@staticmethod
|
| 60 |
+
def _extract_item_row(item_elem: ET.Element) -> Dict[str, Any]:
|
| 61 |
+
title = (item_elem.findtext("title") or "").strip()
|
| 62 |
+
size_text = item_elem.findtext("size") or "0"
|
| 63 |
+
try:
|
| 64 |
+
size_val = int(size_text)
|
| 65 |
+
except ValueError:
|
| 66 |
+
size_val = 0
|
| 67 |
+
|
| 68 |
+
attrs: Dict[str, str] = {}
|
| 69 |
+
for attr in item_elem.findall(f"{{{TORZNAB_NS}}}attr"):
|
| 70 |
+
name = (attr.attrib.get("name") or "").lower().strip()
|
| 71 |
+
val = attr.attrib.get("value") or ""
|
| 72 |
+
if name:
|
| 73 |
+
attrs[name] = val
|
| 74 |
+
|
| 75 |
+
seeders = int(attrs.get("seeders", "0") or 0)
|
| 76 |
+
peers = int(attrs.get("peers", "0") or 0)
|
| 77 |
+
leechers = max(peers - seeders, int(attrs.get("leechers", "0") or 0))
|
| 78 |
+
grabs = int(attrs["grabs"]) if "grabs" in attrs and attrs["grabs"].strip() else None
|
| 79 |
+
infohash = attrs.get("infohash") or ""
|
| 80 |
+
|
| 81 |
+
files_attr = attrs.get("files") or ""
|
| 82 |
+
if files_attr:
|
| 83 |
+
files = [{"path": p.strip(), "size": size_val} for p in files_attr.split(";") if p.strip()]
|
| 84 |
+
else:
|
| 85 |
+
tail = title.rsplit(".", 1)[-1].lower() if "." in title else ""
|
| 86 |
+
known_exts = {"mkv", "mp4", "avi", "flac", "mp3", "m4b", "epub", "pdf", "cbz", "cbr", "iso", "exe", "scr", "vbs", "bat", "cmd", "lnk", "ps1", "msi"}
|
| 87 |
+
if tail in known_exts:
|
| 88 |
+
files = [{"path": title, "size": size_val}]
|
| 89 |
+
else:
|
| 90 |
+
low = title.lower()
|
| 91 |
+
if any(k in low for k in ("flac", "24bit", "lossless")):
|
| 92 |
+
synth_ext = ".flac"
|
| 93 |
+
elif any(k in low for k in ("m4b", "audiobook", "unabridged")):
|
| 94 |
+
synth_ext = ".m4b"
|
| 95 |
+
elif any(k in low for k in ("epub", "ebook")):
|
| 96 |
+
synth_ext = ".epub"
|
| 97 |
+
elif any(k in low for k in ("cbz", "cbr")):
|
| 98 |
+
synth_ext = ".cbz"
|
| 99 |
+
else:
|
| 100 |
+
synth_ext = ".mkv"
|
| 101 |
+
files = [{"path": f"{title}{synth_ext}", "size": size_val}]
|
| 102 |
+
|
| 103 |
+
return {
|
| 104 |
+
"infohash": infohash,
|
| 105 |
+
"name": title,
|
| 106 |
+
"size": size_val,
|
| 107 |
+
"files": files,
|
| 108 |
+
"swarm": {
|
| 109 |
+
"tracker_known": bool(seeders > 0 or leechers > 0 or "seeders" in attrs),
|
| 110 |
+
"tracker_seeders": seeders,
|
| 111 |
+
"tracker_leechers": leechers,
|
| 112 |
+
"tracker_completed": grabs,
|
| 113 |
+
},
|
| 114 |
+
}
|
| 115 |
+
|
| 116 |
+
def should_admit(self, pred: ArmigerUltraPrediction) -> Tuple[bool, str]:
|
| 117 |
+
"""Evaluate whether a Torznab release should be admitted to Sonarr/Radarr/Lidarr/Readarr."""
|
| 118 |
+
if not pred.csam_safety["is_safe"]:
|
| 119 |
+
return False, str(pred.csam_safety.get("reason") or "csam_or_malware_blocked")
|
| 120 |
+
if pred.honeypot_trap["is_honeypot_trap"]:
|
| 121 |
+
return False, "honeypot_trap_blocked"
|
| 122 |
+
if pred.junk == "junk" or not pred.mondrian_gates["passed_all"]:
|
| 123 |
+
return False, str(pred.mondrian_gates.get("safety_gate_reason") or "mondrian_gate_blocked")
|
| 124 |
+
|
| 125 |
+
if self.reject_dead_swarms and pred.swarm_viability["viability_tier"] == "dead":
|
| 126 |
+
return False, "dead_swarm_blocked"
|
| 127 |
+
|
| 128 |
+
if self.allowed_categories and pred.category not in self.allowed_categories:
|
| 129 |
+
return False, f"category_mismatch:{pred.category}"
|
| 130 |
+
|
| 131 |
+
if self.require_english_audio and pred.lang == "non_english":
|
| 132 |
+
has_subs = pred.english_subs == "has_eng_subs"
|
| 133 |
+
if not (self.allow_english_subs and has_subs):
|
| 134 |
+
return False, "non_english_audio_blocked"
|
| 135 |
+
|
| 136 |
+
return True, "admitted"
|
| 137 |
+
|
| 138 |
+
def filter_torznab_xml(self, xml_bytes: bytes) -> Tuple[bytes, Dict[str, Any]]:
|
| 139 |
+
"""Parse upstream Torznab XML, remove blocked `<item>` nodes, and enrich surviving items."""
|
| 140 |
+
root = ET.fromstring(xml_bytes)
|
| 141 |
+
channel = root.find("channel")
|
| 142 |
+
if channel is None:
|
| 143 |
+
return xml_bytes, {"total": 0, "admitted": 0, "blocked": 0, "blocked_reasons": {}}
|
| 144 |
+
|
| 145 |
+
items = channel.findall("item")
|
| 146 |
+
if not items:
|
| 147 |
+
return xml_bytes, {"total": 0, "admitted": 0, "blocked": 0, "blocked_reasons": {}}
|
| 148 |
+
|
| 149 |
+
rows = [self._extract_item_row(it) for it in items]
|
| 150 |
+
predictions = self.pipeline.classify_batch(
|
| 151 |
+
rows,
|
| 152 |
+
use_zero_cue_catalog=self.use_zero_cue_catalog,
|
| 153 |
+
)
|
| 154 |
+
|
| 155 |
+
admitted_count = 0
|
| 156 |
+
blocked_count = 0
|
| 157 |
+
blocked_reasons: Dict[str, int] = {}
|
| 158 |
+
|
| 159 |
+
for item_elem, pred in zip(items, predictions):
|
| 160 |
+
admit, reason = self.should_admit(pred)
|
| 161 |
+
if not admit:
|
| 162 |
+
channel.remove(item_elem)
|
| 163 |
+
blocked_count += 1
|
| 164 |
+
blocked_reasons[reason] = blocked_reasons.get(reason, 0) + 1
|
| 165 |
+
continue
|
| 166 |
+
|
| 167 |
+
admitted_count += 1
|
| 168 |
+
for attr_name, attr_val in (
|
| 169 |
+
("armiger_category", pred.category),
|
| 170 |
+
("armiger_lang", pred.lang),
|
| 171 |
+
("armiger_viability", str(pred.swarm_viability["viability_tier"])),
|
| 172 |
+
("armiger_survival_prob", f"{pred.swarm_viability['survival_net_prob']:.4f}"),
|
| 173 |
+
):
|
| 174 |
+
attr_el = ET.SubElement(item_elem, f"{{{TORZNAB_NS}}}attr")
|
| 175 |
+
attr_el.set("name", attr_name)
|
| 176 |
+
attr_el.set("value", attr_val)
|
| 177 |
+
|
| 178 |
+
filtered_bytes = ET.tostring(root, encoding="utf-8", xml_declaration=True)
|
| 179 |
+
stats = {
|
| 180 |
+
"total": len(items),
|
| 181 |
+
"admitted": admitted_count,
|
| 182 |
+
"blocked": blocked_count,
|
| 183 |
+
"blocked_reasons": blocked_reasons,
|
| 184 |
+
}
|
| 185 |
+
return filtered_bytes, stats
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def create_proxy_server(host: str = "localhost", port: int = 8097) -> ThreadingHTTPServer:
|
| 189 |
+
model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
|
| 190 |
+
variant = os.environ.get("ARMIGER_VARIANT", "int8")
|
| 191 |
+
upstream_url = os.environ.get("UPSTREAM_TORZNAB_URL", "http://dh-waypoint:3333/torznab").rstrip("/")
|
| 192 |
+
pipeline = ArmigerUltraPipeline.from_directory(
|
| 193 |
+
model_dir,
|
| 194 |
+
variant=variant,
|
| 195 |
+
rsi_mode=DHTRSIMode(os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")),
|
| 196 |
+
preload_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
|
| 197 |
+
)
|
| 198 |
+
torznab_filter = ArmigerTorznabFilter(
|
| 199 |
+
pipeline,
|
| 200 |
+
require_english_audio=os.environ.get("ARMIGER_REQUIRE_ENGLISH_AUDIO", "false").lower() == "true",
|
| 201 |
+
allow_english_subs=os.environ.get("ARMIGER_ALLOW_ENGLISH_SUBS", "true").lower() == "true",
|
| 202 |
+
reject_dead_swarms=os.environ.get("ARMIGER_REJECT_DEAD_SWARMS", "true").lower() == "true",
|
| 203 |
+
)
|
| 204 |
+
|
| 205 |
+
class ProxyHandler(BaseHTTPRequestHandler):
|
| 206 |
+
def do_GET(self) -> None:
|
| 207 |
+
if self.path == "/healthz":
|
| 208 |
+
body = json.dumps({"status": "ok", "service": "armiger-torznab-proxy", "variant": variant}).encode("utf-8")
|
| 209 |
+
self.send_response(200)
|
| 210 |
+
self.send_header("Content-Type", "application/json")
|
| 211 |
+
self.send_header("Content-Length", str(len(body)))
|
| 212 |
+
self.end_headers()
|
| 213 |
+
self.wfile.write(body)
|
| 214 |
+
return
|
| 215 |
+
|
| 216 |
+
parsed = urllib.parse.urlsplit(self.path)
|
| 217 |
+
params = urllib.parse.parse_qs(parsed.query)
|
| 218 |
+
t_param = (params.get("t") or [""])[0].lower()
|
| 219 |
+
target = f"{upstream_url}{parsed.path}"
|
| 220 |
+
if parsed.query:
|
| 221 |
+
target = f"{target}?{parsed.query}"
|
| 222 |
+
|
| 223 |
+
req = urllib.request.Request(target, method="GET")
|
| 224 |
+
with urllib.request.urlopen(req, timeout=15.0) as resp:
|
| 225 |
+
raw_xml = resp.read()
|
| 226 |
+
content_type = resp.headers.get("Content-Type", "application/rss+xml; charset=utf-8")
|
| 227 |
+
|
| 228 |
+
if t_param in ("caps", ""):
|
| 229 |
+
out_bytes = raw_xml
|
| 230 |
+
stats = {"total": 0, "admitted": 0, "blocked": 0}
|
| 231 |
+
else:
|
| 232 |
+
out_bytes, stats = torznab_filter.filter_torznab_xml(raw_xml)
|
| 233 |
+
|
| 234 |
+
self.send_response(200)
|
| 235 |
+
self.send_header("Content-Type", content_type)
|
| 236 |
+
self.send_header("X-Armiger-Total", str(stats["total"]))
|
| 237 |
+
self.send_header("X-Armiger-Admitted", str(stats["admitted"]))
|
| 238 |
+
self.send_header("X-Armiger-Blocked", str(stats["blocked"]))
|
| 239 |
+
self.send_header("Content-Length", str(len(out_bytes)))
|
| 240 |
+
self.end_headers()
|
| 241 |
+
self.wfile.write(out_bytes)
|
| 242 |
+
|
| 243 |
+
def log_message(self, fmt: str, *args: Any) -> None:
|
| 244 |
+
return
|
| 245 |
+
|
| 246 |
+
return ThreadingHTTPServer((host, port), ProxyHandler)
|
| 247 |
+
|
| 248 |
+
|
| 249 |
+
if __name__ == "__main__":
|
| 250 |
+
listen_host = os.environ.get("ARMIGER_PROXY_HOST", "localhost")
|
| 251 |
+
listen_port = int(os.environ.get("ARMIGER_PROXY_PORT", "8097"))
|
| 252 |
+
srv = create_proxy_server(listen_host, listen_port)
|
| 253 |
+
print(f"Armiger Torznab proxy listening on {listen_host}:{listen_port}")
|
| 254 |
+
srv.serve_forever()
|
examples/arr-stack/docker-compose.yml
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
services:
|
| 2 |
+
armiger-torznab-proxy:
|
| 3 |
+
image: python:3.12-slim
|
| 4 |
+
container_name: armiger-torznab-proxy
|
| 5 |
+
restart: unless-stopped
|
| 6 |
+
working_dir: /app
|
| 7 |
+
ports:
|
| 8 |
+
- "8097:8097"
|
| 9 |
+
environment:
|
| 10 |
+
ARMIGER_MODEL_DIR: /app
|
| 11 |
+
ARMIGER_VARIANT: "${ARMIGER_VARIANT:-int8}"
|
| 12 |
+
UPSTREAM_TORZNAB_URL: "${UPSTREAM_TORZNAB_URL:-http://prowlarr:9696/1/api}"
|
| 13 |
+
ARMIGER_PROXY_HOST: "localhost"
|
| 14 |
+
ARMIGER_PROXY_PORT: "8097"
|
| 15 |
+
ARMIGER_REQUIRE_ENGLISH_AUDIO: "${ARMIGER_REQUIRE_ENGLISH_AUDIO:-false}"
|
| 16 |
+
ARMIGER_ALLOW_ENGLISH_SUBS: "${ARMIGER_ALLOW_ENGLISH_SUBS:-true}"
|
| 17 |
+
ARMIGER_REJECT_DEAD_SWARMS: "${ARMIGER_REJECT_DEAD_SWARMS:-true}"
|
| 18 |
+
ARMIGER_USE_ZERO_CUE_CATALOG: "true"
|
| 19 |
+
volumes:
|
| 20 |
+
- ../..:/app:ro
|
| 21 |
+
command:
|
| 22 |
+
- sh
|
| 23 |
+
- -c
|
| 24 |
+
- "pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/arr-stack/armiger_torznab_proxy.py"
|
| 25 |
+
|
| 26 |
+
armiger-arr-webhook-receiver:
|
| 27 |
+
image: python:3.12-slim
|
| 28 |
+
container_name: armiger-arr-webhook-receiver
|
| 29 |
+
restart: unless-stopped
|
| 30 |
+
working_dir: /app
|
| 31 |
+
ports:
|
| 32 |
+
- "8098:8098"
|
| 33 |
+
environment:
|
| 34 |
+
ARMIGER_MODEL_DIR: /app
|
| 35 |
+
ARMIGER_VARIANT: "int8"
|
| 36 |
+
ARMIGER_WEBHOOK_HOST: "localhost"
|
| 37 |
+
ARMIGER_WEBHOOK_PORT: "8098"
|
| 38 |
+
ARMIGER_RUN_FFPROBE: "true"
|
| 39 |
+
ARMIGER_DHT_RSI_MODE: "${ARMIGER_DHT_RSI_MODE:-pull_only}"
|
| 40 |
+
volumes:
|
| 41 |
+
- ../..:/app:ro
|
| 42 |
+
- media_library:/data/media:ro
|
| 43 |
+
command:
|
| 44 |
+
- sh
|
| 45 |
+
- -c
|
| 46 |
+
- "apt-get update && apt-get install -y --no-install-recommends ffmpeg && pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/arr-stack/armiger_arr_webhook_receiver.py"
|
| 47 |
+
|
| 48 |
+
volumes:
|
| 49 |
+
media_library:
|
examples/bitmagnet/bitmagnet_armiger_bridge.py
ADDED
|
@@ -0,0 +1,266 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Bitmagnet <-> Armiger Ultra Pre-Ingest / Post-Crawl Classifier & Admission Bridge.
|
| 2 |
+
|
| 3 |
+
Provides two operational modes for upstream Bitmagnet deployments:
|
| 4 |
+
1. HTTP Classifier & Admission Webhook Bridge (`POST /v1/bitmagnet/classify`):
|
| 5 |
+
Accepts batches of newly discovered Bitmagnet torrents, runs `ArmigerUltraPipeline.classify_batch`,
|
| 6 |
+
and returns Bitmagnet-compatible content types (`movie`, `tv_show`, `music`, `ebook`, `audiobook`,
|
| 7 |
+
`comic`, `software`, `game`, `xxx`), structured tags (`armiger:category:*`, `armiger:lang:*`,
|
| 8 |
+
`armiger:viability:*`), and `delete` actions for disguised malware (`.mkv.exe`, `.scr`, `.lnk`),
|
| 9 |
+
fake-seeder honeypot traps, or CSAM/safety gate violations.
|
| 10 |
+
2. GraphQL Mutation & PostgreSQL Reconciliation Helper (`BitmagnetGraphQLSynchronizer`):
|
| 11 |
+
Generates Bitmagnet GraphQL ` torrentContentUpdate ` / ` torrentDelete ` mutations or SQL updates
|
| 12 |
+
so upstream Bitmagnet instances gain Armiger's 95.49% 11-class accuracy (vs. 35.29% built-in rules)
|
| 13 |
+
and 100% fake-seeder malware trap recall.
|
| 14 |
+
"""
|
| 15 |
+
from __future__ import annotations
|
| 16 |
+
|
| 17 |
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 18 |
+
import json
|
| 19 |
+
import os
|
| 20 |
+
from pathlib import Path
|
| 21 |
+
import sys
|
| 22 |
+
from typing import Any, Dict, List, Mapping, Optional, Sequence
|
| 23 |
+
import urllib.request
|
| 24 |
+
|
| 25 |
+
ROOT_DIR = Path(__file__).resolve().parents[2]
|
| 26 |
+
if str(ROOT_DIR) not in sys.path:
|
| 27 |
+
sys.path.insert(0, str(ROOT_DIR))
|
| 28 |
+
|
| 29 |
+
from armiger_dht_rsi import DHTRSIMode
|
| 30 |
+
from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
|
| 31 |
+
|
| 32 |
+
# Map Armiger 11-class labels to Bitmagnet ContentType enum values
|
| 33 |
+
BITMAGNET_CONTENT_TYPE_MAP: Dict[str, Optional[str]] = {
|
| 34 |
+
"movie": "movie",
|
| 35 |
+
"tv_show": "tv_show",
|
| 36 |
+
"music": "music",
|
| 37 |
+
"ebook": "ebook",
|
| 38 |
+
"audiobook": "audiobook",
|
| 39 |
+
"comic": "comic",
|
| 40 |
+
"software": "software",
|
| 41 |
+
"game": "game",
|
| 42 |
+
"course": "ebook",
|
| 43 |
+
"xxx": "xxx",
|
| 44 |
+
"unknown": None,
|
| 45 |
+
}
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
class BitmagnetArmigerBridge:
|
| 49 |
+
"""Classifies, filters, and enriches Bitmagnet torrent items with Armiger Ultra."""
|
| 50 |
+
|
| 51 |
+
def __init__(
|
| 52 |
+
self,
|
| 53 |
+
pipeline: ArmigerUltraPipeline,
|
| 54 |
+
*,
|
| 55 |
+
use_zero_cue_catalog: bool = True,
|
| 56 |
+
prune_honeypots_and_malware: bool = True,
|
| 57 |
+
prune_dead_swarms: bool = False,
|
| 58 |
+
isolate_adult_namespace: bool = True,
|
| 59 |
+
) -> None:
|
| 60 |
+
self.pipeline = pipeline
|
| 61 |
+
self.use_zero_cue_catalog = use_zero_cue_catalog
|
| 62 |
+
self.prune_honeypots_and_malware = prune_honeypots_and_malware
|
| 63 |
+
self.prune_dead_swarms = prune_dead_swarms
|
| 64 |
+
self.isolate_adult_namespace = isolate_adult_namespace
|
| 65 |
+
|
| 66 |
+
@staticmethod
|
| 67 |
+
def _bitmagnet_item_to_row(item: Mapping[str, Any]) -> Dict[str, Any]:
|
| 68 |
+
files_in = item.get("files") if isinstance(item.get("files"), list) else []
|
| 69 |
+
seeders = int(item.get("seeders", item.get("tracker_seeders", 0)) or 0)
|
| 70 |
+
leechers = int(item.get("leechers", item.get("peers", item.get("tracker_leechers", 0))) or 0)
|
| 71 |
+
return {
|
| 72 |
+
"infohash": str(item.get("infoHash") or item.get("info_hash") or item.get("infohash") or ""),
|
| 73 |
+
"name": str(item.get("name") or item.get("title") or ""),
|
| 74 |
+
"size": int(item.get("size") or 0),
|
| 75 |
+
"files": [
|
| 76 |
+
{"path": str(f.get("path") or f.get("name") or ""), "size": int(f.get("size") or 0)}
|
| 77 |
+
if isinstance(f, Mapping)
|
| 78 |
+
else {"path": str(f), "size": 0}
|
| 79 |
+
for f in files_in
|
| 80 |
+
],
|
| 81 |
+
"file_count": int(item.get("filesCount") or item.get("file_count") or len(files_in) or 1),
|
| 82 |
+
"swarm": {
|
| 83 |
+
"tracker_known": bool(seeders > 0 or leechers > 0),
|
| 84 |
+
"tracker_seeders": seeders,
|
| 85 |
+
"tracker_leechers": leechers,
|
| 86 |
+
"tracker_completed": int(item["completed"]) if "completed" in item and item["completed"] is not None else None,
|
| 87 |
+
},
|
| 88 |
+
}
|
| 89 |
+
|
| 90 |
+
def evaluate_batch(self, items: Sequence[Mapping[str, Any]]) -> Dict[str, Any]:
|
| 91 |
+
"""Run Armiger Ultra on a batch of Bitmagnet torrents and return enrichment/prune actions."""
|
| 92 |
+
rows = [self._bitmagnet_item_to_row(it) for it in items]
|
| 93 |
+
predictions: List[ArmigerUltraPrediction] = self.pipeline.classify_batch(
|
| 94 |
+
rows,
|
| 95 |
+
use_zero_cue_catalog=self.use_zero_cue_catalog,
|
| 96 |
+
)
|
| 97 |
+
|
| 98 |
+
actions: List[Dict[str, Any]] = []
|
| 99 |
+
delete_infohashes: List[str] = []
|
| 100 |
+
|
| 101 |
+
for pred in predictions:
|
| 102 |
+
is_malware_or_unsafe = not pred.csam_safety["is_safe"] or pred.junk == "junk"
|
| 103 |
+
is_honeypot = bool(pred.honeypot_trap["is_honeypot_trap"])
|
| 104 |
+
viability_tier = str(pred.swarm_viability["viability_tier"])
|
| 105 |
+
is_dead = viability_tier == "dead"
|
| 106 |
+
|
| 107 |
+
if self.prune_honeypots_and_malware and (is_malware_or_unsafe or is_honeypot):
|
| 108 |
+
reason = (
|
| 109 |
+
pred.csam_safety.get("reason")
|
| 110 |
+
or ("honeypot_trap" if is_honeypot else "junk_or_malware_gate")
|
| 111 |
+
)
|
| 112 |
+
delete_infohashes.append(pred.infohash)
|
| 113 |
+
actions.append({
|
| 114 |
+
"infoHash": pred.infohash,
|
| 115 |
+
"action": "delete",
|
| 116 |
+
"reason": str(reason),
|
| 117 |
+
"category": pred.category,
|
| 118 |
+
"honeypot_trap": is_honeypot,
|
| 119 |
+
"junk": pred.junk,
|
| 120 |
+
})
|
| 121 |
+
continue
|
| 122 |
+
|
| 123 |
+
if self.prune_dead_swarms and is_dead:
|
| 124 |
+
delete_infohashes.append(pred.infohash)
|
| 125 |
+
actions.append({
|
| 126 |
+
"infoHash": pred.infohash,
|
| 127 |
+
"action": "delete",
|
| 128 |
+
"reason": "dead_swarm",
|
| 129 |
+
"category": pred.category,
|
| 130 |
+
"viability_tier": viability_tier,
|
| 131 |
+
})
|
| 132 |
+
continue
|
| 133 |
+
|
| 134 |
+
if self.isolate_adult_namespace and pred.adult == "adult" and pred.category != "xxx":
|
| 135 |
+
bitmagnet_type: Optional[str] = "xxx"
|
| 136 |
+
else:
|
| 137 |
+
bitmagnet_type = BITMAGNET_CONTENT_TYPE_MAP.get(pred.category)
|
| 138 |
+
|
| 139 |
+
tags = [
|
| 140 |
+
f"armiger:{pred.category}",
|
| 141 |
+
f"armiger:lang:{pred.lang}",
|
| 142 |
+
f"armiger:viability:{viability_tier}",
|
| 143 |
+
]
|
| 144 |
+
if pred.anime == "anime":
|
| 145 |
+
tags.append("armiger:anime")
|
| 146 |
+
dub_status = pred.heads["anime"].get("anime_dub_status")
|
| 147 |
+
if dub_status:
|
| 148 |
+
tags.append(f"armiger:dub:{dub_status}")
|
| 149 |
+
if pred.english_subs == "has_eng_subs":
|
| 150 |
+
tags.append("armiger:eng_subs")
|
| 151 |
+
|
| 152 |
+
actions.append({
|
| 153 |
+
"infoHash": pred.infohash,
|
| 154 |
+
"action": "update",
|
| 155 |
+
"contentType": bitmagnet_type,
|
| 156 |
+
"armigerCategory": pred.category,
|
| 157 |
+
"languages": ["en"] if pred.lang == "english" else [],
|
| 158 |
+
"tags": tags,
|
| 159 |
+
"viabilityTier": viability_tier,
|
| 160 |
+
"viabilityProb": float(pred.swarm_viability["viability_prob"]),
|
| 161 |
+
"mondrianPassed": bool(pred.mondrian_gates["passed_all"]),
|
| 162 |
+
})
|
| 163 |
+
|
| 164 |
+
return {
|
| 165 |
+
"total": len(actions),
|
| 166 |
+
"deletedCount": len(delete_infohashes),
|
| 167 |
+
"deleteInfoHashes": delete_infohashes,
|
| 168 |
+
"actions": actions,
|
| 169 |
+
"graphqlDeleteMutation": self.build_graphql_delete_mutation(delete_infohashes),
|
| 170 |
+
}
|
| 171 |
+
|
| 172 |
+
@staticmethod
|
| 173 |
+
def build_graphql_delete_mutation(infohashes: Sequence[str]) -> Optional[Dict[str, Any]]:
|
| 174 |
+
"""Build a Bitmagnet GraphQL `torrentDelete` mutation payload for blocked infohashes."""
|
| 175 |
+
clean = [h.strip().lower() for h in infohashes if h and h.strip()]
|
| 176 |
+
if not clean:
|
| 177 |
+
return None
|
| 178 |
+
return {
|
| 179 |
+
"query": (
|
| 180 |
+
"mutation ArmigerPruneMalwareAndHoneypots($infoHashes: [Hash20!]!) { "
|
| 181 |
+
"torrent { delete(input: { infoHashes: $infoHashes }) { infoHashes } } "
|
| 182 |
+
"}"
|
| 183 |
+
),
|
| 184 |
+
"variables": {"infoHashes": clean},
|
| 185 |
+
}
|
| 186 |
+
|
| 187 |
+
@staticmethod
|
| 188 |
+
def push_graphql_delete(
|
| 189 |
+
graphql_endpoint: str,
|
| 190 |
+
infohashes: Sequence[str],
|
| 191 |
+
timeout: float = 5.0,
|
| 192 |
+
) -> Optional[Dict[str, Any]]:
|
| 193 |
+
"""Execute the `torrentDelete` GraphQL mutation against a running Bitmagnet endpoint."""
|
| 194 |
+
mutation = BitmagnetArmigerBridge.build_graphql_delete_mutation(infohashes)
|
| 195 |
+
if mutation is None:
|
| 196 |
+
return None
|
| 197 |
+
req = urllib.request.Request(
|
| 198 |
+
graphql_endpoint,
|
| 199 |
+
data=json.dumps(mutation).encode("utf-8"),
|
| 200 |
+
headers={"Content-Type": "application/json"},
|
| 201 |
+
method="POST",
|
| 202 |
+
)
|
| 203 |
+
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
| 204 |
+
return json.loads(resp.read().decode("utf-8"))
|
| 205 |
+
|
| 206 |
+
|
| 207 |
+
def create_server(host: str = "localhost", port: int = 8096) -> ThreadingHTTPServer:
|
| 208 |
+
model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
|
| 209 |
+
variant = os.environ.get("ARMIGER_VARIANT", "fp32")
|
| 210 |
+
pipeline = ArmigerUltraPipeline.from_directory(
|
| 211 |
+
model_dir,
|
| 212 |
+
variant=variant,
|
| 213 |
+
rsi_mode=DHTRSIMode(os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")),
|
| 214 |
+
preload_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
|
| 215 |
+
)
|
| 216 |
+
bridge = BitmagnetArmigerBridge(
|
| 217 |
+
pipeline,
|
| 218 |
+
use_zero_cue_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
|
| 219 |
+
prune_honeypots_and_malware=os.environ.get("ARMIGER_PRUNE_MALWARE", "true").lower() == "true",
|
| 220 |
+
prune_dead_swarms=os.environ.get("ARMIGER_PRUNE_DEAD_SWARMS", "false").lower() == "true",
|
| 221 |
+
)
|
| 222 |
+
|
| 223 |
+
class Handler(BaseHTTPRequestHandler):
|
| 224 |
+
def _send_json(self, status: int, payload: Dict[str, Any]) -> None:
|
| 225 |
+
body = json.dumps(payload).encode("utf-8")
|
| 226 |
+
self.send_response(status)
|
| 227 |
+
self.send_header("Content-Type", "application/json")
|
| 228 |
+
self.send_header("Content-Length", str(len(body)))
|
| 229 |
+
self.end_headers()
|
| 230 |
+
self.wfile.write(body)
|
| 231 |
+
|
| 232 |
+
def do_GET(self) -> None:
|
| 233 |
+
if self.path == "/healthz":
|
| 234 |
+
self._send_json(200, {"status": "ok", "service": "bitmagnet-armiger-bridge", "variant": variant})
|
| 235 |
+
else:
|
| 236 |
+
self._send_json(404, {"error": "not_found"})
|
| 237 |
+
|
| 238 |
+
def do_POST(self) -> None:
|
| 239 |
+
length = int(self.headers.get("Content-Length", "0"))
|
| 240 |
+
raw = self.rfile.read(length) if length > 0 else b"{}"
|
| 241 |
+
payload = json.loads(raw.decode("utf-8"))
|
| 242 |
+
if self.path == "/v1/bitmagnet/classify":
|
| 243 |
+
items = payload.get("torrents") or payload.get("items") or []
|
| 244 |
+
result = bridge.evaluate_batch(items)
|
| 245 |
+
gql_url = os.environ.get("BITMAGNET_GRAPHQL_URL", "").strip()
|
| 246 |
+
if gql_url and result["deleteInfoHashes"]:
|
| 247 |
+
try:
|
| 248 |
+
bridge.push_graphql_delete(gql_url, result["deleteInfoHashes"])
|
| 249 |
+
except Exception as exc:
|
| 250 |
+
result["graphqlPushError"] = str(exc)
|
| 251 |
+
self._send_json(200, result)
|
| 252 |
+
else:
|
| 253 |
+
self._send_json(404, {"error": "not_found"})
|
| 254 |
+
|
| 255 |
+
def log_message(self, fmt: str, *args: Any) -> None:
|
| 256 |
+
return
|
| 257 |
+
|
| 258 |
+
return ThreadingHTTPServer((host, port), Handler)
|
| 259 |
+
|
| 260 |
+
|
| 261 |
+
if __name__ == "__main__":
|
| 262 |
+
listen_host = os.environ.get("ARMIGER_BRIDGE_HOST", "localhost")
|
| 263 |
+
listen_port = int(os.environ.get("ARMIGER_BRIDGE_PORT", "8096"))
|
| 264 |
+
srv = create_server(listen_host, listen_port)
|
| 265 |
+
print(f"Bitmagnet <-> Armiger Ultra bridge listening on {listen_host}:{listen_port}")
|
| 266 |
+
srv.serve_forever()
|
examples/bitmagnet/docker-compose.yml
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
services:
|
| 2 |
+
bitmagnet:
|
| 3 |
+
image: ghcr.io/bitmagnet-io/bitmagnet:latest
|
| 4 |
+
container_name: bitmagnet
|
| 5 |
+
restart: unless-stopped
|
| 6 |
+
depends_on:
|
| 7 |
+
postgres:
|
| 8 |
+
condition: service_healthy
|
| 9 |
+
ports:
|
| 10 |
+
- "3333:3333"
|
| 11 |
+
- "3334:3334/tcp"
|
| 12 |
+
- "3334:3334/udp"
|
| 13 |
+
environment:
|
| 14 |
+
POSTGRES_HOST: postgres
|
| 15 |
+
POSTGRES_NAME: bitmagnet
|
| 16 |
+
POSTGRES_USER: postgres
|
| 17 |
+
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
|
| 18 |
+
command:
|
| 19 |
+
- worker
|
| 20 |
+
- run
|
| 21 |
+
- --keys=http_server
|
| 22 |
+
- --keys=queue_server
|
| 23 |
+
- --keys=dht_crawler
|
| 24 |
+
|
| 25 |
+
armiger-bitmagnet-bridge:
|
| 26 |
+
image: python:3.12-slim
|
| 27 |
+
container_name: armiger-bitmagnet-bridge
|
| 28 |
+
restart: unless-stopped
|
| 29 |
+
working_dir: /app
|
| 30 |
+
depends_on:
|
| 31 |
+
- bitmagnet
|
| 32 |
+
ports:
|
| 33 |
+
- "8096:8096"
|
| 34 |
+
environment:
|
| 35 |
+
ARMIGER_MODEL_DIR: /app
|
| 36 |
+
ARMIGER_VARIANT: "${ARMIGER_VARIANT:-fp32}"
|
| 37 |
+
ARMIGER_BRIDGE_HOST: "localhost"
|
| 38 |
+
ARMIGER_BRIDGE_PORT: "8096"
|
| 39 |
+
BITMAGNET_GRAPHQL_URL: "http://bitmagnet:3333/graphql"
|
| 40 |
+
ARMIGER_PRUNE_MALWARE: "true"
|
| 41 |
+
ARMIGER_PRUNE_DEAD_SWARMS: "false"
|
| 42 |
+
volumes:
|
| 43 |
+
- ../..:/app:ro
|
| 44 |
+
command:
|
| 45 |
+
- sh
|
| 46 |
+
- -c
|
| 47 |
+
- "pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/bitmagnet/bitmagnet_armiger_bridge.py"
|
| 48 |
+
|
| 49 |
+
armiger-torznab-proxy:
|
| 50 |
+
image: python:3.12-slim
|
| 51 |
+
container_name: armiger-torznab-proxy
|
| 52 |
+
restart: unless-stopped
|
| 53 |
+
working_dir: /app
|
| 54 |
+
depends_on:
|
| 55 |
+
- bitmagnet
|
| 56 |
+
ports:
|
| 57 |
+
- "8097:8097"
|
| 58 |
+
environment:
|
| 59 |
+
ARMIGER_MODEL_DIR: /app
|
| 60 |
+
ARMIGER_VARIANT: "${ARMIGER_VARIANT:-int8}"
|
| 61 |
+
UPSTREAM_TORZNAB_URL: "http://bitmagnet:3333/torznab"
|
| 62 |
+
ARMIGER_PROXY_PORT: "8097"
|
| 63 |
+
ARMIGER_REQUIRE_ENGLISH_AUDIO: "false"
|
| 64 |
+
ARMIGER_ALLOW_ENGLISH_SUBS: "true"
|
| 65 |
+
ARMIGER_REJECT_DEAD_SWARMS: "true"
|
| 66 |
+
volumes:
|
| 67 |
+
- ../..:/app:ro
|
| 68 |
+
command:
|
| 69 |
+
- sh
|
| 70 |
+
- -c
|
| 71 |
+
- "pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/arr-stack/armiger_torznab_proxy.py"
|
| 72 |
+
|
| 73 |
+
postgres:
|
| 74 |
+
image: postgres:16-alpine
|
| 75 |
+
container_name: bitmagnet-postgres
|
| 76 |
+
restart: unless-stopped
|
| 77 |
+
environment:
|
| 78 |
+
POSTGRES_DB: bitmagnet
|
| 79 |
+
POSTGRES_USER: postgres
|
| 80 |
+
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
|
| 81 |
+
volumes:
|
| 82 |
+
- bitmagnet_pgdata:/var/lib/postgresql/data
|
| 83 |
+
healthcheck:
|
| 84 |
+
test: ["CMD-SHELL", "pg_isready -d bitmagnet -U postgres"]
|
| 85 |
+
interval: 10s
|
| 86 |
+
timeout: 5s
|
| 87 |
+
retries: 5
|
| 88 |
+
|
| 89 |
+
volumes:
|
| 90 |
+
bitmagnet_pgdata:
|
examples/dh-waypoint/07-armiger-ultra.plugin.yml
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Drop this file into DH_WAYPOINT_PROCESSORS_PLUGIN_DIR (default /etc/dh-waypoint/plugins.d).
|
| 2 |
+
#
|
| 3 |
+
# Full Armiger Ultra integration wiring:
|
| 4 |
+
# 1. Batch classification (`POST /v1/classify`, `dh-waypoint.processor.v1`)
|
| 5 |
+
# 2. Cohort admission gate (`POST /v1/admit`, `dh-waypoint.processors.admission.v1`)
|
| 6 |
+
# 3. 4-tier closed-loop outcome feedback (`POST /v1/feedback`, `dh-waypoint.processors.feedback.v1`)
|
| 7 |
+
|
| 8 |
+
processor:
|
| 9 |
+
name: armiger-ultra
|
| 10 |
+
endpoint: http://armiger-ultra:8095/v1/classify
|
| 11 |
+
timeout: 5s
|
| 12 |
+
max_batch: 64
|
| 13 |
+
required: false
|
| 14 |
+
forward_files: true
|
| 15 |
+
max_files: 64
|
| 16 |
+
forward_trackers: true
|
| 17 |
+
forward_swarm: true
|
| 18 |
+
|
| 19 |
+
admission:
|
| 20 |
+
enabled: true
|
| 21 |
+
endpoint: http://armiger-ultra:8095/v1/admit
|
| 22 |
+
timeout: 3s
|
| 23 |
+
max_batch: 64
|
| 24 |
+
max_files: 64
|
| 25 |
+
fail_open: true
|
| 26 |
+
|
| 27 |
+
learning:
|
| 28 |
+
enabled: true
|
| 29 |
+
feedback_endpoint: http://armiger-ultra:8095/v1/feedback
|
| 30 |
+
feedback_timeout: 5s
|
| 31 |
+
feedback_batch_size: 64
|
| 32 |
+
flush_interval: 15s
|
| 33 |
+
queue_size: 2048
|
examples/dh-waypoint/armiger_waypoint_processor.py
ADDED
|
@@ -0,0 +1,270 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""DH-Waypoint processorapi/v1 HTTP Service for Armiger & Armiger Ultra.
|
| 2 |
+
|
| 3 |
+
Implements all three DH-Waypoint external processor contracts:
|
| 4 |
+
- `POST /v1/classify` (`dh-waypoint.processor.v1`)
|
| 5 |
+
- `POST /v1/admit` (`dh-waypoint.processors.admission.v1`)
|
| 6 |
+
- `POST /v1/feedback` (`dh-waypoint.processors.feedback.v1`)
|
| 7 |
+
- `GET /healthz`
|
| 8 |
+
"""
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
| 12 |
+
import json
|
| 13 |
+
import os
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
import sys
|
| 16 |
+
from typing import Any, Dict, List, Mapping, Optional
|
| 17 |
+
|
| 18 |
+
ROOT_DIR = Path(__file__).resolve().parents[2]
|
| 19 |
+
if str(ROOT_DIR) not in sys.path:
|
| 20 |
+
sys.path.insert(0, str(ROOT_DIR))
|
| 21 |
+
|
| 22 |
+
from armiger_dht_rsi import CandidateObservation, DHTRSIMode
|
| 23 |
+
from armiger_ultra_pipeline import ArmigerUltraPipeline, ArmigerUltraPrediction
|
| 24 |
+
|
| 25 |
+
# Map Armiger 11-class labels to DH-Waypoint canonical content types
|
| 26 |
+
WAYPOINT_CONTENT_TYPE_MAP: Dict[str, str] = {
|
| 27 |
+
"movie": "movie",
|
| 28 |
+
"tv_show": "tv_show",
|
| 29 |
+
"music": "music",
|
| 30 |
+
"ebook": "ebook",
|
| 31 |
+
"audiobook": "audiobook",
|
| 32 |
+
"comic": "comic",
|
| 33 |
+
"software": "software",
|
| 34 |
+
"game": "game",
|
| 35 |
+
"course": "ebook",
|
| 36 |
+
"xxx": "xxx",
|
| 37 |
+
"unknown": "unknown",
|
| 38 |
+
}
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class ArmigerWaypointAdapter:
|
| 42 |
+
"""Translates between DH-Waypoint processorapi/v1 envelopes and ArmigerUltraPipeline."""
|
| 43 |
+
|
| 44 |
+
def __init__(
|
| 45 |
+
self,
|
| 46 |
+
pipeline: ArmigerUltraPipeline,
|
| 47 |
+
*,
|
| 48 |
+
use_zero_cue_catalog: bool = True,
|
| 49 |
+
reject_dead_swarms: bool = False,
|
| 50 |
+
) -> None:
|
| 51 |
+
self.pipeline = pipeline
|
| 52 |
+
self.use_zero_cue_catalog = use_zero_cue_catalog
|
| 53 |
+
self.reject_dead_swarms = reject_dead_swarms
|
| 54 |
+
|
| 55 |
+
@staticmethod
|
| 56 |
+
def _torrent_to_row(t: Mapping[str, Any]) -> Dict[str, Any]:
|
| 57 |
+
swarm_in = t.get("swarm") if isinstance(t.get("swarm"), Mapping) else {}
|
| 58 |
+
seeders = int(swarm_in.get("seeders", t.get("seeders", 0)) or 0)
|
| 59 |
+
leechers = int(swarm_in.get("leechers", t.get("leechers", 0)) or 0)
|
| 60 |
+
files = t.get("files") if isinstance(t.get("files"), list) else []
|
| 61 |
+
return {
|
| 62 |
+
"infohash": str(t.get("info_hash") or t.get("infohash") or ""),
|
| 63 |
+
"name": str(t.get("name") or t.get("title") or ""),
|
| 64 |
+
"size": int(t.get("size") or 0),
|
| 65 |
+
"files": files,
|
| 66 |
+
"file_count": int(t.get("files_count") or len(files) or 1),
|
| 67 |
+
"sources": t.get("sources") if isinstance(t.get("sources"), list) else [],
|
| 68 |
+
"swarm": {
|
| 69 |
+
"tracker_known": bool(seeders > 0 or leechers > 0 or swarm_in),
|
| 70 |
+
"tracker_seeders": seeders,
|
| 71 |
+
"tracker_leechers": leechers,
|
| 72 |
+
"tracker_completed": int(swarm_in["completed"]) if "completed" in swarm_in and swarm_in["completed"] is not None else None,
|
| 73 |
+
},
|
| 74 |
+
}
|
| 75 |
+
|
| 76 |
+
def handle_classify_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
|
| 77 |
+
"""Handle `POST /v1/classify` (`dh-waypoint.processor.v1`)."""
|
| 78 |
+
torrents = request_body.get("torrents") or []
|
| 79 |
+
rows = [self._torrent_to_row(t) for t in torrents]
|
| 80 |
+
predictions: List[ArmigerUltraPrediction] = self.pipeline.classify_batch(
|
| 81 |
+
rows,
|
| 82 |
+
use_zero_cue_catalog=self.use_zero_cue_catalog,
|
| 83 |
+
)
|
| 84 |
+
|
| 85 |
+
decisions: List[Dict[str, Any]] = []
|
| 86 |
+
for pred in predictions:
|
| 87 |
+
gates_passed = bool(pred.mondrian_gates["passed_all"])
|
| 88 |
+
viability_tier = str(pred.swarm_viability["viability_tier"])
|
| 89 |
+
if not gates_passed:
|
| 90 |
+
reason = (
|
| 91 |
+
pred.csam_safety.get("reason")
|
| 92 |
+
or pred.mondrian_gates.get("safety_gate_reason")
|
| 93 |
+
or ("honeypot_trap_blocked" if pred.honeypot_trap["is_honeypot_trap"] else "safety_gate_rejected")
|
| 94 |
+
)
|
| 95 |
+
decisions.append({
|
| 96 |
+
"info_hash": pred.infohash,
|
| 97 |
+
"action": "delete",
|
| 98 |
+
"reason": str(reason),
|
| 99 |
+
})
|
| 100 |
+
continue
|
| 101 |
+
|
| 102 |
+
content_type = WAYPOINT_CONTENT_TYPE_MAP.get(pred.category, "unknown")
|
| 103 |
+
tags = [
|
| 104 |
+
f"armiger:category:{pred.category}",
|
| 105 |
+
f"armiger:lang:{pred.lang}",
|
| 106 |
+
f"armiger:viability:{viability_tier}",
|
| 107 |
+
]
|
| 108 |
+
if pred.anime == "anime":
|
| 109 |
+
tags.append("armiger:anime")
|
| 110 |
+
dub_status = pred.heads["anime"].get("anime_dub_status")
|
| 111 |
+
if dub_status:
|
| 112 |
+
tags.append(f"armiger:anime_dub:{dub_status}")
|
| 113 |
+
if pred.english_subs == "has_eng_subs":
|
| 114 |
+
tags.append("armiger:eng_subs")
|
| 115 |
+
|
| 116 |
+
decisions.append({
|
| 117 |
+
"info_hash": pred.infohash,
|
| 118 |
+
"action": "enrich",
|
| 119 |
+
"content_type": content_type,
|
| 120 |
+
"languages": ["en"] if pred.lang == "english" else [],
|
| 121 |
+
"tags": tags,
|
| 122 |
+
"attributes": {
|
| 123 |
+
"armiger.category": pred.category,
|
| 124 |
+
"armiger.category_confidence": f"{pred.heads['category']['confidence']:.4f}",
|
| 125 |
+
"armiger.lang": pred.lang,
|
| 126 |
+
"armiger.viability_tier": viability_tier,
|
| 127 |
+
"armiger.viability_prob": f"{pred.swarm_viability['viability_prob']:.4f}",
|
| 128 |
+
"armiger.survival_net_prob": f"{pred.swarm_viability['survival_net_prob']:.4f}",
|
| 129 |
+
"armiger.catalog_source": str(pred.plugin_trace["plugin_3_zero_cue_catalog"]["resolution_source"]),
|
| 130 |
+
},
|
| 131 |
+
})
|
| 132 |
+
|
| 133 |
+
return {
|
| 134 |
+
"schema": "dh-waypoint.processor.v1",
|
| 135 |
+
"decisions": decisions,
|
| 136 |
+
}
|
| 137 |
+
|
| 138 |
+
def handle_admission_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
|
| 139 |
+
"""Handle `POST /v1/admit` (`dh-waypoint.processors.admission.v1`)."""
|
| 140 |
+
candidates = request_body.get("candidates") or []
|
| 141 |
+
rows = [self._torrent_to_row(c) for c in candidates]
|
| 142 |
+
predictions = self.pipeline.classify_batch(
|
| 143 |
+
rows,
|
| 144 |
+
use_zero_cue_catalog=self.use_zero_cue_catalog,
|
| 145 |
+
)
|
| 146 |
+
|
| 147 |
+
decisions: List[Dict[str, Any]] = []
|
| 148 |
+
for cand, pred in zip(candidates, predictions):
|
| 149 |
+
info_hash = str(cand.get("info_hash") or pred.infohash)
|
| 150 |
+
gates_passed = bool(pred.mondrian_gates["passed_all"])
|
| 151 |
+
viability_tier = str(pred.swarm_viability["viability_tier"])
|
| 152 |
+
admit = gates_passed and (not self.reject_dead_swarms or viability_tier != "dead")
|
| 153 |
+
reason = "passed_mondrian_conformal_gates"
|
| 154 |
+
if not gates_passed:
|
| 155 |
+
reason = str(
|
| 156 |
+
pred.csam_safety.get("reason")
|
| 157 |
+
or pred.mondrian_gates.get("safety_gate_reason")
|
| 158 |
+
or "honeypot_or_safety_gate_blocked"
|
| 159 |
+
)
|
| 160 |
+
elif not admit:
|
| 161 |
+
reason = "dead_swarm_rejected"
|
| 162 |
+
|
| 163 |
+
decisions.append({
|
| 164 |
+
"info_hash": info_hash,
|
| 165 |
+
"admit": admit,
|
| 166 |
+
"cohort": pred.category if admit else "rejected",
|
| 167 |
+
"score": float(pred.swarm_viability["viability_prob"] if admit else 0.0),
|
| 168 |
+
"reason": reason,
|
| 169 |
+
})
|
| 170 |
+
|
| 171 |
+
return {
|
| 172 |
+
"schema": "dh-waypoint.processors.admission.v1",
|
| 173 |
+
"decisions": decisions,
|
| 174 |
+
}
|
| 175 |
+
|
| 176 |
+
def handle_feedback_batch(self, request_body: Mapping[str, Any]) -> Dict[str, Any]:
|
| 177 |
+
"""Handle `POST /v1/feedback` (`dh-waypoint.processors.feedback.v1`)."""
|
| 178 |
+
events = request_body.get("events") or []
|
| 179 |
+
accepted = 0
|
| 180 |
+
verified = 0
|
| 181 |
+
for ev in events:
|
| 182 |
+
accepted += 1
|
| 183 |
+
tier = str(ev.get("tier") or "")
|
| 184 |
+
outcome = str(ev.get("outcome") or "")
|
| 185 |
+
if tier in ("t3_files", "t4_playback") and outcome in ("verified", "imported", "completed"):
|
| 186 |
+
cand = CandidateObservation(
|
| 187 |
+
source_adapter="ffprobe",
|
| 188 |
+
raw_title=str(ev.get("name") or ev.get("info_hash") or "verified_item"),
|
| 189 |
+
claimed_category=str(ev.get("content_type") or "movie"),
|
| 190 |
+
download_completed=True,
|
| 191 |
+
bytes_downloaded=int(ev.get("size") or 100_000_000),
|
| 192 |
+
total_bytes=int(ev.get("size") or 100_000_000),
|
| 193 |
+
catalog_id_type="movie_catalog" if ev.get("content_type") == "movie" else "series_catalog",
|
| 194 |
+
catalog_id_value=str(ev.get("info_hash") or "verified"),
|
| 195 |
+
stream_metadata={
|
| 196 |
+
"video_codec": str((ev.get("attributes") or {}).get("video_codec", "hevc")),
|
| 197 |
+
"audio_codec": str((ev.get("attributes") or {}).get("audio_codec", "eac3")),
|
| 198 |
+
"audio_languages": [str((ev.get("attributes") or {}).get("audio_lang", "eng"))],
|
| 199 |
+
},
|
| 200 |
+
)
|
| 201 |
+
ok, _, _ = self.pipeline.ingest_rsi_feedback(cand)
|
| 202 |
+
if ok:
|
| 203 |
+
verified += 1
|
| 204 |
+
|
| 205 |
+
return {
|
| 206 |
+
"schema": "dh-waypoint.processors.feedback.v1",
|
| 207 |
+
"accepted": accepted,
|
| 208 |
+
"verified": verified,
|
| 209 |
+
}
|
| 210 |
+
|
| 211 |
+
|
| 212 |
+
def create_server(host: str = "localhost", port: int = 8095) -> ThreadingHTTPServer:
|
| 213 |
+
model_dir = os.environ.get("ARMIGER_MODEL_DIR", str(ROOT_DIR))
|
| 214 |
+
variant = os.environ.get("ARMIGER_VARIANT", "fp32")
|
| 215 |
+
rsi_mode_str = os.environ.get("ARMIGER_DHT_RSI_MODE", "pull_only")
|
| 216 |
+
pipeline = ArmigerUltraPipeline.from_directory(
|
| 217 |
+
model_dir,
|
| 218 |
+
variant=variant,
|
| 219 |
+
adult_threshold=float(os.environ.get("ARMIGER_ADULT_THRESHOLD", "0.35")),
|
| 220 |
+
junk_threshold=float(os.environ.get("ARMIGER_JUNK_THRESHOLD", "0.25")),
|
| 221 |
+
scope_threshold=float(os.environ.get("ARMIGER_SCOPE_THRESHOLD", "0.35")),
|
| 222 |
+
rsi_mode=DHTRSIMode(rsi_mode_str),
|
| 223 |
+
preload_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
|
| 224 |
+
)
|
| 225 |
+
adapter = ArmigerWaypointAdapter(
|
| 226 |
+
pipeline,
|
| 227 |
+
use_zero_cue_catalog=os.environ.get("ARMIGER_USE_ZERO_CUE_CATALOG", "true").lower() == "true",
|
| 228 |
+
reject_dead_swarms=os.environ.get("ARMIGER_REJECT_DEAD_SWARMS", "false").lower() == "true",
|
| 229 |
+
)
|
| 230 |
+
|
| 231 |
+
class Handler(BaseHTTPRequestHandler):
|
| 232 |
+
def _send_json(self, status: int, payload: Dict[str, Any]) -> None:
|
| 233 |
+
body = json.dumps(payload).encode("utf-8")
|
| 234 |
+
self.send_response(status)
|
| 235 |
+
self.send_header("Content-Type", "application/json")
|
| 236 |
+
self.send_header("Content-Length", str(len(body)))
|
| 237 |
+
self.end_headers()
|
| 238 |
+
self.wfile.write(body)
|
| 239 |
+
|
| 240 |
+
def do_GET(self) -> None:
|
| 241 |
+
if self.path == "/healthz":
|
| 242 |
+
self._send_json(200, {"status": "ok", "engine": "armiger-ultra", "variant": variant})
|
| 243 |
+
else:
|
| 244 |
+
self._send_json(404, {"error": "not_found"})
|
| 245 |
+
|
| 246 |
+
def do_POST(self) -> None:
|
| 247 |
+
length = int(self.headers.get("Content-Length", "0"))
|
| 248 |
+
raw = self.rfile.read(length) if length > 0 else b"{}"
|
| 249 |
+
payload = json.loads(raw.decode("utf-8"))
|
| 250 |
+
if self.path == "/v1/classify":
|
| 251 |
+
self._send_json(200, adapter.handle_classify_batch(payload))
|
| 252 |
+
elif self.path == "/v1/admit":
|
| 253 |
+
self._send_json(200, adapter.handle_admission_batch(payload))
|
| 254 |
+
elif self.path == "/v1/feedback":
|
| 255 |
+
self._send_json(200, adapter.handle_feedback_batch(payload))
|
| 256 |
+
else:
|
| 257 |
+
self._send_json(404, {"error": "not_found"})
|
| 258 |
+
|
| 259 |
+
def log_message(self, fmt: str, *args: Any) -> None:
|
| 260 |
+
return
|
| 261 |
+
|
| 262 |
+
return ThreadingHTTPServer((host, port), Handler)
|
| 263 |
+
|
| 264 |
+
|
| 265 |
+
if __name__ == "__main__":
|
| 266 |
+
listen_host = os.environ.get("ARMIGER_HOST", "localhost")
|
| 267 |
+
listen_port = int(os.environ.get("ARMIGER_PORT", "8095"))
|
| 268 |
+
srv = create_server(listen_host, listen_port)
|
| 269 |
+
print(f"Armiger DH-Waypoint processor listening on {listen_host}:{listen_port}")
|
| 270 |
+
srv.serve_forever()
|
examples/dh-waypoint/docker-compose.yml
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
services:
|
| 2 |
+
dh-waypoint:
|
| 3 |
+
image: ghcr.io/spencernorton/dh-waypoint:v5.6.0
|
| 4 |
+
container_name: dh-waypoint
|
| 5 |
+
restart: unless-stopped
|
| 6 |
+
depends_on:
|
| 7 |
+
postgres:
|
| 8 |
+
condition: service_healthy
|
| 9 |
+
armiger-ultra:
|
| 10 |
+
condition: service_healthy
|
| 11 |
+
ports:
|
| 12 |
+
- "3333:3333"
|
| 13 |
+
- "3334:3334/tcp"
|
| 14 |
+
- "3334:3334/udp"
|
| 15 |
+
environment:
|
| 16 |
+
POSTGRES_HOST: postgres
|
| 17 |
+
POSTGRES_PORT: "5432"
|
| 18 |
+
POSTGRES_DB: dh_waypoint
|
| 19 |
+
POSTGRES_USER: postgres
|
| 20 |
+
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
|
| 21 |
+
DH_WAYPOINT_PROCESSORS_PLUGIN_DIR: /etc/dh-waypoint/plugins.d
|
| 22 |
+
volumes:
|
| 23 |
+
- ./07-armiger-ultra.plugin.yml:/etc/dh-waypoint/plugins.d/07-armiger-ultra.plugin.yml:ro
|
| 24 |
+
command:
|
| 25 |
+
- worker
|
| 26 |
+
- run
|
| 27 |
+
- --keys=http_server
|
| 28 |
+
- --keys=queue_server
|
| 29 |
+
- --keys=dht_crawler
|
| 30 |
+
|
| 31 |
+
armiger-ultra:
|
| 32 |
+
image: python:3.12-slim
|
| 33 |
+
container_name: armiger-ultra
|
| 34 |
+
restart: unless-stopped
|
| 35 |
+
working_dir: /app
|
| 36 |
+
ports:
|
| 37 |
+
- "8095:8095"
|
| 38 |
+
environment:
|
| 39 |
+
ARMIGER_MODEL_DIR: /app
|
| 40 |
+
ARMIGER_VARIANT: "${ARMIGER_VARIANT:-fp32}"
|
| 41 |
+
ARMIGER_PORT: "8095"
|
| 42 |
+
ARMIGER_ADULT_THRESHOLD: "0.35"
|
| 43 |
+
ARMIGER_JUNK_THRESHOLD: "0.25"
|
| 44 |
+
ARMIGER_SCOPE_THRESHOLD: "0.35"
|
| 45 |
+
ARMIGER_USE_ZERO_CUE_CATALOG: "true"
|
| 46 |
+
ARMIGER_DHT_RSI_MODE: "${ARMIGER_DHT_RSI_MODE:-pull_only}"
|
| 47 |
+
volumes:
|
| 48 |
+
- ../..:/app:ro
|
| 49 |
+
- armiger_state:/var/lib/armiger
|
| 50 |
+
command:
|
| 51 |
+
- sh
|
| 52 |
+
- -c
|
| 53 |
+
- "pip install --no-cache-dir numpy onnxruntime cryptography && python3 examples/dh-waypoint/armiger_waypoint_processor.py"
|
| 54 |
+
healthcheck:
|
| 55 |
+
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8095/healthz')"]
|
| 56 |
+
interval: 15s
|
| 57 |
+
timeout: 5s
|
| 58 |
+
retries: 5
|
| 59 |
+
|
| 60 |
+
postgres:
|
| 61 |
+
image: postgres:16-alpine
|
| 62 |
+
container_name: dh-waypoint-postgres
|
| 63 |
+
restart: unless-stopped
|
| 64 |
+
environment:
|
| 65 |
+
POSTGRES_DB: dh_waypoint
|
| 66 |
+
POSTGRES_USER: postgres
|
| 67 |
+
POSTGRES_PASSWORD: "${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}"
|
| 68 |
+
volumes:
|
| 69 |
+
- dh_waypoint_pgdata:/var/lib/postgresql/data
|
| 70 |
+
healthcheck:
|
| 71 |
+
test: ["CMD-SHELL", "pg_isready -d dh_waypoint -U postgres"]
|
| 72 |
+
interval: 10s
|
| 73 |
+
timeout: 5s
|
| 74 |
+
retries: 5
|
| 75 |
+
|
| 76 |
+
volumes:
|
| 77 |
+
armiger_state:
|
| 78 |
+
dh_waypoint_pgdata:
|