armiger-ultra / docs /plugins /dht-db-sync.md
spencernorton's picture
Rename zero_cue_2000 and multilang_3000 evaluation split labels
44d3f66 verified
|
Raw History Blame Contribute Delete
13.9 kB

Armiger Plugin — Modular, CSAM-Protected, Encrypted DHT Database Sync (armiger-dht-db-sync)

Decentralized BEP-44 / BEP-46 database synchronization engine featuring multi-layer CSAM & abuse blocking, Ed25519-signed blinded SHA-256 tombstone purging, modular fine-grained category & language subscriptions with isolated xxx containment, X25519 + ChaCha20-Poly1305 AEAD encryption, and $K$-of-$N$ distinct-ASN Sybil quorum verification.
Designed for decentralized indexers (Bitmagnet and self-hosted library fleets) powered by NorviTech/armiger-ultra.

Compatible with Armiger Ultra Compatible with Armiger Licence Apache-2.0 Tests Passing


Architecture & Security Invariants

1. Multi-Layer CSAM & Harmful-Content Protection

  • Multi-Stage ContentFilter Parity: Full unicode NFKD decomposition, diacritic stripping, zero-width / bidirectional formatting removal, and leetspeak deobfuscation (0→o, 1→i, 3→e, 4/@→a, $→s, 7/+→t).
  • Deterministic & Structural Pre-Ingest Blocker: Inspects titles, file lists, and tags before shard generation. Blocks child sexual abuse material (CSAM / CSAE), non-consensual sexual exploitation, and extreme violence / snuff. Disguised executable droppers (.mp4.exe, .scr, .bat, .vbs, .ps1) in media categories are unconditionally rejected.
  • Ed25519-Signed Blinded SHA-256 Tombstone Purge Engine: Revocations use SHA-256("armiger-tombstone-v1:" || infohash). Manifests require strictly increasing sequence counters (seq > current_seq) and valid Ed25519 root signatures. Nodes automatically sweep and purge matching entries from local databases and ban re-ingestion.
  • Mandatory Armiger Conformal Safety Gate: Mondrian conformal gates (junk < 0.1010, scope < 0.4770, adult >= 0.5451 routed strictly to adult containment).

2. Modular Sharded Subscription Architecture

  • Partition Dimensions: Partitioned across Category (10 modules) $\times$ Language (14 modules) $\times$ Quality Tier (7 tiers) $\times$ Time Epoch.
  • 10 Category Modules: movie, tv_show, music, game, software, anime, book, comic, course, and xxx.
  • Strict xxx Isolation & Opt-In: Adult content is strictly disabled by default (enable_xxx=False). Adult shards use a segregated namespace and storage table (adult_torrents). Mainstream searches never query or return adult records, and mainstream shards never carry adult entries.
  • Selective Shard Download: Subscribers configure exact subsets (e.g. categories={"movie", "tv_show"}, languages={"en", "ja"}). DHT lookups only fetch subscribed shards—zero network bandwidth or storage is spent on unwanted modules.

3. X25519 + ChaCha20-Poly1305 AEAD Encryption

  • Key Derivation: HKDF-SHA256 derives 32-byte ChaCha20-Poly1305 keys salted by shard_id and bounded by epoch.
  • Authenticated Associated Data (AAD): armiger-shard-aad-v1:{shard_id}:{epoch}:{seq}:{merkle_root} is bound to every ciphertext block.
  • Anti-Transplant & Replay Defense: Any tampering, sequence alteration, or cross-shard ciphertext transplant immediately fails Poly1305 authentication.

4. BEP-44 / BEP-46 Root-of-Trust + K-of-N Distinct-ASN Quorum

  • BEP-44 Mutable DHT Pointers: Target key calculated via SHA1(pubkey || salt).
  • BEP-46 Infohash Pointers: Canonical infohash references encrypted micro-shards.
  • Sybil-Resistant $K$-of-$N$ Quorum: Requires $K \ge 5$ cryptographically valid Ed25519 attestations from $\ge 5$ distinct Autonomous System Numbers (ASNs) (e.g., AS15169 Google, AS13335 Cloudflare, AS16509 Amazon, AS7922 Comcast, AS3356 Lumen). Multi-node Sybil attacks within the same ASN or /24 subnet are automatically rejected.
  • Binary Merkle Tree Verification: Each shard payload forms a binary Merkle tree of canonical record hashes. Manifest carries signed merkle_root. Supports individual inclusion proofs for auditability.

5. Anonymity & Dynamic Opt-In/Opt-Out

  • Zero PII: No IP addresses, user tokens, passkeys, or client headers are stored or broadcast.
  • Sanitized Metadata: Private tracker passkeys ([passkey=...], ?passkey=...) are stripped prior to hashing.
  • Sync Modes: OFF, PULL_ONLY (download only), SUBSCRIBE_AND_SEED (distribute subscribed shards), PUBLISHER.

Quickstart

from armiger_dht_db_sync import (
    ArmigerDHTDatabaseSync,
    SubscriptionConfig,
    SyncMode,
    create_mock_quorum_attestations,
    generate_ed25519_keypair,
)
from csam_guard import sign_tombstone_list

# 1. Initialize root keys and subscription
root_priv, root_pub = generate_ed25519_keypair()
sub_cfg = SubscriptionConfig(
    categories={"movie", "anime"},
    languages={"en", "ja"},
    quality_tiers={"uhd_4k", "hd_1080p"},
    epochs={"2026_q4"},
    enable_xxx=False,  # Adult content strictly excluded
    sync_mode=SyncMode.PULL_ONLY,
)

sync_engine = ArmigerDHTDatabaseSync(
    subscription=sub_cfg,
    trusted_root_pubkeys_hex=[root_pub.hex()],
)

# 2. Ingest candidate record with CSAM & safety checks
ok, msg, record = sync_engine.ingest_candidate_record(
    infohash="0123456789abcdef0123456789abcdef01234567",
    title="Spring.2019.1080p.WEBM.AV1-OPEN",
    category="anime",
    language="ja",
    quality_tier="hd_1080p",
    epoch="2026_q4",
    p_adult=0.01,
    p_junk=0.001,
    p_excluded_scope=0.01,
)
print("Ingestion result:", ok, msg)

# 3. Search local database
results = sync_engine.db.search(query="Spring", category="anime")
print("Found torrents:", len(results))

# 4. Apply Ed25519-signed blinded tombstone purge
tombstone_manifest = sign_tombstone_list(
    seq=1,
    infohashes_or_blinded=["0123456789abcdef0123456789abcdef01234567"],
    private_key_bytes=root_priv,
)
ok, msg, new_count, purged = sync_engine.apply_signed_tombstones(tombstone_manifest)
print("Tombstone purge executed. Purged records:", purged)

Test Suite Execution

Run the comprehensive 27-point test suite:

python3 test_dht_db_sync.py -v

All 27 test cases pass cleanly:

test_ciphertext_tampering_defense ... ok
test_cross_shard_transplant_defense ... ok
test_key_derivation_deterministic_and_unique ... ok
test_round_trip_encryption_decryption ... ok
test_merkle_tree_construction_and_inclusion_proofs ... ok
test_quorum_verification_success ... ok
test_sybil_attack_rejected_duplicate_asns ... ok
test_armiger_mondrian_safety_gate ... ok
test_direct_csam_detection ... ok
test_disguised_malware_dropper_in_media_torrents ... ok
test_nonconsensual_and_extreme_violence_detection ... ok
test_private_tracker_passkey_sanitization ... ok
test_signed_tombstone_purge_lifecycle ... ok
test_text_normalization_and_leetspeak ... ok
test_bep44_target_derivation ... ok
test_ciphertext_sha256_mismatch_rejected ... ok
test_full_publish_subscribe_and_shard_application ... ok
test_merkle_root_mismatch_rejected ... ok
test_pre_ingest_reinspection_drops_bad_records_in_shard ... ok
test_stale_or_downgrade_sequence_rejected ... ok
test_sync_mode_off_disables_all_operations ... ok
test_untrusted_root_key_rejected ... ok
test_x25519_ecdh_key_exchange ... ok
test_database_table_segregation ... ok
test_selective_shard_filtering ... ok
test_strict_xxx_isolation_in_subscription ... ok
test_taxonomy_definitions ... ok

Ran 27 tests in 0.011s - OK

Part of the Armiger Modular Plugin Suite

Plugin Repository Bundled in Armiger Ultra? Footprint Core Capability & Measured Impact (Unmutated Ground-Truth Keys)
NorviTech/armiger-swarm-survival Yes (swarm_adapter.pt, swarm_survival.py) 167 KB (132 KB .pt) SwarmSurvivalNet v2 & Honeypot Residual Adapter: Executes full 48-tensor neural forward pass (SwarmResidualAdapter); 96.83% Multi-Source Corpus AUROC (N=10,278, vs. 53.02% raw seeders), 92.42% / 93.03% Public-Tracker AUROC (N=757 / N=493, vs. 85.17% raw seeders), 100.0% fake-seeder malware trap recall, 0.00% content-category flip guarantee.
NorviTech/armiger-swarm-priors Yes (swarm_priors.py) 29.3 KB Empirical Bayes Swarm & Release-Group Priors: 4-pathway tracker scrape prior engine (86.64% formula AUROC), mandatory modality dropout (p=0.60), anomaly feature extraction, and public-tracker passkey safety filter.
NorviTech/armiger-zero-cue-catalog Yes (zero_cue_plugin.py, validated_zero_cue_catalog.json) 26.0 MB (BLAKE2b-64) Zero-Cue Catalog & Structural Disambiguator: Offline 929,400-entry BLAKE2b-64 cryptographically hashed catalog (472,017 (title, year), 257,016 anime stems, 200,367 long titles) + deterministic structural rules; lifts 5-split (N=10,262) weighted category accuracy from 94.00% (Standalone Neural) to 95.42% (Fair Blind Hybrid), beyond_confirmation (N=1,025) from 92.88% to 95.12%, and zero_cue_2000 (N=2,000) from 87.40% to 91.50% (+4.10 pp).
NorviTech/armiger-audio-guard Yes (audio_language_guard.py) 3.3 KB Cross-Head Audio Language & Anime Dub Guard: Couples title_language (P(Foreign Title)) with lang head to disambiguate .en. prepositions, <Foreign>.ENG.Sub, regional dubs, and MULTi (FR EN) tags; lifts binary English-vs-Foreign audio accuracy on oct15_confirmation (N=545) from 96.88% (Standalone Neural) to 97.80% (92.75% 3-class N=772).
NorviTech/armiger-calibration-dht Yes (calibration.json, calibration_and_dht.py, armiger_dht_rsi.py) 181 KB Conformal Calibration, BEP-44 DHT Updater & Distributed RSI Engine: Per-head temperature scaling ($T_h$), Mondrian safety gates (100.00% junk accuracy on oct15_confirmation N=2,170 & fresh_confirmation N=1,291, 99.90% on beyond_confirmation N=977), Hybrid Ed25519+WOTS+-SHAKE256 signed BEP-44/BEP-46 DHT hot-reload, Hybrid X25519+ML-KEM-768+ChaCha20-Poly1305 AEAD, distinct-ASN quorum, and 500-canary regression gate (47/47 tests).
NorviTech/armiger-dht-db-sync Yes (csam_guard.py, armiger_dht_db_sync.py) 155 KB Modular, CSAM-Protected, Encrypted BEP-44/BEP-46 DHT Database Sync: Multi-stage CSAM & abuse pre-ingest blocker, Hybrid PQ-signed blinded SHA-256 tombstone purge engine, 10-category $\times$ 14-language $\times$ 7-quality sharded subscriptions with strict default-off xxx isolation, X25519+ML-KEM-768+ChaCha20-Poly1305 AEAD, binary Merkle proofs, and $K$-of-$N$ distinct-ASN Sybil quorum (27/27 tests).

Intended Use & Legal Compliance

Armiger, Armiger Ultra, and all bundled plugins are open-source machine learning and cryptographic metadata-classification utilities designed strictly for lawful use cases, including:

  • Organizing and indexing licensed, public-domain, Creative Commons, and open-source digital media, software distributions (e.g., Linux ISOs, open-source games), and educational archives (e.g., OpenCourseWare, LibriVox, Project Gutenberg).
  • Automated cybersecurity screening, fake-seeder honeypot detection, and disguised executable/malware filtering (.mkv.exe, .scr, .vbs) in peer-to-peer and decentralized storage research.
  • Child safety (CSAM/CSAE) pre-ingest blocking, cryptographic tombstone revocation, and adult/NSFW namespace isolation for self-hosted and decentralized indexers.

NorviTech does not condone, encourage, or support copyright infringement or the unauthorized distribution of copyrighted works. Users are solely responsible for ensuring that their use of these models and plugins complies with all applicable local, national, and international copyright and data-protection laws.


Part of the NorviTech Suite — open-source apps for the Linux desktop and the self-hosted stack

Armiger Ultra · Armiger · Swarm Survival · Swarm Priors · Zero-Cue Catalog · Audio Guard · Calibration & DHT RSI · DHT DB Sync · norvitech.com