DibaGuard / docs /CLEAN_MODELS.md
DibaAi's picture
DibaGuard: public licence-clean release (Dibachain)
5d97275 verified
|
Raw History Blame Contribute Delete
34 kB

DibaGuard licence-clean models (roadmap 16)

Open-release copy (DibaAi/DibaGuard-Open). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (http_v2) were measured with that model, which is not part of the open release; the open release ships the licence-clean network models described in CLEAN_MODELS.md. Licence statements inside this document are superseded by DATA_LICENSES.md, CLEAN_MODELS.md and the repository's LICENSE, LICENSE-WEIGHTS and NOTICE files. Sections of the original that were written in Persian were removed from this copy; nothing else was edited.

Status as of 2026-10-31 (final). Goal: a model set that may ship in a commercial, closed product, trained only on sources that docs/DATA_LICENSES.md (the licence audit of the same day) marks GREEN. Nothing shipped was touched: the clean models sit next to the shipped ones under clean_* names in DibaAi/dibaguard-results-private, the clean corpora under new folders in DibaAi/dibaguard-data-private. Fetched reports: D:/build-tmp/guard/clean/.

0. Verdict and head-to-head metrics

Network side (tier-1 prior, tier-2 GRU, entity detectors): the clean set can replace the shipped one, with one stated weakness (IoT / UNSW-NB15-style networks, below) and one legal condition (LUFlow is CC BY-SA 4.0; the LUFlow-trained parts need sign-off or a retrain without LUFlow, section 6). HTTP side: it cannot yet. The clean HTTP model is strong on traffic like its own (SR-BH, ModSecurity days 21-30 with an SR-BH reference) but collapses on an application it never saw (the shipped test suite, the attack-simulation lab, a zanbil.ir reference): keep the shipped HTTP model out of a commercial build and do not ship the clean one in its place until the gaps in section 6 are closed. A third attempt (own attack grammar, benign clients without a User-Agent) traded one failure for another (section 0.4) and is not a replacement either.

How these numbers were made (guard_jobs/clean_eval.py, Kaggle CPU runs guard-clean-eval-flows, guard-clean-eval-http; reports clean_vs_shipped_flows.json, clean_vs_shipped_http.json): both model sets score the same rows. Flows: the site reference is the testbed's cal split (benign rows), the scored rows its later test split, threshold = conformal quantile of the reference at alpha = 1e-3 for both models. HTTP: profile and threshold from half of the suite's benign rows (the same 512 profile rows for both models), FPR on the other half; attack-only suites use a named benign reference. 95 % CIs are cluster bootstraps (clusters = source address for flows, payload group or client for HTTP), so one attacker's many rows cannot narrow them; CIs on a single cluster collapse to a point. "Seen" = the model's training data contained (other rows of) that testbed. RED / AMBER rows (nfv3_*, http_v1, zanbil) were scored for this comparison only, never trained on; those numbers are internal.

0.1 Tier-1 prior (nf_entity_mono, the engine's per-flow prior), per-flow, alpha = 1e-3

For GREEN testbeds the clean side is a leave-out retrain of the same model (clean_lodo_mono, trained here without the testbed; SIMARGL 2021 and 2022 are held out together because they are one network a year apart), so neither side saw the scored network.

scored network (rows: benign / attack) seen by shipped AUROC [95 % CI] clean AUROC [95 % CI] shipped FPR / TPR clean FPR / TPR [TPR CI]
SIMARGL2022 (337k / 914k) neither 0.968 [0.836, 1.000] 0.999 [0.999, 1.000] 0.0023 / 0.52 0.0010 / 0.49 [0.01, 1.00]
APPRAISE (767k / 483k) neither 0.614 [0.376, 0.828] 0.985 [0.224, 0.989] 0.0072 / 0.00 0.0020 / 0.94 [0.00, 0.94]
ACI-IoT-2023, author release (166k / 142k) neither 0.622 [0.422, 0.757] 0.644 [0.273, 0.779] 0.0011 / 0.0003 0.0002 / 0.17 [0.00, 0.42]
CSE-CIC-IDS2018 Tue 20-02, AWS (1.09M / 160k) neither 0.949 [0.925, 0.968] 1.000 [1.000, 1.000] 0.0000 / 0.00 0.0037 / 1.00 (FPR above alpha)
SIMARGL2021 (654k / 596k) shipped 0.911 [0.789, 0.991] 0.932 [0.857, 1.000] 0.00005 / 0.00 0.0005 / 0.00
NF-CSE-CIC-IDS2018-v3 (1.11M / 141k) shipped 0.987 [0.982, 0.993] 0.994 [0.993, 0.996] 0.0000 / 0.00 0.00004 / 0.00
NF-ToN-IoT-v3 (713k / 537k) shipped 0.955 [0.801, 0.998] 0.879 [0.000, 0.950] 0.0002 / 0.004 0.0000 / 0.00
NF-UNSW-NB15-v3 (540k / 52k) shipped 0.958 [0.923, 0.979] 0.066 [0.015, 0.102] 0.0005 / 0.00 0.0000 / 0.00
NF-BoT-IoT-v3 (403 / 1.25M) shipped 0.996 0.993 β€” (403 benign) β€”

Per family where it matters: SIMARGL2022 DoS 1.00 (shipped) vs 0.01 (clean leave-out), port scans 0.00 vs 1.00; APPRAISE reconnaissance 0.00 vs 0.94. Per-flow TPR at alpha = 1e-3 is near zero for both priors on most networks: the prior is not where the engine detects (it feeds the per-source evidence tier, next table).

Per-source evidence (what the engine alarms on), SIMARGL2021, unseen network, Mondrian-fused tiers, judged per (source, minute) at alpha = 1e-3: clean (SIMARGL 2021 + 2022 held out, run guard-clean-seq-4): FPR 0.00081 [0.0007, 0.0010], attacking source-minutes caught 8 / 8 [0.68, 1.00] (Wilson), every attacking source caught, median 447 flows to the first alarm; shipped (its own leave-one-out report seq_v1_realworld): FPR 0.0029 [0.0026, 0.0032], 16 / 17 [0.73, 0.99], every attacking source caught. The clean set holds the false-alarm budget better (0.08 % vs 0.29 % for a 0.1 % target) at the same detection; note that the two runs sample different attacking sources.

0.2 Entity detectors (fallback tier 1) and the LUFlow detector, alpha = 1e-3

scored network seen by shipped AUROC / FPR / TPR clean AUROC / FPR / TPR
APPRAISE clean 0.189 / 0.0004 / 0.00 0.9996 / 0.0008 / 0.999
ACI-IoT-2023 clean 0.342 / 0.0026 / 0.046 0.675 / 0.036 / 0.075
SIMARGL2022 clean 0.9995 / 0.0003 / 0.88 1.000 / 0.0004 / 0.9999
CSE-CIC-IDS2018 AWS clean 0.9999 / 0.0012 / 1.00 1.000 / 0.0007 / 0.9999
NF-CSE-CIC-IDS2018-v3 shipped 0.997 / 0.0006 / 0.98 0.965 / 0.0030 / 0.005
NF-ToN-IoT-v3 shipped 0.850 / 0.0014 / 0.14 0.092 / 0.0000 / 0.00
NF-UNSW-NB15-v3 shipped 1.000 / 0.0003 / 1.00 0.066 / 0.0027 / 0.018
LUFlow 2020 (lu detector) both 0.984 / 0.0074 / 0.41 0.9998 / 0.0030 / 0.99
LUFlow 2021 (lu detector) clean 0.984 / 0.0002 / 0.39 0.999 / 0.00004 / 0.95

Each detector is strong where it trained and weak elsewhere; no row here is a "neither saw it" comparison except through the leave-out reports (clean_entity_v1_grouped.json): with the whole SIMARGL family held out, the clean ent detector scores SIMARGL2021 at AUROC 0.947, TPR 0.69 at FPR 2e-4, entity-minute AUROC 0.995; the shipped detector's leave-one-out on SIMARGL2021 (card) was 0.538, TPR 0.27, entity-minute 0.865. The LUFlow detector gain is mostly the clock fix of section 4.

0.3 Tier-2 GRU (seqnet)

No same-row head-to-head (scoring needs the sequence windows of the training job); leave-out numbers from the training reports, noctx, site-calibrated at 1e-3: SIMARGL2021 with its sibling held out β€” clean AUROC 0.991, entity-minute 0.9999, fused with the trees AUROC 1.000 at FPR 0.0024 / TPR 1.00; shipped (SIMARGL2021 alone held out) AUROC 0.822, entity-minute 0.939, fused 0.708 at FPR 0.024 / TPR 0.48. Other clean leave-outs: APPRAISE 0.987, ACI-IoT 0.934, CSE-CIC-IDS2018 AWS 0.899. Weakness: trained on NetFlow-style testbeds only, the GRU is inverted on LUFlow (AUROC 0.0002 with both LUFlow years held out); with one LUFlow year in training it reaches 0.80 / 0.999. The exported clean GRU is trained on all seven testbeds, LUFlow included.

0.4 HTTP (http_tier1_v2), site-calibrated alpha = 1e-3, same rows for both models

suite (benign / attack rows) seen by shipped AUROC [CI] clean AUROC [CI] shipped FPR / TPR [TPR CI] clean FPR / TPR [TPR CI]
WordPress POST, SR-BH subset (1.5k / 16.4k) neither (clean saw other requests of the same site) 0.817 [0.791, 0.840] 0.915 [0.904, 0.922] 0.000 / 0.002 [0.001, 0.002] 0.0019 / 0.31 [0.30, 0.32]
ModSecurity 2025 days 21-30, reference SR-BH benign (34.6k attacks) days 1-20 by both β€” β€” β€” / 0.76 [0.66, 0.85] β€” / 0.96 [0.95, 0.97]
same attacks, reference zanbil.ir benign days 1-20 by both β€” β€” β€” / 0.91 [0.84, 0.96] β€” / 0.19 [0.14, 0.24]
attack-simulation lab, DVWA (229 / 30k) neither 0.998 [0.997, 0.998] 0.422 [0.406, 0.437] β€” β€”
shipped test split (58k / 52k) shipped 0.997 [0.997, 0.997] 0.803 [0.799, 0.808] 0.0010 / 0.87 [0.86, 0.87] 0.0011 / 0.06 [0.06, 0.06]
shipped evasion suite (17k obfuscated SQLi / XSS) shipped β€” β€” β€” / 0.86 [0.86, 0.87] β€” / 0.08 [0.08, 0.09]
Persian benign, zanbil (4.5k) shipped β€” β€” FPR 0.000 (own threshold 0.000) FPR 0.000 (own threshold 0.33)
clean test split (37k / 84k) clean 0.821 [0.815, 0.825] 0.992 [0.992, 0.993] 0.0008 / 0.22 [0.22, 0.23] 0.0011 / 0.93 [0.93, 0.93]
clean evasion suite (own mutations, 1.8k) clean β€” β€” β€” / 0.50 [0.48, 0.52] β€” / 0.76 [0.74, 0.78]
Persian benign, own generator (8.9k) clean β€” β€” FPR 0.0002 FPR 0.000

Per family on the shipped test split (TPR at the site threshold, shipped vs clean): XSS 0.89 vs 0.02, SQLi 0.88 vs 0.10, command injection 0.96 vs 0.004, traversal 0.93 vs 0.21. On the clean test split: SQLi 0.21 vs 0.99, XSS 0.16 vs 0.51, traversal 0.75 vs 0.85, LFI 0.71 vs 0.60, command injection 0.20 vs 0.75. ModSecurity per family with the SR-BH reference (shipped vs clean): SQLi 0.23 vs 1.00, protocol 0.66 vs 0.98, bad bots 0.90 vs 0.81; with the zanbil reference the clean model loses every family (bots 0.01, LFI 0.17, SQLi 0.03).

What this says: each HTTP model is good on its own data distribution and poor on the other's, and the clean model does not generalise to a site it never saw: on zanbil.ir's benign traffic its scores are high enough that the site threshold rises above most attacks, on the DVWA lab it ranks attacks below benign, and with no site calibration (its own threshold) it would flag a third of real Persian requests. Two spot checks show why: a benign GET /shop?item=12 without a User-Agent gets logit +26.7 (SR-BH attack tools rarely send one, SR-BH browsers always do), and a plain <script>alert(1)</script> in a query is missed (SR-BH has no XSS label at all; the GREEN XSS lists are small).

Third attempt (http_clean3: + own attack grammar of 16k payloads, 15 % of synthetic benign clients without a User-Agent; run guard-clean-http-3f, CPU, model clean_http_tier1_v2_c3f.npz; head-to-head clean_vs_shipped_http_c3.json), TPR at the site threshold, run 2 -> run 3 (shipped): WordPress POST 0.31 -> 0.19 (0.002); ModSecurity, SR-BH reference 0.96 -> 0.97 (0.76); ModSecurity, zanbil reference 0.19 -> 0.38 (0.91); lab AUROC 0.42 -> 0.87 (0.998); shipped test split 0.06 -> 0.05 (0.87); shipped evasion 0.08 -> 0.08 (0.86); clean test split 0.93 -> 0.71 (0.22); clean evasion 0.76 -> 0.23 (0.50); Persian zanbil FPR at the model's own threshold 0.33 -> 0.022 (0.000). The no-User-Agent false alarm is gone (logit 26.7 -> 3.1, threshold 8.7), but at its own threshold run 3 now misses the plain UNION SQLi, the plain XSS and the ../../etc/passwd spot checks. Neither clean HTTP model is fit to ship; clean_http_tier1_v2.npz (run 2) stays the reference clean HTTP artefact.

0.5 Can the clean set replace the shipped one?

part replace? why
tier-1 prior nf_entity_mono yes equal or better on every GREEN network neither saw (AUROC 0.97 -> 0.999, 0.61 -> 0.985, 0.62 -> 0.64, 0.95 -> 1.00); better calibrated per-source evidence on SIMARGL2021; worse on UNSW-NB15 (0.96 -> 0.07) and ToN-IoT (0.96 -> 0.88), which are the shipped model's own training networks and have no GREEN twin
tier-2 GRU seqnet yes leave-out results on GREEN networks at or above the shipped model's; same size (381 KB), parity 4.6e-5
entity detectors (fallback) yes as the prior; the LUFlow detector is clearly better (clock fix)
HTTP http_tier1_v2 no shipped test split TPR 0.87 -> 0.06 (run 2) / 0.05 (run 3), lab AUROC 0.998 -> 0.42 / 0.87, ModSecurity with a zanbil reference 0.91 -> 0.19 / 0.38; better only on SR-BH-like traffic

1. What each clean model is

The v2 engine (dibaguard/engine.py) loads three files: seq_v1/nf_entity_mono.npz (tier-1 prior; falls back to entity_v1/nf_entity_detector.npz), seq_v1/seqnet.npz (tier-2 GRU) and http_v2/http_tier1_v2.npz. Each has a clean twin. To run the engine on the clean set, copy the clean_* files under the shipped names into a separate model folder (for example models_clean/seq_v1/nf_entity_mono.npz) and Guard.load that folder; no runtime change is needed. Every clean model carries data: "licence-clean (roadmap 16)", the per-source licence and the attribution block (notice) in its own metadata.

clean artefact (results repo) replaces trained on (licence) run
models/seq_v1/clean_nf_entity_mono.npz seq_v1/nf_entity_mono.npz (monotone prior, tier 1) SIMARGL2021, SIMARGL2022, APPRAISE, ACI-IoT-2023 (CC BY 4.0); CSE-CIC-IDS2018 Tue 20-02 from AWS (CIC grant); LUFlow 2020 + 2021 (CC BY-SA 4.0) guard-clean-seq-1
models/seq_v1/clean_seqnet.npz + clean_seq_v1_info.json seq_v1/seqnet.npz (GRU d=64, tier 2) same seven testbeds guard-clean-seq-3
models/entity_v1/clean_nf_entity_detector.npz entity_v1/nf_entity_detector.npz the five nf testbeds above guard-clean-entity-1
models/entity_v1/clean_lu_entity_detector.npz entity_v1/lu_entity_detector.npz LUFlow 2020 + 2021 (CC BY-SA 4.0) guard-clean-entity-1
models/http_v2/clean_http_tier1_v2.npz http_v2/http_tier1_v2.npz SR-BH 2020 (CC0), ModSecurity 2025 days 1-20 (CC BY 4.0), PayloadsAllTheThings + SecLists payloads (MIT), OWASP CRS regression tests (Apache-2.0), own synthetic traffic guard-clean-http-2

Not retrained (legacy, not loaded by the v2 engine, still RED as sourced): flows_v1/* and http_v1/http_tier1. See section 6.

2. Data actually used, with licences and the attribution each one requires

2.1 Flows with addresses and timestamps: entity_clean/ (data repo)

Built by guard_jobs/entity_build.py --clean (launched as --job entity_clean, which attaches only GREEN Kaggle datasets to the kernel), runs guard-clean-entity-build and guard-clean-entity-fix. Every row has licence; the builder raises before writing if any row is outside the allowlist.

testbed source as read (owner's own release) licence rows span sources families cite
simargl2021 Kaggle h2020simargl/simargl2021-network-intrusion-detection-dataset CC BY 4.0 8,000,000 443 h 46,873 benign 3.92 M, portscan 2.50 M, DoS (RUDY, slowloris) 1.58 M Mihailescu et al., Sensors 21(13):4319, 2021
simargl2022 Kaggle h2020simargl/simargl2022 (days 15, 16, 19 Mar 2022) CC BY 4.0 8,000,000 100 h 35,512 benign 3.30 M, portscan 3.88 M, dos 0.82 M H2020 SIMARGL, SIMARGL2022 dataset (CC BY 4.0)
appraise Kaggle ittibydgoszcz/appraise-h2020-real-labelled-netflow-dataset CC BY 4.0 7,570,072 1,912 h 1,339 benign 4.78 M, recon 2.36 M, ddos 0.43 M, bruteforce 949 ITTI, APPRAISE H2020 real labelled NetFlow dataset (CC BY 4.0)
aci2023 Kaggle emilynack/aci-iot-network-traffic-dataset-2023 (ACI-IoT-2023.csv, the author's release) CC BY 4.0 1,231,411 102 h 159,381 benign 0.33 M, portscan 0.44 M, floods 0.29 M, recon 0.15 M, slowloris, dictionary Nack et al., MILCOM 2024; doi:10.21227/qacj-3x32
cic2018aws CSE-CIC-IDS2018, AWS open-data bucket (publisher's copy), processed day Thuesday-20-02-2018 CIC grant: redistribute / republish in any form, citation required 7,948,748 12 h 31,291 benign 7.37 M, DDoS LOIC-HTTP 0.58 M Sharafaldin, Habibi Lashkari, Ghorbani, ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018
lu2020 Kaggle mryanm/luflow-network-intrusion-detection-data-set (author's release), 2020 files CC BY-SA 4.0 6,856,505 180 h 1,249 benign 5.12 M, malicious 1.74 M Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517
lu2021 same, 2021 files CC BY-SA 4.0 6,430,074 2,193 h 691 benign 5.40 M, malicious 1.03 M as lu2020

lu2022 was built too but is not used: the 2022 files carry a single source address, so there is no per-source stream. It stays in entity_clean/ and every trainer run passes the testbed list explicitly.

2.2 HTTP: http_clean2/ (data repo; http_clean/ is the first, superseded build)

Built by guard_jobs/http_clean.py (--job http_clean, or http_build --clean), run guard-clean-http-build3 with --prefix http_clean2 --per-file 20000 --per-family 40000 --placements 2. 1,006,457 rows; by licence: CC0 776,942, own 132,738, MIT 61,719, Apache-2.0 35,058.

source as read licence what it contributes cite / notice
srbh SR-BH 2020 multi-label, Harvard Dataverse doi:10.7910/DVN/OGOIXX, file data_capec_multilabel.csv (licence read from the Dataverse API: CC0-1.0) CC0 1.0 878,968 real requests to a WordPress honeypot (357,536 attacks, CAPEC labels mapped to families), benign carriers, the wild suite (last 20 % of time) Sureda Riera et al., Computers & Security 2022, doi:10.1016/j.cose.2022.102788
wp_post SR-BH 2020 WordPress POST subset, Kaggle ahmeterenceylan/http-request-dataset CC0 upstream (uploader label MIT) held-out suite, never trained on; SR-BH's own 28,381 POST requests are left out of srbh for that reason as srbh
modsec ModSecurity thirty-day archive, Zenodo 17178461 (http_v2/modsec.parquet, already built) CC BY 4.0 days 1-20 join training (--modsec-train 1, as the shipped http_v2_final2), days 21-30 are the modsec suite Lucz, G. (2025), doi:10.5281/zenodo.17178461
patt PayloadsAllTheThings */Intruder(s)/* lists, commit 08ea40e7 MIT attack payloads placed into benign carriers MIT notice
seclists SecLists Fuzzing/* payload lists (family by file keyword), commit 0008f434 MIT attack payloads placed into benign carriers MIT notice
crs OWASP CRS tests/regression/tests for rule files 913, 921, 930-944, commit ca679721 Apache-2.0 3,798 attack requests (positive tests) and 802 benign hard negatives (negative tests) Apache-2.0 + NOTICE
synth own generator in guard_jobs/http_clean.py own 80,000 benign shop / blog / API requests (English and Persian parameters, JSON bodies) as rows and carriers; benign values placed exactly like the attack payloads; the evasion variants (own mutation engine) -
lab Kaggle moradrawashdeh/attack-simulation-lab (uploader's own lab) Apache-2.0 evaluation suite only (labels are per host, not per request) Apache-2.0 notice

Two cleaning rules were added to the payload lists (both data quality, not licence): a list line is kept only if it carries the family's active syntax (http_clean.ACTIVE; the Intruder / Fuzzing folders also hold plain word lists such as event-handler names, bare file paths and LDAP attribute names, which are not attacks by themselves), and files named after the PortSwigger cheat sheet are dropped (the audit treats PortSwigger-derived content as unlicensed, row 17).

2.3 Not used, and why

candidate reason
NF-UQ-NIDS-v2, NF-*-v3 (athena21, thanhphuc1810 "MIT", seyhed, ...) RED (UQ academic-only terms; uploader labels grant nothing)
dhoogla CIC-IDS2017 / CSE-CIC-IDS2018 / CIC-DDoS2019, dhoogla LUFlow RED as sourced (CC BY-NC-SA mirrors)
CIC-IDS2017 and CIC-DDoS2019 originals Not available from the publisher without a manual step: every cicresearch.ca/CICDataset/... path probed from a Kaggle kernel redirects to the UNB dataset index (download behind UNB's registration form); no Kaggle copy is published by UNB/CIC. Not used.
CSE-CIC-IDS2018 other days the AWS processed CSVs of the other nine days have no Src IP / Dst IP (unusable for per-source streams; usable for a single-flow cic view, see section 6)
ACI-IoT-2023 / IoMT-2024 CIC-BCCC-NRC copies (phungvannamanh, amineipad) AMBER (uploader MIT label, CIC/BCCC terms unconfirmed); the author's own ACI release is used instead
MU-IoT GREEN, but the release has normalised features only (no addresses, no timestamps)
zanbil.ir AMBER (raw log withheld by the owner, real client IPs)
CSIC 2010, mreowie, obf_sqli, obf_xss, syed_xss, syed_sqli, sambit RED / AMBER
own red-team generators (scripts/redteam_*.py, dibaguard.shield.playbook) GREEN and they do produce flow tables, but they are the live lab's own test traffic; training on them would grade the models on their training data. Kept as the independent lab check.

3. The licence gate

  • guard_jobs/licences.py: CLEAN = {cc-by-4.0, cc-by-sa-4.0, cc0-1.0, mit, apache-2.0, cic-unb-cite, own}; GREEN = source id -> (licence, the origin the audit cleared, attribution); OTHER = the audit's tags for every RED / AMBER source.
  • Every corpus row has a licence column (entity_build now writes it for every testbed, clean or not; http_build's tags were corrected to the audit's values).
  • entity_build --clean: refuses a testbed that is not GREEN or whose Kaggle ref differs from the cleared origin; require_clean on every testbed's rows before the parquet is written.
  • http_clean: require_source on every row's source, require_clean on the whole frame before writing.
  • entity_train --clean, seq_train --clean, http_train2 --clean: read only entity_clean/ / http_clean*, re-check every loaded testbed / split (source GREEN, every row's licence in CLEAN), raise otherwise; outputs are clean_* files from separate scratch folders, so a clean run cannot upload a shipped file name.
  • tests/test_clean.py builds every clean loader on synthetic files, checks the outputs, and checks that a RED testbed (nfv3_cic2018, aciiot2023) and a single tampered row (uq-academic-only) are refused.

4. Loader defects found on the way (fixed in --clean mode only; the shipped corpora keep their bytes)

  1. LUFlow time unit. The author's release stores time_start in microseconds, plus a few rows on other scales or at 0. The builder divided by 1000, so the shipped entity_v1/lu2020 stream ran on a 1000x stretched clock with a near-zero first row (span_h 442,511,328 in its manifest). Every horizon of the entity state (10 s / 2 min / 30 min) was therefore effectively 1000x shorter on LUFlow for lu_entity_detector, seqnet and nf_entity_mono. Clean build: per-row scaling and a plausibility window (lu2020 now spans 180 h).
  2. SYN-only on exporters without a client/server flag split (SIMARGL, APPRAISE): client_flags was 0, so the SYN-only share was always 0 on SIMARGL2021, a dataset that is one-third SYN scans. Clean build: rebuilt from the cumulative TCP_FLAGS and whether the server sent any packet.
  3. ACI-IoT-2023 author timestamps mix spellings across days (2023-01-11 = 1 Nov). Clean build: each row is read every plausible way and the reading inside the README's capture week is kept (span 102 h, was 5,669 h).
  4. CICFlowMeter rows with an unparseable time became -9.2e9 s and sorted first; now dropped.

5. Training-protocol results (leave-one-dataset-out, as the training jobs report them)

Reports in D:/build-tmp/guard/clean/. These are each model set's own leave-out numbers, on different testbeds than the shipped reports, so they are context for section 0, not a head-to-head.

  • clean_entity_v1_train.json / clean_entity_v1_grouped.json (runs guard-clean-entity-1, -2): nf view, ent features, held-out flow AUROC β€” SIMARGL2021 0.966 (0.947 with SIMARGL2022 also held out), SIMARGL2022 0.996 (0.979 grouped), APPRAISE 0.934, ACI-IoT 0.969, CSE-CIC-IDS2018 AWS 0.997; lu view LUFlow 2020 0.997, 2021 0.995. Site-calibrated realised FPR at 1e-3 departs from the target on SIMARGL2021 (0.12 single, 2e-4 grouped), SIMARGL2022 grouped (0.019) and ACI-IoT (0.021): few sources with long-lived state are not exchangeable between periods, the limit the card already states for UNSW-NB15.
  • clean_seq_v1_train.json / _grouped / _keepatt (runs guard-clean-seq-1, -2, -4): GRU and fused numbers in section 0.3; anomaly tier: monotone prior leave-out AUROC SIMARGL2022 1.000, CSE-CIC-IDS2018 AWS 0.9998, APPRAISE 0.24 on the source-capped sample of seq-1 (591 attack flows in its test split) and 0.992 when every attacking source is kept (seq-4), ACI-IoT 0.71; the label-free site-only tier (quant) 0.99-1.00 on SIMARGL / CSE-CIC and 0.81 on ACI-IoT. In seq-1 the SIMARGL2021 test split lost its only attacker to the random source sampling (the shipped run happened to keep one), which is why seq-4 re-ran the anomaly protocol with --keep-attack-sources 1. LUFlow anomaly folds are one-sided after source capping (as in the shipped report).
  • clean_seq_v1_export.json (guard-clean-seq-3): exported clean GRU, d = 64, 95,293 parameters, 381 KB, numpy / torch parity 4.6e-5, 130k flows/s on one core.
  • clean_http_v2_train.json (guard-clean-http-2, corpus http_clean2): leave-one-source-out SR-BH AUROC 0.873 (no site context; 0.42 with context) / site TPR 0.24 at FPR 7e-4, novelty tier 0.984, fused 0.974; WordPress POST 0.947 / 0.17 (shipped: 0.693 / 0.17); CRS 0.57; lab 0.14 (novelty 0.996). Final model: test AUROC 0.992, evasion TPR 0.77, ModSecurity last ten days caught 0.958 with the SR-BH reference, 3,387 clients 0.895; 7,849 KB; 1,888 req/s on one core in Python. clean_http_v2_train_run1_corpus_v1.json is the first attempt (corpus http_clean, before the payload-list cleaning): XSS 0.18, LFI 0.15, evasion 0.29, kept for the record.

6. Gaps: what still depends on data that is not cleared

  1. HTTP model. The clean HTTP model is not a replacement (section 0.4). What it lacks is benign and attack variety from applications other than one WordPress honeypot: the shipped model's strength on CSIC / zanbil / payload-set traffic comes from RED / AMBER data (CSIC 2010, zanbil.ir, mreowie, obfuscated SQLi / XSS sets). Routes, in order of value: (a) the author's written OK for zanbil.ir (CC0, raw log withheld; DATA_LICENSES Β§4.3), which restores a real non-WordPress benign site; (b) CSIC 2010 permission; (c) more GREEN benign applications (requests generated from the CC0 APIs.guru OpenAPI directory, CRS negative tests at scale); (d) more own attack grammar and WAF-A-MoLE (MIT) mutation of GREEN payloads. Until then the commercial build has no licence-clean HTTP model; the site-only novelty tier (no training data) and the playbook signatures remain clean.
  2. IoT / UNSW-NB15-style networks. The clean prior is weak on NF-UNSW-NB15 (AUROC 0.07) and below the shipped model on NF-ToN-IoT. There is no GREEN twin of UNSW-NB15, BoT-IoT or ToN-IoT (all UNSW Canberra, academic-only); ToN-IoT is "allowable after asking the author" β€” the one permission worth asking for the network side. MU-IoT (CC BY 4.0) has no addresses; CIC-IoT2023 from UNB would need its PCAPs re-flowed.
  3. CIC-IDS2017 and CIC-DDoS2019 could not be fetched from the publisher without the registration form; only the CSE-CIC-IDS2018 day with addresses (Tuesday 20-02) is in. Downloading the UNB originals by hand (GREEN with citation) and re-flowing the 2018 PCAPs from AWS would add a CIC testbed with every attack family.
  4. Legacy artefacts not retrained: flows_v1/* (single-flow nf / cic / lu detectors) and http_v1/http_tier1 are still RED as sourced. They are not loaded by the v2 engine. A clean cic view is possible from the ten AWS CSE-CIC-IDS2018 processed CSVs (no addresses needed for single flows), a clean lu view from the author's LUFlow.
  5. Go runtime files: the clean models exist as .npz; the flat .bin / .bin.json exports and parity fixtures for the Go runtime (dibaguard.parity.npz_to_flat) were not produced for them.
  6. Open licence points that touch the clean set (from DATA_LICENSES Β§5, unchanged): LUFlow is CC BY-SA 4.0 β€” whether trained weights are Adapted Material is unsettled, so the LUFlow-trained parts (clean lu detector, and the GRU / prior, which include LUFlow 2020-21) need legal sign-off or a retrain without LUFlow; the uploader identities of the SIMARGL, LUFlow and ACI-IoT Kaggle accounts are inferred, not verified; the CIC FAQ wording should be confirmed by e-mail; PayloadsAllTheThings / SecLists files credit upstream cheat sheets whose own terms were not checked file by file (the PortSwigger-named files were dropped).
  7. Shipped models carry the old loader defects (section 4): the shipped LUFlow-trained parts ran on a 1000x stretched clock and SIMARGL2021 without the SYN-only signal. A rebuild of entity_v1 with the --clean fixes applied to every testbed would be needed before comparing future shipped retrains to these numbers.

7. Runs, files and how to check them

run (account) job and arguments output
guard-clean-survey (1) survey_clean publisher channels, SR-BH, payload repos, lab, ACI author release
guard-clean-entity-build (1), guard-clean-entity-fix (1) entity_clean (entity_build --clean); fix run --testbeds aci2023,lu2020,lu2021 entity_clean/*.parquet + manifest
guard-clean-http-build (2, stray first version), -build2 (2), -build3 (2), -build4 (2) http_clean; build3 --prefix http_clean2 --per-file 20000 --per-family 40000 --placements 2; build4 adds --own-attacks 1 --no-ua-share 0.15 --prefix http_clean3 http_clean/, http_clean2/, http_clean3/
guard-clean-entity-1 (1, GPU) entity_train --clean --views nf,lu --sets base,ent,both --testbeds <7> --iters 1000 --max-rows 1500000 models/entity_v1/clean_{nf,lu}_entity_detector.npz, reports/clean_entity_v1_train.json
guard-clean-entity-2 (1, GPU) same, --lodo "simargl2021+simargl2022,..." --no-export reports/clean_entity_v1_grouped.json
guard-clean-seq-1 (1, GPU) seq_train --clean --d 128 --ctx 0 --fset ent --protocols temporal,lodo,anomaly --iters 300 --rolling 1800 models/seq_v1/clean_nf_entity_mono.npz, reports/clean_seq_v1_train.json
guard-clean-seq-2 (1, GPU) grouped leave-out, --export-mono 0 reports/clean_seq_v1_grouped.json
guard-clean-seq-3 (2, GPU) --d 64 --protocols export models/seq_v1/clean_seqnet.npz, clean_seq_v1_info.json
guard-clean-seq-4 (1, GPU) anomaly protocol with --keep-attack-sources 1 reports/clean_seq_v1_keepatt.json
guard-clean-http-1 (2, GPU) http_train2 --clean on http_clean; first version killed (out of memory at the SR-BH leave-out, fixed by --loso-cap) reports/clean_http_v2_train.json (renamed locally _run1_corpus_v1)
guard-clean-http-2 (2, GPU) http_train2 --clean --corpus http_clean2 --protocols loso,final --ctx 0,1 --modsec-train 1 models/http_v2/clean_http_tier1_v2.npz, reports/clean_http_v2_train.json
guard-clean-http-3 (1, CPU), guard-clean-http-3f (2, CPU) same on http_clean3 (--loso srbh,wp_post,crs); -3f final only models/http_v2/clean_http_tier1_v2_c3.npz / _c3f.npz, reports/clean_http_v2_train3*.json
guard-clean-eval-flows (1, CPU), guard-clean-eval-http (2, CPU), guard-clean-eval-http3 (2, CPU) clean_eval --parts flows / --parts http / --parts http --clean-http models/http_v2/clean_http_tier1_v2_c3f.npz reports/clean_vs_shipped_{flows,http,http_c3}.json

Check any run: python kag.py status --run-name <run> --result (from D:/diba-slm/guard); full log after the kernel ends: python kag.py log --run-name <run>. Reports and models are in DibaAi/dibaguard-results-private, corpora in DibaAi/dibaguard-data-private (both private).

Code added or changed (training side only, guard_jobs/ and tests/): licences.py (new), entity_clean.py (new), http_clean.py (new), clean_eval.py (new), survey_clean.py (new); --clean and related options in entity_build.py, entity_train.py, seq_train.py, http_build.py, http_train2.py; tests/test_clean.py. Default (non-clean) behaviour of the existing jobs is unchanged except that entity_build now writes a licence column and http_build's licence tags carry the audit's values.

8. Attribution block for a commercial release of the clean network models

CSE-CIC-IDS2018 (Sharafaldin, Habibi Lashkari, Ghorbani, ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018); RoEduNet-SIMARGL2021 (Mihailescu et al., Sensors 21(13):4319, 2021, CC BY 4.0); SIMARGL2022 (H2020 SIMARGL, CC BY 4.0); APPRAISE H2020 real labelled NetFlow (ITTI, CC BY 4.0); ACI-IoT-2023 (Nack et al., MILCOM 2024, doi:10.21227/qacj-3x32, CC BY 4.0); LUFlow (Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517, CC BY-SA 4.0). For an HTTP model built from the clean corpus, add SR-BH 2020 (CC0; cite doi:10.1016/j.cose.2022.102788), the ModSecurity archive (doi:10.5281/zenodo.17178461, CC BY 4.0), and the MIT / Apache-2.0 notices of PayloadsAllTheThings, SecLists and the OWASP CRS. The same block is stored in every clean model's metadata (notice).