Download docs/CLEAN_MODELS.md from Dibachain/DibaGuard: direct link, hf CLI and curl.
- Browser
- Download file 34 kB
-
https://huggingface.co/Dibachain/DibaGuard/resolve/main/docs/CLEAN_MODELS.md
- Command line
-
hf download hf://Dibachain/DibaGuard/docs/CLEAN_MODELS.md
-
curl -L -o CLEAN_MODELS.md https://huggingface.co/Dibachain/DibaGuard/resolve/main/docs/CLEAN_MODELS.md
DibaGuard licence-clean models (roadmap 16)
Open-release copy (DibaAi/DibaGuard-Open). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (
http_v2) were measured with that model, which is not part of the open release; the open release ships the licence-clean network models described inCLEAN_MODELS.md. Licence statements inside this document are superseded byDATA_LICENSES.md,CLEAN_MODELS.mdand the repository'sLICENSE,LICENSE-WEIGHTSandNOTICEfiles. Sections of the original that were written in Persian were removed from this copy; nothing else was edited.
Status as of 2026-10-31 (final). Goal: a model set that may ship in a commercial, closed product, trained only on
sources that docs/DATA_LICENSES.md (the licence audit of the same day) marks GREEN. Nothing shipped was touched:
the clean models sit next to the shipped ones under clean_* names in DibaAi/dibaguard-results-private, the clean
corpora under new folders in DibaAi/dibaguard-data-private. Fetched reports: D:/build-tmp/guard/clean/.
0. Verdict and head-to-head metrics
Network side (tier-1 prior, tier-2 GRU, entity detectors): the clean set can replace the shipped one, with one stated weakness (IoT / UNSW-NB15-style networks, below) and one legal condition (LUFlow is CC BY-SA 4.0; the LUFlow-trained parts need sign-off or a retrain without LUFlow, section 6). HTTP side: it cannot yet. The clean HTTP model is strong on traffic like its own (SR-BH, ModSecurity days 21-30 with an SR-BH reference) but collapses on an application it never saw (the shipped test suite, the attack-simulation lab, a zanbil.ir reference): keep the shipped HTTP model out of a commercial build and do not ship the clean one in its place until the gaps in section 6 are closed. A third attempt (own attack grammar, benign clients without a User-Agent) traded one failure for another (section 0.4) and is not a replacement either.
How these numbers were made (guard_jobs/clean_eval.py, Kaggle CPU runs guard-clean-eval-flows,
guard-clean-eval-http; reports clean_vs_shipped_flows.json, clean_vs_shipped_http.json): both model sets score
the same rows. Flows: the site reference is the testbed's cal split (benign rows), the scored rows its later
test split, threshold = conformal quantile of the reference at alpha = 1e-3 for both models. HTTP: profile and
threshold from half of the suite's benign rows (the same 512 profile rows for both models), FPR on the other half;
attack-only suites use a named benign reference. 95 % CIs are cluster bootstraps (clusters = source address for
flows, payload group or client for HTTP), so one attacker's many rows cannot narrow them; CIs on a single cluster
collapse to a point. "Seen" = the model's training data contained (other rows of) that testbed. RED / AMBER rows
(nfv3_*, http_v1, zanbil) were scored for this comparison only, never trained on; those numbers are internal.
0.1 Tier-1 prior (nf_entity_mono, the engine's per-flow prior), per-flow, alpha = 1e-3
For GREEN testbeds the clean side is a leave-out retrain of the same model (clean_lodo_mono, trained here without
the testbed; SIMARGL 2021 and 2022 are held out together because they are one network a year apart), so neither
side saw the scored network.
| scored network (rows: benign / attack) | seen by | shipped AUROC [95 % CI] | clean AUROC [95 % CI] | shipped FPR / TPR | clean FPR / TPR [TPR CI] |
|---|---|---|---|---|---|
| SIMARGL2022 (337k / 914k) | neither | 0.968 [0.836, 1.000] | 0.999 [0.999, 1.000] | 0.0023 / 0.52 | 0.0010 / 0.49 [0.01, 1.00] |
| APPRAISE (767k / 483k) | neither | 0.614 [0.376, 0.828] | 0.985 [0.224, 0.989] | 0.0072 / 0.00 | 0.0020 / 0.94 [0.00, 0.94] |
| ACI-IoT-2023, author release (166k / 142k) | neither | 0.622 [0.422, 0.757] | 0.644 [0.273, 0.779] | 0.0011 / 0.0003 | 0.0002 / 0.17 [0.00, 0.42] |
| CSE-CIC-IDS2018 Tue 20-02, AWS (1.09M / 160k) | neither | 0.949 [0.925, 0.968] | 1.000 [1.000, 1.000] | 0.0000 / 0.00 | 0.0037 / 1.00 (FPR above alpha) |
| SIMARGL2021 (654k / 596k) | shipped | 0.911 [0.789, 0.991] | 0.932 [0.857, 1.000] | 0.00005 / 0.00 | 0.0005 / 0.00 |
| NF-CSE-CIC-IDS2018-v3 (1.11M / 141k) | shipped | 0.987 [0.982, 0.993] | 0.994 [0.993, 0.996] | 0.0000 / 0.00 | 0.00004 / 0.00 |
| NF-ToN-IoT-v3 (713k / 537k) | shipped | 0.955 [0.801, 0.998] | 0.879 [0.000, 0.950] | 0.0002 / 0.004 | 0.0000 / 0.00 |
| NF-UNSW-NB15-v3 (540k / 52k) | shipped | 0.958 [0.923, 0.979] | 0.066 [0.015, 0.102] | 0.0005 / 0.00 | 0.0000 / 0.00 |
| NF-BoT-IoT-v3 (403 / 1.25M) | shipped | 0.996 | 0.993 | β (403 benign) | β |
Per family where it matters: SIMARGL2022 DoS 1.00 (shipped) vs 0.01 (clean leave-out), port scans 0.00 vs 1.00; APPRAISE reconnaissance 0.00 vs 0.94. Per-flow TPR at alpha = 1e-3 is near zero for both priors on most networks: the prior is not where the engine detects (it feeds the per-source evidence tier, next table).
Per-source evidence (what the engine alarms on), SIMARGL2021, unseen network, Mondrian-fused tiers, judged per
(source, minute) at alpha = 1e-3: clean (SIMARGL 2021 + 2022 held out, run guard-clean-seq-4): FPR 0.00081
[0.0007, 0.0010], attacking source-minutes caught 8 / 8 [0.68, 1.00] (Wilson), every attacking source caught, median
447 flows to the first alarm; shipped (its own leave-one-out report seq_v1_realworld): FPR 0.0029 [0.0026, 0.0032],
16 / 17 [0.73, 0.99], every attacking source caught. The clean set holds the false-alarm budget better (0.08 % vs
0.29 % for a 0.1 % target) at the same detection; note that the two runs sample different attacking sources.
0.2 Entity detectors (fallback tier 1) and the LUFlow detector, alpha = 1e-3
| scored network | seen by | shipped AUROC / FPR / TPR | clean AUROC / FPR / TPR |
|---|---|---|---|
| APPRAISE | clean | 0.189 / 0.0004 / 0.00 | 0.9996 / 0.0008 / 0.999 |
| ACI-IoT-2023 | clean | 0.342 / 0.0026 / 0.046 | 0.675 / 0.036 / 0.075 |
| SIMARGL2022 | clean | 0.9995 / 0.0003 / 0.88 | 1.000 / 0.0004 / 0.9999 |
| CSE-CIC-IDS2018 AWS | clean | 0.9999 / 0.0012 / 1.00 | 1.000 / 0.0007 / 0.9999 |
| NF-CSE-CIC-IDS2018-v3 | shipped | 0.997 / 0.0006 / 0.98 | 0.965 / 0.0030 / 0.005 |
| NF-ToN-IoT-v3 | shipped | 0.850 / 0.0014 / 0.14 | 0.092 / 0.0000 / 0.00 |
| NF-UNSW-NB15-v3 | shipped | 1.000 / 0.0003 / 1.00 | 0.066 / 0.0027 / 0.018 |
| LUFlow 2020 (lu detector) | both | 0.984 / 0.0074 / 0.41 | 0.9998 / 0.0030 / 0.99 |
| LUFlow 2021 (lu detector) | clean | 0.984 / 0.0002 / 0.39 | 0.999 / 0.00004 / 0.95 |
Each detector is strong where it trained and weak elsewhere; no row here is a "neither saw it" comparison except
through the leave-out reports (clean_entity_v1_grouped.json): with the whole SIMARGL family held out, the clean
ent detector scores SIMARGL2021 at AUROC 0.947, TPR 0.69 at FPR 2e-4, entity-minute AUROC 0.995; the shipped
detector's leave-one-out on SIMARGL2021 (card) was 0.538, TPR 0.27, entity-minute 0.865. The LUFlow detector gain is
mostly the clock fix of section 4.
0.3 Tier-2 GRU (seqnet)
No same-row head-to-head (scoring needs the sequence windows of the training job); leave-out numbers from the
training reports, noctx, site-calibrated at 1e-3: SIMARGL2021 with its sibling held out β clean AUROC 0.991,
entity-minute 0.9999, fused with the trees AUROC 1.000 at FPR 0.0024 / TPR 1.00; shipped (SIMARGL2021 alone held
out) AUROC 0.822, entity-minute 0.939, fused 0.708 at FPR 0.024 / TPR 0.48. Other clean leave-outs: APPRAISE 0.987,
ACI-IoT 0.934, CSE-CIC-IDS2018 AWS 0.899. Weakness: trained on NetFlow-style testbeds only, the GRU is inverted on
LUFlow (AUROC 0.0002 with both LUFlow years held out); with one LUFlow year in training it reaches 0.80 / 0.999.
The exported clean GRU is trained on all seven testbeds, LUFlow included.
0.4 HTTP (http_tier1_v2), site-calibrated alpha = 1e-3, same rows for both models
| suite (benign / attack rows) | seen by | shipped AUROC [CI] | clean AUROC [CI] | shipped FPR / TPR [TPR CI] | clean FPR / TPR [TPR CI] |
|---|---|---|---|---|---|
| WordPress POST, SR-BH subset (1.5k / 16.4k) | neither (clean saw other requests of the same site) | 0.817 [0.791, 0.840] | 0.915 [0.904, 0.922] | 0.000 / 0.002 [0.001, 0.002] | 0.0019 / 0.31 [0.30, 0.32] |
| ModSecurity 2025 days 21-30, reference SR-BH benign (34.6k attacks) | days 1-20 by both | β | β | β / 0.76 [0.66, 0.85] | β / 0.96 [0.95, 0.97] |
| same attacks, reference zanbil.ir benign | days 1-20 by both | β | β | β / 0.91 [0.84, 0.96] | β / 0.19 [0.14, 0.24] |
| attack-simulation lab, DVWA (229 / 30k) | neither | 0.998 [0.997, 0.998] | 0.422 [0.406, 0.437] | β | β |
| shipped test split (58k / 52k) | shipped | 0.997 [0.997, 0.997] | 0.803 [0.799, 0.808] | 0.0010 / 0.87 [0.86, 0.87] | 0.0011 / 0.06 [0.06, 0.06] |
| shipped evasion suite (17k obfuscated SQLi / XSS) | shipped | β | β | β / 0.86 [0.86, 0.87] | β / 0.08 [0.08, 0.09] |
| Persian benign, zanbil (4.5k) | shipped | β | β | FPR 0.000 (own threshold 0.000) | FPR 0.000 (own threshold 0.33) |
| clean test split (37k / 84k) | clean | 0.821 [0.815, 0.825] | 0.992 [0.992, 0.993] | 0.0008 / 0.22 [0.22, 0.23] | 0.0011 / 0.93 [0.93, 0.93] |
| clean evasion suite (own mutations, 1.8k) | clean | β | β | β / 0.50 [0.48, 0.52] | β / 0.76 [0.74, 0.78] |
| Persian benign, own generator (8.9k) | clean | β | β | FPR 0.0002 | FPR 0.000 |
Per family on the shipped test split (TPR at the site threshold, shipped vs clean): XSS 0.89 vs 0.02, SQLi 0.88 vs 0.10, command injection 0.96 vs 0.004, traversal 0.93 vs 0.21. On the clean test split: SQLi 0.21 vs 0.99, XSS 0.16 vs 0.51, traversal 0.75 vs 0.85, LFI 0.71 vs 0.60, command injection 0.20 vs 0.75. ModSecurity per family with the SR-BH reference (shipped vs clean): SQLi 0.23 vs 1.00, protocol 0.66 vs 0.98, bad bots 0.90 vs 0.81; with the zanbil reference the clean model loses every family (bots 0.01, LFI 0.17, SQLi 0.03).
What this says: each HTTP model is good on its own data distribution and poor on the other's, and the clean model
does not generalise to a site it never saw: on zanbil.ir's benign traffic its scores are high enough that the
site threshold rises above most attacks, on the DVWA lab it ranks attacks below benign, and with no site
calibration (its own threshold) it would flag a third of real Persian requests. Two spot checks show why: a benign
GET /shop?item=12 without a User-Agent gets logit +26.7 (SR-BH attack tools rarely send one, SR-BH browsers always
do), and a plain <script>alert(1)</script> in a query is missed (SR-BH has no XSS label at all; the GREEN XSS
lists are small).
Third attempt (http_clean3: + own attack grammar of 16k payloads, 15 % of synthetic benign clients without a
User-Agent; run guard-clean-http-3f, CPU, model clean_http_tier1_v2_c3f.npz; head-to-head
clean_vs_shipped_http_c3.json), TPR at the site threshold, run 2 -> run 3 (shipped): WordPress POST 0.31 -> 0.19
(0.002); ModSecurity, SR-BH reference 0.96 -> 0.97 (0.76); ModSecurity, zanbil reference 0.19 -> 0.38 (0.91); lab
AUROC 0.42 -> 0.87 (0.998); shipped test split 0.06 -> 0.05 (0.87); shipped evasion 0.08 -> 0.08 (0.86); clean test
split 0.93 -> 0.71 (0.22); clean evasion 0.76 -> 0.23 (0.50); Persian zanbil FPR at the model's own threshold
0.33 -> 0.022 (0.000). The no-User-Agent false alarm is gone (logit 26.7 -> 3.1, threshold 8.7), but at its own
threshold run 3 now misses the plain UNION SQLi, the plain XSS and the ../../etc/passwd spot checks. Neither clean
HTTP model is fit to ship; clean_http_tier1_v2.npz (run 2) stays the reference clean HTTP artefact.
0.5 Can the clean set replace the shipped one?
| part | replace? | why |
|---|---|---|
tier-1 prior nf_entity_mono |
yes | equal or better on every GREEN network neither saw (AUROC 0.97 -> 0.999, 0.61 -> 0.985, 0.62 -> 0.64, 0.95 -> 1.00); better calibrated per-source evidence on SIMARGL2021; worse on UNSW-NB15 (0.96 -> 0.07) and ToN-IoT (0.96 -> 0.88), which are the shipped model's own training networks and have no GREEN twin |
tier-2 GRU seqnet |
yes | leave-out results on GREEN networks at or above the shipped model's; same size (381 KB), parity 4.6e-5 |
| entity detectors (fallback) | yes | as the prior; the LUFlow detector is clearly better (clock fix) |
HTTP http_tier1_v2 |
no | shipped test split TPR 0.87 -> 0.06 (run 2) / 0.05 (run 3), lab AUROC 0.998 -> 0.42 / 0.87, ModSecurity with a zanbil reference 0.91 -> 0.19 / 0.38; better only on SR-BH-like traffic |
1. What each clean model is
The v2 engine (dibaguard/engine.py) loads three files: seq_v1/nf_entity_mono.npz (tier-1 prior; falls back to
entity_v1/nf_entity_detector.npz), seq_v1/seqnet.npz (tier-2 GRU) and http_v2/http_tier1_v2.npz. Each has a
clean twin. To run the engine on the clean set, copy the clean_* files under the shipped names into a separate
model folder (for example models_clean/seq_v1/nf_entity_mono.npz) and Guard.load that folder; no runtime change
is needed. Every clean model carries data: "licence-clean (roadmap 16)", the per-source licence and the
attribution block (notice) in its own metadata.
| clean artefact (results repo) | replaces | trained on (licence) | run |
|---|---|---|---|
models/seq_v1/clean_nf_entity_mono.npz |
seq_v1/nf_entity_mono.npz (monotone prior, tier 1) |
SIMARGL2021, SIMARGL2022, APPRAISE, ACI-IoT-2023 (CC BY 4.0); CSE-CIC-IDS2018 Tue 20-02 from AWS (CIC grant); LUFlow 2020 + 2021 (CC BY-SA 4.0) | guard-clean-seq-1 |
models/seq_v1/clean_seqnet.npz + clean_seq_v1_info.json |
seq_v1/seqnet.npz (GRU d=64, tier 2) |
same seven testbeds | guard-clean-seq-3 |
models/entity_v1/clean_nf_entity_detector.npz |
entity_v1/nf_entity_detector.npz |
the five nf testbeds above | guard-clean-entity-1 |
models/entity_v1/clean_lu_entity_detector.npz |
entity_v1/lu_entity_detector.npz |
LUFlow 2020 + 2021 (CC BY-SA 4.0) | guard-clean-entity-1 |
models/http_v2/clean_http_tier1_v2.npz |
http_v2/http_tier1_v2.npz |
SR-BH 2020 (CC0), ModSecurity 2025 days 1-20 (CC BY 4.0), PayloadsAllTheThings + SecLists payloads (MIT), OWASP CRS regression tests (Apache-2.0), own synthetic traffic | guard-clean-http-2 |
Not retrained (legacy, not loaded by the v2 engine, still RED as sourced): flows_v1/* and http_v1/http_tier1.
See section 6.
2. Data actually used, with licences and the attribution each one requires
2.1 Flows with addresses and timestamps: entity_clean/ (data repo)
Built by guard_jobs/entity_build.py --clean (launched as --job entity_clean, which attaches only GREEN Kaggle
datasets to the kernel), runs guard-clean-entity-build and guard-clean-entity-fix. Every row has
licence; the builder raises before writing if any row is outside the allowlist.
| testbed | source as read (owner's own release) | licence | rows | span | sources | families | cite |
|---|---|---|---|---|---|---|---|
| simargl2021 | Kaggle h2020simargl/simargl2021-network-intrusion-detection-dataset |
CC BY 4.0 | 8,000,000 | 443 h | 46,873 | benign 3.92 M, portscan 2.50 M, DoS (RUDY, slowloris) 1.58 M | Mihailescu et al., Sensors 21(13):4319, 2021 |
| simargl2022 | Kaggle h2020simargl/simargl2022 (days 15, 16, 19 Mar 2022) |
CC BY 4.0 | 8,000,000 | 100 h | 35,512 | benign 3.30 M, portscan 3.88 M, dos 0.82 M | H2020 SIMARGL, SIMARGL2022 dataset (CC BY 4.0) |
| appraise | Kaggle ittibydgoszcz/appraise-h2020-real-labelled-netflow-dataset |
CC BY 4.0 | 7,570,072 | 1,912 h | 1,339 | benign 4.78 M, recon 2.36 M, ddos 0.43 M, bruteforce 949 | ITTI, APPRAISE H2020 real labelled NetFlow dataset (CC BY 4.0) |
| aci2023 | Kaggle emilynack/aci-iot-network-traffic-dataset-2023 (ACI-IoT-2023.csv, the author's release) |
CC BY 4.0 | 1,231,411 | 102 h | 159,381 | benign 0.33 M, portscan 0.44 M, floods 0.29 M, recon 0.15 M, slowloris, dictionary | Nack et al., MILCOM 2024; doi:10.21227/qacj-3x32 |
| cic2018aws | CSE-CIC-IDS2018, AWS open-data bucket (publisher's copy), processed day Thuesday-20-02-2018 |
CIC grant: redistribute / republish in any form, citation required | 7,948,748 | 12 h | 31,291 | benign 7.37 M, DDoS LOIC-HTTP 0.58 M | Sharafaldin, Habibi Lashkari, Ghorbani, ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018 |
| lu2020 | Kaggle mryanm/luflow-network-intrusion-detection-data-set (author's release), 2020 files |
CC BY-SA 4.0 | 6,856,505 | 180 h | 1,249 | benign 5.12 M, malicious 1.74 M | Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517 |
| lu2021 | same, 2021 files | CC BY-SA 4.0 | 6,430,074 | 2,193 h | 691 | benign 5.40 M, malicious 1.03 M | as lu2020 |
lu2022 was built too but is not used: the 2022 files carry a single source address, so there is no per-source
stream. It stays in entity_clean/ and every trainer run passes the testbed list explicitly.
2.2 HTTP: http_clean2/ (data repo; http_clean/ is the first, superseded build)
Built by guard_jobs/http_clean.py (--job http_clean, or http_build --clean), run guard-clean-http-build3
with --prefix http_clean2 --per-file 20000 --per-family 40000 --placements 2. 1,006,457 rows; by licence: CC0
776,942, own 132,738, MIT 61,719, Apache-2.0 35,058.
| source | as read | licence | what it contributes | cite / notice |
|---|---|---|---|---|
| srbh | SR-BH 2020 multi-label, Harvard Dataverse doi:10.7910/DVN/OGOIXX, file data_capec_multilabel.csv (licence read from the Dataverse API: CC0-1.0) |
CC0 1.0 | 878,968 real requests to a WordPress honeypot (357,536 attacks, CAPEC labels mapped to families), benign carriers, the wild suite (last 20 % of time) |
Sureda Riera et al., Computers & Security 2022, doi:10.1016/j.cose.2022.102788 |
| wp_post | SR-BH 2020 WordPress POST subset, Kaggle ahmeterenceylan/http-request-dataset |
CC0 upstream (uploader label MIT) | held-out suite, never trained on; SR-BH's own 28,381 POST requests are left out of srbh for that reason |
as srbh |
| modsec | ModSecurity thirty-day archive, Zenodo 17178461 (http_v2/modsec.parquet, already built) |
CC BY 4.0 | days 1-20 join training (--modsec-train 1, as the shipped http_v2_final2), days 21-30 are the modsec suite |
Lucz, G. (2025), doi:10.5281/zenodo.17178461 |
| patt | PayloadsAllTheThings */Intruder(s)/* lists, commit 08ea40e7 |
MIT | attack payloads placed into benign carriers | MIT notice |
| seclists | SecLists Fuzzing/* payload lists (family by file keyword), commit 0008f434 |
MIT | attack payloads placed into benign carriers | MIT notice |
| crs | OWASP CRS tests/regression/tests for rule files 913, 921, 930-944, commit ca679721 |
Apache-2.0 | 3,798 attack requests (positive tests) and 802 benign hard negatives (negative tests) | Apache-2.0 + NOTICE |
| synth | own generator in guard_jobs/http_clean.py |
own | 80,000 benign shop / blog / API requests (English and Persian parameters, JSON bodies) as rows and carriers; benign values placed exactly like the attack payloads; the evasion variants (own mutation engine) | - |
| lab | Kaggle moradrawashdeh/attack-simulation-lab (uploader's own lab) |
Apache-2.0 | evaluation suite only (labels are per host, not per request) | Apache-2.0 notice |
Two cleaning rules were added to the payload lists (both data quality, not licence): a list line is kept only if it
carries the family's active syntax (http_clean.ACTIVE; the Intruder / Fuzzing folders also hold plain word lists
such as event-handler names, bare file paths and LDAP attribute names, which are not attacks by themselves), and
files named after the PortSwigger cheat sheet are dropped (the audit treats PortSwigger-derived content as
unlicensed, row 17).
2.3 Not used, and why
| candidate | reason |
|---|---|
| NF-UQ-NIDS-v2, NF-*-v3 (athena21, thanhphuc1810 "MIT", seyhed, ...) | RED (UQ academic-only terms; uploader labels grant nothing) |
| dhoogla CIC-IDS2017 / CSE-CIC-IDS2018 / CIC-DDoS2019, dhoogla LUFlow | RED as sourced (CC BY-NC-SA mirrors) |
| CIC-IDS2017 and CIC-DDoS2019 originals | Not available from the publisher without a manual step: every cicresearch.ca/CICDataset/... path probed from a Kaggle kernel redirects to the UNB dataset index (download behind UNB's registration form); no Kaggle copy is published by UNB/CIC. Not used. |
| CSE-CIC-IDS2018 other days | the AWS processed CSVs of the other nine days have no Src IP / Dst IP (unusable for per-source streams; usable for a single-flow cic view, see section 6) |
| ACI-IoT-2023 / IoMT-2024 CIC-BCCC-NRC copies (phungvannamanh, amineipad) | AMBER (uploader MIT label, CIC/BCCC terms unconfirmed); the author's own ACI release is used instead |
| MU-IoT | GREEN, but the release has normalised features only (no addresses, no timestamps) |
| zanbil.ir | AMBER (raw log withheld by the owner, real client IPs) |
| CSIC 2010, mreowie, obf_sqli, obf_xss, syed_xss, syed_sqli, sambit | RED / AMBER |
own red-team generators (scripts/redteam_*.py, dibaguard.shield.playbook) |
GREEN and they do produce flow tables, but they are the live lab's own test traffic; training on them would grade the models on their training data. Kept as the independent lab check. |
3. The licence gate
guard_jobs/licences.py:CLEAN= {cc-by-4.0, cc-by-sa-4.0, cc0-1.0, mit, apache-2.0, cic-unb-cite, own};GREEN= source id -> (licence, the origin the audit cleared, attribution);OTHER= the audit's tags for every RED / AMBER source.- Every corpus row has a
licencecolumn (entity_buildnow writes it for every testbed, clean or not;http_build's tags were corrected to the audit's values). entity_build --clean: refuses a testbed that is not GREEN or whose Kaggle ref differs from the cleared origin;require_cleanon every testbed's rows before the parquet is written.http_clean:require_sourceon every row's source,require_cleanon the whole frame before writing.entity_train --clean,seq_train --clean,http_train2 --clean: read onlyentity_clean//http_clean*, re-check every loaded testbed / split (source GREEN, every row's licence inCLEAN), raise otherwise; outputs areclean_*files from separate scratch folders, so a clean run cannot upload a shipped file name.tests/test_clean.pybuilds every clean loader on synthetic files, checks the outputs, and checks that a RED testbed (nfv3_cic2018,aciiot2023) and a single tampered row (uq-academic-only) are refused.
4. Loader defects found on the way (fixed in --clean mode only; the shipped corpora keep their bytes)
- LUFlow time unit. The author's release stores
time_startin microseconds, plus a few rows on other scales or at 0. The builder divided by 1000, so the shippedentity_v1/lu2020stream ran on a 1000x stretched clock with a near-zero first row (span_h442,511,328 in its manifest). Every horizon of the entity state (10 s / 2 min / 30 min) was therefore effectively 1000x shorter on LUFlow forlu_entity_detector,seqnetandnf_entity_mono. Clean build: per-row scaling and a plausibility window (lu2020 now spans 180 h). - SYN-only on exporters without a client/server flag split (SIMARGL, APPRAISE):
client_flagswas 0, so the SYN-only share was always 0 on SIMARGL2021, a dataset that is one-third SYN scans. Clean build: rebuilt from the cumulativeTCP_FLAGSand whether the server sent any packet. - ACI-IoT-2023 author timestamps mix spellings across days (
2023-01-11= 1 Nov). Clean build: each row is read every plausible way and the reading inside the README's capture week is kept (span 102 h, was 5,669 h). - CICFlowMeter rows with an unparseable time became -9.2e9 s and sorted first; now dropped.
5. Training-protocol results (leave-one-dataset-out, as the training jobs report them)
Reports in D:/build-tmp/guard/clean/. These are each model set's own leave-out numbers, on different testbeds
than the shipped reports, so they are context for section 0, not a head-to-head.
clean_entity_v1_train.json/clean_entity_v1_grouped.json(runsguard-clean-entity-1,-2): nf view,entfeatures, held-out flow AUROC β SIMARGL2021 0.966 (0.947 with SIMARGL2022 also held out), SIMARGL2022 0.996 (0.979 grouped), APPRAISE 0.934, ACI-IoT 0.969, CSE-CIC-IDS2018 AWS 0.997; lu view LUFlow 2020 0.997, 2021 0.995. Site-calibrated realised FPR at 1e-3 departs from the target on SIMARGL2021 (0.12 single, 2e-4 grouped), SIMARGL2022 grouped (0.019) and ACI-IoT (0.021): few sources with long-lived state are not exchangeable between periods, the limit the card already states for UNSW-NB15.clean_seq_v1_train.json/_grouped/_keepatt(runsguard-clean-seq-1,-2,-4): GRU and fused numbers in section 0.3; anomaly tier: monotone prior leave-out AUROC SIMARGL2022 1.000, CSE-CIC-IDS2018 AWS 0.9998, APPRAISE 0.24 on the source-capped sample ofseq-1(591 attack flows in its test split) and 0.992 when every attacking source is kept (seq-4), ACI-IoT 0.71; the label-free site-only tier (quant) 0.99-1.00 on SIMARGL / CSE-CIC and 0.81 on ACI-IoT. Inseq-1the SIMARGL2021 test split lost its only attacker to the random source sampling (the shipped run happened to keep one), which is whyseq-4re-ran the anomaly protocol with--keep-attack-sources 1. LUFlow anomaly folds are one-sided after source capping (as in the shipped report).clean_seq_v1_export.json(guard-clean-seq-3): exported clean GRU, d = 64, 95,293 parameters, 381 KB, numpy / torch parity 4.6e-5, 130k flows/s on one core.clean_http_v2_train.json(guard-clean-http-2, corpushttp_clean2): leave-one-source-out SR-BH AUROC 0.873 (no site context; 0.42 with context) / site TPR 0.24 at FPR 7e-4, novelty tier 0.984, fused 0.974; WordPress POST 0.947 / 0.17 (shipped: 0.693 / 0.17); CRS 0.57; lab 0.14 (novelty 0.996). Final model: test AUROC 0.992, evasion TPR 0.77, ModSecurity last ten days caught 0.958 with the SR-BH reference, 3,387 clients 0.895; 7,849 KB; 1,888 req/s on one core in Python.clean_http_v2_train_run1_corpus_v1.jsonis the first attempt (corpushttp_clean, before the payload-list cleaning): XSS 0.18, LFI 0.15, evasion 0.29, kept for the record.
6. Gaps: what still depends on data that is not cleared
- HTTP model. The clean HTTP model is not a replacement (section 0.4). What it lacks is benign and attack variety from applications other than one WordPress honeypot: the shipped model's strength on CSIC / zanbil / payload-set traffic comes from RED / AMBER data (CSIC 2010, zanbil.ir, mreowie, obfuscated SQLi / XSS sets). Routes, in order of value: (a) the author's written OK for zanbil.ir (CC0, raw log withheld; DATA_LICENSES Β§4.3), which restores a real non-WordPress benign site; (b) CSIC 2010 permission; (c) more GREEN benign applications (requests generated from the CC0 APIs.guru OpenAPI directory, CRS negative tests at scale); (d) more own attack grammar and WAF-A-MoLE (MIT) mutation of GREEN payloads. Until then the commercial build has no licence-clean HTTP model; the site-only novelty tier (no training data) and the playbook signatures remain clean.
- IoT / UNSW-NB15-style networks. The clean prior is weak on NF-UNSW-NB15 (AUROC 0.07) and below the shipped model on NF-ToN-IoT. There is no GREEN twin of UNSW-NB15, BoT-IoT or ToN-IoT (all UNSW Canberra, academic-only); ToN-IoT is "allowable after asking the author" β the one permission worth asking for the network side. MU-IoT (CC BY 4.0) has no addresses; CIC-IoT2023 from UNB would need its PCAPs re-flowed.
- CIC-IDS2017 and CIC-DDoS2019 could not be fetched from the publisher without the registration form; only the CSE-CIC-IDS2018 day with addresses (Tuesday 20-02) is in. Downloading the UNB originals by hand (GREEN with citation) and re-flowing the 2018 PCAPs from AWS would add a CIC testbed with every attack family.
- Legacy artefacts not retrained:
flows_v1/*(single-flow nf / cic / lu detectors) andhttp_v1/http_tier1are still RED as sourced. They are not loaded by the v2 engine. A cleancicview is possible from the ten AWS CSE-CIC-IDS2018 processed CSVs (no addresses needed for single flows), a cleanluview from the author's LUFlow. - Go runtime files: the clean models exist as
.npz; the flat.bin/.bin.jsonexports and parity fixtures for the Go runtime (dibaguard.parity.npz_to_flat) were not produced for them. - Open licence points that touch the clean set (from DATA_LICENSES Β§5, unchanged): LUFlow is CC BY-SA 4.0 β
whether trained weights are Adapted Material is unsettled, so the LUFlow-trained parts (clean
ludetector, and the GRU / prior, which include LUFlow 2020-21) need legal sign-off or a retrain without LUFlow; the uploader identities of the SIMARGL, LUFlow and ACI-IoT Kaggle accounts are inferred, not verified; the CIC FAQ wording should be confirmed by e-mail; PayloadsAllTheThings / SecLists files credit upstream cheat sheets whose own terms were not checked file by file (the PortSwigger-named files were dropped). - Shipped models carry the old loader defects (section 4): the shipped LUFlow-trained parts ran on a 1000x
stretched clock and SIMARGL2021 without the SYN-only signal. A rebuild of
entity_v1with the--cleanfixes applied to every testbed would be needed before comparing future shipped retrains to these numbers.
7. Runs, files and how to check them
| run (account) | job and arguments | output |
|---|---|---|
guard-clean-survey (1) |
survey_clean |
publisher channels, SR-BH, payload repos, lab, ACI author release |
guard-clean-entity-build (1), guard-clean-entity-fix (1) |
entity_clean (entity_build --clean); fix run --testbeds aci2023,lu2020,lu2021 |
entity_clean/*.parquet + manifest |
guard-clean-http-build (2, stray first version), -build2 (2), -build3 (2), -build4 (2) |
http_clean; build3 --prefix http_clean2 --per-file 20000 --per-family 40000 --placements 2; build4 adds --own-attacks 1 --no-ua-share 0.15 --prefix http_clean3 |
http_clean/, http_clean2/, http_clean3/ |
guard-clean-entity-1 (1, GPU) |
entity_train --clean --views nf,lu --sets base,ent,both --testbeds <7> --iters 1000 --max-rows 1500000 |
models/entity_v1/clean_{nf,lu}_entity_detector.npz, reports/clean_entity_v1_train.json |
guard-clean-entity-2 (1, GPU) |
same, --lodo "simargl2021+simargl2022,..." --no-export |
reports/clean_entity_v1_grouped.json |
guard-clean-seq-1 (1, GPU) |
seq_train --clean --d 128 --ctx 0 --fset ent --protocols temporal,lodo,anomaly --iters 300 --rolling 1800 |
models/seq_v1/clean_nf_entity_mono.npz, reports/clean_seq_v1_train.json |
guard-clean-seq-2 (1, GPU) |
grouped leave-out, --export-mono 0 |
reports/clean_seq_v1_grouped.json |
guard-clean-seq-3 (2, GPU) |
--d 64 --protocols export |
models/seq_v1/clean_seqnet.npz, clean_seq_v1_info.json |
guard-clean-seq-4 (1, GPU) |
anomaly protocol with --keep-attack-sources 1 |
reports/clean_seq_v1_keepatt.json |
guard-clean-http-1 (2, GPU) |
http_train2 --clean on http_clean; first version killed (out of memory at the SR-BH leave-out, fixed by --loso-cap) |
reports/clean_http_v2_train.json (renamed locally _run1_corpus_v1) |
guard-clean-http-2 (2, GPU) |
http_train2 --clean --corpus http_clean2 --protocols loso,final --ctx 0,1 --modsec-train 1 |
models/http_v2/clean_http_tier1_v2.npz, reports/clean_http_v2_train.json |
guard-clean-http-3 (1, CPU), guard-clean-http-3f (2, CPU) |
same on http_clean3 (--loso srbh,wp_post,crs); -3f final only |
models/http_v2/clean_http_tier1_v2_c3.npz / _c3f.npz, reports/clean_http_v2_train3*.json |
guard-clean-eval-flows (1, CPU), guard-clean-eval-http (2, CPU), guard-clean-eval-http3 (2, CPU) |
clean_eval --parts flows / --parts http / --parts http --clean-http models/http_v2/clean_http_tier1_v2_c3f.npz |
reports/clean_vs_shipped_{flows,http,http_c3}.json |
Check any run: python kag.py status --run-name <run> --result (from D:/diba-slm/guard); full log after the
kernel ends: python kag.py log --run-name <run>. Reports and models are in DibaAi/dibaguard-results-private,
corpora in DibaAi/dibaguard-data-private (both private).
Code added or changed (training side only, guard_jobs/ and tests/): licences.py (new), entity_clean.py
(new), http_clean.py (new), clean_eval.py (new), survey_clean.py (new); --clean and related options in
entity_build.py, entity_train.py, seq_train.py, http_build.py, http_train2.py; tests/test_clean.py.
Default (non-clean) behaviour of the existing jobs is unchanged except that entity_build now writes a licence
column and http_build's licence tags carry the audit's values.
8. Attribution block for a commercial release of the clean network models
CSE-CIC-IDS2018 (Sharafaldin, Habibi Lashkari, Ghorbani, ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018);
RoEduNet-SIMARGL2021 (Mihailescu et al., Sensors 21(13):4319, 2021, CC BY 4.0); SIMARGL2022 (H2020 SIMARGL, CC BY
4.0); APPRAISE H2020 real labelled NetFlow (ITTI, CC BY 4.0); ACI-IoT-2023 (Nack et al., MILCOM 2024,
doi:10.21227/qacj-3x32, CC BY 4.0); LUFlow (Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517, CC BY-SA
4.0). For an HTTP model built from the clean corpus, add SR-BH 2020 (CC0; cite doi:10.1016/j.cose.2022.102788),
the ModSecurity archive (doi:10.5281/zenodo.17178461, CC BY 4.0), and the MIT / Apache-2.0 notices of
PayloadsAllTheThings, SecLists and the OWASP CRS. The same block is stored in every clean model's metadata
(notice).