|
Download docs/CLEAN_MODELS.md from Dibachain/DibaGuard: direct link, hf CLI and curl.
- Browser
- Download file 34 kB
-
https://huggingface.co/Dibachain/DibaGuard/resolve/main/docs/CLEAN_MODELS.md
- Command line
-
hf download hf://Dibachain/DibaGuard/docs/CLEAN_MODELS.md
-
curl -L -o CLEAN_MODELS.md https://huggingface.co/Dibachain/DibaGuard/resolve/main/docs/CLEAN_MODELS.md
34 kB
| # DibaGuard licence-clean models (roadmap 16) | |
| > **Open-release copy** ([DibaAi/DibaGuard-Open](https://huggingface.co/DibaAi/DibaGuard-Open)). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (`http_v2`) were measured with that model, which is **not** part of the open release; the open release ships the licence-clean network models described in `CLEAN_MODELS.md`. Licence statements inside this document are superseded by `DATA_LICENSES.md`, `CLEAN_MODELS.md` and the repository's `LICENSE`, `LICENSE-WEIGHTS` and `NOTICE` files. Sections of the original that were written in Persian were removed from this copy; nothing else was edited. | |
| Status as of **2026-10-31** (final). Goal: a model set that may ship in a commercial, closed product, trained **only** on | |
| sources that `docs/DATA_LICENSES.md` (the licence audit of the same day) marks GREEN. Nothing shipped was touched: | |
| the clean models sit next to the shipped ones under `clean_*` names in `DibaAi/dibaguard-results-private`, the clean | |
| corpora under new folders in `DibaAi/dibaguard-data-private`. Fetched reports: `D:/build-tmp/guard/clean/`. | |
| ## 0. Verdict and head-to-head metrics | |
| **Network side (tier-1 prior, tier-2 GRU, entity detectors): the clean set can replace the shipped one**, with one | |
| stated weakness (IoT / UNSW-NB15-style networks, below) and one legal condition (LUFlow is CC BY-SA 4.0; the | |
| LUFlow-trained parts need sign-off or a retrain without LUFlow, section 6). **HTTP side: it cannot yet.** The clean HTTP model is | |
| strong on traffic like its own (SR-BH, ModSecurity days 21-30 with an SR-BH reference) but collapses on an | |
| application it never saw (the shipped test suite, the attack-simulation lab, a zanbil.ir reference): keep the | |
| shipped HTTP model out of a commercial build and do not ship the clean one in its place until the gaps in section 6 | |
| are closed. A third attempt (own attack grammar, benign clients without a User-Agent) traded one failure for another (section 0.4) and is not a replacement either. | |
| How these numbers were made (`guard_jobs/clean_eval.py`, Kaggle CPU runs `guard-clean-eval-flows`, | |
| `guard-clean-eval-http`; reports `clean_vs_shipped_flows.json`, `clean_vs_shipped_http.json`): both model sets score | |
| **the same rows**. Flows: the site reference is the testbed's `cal` split (benign rows), the scored rows its later | |
| `test` split, threshold = conformal quantile of the reference at alpha = 1e-3 for both models. HTTP: profile and | |
| threshold from half of the suite's benign rows (the same 512 profile rows for both models), FPR on the other half; | |
| attack-only suites use a named benign reference. 95 % CIs are cluster bootstraps (clusters = source address for | |
| flows, payload group or client for HTTP), so one attacker's many rows cannot narrow them; CIs on a single cluster | |
| collapse to a point. "Seen" = the model's training data contained (other rows of) that testbed. RED / AMBER rows | |
| (nfv3_*, http_v1, zanbil) were scored for this comparison only, never trained on; those numbers are internal. | |
| ### 0.1 Tier-1 prior (`nf_entity_mono`, the engine's per-flow prior), per-flow, alpha = 1e-3 | |
| For GREEN testbeds the clean side is a leave-out retrain of the same model (`clean_lodo_mono`, trained here without | |
| the testbed; SIMARGL 2021 and 2022 are held out together because they are one network a year apart), so neither | |
| side saw the scored network. | |
| | scored network (rows: benign / attack) | seen by | shipped AUROC [95 % CI] | clean AUROC [95 % CI] | shipped FPR / TPR | clean FPR / TPR [TPR CI] | | |
| |---|---|---|---|---|---| | |
| | SIMARGL2022 (337k / 914k) | neither | 0.968 [0.836, 1.000] | **0.999** [0.999, 1.000] | 0.0023 / 0.52 | 0.0010 / 0.49 [0.01, 1.00] | | |
| | APPRAISE (767k / 483k) | neither | 0.614 [0.376, 0.828] | **0.985** [0.224, 0.989] | 0.0072 / 0.00 | 0.0020 / **0.94** [0.00, 0.94] | | |
| | ACI-IoT-2023, author release (166k / 142k) | neither | 0.622 [0.422, 0.757] | 0.644 [0.273, 0.779] | 0.0011 / 0.0003 | 0.0002 / 0.17 [0.00, 0.42] | | |
| | CSE-CIC-IDS2018 Tue 20-02, AWS (1.09M / 160k) | neither | 0.949 [0.925, 0.968] | **1.000** [1.000, 1.000] | 0.0000 / 0.00 | 0.0037 / 1.00 (FPR above alpha) | | |
| | SIMARGL2021 (654k / 596k) | shipped | 0.911 [0.789, 0.991] | 0.932 [0.857, 1.000] | 0.00005 / 0.00 | 0.0005 / 0.00 | | |
| | NF-CSE-CIC-IDS2018-v3 (1.11M / 141k) | shipped | 0.987 [0.982, 0.993] | **0.994** [0.993, 0.996] | 0.0000 / 0.00 | 0.00004 / 0.00 | | |
| | NF-ToN-IoT-v3 (713k / 537k) | shipped | **0.955** [0.801, 0.998] | 0.879 [0.000, 0.950] | 0.0002 / 0.004 | 0.0000 / 0.00 | | |
| | NF-UNSW-NB15-v3 (540k / 52k) | shipped | **0.958** [0.923, 0.979] | 0.066 [0.015, 0.102] | 0.0005 / 0.00 | 0.0000 / 0.00 | | |
| | NF-BoT-IoT-v3 (403 / 1.25M) | shipped | 0.996 | 0.993 | β (403 benign) | β | | |
| Per family where it matters: SIMARGL2022 DoS 1.00 (shipped) vs 0.01 (clean leave-out), port scans 0.00 vs 1.00; | |
| APPRAISE reconnaissance 0.00 vs 0.94. Per-flow TPR at alpha = 1e-3 is near zero for both priors on most networks: | |
| the prior is not where the engine detects (it feeds the per-source evidence tier, next table). | |
| **Per-source evidence (what the engine alarms on), SIMARGL2021, unseen network, Mondrian-fused tiers, judged per | |
| (source, minute) at alpha = 1e-3:** clean (SIMARGL 2021 + 2022 held out, run `guard-clean-seq-4`): FPR 0.00081 | |
| [0.0007, 0.0010], attacking source-minutes caught 8 / 8 [0.68, 1.00] (Wilson), every attacking source caught, median | |
| 447 flows to the first alarm; shipped (its own leave-one-out report `seq_v1_realworld`): FPR 0.0029 [0.0026, 0.0032], | |
| 16 / 17 [0.73, 0.99], every attacking source caught. The clean set holds the false-alarm budget better (0.08 % vs | |
| 0.29 % for a 0.1 % target) at the same detection; note that the two runs sample different attacking sources. | |
| ### 0.2 Entity detectors (fallback tier 1) and the LUFlow detector, alpha = 1e-3 | |
| | scored network | seen by | shipped AUROC / FPR / TPR | clean AUROC / FPR / TPR | | |
| |---|---|---|---| | |
| | APPRAISE | clean | 0.189 / 0.0004 / 0.00 | 0.9996 / 0.0008 / 0.999 | | |
| | ACI-IoT-2023 | clean | 0.342 / 0.0026 / 0.046 | 0.675 / 0.036 / 0.075 | | |
| | SIMARGL2022 | clean | 0.9995 / 0.0003 / 0.88 | 1.000 / 0.0004 / 0.9999 | | |
| | CSE-CIC-IDS2018 AWS | clean | 0.9999 / 0.0012 / 1.00 | 1.000 / 0.0007 / 0.9999 | | |
| | NF-CSE-CIC-IDS2018-v3 | shipped | 0.997 / 0.0006 / 0.98 | 0.965 / 0.0030 / 0.005 | | |
| | NF-ToN-IoT-v3 | shipped | 0.850 / 0.0014 / 0.14 | 0.092 / 0.0000 / 0.00 | | |
| | NF-UNSW-NB15-v3 | shipped | 1.000 / 0.0003 / 1.00 | 0.066 / 0.0027 / 0.018 | | |
| | LUFlow 2020 (lu detector) | both | 0.984 / 0.0074 / 0.41 | **0.9998 / 0.0030 / 0.99** | | |
| | LUFlow 2021 (lu detector) | clean | 0.984 / 0.0002 / 0.39 | 0.999 / 0.00004 / 0.95 | | |
| Each detector is strong where it trained and weak elsewhere; no row here is a "neither saw it" comparison except | |
| through the leave-out reports (`clean_entity_v1_grouped.json`): with the whole SIMARGL family held out, the clean | |
| `ent` detector scores SIMARGL2021 at AUROC 0.947, TPR 0.69 at FPR 2e-4, entity-minute AUROC 0.995; the shipped | |
| detector's leave-one-out on SIMARGL2021 (card) was 0.538, TPR 0.27, entity-minute 0.865. The LUFlow detector gain is | |
| mostly the clock fix of section 4. | |
| ### 0.3 Tier-2 GRU (`seqnet`) | |
| No same-row head-to-head (scoring needs the sequence windows of the training job); leave-out numbers from the | |
| training reports, `noctx`, site-calibrated at 1e-3: SIMARGL2021 with its sibling held out β clean AUROC 0.991, | |
| entity-minute 0.9999, fused with the trees AUROC 1.000 at FPR 0.0024 / TPR 1.00; shipped (SIMARGL2021 alone held | |
| out) AUROC 0.822, entity-minute 0.939, fused 0.708 at FPR 0.024 / TPR 0.48. Other clean leave-outs: APPRAISE 0.987, | |
| ACI-IoT 0.934, CSE-CIC-IDS2018 AWS 0.899. Weakness: trained on NetFlow-style testbeds only, the GRU is inverted on | |
| LUFlow (AUROC 0.0002 with both LUFlow years held out); with one LUFlow year in training it reaches 0.80 / 0.999. | |
| The exported clean GRU is trained on all seven testbeds, LUFlow included. | |
| ### 0.4 HTTP (`http_tier1_v2`), site-calibrated alpha = 1e-3, same rows for both models | |
| | suite (benign / attack rows) | seen by | shipped AUROC [CI] | clean AUROC [CI] | shipped FPR / TPR [TPR CI] | clean FPR / TPR [TPR CI] | | |
| |---|---|---|---|---|---| | |
| | WordPress POST, SR-BH subset (1.5k / 16.4k) | neither (clean saw other requests of the same site) | 0.817 [0.791, 0.840] | **0.915** [0.904, 0.922] | 0.000 / 0.002 [0.001, 0.002] | 0.0019 / **0.31** [0.30, 0.32] | | |
| | ModSecurity 2025 days 21-30, reference SR-BH benign (34.6k attacks) | days 1-20 by both | β | β | β / 0.76 [0.66, 0.85] | β / **0.96** [0.95, 0.97] | | |
| | same attacks, reference zanbil.ir benign | days 1-20 by both | β | β | β / **0.91** [0.84, 0.96] | β / 0.19 [0.14, 0.24] | | |
| | attack-simulation lab, DVWA (229 / 30k) | neither | **0.998** [0.997, 0.998] | 0.422 [0.406, 0.437] | β | β | | |
| | shipped test split (58k / 52k) | shipped | **0.997** [0.997, 0.997] | 0.803 [0.799, 0.808] | 0.0010 / **0.87** [0.86, 0.87] | 0.0011 / 0.06 [0.06, 0.06] | | |
| | shipped evasion suite (17k obfuscated SQLi / XSS) | shipped | β | β | β / **0.86** [0.86, 0.87] | β / 0.08 [0.08, 0.09] | | |
| | Persian benign, zanbil (4.5k) | shipped | β | β | FPR 0.000 (own threshold 0.000) | FPR 0.000 (own threshold **0.33**) | | |
| | clean test split (37k / 84k) | clean | 0.821 [0.815, 0.825] | **0.992** [0.992, 0.993] | 0.0008 / 0.22 [0.22, 0.23] | 0.0011 / **0.93** [0.93, 0.93] | | |
| | clean evasion suite (own mutations, 1.8k) | clean | β | β | β / 0.50 [0.48, 0.52] | β / **0.76** [0.74, 0.78] | | |
| | Persian benign, own generator (8.9k) | clean | β | β | FPR 0.0002 | FPR 0.000 | | |
| Per family on the shipped test split (TPR at the site threshold, shipped vs clean): XSS 0.89 vs 0.02, SQLi 0.88 vs | |
| 0.10, command injection 0.96 vs 0.004, traversal 0.93 vs 0.21. On the clean test split: SQLi 0.21 vs 0.99, XSS 0.16 | |
| vs 0.51, traversal 0.75 vs 0.85, LFI 0.71 vs 0.60, command injection 0.20 vs 0.75. ModSecurity per family with the | |
| SR-BH reference (shipped vs clean): SQLi 0.23 vs 1.00, protocol 0.66 vs 0.98, bad bots 0.90 vs 0.81; with the zanbil | |
| reference the clean model loses every family (bots 0.01, LFI 0.17, SQLi 0.03). | |
| What this says: each HTTP model is good on its own data distribution and poor on the other's, and the clean model | |
| **does not generalise to a site it never saw**: on zanbil.ir's benign traffic its scores are high enough that the | |
| site threshold rises above most attacks, on the DVWA lab it ranks attacks below benign, and with no site | |
| calibration (its own threshold) it would flag a third of real Persian requests. Two spot checks show why: a benign | |
| `GET /shop?item=12` without a User-Agent gets logit +26.7 (SR-BH attack tools rarely send one, SR-BH browsers always | |
| do), and a plain `<script>alert(1)</script>` in a query is missed (SR-BH has no XSS label at all; the GREEN XSS | |
| lists are small). | |
| **Third attempt** (`http_clean3`: + own attack grammar of 16k payloads, 15 % of synthetic benign clients without a | |
| User-Agent; run `guard-clean-http-3f`, CPU, model `clean_http_tier1_v2_c3f.npz`; head-to-head | |
| `clean_vs_shipped_http_c3.json`), TPR at the site threshold, run 2 -> run 3 (shipped): WordPress POST 0.31 -> 0.19 | |
| (0.002); ModSecurity, SR-BH reference 0.96 -> 0.97 (0.76); ModSecurity, zanbil reference 0.19 -> 0.38 (0.91); lab | |
| AUROC 0.42 -> 0.87 (0.998); shipped test split 0.06 -> 0.05 (0.87); shipped evasion 0.08 -> 0.08 (0.86); clean test | |
| split 0.93 -> 0.71 (0.22); clean evasion 0.76 -> 0.23 (0.50); Persian zanbil FPR at the model's own threshold | |
| 0.33 -> 0.022 (0.000). The no-User-Agent false alarm is gone (logit 26.7 -> 3.1, threshold 8.7), but at its own | |
| threshold run 3 now misses the plain UNION SQLi, the plain XSS and the `../../etc/passwd` spot checks. Neither clean | |
| HTTP model is fit to ship; `clean_http_tier1_v2.npz` (run 2) stays the reference clean HTTP artefact. | |
| ### 0.5 Can the clean set replace the shipped one? | |
| | part | replace? | why | | |
| |---|---|---| | |
| | tier-1 prior `nf_entity_mono` | **yes** | equal or better on every GREEN network neither saw (AUROC 0.97 -> 0.999, 0.61 -> 0.985, 0.62 -> 0.64, 0.95 -> 1.00); better calibrated per-source evidence on SIMARGL2021; worse on UNSW-NB15 (0.96 -> 0.07) and ToN-IoT (0.96 -> 0.88), which are the shipped model's own training networks and have no GREEN twin | | |
| | tier-2 GRU `seqnet` | **yes** | leave-out results on GREEN networks at or above the shipped model's; same size (381 KB), parity 4.6e-5 | | |
| | entity detectors (fallback) | yes | as the prior; the LUFlow detector is clearly better (clock fix) | | |
| | HTTP `http_tier1_v2` | **no** | shipped test split TPR 0.87 -> 0.06 (run 2) / 0.05 (run 3), lab AUROC 0.998 -> 0.42 / 0.87, ModSecurity with a zanbil reference 0.91 -> 0.19 / 0.38; better only on SR-BH-like traffic | | |
| ## 1. What each clean model is | |
| The v2 engine (`dibaguard/engine.py`) loads three files: `seq_v1/nf_entity_mono.npz` (tier-1 prior; falls back to | |
| `entity_v1/nf_entity_detector.npz`), `seq_v1/seqnet.npz` (tier-2 GRU) and `http_v2/http_tier1_v2.npz`. Each has a | |
| clean twin. To run the engine on the clean set, copy the `clean_*` files under the shipped names into a separate | |
| model folder (for example `models_clean/seq_v1/nf_entity_mono.npz`) and `Guard.load` that folder; no runtime change | |
| is needed. Every clean model carries `data: "licence-clean (roadmap 16)"`, the per-source licence and the | |
| attribution block (`notice`) in its own metadata. | |
| | clean artefact (results repo) | replaces | trained on (licence) | run | | |
| |---|---|---|---| | |
| | `models/seq_v1/clean_nf_entity_mono.npz` | `seq_v1/nf_entity_mono.npz` (monotone prior, tier 1) | SIMARGL2021, SIMARGL2022, APPRAISE, ACI-IoT-2023 (CC BY 4.0); CSE-CIC-IDS2018 Tue 20-02 from AWS (CIC grant); LUFlow 2020 + 2021 (CC BY-SA 4.0) | `guard-clean-seq-1` | | |
| | `models/seq_v1/clean_seqnet.npz` + `clean_seq_v1_info.json` | `seq_v1/seqnet.npz` (GRU d=64, tier 2) | same seven testbeds | `guard-clean-seq-3` | | |
| | `models/entity_v1/clean_nf_entity_detector.npz` | `entity_v1/nf_entity_detector.npz` | the five nf testbeds above | `guard-clean-entity-1` | | |
| | `models/entity_v1/clean_lu_entity_detector.npz` | `entity_v1/lu_entity_detector.npz` | LUFlow 2020 + 2021 (CC BY-SA 4.0) | `guard-clean-entity-1` | | |
| | `models/http_v2/clean_http_tier1_v2.npz` | `http_v2/http_tier1_v2.npz` | SR-BH 2020 (CC0), ModSecurity 2025 days 1-20 (CC BY 4.0), PayloadsAllTheThings + SecLists payloads (MIT), OWASP CRS regression tests (Apache-2.0), own synthetic traffic | `guard-clean-http-2` | | |
| Not retrained (legacy, not loaded by the v2 engine, still RED as sourced): `flows_v1/*` and `http_v1/http_tier1`. | |
| See section 6. | |
| ## 2. Data actually used, with licences and the attribution each one requires | |
| ### 2.1 Flows with addresses and timestamps: `entity_clean/` (data repo) | |
| Built by `guard_jobs/entity_build.py --clean` (launched as `--job entity_clean`, which attaches only GREEN Kaggle | |
| datasets to the kernel), runs `guard-clean-entity-build` and `guard-clean-entity-fix`. Every row has | |
| `licence`; the builder raises before writing if any row is outside the allowlist. | |
| | testbed | source as read (owner's own release) | licence | rows | span | sources | families | cite | | |
| |---|---|---|---|---|---|---|---| | |
| | simargl2021 | Kaggle `h2020simargl/simargl2021-network-intrusion-detection-dataset` | CC BY 4.0 | 8,000,000 | 443 h | 46,873 | benign 3.92 M, portscan 2.50 M, DoS (RUDY, slowloris) 1.58 M | Mihailescu et al., Sensors 21(13):4319, 2021 | | |
| | simargl2022 | Kaggle `h2020simargl/simargl2022` (days 15, 16, 19 Mar 2022) | CC BY 4.0 | 8,000,000 | 100 h | 35,512 | benign 3.30 M, portscan 3.88 M, dos 0.82 M | H2020 SIMARGL, SIMARGL2022 dataset (CC BY 4.0) | | |
| | appraise | Kaggle `ittibydgoszcz/appraise-h2020-real-labelled-netflow-dataset` | CC BY 4.0 | 7,570,072 | 1,912 h | 1,339 | benign 4.78 M, recon 2.36 M, ddos 0.43 M, bruteforce 949 | ITTI, APPRAISE H2020 real labelled NetFlow dataset (CC BY 4.0) | | |
| | aci2023 | Kaggle `emilynack/aci-iot-network-traffic-dataset-2023` (`ACI-IoT-2023.csv`, the author's release) | CC BY 4.0 | 1,231,411 | 102 h | 159,381 | benign 0.33 M, portscan 0.44 M, floods 0.29 M, recon 0.15 M, slowloris, dictionary | Nack et al., MILCOM 2024; doi:10.21227/qacj-3x32 | | |
| | cic2018aws | CSE-CIC-IDS2018, AWS open-data bucket (publisher's copy), processed day `Thuesday-20-02-2018` | CIC grant: redistribute / republish in any form, citation required | 7,948,748 | 12 h | 31,291 | benign 7.37 M, DDoS LOIC-HTTP 0.58 M | Sharafaldin, Habibi Lashkari, Ghorbani, ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018 | | |
| | lu2020 | Kaggle `mryanm/luflow-network-intrusion-detection-data-set` (author's release), 2020 files | CC BY-SA 4.0 | 6,856,505 | 180 h | 1,249 | benign 5.12 M, malicious 1.74 M | Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517 | | |
| | lu2021 | same, 2021 files | CC BY-SA 4.0 | 6,430,074 | 2,193 h | 691 | benign 5.40 M, malicious 1.03 M | as lu2020 | | |
| `lu2022` was built too but is not used: the 2022 files carry a single source address, so there is no per-source | |
| stream. It stays in `entity_clean/` and every trainer run passes the testbed list explicitly. | |
| ### 2.2 HTTP: `http_clean2/` (data repo; `http_clean/` is the first, superseded build) | |
| Built by `guard_jobs/http_clean.py` (`--job http_clean`, or `http_build --clean`), run `guard-clean-http-build3` | |
| with `--prefix http_clean2 --per-file 20000 --per-family 40000 --placements 2`. 1,006,457 rows; by licence: CC0 | |
| 776,942, own 132,738, MIT 61,719, Apache-2.0 35,058. | |
| | source | as read | licence | what it contributes | cite / notice | | |
| |---|---|---|---|---| | |
| | srbh | SR-BH 2020 multi-label, Harvard Dataverse doi:10.7910/DVN/OGOIXX, file `data_capec_multilabel.csv` (licence read from the Dataverse API: CC0-1.0) | CC0 1.0 | 878,968 real requests to a WordPress honeypot (357,536 attacks, CAPEC labels mapped to families), benign carriers, the `wild` suite (last 20 % of time) | Sureda Riera et al., Computers & Security 2022, doi:10.1016/j.cose.2022.102788 | | |
| | wp_post | SR-BH 2020 WordPress POST subset, Kaggle `ahmeterenceylan/http-request-dataset` | CC0 upstream (uploader label MIT) | held-out suite, never trained on; SR-BH's own 28,381 POST requests are left out of `srbh` for that reason | as srbh | | |
| | modsec | ModSecurity thirty-day archive, Zenodo 17178461 (`http_v2/modsec.parquet`, already built) | CC BY 4.0 | days 1-20 join training (`--modsec-train 1`, as the shipped `http_v2_final2`), days 21-30 are the `modsec` suite | Lucz, G. (2025), doi:10.5281/zenodo.17178461 | | |
| | patt | PayloadsAllTheThings `*/Intruder(s)/*` lists, commit `08ea40e7` | MIT | attack payloads placed into benign carriers | MIT notice | | |
| | seclists | SecLists `Fuzzing/*` payload lists (family by file keyword), commit `0008f434` | MIT | attack payloads placed into benign carriers | MIT notice | | |
| | crs | OWASP CRS `tests/regression/tests` for rule files 913, 921, 930-944, commit `ca679721` | Apache-2.0 | 3,798 attack requests (positive tests) and 802 benign hard negatives (negative tests) | Apache-2.0 + NOTICE | | |
| | synth | own generator in `guard_jobs/http_clean.py` | own | 80,000 benign shop / blog / API requests (English and Persian parameters, JSON bodies) as rows and carriers; benign values placed exactly like the attack payloads; the evasion variants (own mutation engine) | - | | |
| | lab | Kaggle `moradrawashdeh/attack-simulation-lab` (uploader's own lab) | Apache-2.0 | evaluation suite only (labels are per host, not per request) | Apache-2.0 notice | | |
| Two cleaning rules were added to the payload lists (both data quality, not licence): a list line is kept only if it | |
| carries the family's active syntax (`http_clean.ACTIVE`; the Intruder / Fuzzing folders also hold plain word lists | |
| such as event-handler names, bare file paths and LDAP attribute names, which are not attacks by themselves), and | |
| files named after the PortSwigger cheat sheet are dropped (the audit treats PortSwigger-derived content as | |
| unlicensed, row 17). | |
| ### 2.3 Not used, and why | |
| | candidate | reason | | |
| |---|---| | |
| | NF-UQ-NIDS-v2, NF-*-v3 (athena21, thanhphuc1810 "MIT", seyhed, ...) | RED (UQ academic-only terms; uploader labels grant nothing) | | |
| | dhoogla CIC-IDS2017 / CSE-CIC-IDS2018 / CIC-DDoS2019, dhoogla LUFlow | RED as sourced (CC BY-NC-SA mirrors) | | |
| | CIC-IDS2017 and CIC-DDoS2019 originals | **Not available from the publisher without a manual step**: every `cicresearch.ca/CICDataset/...` path probed from a Kaggle kernel redirects to the UNB dataset index (download behind UNB's registration form); no Kaggle copy is published by UNB/CIC. Not used. | | |
| | CSE-CIC-IDS2018 other days | the AWS processed CSVs of the other nine days have no Src IP / Dst IP (unusable for per-source streams; usable for a single-flow `cic` view, see section 6) | | |
| | ACI-IoT-2023 / IoMT-2024 CIC-BCCC-NRC copies (phungvannamanh, amineipad) | AMBER (uploader MIT label, CIC/BCCC terms unconfirmed); the author's own ACI release is used instead | | |
| | MU-IoT | GREEN, but the release has normalised features only (no addresses, no timestamps) | | |
| | zanbil.ir | AMBER (raw log withheld by the owner, real client IPs) | | |
| | CSIC 2010, mreowie, obf_sqli, obf_xss, syed_xss, syed_sqli, sambit | RED / AMBER | | |
| | own red-team generators (`scripts/redteam_*.py`, `dibaguard.shield.playbook`) | GREEN and they do produce flow tables, but they are the live lab's own test traffic; training on them would grade the models on their training data. Kept as the independent lab check. | | |
| ## 3. The licence gate | |
| * `guard_jobs/licences.py`: `CLEAN` = {cc-by-4.0, cc-by-sa-4.0, cc0-1.0, mit, apache-2.0, cic-unb-cite, own}; `GREEN` | |
| = source id -> (licence, the origin the audit cleared, attribution); `OTHER` = the audit's tags for every RED / | |
| AMBER source. | |
| * Every corpus row has a `licence` column (`entity_build` now writes it for every testbed, clean or not; `http_build`'s | |
| tags were corrected to the audit's values). | |
| * `entity_build --clean`: refuses a testbed that is not GREEN or whose Kaggle ref differs from the cleared origin; | |
| `require_clean` on every testbed's rows before the parquet is written. | |
| * `http_clean`: `require_source` on every row's source, `require_clean` on the whole frame before writing. | |
| * `entity_train --clean`, `seq_train --clean`, `http_train2 --clean`: read only `entity_clean/` / `http_clean*`, | |
| re-check every loaded testbed / split (source GREEN, every row's licence in `CLEAN`), raise otherwise; outputs are | |
| `clean_*` files from separate scratch folders, so a clean run cannot upload a shipped file name. | |
| * `tests/test_clean.py` builds every clean loader on synthetic files, checks the outputs, and checks that a RED | |
| testbed (`nfv3_cic2018`, `aciiot2023`) and a single tampered row (`uq-academic-only`) are refused. | |
| ## 4. Loader defects found on the way (fixed in `--clean` mode only; the shipped corpora keep their bytes) | |
| 1. **LUFlow time unit.** The author's release stores `time_start` in microseconds, plus a few rows on other scales | |
| or at 0. The builder divided by 1000, so the shipped `entity_v1/lu2020` stream ran on a 1000x stretched clock | |
| with a near-zero first row (`span_h` 442,511,328 in its manifest). Every horizon of the entity state (10 s / 2 min / | |
| 30 min) was therefore effectively 1000x shorter on LUFlow for `lu_entity_detector`, `seqnet` and | |
| `nf_entity_mono`. Clean build: per-row scaling and a plausibility window (lu2020 now spans 180 h). | |
| 2. **SYN-only on exporters without a client/server flag split** (SIMARGL, APPRAISE): `client_flags` was 0, so the | |
| SYN-only share was always 0 on SIMARGL2021, a dataset that is one-third SYN scans. Clean build: rebuilt from | |
| the cumulative `TCP_FLAGS` and whether the server sent any packet. | |
| 3. **ACI-IoT-2023 author timestamps** mix spellings across days (`2023-01-11` = 1 Nov). Clean build: each row is | |
| read every plausible way and the reading inside the README's capture week is kept (span 102 h, was 5,669 h). | |
| 4. CICFlowMeter rows with an unparseable time became -9.2e9 s and sorted first; now dropped. | |
| ## 5. Training-protocol results (leave-one-dataset-out, as the training jobs report them) | |
| Reports in `D:/build-tmp/guard/clean/`. These are each model set's own leave-out numbers, on different testbeds | |
| than the shipped reports, so they are context for section 0, not a head-to-head. | |
| * `clean_entity_v1_train.json` / `clean_entity_v1_grouped.json` (runs `guard-clean-entity-1`, `-2`): nf view, | |
| `ent` features, held-out flow AUROC β SIMARGL2021 0.966 (0.947 with SIMARGL2022 also held out), SIMARGL2022 0.996 | |
| (0.979 grouped), APPRAISE 0.934, ACI-IoT 0.969, CSE-CIC-IDS2018 AWS 0.997; lu view LUFlow 2020 0.997, 2021 0.995. | |
| Site-calibrated realised FPR at 1e-3 departs from the target on SIMARGL2021 (0.12 single, 2e-4 grouped), SIMARGL2022 | |
| grouped (0.019) and ACI-IoT (0.021): few sources with long-lived state are not exchangeable between periods, the | |
| limit the card already states for UNSW-NB15. | |
| * `clean_seq_v1_train.json` / `_grouped` / `_keepatt` (runs `guard-clean-seq-1`, `-2`, `-4`): GRU and fused numbers | |
| in section 0.3; anomaly tier: monotone prior leave-out AUROC SIMARGL2022 1.000, CSE-CIC-IDS2018 AWS 0.9998, | |
| APPRAISE 0.24 on the source-capped sample of `seq-1` (591 attack flows in its test split) and 0.992 when every | |
| attacking source is kept (`seq-4`), ACI-IoT 0.71; the label-free site-only tier (quant) 0.99-1.00 on SIMARGL / | |
| CSE-CIC and 0.81 on ACI-IoT. In `seq-1` the SIMARGL2021 test split lost its only attacker to the random source | |
| sampling (the shipped run happened to keep one), which is why `seq-4` re-ran the anomaly protocol with | |
| `--keep-attack-sources 1`. LUFlow anomaly folds are one-sided after source capping (as in the shipped report). | |
| * `clean_seq_v1_export.json` (`guard-clean-seq-3`): exported clean GRU, d = 64, 95,293 parameters, 381 KB, numpy / | |
| torch parity 4.6e-5, 130k flows/s on one core. | |
| * `clean_http_v2_train.json` (`guard-clean-http-2`, corpus `http_clean2`): leave-one-source-out SR-BH AUROC 0.873 | |
| (no site context; 0.42 with context) / site TPR 0.24 at FPR 7e-4, novelty tier 0.984, fused 0.974; WordPress POST | |
| 0.947 / 0.17 (shipped: 0.693 / 0.17); CRS 0.57; lab 0.14 (novelty 0.996). Final model: test AUROC 0.992, evasion | |
| TPR 0.77, ModSecurity last ten days caught 0.958 with the SR-BH reference, 3,387 clients 0.895; 7,849 KB; 1,888 req/s | |
| on one core in Python. `clean_http_v2_train_run1_corpus_v1.json` is the first attempt (corpus `http_clean`, before | |
| the payload-list cleaning): XSS 0.18, LFI 0.15, evasion 0.29, kept for the record. | |
| ## 6. Gaps: what still depends on data that is not cleared | |
| 1. **HTTP model.** The clean HTTP model is not a replacement (section 0.4). What it lacks is benign and attack | |
| variety from applications other than one WordPress honeypot: the shipped model's strength on CSIC / zanbil / | |
| payload-set traffic comes from RED / AMBER data (CSIC 2010, zanbil.ir, mreowie, obfuscated SQLi / XSS sets). | |
| Routes, in order of value: (a) the author's written OK for zanbil.ir (CC0, raw log withheld; DATA_LICENSES Β§4.3), | |
| which restores a real non-WordPress benign site; (b) CSIC 2010 permission; (c) more GREEN benign applications | |
| (requests generated from the CC0 APIs.guru OpenAPI directory, CRS negative tests at scale); (d) more own attack | |
| grammar and WAF-A-MoLE (MIT) mutation of GREEN payloads. Until then the commercial build has **no** licence-clean | |
| HTTP model; the site-only novelty tier (no training data) and the playbook signatures remain clean. | |
| 2. **IoT / UNSW-NB15-style networks.** The clean prior is weak on NF-UNSW-NB15 (AUROC 0.07) and below the shipped | |
| model on NF-ToN-IoT. There is no GREEN twin of UNSW-NB15, BoT-IoT or ToN-IoT (all UNSW Canberra, academic-only); | |
| ToN-IoT is "allowable after asking the author" β the one permission worth asking for the network side. MU-IoT | |
| (CC BY 4.0) has no addresses; CIC-IoT2023 from UNB would need its PCAPs re-flowed. | |
| 3. **CIC-IDS2017 and CIC-DDoS2019** could not be fetched from the publisher without the registration form; only the | |
| CSE-CIC-IDS2018 day with addresses (Tuesday 20-02) is in. Downloading the UNB originals by hand (GREEN with | |
| citation) and re-flowing the 2018 PCAPs from AWS would add a CIC testbed with every attack family. | |
| 4. **Legacy artefacts not retrained:** `flows_v1/*` (single-flow nf / cic / lu detectors) and `http_v1/http_tier1` | |
| are still RED as sourced. They are not loaded by the v2 engine. A clean `cic` view is possible from the ten AWS | |
| CSE-CIC-IDS2018 processed CSVs (no addresses needed for single flows), a clean `lu` view from the author's LUFlow. | |
| 5. **Go runtime files:** the clean models exist as `.npz`; the flat `.bin` / `.bin.json` exports and parity fixtures | |
| for the Go runtime (`dibaguard.parity.npz_to_flat`) were not produced for them. | |
| 6. **Open licence points that touch the clean set** (from DATA_LICENSES Β§5, unchanged): LUFlow is CC BY-SA 4.0 β | |
| whether trained weights are Adapted Material is unsettled, so the LUFlow-trained parts (clean `lu` detector, and | |
| the GRU / prior, which include LUFlow 2020-21) need legal sign-off or a retrain without LUFlow; the uploader | |
| identities of the SIMARGL, LUFlow and ACI-IoT Kaggle accounts are inferred, not verified; the CIC FAQ wording | |
| should be confirmed by e-mail; PayloadsAllTheThings / SecLists files credit upstream cheat sheets whose own terms | |
| were not checked file by file (the PortSwigger-named files were dropped). | |
| 7. **Shipped models carry the old loader defects** (section 4): the shipped LUFlow-trained parts ran on a 1000x | |
| stretched clock and SIMARGL2021 without the SYN-only signal. A rebuild of `entity_v1` with the `--clean` fixes | |
| applied to every testbed would be needed before comparing future shipped retrains to these numbers. | |
| ## 7. Runs, files and how to check them | |
| | run (account) | job and arguments | output | | |
| |---|---|---| | |
| | `guard-clean-survey` (1) | `survey_clean` | publisher channels, SR-BH, payload repos, lab, ACI author release | | |
| | `guard-clean-entity-build` (1), `guard-clean-entity-fix` (1) | `entity_clean` (`entity_build --clean`); fix run `--testbeds aci2023,lu2020,lu2021` | `entity_clean/*.parquet` + manifest | | |
| | `guard-clean-http-build` (2, stray first version), `-build2` (2), `-build3` (2), `-build4` (2) | `http_clean`; build3 `--prefix http_clean2 --per-file 20000 --per-family 40000 --placements 2`; build4 adds `--own-attacks 1 --no-ua-share 0.15 --prefix http_clean3` | `http_clean/`, `http_clean2/`, `http_clean3/` | | |
| | `guard-clean-entity-1` (1, GPU) | `entity_train --clean --views nf,lu --sets base,ent,both --testbeds <7> --iters 1000 --max-rows 1500000` | `models/entity_v1/clean_{nf,lu}_entity_detector.npz`, `reports/clean_entity_v1_train.json` | | |
| | `guard-clean-entity-2` (1, GPU) | same, `--lodo "simargl2021+simargl2022,..." --no-export` | `reports/clean_entity_v1_grouped.json` | | |
| | `guard-clean-seq-1` (1, GPU) | `seq_train --clean --d 128 --ctx 0 --fset ent --protocols temporal,lodo,anomaly --iters 300 --rolling 1800` | `models/seq_v1/clean_nf_entity_mono.npz`, `reports/clean_seq_v1_train.json` | | |
| | `guard-clean-seq-2` (1, GPU) | grouped leave-out, `--export-mono 0` | `reports/clean_seq_v1_grouped.json` | | |
| | `guard-clean-seq-3` (2, GPU) | `--d 64 --protocols export` | `models/seq_v1/clean_seqnet.npz`, `clean_seq_v1_info.json` | | |
| | `guard-clean-seq-4` (1, GPU) | anomaly protocol with `--keep-attack-sources 1` | `reports/clean_seq_v1_keepatt.json` | | |
| | `guard-clean-http-1` (2, GPU) | `http_train2 --clean` on `http_clean`; first version killed (out of memory at the SR-BH leave-out, fixed by `--loso-cap`) | `reports/clean_http_v2_train.json` (renamed locally `_run1_corpus_v1`) | | |
| | `guard-clean-http-2` (2, GPU) | `http_train2 --clean --corpus http_clean2 --protocols loso,final --ctx 0,1 --modsec-train 1` | `models/http_v2/clean_http_tier1_v2.npz`, `reports/clean_http_v2_train.json` | | |
| | `guard-clean-http-3` (1, CPU), `guard-clean-http-3f` (2, CPU) | same on `http_clean3` (`--loso srbh,wp_post,crs`); `-3f` final only | `models/http_v2/clean_http_tier1_v2_c3.npz` / `_c3f.npz`, `reports/clean_http_v2_train3*.json` | | |
| | `guard-clean-eval-flows` (1, CPU), `guard-clean-eval-http` (2, CPU), `guard-clean-eval-http3` (2, CPU) | `clean_eval --parts flows` / `--parts http` / `--parts http --clean-http models/http_v2/clean_http_tier1_v2_c3f.npz` | `reports/clean_vs_shipped_{flows,http,http_c3}.json` | | |
| Check any run: `python kag.py status --run-name <run> --result` (from `D:/diba-slm/guard`); full log after the | |
| kernel ends: `python kag.py log --run-name <run>`. Reports and models are in `DibaAi/dibaguard-results-private`, | |
| corpora in `DibaAi/dibaguard-data-private` (both private). | |
| Code added or changed (training side only, `guard_jobs/` and `tests/`): `licences.py` (new), `entity_clean.py` | |
| (new), `http_clean.py` (new), `clean_eval.py` (new), `survey_clean.py` (new); `--clean` and related options in | |
| `entity_build.py`, `entity_train.py`, `seq_train.py`, `http_build.py`, `http_train2.py`; `tests/test_clean.py`. | |
| Default (non-clean) behaviour of the existing jobs is unchanged except that `entity_build` now writes a `licence` | |
| column and `http_build`'s licence tags carry the audit's values. | |
| ## 8. Attribution block for a commercial release of the clean network models | |
| CSE-CIC-IDS2018 (Sharafaldin, Habibi Lashkari, Ghorbani, ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018); | |
| RoEduNet-SIMARGL2021 (Mihailescu et al., Sensors 21(13):4319, 2021, CC BY 4.0); SIMARGL2022 (H2020 SIMARGL, CC BY | |
| 4.0); APPRAISE H2020 real labelled NetFlow (ITTI, CC BY 4.0); ACI-IoT-2023 (Nack et al., MILCOM 2024, | |
| doi:10.21227/qacj-3x32, CC BY 4.0); LUFlow (Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517, CC BY-SA | |
| 4.0). For an HTTP model built from the clean corpus, add SR-BH 2020 (CC0; cite doi:10.1016/j.cose.2022.102788), | |
| the ModSecurity archive (doi:10.5281/zenodo.17178461, CC BY 4.0), and the MIT / Apache-2.0 notices of | |
| PayloadsAllTheThings, SecLists and the OWASP CRS. The same block is stored in every clean model's metadata | |
| (`notice`). | |