# DibaGuard licence-clean models (roadmap 16)
> **Open-release copy** ([DibaAi/DibaGuard-Open](https://huggingface.co/DibaAi/DibaGuard-Open)). This document was written for the private build of DibaGuard. Numbers that involve the HTTP request model (`http_v2`) were measured with that model, which is **not** part of the open release; the open release ships the licence-clean network models described in `CLEAN_MODELS.md`. Licence statements inside this document are superseded by `DATA_LICENSES.md`, `CLEAN_MODELS.md` and the repository's `LICENSE`, `LICENSE-WEIGHTS` and `NOTICE` files. Sections of the original that were written in Persian were removed from this copy; nothing else was edited.
Status as of **2026-10-31** (final). Goal: a model set that may ship in a commercial, closed product, trained **only** on
sources that `docs/DATA_LICENSES.md` (the licence audit of the same day) marks GREEN. Nothing shipped was touched:
the clean models sit next to the shipped ones under `clean_*` names in `DibaAi/dibaguard-results-private`, the clean
corpora under new folders in `DibaAi/dibaguard-data-private`. Fetched reports: `D:/build-tmp/guard/clean/`.
## 0. Verdict and head-to-head metrics
**Network side (tier-1 prior, tier-2 GRU, entity detectors): the clean set can replace the shipped one**, with one
stated weakness (IoT / UNSW-NB15-style networks, below) and one legal condition (LUFlow is CC BY-SA 4.0; the
LUFlow-trained parts need sign-off or a retrain without LUFlow, section 6). **HTTP side: it cannot yet.** The clean HTTP model is
strong on traffic like its own (SR-BH, ModSecurity days 21-30 with an SR-BH reference) but collapses on an
application it never saw (the shipped test suite, the attack-simulation lab, a zanbil.ir reference): keep the
shipped HTTP model out of a commercial build and do not ship the clean one in its place until the gaps in section 6
are closed. A third attempt (own attack grammar, benign clients without a User-Agent) traded one failure for another (section 0.4) and is not a replacement either.
How these numbers were made (`guard_jobs/clean_eval.py`, Kaggle CPU runs `guard-clean-eval-flows`,
`guard-clean-eval-http`; reports `clean_vs_shipped_flows.json`, `clean_vs_shipped_http.json`): both model sets score
**the same rows**. Flows: the site reference is the testbed's `cal` split (benign rows), the scored rows its later
`test` split, threshold = conformal quantile of the reference at alpha = 1e-3 for both models. HTTP: profile and
threshold from half of the suite's benign rows (the same 512 profile rows for both models), FPR on the other half;
attack-only suites use a named benign reference. 95 % CIs are cluster bootstraps (clusters = source address for
flows, payload group or client for HTTP), so one attacker's many rows cannot narrow them; CIs on a single cluster
collapse to a point. "Seen" = the model's training data contained (other rows of) that testbed. RED / AMBER rows
(nfv3_*, http_v1, zanbil) were scored for this comparison only, never trained on; those numbers are internal.
### 0.1 Tier-1 prior (`nf_entity_mono`, the engine's per-flow prior), per-flow, alpha = 1e-3
For GREEN testbeds the clean side is a leave-out retrain of the same model (`clean_lodo_mono`, trained here without
the testbed; SIMARGL 2021 and 2022 are held out together because they are one network a year apart), so neither
side saw the scored network.
| scored network (rows: benign / attack) | seen by | shipped AUROC [95 % CI] | clean AUROC [95 % CI] | shipped FPR / TPR | clean FPR / TPR [TPR CI] |
|---|---|---|---|---|---|
| SIMARGL2022 (337k / 914k) | neither | 0.968 [0.836, 1.000] | **0.999** [0.999, 1.000] | 0.0023 / 0.52 | 0.0010 / 0.49 [0.01, 1.00] |
| APPRAISE (767k / 483k) | neither | 0.614 [0.376, 0.828] | **0.985** [0.224, 0.989] | 0.0072 / 0.00 | 0.0020 / **0.94** [0.00, 0.94] |
| ACI-IoT-2023, author release (166k / 142k) | neither | 0.622 [0.422, 0.757] | 0.644 [0.273, 0.779] | 0.0011 / 0.0003 | 0.0002 / 0.17 [0.00, 0.42] |
| CSE-CIC-IDS2018 Tue 20-02, AWS (1.09M / 160k) | neither | 0.949 [0.925, 0.968] | **1.000** [1.000, 1.000] | 0.0000 / 0.00 | 0.0037 / 1.00 (FPR above alpha) |
| SIMARGL2021 (654k / 596k) | shipped | 0.911 [0.789, 0.991] | 0.932 [0.857, 1.000] | 0.00005 / 0.00 | 0.0005 / 0.00 |
| NF-CSE-CIC-IDS2018-v3 (1.11M / 141k) | shipped | 0.987 [0.982, 0.993] | **0.994** [0.993, 0.996] | 0.0000 / 0.00 | 0.00004 / 0.00 |
| NF-ToN-IoT-v3 (713k / 537k) | shipped | **0.955** [0.801, 0.998] | 0.879 [0.000, 0.950] | 0.0002 / 0.004 | 0.0000 / 0.00 |
| NF-UNSW-NB15-v3 (540k / 52k) | shipped | **0.958** [0.923, 0.979] | 0.066 [0.015, 0.102] | 0.0005 / 0.00 | 0.0000 / 0.00 |
| NF-BoT-IoT-v3 (403 / 1.25M) | shipped | 0.996 | 0.993 | — (403 benign) | — |
Per family where it matters: SIMARGL2022 DoS 1.00 (shipped) vs 0.01 (clean leave-out), port scans 0.00 vs 1.00;
APPRAISE reconnaissance 0.00 vs 0.94. Per-flow TPR at alpha = 1e-3 is near zero for both priors on most networks:
the prior is not where the engine detects (it feeds the per-source evidence tier, next table).
**Per-source evidence (what the engine alarms on), SIMARGL2021, unseen network, Mondrian-fused tiers, judged per
(source, minute) at alpha = 1e-3:** clean (SIMARGL 2021 + 2022 held out, run `guard-clean-seq-4`): FPR 0.00081
[0.0007, 0.0010], attacking source-minutes caught 8 / 8 [0.68, 1.00] (Wilson), every attacking source caught, median
447 flows to the first alarm; shipped (its own leave-one-out report `seq_v1_realworld`): FPR 0.0029 [0.0026, 0.0032],
16 / 17 [0.73, 0.99], every attacking source caught. The clean set holds the false-alarm budget better (0.08 % vs
0.29 % for a 0.1 % target) at the same detection; note that the two runs sample different attacking sources.
### 0.2 Entity detectors (fallback tier 1) and the LUFlow detector, alpha = 1e-3
| scored network | seen by | shipped AUROC / FPR / TPR | clean AUROC / FPR / TPR |
|---|---|---|---|
| APPRAISE | clean | 0.189 / 0.0004 / 0.00 | 0.9996 / 0.0008 / 0.999 |
| ACI-IoT-2023 | clean | 0.342 / 0.0026 / 0.046 | 0.675 / 0.036 / 0.075 |
| SIMARGL2022 | clean | 0.9995 / 0.0003 / 0.88 | 1.000 / 0.0004 / 0.9999 |
| CSE-CIC-IDS2018 AWS | clean | 0.9999 / 0.0012 / 1.00 | 1.000 / 0.0007 / 0.9999 |
| NF-CSE-CIC-IDS2018-v3 | shipped | 0.997 / 0.0006 / 0.98 | 0.965 / 0.0030 / 0.005 |
| NF-ToN-IoT-v3 | shipped | 0.850 / 0.0014 / 0.14 | 0.092 / 0.0000 / 0.00 |
| NF-UNSW-NB15-v3 | shipped | 1.000 / 0.0003 / 1.00 | 0.066 / 0.0027 / 0.018 |
| LUFlow 2020 (lu detector) | both | 0.984 / 0.0074 / 0.41 | **0.9998 / 0.0030 / 0.99** |
| LUFlow 2021 (lu detector) | clean | 0.984 / 0.0002 / 0.39 | 0.999 / 0.00004 / 0.95 |
Each detector is strong where it trained and weak elsewhere; no row here is a "neither saw it" comparison except
through the leave-out reports (`clean_entity_v1_grouped.json`): with the whole SIMARGL family held out, the clean
`ent` detector scores SIMARGL2021 at AUROC 0.947, TPR 0.69 at FPR 2e-4, entity-minute AUROC 0.995; the shipped
detector's leave-one-out on SIMARGL2021 (card) was 0.538, TPR 0.27, entity-minute 0.865. The LUFlow detector gain is
mostly the clock fix of section 4.
### 0.3 Tier-2 GRU (`seqnet`)
No same-row head-to-head (scoring needs the sequence windows of the training job); leave-out numbers from the
training reports, `noctx`, site-calibrated at 1e-3: SIMARGL2021 with its sibling held out — clean AUROC 0.991,
entity-minute 0.9999, fused with the trees AUROC 1.000 at FPR 0.0024 / TPR 1.00; shipped (SIMARGL2021 alone held
out) AUROC 0.822, entity-minute 0.939, fused 0.708 at FPR 0.024 / TPR 0.48. Other clean leave-outs: APPRAISE 0.987,
ACI-IoT 0.934, CSE-CIC-IDS2018 AWS 0.899. Weakness: trained on NetFlow-style testbeds only, the GRU is inverted on
LUFlow (AUROC 0.0002 with both LUFlow years held out); with one LUFlow year in training it reaches 0.80 / 0.999.
The exported clean GRU is trained on all seven testbeds, LUFlow included.
### 0.4 HTTP (`http_tier1_v2`), site-calibrated alpha = 1e-3, same rows for both models
| suite (benign / attack rows) | seen by | shipped AUROC [CI] | clean AUROC [CI] | shipped FPR / TPR [TPR CI] | clean FPR / TPR [TPR CI] |
|---|---|---|---|---|---|
| WordPress POST, SR-BH subset (1.5k / 16.4k) | neither (clean saw other requests of the same site) | 0.817 [0.791, 0.840] | **0.915** [0.904, 0.922] | 0.000 / 0.002 [0.001, 0.002] | 0.0019 / **0.31** [0.30, 0.32] |
| ModSecurity 2025 days 21-30, reference SR-BH benign (34.6k attacks) | days 1-20 by both | — | — | — / 0.76 [0.66, 0.85] | — / **0.96** [0.95, 0.97] |
| same attacks, reference zanbil.ir benign | days 1-20 by both | — | — | — / **0.91** [0.84, 0.96] | — / 0.19 [0.14, 0.24] |
| attack-simulation lab, DVWA (229 / 30k) | neither | **0.998** [0.997, 0.998] | 0.422 [0.406, 0.437] | — | — |
| shipped test split (58k / 52k) | shipped | **0.997** [0.997, 0.997] | 0.803 [0.799, 0.808] | 0.0010 / **0.87** [0.86, 0.87] | 0.0011 / 0.06 [0.06, 0.06] |
| shipped evasion suite (17k obfuscated SQLi / XSS) | shipped | — | — | — / **0.86** [0.86, 0.87] | — / 0.08 [0.08, 0.09] |
| Persian benign, zanbil (4.5k) | shipped | — | — | FPR 0.000 (own threshold 0.000) | FPR 0.000 (own threshold **0.33**) |
| clean test split (37k / 84k) | clean | 0.821 [0.815, 0.825] | **0.992** [0.992, 0.993] | 0.0008 / 0.22 [0.22, 0.23] | 0.0011 / **0.93** [0.93, 0.93] |
| clean evasion suite (own mutations, 1.8k) | clean | — | — | — / 0.50 [0.48, 0.52] | — / **0.76** [0.74, 0.78] |
| Persian benign, own generator (8.9k) | clean | — | — | FPR 0.0002 | FPR 0.000 |
Per family on the shipped test split (TPR at the site threshold, shipped vs clean): XSS 0.89 vs 0.02, SQLi 0.88 vs
0.10, command injection 0.96 vs 0.004, traversal 0.93 vs 0.21. On the clean test split: SQLi 0.21 vs 0.99, XSS 0.16
vs 0.51, traversal 0.75 vs 0.85, LFI 0.71 vs 0.60, command injection 0.20 vs 0.75. ModSecurity per family with the
SR-BH reference (shipped vs clean): SQLi 0.23 vs 1.00, protocol 0.66 vs 0.98, bad bots 0.90 vs 0.81; with the zanbil
reference the clean model loses every family (bots 0.01, LFI 0.17, SQLi 0.03).
What this says: each HTTP model is good on its own data distribution and poor on the other's, and the clean model
**does not generalise to a site it never saw**: on zanbil.ir's benign traffic its scores are high enough that the
site threshold rises above most attacks, on the DVWA lab it ranks attacks below benign, and with no site
calibration (its own threshold) it would flag a third of real Persian requests. Two spot checks show why: a benign
`GET /shop?item=12` without a User-Agent gets logit +26.7 (SR-BH attack tools rarely send one, SR-BH browsers always
do), and a plain `` in a query is missed (SR-BH has no XSS label at all; the GREEN XSS
lists are small).
**Third attempt** (`http_clean3`: + own attack grammar of 16k payloads, 15 % of synthetic benign clients without a
User-Agent; run `guard-clean-http-3f`, CPU, model `clean_http_tier1_v2_c3f.npz`; head-to-head
`clean_vs_shipped_http_c3.json`), TPR at the site threshold, run 2 -> run 3 (shipped): WordPress POST 0.31 -> 0.19
(0.002); ModSecurity, SR-BH reference 0.96 -> 0.97 (0.76); ModSecurity, zanbil reference 0.19 -> 0.38 (0.91); lab
AUROC 0.42 -> 0.87 (0.998); shipped test split 0.06 -> 0.05 (0.87); shipped evasion 0.08 -> 0.08 (0.86); clean test
split 0.93 -> 0.71 (0.22); clean evasion 0.76 -> 0.23 (0.50); Persian zanbil FPR at the model's own threshold
0.33 -> 0.022 (0.000). The no-User-Agent false alarm is gone (logit 26.7 -> 3.1, threshold 8.7), but at its own
threshold run 3 now misses the plain UNION SQLi, the plain XSS and the `../../etc/passwd` spot checks. Neither clean
HTTP model is fit to ship; `clean_http_tier1_v2.npz` (run 2) stays the reference clean HTTP artefact.
### 0.5 Can the clean set replace the shipped one?
| part | replace? | why |
|---|---|---|
| tier-1 prior `nf_entity_mono` | **yes** | equal or better on every GREEN network neither saw (AUROC 0.97 -> 0.999, 0.61 -> 0.985, 0.62 -> 0.64, 0.95 -> 1.00); better calibrated per-source evidence on SIMARGL2021; worse on UNSW-NB15 (0.96 -> 0.07) and ToN-IoT (0.96 -> 0.88), which are the shipped model's own training networks and have no GREEN twin |
| tier-2 GRU `seqnet` | **yes** | leave-out results on GREEN networks at or above the shipped model's; same size (381 KB), parity 4.6e-5 |
| entity detectors (fallback) | yes | as the prior; the LUFlow detector is clearly better (clock fix) |
| HTTP `http_tier1_v2` | **no** | shipped test split TPR 0.87 -> 0.06 (run 2) / 0.05 (run 3), lab AUROC 0.998 -> 0.42 / 0.87, ModSecurity with a zanbil reference 0.91 -> 0.19 / 0.38; better only on SR-BH-like traffic |
## 1. What each clean model is
The v2 engine (`dibaguard/engine.py`) loads three files: `seq_v1/nf_entity_mono.npz` (tier-1 prior; falls back to
`entity_v1/nf_entity_detector.npz`), `seq_v1/seqnet.npz` (tier-2 GRU) and `http_v2/http_tier1_v2.npz`. Each has a
clean twin. To run the engine on the clean set, copy the `clean_*` files under the shipped names into a separate
model folder (for example `models_clean/seq_v1/nf_entity_mono.npz`) and `Guard.load` that folder; no runtime change
is needed. Every clean model carries `data: "licence-clean (roadmap 16)"`, the per-source licence and the
attribution block (`notice`) in its own metadata.
| clean artefact (results repo) | replaces | trained on (licence) | run |
|---|---|---|---|
| `models/seq_v1/clean_nf_entity_mono.npz` | `seq_v1/nf_entity_mono.npz` (monotone prior, tier 1) | SIMARGL2021, SIMARGL2022, APPRAISE, ACI-IoT-2023 (CC BY 4.0); CSE-CIC-IDS2018 Tue 20-02 from AWS (CIC grant); LUFlow 2020 + 2021 (CC BY-SA 4.0) | `guard-clean-seq-1` |
| `models/seq_v1/clean_seqnet.npz` + `clean_seq_v1_info.json` | `seq_v1/seqnet.npz` (GRU d=64, tier 2) | same seven testbeds | `guard-clean-seq-3` |
| `models/entity_v1/clean_nf_entity_detector.npz` | `entity_v1/nf_entity_detector.npz` | the five nf testbeds above | `guard-clean-entity-1` |
| `models/entity_v1/clean_lu_entity_detector.npz` | `entity_v1/lu_entity_detector.npz` | LUFlow 2020 + 2021 (CC BY-SA 4.0) | `guard-clean-entity-1` |
| `models/http_v2/clean_http_tier1_v2.npz` | `http_v2/http_tier1_v2.npz` | SR-BH 2020 (CC0), ModSecurity 2025 days 1-20 (CC BY 4.0), PayloadsAllTheThings + SecLists payloads (MIT), OWASP CRS regression tests (Apache-2.0), own synthetic traffic | `guard-clean-http-2` |
Not retrained (legacy, not loaded by the v2 engine, still RED as sourced): `flows_v1/*` and `http_v1/http_tier1`.
See section 6.
## 2. Data actually used, with licences and the attribution each one requires
### 2.1 Flows with addresses and timestamps: `entity_clean/` (data repo)
Built by `guard_jobs/entity_build.py --clean` (launched as `--job entity_clean`, which attaches only GREEN Kaggle
datasets to the kernel), runs `guard-clean-entity-build` and `guard-clean-entity-fix`. Every row has
`licence`; the builder raises before writing if any row is outside the allowlist.
| testbed | source as read (owner's own release) | licence | rows | span | sources | families | cite |
|---|---|---|---|---|---|---|---|
| simargl2021 | Kaggle `h2020simargl/simargl2021-network-intrusion-detection-dataset` | CC BY 4.0 | 8,000,000 | 443 h | 46,873 | benign 3.92 M, portscan 2.50 M, DoS (RUDY, slowloris) 1.58 M | Mihailescu et al., Sensors 21(13):4319, 2021 |
| simargl2022 | Kaggle `h2020simargl/simargl2022` (days 15, 16, 19 Mar 2022) | CC BY 4.0 | 8,000,000 | 100 h | 35,512 | benign 3.30 M, portscan 3.88 M, dos 0.82 M | H2020 SIMARGL, SIMARGL2022 dataset (CC BY 4.0) |
| appraise | Kaggle `ittibydgoszcz/appraise-h2020-real-labelled-netflow-dataset` | CC BY 4.0 | 7,570,072 | 1,912 h | 1,339 | benign 4.78 M, recon 2.36 M, ddos 0.43 M, bruteforce 949 | ITTI, APPRAISE H2020 real labelled NetFlow dataset (CC BY 4.0) |
| aci2023 | Kaggle `emilynack/aci-iot-network-traffic-dataset-2023` (`ACI-IoT-2023.csv`, the author's release) | CC BY 4.0 | 1,231,411 | 102 h | 159,381 | benign 0.33 M, portscan 0.44 M, floods 0.29 M, recon 0.15 M, slowloris, dictionary | Nack et al., MILCOM 2024; doi:10.21227/qacj-3x32 |
| cic2018aws | CSE-CIC-IDS2018, AWS open-data bucket (publisher's copy), processed day `Thuesday-20-02-2018` | CIC grant: redistribute / republish in any form, citation required | 7,948,748 | 12 h | 31,291 | benign 7.37 M, DDoS LOIC-HTTP 0.58 M | Sharafaldin, Habibi Lashkari, Ghorbani, ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018 |
| lu2020 | Kaggle `mryanm/luflow-network-intrusion-detection-data-set` (author's release), 2020 files | CC BY-SA 4.0 | 6,856,505 | 180 h | 1,249 | benign 5.12 M, malicious 1.74 M | Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517 |
| lu2021 | same, 2021 files | CC BY-SA 4.0 | 6,430,074 | 2,193 h | 691 | benign 5.40 M, malicious 1.03 M | as lu2020 |
`lu2022` was built too but is not used: the 2022 files carry a single source address, so there is no per-source
stream. It stays in `entity_clean/` and every trainer run passes the testbed list explicitly.
### 2.2 HTTP: `http_clean2/` (data repo; `http_clean/` is the first, superseded build)
Built by `guard_jobs/http_clean.py` (`--job http_clean`, or `http_build --clean`), run `guard-clean-http-build3`
with `--prefix http_clean2 --per-file 20000 --per-family 40000 --placements 2`. 1,006,457 rows; by licence: CC0
776,942, own 132,738, MIT 61,719, Apache-2.0 35,058.
| source | as read | licence | what it contributes | cite / notice |
|---|---|---|---|---|
| srbh | SR-BH 2020 multi-label, Harvard Dataverse doi:10.7910/DVN/OGOIXX, file `data_capec_multilabel.csv` (licence read from the Dataverse API: CC0-1.0) | CC0 1.0 | 878,968 real requests to a WordPress honeypot (357,536 attacks, CAPEC labels mapped to families), benign carriers, the `wild` suite (last 20 % of time) | Sureda Riera et al., Computers & Security 2022, doi:10.1016/j.cose.2022.102788 |
| wp_post | SR-BH 2020 WordPress POST subset, Kaggle `ahmeterenceylan/http-request-dataset` | CC0 upstream (uploader label MIT) | held-out suite, never trained on; SR-BH's own 28,381 POST requests are left out of `srbh` for that reason | as srbh |
| modsec | ModSecurity thirty-day archive, Zenodo 17178461 (`http_v2/modsec.parquet`, already built) | CC BY 4.0 | days 1-20 join training (`--modsec-train 1`, as the shipped `http_v2_final2`), days 21-30 are the `modsec` suite | Lucz, G. (2025), doi:10.5281/zenodo.17178461 |
| patt | PayloadsAllTheThings `*/Intruder(s)/*` lists, commit `08ea40e7` | MIT | attack payloads placed into benign carriers | MIT notice |
| seclists | SecLists `Fuzzing/*` payload lists (family by file keyword), commit `0008f434` | MIT | attack payloads placed into benign carriers | MIT notice |
| crs | OWASP CRS `tests/regression/tests` for rule files 913, 921, 930-944, commit `ca679721` | Apache-2.0 | 3,798 attack requests (positive tests) and 802 benign hard negatives (negative tests) | Apache-2.0 + NOTICE |
| synth | own generator in `guard_jobs/http_clean.py` | own | 80,000 benign shop / blog / API requests (English and Persian parameters, JSON bodies) as rows and carriers; benign values placed exactly like the attack payloads; the evasion variants (own mutation engine) | - |
| lab | Kaggle `moradrawashdeh/attack-simulation-lab` (uploader's own lab) | Apache-2.0 | evaluation suite only (labels are per host, not per request) | Apache-2.0 notice |
Two cleaning rules were added to the payload lists (both data quality, not licence): a list line is kept only if it
carries the family's active syntax (`http_clean.ACTIVE`; the Intruder / Fuzzing folders also hold plain word lists
such as event-handler names, bare file paths and LDAP attribute names, which are not attacks by themselves), and
files named after the PortSwigger cheat sheet are dropped (the audit treats PortSwigger-derived content as
unlicensed, row 17).
### 2.3 Not used, and why
| candidate | reason |
|---|---|
| NF-UQ-NIDS-v2, NF-*-v3 (athena21, thanhphuc1810 "MIT", seyhed, ...) | RED (UQ academic-only terms; uploader labels grant nothing) |
| dhoogla CIC-IDS2017 / CSE-CIC-IDS2018 / CIC-DDoS2019, dhoogla LUFlow | RED as sourced (CC BY-NC-SA mirrors) |
| CIC-IDS2017 and CIC-DDoS2019 originals | **Not available from the publisher without a manual step**: every `cicresearch.ca/CICDataset/...` path probed from a Kaggle kernel redirects to the UNB dataset index (download behind UNB's registration form); no Kaggle copy is published by UNB/CIC. Not used. |
| CSE-CIC-IDS2018 other days | the AWS processed CSVs of the other nine days have no Src IP / Dst IP (unusable for per-source streams; usable for a single-flow `cic` view, see section 6) |
| ACI-IoT-2023 / IoMT-2024 CIC-BCCC-NRC copies (phungvannamanh, amineipad) | AMBER (uploader MIT label, CIC/BCCC terms unconfirmed); the author's own ACI release is used instead |
| MU-IoT | GREEN, but the release has normalised features only (no addresses, no timestamps) |
| zanbil.ir | AMBER (raw log withheld by the owner, real client IPs) |
| CSIC 2010, mreowie, obf_sqli, obf_xss, syed_xss, syed_sqli, sambit | RED / AMBER |
| own red-team generators (`scripts/redteam_*.py`, `dibaguard.shield.playbook`) | GREEN and they do produce flow tables, but they are the live lab's own test traffic; training on them would grade the models on their training data. Kept as the independent lab check. |
## 3. The licence gate
* `guard_jobs/licences.py`: `CLEAN` = {cc-by-4.0, cc-by-sa-4.0, cc0-1.0, mit, apache-2.0, cic-unb-cite, own}; `GREEN`
= source id -> (licence, the origin the audit cleared, attribution); `OTHER` = the audit's tags for every RED /
AMBER source.
* Every corpus row has a `licence` column (`entity_build` now writes it for every testbed, clean or not; `http_build`'s
tags were corrected to the audit's values).
* `entity_build --clean`: refuses a testbed that is not GREEN or whose Kaggle ref differs from the cleared origin;
`require_clean` on every testbed's rows before the parquet is written.
* `http_clean`: `require_source` on every row's source, `require_clean` on the whole frame before writing.
* `entity_train --clean`, `seq_train --clean`, `http_train2 --clean`: read only `entity_clean/` / `http_clean*`,
re-check every loaded testbed / split (source GREEN, every row's licence in `CLEAN`), raise otherwise; outputs are
`clean_*` files from separate scratch folders, so a clean run cannot upload a shipped file name.
* `tests/test_clean.py` builds every clean loader on synthetic files, checks the outputs, and checks that a RED
testbed (`nfv3_cic2018`, `aciiot2023`) and a single tampered row (`uq-academic-only`) are refused.
## 4. Loader defects found on the way (fixed in `--clean` mode only; the shipped corpora keep their bytes)
1. **LUFlow time unit.** The author's release stores `time_start` in microseconds, plus a few rows on other scales
or at 0. The builder divided by 1000, so the shipped `entity_v1/lu2020` stream ran on a 1000x stretched clock
with a near-zero first row (`span_h` 442,511,328 in its manifest). Every horizon of the entity state (10 s / 2 min /
30 min) was therefore effectively 1000x shorter on LUFlow for `lu_entity_detector`, `seqnet` and
`nf_entity_mono`. Clean build: per-row scaling and a plausibility window (lu2020 now spans 180 h).
2. **SYN-only on exporters without a client/server flag split** (SIMARGL, APPRAISE): `client_flags` was 0, so the
SYN-only share was always 0 on SIMARGL2021, a dataset that is one-third SYN scans. Clean build: rebuilt from
the cumulative `TCP_FLAGS` and whether the server sent any packet.
3. **ACI-IoT-2023 author timestamps** mix spellings across days (`2023-01-11` = 1 Nov). Clean build: each row is
read every plausible way and the reading inside the README's capture week is kept (span 102 h, was 5,669 h).
4. CICFlowMeter rows with an unparseable time became -9.2e9 s and sorted first; now dropped.
## 5. Training-protocol results (leave-one-dataset-out, as the training jobs report them)
Reports in `D:/build-tmp/guard/clean/`. These are each model set's own leave-out numbers, on different testbeds
than the shipped reports, so they are context for section 0, not a head-to-head.
* `clean_entity_v1_train.json` / `clean_entity_v1_grouped.json` (runs `guard-clean-entity-1`, `-2`): nf view,
`ent` features, held-out flow AUROC — SIMARGL2021 0.966 (0.947 with SIMARGL2022 also held out), SIMARGL2022 0.996
(0.979 grouped), APPRAISE 0.934, ACI-IoT 0.969, CSE-CIC-IDS2018 AWS 0.997; lu view LUFlow 2020 0.997, 2021 0.995.
Site-calibrated realised FPR at 1e-3 departs from the target on SIMARGL2021 (0.12 single, 2e-4 grouped), SIMARGL2022
grouped (0.019) and ACI-IoT (0.021): few sources with long-lived state are not exchangeable between periods, the
limit the card already states for UNSW-NB15.
* `clean_seq_v1_train.json` / `_grouped` / `_keepatt` (runs `guard-clean-seq-1`, `-2`, `-4`): GRU and fused numbers
in section 0.3; anomaly tier: monotone prior leave-out AUROC SIMARGL2022 1.000, CSE-CIC-IDS2018 AWS 0.9998,
APPRAISE 0.24 on the source-capped sample of `seq-1` (591 attack flows in its test split) and 0.992 when every
attacking source is kept (`seq-4`), ACI-IoT 0.71; the label-free site-only tier (quant) 0.99-1.00 on SIMARGL /
CSE-CIC and 0.81 on ACI-IoT. In `seq-1` the SIMARGL2021 test split lost its only attacker to the random source
sampling (the shipped run happened to keep one), which is why `seq-4` re-ran the anomaly protocol with
`--keep-attack-sources 1`. LUFlow anomaly folds are one-sided after source capping (as in the shipped report).
* `clean_seq_v1_export.json` (`guard-clean-seq-3`): exported clean GRU, d = 64, 95,293 parameters, 381 KB, numpy /
torch parity 4.6e-5, 130k flows/s on one core.
* `clean_http_v2_train.json` (`guard-clean-http-2`, corpus `http_clean2`): leave-one-source-out SR-BH AUROC 0.873
(no site context; 0.42 with context) / site TPR 0.24 at FPR 7e-4, novelty tier 0.984, fused 0.974; WordPress POST
0.947 / 0.17 (shipped: 0.693 / 0.17); CRS 0.57; lab 0.14 (novelty 0.996). Final model: test AUROC 0.992, evasion
TPR 0.77, ModSecurity last ten days caught 0.958 with the SR-BH reference, 3,387 clients 0.895; 7,849 KB; 1,888 req/s
on one core in Python. `clean_http_v2_train_run1_corpus_v1.json` is the first attempt (corpus `http_clean`, before
the payload-list cleaning): XSS 0.18, LFI 0.15, evasion 0.29, kept for the record.
## 6. Gaps: what still depends on data that is not cleared
1. **HTTP model.** The clean HTTP model is not a replacement (section 0.4). What it lacks is benign and attack
variety from applications other than one WordPress honeypot: the shipped model's strength on CSIC / zanbil /
payload-set traffic comes from RED / AMBER data (CSIC 2010, zanbil.ir, mreowie, obfuscated SQLi / XSS sets).
Routes, in order of value: (a) the author's written OK for zanbil.ir (CC0, raw log withheld; DATA_LICENSES §4.3),
which restores a real non-WordPress benign site; (b) CSIC 2010 permission; (c) more GREEN benign applications
(requests generated from the CC0 APIs.guru OpenAPI directory, CRS negative tests at scale); (d) more own attack
grammar and WAF-A-MoLE (MIT) mutation of GREEN payloads. Until then the commercial build has **no** licence-clean
HTTP model; the site-only novelty tier (no training data) and the playbook signatures remain clean.
2. **IoT / UNSW-NB15-style networks.** The clean prior is weak on NF-UNSW-NB15 (AUROC 0.07) and below the shipped
model on NF-ToN-IoT. There is no GREEN twin of UNSW-NB15, BoT-IoT or ToN-IoT (all UNSW Canberra, academic-only);
ToN-IoT is "allowable after asking the author" — the one permission worth asking for the network side. MU-IoT
(CC BY 4.0) has no addresses; CIC-IoT2023 from UNB would need its PCAPs re-flowed.
3. **CIC-IDS2017 and CIC-DDoS2019** could not be fetched from the publisher without the registration form; only the
CSE-CIC-IDS2018 day with addresses (Tuesday 20-02) is in. Downloading the UNB originals by hand (GREEN with
citation) and re-flowing the 2018 PCAPs from AWS would add a CIC testbed with every attack family.
4. **Legacy artefacts not retrained:** `flows_v1/*` (single-flow nf / cic / lu detectors) and `http_v1/http_tier1`
are still RED as sourced. They are not loaded by the v2 engine. A clean `cic` view is possible from the ten AWS
CSE-CIC-IDS2018 processed CSVs (no addresses needed for single flows), a clean `lu` view from the author's LUFlow.
5. **Go runtime files:** the clean models exist as `.npz`; the flat `.bin` / `.bin.json` exports and parity fixtures
for the Go runtime (`dibaguard.parity.npz_to_flat`) were not produced for them.
6. **Open licence points that touch the clean set** (from DATA_LICENSES §5, unchanged): LUFlow is CC BY-SA 4.0 —
whether trained weights are Adapted Material is unsettled, so the LUFlow-trained parts (clean `lu` detector, and
the GRU / prior, which include LUFlow 2020-21) need legal sign-off or a retrain without LUFlow; the uploader
identities of the SIMARGL, LUFlow and ACI-IoT Kaggle accounts are inferred, not verified; the CIC FAQ wording
should be confirmed by e-mail; PayloadsAllTheThings / SecLists files credit upstream cheat sheets whose own terms
were not checked file by file (the PortSwigger-named files were dropped).
7. **Shipped models carry the old loader defects** (section 4): the shipped LUFlow-trained parts ran on a 1000x
stretched clock and SIMARGL2021 without the SYN-only signal. A rebuild of `entity_v1` with the `--clean` fixes
applied to every testbed would be needed before comparing future shipped retrains to these numbers.
## 7. Runs, files and how to check them
| run (account) | job and arguments | output |
|---|---|---|
| `guard-clean-survey` (1) | `survey_clean` | publisher channels, SR-BH, payload repos, lab, ACI author release |
| `guard-clean-entity-build` (1), `guard-clean-entity-fix` (1) | `entity_clean` (`entity_build --clean`); fix run `--testbeds aci2023,lu2020,lu2021` | `entity_clean/*.parquet` + manifest |
| `guard-clean-http-build` (2, stray first version), `-build2` (2), `-build3` (2), `-build4` (2) | `http_clean`; build3 `--prefix http_clean2 --per-file 20000 --per-family 40000 --placements 2`; build4 adds `--own-attacks 1 --no-ua-share 0.15 --prefix http_clean3` | `http_clean/`, `http_clean2/`, `http_clean3/` |
| `guard-clean-entity-1` (1, GPU) | `entity_train --clean --views nf,lu --sets base,ent,both --testbeds <7> --iters 1000 --max-rows 1500000` | `models/entity_v1/clean_{nf,lu}_entity_detector.npz`, `reports/clean_entity_v1_train.json` |
| `guard-clean-entity-2` (1, GPU) | same, `--lodo "simargl2021+simargl2022,..." --no-export` | `reports/clean_entity_v1_grouped.json` |
| `guard-clean-seq-1` (1, GPU) | `seq_train --clean --d 128 --ctx 0 --fset ent --protocols temporal,lodo,anomaly --iters 300 --rolling 1800` | `models/seq_v1/clean_nf_entity_mono.npz`, `reports/clean_seq_v1_train.json` |
| `guard-clean-seq-2` (1, GPU) | grouped leave-out, `--export-mono 0` | `reports/clean_seq_v1_grouped.json` |
| `guard-clean-seq-3` (2, GPU) | `--d 64 --protocols export` | `models/seq_v1/clean_seqnet.npz`, `clean_seq_v1_info.json` |
| `guard-clean-seq-4` (1, GPU) | anomaly protocol with `--keep-attack-sources 1` | `reports/clean_seq_v1_keepatt.json` |
| `guard-clean-http-1` (2, GPU) | `http_train2 --clean` on `http_clean`; first version killed (out of memory at the SR-BH leave-out, fixed by `--loso-cap`) | `reports/clean_http_v2_train.json` (renamed locally `_run1_corpus_v1`) |
| `guard-clean-http-2` (2, GPU) | `http_train2 --clean --corpus http_clean2 --protocols loso,final --ctx 0,1 --modsec-train 1` | `models/http_v2/clean_http_tier1_v2.npz`, `reports/clean_http_v2_train.json` |
| `guard-clean-http-3` (1, CPU), `guard-clean-http-3f` (2, CPU) | same on `http_clean3` (`--loso srbh,wp_post,crs`); `-3f` final only | `models/http_v2/clean_http_tier1_v2_c3.npz` / `_c3f.npz`, `reports/clean_http_v2_train3*.json` |
| `guard-clean-eval-flows` (1, CPU), `guard-clean-eval-http` (2, CPU), `guard-clean-eval-http3` (2, CPU) | `clean_eval --parts flows` / `--parts http` / `--parts http --clean-http models/http_v2/clean_http_tier1_v2_c3f.npz` | `reports/clean_vs_shipped_{flows,http,http_c3}.json` |
Check any run: `python kag.py status --run-name --result` (from `D:/diba-slm/guard`); full log after the
kernel ends: `python kag.py log --run-name `. Reports and models are in `DibaAi/dibaguard-results-private`,
corpora in `DibaAi/dibaguard-data-private` (both private).
Code added or changed (training side only, `guard_jobs/` and `tests/`): `licences.py` (new), `entity_clean.py`
(new), `http_clean.py` (new), `clean_eval.py` (new), `survey_clean.py` (new); `--clean` and related options in
`entity_build.py`, `entity_train.py`, `seq_train.py`, `http_build.py`, `http_train2.py`; `tests/test_clean.py`.
Default (non-clean) behaviour of the existing jobs is unchanged except that `entity_build` now writes a `licence`
column and `http_build`'s licence tags carry the audit's values.
## 8. Attribution block for a commercial release of the clean network models
CSE-CIC-IDS2018 (Sharafaldin, Habibi Lashkari, Ghorbani, ICISSP 2018; https://registry.opendata.aws/cse-cic-ids2018);
RoEduNet-SIMARGL2021 (Mihailescu et al., Sensors 21(13):4319, 2021, CC BY 4.0); SIMARGL2022 (H2020 SIMARGL, CC BY
4.0); APPRAISE H2020 real labelled NetFlow (ITTI, CC BY 4.0); ACI-IoT-2023 (Nack et al., MILCOM 2024,
doi:10.21227/qacj-3x32, CC BY 4.0); LUFlow (Mills et al., IEEE TNSM 2021, doi:10.1109/TNSM.2021.3091517, CC BY-SA
4.0). For an HTTP model built from the clean corpus, add SR-BH 2020 (CC0; cite doi:10.1016/j.cose.2022.102788),
the ModSecurity archive (doi:10.5281/zenodo.17178461, CC BY 4.0), and the MIT / Apache-2.0 notices of
PayloadsAllTheThings, SecLists and the OWASP CRS. The same block is stored in every clean model's metadata
(`notice`).