| # dataset | |
| 100 records — 5 security patterns × 10 repositories × 2 languages (Java, Python) — | |
| with their ground truth, the codebases, the outputs of every experiment, and the | |
| patterns themselves. | |
| ``` | |
| dataset/ | |
| ├── manifest_java_full_20261005.json the 50 Java records | |
| ├── manifest_python_full_20261005.json the 50 Python records | |
| ├── ground_truth/ 1. role mapping + interactions, per record | |
| │ ├── java/<pattern>__<repo>.json | |
| │ ├── python/<pattern>__<repo>.json | |
| │ └── summary.csv | |
| ├── codebase/ 2. the code each record is recovered from | |
| │ ├── repos/<language>/<repo>/ | |
| │ ├── cpg/<repo>__<JAVASRC|PYTHONSRC>.cpg | |
| │ ├── repos.csv provenance: GitHub URL, commit, license | |
| │ └── licenses/<language>/<repo>/ each repo's license and NOTICE files | |
| ├── experiment_outputs/ 3. 4 experiments × 4 models × 2 languages | |
| │ └── <experiment>/<model>/<language>/ | |
| │ ├── evaluation.json | |
| │ └── <pattern>__<repo>/recovery.json | |
| ├── security_patterns/ 4. the 5 patterns | |
| │ └── <pattern>/{01_text.md, 02_diagram.md, 03_subflows.md, 04_pattern.json} | |
| ├── Recovery_Conversation_Session.txt 5. the session that drafted the role mappings | |
| └── LICENSE MIT, for this dataset's own content (see License) | |
| ``` | |
| ## The paper | |
| This is the dataset of the paper *A Hybrid Approach for Automated Security | |
| Pattern Recovery*: the groundtruth of its Section IV-B, the codebases and CPGs | |
| the recovery runs on, and the outputs and scores of every experiment of its | |
| Section V. The code that produced and scores them is | |
| [secpatrec](https://github.com/anonymousspottedtoad/secpatrec), which installs the | |
| interaction walk and the metrics. The paper's appendix is at | |
| <https://doi.org/10.5281/zenodo.23664769>; its Appendix G lists all the artifacts | |
| and shows how the reported numbers are recomputed from this dataset. | |
| | Paper | Here | | |
| | ----- | ---- | | |
| | recovery instance (100 of them) | record: one pattern in one repository, listed in the manifests | | |
| | the specification *P* of P1 to P5 | `security_patterns/<pattern>/04_pattern.json`: P1 `authentication_pwd`, P2 `verifiable_token_based_authentication`, P3 `obscure_token_based_access_control`, P4 `session_based_access_control`, P5 `limit_request_rate` | | |
| | the groundtruth mapping *M*, statuses *D* and interaction sequence *S* | `role_mappings` (`entity`, `status`) and `interaction_order` in `ground_truth/` | | |
| | RDA, MLS, IRC | Levels 1, 2 and 3 in `evaluation.json` | | |
| | GPT-6.1, Muse Spark, DeepSeek, GLM | the `<model>` folders `gpt-6.1-sol`, `muse-spark-1.3-contributor`, `deepseek-v4-flash-0731`, `glm-5.3-flash` | | |
| | SecPatRec, the one-shot baseline, and RQ4's settings without lexical or semantic search | the `<experiment>` folders `pipeline`, `baseline`, `ablation_no_grep`, `ablation_no_search` | | |
| | the Claude Code session of Appendix D-A | `Recovery_Conversation_Session.txt` | | |
| ## Manifests | |
| Each manifest is `{"runs": [...]}`, one entry per record: `pattern`, `repo` and | |
| `language`, plus the codebase's provenance — `url` (its GitHub repository), | |
| `commit` (the commit it was cloned at) and `license` (see | |
| [2. Codebase](#2-codebase)). The rest of a record follows from `pattern`, `repo` | |
| and `language`: | |
| | | | | |
| | - | - | | |
| | codebase | `codebase/repos/<language>/<repo>/` | | |
| | CPG | `codebase/cpg/<repo>__<JAVASRC\|PYTHONSRC>.cpg` | | |
| | pattern | `security_patterns/<pattern>/04_pattern.json` | | |
| | ground truth | `ground_truth/<language>/<pattern>__<repo>.json` | | |
| The pipeline (`secpatrec`) runs the experiments on this dataset with | |
| `secpat evaluate`, `secpat interaction` and `secpat recover --models`, each part | |
| of the dataset named by its own flag. Its README covers installing, downloading | |
| this dataset and running. | |
| ## 1. Ground truth | |
| One JSON file per record, named `<pattern>__<repo>.json`, with two fields: | |
| | Field | Holds | | |
| | ----- | ----- | | |
| | `role_mappings` | one row per role the annotators recorded (see below): `patternRole`, `status` (`present`, `external`, `missing`), `entity`, `file`, `lines`, `notes` — the same schema as a recovery's `recovery.json` | | |
| | `interaction_order` | the order the roles interact in: `entryRole`, `entryRule`, and `interactions` ranked from the entry role, each with `from`, `to`, the call site (`file`, `line`), its call `path` and the `mechanism` that reached it — again the same schema as `recovery.json` | | |
| The 100 files hold 758 rows: 497 for roles the pattern requires, 200 for Subject | |
| and System (Entity and System in `limit_request_rate`), which every file records, | |
| and 61 for roles of the patterns' other flows that `04_pattern.json` does not list, | |
| such as the Registrar of P1. The paper's Table III counts only the roles that | |
| `04_pattern.json` requires, 500 in all, and a required role with no row counts as | |
| `missing`, which is the case for the Pepper store in three P1 records. Counted this | |
| way, the files give 334 present, 75 external and 91 missing roles, and 280 | |
| interactions. The metrics read the files the same way. | |
| `interaction_order` is derived from `role_mappings` by walking the repo's CPG with | |
| Joern 4.0.627, using the walk scripts in `secpat_interaction_recovery` | |
| (`walk.sc` a81b989f…, `java.sc` eb906608…, `python.sc` b7038c5f…). All 100 records | |
| have an order. The annotators checked it in the annotation interface (the paper's | |
| Appendix D-B), which derives the order again from the mapping whenever they change | |
| a mapping and saves the order derived from the saved mapping. So every stored | |
| interaction is the walk's output, and its `mechanism` says how the walk reached it. | |
| Level 3 does not read this order: it walks the groundtruth mapping again over the | |
| roles both sides detected (see `secpat_evaluation_metrics`). | |
| `summary.csv` has one row per record: the number of interactions and the order as | |
| `From>To → …`. | |
| ## 2. Codebase | |
| - `repos/<language>/<repo>/` — the 93 repositories (44 Java, 49 Python); five | |
| serve two patterns and one serves three. Only the `.java` / `.py` source files | |
| are kept. | |
| - `cpg/` — one Joern 4.0.627 code property graph per repository. Point | |
| `JOERN_CPG_ROOT` here to run Level 3 and stage 3 without rebuilding them. Java | |
| CPGs leave out `test/` folders (the `javasrc2cpg` default). | |
| - `repos.csv` — one row per repository: `language`, `repo`, `github` | |
| (`owner/name`), `url`, `commit`, `license`, `license_source` (where the license | |
| was read: the license file, or a `pom.xml` / `pyproject.toml` declaration), | |
| `license_files` and `records` (the patterns it serves). Each repository is its | |
| GitHub repository at `commit`. | |
| - `licenses/<language>/<repo>/` — the license and NOTICE files of each repository | |
| at `commit`, at their original paths. `repos/` keeps only source files, so | |
| these are copied here. | |
| `license` is an SPDX identifier as GitHub reports it, read at `commit`. Two values | |
| are not licenses: `NONE` — the repository has no license file or declaration at | |
| that commit — and `NOASSERTION` — only some of its modules declare one. | |
| ## 3. Experiment outputs | |
| | `<experiment>` | Setting | | |
| | -------------- | ------- | | |
| | `pipeline` | the full recovery pipeline (`secpat recover`) | | |
| | `baseline` | one-shot recovery: the whole repository packed into one prompt (`secpat baseline`) | | |
| | `ablation_no_grep` | the pipeline with the `grep` tool disabled (keyword search) | | |
| | `ablation_no_search` | the pipeline with the `search` tool disabled (semantic search) | | |
| `<model>` is one of `gpt-6.1-sol`, `deepseek-v4-flash-0731`, `glm-5.3-flash`, | |
| `muse-spark-1.3-contributor`. | |
| Each `<experiment>/<model>/<language>/` folder holds one subfolder per record | |
| with its `recovery.json`: `role_mappings` and `interaction_order`, in the same | |
| schema as the ground truth. | |
| - **Pipeline and ablations:** Level 3 does not read the stored | |
| `interaction_order`. It derives the recovery's order from `role_mappings` on | |
| the CPG, with the same walk as the ground truth's. The ablation runs, and some | |
| pipeline runs, skipped the order at recovery time, so theirs reads | |
| `available: false`. | |
| - **Baseline:** `interaction_order` is the order the model stated in its answer, | |
| and Level 3 scores that (`--recovery-order stored`). | |
| `evaluation.json` is the batch's score, one entry per record with Levels 1–3: | |
| the per-record scores behind the paper's results for RQ1, RQ3 and RQ4. Their | |
| means are the reported RDA, MLS and IRC; for the pipeline over the four models, | |
| 0.84, 0.84 and 0.66. The paper's token counts, costs, steps and memory use (RQ2, | |
| Figs. 38 and 39 of RQ3, and part of Table X of RQ4) are not in this dataset. | |
| - **Baseline:** 16 records (10 Java, 6 Python, from 14 repositories) have a one-shot prompt over the | |
| 900K-token input limit (counted with the `o200k_base` tokenizer). They were not | |
| run, have no folder, and score 0 on every level. | |
| - The other experiments ran all 50 records per language. | |
| To score these recoveries again, see Experiment 1 in the `secpatrec` README. A | |
| fresh score leaves the baseline's 16 unrun records unscored; `evaluation.json` | |
| counts them as 0. | |
| ## 4. Security patterns | |
| The five patterns, each as `01_text.md` (the catalogue page), `02_diagram.md`, | |
| `03_subflows.md` and `04_pattern.json` (the roles and the behaviour walkthrough | |
| the recovery and the interaction walk read): | |
| - `authentication_pwd` — password-based authentication | |
| - `verifiable_token_based_authentication` | |
| - `obscure_token_based_access_control` | |
| - `session_based_access_control` | |
| - `limit_request_rate` | |
| ## 5. Recovery conversation session | |
| `Recovery_Conversation_Session.txt` is the exported Claude Code session in which | |
| the ground truth's role mappings were first drafted (the paper's Appendix D-A). A planner agent writes one | |
| language-independent recovery plan, tries it on one or two repositories, then | |
| hands the plan, the pattern and the repository to one sub-agent per repository. | |
| The session runs Java first and then Python, in batches of up to 20 repositories | |
| per pattern, so it covers more repositories than the 100 records kept here. | |
| ## License | |
| - **This dataset's own content** — the ground truth, the manifests, the experiment | |
| outputs and the recovery conversation session — is released under the MIT | |
| License (see `LICENSE`). | |
| - **`security_patterns/`** is derived from the security pattern catalogue of | |
| Van Den Berghe et al., © 2021-2022 imec-DistriNet | |
| (<https://securitypatterns.distrinet-research.be/>), licensed under | |
| [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/): | |
| `01_text.md` is the catalogue page, and `02_diagram.md`, `03_subflows.md` and | |
| `04_pattern.json` are adaptations of it. It is shared under the same license. | |
| - **`codebase/`** holds third-party code. Each repository keeps its own license, | |
| listed in `codebase/repos.csv`, with its license files in `codebase/licenses/`. | |
Xet Storage Details
- Size:
- 11 kB
- Xet hash:
- 2fc8d8ea843e7a650681db5317c6558d396254d40e925533568850f5efdfb302
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.