Buckets:

1.07 GB
45,174 files
Updated 19 days ago
Name
Size
security_patterns
ground_truth
experiment_outputs
codebase
manifest_python_full_20261005.json13.6 kB
xet
manifest_java_full_20261005.json14 kB
xet
Recovery_Conversation_Session.txt768 kB
xet
README.md11 kB
xet
LICENSE1.07 kB
xet
README.md

dataset

100 records — 5 security patterns × 10 repositories × 2 languages (Java, Python) — with their ground truth, the codebases, the outputs of every experiment, and the patterns themselves.

dataset/
├── manifest_java_full_20261005.json     the 50 Java records
├── manifest_python_full_20261005.json   the 50 Python records
├── ground_truth/                        1. role mapping + interactions, per record
│   ├── java/<pattern>__<repo>.json
│   ├── python/<pattern>__<repo>.json
│   └── summary.csv
├── codebase/                            2. the code each record is recovered from
│   ├── repos/<language>/<repo>/
│   ├── cpg/<repo>__<JAVASRC|PYTHONSRC>.cpg
│   ├── repos.csv                        provenance: GitHub URL, commit, license
│   └── licenses/<language>/<repo>/      each repo's license and NOTICE files
├── experiment_outputs/                  3. 4 experiments × 4 models × 2 languages
│   └── <experiment>/<model>/<language>/
│       ├── evaluation.json
│       └── <pattern>__<repo>/recovery.json
├── security_patterns/                   4. the 5 patterns
│   └── <pattern>/{01_text.md, 02_diagram.md, 03_subflows.md, 04_pattern.json}
├── Recovery_Conversation_Session.txt    5. the session that drafted the role mappings
└── LICENSE                              MIT, for this dataset's own content (see License)

The paper

This is the dataset of the paper A Hybrid Approach for Automated Security Pattern Recovery: the groundtruth of its Section IV-B, the codebases and CPGs the recovery runs on, and the outputs and scores of every experiment of its Section V. The code that produced and scores them is secpatrec, which installs the interaction walk and the metrics. The paper's appendix is at https://doi.org/10.5281/zenodo.23664769; its Appendix G lists all the artifacts and shows how the reported numbers are recomputed from this dataset.

Paper Here
recovery instance (100 of them) record: one pattern in one repository, listed in the manifests
the specification P of P1 to P5 security_patterns/<pattern>/04_pattern.json: P1 authentication_pwd, P2 verifiable_token_based_authentication, P3 obscure_token_based_access_control, P4 session_based_access_control, P5 limit_request_rate
the groundtruth mapping M, statuses D and interaction sequence S role_mappings (entity, status) and interaction_order in ground_truth/
RDA, MLS, IRC Levels 1, 2 and 3 in evaluation.json
GPT-6.1, Muse Spark, DeepSeek, GLM the <model> folders gpt-6.1-sol, muse-spark-1.3-contributor, deepseek-v4-flash-0731, glm-5.3-flash
SecPatRec, the one-shot baseline, and RQ4's settings without lexical or semantic search the <experiment> folders pipeline, baseline, ablation_no_grep, ablation_no_search
the Claude Code session of Appendix D-A Recovery_Conversation_Session.txt

Manifests

Each manifest is {"runs": [...]}, one entry per record: pattern, repo and language, plus the codebase's provenance — url (its GitHub repository), commit (the commit it was cloned at) and license (see 2. Codebase). The rest of a record follows from pattern, repo and language:

codebase codebase/repos/<language>/<repo>/
CPG codebase/cpg/<repo>__<JAVASRC|PYTHONSRC>.cpg
pattern security_patterns/<pattern>/04_pattern.json
ground truth ground_truth/<language>/<pattern>__<repo>.json

The pipeline (secpatrec) runs the experiments on this dataset with secpat evaluate, secpat interaction and secpat recover --models, each part of the dataset named by its own flag. Its README covers installing, downloading this dataset and running.

1. Ground truth

One JSON file per record, named <pattern>__<repo>.json, with two fields:

Field Holds
role_mappings one row per role the annotators recorded (see below): patternRole, status (present, external, missing), entity, file, lines, notes — the same schema as a recovery's recovery.json
interaction_order the order the roles interact in: entryRole, entryRule, and interactions ranked from the entry role, each with from, to, the call site (file, line), its call path and the mechanism that reached it — again the same schema as recovery.json

The 100 files hold 758 rows: 497 for roles the pattern requires, 200 for Subject and System (Entity and System in limit_request_rate), which every file records, and 61 for roles of the patterns' other flows that 04_pattern.json does not list, such as the Registrar of P1. The paper's Table III counts only the roles that 04_pattern.json requires, 500 in all, and a required role with no row counts as missing, which is the case for the Pepper store in three P1 records. Counted this way, the files give 334 present, 75 external and 91 missing roles, and 280 interactions. The metrics read the files the same way.

interaction_order is derived from role_mappings by walking the repo's CPG with Joern 4.0.627, using the walk scripts in secpat_interaction_recovery (walk.sc a81b989f…, java.sc eb906608…, python.sc b7038c5f…). All 100 records have an order. The annotators checked it in the annotation interface (the paper's Appendix D-B), which derives the order again from the mapping whenever they change a mapping and saves the order derived from the saved mapping. So every stored interaction is the walk's output, and its mechanism says how the walk reached it. Level 3 does not read this order: it walks the groundtruth mapping again over the roles both sides detected (see secpat_evaluation_metrics).

summary.csv has one row per record: the number of interactions and the order as From>To → ….

2. Codebase

  • repos/<language>/<repo>/ — the 93 repositories (44 Java, 49 Python); five serve two patterns and one serves three. Only the .java / .py source files are kept.
  • cpg/ — one Joern 4.0.627 code property graph per repository. Point JOERN_CPG_ROOT here to run Level 3 and stage 3 without rebuilding them. Java CPGs leave out test/ folders (the javasrc2cpg default).
  • repos.csv — one row per repository: language, repo, github (owner/name), url, commit, license, license_source (where the license was read: the license file, or a pom.xml / pyproject.toml declaration), license_files and records (the patterns it serves). Each repository is its GitHub repository at commit.
  • licenses/<language>/<repo>/ — the license and NOTICE files of each repository at commit, at their original paths. repos/ keeps only source files, so these are copied here.

license is an SPDX identifier as GitHub reports it, read at commit. Two values are not licenses: NONE — the repository has no license file or declaration at that commit — and NOASSERTION — only some of its modules declare one.

3. Experiment outputs

<experiment> Setting
pipeline the full recovery pipeline (secpat recover)
baseline one-shot recovery: the whole repository packed into one prompt (secpat baseline)
ablation_no_grep the pipeline with the grep tool disabled (keyword search)
ablation_no_search the pipeline with the search tool disabled (semantic search)

<model> is one of gpt-6.1-sol, deepseek-v4-flash-0731, glm-5.3-flash, muse-spark-1.3-contributor.

Each <experiment>/<model>/<language>/ folder holds one subfolder per record with its recovery.json: role_mappings and interaction_order, in the same schema as the ground truth.

  • Pipeline and ablations: Level 3 does not read the stored interaction_order. It derives the recovery's order from role_mappings on the CPG, with the same walk as the ground truth's. The ablation runs, and some pipeline runs, skipped the order at recovery time, so theirs reads available: false.
  • Baseline: interaction_order is the order the model stated in its answer, and Level 3 scores that (--recovery-order stored).

evaluation.json is the batch's score, one entry per record with Levels 1–3: the per-record scores behind the paper's results for RQ1, RQ3 and RQ4. Their means are the reported RDA, MLS and IRC; for the pipeline over the four models, 0.84, 0.84 and 0.66. The paper's token counts, costs, steps and memory use (RQ2, Figs. 38 and 39 of RQ3, and part of Table X of RQ4) are not in this dataset.

  • Baseline: 16 records (10 Java, 6 Python, from 14 repositories) have a one-shot prompt over the 900K-token input limit (counted with the o200k_base tokenizer). They were not run, have no folder, and score 0 on every level.
  • The other experiments ran all 50 records per language.

To score these recoveries again, see Experiment 1 in the secpatrec README. A fresh score leaves the baseline's 16 unrun records unscored; evaluation.json counts them as 0.

4. Security patterns

The five patterns, each as 01_text.md (the catalogue page), 02_diagram.md, 03_subflows.md and 04_pattern.json (the roles and the behaviour walkthrough the recovery and the interaction walk read):

  • authentication_pwd — password-based authentication
  • verifiable_token_based_authentication
  • obscure_token_based_access_control
  • session_based_access_control
  • limit_request_rate

5. Recovery conversation session

Recovery_Conversation_Session.txt is the exported Claude Code session in which the ground truth's role mappings were first drafted (the paper's Appendix D-A). A planner agent writes one language-independent recovery plan, tries it on one or two repositories, then hands the plan, the pattern and the repository to one sub-agent per repository. The session runs Java first and then Python, in batches of up to 20 repositories per pattern, so it covers more repositories than the 100 records kept here.

License

  • This dataset's own content — the ground truth, the manifests, the experiment outputs and the recovery conversation session — is released under the MIT License (see LICENSE).
  • security_patterns/ is derived from the security pattern catalogue of Van Den Berghe et al., © 2021-2022 imec-DistriNet (https://securitypatterns.distrinet-research.be/), licensed under CC BY-NC-SA 4.0: 01_text.md is the catalogue page, and 02_diagram.md, 03_subflows.md and 04_pattern.json are adaptations of it. It is shared under the same license.
  • codebase/ holds third-party code. Each repository keeps its own license, listed in codebase/repos.csv, with its license files in codebase/licenses/.
Total size
1.07 GB
Files
45,174
Last updated
Nov 4
Pre-warmed CDN
US EU US EU

Contributors