dataset
100 records — 5 security patterns × 10 repositories × 2 languages (Java, Python) — with their ground truth, the codebases, the outputs of every experiment, and the patterns themselves.
dataset/
├── manifest_java_full_20261005.json the 50 Java records
├── manifest_python_full_20261005.json the 50 Python records
├── ground_truth/ 1. role mapping + interactions, per record
│ ├── java/<pattern>__<repo>.json
│ ├── python/<pattern>__<repo>.json
│ └── summary.csv
├── codebase/ 2. the code each record is recovered from
│ ├── repos/<language>/<repo>/
│ ├── cpg/<repo>__<JAVASRC|PYTHONSRC>.cpg
│ ├── repos.csv provenance: GitHub URL, commit, license
│ └── licenses/<language>/<repo>/ each repo's license and NOTICE files
├── experiment_outputs/ 3. 4 experiments × 4 models × 2 languages
│ └── <experiment>/<model>/<language>/
│ ├── evaluation.json
│ └── <pattern>__<repo>/recovery.json
├── security_patterns/ 4. the 5 patterns
│ └── <pattern>/{01_text.md, 02_diagram.md, 03_subflows.md, 04_pattern.json}
├── Recovery_Conversation_Session.txt 5. the session that drafted the role mappings
└── LICENSE MIT, for this dataset's own content (see License)
The paper
This is the dataset of the paper A Hybrid Approach for Automated Security Pattern Recovery: the groundtruth of its Section IV-B, the codebases and CPGs the recovery runs on, and the outputs and scores of every experiment of its Section V. The code that produced and scores them is secpatrec, which installs the interaction walk and the metrics. The paper's appendix is at https://doi.org/10.5281/zenodo.23664769; its Appendix G lists all the artifacts and shows how the reported numbers are recomputed from this dataset.
| Paper | Here |
|---|---|
| recovery instance (100 of them) | record: one pattern in one repository, listed in the manifests |
| the specification P of P1 to P5 | security_patterns/<pattern>/04_pattern.json: P1 authentication_pwd, P2 verifiable_token_based_authentication, P3 obscure_token_based_access_control, P4 session_based_access_control, P5 limit_request_rate |
| the groundtruth mapping M, statuses D and interaction sequence S | role_mappings (entity, status) and interaction_order in ground_truth/ |
| RDA, MLS, IRC | Levels 1, 2 and 3 in evaluation.json |
| GPT-6.1, Muse Spark, DeepSeek, GLM | the <model> folders gpt-6.1-sol, muse-spark-1.3-contributor, deepseek-v4-flash-0731, glm-5.3-flash |
| SecPatRec, the one-shot baseline, and RQ4's settings without lexical or semantic search | the <experiment> folders pipeline, baseline, ablation_no_grep, ablation_no_search |
| the Claude Code session of Appendix D-A | Recovery_Conversation_Session.txt |
Manifests
Each manifest is {"runs": [...]}, one entry per record: pattern, repo and
language, plus the codebase's provenance — url (its GitHub repository),
commit (the commit it was cloned at) and license (see
2. Codebase). The rest of a record follows from pattern, repo
and language:
| codebase | codebase/repos/<language>/<repo>/ |
| CPG | codebase/cpg/<repo>__<JAVASRC|PYTHONSRC>.cpg |
| pattern | security_patterns/<pattern>/04_pattern.json |
| ground truth | ground_truth/<language>/<pattern>__<repo>.json |
The pipeline (secpatrec) runs the experiments on this dataset with
secpat evaluate, secpat interaction and secpat recover --models, each part
of the dataset named by its own flag. Its README covers installing, downloading
this dataset and running.
1. Ground truth
One JSON file per record, named <pattern>__<repo>.json, with two fields:
| Field | Holds |
|---|---|
role_mappings |
one row per role the annotators recorded (see below): patternRole, status (present, external, missing), entity, file, lines, notes — the same schema as a recovery's recovery.json |
interaction_order |
the order the roles interact in: entryRole, entryRule, and interactions ranked from the entry role, each with from, to, the call site (file, line), its call path and the mechanism that reached it — again the same schema as recovery.json |
The 100 files hold 758 rows: 497 for roles the pattern requires, 200 for Subject
and System (Entity and System in limit_request_rate), which every file records,
and 61 for roles of the patterns' other flows that 04_pattern.json does not list,
such as the Registrar of P1. The paper's Table III counts only the roles that
04_pattern.json requires, 500 in all, and a required role with no row counts as
missing, which is the case for the Pepper store in three P1 records. Counted this
way, the files give 334 present, 75 external and 91 missing roles, and 280
interactions. The metrics read the files the same way.
interaction_order is derived from role_mappings by walking the repo's CPG with
Joern 4.0.627, using the walk scripts in secpat_interaction_recovery
(walk.sc a81b989f…, java.sc eb906608…, python.sc b7038c5f…). All 100 records
have an order. The annotators checked it in the annotation interface (the paper's
Appendix D-B), which derives the order again from the mapping whenever they change
a mapping and saves the order derived from the saved mapping. So every stored
interaction is the walk's output, and its mechanism says how the walk reached it.
Level 3 does not read this order: it walks the groundtruth mapping again over the
roles both sides detected (see secpat_evaluation_metrics).
summary.csv has one row per record: the number of interactions and the order as
From>To → ….
2. Codebase
repos/<language>/<repo>/— the 93 repositories (44 Java, 49 Python); five serve two patterns and one serves three. Only the.java/.pysource files are kept.cpg/— one Joern 4.0.627 code property graph per repository. PointJOERN_CPG_ROOThere to run Level 3 and stage 3 without rebuilding them. Java CPGs leave outtest/folders (thejavasrc2cpgdefault).repos.csv— one row per repository:language,repo,github(owner/name),url,commit,license,license_source(where the license was read: the license file, or apom.xml/pyproject.tomldeclaration),license_filesandrecords(the patterns it serves). Each repository is its GitHub repository atcommit.licenses/<language>/<repo>/— the license and NOTICE files of each repository atcommit, at their original paths.repos/keeps only source files, so these are copied here.
license is an SPDX identifier as GitHub reports it, read at commit. Two values
are not licenses: NONE — the repository has no license file or declaration at
that commit — and NOASSERTION — only some of its modules declare one.
3. Experiment outputs
<experiment> |
Setting |
|---|---|
pipeline |
the full recovery pipeline (secpat recover) |
baseline |
one-shot recovery: the whole repository packed into one prompt (secpat baseline) |
ablation_no_grep |
the pipeline with the grep tool disabled (keyword search) |
ablation_no_search |
the pipeline with the search tool disabled (semantic search) |
<model> is one of gpt-6.1-sol, deepseek-v4-flash-0731, glm-5.3-flash,
muse-spark-1.3-contributor.
Each <experiment>/<model>/<language>/ folder holds one subfolder per record
with its recovery.json: role_mappings and interaction_order, in the same
schema as the ground truth.
- Pipeline and ablations: Level 3 does not read the stored
interaction_order. It derives the recovery's order fromrole_mappingson the CPG, with the same walk as the ground truth's. The ablation runs, and some pipeline runs, skipped the order at recovery time, so theirs readsavailable: false. - Baseline:
interaction_orderis the order the model stated in its answer, and Level 3 scores that (--recovery-order stored).
evaluation.json is the batch's score, one entry per record with Levels 1–3:
the per-record scores behind the paper's results for RQ1, RQ3 and RQ4. Their
means are the reported RDA, MLS and IRC; for the pipeline over the four models,
0.84, 0.84 and 0.66. The paper's token counts, costs, steps and memory use (RQ2,
Figs. 38 and 39 of RQ3, and part of Table X of RQ4) are not in this dataset.
- Baseline: 16 records (10 Java, 6 Python, from 14 repositories) have a one-shot prompt over the
900K-token input limit (counted with the
o200k_basetokenizer). They were not run, have no folder, and score 0 on every level. - The other experiments ran all 50 records per language.
To score these recoveries again, see Experiment 1 in the secpatrec README. A
fresh score leaves the baseline's 16 unrun records unscored; evaluation.json
counts them as 0.
4. Security patterns
The five patterns, each as 01_text.md (the catalogue page), 02_diagram.md,
03_subflows.md and 04_pattern.json (the roles and the behaviour walkthrough
the recovery and the interaction walk read):
authentication_pwd— password-based authenticationverifiable_token_based_authenticationobscure_token_based_access_controlsession_based_access_controllimit_request_rate
5. Recovery conversation session
Recovery_Conversation_Session.txt is the exported Claude Code session in which
the ground truth's role mappings were first drafted (the paper's Appendix D-A). A planner agent writes one
language-independent recovery plan, tries it on one or two repositories, then
hands the plan, the pattern and the repository to one sub-agent per repository.
The session runs Java first and then Python, in batches of up to 20 repositories
per pattern, so it covers more repositories than the 100 records kept here.
License
- This dataset's own content — the ground truth, the manifests, the experiment
outputs and the recovery conversation session — is released under the MIT
License (see
LICENSE). security_patterns/is derived from the security pattern catalogue of Van Den Berghe et al., © 2021-2022 imec-DistriNet (https://securitypatterns.distrinet-research.be/), licensed under CC BY-NC-SA 4.0:01_text.mdis the catalogue page, and02_diagram.md,03_subflows.mdand04_pattern.jsonare adaptations of it. It is shared under the same license.codebase/holds third-party code. Each repository keeps its own license, listed incodebase/repos.csv, with its license files incodebase/licenses/.
Xet Storage Details
- Size:
- 11 kB
- Xet hash:
- 2fc8d8ea843e7a650681db5317c6558d396254d40e925533568850f5efdfb302
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.