Buckets:

|
download
raw
11 kB
# dataset
100 records — 5 security patterns × 10 repositories × 2 languages (Java, Python) —
with their ground truth, the codebases, the outputs of every experiment, and the
patterns themselves.
```
dataset/
├── manifest_java_full_20261005.json the 50 Java records
├── manifest_python_full_20261005.json the 50 Python records
├── ground_truth/ 1. role mapping + interactions, per record
│ ├── java/<pattern>__<repo>.json
│ ├── python/<pattern>__<repo>.json
│ └── summary.csv
├── codebase/ 2. the code each record is recovered from
│ ├── repos/<language>/<repo>/
│ ├── cpg/<repo>__<JAVASRC|PYTHONSRC>.cpg
│ ├── repos.csv provenance: GitHub URL, commit, license
│ └── licenses/<language>/<repo>/ each repo's license and NOTICE files
├── experiment_outputs/ 3. 4 experiments × 4 models × 2 languages
│ └── <experiment>/<model>/<language>/
│ ├── evaluation.json
│ └── <pattern>__<repo>/recovery.json
├── security_patterns/ 4. the 5 patterns
│ └── <pattern>/{01_text.md, 02_diagram.md, 03_subflows.md, 04_pattern.json}
├── Recovery_Conversation_Session.txt 5. the session that drafted the role mappings
└── LICENSE MIT, for this dataset's own content (see License)
```
## The paper
This is the dataset of the paper *A Hybrid Approach for Automated Security
Pattern Recovery*: the groundtruth of its Section IV-B, the codebases and CPGs
the recovery runs on, and the outputs and scores of every experiment of its
Section V. The code that produced and scores them is
[secpatrec](https://github.com/anonymousspottedtoad/secpatrec), which installs the
interaction walk and the metrics. The paper's appendix is at
<https://doi.org/10.5281/zenodo.23664769>; its Appendix G lists all the artifacts
and shows how the reported numbers are recomputed from this dataset.
| Paper | Here |
| ----- | ---- |
| recovery instance (100 of them) | record: one pattern in one repository, listed in the manifests |
| the specification *P* of P1 to P5 | `security_patterns/<pattern>/04_pattern.json`: P1 `authentication_pwd`, P2 `verifiable_token_based_authentication`, P3 `obscure_token_based_access_control`, P4 `session_based_access_control`, P5 `limit_request_rate` |
| the groundtruth mapping *M*, statuses *D* and interaction sequence *S* | `role_mappings` (`entity`, `status`) and `interaction_order` in `ground_truth/` |
| RDA, MLS, IRC | Levels 1, 2 and 3 in `evaluation.json` |
| GPT-6.1, Muse Spark, DeepSeek, GLM | the `<model>` folders `gpt-6.1-sol`, `muse-spark-1.3-contributor`, `deepseek-v4-flash-0731`, `glm-5.3-flash` |
| SecPatRec, the one-shot baseline, and RQ4's settings without lexical or semantic search | the `<experiment>` folders `pipeline`, `baseline`, `ablation_no_grep`, `ablation_no_search` |
| the Claude Code session of Appendix D-A | `Recovery_Conversation_Session.txt` |
## Manifests
Each manifest is `{"runs": [...]}`, one entry per record: `pattern`, `repo` and
`language`, plus the codebase's provenance — `url` (its GitHub repository),
`commit` (the commit it was cloned at) and `license` (see
[2. Codebase](#2-codebase)). The rest of a record follows from `pattern`, `repo`
and `language`:
| | |
| - | - |
| codebase | `codebase/repos/<language>/<repo>/` |
| CPG | `codebase/cpg/<repo>__<JAVASRC\|PYTHONSRC>.cpg` |
| pattern | `security_patterns/<pattern>/04_pattern.json` |
| ground truth | `ground_truth/<language>/<pattern>__<repo>.json` |
The pipeline (`secpatrec`) runs the experiments on this dataset with
`secpat evaluate`, `secpat interaction` and `secpat recover --models`, each part
of the dataset named by its own flag. Its README covers installing, downloading
this dataset and running.
## 1. Ground truth
One JSON file per record, named `<pattern>__<repo>.json`, with two fields:
| Field | Holds |
| ----- | ----- |
| `role_mappings` | one row per role the annotators recorded (see below): `patternRole`, `status` (`present`, `external`, `missing`), `entity`, `file`, `lines`, `notes` — the same schema as a recovery's `recovery.json` |
| `interaction_order` | the order the roles interact in: `entryRole`, `entryRule`, and `interactions` ranked from the entry role, each with `from`, `to`, the call site (`file`, `line`), its call `path` and the `mechanism` that reached it — again the same schema as `recovery.json` |
The 100 files hold 758 rows: 497 for roles the pattern requires, 200 for Subject
and System (Entity and System in `limit_request_rate`), which every file records,
and 61 for roles of the patterns' other flows that `04_pattern.json` does not list,
such as the Registrar of P1. The paper's Table III counts only the roles that
`04_pattern.json` requires, 500 in all, and a required role with no row counts as
`missing`, which is the case for the Pepper store in three P1 records. Counted this
way, the files give 334 present, 75 external and 91 missing roles, and 280
interactions. The metrics read the files the same way.
`interaction_order` is derived from `role_mappings` by walking the repo's CPG with
Joern 4.0.627, using the walk scripts in `secpat_interaction_recovery`
(`walk.sc` a81b989f…, `java.sc` eb906608…, `python.sc` b7038c5f…). All 100 records
have an order. The annotators checked it in the annotation interface (the paper's
Appendix D-B), which derives the order again from the mapping whenever they change
a mapping and saves the order derived from the saved mapping. So every stored
interaction is the walk's output, and its `mechanism` says how the walk reached it.
Level 3 does not read this order: it walks the groundtruth mapping again over the
roles both sides detected (see `secpat_evaluation_metrics`).
`summary.csv` has one row per record: the number of interactions and the order as
`From>To → …`.
## 2. Codebase
- `repos/<language>/<repo>/` — the 93 repositories (44 Java, 49 Python); five
serve two patterns and one serves three. Only the `.java` / `.py` source files
are kept.
- `cpg/` — one Joern 4.0.627 code property graph per repository. Point
`JOERN_CPG_ROOT` here to run Level 3 and stage 3 without rebuilding them. Java
CPGs leave out `test/` folders (the `javasrc2cpg` default).
- `repos.csv` — one row per repository: `language`, `repo`, `github`
(`owner/name`), `url`, `commit`, `license`, `license_source` (where the license
was read: the license file, or a `pom.xml` / `pyproject.toml` declaration),
`license_files` and `records` (the patterns it serves). Each repository is its
GitHub repository at `commit`.
- `licenses/<language>/<repo>/` — the license and NOTICE files of each repository
at `commit`, at their original paths. `repos/` keeps only source files, so
these are copied here.
`license` is an SPDX identifier as GitHub reports it, read at `commit`. Two values
are not licenses: `NONE` — the repository has no license file or declaration at
that commit — and `NOASSERTION` — only some of its modules declare one.
## 3. Experiment outputs
| `<experiment>` | Setting |
| -------------- | ------- |
| `pipeline` | the full recovery pipeline (`secpat recover`) |
| `baseline` | one-shot recovery: the whole repository packed into one prompt (`secpat baseline`) |
| `ablation_no_grep` | the pipeline with the `grep` tool disabled (keyword search) |
| `ablation_no_search` | the pipeline with the `search` tool disabled (semantic search) |
`<model>` is one of `gpt-6.1-sol`, `deepseek-v4-flash-0731`, `glm-5.3-flash`,
`muse-spark-1.3-contributor`.
Each `<experiment>/<model>/<language>/` folder holds one subfolder per record
with its `recovery.json`: `role_mappings` and `interaction_order`, in the same
schema as the ground truth.
- **Pipeline and ablations:** Level 3 does not read the stored
`interaction_order`. It derives the recovery's order from `role_mappings` on
the CPG, with the same walk as the ground truth's. The ablation runs, and some
pipeline runs, skipped the order at recovery time, so theirs reads
`available: false`.
- **Baseline:** `interaction_order` is the order the model stated in its answer,
and Level 3 scores that (`--recovery-order stored`).
`evaluation.json` is the batch's score, one entry per record with Levels 1–3:
the per-record scores behind the paper's results for RQ1, RQ3 and RQ4. Their
means are the reported RDA, MLS and IRC; for the pipeline over the four models,
0.84, 0.84 and 0.66. The paper's token counts, costs, steps and memory use (RQ2,
Figs. 38 and 39 of RQ3, and part of Table X of RQ4) are not in this dataset.
- **Baseline:** 16 records (10 Java, 6 Python, from 14 repositories) have a one-shot prompt over the
900K-token input limit (counted with the `o200k_base` tokenizer). They were not
run, have no folder, and score 0 on every level.
- The other experiments ran all 50 records per language.
To score these recoveries again, see Experiment 1 in the `secpatrec` README. A
fresh score leaves the baseline's 16 unrun records unscored; `evaluation.json`
counts them as 0.
## 4. Security patterns
The five patterns, each as `01_text.md` (the catalogue page), `02_diagram.md`,
`03_subflows.md` and `04_pattern.json` (the roles and the behaviour walkthrough
the recovery and the interaction walk read):
- `authentication_pwd` — password-based authentication
- `verifiable_token_based_authentication`
- `obscure_token_based_access_control`
- `session_based_access_control`
- `limit_request_rate`
## 5. Recovery conversation session
`Recovery_Conversation_Session.txt` is the exported Claude Code session in which
the ground truth's role mappings were first drafted (the paper's Appendix D-A). A planner agent writes one
language-independent recovery plan, tries it on one or two repositories, then
hands the plan, the pattern and the repository to one sub-agent per repository.
The session runs Java first and then Python, in batches of up to 20 repositories
per pattern, so it covers more repositories than the 100 records kept here.
## License
- **This dataset's own content** — the ground truth, the manifests, the experiment
outputs and the recovery conversation session — is released under the MIT
License (see `LICENSE`).
- **`security_patterns/`** is derived from the security pattern catalogue of
Van Den Berghe et al., © 2021-2022 imec-DistriNet
(<https://securitypatterns.distrinet-research.be/>), licensed under
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/):
`01_text.md` is the catalogue page, and `02_diagram.md`, `03_subflows.md` and
`04_pattern.json` are adaptations of it. It is shared under the same license.
- **`codebase/`** holds third-party code. Each repository keeps its own license,
listed in `codebase/repos.csv`, with its license files in `codebase/licenses/`.

Xet Storage Details

Size:
11 kB
·
Xet hash:
2fc8d8ea843e7a650681db5317c6558d396254d40e925533568850f5efdfb302

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.