Buckets:
| # human_verified_bucket | |
| 100 records — 5 security patterns × 10 repositories × 2 languages (Java, Python) — | |
| with their human-verified role mappings and interaction sequences, the codebases and | |
| their CPGs, the outputs of the one-shot baseline, and the patterns themselves. | |
| ``` | |
| human_verified_bucket/ | |
| ├── manifest.json ································ 100 records: URL, commit, license | |
| ├── codebase/ | |
| │ ├── <language>/<repo>/ ········································· 93 repositories | |
| │ ├── repos.csv | |
| │ └── licenses/<language>/<repo>/ | |
| ├── cpg/ ··························································· 93 CPGs, 818 MB | |
| │ └── <repo>__<JAVASRC|PYTHONSRC>.cpg | |
| ├── human_annotated/ | |
| │ ├── mappings/ ············································· 100 pairs, 500 roles | |
| │ │ └── <language>/<pattern>__<repo>.json | |
| │ ├── interactions/ ······························ 100 sequences, 280 interactions | |
| │ │ └── <language>/<pattern>__<repo>.json | |
| │ └── summary.csv | |
| ├── security_patterns/<pattern>/ ········································ 5 patterns | |
| │ ├── 01_text.md 02_diagram.md 03_subflows.md | |
| │ └── 04_pattern.json | |
| ├── experiment_outputs/<model>/<language>/ ·································· 4 LLMs | |
| ├── scripts/score.py ············································ recomputes Table I | |
| └── figures/usage-overview.png ················· the figure in 3. Experiment outputs | |
| ``` | |
| **Field names.** This bucket renames four fields of the original release | |
| (`securityPatternsRecovery`) to the terms of the SecPat paper: `patternRole` → | |
| `role`, `entity` → `symbol`, `interaction_order` → `interaction_sequence`, and an | |
| interaction's `rank` → `order`. `lines` keeps its name, since some values are line | |
| ranges. The `secpatrec` tools and the scorer still read the original names, so | |
| `scripts/score.py` renames the fields back when it passes the records to the | |
| scorer, and `evaluation.json`, the scorer's own output, keeps the original names. | |
| ## The papers | |
| This bucket contains the human-verified records of SecPat, the dataset of the paper | |
| *SecPat: A Dataset of Security Pattern Implementations in Open-Source Web | |
| Applications*. Two annotators verified these records (Section II-D of that paper), | |
| and the paper benchmarks four LLMs on them with the one-shot baseline (Section III), | |
| whose outputs and scores are in `experiment_outputs/`. | |
| The same records are the groundtruth of the paper *A Hybrid Approach for Automated | |
| Security Pattern Recovery* (Section IV-B), whose appendix is at | |
| <https://doi.org/10.5281/zenodo.23664769>. Its Appendix G lists all the artifacts | |
| and shows how the reported numbers are recomputed from this dataset. The code that | |
| produced and scores the records is | |
| [secpatrec](https://github.com/anonymousspottedtoad/secpatrec), which installs the | |
| interaction traversal and the metrics. | |
| | Papers | Here | | |
| | ----- | ---- | | |
| | record, or recovery instance in the technical paper (100 of them) | one pattern in one repository, listed in `manifest.json` | | |
| | the specification *P* of P1 to P5 | `security_patterns/<pattern>/04_pattern.json`: P1 `authentication_pwd`, P2 `verifiable_token_based_authentication`, P3 `obscure_token_based_access_control`, P4 `session_based_access_control`, P5 `limit_request_rate` | | |
| | the role mapping and interaction sequence of a record (*M*, *D* and *S* in the technical paper) | `role_mappings` (`symbol`, `status`) in `human_annotated/mappings/` and `interaction_sequence` in `human_annotated/interactions/` | | |
| | RDA, MLS, IRC | Levels 1, 2 and 3 in `evaluation.json` | | |
| | GPT-6.1, Muse Spark, DeepSeek, GLM | the `<model>` folders `gpt-6.1-sol`, `muse-spark-1.3-contributor`, `deepseek-v4-flash-0731`, `glm-5.3-flash` | | |
| | the one-shot baseline | `experiment_outputs/` (the pipeline and ablation runs stay in the original release) | | |
| ## Recomputing Table I | |
| Table I of the SecPat paper reports the mean RDA, MLS and IRC of four LLMs over the | |
| 100 records of this bucket. You can recompute it with our scoring tool, | |
| [secpat_evaluation_metrics](https://github.com/anonymousspottedtoad/secpat_evaluation_metrics), | |
| which compares each LLM answer in `experiment_outputs/` with the human-annotated | |
| record of the same repository and pattern. It needs only Python 3.11 or newer, and | |
| the scoring takes about 30 seconds once the bucket is downloaded. | |
| **1. Install the scoring tool** in a virtual environment. pip installs it from its | |
| GitHub release tag, together with | |
| [secpat_interaction_recovery](https://github.com/anonymousspottedtoad/secpat_interaction_recovery), | |
| whose interaction types the scorer uses: | |
| ```bash | |
| python3 -m venv .venv | |
| source .venv/bin/activate | |
| pip install --upgrade pip huggingface_hub | |
| pip install \ | |
| "secpat-interaction-order @ git+https://github.com/anonymousspottedtoad/secpat_interaction_recovery.git@v0.1.0" \ | |
| "secpat-scoring @ git+https://github.com/anonymousspottedtoad/secpat_evaluation_metrics.git@v0.23.0" | |
| ``` | |
| **2. Download this bucket** without `cpg/`, since the scoring does not read the | |
| CPGs. The download contains about 44,000 small source files (234 MB) and takes | |
| about 15 minutes: | |
| ```bash | |
| hf buckets sync hf://buckets/AnonymousSpottedToad/human_verified_bucket human_verified_bucket --exclude "cpg/*" | |
| ``` | |
| **3. Score the four LLMs** with `scripts/score.py`: | |
| ```bash | |
| python human_verified_bucket/scripts/score.py human_verified_bucket | |
| ``` | |
| For IRC, the script compares the interaction sequence that the LLM returned with the | |
| human-verified sequence in `human_annotated/interactions/`, both narrowed to the | |
| roles that the LLM and the annotators both detected. The 16 records whose one-shot | |
| prompt exceeded the input limit have no answer and count as 0, as in the paper. The | |
| script prints the means over the 100 records, which are the numbers of Table I, and | |
| compares every record with the bucket's own `evaluation.json` in its last column: | |
| ``` | |
| Model RDA MLS IRC records not run same as evaluation.json | |
| GPT-6.1 0.740 0.704 0.511 100 16 100/100 | |
| Muse Spark 0.715 0.691 0.509 100 16 100/100 | |
| DeepSeek-V4-Flash 0.672 0.646 0.515 100 16 100/100 | |
| GLM-5.3-Flash 0.705 0.688 0.510 100 16 100/100 | |
| ``` | |
| `scores.json` then contains the three scores of every record and model, and | |
| `--models gpt-6.1-sol` scores a single LLM. | |
| ## Manifest | |
| `manifest.json` is `{"runs": [...]}`, one entry per record: `pattern`, `repo` and | |
| `language`, plus the codebase's provenance — `url` (its GitHub repository), | |
| `commit` (the commit it was cloned at) and `license` (see | |
| [2. Codebase](#2-codebase)). The rest of a record follows from `pattern`, `repo` | |
| and `language`: | |
| | | | | |
| | - | - | | |
| | codebase | `codebase/<language>/<repo>/` | | |
| | CPG | `cpg/<repo>__<JAVASRC\|PYTHONSRC>.cpg` | | |
| | pattern | `security_patterns/<pattern>/04_pattern.json` | | |
| | role mapping | `human_annotated/mappings/<language>/<pattern>__<repo>.json` | | |
| | interaction sequence | `human_annotated/interactions/<language>/<pattern>__<repo>.json` | | |
| The pipeline (`secpatrec`) runs the experiments on this dataset with | |
| `secpat evaluate`, `secpat interaction` and `secpat recover --models`, each part | |
| of the dataset named by its own flag. Its README covers installing, downloading | |
| this dataset and running. | |
| ## How the records were drafted | |
| Before the human verification, Claude Code (Opus 5.5) drafted a role mapping for | |
| each sampled (repository, pattern) pair. It received the catalogue text and diagram | |
| of each pattern (`01_text.md` and `02_diagram.md` in `security_patterns/<pattern>/`) | |
| and spawned one subagent per pair. The subagent marked each role of the pattern as | |
| `present`, `external` or `missing` and located the code symbol that represents it, | |
| as the following pseudocode shows. | |
| ``` | |
| Input: R[l][p], the sampled codebases of language l paired with pattern p | |
| for each language l in {Python, Java}: | |
| for each pattern p in {P1, ..., P5}: | |
| for each codebase r in R[l][p]: | |
| spawn a subagent to analyse the code of r | |
| for each role ρ of p: # inside the subagent | |
| s[ρ] ← present, external or missing | |
| a[ρ] ← location of ρ in r and its AST node | |
| ``` | |
| Our traversal rules then derived the interaction sequence of each of the 502 drafts | |
| from the CPG of its repository. The traversal found at least one interaction for 281 | |
| drafts, and we sampled the 100 records of this bucket from them, 10 per pattern and | |
| language, for two annotators to verify (see the next section). | |
| ## 1. Human-annotated records | |
| Two JSON files per record, both named `<pattern>__<repo>.json`: | |
| | File | Contains | | |
| | ----- | ----- | | |
| | `mappings/<language>/` | `role_mappings`: one row per role the annotators recorded (see below): `role`, `status` (`present`, `external`, `missing`), `symbol`, `file`, `lines`, `notes` — the same schema as a recovery's `recovery.json` | | |
| | `interactions/<language>/` | `interaction_sequence`: the order the roles interact in: `entryRole`, `entryRule`, and `interactions` numbered by `order` from the entry role, each with `from`, `to`, the call site (`file`, `line`), its call `path` and the `mechanism` that reached it — again the same schema as `recovery.json` | | |
| The 100 files contain 758 rows: 497 for roles the pattern requires, 200 for Subject | |
| and System (Entity and System in `limit_request_rate`), which every file records, | |
| and 61 for roles of the patterns' other flows that `04_pattern.json` does not list, | |
| such as the Registrar of P1. Both papers (Section II-D of the SecPat paper, Table III | |
| of the technical paper) count only the roles that `04_pattern.json` requires, 500 in all, and a required role with no row counts as | |
| `missing`, which is the case for the Pepper store in three P1 records. Counted this | |
| way, the files contain 334 present, 75 external and 91 missing roles, and 280 | |
| interactions. The metrics read the files the same way. | |
| `interaction_sequence` is derived from `role_mappings` by traversing the repository's | |
| CPG with Joern 4.0.627, using the traversal scripts in `secpat_interaction_recovery` | |
| (`walk.sc` a81b989f…, `java.sc` eb906608…, `python.sc` b7038c5f…). All 100 records | |
| have an order. The annotators checked it in our annotation tool (Section II-D of the | |
| SecPat paper), which traverses the CPG again whenever they change a mapping and | |
| saves the order derived from the saved mapping. So every stored interaction is the | |
| traversal's output, and its `mechanism` says how the traversal reached it. To | |
| compute IRC, `scripts/score.py` compares this sequence with the LLM's, both narrowed | |
| to the roles that both sides detected. We computed the `evaluation.json` files in | |
| `experiment_outputs/` by traversing the CPG again from the ground-truth mapping over | |
| the same roles instead, and both ways produce the same score for every record. | |
| `human_annotated/summary.csv` has one row per record: the number of interactions and the order as | |
| `From>To → …`. | |
| ## 2. Codebase | |
| - `codebase/<language>/<repo>/` — the 93 repositories (44 Java, 49 Python); five | |
| serve two patterns and one serves three. Only the `.java` / `.py` source files | |
| are kept. | |
| - `cpg/` — one Joern 4.0.627 code property graph per repository. Point | |
| `JOERN_CPG_ROOT` here to run Level 3 and stage 3 without rebuilding them. Java | |
| CPGs leave out `test/` folders (the `javasrc2cpg` default). | |
| - `codebase/repos.csv` — one row per repository: `language`, `repo`, `github` | |
| (`owner/name`), `url`, `commit`, `license`, `license_source` (where the license | |
| was read: the license file, or a `pom.xml` / `pyproject.toml` declaration), | |
| `license_files` and `records` (the patterns it serves). Each repository is its | |
| GitHub repository at `commit`. | |
| - `codebase/licenses/<language>/<repo>/` — the license and NOTICE files of each repository | |
| at `commit`, at their original paths. `codebase/<language>/` keeps only source | |
| files, so these are copied here. | |
| `license` is an SPDX identifier as GitHub reports it, read at `commit`. Two values | |
| are not licenses: `NONE` — the repository has no license file or declaration at | |
| that commit — and `NOASSERTION` — only some of its modules declare one. | |
| ## 3. Experiment outputs | |
| The one-shot baseline, which puts the whole repository into one prompt | |
| (`secpat baseline`). The pipeline and ablation runs are in the original release, | |
| `securityPatternsRecovery`. | |
|  | |
| *Benchmarking an LLM on `human_verified_bucket` with the one-shot baseline. Each | |
| LLM receives the codebase and the pattern of a record in one prompt (1, 2) and | |
| answers with a role mapping and an interaction sequence (3), which the scoring | |
| tool compares with the ground truth by RDA, MLS and IRC (4). In this example on | |
| P5 Rate limiting in `loadcoder`, the answer marks the History provider as | |
| external, which lowers RDA to 0.75.* | |
| `<model>` is one of `gpt-6.1-sol`, `deepseek-v4-flash-0731`, `glm-5.3-flash`, | |
| `muse-spark-1.3-contributor`. | |
| Each `<model>/<language>/` folder contains one subfolder per record with its | |
| `recovery.json`: `role_mappings` and `interaction_sequence`, in the same schema as | |
| the human-annotated records. `interaction_sequence` is the order the model returned | |
| in its answer, and Level 3 scores that (`--recovery-order stored`). | |
| `evaluation.json` is the batch's score, one entry per record with Levels 1–3. | |
| Their means are the reported RDA, MLS and IRC. 16 records (10 Java, 6 Python, | |
| from 14 repositories) have a one-shot prompt over the 900K-token input limit | |
| (counted with the `o200k_base` tokenizer). They were not run, have no folder, and | |
| score 0 on every level. | |
| To score these answers again, follow [Recomputing Table I](#recomputing-table-i). | |
| ## 4. Security patterns | |
| The five patterns, each as `01_text.md` (the catalogue page), `02_diagram.md`, | |
| `03_subflows.md` and `04_pattern.json` (the roles and the expected behaviour that | |
| the recovery and the interaction traversal read): | |
| - `authentication_pwd` — password-based authentication | |
| - `verifiable_token_based_authentication` | |
| - `obscure_token_based_access_control` | |
| - `session_based_access_control` | |
| - `limit_request_rate` | |
| ## License | |
| - **This dataset's own content** — the human-annotated records, the manifest and | |
| the experiment outputs — is released under the MIT License (see `LICENSE`). | |
| - **`security_patterns/`** is derived from the security pattern catalogue of | |
| Van Den Berghe et al., © 2021-2022 imec-DistriNet | |
| (<https://securitypatterns.distrinet-research.be/>), licensed under | |
| [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/): | |
| `01_text.md` is the catalogue page, and `02_diagram.md`, `03_subflows.md` and | |
| `04_pattern.json` are adaptations of it. It is shared under the same license. | |
| - **`codebase/`** and **`cpg/`** contain third-party code and graphs built from it. | |
| Each repository keeps its own license, listed in `codebase/repos.csv`, with its | |
| license files in `codebase/licenses/`. | |
Xet Storage Details
- Size:
- 15.6 kB
- Xet hash:
- 7a6381d1d32a952643afb92f321111eb0cc3d72e57323759e463cc89d6b292f8
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.