Buckets:
human_verified_bucket
100 records — 5 security patterns × 10 repositories × 2 languages (Java, Python) — with their human-verified role mappings and interaction sequences, the codebases and their CPGs, the outputs of the one-shot baseline, and the patterns themselves.
human_verified_bucket/
├── manifest.json ································ 100 records: URL, commit, license
├── codebase/
│ ├── <language>/<repo>/ ········································· 93 repositories
│ ├── repos.csv
│ └── licenses/<language>/<repo>/
├── cpg/ ··························································· 93 CPGs, 818 MB
│ └── <repo>__<JAVASRC|PYTHONSRC>.cpg
├── human_annotated/
│ ├── mappings/ ············································· 100 pairs, 500 roles
│ │ └── <language>/<pattern>__<repo>.json
│ ├── interactions/ ······························ 100 sequences, 280 interactions
│ │ └── <language>/<pattern>__<repo>.json
│ └── summary.csv
├── security_patterns/<pattern>/ ········································ 5 patterns
│ ├── 01_text.md 02_diagram.md 03_subflows.md
│ └── 04_pattern.json
├── experiment_outputs/<model>/<language>/ ·································· 4 LLMs
├── scripts/score.py ············································ recomputes Table I
└── figures/usage-overview.png ················· the figure in 3. Experiment outputs
Field names. This bucket renames four fields of the original release
(securityPatternsRecovery) to the terms of the SecPat paper: patternRole →
role, entity → symbol, interaction_order → interaction_sequence, and an
interaction's rank → order. lines keeps its name, since some values are line
ranges. The secpatrec tools and the scorer still read the original names, so
scripts/score.py renames the fields back when it passes the records to the
scorer, and evaluation.json, the scorer's own output, keeps the original names.
The papers
This bucket contains the human-verified records of SecPat, the dataset of the paper
SecPat: A Dataset of Security Pattern Implementations in Open-Source Web
Applications. Two annotators verified these records (Section II-D of that paper),
and the paper benchmarks four LLMs on them with the one-shot baseline (Section III),
whose outputs and scores are in experiment_outputs/.
The same records are the groundtruth of the paper A Hybrid Approach for Automated Security Pattern Recovery (Section IV-B), whose appendix is at https://doi.org/10.5281/zenodo.23664769. Its Appendix G lists all the artifacts and shows how the reported numbers are recomputed from this dataset. The code that produced and scores the records is secpatrec, which installs the interaction traversal and the metrics.
| Papers | Here |
|---|---|
| record, or recovery instance in the technical paper (100 of them) | one pattern in one repository, listed in manifest.json |
| the specification P of P1 to P5 | security_patterns/<pattern>/04_pattern.json: P1 authentication_pwd, P2 verifiable_token_based_authentication, P3 obscure_token_based_access_control, P4 session_based_access_control, P5 limit_request_rate |
| the role mapping and interaction sequence of a record (M, D and S in the technical paper) | role_mappings (symbol, status) in human_annotated/mappings/ and interaction_sequence in human_annotated/interactions/ |
| RDA, MLS, IRC | Levels 1, 2 and 3 in evaluation.json |
| GPT-6.1, Muse Spark, DeepSeek, GLM | the <model> folders gpt-6.1-sol, muse-spark-1.3-contributor, deepseek-v4-flash-0731, glm-5.3-flash |
| the one-shot baseline | experiment_outputs/ (the pipeline and ablation runs stay in the original release) |
Recomputing Table I
Table I of the SecPat paper reports the mean RDA, MLS and IRC of four LLMs over the
100 records of this bucket. You can recompute it with our scoring tool,
secpat_evaluation_metrics,
which compares each LLM answer in experiment_outputs/ with the human-annotated
record of the same repository and pattern. It needs only Python 3.11 or newer, and
the scoring takes about 30 seconds once the bucket is downloaded.
1. Install the scoring tool in a virtual environment. pip installs it from its GitHub release tag, together with secpat_interaction_recovery, whose interaction types the scorer uses:
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip huggingface_hub
pip install \
"secpat-interaction-order @ git+https://github.com/anonymousspottedtoad/secpat_interaction_recovery.git@v0.1.0" \
"secpat-scoring @ git+https://github.com/anonymousspottedtoad/secpat_evaluation_metrics.git@v0.23.0"
2. Download this bucket without cpg/, since the scoring does not read the
CPGs. The download contains about 44,000 small source files (234 MB) and takes
about 15 minutes:
hf buckets sync hf://buckets/AnonymousSpottedToad/human_verified_bucket human_verified_bucket --exclude "cpg/*"
3. Score the four LLMs with scripts/score.py:
python human_verified_bucket/scripts/score.py human_verified_bucket
For IRC, the script compares the interaction sequence that the LLM returned with the
human-verified sequence in human_annotated/interactions/, both narrowed to the
roles that the LLM and the annotators both detected. The 16 records whose one-shot
prompt exceeded the input limit have no answer and count as 0, as in the paper. The
script prints the means over the 100 records, which are the numbers of Table I, and
compares every record with the bucket's own evaluation.json in its last column:
Model RDA MLS IRC records not run same as evaluation.json
GPT-6.1 0.740 0.704 0.511 100 16 100/100
Muse Spark 0.715 0.691 0.509 100 16 100/100
DeepSeek-V4-Flash 0.672 0.646 0.515 100 16 100/100
GLM-5.3-Flash 0.705 0.688 0.510 100 16 100/100
scores.json then contains the three scores of every record and model, and
--models gpt-6.1-sol scores a single LLM.
Manifest
manifest.json is {"runs": [...]}, one entry per record: pattern, repo and
language, plus the codebase's provenance — url (its GitHub repository),
commit (the commit it was cloned at) and license (see
2. Codebase). The rest of a record follows from pattern, repo
and language:
| codebase | codebase/<language>/<repo>/ |
| CPG | cpg/<repo>__<JAVASRC|PYTHONSRC>.cpg |
| pattern | security_patterns/<pattern>/04_pattern.json |
| role mapping | human_annotated/mappings/<language>/<pattern>__<repo>.json |
| interaction sequence | human_annotated/interactions/<language>/<pattern>__<repo>.json |
The pipeline (secpatrec) runs the experiments on this dataset with
secpat evaluate, secpat interaction and secpat recover --models, each part
of the dataset named by its own flag. Its README covers installing, downloading
this dataset and running.
How the records were drafted
Before the human verification, Claude Code (Opus 5.5) drafted a role mapping for
each sampled (repository, pattern) pair. It received the catalogue text and diagram
of each pattern (01_text.md and 02_diagram.md in security_patterns/<pattern>/)
and spawned one subagent per pair. The subagent marked each role of the pattern as
present, external or missing and located the code symbol that represents it,
as the following pseudocode shows.
Input: R[l][p], the sampled codebases of language l paired with pattern p
for each language l in {Python, Java}:
for each pattern p in {P1, ..., P5}:
for each codebase r in R[l][p]:
spawn a subagent to analyse the code of r
for each role ρ of p: # inside the subagent
s[ρ] ← present, external or missing
a[ρ] ← location of ρ in r and its AST node
Our traversal rules then derived the interaction sequence of each of the 502 drafts from the CPG of its repository. The traversal found at least one interaction for 281 drafts, and we sampled the 100 records of this bucket from them, 10 per pattern and language, for two annotators to verify (see the next section).
1. Human-annotated records
Two JSON files per record, both named <pattern>__<repo>.json:
| File | Contains |
|---|---|
mappings/<language>/ |
role_mappings: one row per role the annotators recorded (see below): role, status (present, external, missing), symbol, file, lines, notes — the same schema as a recovery's recovery.json |
interactions/<language>/ |
interaction_sequence: the order the roles interact in: entryRole, entryRule, and interactions numbered by order from the entry role, each with from, to, the call site (file, line), its call path and the mechanism that reached it — again the same schema as recovery.json |
The 100 files contain 758 rows: 497 for roles the pattern requires, 200 for Subject
and System (Entity and System in limit_request_rate), which every file records,
and 61 for roles of the patterns' other flows that 04_pattern.json does not list,
such as the Registrar of P1. Both papers (Section II-D of the SecPat paper, Table III
of the technical paper) count only the roles that 04_pattern.json requires, 500 in all, and a required role with no row counts as
missing, which is the case for the Pepper store in three P1 records. Counted this
way, the files contain 334 present, 75 external and 91 missing roles, and 280
interactions. The metrics read the files the same way.
interaction_sequence is derived from role_mappings by traversing the repository's
CPG with Joern 4.0.627, using the traversal scripts in secpat_interaction_recovery
(walk.sc a81b989f…, java.sc eb906608…, python.sc b7038c5f…). All 100 records
have an order. The annotators checked it in our annotation tool (Section II-D of the
SecPat paper), which traverses the CPG again whenever they change a mapping and
saves the order derived from the saved mapping. So every stored interaction is the
traversal's output, and its mechanism says how the traversal reached it. To
compute IRC, scripts/score.py compares this sequence with the LLM's, both narrowed
to the roles that both sides detected. We computed the evaluation.json files in
experiment_outputs/ by traversing the CPG again from the ground-truth mapping over
the same roles instead, and both ways produce the same score for every record.
human_annotated/summary.csv has one row per record: the number of interactions and the order as
From>To → ….
2. Codebase
codebase/<language>/<repo>/— the 93 repositories (44 Java, 49 Python); five serve two patterns and one serves three. Only the.java/.pysource files are kept.cpg/— one Joern 4.0.627 code property graph per repository. PointJOERN_CPG_ROOThere to run Level 3 and stage 3 without rebuilding them. Java CPGs leave outtest/folders (thejavasrc2cpgdefault).codebase/repos.csv— one row per repository:language,repo,github(owner/name),url,commit,license,license_source(where the license was read: the license file, or apom.xml/pyproject.tomldeclaration),license_filesandrecords(the patterns it serves). Each repository is its GitHub repository atcommit.codebase/licenses/<language>/<repo>/— the license and NOTICE files of each repository atcommit, at their original paths.codebase/<language>/keeps only source files, so these are copied here.
license is an SPDX identifier as GitHub reports it, read at commit. Two values
are not licenses: NONE — the repository has no license file or declaration at
that commit — and NOASSERTION — only some of its modules declare one.
3. Experiment outputs
The one-shot baseline, which puts the whole repository into one prompt
(secpat baseline). The pipeline and ablation runs are in the original release,
securityPatternsRecovery.
Benchmarking an LLM on human_verified_bucket with the one-shot baseline. Each
LLM receives the codebase and the pattern of a record in one prompt (1, 2) and
answers with a role mapping and an interaction sequence (3), which the scoring
tool compares with the ground truth by RDA, MLS and IRC (4). In this example on
P5 Rate limiting in loadcoder, the answer marks the History provider as
external, which lowers RDA to 0.75.
<model> is one of gpt-6.1-sol, deepseek-v4-flash-0731, glm-5.3-flash,
muse-spark-1.3-contributor.
Each <model>/<language>/ folder contains one subfolder per record with its
recovery.json: role_mappings and interaction_sequence, in the same schema as
the human-annotated records. interaction_sequence is the order the model returned
in its answer, and Level 3 scores that (--recovery-order stored).
evaluation.json is the batch's score, one entry per record with Levels 1–3.
Their means are the reported RDA, MLS and IRC. 16 records (10 Java, 6 Python,
from 14 repositories) have a one-shot prompt over the 900K-token input limit
(counted with the o200k_base tokenizer). They were not run, have no folder, and
score 0 on every level.
To score these answers again, follow Recomputing Table I.
4. Security patterns
The five patterns, each as 01_text.md (the catalogue page), 02_diagram.md,
03_subflows.md and 04_pattern.json (the roles and the expected behaviour that
the recovery and the interaction traversal read):
authentication_pwd— password-based authenticationverifiable_token_based_authenticationobscure_token_based_access_controlsession_based_access_controllimit_request_rate
License
- This dataset's own content — the human-annotated records, the manifest and
the experiment outputs — is released under the MIT License (see
LICENSE). security_patterns/is derived from the security pattern catalogue of Van Den Berghe et al., © 2021-2022 imec-DistriNet (https://securitypatterns.distrinet-research.be/), licensed under CC BY-NC-SA 4.0:01_text.mdis the catalogue page, and02_diagram.md,03_subflows.mdand04_pattern.jsonare adaptations of it. It is shared under the same license.codebase/andcpg/contain third-party code and graphs built from it. Each repository keeps its own license, listed incodebase/repos.csv, with its license files incodebase/licenses/.
Xet Storage Details
- Size:
- 15.6 kB
- Xet hash:
- 7a6381d1d32a952643afb92f321111eb0cc3d72e57323759e463cc89d6b292f8
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.
