Buckets:

|
download
raw
15.6 kB

human_verified_bucket

100 records — 5 security patterns × 10 repositories × 2 languages (Java, Python) — with their human-verified role mappings and interaction sequences, the codebases and their CPGs, the outputs of the one-shot baseline, and the patterns themselves.

human_verified_bucket/
├── manifest.json ································ 100 records: URL, commit, license
├── codebase/
│   ├── <language>/<repo>/ ········································· 93 repositories
│   ├── repos.csv
│   └── licenses/<language>/<repo>/
├── cpg/ ··························································· 93 CPGs, 818 MB
│   └── <repo>__<JAVASRC|PYTHONSRC>.cpg
├── human_annotated/
│   ├── mappings/ ············································· 100 pairs, 500 roles
│   │   └── <language>/<pattern>__<repo>.json
│   ├── interactions/ ······························ 100 sequences, 280 interactions
│   │   └── <language>/<pattern>__<repo>.json
│   └── summary.csv
├── security_patterns/<pattern>/ ········································ 5 patterns
│   ├── 01_text.md   02_diagram.md   03_subflows.md
│   └── 04_pattern.json
├── experiment_outputs/<model>/<language>/ ·································· 4 LLMs
├── scripts/score.py ············································ recomputes Table I
└── figures/usage-overview.png ················· the figure in 3. Experiment outputs

Field names. This bucket renames four fields of the original release (securityPatternsRecovery) to the terms of the SecPat paper: patternRole → role, entity → symbol, interaction_order → interaction_sequence, and an interaction's rank → order. lines keeps its name, since some values are line ranges. The secpatrec tools and the scorer still read the original names, so scripts/score.py renames the fields back when it passes the records to the scorer, and evaluation.json, the scorer's own output, keeps the original names.

The papers

This bucket contains the human-verified records of SecPat, the dataset of the paper SecPat: A Dataset of Security Pattern Implementations in Open-Source Web Applications. Two annotators verified these records (Section II-D of that paper), and the paper benchmarks four LLMs on them with the one-shot baseline (Section III), whose outputs and scores are in experiment_outputs/.

The same records are the groundtruth of the paper A Hybrid Approach for Automated Security Pattern Recovery (Section IV-B), whose appendix is at https://doi.org/10.5281/zenodo.23664769. Its Appendix G lists all the artifacts and shows how the reported numbers are recomputed from this dataset. The code that produced and scores the records is secpatrec, which installs the interaction traversal and the metrics.

Papers Here
record, or recovery instance in the technical paper (100 of them) one pattern in one repository, listed in manifest.json
the specification P of P1 to P5 security_patterns/<pattern>/04_pattern.json: P1 authentication_pwd, P2 verifiable_token_based_authentication, P3 obscure_token_based_access_control, P4 session_based_access_control, P5 limit_request_rate
the role mapping and interaction sequence of a record (M, D and S in the technical paper) role_mappings (symbol, status) in human_annotated/mappings/ and interaction_sequence in human_annotated/interactions/
RDA, MLS, IRC Levels 1, 2 and 3 in evaluation.json
GPT-6.1, Muse Spark, DeepSeek, GLM the <model> folders gpt-6.1-sol, muse-spark-1.3-contributor, deepseek-v4-flash-0731, glm-5.3-flash
the one-shot baseline experiment_outputs/ (the pipeline and ablation runs stay in the original release)

Recomputing Table I

Table I of the SecPat paper reports the mean RDA, MLS and IRC of four LLMs over the 100 records of this bucket. You can recompute it with our scoring tool, secpat_evaluation_metrics, which compares each LLM answer in experiment_outputs/ with the human-annotated record of the same repository and pattern. It needs only Python 3.11 or newer, and the scoring takes about 30 seconds once the bucket is downloaded.

1. Install the scoring tool in a virtual environment. pip installs it from its GitHub release tag, together with secpat_interaction_recovery, whose interaction types the scorer uses:

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip huggingface_hub
pip install \
  "secpat-interaction-order @ git+https://github.com/anonymousspottedtoad/secpat_interaction_recovery.git@v0.1.0" \
  "secpat-scoring @ git+https://github.com/anonymousspottedtoad/secpat_evaluation_metrics.git@v0.23.0"

2. Download this bucket without cpg/, since the scoring does not read the CPGs. The download contains about 44,000 small source files (234 MB) and takes about 15 minutes:

hf buckets sync hf://buckets/AnonymousSpottedToad/human_verified_bucket human_verified_bucket --exclude "cpg/*"

3. Score the four LLMs with scripts/score.py:

python human_verified_bucket/scripts/score.py human_verified_bucket

For IRC, the script compares the interaction sequence that the LLM returned with the human-verified sequence in human_annotated/interactions/, both narrowed to the roles that the LLM and the annotators both detected. The 16 records whose one-shot prompt exceeded the input limit have no answer and count as 0, as in the paper. The script prints the means over the 100 records, which are the numbers of Table I, and compares every record with the bucket's own evaluation.json in its last column:

Model                   RDA    MLS    IRC  records  not run  same as evaluation.json
GPT-6.1               0.740  0.704  0.511      100       16                  100/100
Muse Spark            0.715  0.691  0.509      100       16                  100/100
DeepSeek-V4-Flash     0.672  0.646  0.515      100       16                  100/100
GLM-5.3-Flash         0.705  0.688  0.510      100       16                  100/100

scores.json then contains the three scores of every record and model, and --models gpt-6.1-sol scores a single LLM.

Manifest

manifest.json is {"runs": [...]}, one entry per record: pattern, repo and language, plus the codebase's provenance — url (its GitHub repository), commit (the commit it was cloned at) and license (see 2. Codebase). The rest of a record follows from pattern, repo and language:

codebase codebase/<language>/<repo>/
CPG cpg/<repo>__<JAVASRC|PYTHONSRC>.cpg
pattern security_patterns/<pattern>/04_pattern.json
role mapping human_annotated/mappings/<language>/<pattern>__<repo>.json
interaction sequence human_annotated/interactions/<language>/<pattern>__<repo>.json

The pipeline (secpatrec) runs the experiments on this dataset with secpat evaluate, secpat interaction and secpat recover --models, each part of the dataset named by its own flag. Its README covers installing, downloading this dataset and running.

How the records were drafted

Before the human verification, Claude Code (Opus 5.5) drafted a role mapping for each sampled (repository, pattern) pair. It received the catalogue text and diagram of each pattern (01_text.md and 02_diagram.md in security_patterns/<pattern>/) and spawned one subagent per pair. The subagent marked each role of the pattern as present, external or missing and located the code symbol that represents it, as the following pseudocode shows.

Input: R[l][p], the sampled codebases of language l paired with pattern p

for each language l in {Python, Java}:
    for each pattern p in {P1, ..., P5}:
        for each codebase r in R[l][p]:
            spawn a subagent to analyse the code of r
            for each role ρ of p:                         # inside the subagent
                s[ρ] ← present, external or missing
                a[ρ] ← location of ρ in r and its AST node

Our traversal rules then derived the interaction sequence of each of the 502 drafts from the CPG of its repository. The traversal found at least one interaction for 281 drafts, and we sampled the 100 records of this bucket from them, 10 per pattern and language, for two annotators to verify (see the next section).

1. Human-annotated records

Two JSON files per record, both named <pattern>__<repo>.json:

File Contains
mappings/<language>/ role_mappings: one row per role the annotators recorded (see below): role, status (present, external, missing), symbol, file, lines, notes — the same schema as a recovery's recovery.json
interactions/<language>/ interaction_sequence: the order the roles interact in: entryRole, entryRule, and interactions numbered by order from the entry role, each with from, to, the call site (file, line), its call path and the mechanism that reached it — again the same schema as recovery.json

The 100 files contain 758 rows: 497 for roles the pattern requires, 200 for Subject and System (Entity and System in limit_request_rate), which every file records, and 61 for roles of the patterns' other flows that 04_pattern.json does not list, such as the Registrar of P1. Both papers (Section II-D of the SecPat paper, Table III of the technical paper) count only the roles that 04_pattern.json requires, 500 in all, and a required role with no row counts as missing, which is the case for the Pepper store in three P1 records. Counted this way, the files contain 334 present, 75 external and 91 missing roles, and 280 interactions. The metrics read the files the same way.

interaction_sequence is derived from role_mappings by traversing the repository's CPG with Joern 4.0.627, using the traversal scripts in secpat_interaction_recovery (walk.sc a81b989f…, java.sc eb906608…, python.sc b7038c5f…). All 100 records have an order. The annotators checked it in our annotation tool (Section II-D of the SecPat paper), which traverses the CPG again whenever they change a mapping and saves the order derived from the saved mapping. So every stored interaction is the traversal's output, and its mechanism says how the traversal reached it. To compute IRC, scripts/score.py compares this sequence with the LLM's, both narrowed to the roles that both sides detected. We computed the evaluation.json files in experiment_outputs/ by traversing the CPG again from the ground-truth mapping over the same roles instead, and both ways produce the same score for every record.

human_annotated/summary.csv has one row per record: the number of interactions and the order as From>To → ….

2. Codebase

  • codebase/<language>/<repo>/ — the 93 repositories (44 Java, 49 Python); five serve two patterns and one serves three. Only the .java / .py source files are kept.
  • cpg/ — one Joern 4.0.627 code property graph per repository. Point JOERN_CPG_ROOT here to run Level 3 and stage 3 without rebuilding them. Java CPGs leave out test/ folders (the javasrc2cpg default).
  • codebase/repos.csv — one row per repository: language, repo, github (owner/name), url, commit, license, license_source (where the license was read: the license file, or a pom.xml / pyproject.toml declaration), license_files and records (the patterns it serves). Each repository is its GitHub repository at commit.
  • codebase/licenses/<language>/<repo>/ — the license and NOTICE files of each repository at commit, at their original paths. codebase/<language>/ keeps only source files, so these are copied here.

license is an SPDX identifier as GitHub reports it, read at commit. Two values are not licenses: NONE — the repository has no license file or declaration at that commit — and NOASSERTION — only some of its modules declare one.

3. Experiment outputs

The one-shot baseline, which puts the whole repository into one prompt (secpat baseline). The pipeline and ablation runs are in the original release, securityPatternsRecovery.

Benchmarking an LLM on human_verified_bucket with the one-shot baseline

Benchmarking an LLM on human_verified_bucket with the one-shot baseline. Each LLM receives the codebase and the pattern of a record in one prompt (1, 2) and answers with a role mapping and an interaction sequence (3), which the scoring tool compares with the ground truth by RDA, MLS and IRC (4). In this example on P5 Rate limiting in loadcoder, the answer marks the History provider as external, which lowers RDA to 0.75.

<model> is one of gpt-6.1-sol, deepseek-v4-flash-0731, glm-5.3-flash, muse-spark-1.3-contributor.

Each <model>/<language>/ folder contains one subfolder per record with its recovery.json: role_mappings and interaction_sequence, in the same schema as the human-annotated records. interaction_sequence is the order the model returned in its answer, and Level 3 scores that (--recovery-order stored).

evaluation.json is the batch's score, one entry per record with Levels 1–3. Their means are the reported RDA, MLS and IRC. 16 records (10 Java, 6 Python, from 14 repositories) have a one-shot prompt over the 900K-token input limit (counted with the o200k_base tokenizer). They were not run, have no folder, and score 0 on every level.

To score these answers again, follow Recomputing Table I.

4. Security patterns

The five patterns, each as 01_text.md (the catalogue page), 02_diagram.md, 03_subflows.md and 04_pattern.json (the roles and the expected behaviour that the recovery and the interaction traversal read):

  • authentication_pwd — password-based authentication
  • verifiable_token_based_authentication
  • obscure_token_based_access_control
  • session_based_access_control
  • limit_request_rate

License

  • This dataset's own content — the human-annotated records, the manifest and the experiment outputs — is released under the MIT License (see LICENSE).
  • security_patterns/ is derived from the security pattern catalogue of Van Den Berghe et al., © 2021-2022 imec-DistriNet (https://securitypatterns.distrinet-research.be/), licensed under CC BY-NC-SA 4.0: 01_text.md is the catalogue page, and 02_diagram.md, 03_subflows.md and 04_pattern.json are adaptations of it. It is shared under the same license.
  • codebase/ and cpg/ contain third-party code and graphs built from it. Each repository keeps its own license, listed in codebase/repos.csv, with its license files in codebase/licenses/.

Xet Storage Details

Size:
15.6 kB
·
Xet hash:
7a6381d1d32a952643afb92f321111eb0cc3d72e57323759e463cc89d6b292f8

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.