Buckets:

|
download
raw
15.6 kB
# human_verified_bucket
100 records — 5 security patterns × 10 repositories × 2 languages (Java, Python) —
with their human-verified role mappings and interaction sequences, the codebases and
their CPGs, the outputs of the one-shot baseline, and the patterns themselves.
```
human_verified_bucket/
├── manifest.json ································ 100 records: URL, commit, license
├── codebase/
│ ├── <language>/<repo>/ ········································· 93 repositories
│ ├── repos.csv
│ └── licenses/<language>/<repo>/
├── cpg/ ··························································· 93 CPGs, 818 MB
│ └── <repo>__<JAVASRC|PYTHONSRC>.cpg
├── human_annotated/
│ ├── mappings/ ············································· 100 pairs, 500 roles
│ │ └── <language>/<pattern>__<repo>.json
│ ├── interactions/ ······························ 100 sequences, 280 interactions
│ │ └── <language>/<pattern>__<repo>.json
│ └── summary.csv
├── security_patterns/<pattern>/ ········································ 5 patterns
│ ├── 01_text.md 02_diagram.md 03_subflows.md
│ └── 04_pattern.json
├── experiment_outputs/<model>/<language>/ ·································· 4 LLMs
├── scripts/score.py ············································ recomputes Table I
└── figures/usage-overview.png ················· the figure in 3. Experiment outputs
```
**Field names.** This bucket renames four fields of the original release
(`securityPatternsRecovery`) to the terms of the SecPat paper: `patternRole` →
`role`, `entity` → `symbol`, `interaction_order` → `interaction_sequence`, and an
interaction's `rank` → `order`. `lines` keeps its name, since some values are line
ranges. The `secpatrec` tools and the scorer still read the original names, so
`scripts/score.py` renames the fields back when it passes the records to the
scorer, and `evaluation.json`, the scorer's own output, keeps the original names.
## The papers
This bucket contains the human-verified records of SecPat, the dataset of the paper
*SecPat: A Dataset of Security Pattern Implementations in Open-Source Web
Applications*. Two annotators verified these records (Section II-D of that paper),
and the paper benchmarks four LLMs on them with the one-shot baseline (Section III),
whose outputs and scores are in `experiment_outputs/`.
The same records are the groundtruth of the paper *A Hybrid Approach for Automated
Security Pattern Recovery* (Section IV-B), whose appendix is at
<https://doi.org/10.5281/zenodo.23664769>. Its Appendix G lists all the artifacts
and shows how the reported numbers are recomputed from this dataset. The code that
produced and scores the records is
[secpatrec](https://github.com/anonymousspottedtoad/secpatrec), which installs the
interaction traversal and the metrics.
| Papers | Here |
| ----- | ---- |
| record, or recovery instance in the technical paper (100 of them) | one pattern in one repository, listed in `manifest.json` |
| the specification *P* of P1 to P5 | `security_patterns/<pattern>/04_pattern.json`: P1 `authentication_pwd`, P2 `verifiable_token_based_authentication`, P3 `obscure_token_based_access_control`, P4 `session_based_access_control`, P5 `limit_request_rate` |
| the role mapping and interaction sequence of a record (*M*, *D* and *S* in the technical paper) | `role_mappings` (`symbol`, `status`) in `human_annotated/mappings/` and `interaction_sequence` in `human_annotated/interactions/` |
| RDA, MLS, IRC | Levels 1, 2 and 3 in `evaluation.json` |
| GPT-6.1, Muse Spark, DeepSeek, GLM | the `<model>` folders `gpt-6.1-sol`, `muse-spark-1.3-contributor`, `deepseek-v4-flash-0731`, `glm-5.3-flash` |
| the one-shot baseline | `experiment_outputs/` (the pipeline and ablation runs stay in the original release) |
## Recomputing Table I
Table I of the SecPat paper reports the mean RDA, MLS and IRC of four LLMs over the
100 records of this bucket. You can recompute it with our scoring tool,
[secpat_evaluation_metrics](https://github.com/anonymousspottedtoad/secpat_evaluation_metrics),
which compares each LLM answer in `experiment_outputs/` with the human-annotated
record of the same repository and pattern. It needs only Python 3.11 or newer, and
the scoring takes about 30 seconds once the bucket is downloaded.
**1. Install the scoring tool** in a virtual environment. pip installs it from its
GitHub release tag, together with
[secpat_interaction_recovery](https://github.com/anonymousspottedtoad/secpat_interaction_recovery),
whose interaction types the scorer uses:
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip huggingface_hub
pip install \
"secpat-interaction-order @ git+https://github.com/anonymousspottedtoad/secpat_interaction_recovery.git@v0.1.0" \
"secpat-scoring @ git+https://github.com/anonymousspottedtoad/secpat_evaluation_metrics.git@v0.23.0"
```
**2. Download this bucket** without `cpg/`, since the scoring does not read the
CPGs. The download contains about 44,000 small source files (234 MB) and takes
about 15 minutes:
```bash
hf buckets sync hf://buckets/AnonymousSpottedToad/human_verified_bucket human_verified_bucket --exclude "cpg/*"
```
**3. Score the four LLMs** with `scripts/score.py`:
```bash
python human_verified_bucket/scripts/score.py human_verified_bucket
```
For IRC, the script compares the interaction sequence that the LLM returned with the
human-verified sequence in `human_annotated/interactions/`, both narrowed to the
roles that the LLM and the annotators both detected. The 16 records whose one-shot
prompt exceeded the input limit have no answer and count as 0, as in the paper. The
script prints the means over the 100 records, which are the numbers of Table I, and
compares every record with the bucket's own `evaluation.json` in its last column:
```
Model RDA MLS IRC records not run same as evaluation.json
GPT-6.1 0.740 0.704 0.511 100 16 100/100
Muse Spark 0.715 0.691 0.509 100 16 100/100
DeepSeek-V4-Flash 0.672 0.646 0.515 100 16 100/100
GLM-5.3-Flash 0.705 0.688 0.510 100 16 100/100
```
`scores.json` then contains the three scores of every record and model, and
`--models gpt-6.1-sol` scores a single LLM.
## Manifest
`manifest.json` is `{"runs": [...]}`, one entry per record: `pattern`, `repo` and
`language`, plus the codebase's provenance — `url` (its GitHub repository),
`commit` (the commit it was cloned at) and `license` (see
[2. Codebase](#2-codebase)). The rest of a record follows from `pattern`, `repo`
and `language`:
| | |
| - | - |
| codebase | `codebase/<language>/<repo>/` |
| CPG | `cpg/<repo>__<JAVASRC\|PYTHONSRC>.cpg` |
| pattern | `security_patterns/<pattern>/04_pattern.json` |
| role mapping | `human_annotated/mappings/<language>/<pattern>__<repo>.json` |
| interaction sequence | `human_annotated/interactions/<language>/<pattern>__<repo>.json` |
The pipeline (`secpatrec`) runs the experiments on this dataset with
`secpat evaluate`, `secpat interaction` and `secpat recover --models`, each part
of the dataset named by its own flag. Its README covers installing, downloading
this dataset and running.
## How the records were drafted
Before the human verification, Claude Code (Opus 5.5) drafted a role mapping for
each sampled (repository, pattern) pair. It received the catalogue text and diagram
of each pattern (`01_text.md` and `02_diagram.md` in `security_patterns/<pattern>/`)
and spawned one subagent per pair. The subagent marked each role of the pattern as
`present`, `external` or `missing` and located the code symbol that represents it,
as the following pseudocode shows.
```
Input: R[l][p], the sampled codebases of language l paired with pattern p
for each language l in {Python, Java}:
for each pattern p in {P1, ..., P5}:
for each codebase r in R[l][p]:
spawn a subagent to analyse the code of r
for each role ρ of p: # inside the subagent
s[ρ] ← present, external or missing
a[ρ] ← location of ρ in r and its AST node
```
Our traversal rules then derived the interaction sequence of each of the 502 drafts
from the CPG of its repository. The traversal found at least one interaction for 281
drafts, and we sampled the 100 records of this bucket from them, 10 per pattern and
language, for two annotators to verify (see the next section).
## 1. Human-annotated records
Two JSON files per record, both named `<pattern>__<repo>.json`:
| File | Contains |
| ----- | ----- |
| `mappings/<language>/` | `role_mappings`: one row per role the annotators recorded (see below): `role`, `status` (`present`, `external`, `missing`), `symbol`, `file`, `lines`, `notes` — the same schema as a recovery's `recovery.json` |
| `interactions/<language>/` | `interaction_sequence`: the order the roles interact in: `entryRole`, `entryRule`, and `interactions` numbered by `order` from the entry role, each with `from`, `to`, the call site (`file`, `line`), its call `path` and the `mechanism` that reached it — again the same schema as `recovery.json` |
The 100 files contain 758 rows: 497 for roles the pattern requires, 200 for Subject
and System (Entity and System in `limit_request_rate`), which every file records,
and 61 for roles of the patterns' other flows that `04_pattern.json` does not list,
such as the Registrar of P1. Both papers (Section II-D of the SecPat paper, Table III
of the technical paper) count only the roles that `04_pattern.json` requires, 500 in all, and a required role with no row counts as
`missing`, which is the case for the Pepper store in three P1 records. Counted this
way, the files contain 334 present, 75 external and 91 missing roles, and 280
interactions. The metrics read the files the same way.
`interaction_sequence` is derived from `role_mappings` by traversing the repository's
CPG with Joern 4.0.627, using the traversal scripts in `secpat_interaction_recovery`
(`walk.sc` a81b989f…, `java.sc` eb906608…, `python.sc` b7038c5f…). All 100 records
have an order. The annotators checked it in our annotation tool (Section II-D of the
SecPat paper), which traverses the CPG again whenever they change a mapping and
saves the order derived from the saved mapping. So every stored interaction is the
traversal's output, and its `mechanism` says how the traversal reached it. To
compute IRC, `scripts/score.py` compares this sequence with the LLM's, both narrowed
to the roles that both sides detected. We computed the `evaluation.json` files in
`experiment_outputs/` by traversing the CPG again from the ground-truth mapping over
the same roles instead, and both ways produce the same score for every record.
`human_annotated/summary.csv` has one row per record: the number of interactions and the order as
`From>To → …`.
## 2. Codebase
- `codebase/<language>/<repo>/` — the 93 repositories (44 Java, 49 Python); five
serve two patterns and one serves three. Only the `.java` / `.py` source files
are kept.
- `cpg/` — one Joern 4.0.627 code property graph per repository. Point
`JOERN_CPG_ROOT` here to run Level 3 and stage 3 without rebuilding them. Java
CPGs leave out `test/` folders (the `javasrc2cpg` default).
- `codebase/repos.csv` — one row per repository: `language`, `repo`, `github`
(`owner/name`), `url`, `commit`, `license`, `license_source` (where the license
was read: the license file, or a `pom.xml` / `pyproject.toml` declaration),
`license_files` and `records` (the patterns it serves). Each repository is its
GitHub repository at `commit`.
- `codebase/licenses/<language>/<repo>/` — the license and NOTICE files of each repository
at `commit`, at their original paths. `codebase/<language>/` keeps only source
files, so these are copied here.
`license` is an SPDX identifier as GitHub reports it, read at `commit`. Two values
are not licenses: `NONE` — the repository has no license file or declaration at
that commit — and `NOASSERTION` — only some of its modules declare one.
## 3. Experiment outputs
The one-shot baseline, which puts the whole repository into one prompt
(`secpat baseline`). The pipeline and ablation runs are in the original release,
`securityPatternsRecovery`.
![Benchmarking an LLM on human_verified_bucket with the one-shot baseline](https://huggingface.co/buckets/AnonymousSpottedToad/human_verified_bucket/resolve/figures/usage-overview.png)
*Benchmarking an LLM on `human_verified_bucket` with the one-shot baseline. Each
LLM receives the codebase and the pattern of a record in one prompt (1, 2) and
answers with a role mapping and an interaction sequence (3), which the scoring
tool compares with the ground truth by RDA, MLS and IRC (4). In this example on
P5 Rate limiting in `loadcoder`, the answer marks the History provider as
external, which lowers RDA to 0.75.*
`<model>` is one of `gpt-6.1-sol`, `deepseek-v4-flash-0731`, `glm-5.3-flash`,
`muse-spark-1.3-contributor`.
Each `<model>/<language>/` folder contains one subfolder per record with its
`recovery.json`: `role_mappings` and `interaction_sequence`, in the same schema as
the human-annotated records. `interaction_sequence` is the order the model returned
in its answer, and Level 3 scores that (`--recovery-order stored`).
`evaluation.json` is the batch's score, one entry per record with Levels 1–3.
Their means are the reported RDA, MLS and IRC. 16 records (10 Java, 6 Python,
from 14 repositories) have a one-shot prompt over the 900K-token input limit
(counted with the `o200k_base` tokenizer). They were not run, have no folder, and
score 0 on every level.
To score these answers again, follow [Recomputing Table I](#recomputing-table-i).
## 4. Security patterns
The five patterns, each as `01_text.md` (the catalogue page), `02_diagram.md`,
`03_subflows.md` and `04_pattern.json` (the roles and the expected behaviour that
the recovery and the interaction traversal read):
- `authentication_pwd` — password-based authentication
- `verifiable_token_based_authentication`
- `obscure_token_based_access_control`
- `session_based_access_control`
- `limit_request_rate`
## License
- **This dataset's own content** — the human-annotated records, the manifest and
the experiment outputs — is released under the MIT License (see `LICENSE`).
- **`security_patterns/`** is derived from the security pattern catalogue of
Van Den Berghe et al., © 2021-2022 imec-DistriNet
(<https://securitypatterns.distrinet-research.be/>), licensed under
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/):
`01_text.md` is the catalogue page, and `02_diagram.md`, `03_subflows.md` and
`04_pattern.json` are adaptations of it. It is shared under the same license.
- **`codebase/`** and **`cpg/`** contain third-party code and graphs built from it.
Each repository keeps its own license, listed in `codebase/repos.csv`, with its
license files in `codebase/licenses/`.

Xet Storage Details

Size:
15.6 kB
·
Xet hash:
7a6381d1d32a952643afb92f321111eb0cc3d72e57323759e463cc89d6b292f8

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.