HolyCyber — defensive cybersecurity instruction dataset

A long-form, fully grounded instruction dataset for a defensive cybersecurity assistant. Every row is a self-contained briefing of at least 2,000 tokens (user + assistant, tiktoken cl100k_base), written from authoritative public sources and shipped with the provenance needed to check it.

At a glance

Layout

dataset/
  train.jsonl  val.jsonl  test.jsonl      # the global dataset
  samples.jsonl                            # 17 full rows, one per domain
  SAMPLES.md                               # the same rows, rendered for reading
  DATASET_CARD.md                          # global datasheet
  README.md                                # this file
  VERSION                                  # dataset version
  layers/
    INDEX.md                               # per-domain table
    <domain>/{train,val,test}.jsonl        # one focused dataset per domain
    <domain>/DATASET_CARD.md               # per-domain datasheet

Domains

Each layer is built on its own, from that domain's authoritative sources only, and can be used without the others.

Core share is the median percentage of an answer that is the entity's own material; the rest is a labelled reference appendix (## Reference: ...). The column is there because it is the honest measure of how much of a row is about the thing it is named after.

Record format

One JSON object per line. messages is the usual three-turn chat.

{
  "id": "6d408e38f3cb8dff",
  "row_key": "detection_engineering:attack:T1568:detection_engineering:detection_strategy:1",
  "domain": "detection_engineering",
  "subdomain": "detection_strategy",
  "task_type": "detection_engineering",
  "difficulty": "advanced",
  "group_key": "attack:T1568",
  "source": "mitre-attack",
  "license": "MITRE ATT&CK Terms of Use",
  "split": "train",
  "canary": "…",
  "tokens": {"user": 180, "assistant": 2350, "total": 2530},
  "core_share": 0.41,
  "appendix": true,
  "references": [{"title": "ATT&CK T1568 Dynamic Resolution", "url": "https://…"}],
  "messages": [
    {"role": "system", "content": "…"},
    {"role": "user", "content": "How would you build detection for …"},
    {"role": "assistant", "content": "## Summary\n…"}
  ]
}

Using the metadata

tokens, core_share and appendix are measured per row, so a training or evaluation run can filter or mask without re-tokenising anything:

from datasets import load_dataset

ds = load_dataset("json", data_files={
    "train": "dataset/train.jsonl",
    "validation": "dataset/val.jsonl",
    "test": "dataset/test.jsonl",
})

# rows whose answer is mostly the entity itself, not the appendix
core = ds["train"].filter(lambda r: r["core_share"] >= 0.5)

# token-budget a batch
short = ds["train"].filter(lambda r: r["tokens"]["total"] <= 2600)

# the appendix is a labelled block, so masking it is a text operation
def mask_reference(answer: str) -> str:
    i = answer.find("\n## Reference: ")
    return answer[:i] if i != -1 else answer

What a row contains

Answers are built from named sections, not free prose, so a row can be checked section by section. The most frequent topic headings across the dataset (structural labels such as Answer or Context, and the appendix, excluded):

Samples

SAMPLES.md shows one complete row per domain (the row closest to that layer's median size, from the test split), and samples.jsonl holds the same 17 rows in full. They are there so a reviewer can judge the text without loading 1.2 GB.

Quality controls

Guaranteed by the build and re-checked by scripts/verify_layers.py:

• 2,000-token floor on every row (user + assistant).
• Unique rows: unique ids, unique assistant texts, unique questions.
• No split leakage: the split is cut on group_key (the source entity).
• Grounded: every fact comes from MITRE ATT&CK, D3FEND, CWE, CAPEC, ATLAS, SigmaHQ, CISA KEV, an RFC, a NIST publication or the cited arXiv paper.
• Decontaminated: no 25-gram overlap with the benchmark corpora, re-checked by an independent pass.
• Defensive only: no functional exploits, no malware, no weaponised jailbreak prompts.
• Provenance: every corpus document has a row in research/fetch_manifest.csv (url, date, licence).
• Measured metadata: tokens sums to the build total exactly; row_key is unique in every layer and never crosses a split; 40,144 rows (77.5%) carry an appendix, the other 11,661 are core-only.

Measured, not claimed — research/AUDIT_REPORT.md has the numbers:

• Reference-appendix share. Framework entities are short (a KEV entry is a few hundred tokens), so their rows are completed with other entities' records. The appendix is labelled and rotated, but it is still reused text.
• Paragraph reuse. Rows are unique; some paragraphs are not. The audit reports the exact share, including how much crosses the train/test boundary.
• Task mix. Paper-heavy domains are close to single-task by nature.

Usage

from datasets import load_dataset

ds = load_dataset("json", data_files={
    "train": "dataset/train.jsonl",
    "validation": "dataset/val.jsonl",
    "test": "dataset/test.jsonl",
})

For supervised fine-tuning, mask the ## Reference: blocks if you want the loss computed on the entity's own material only — the appendix is deliberately marked so this is a two-line filter.

Corpus

All sources are public:

research/SOURCE_TABLE.md is the generated source audit (licence, retrieval date, label method, known problems).

Building

python scripts\domains.py                      # every domain
python scripts\domains.py identity_access      # one or more domains
$env:DRY="1"; python scripts\domains.py         # size preview
$env:FORCE="1"; python scripts\domains.py       # rebuild existing layers
python scripts\verify_layers.py                 # independent verification
python scripts\validate_ground_truth.py         # quotation / corpus-vocabulary check
python scripts\run_audit.py                     # four-stage independent audit

The build is resumable: a layer that already exists in build/layers/stats.json and has a DATASET_CARD.md is skipped unless FORCE=1.

Licence

The dataset is a derivative of its sources, and the sources do not share one licence:

Because OWASP material is share-alike, the conservative choice for a redistribution is CC BY-SA 4.0 with attribution to the sources above and to the per-row license field. arXiv licences vary per paper: check license/references before republishing a row that quotes a paper.

Changelog

Limits of this release

• Built by a deterministic pipeline, not reviewed row-by-row by a domain expert. Grounding bounds invention; it does not by itself prove correctness.
• The 2,000-row × 2,000-token floor is above what the smallest domains can support from their own records alone, which is why the reference appendix exists. research/AUDIT_REPORT.md quantifies the consequence instead of hiding it.
• In a few RFC rows, a resource-record dump from the typeset PDF is parsed as a section heading, so it appears as a heading and as a question bullet. The audit records the example; removing it needs a page-furniture filter in the document parser.
• English only.

Related public datasets

Six public sets were reviewed; what each offers and what was taken from it is recorded in research/EXTERNAL_REVIEW.md (Part A: Vyber07/cyber-security, gated; Part B: the four below, measured in full).

The comparison is also a contamination check: none of the four shares a verbatim paragraph (≥120 characters, normalised) with this dataset's 305,232 unique paragraphs. Numbers and method: scripts/external_compare.py, build/external/compare.json.

The plan documents