SpecMine

A large-scale corpus of Spec-Driven Development (SDD) artifacts mined from public GitHub: the Markdown specifications that AI coding agents turn into code, attributed to the tool that produced them, enriched with repository metadata and full commit history, and, for a subsample of repositories, followed into the pull requests that change them.

This is the per-table Parquet mirror of SpecMine v1.0 (July 2026 snapshot). The citable archive of record (with the MySQL dump, CSV, and spec-text JSONL) is on Zenodo; the schema, loaders, an example notebook, and a 500-repository sample are on GitHub at https://github.com/shyamagarwal13/specmine-official. SpecMine is the dataset for the MSR 2027 Mining Challenge.

Tables (select a config above)

Spec text is not on Hugging Face; it ships as specs.jsonl.gz in the Zenodo record and joins to spec_files on file_url_sha16.

Usage

from datasets import load_dataset
prs = load_dataset("USER/specmine", "pull_requests", split="train")

Or read a table directly:

import pandas as pd
prs = pd.read_parquet("hf://datasets/USER/specmine/data/pull_requests.parquet")

License and ethics

Compilation released under CC BY 4.0. Underlying file content remains under each source repository's own license (see repo_license_spdx_id). Commit emails, avatar URLs, and GPG signatures are removed; public logins and numeric IDs are kept. Please do not attempt deanonymization.

Citation

@inproceedings{specmine2027,
  author    = {Agarwal, Shyam and Vasilescu, Bogdan},
  title     = {SpecMine: A Large-Scale Corpus of Spec-Driven Development Artifacts},
  year      = {2027},
  booktitle = {Proceedings of the 24th International Conference on Mining Software Repositories},
  series    = {MSR '27}
}