code/docs/data.md

# Data: what is here, and what is allowed into training

This is a small, auditable bootstrap collection. Downloaded GDScript is reference
material, not a set of completed games and not supervised training data. A row
becomes a training example only after it contains a complete project and passes
the validation and duplicate checks below. No quantity of reference scripts is
reported as a quantity of playable-game examples.

## Inventory

| File | Contents | Training use |
| --- | --- | --- |
| `data/reference_gd.jsonl` | 180 permissively licensed `.gd` files from two pinned public GitHub repositories | Optional teacher reference; not passed directly to SFT |
| `data/reference_manifest.json` | Commits, full notice paths, accepted/rejected counts, and filter reasons | Provenance |
| `data/licenses/` | Unmodified complete root license texts, including original copyright notices | Keep with any redistribution of the reference corpus |
| `data/teacher_prompts.jsonl` | 100 original, mechanically diverse teacher instructions | Teacher input; these are not generated or validated projects |
| `data/teacher_prompt_manifest.json` | Prompt count, digest, origin, and duplicate policy | Provenance |
| `data/codex_teacher_batch_root.jsonl` | 8 original Codex-teacher projects for instruction IDs 001–008 | Candidates; Linux admission required |
| `data/codex_teacher_batch_a.jsonl` | 32 original Codex-teacher projects for IDs 009–020 and 045–064 | Candidates; Linux admission required |
| `data/codex_teacher_batch_b.jsonl` | 32 original Codex-teacher projects for IDs 021–032 and 065–084 | Candidates; Linux admission required |
| `data/codex_teacher_batch_c.jsonl` | 28 original Codex-teacher projects for IDs 033–044 and 085–100 | Candidates; Linux admission required |
| `data/synthetic_candidates.jsonl` | Nine additional original seed projects written by the in-session assistant | Candidates; Linux admission required |
| `data/teacher_provenance.json` | Origin of the nine initial seeds | Provenance; these are not outputs of the separate Qwen teacher pilot |
| `data/train.jsonl` / `data/dev.jsonl` | Generated in the Linux runtime from accepted, deduplicated complete projects | Training / development loss only |
| `data/eval_prompts.jsonl` | 500 frozen prompts with no target outputs | Blind generation benchmark only |
| `data/eval_manifest.json` | Benchmark design, policy, and SHA-256 digest | Split integrity |
| `data/validation_manifest.jsonl` | Generated by Linux admission: per-candidate engine results and output fingerprints | Admission evidence, not an authored pass claim |
| `data/rejected_candidates.jsonl` | Generated by admission: rejections with reasons | Audit |
| `data/dataset_manifest.json` | Generated after a successful build: exact accepted train/dev counts and file digests | Current admitted dataset inventory |

The four Codex batches now contain **100 distinct complete-project candidates**,
one for every teacher instruction. The nine earlier seeds are separate, giving
109 authored candidates before validation and deduplication. This is still a
bootstrap collection, not evidence that a useful game generator has been trained.
Linux import and full 60-second runtime admission accepted 108 and rejected one:
97 training examples and 11 development examples. The produced dataset manifest
records the exact file hashes; these are teacher-data checks, not student evaluation. Attempts, static-contract passes, and
rejected candidates are not training rows or student evaluation results.

The earlier macOS startup-failure logs were moved to
[`artifacts/macos-startup-failure-evidence/`](../artifacts/macos-startup-failure-evidence/).
They describe sandbox/runtime startup failures, not successful admission or model
evaluation. There is no macOS validation manifest left in the source `data/`
directory. The Linux admission files listed above are generated outputs, not
prewritten evidence bundled to imply checks have passed.

## GitHub collection and attribution

The checked-in collection uses these exact revisions:

| Repository | Commit | Kept files | License |
| --- | --- | ---: | --- |
| [Godot demo projects](https://github.com/godotengine/godot-demo-projects/tree/3e08537616661a5883831628decab4c526260289) | `3e08537616661a5883831628decab4c526260289` | 160 | [Full MIT notice](../data/licenses/godotengine__godot-demo-projects/3e08537616661a5883831628decab4c526260289/LICENSE.md) |
| [Godot Dialogue Manager](https://github.com/nathanhoad/godot_dialogue_manager/tree/8ffe461f2a69180a35b2f00f31b563bfe6231da8) | `8ffe461f2a69180a35b2f00f31b563bfe6231da8` | 20 | [Full MIT notice](../data/licenses/nathanhoad__godot_dialogue_manager/8ffe461f2a69180a35b2f00f31b563bfe6231da8/LICENSE) |

Each row records repository, pinned commit, original path, direct source URL,
source text, SHA-256 digest, Git blob identifier, license identifier, and paths
to the complete notices. The dialogue-manager addon has its own nested notice,
so this conservative collector excluded that subtree; the kept files are outside
that boundary. The demos are code references and may use a different Godot minor
version from the training runtime. They are not claimed to compile independently.

Only ordinary `.gd` text blobs are downloaded. No source archive, textures, models,
music, sounds, fonts, or scene assets are fetched. Public availability is never
treated as a license. A repository must have an unambiguous full MIT,
Apache-2.0, or CC0-1.0 root grant. This snapshot happens to contain MIT only.
GPL, LGPL, AGPL, other copyleft, noncommercial, unknown, and mixed root grants
are refused. Files under any nested license/notice boundary are excluded, even
when that nested grant might be permissible. Relevant README and CREDITS files
are scanned for restrictive licenses, and their affected subtrees are excluded.
Conflicting file comment headers and SPDX expressions are also excluded.

These checks deliberately prefer false exclusions to guessing. They do not prove
that every upstream contributor had the right to license every line. Report any
suspect file with its row ID, and remove it and derived data while investigating.
Do not add a repository by merely trusting GitHub's license badge.

Reproduce the exact reference corpus with:

```bash
python scripts/collect_github.py \
  --repo godotengine/godot-demo-projects@3e08537616661a5883831628decab4c526260289 \
  --repo nathanhoad/godot_dialogue_manager@8ffe461f2a69180a35b2f00f31b563bfe6231da8 \
  --max-files-per-repo 180
```

The cap samples eligible paths evenly across each repository's sorted file tree;
license exclusions and duplicate removal reduce the final count. API access is
unauthenticated and uses no paid service. GitHub rate limiting may require waiting
and retrying; the script does not bypass limits.

## The Stack v2: GDScript exists, but this build does not use it

The official [dataset card](https://huggingface.co/datasets/bigcode/the-stack-v2)
and its [configuration list](https://huggingface.co/datasets/bigcode/the-stack-v2/blob/main/README.md)
include GDScript. Access is gated by terms. The card requires honoring original
source licenses and Software Heritage training principles, and it says bulk
download requires an agreement with Software Heritage/INRIA. Its permissive
license set is also broader than this project's MIT / Apache / CC0 allowlist.

That is a licensing/access dependency, not permission to scrape around the gate.
No Stack v2 rows were downloaded, no terms were accepted on the user's behalf,
and no blanket relicensing is claimed. If these conditions are resolved later,
an importer still needs original-license filtering, source identifiers, notices,
removal-request handling, and the same duplicate and validation gates. The direct
GitHub corpus lets development continue without assuming those conditions away.

## Teacher references and synthetic provenance

`teacher_prompts.jsonl` has exactly `id` and `instruction` per row. Its 100 prompts
ask for distinct interactions and combinations rather than 100 differently named
versions of one project. They contain no frozen eval IDs or answers. Every prompt
is checked against the 500 held-out instructions and other teacher instructions
with normalized token 3-gram Jaccard similarity; values at or above 0.78 are refused.
This is a documented lexical check, not a claim to eliminate all semantic overlap.

A teacher generator may select relevant `code` strings from `reference_gd.jsonl`.
For every attempt, save the teacher model ID and revision, generation parameters,
input prompt digest, output digest, selected reference row IDs, complete license
notice paths, and validation outcome in a sidecar provenance file. Keep source
copyright notices in any copied or adapted snippets and redistribute their full
licenses with the dataset. Do not label unvalidated attempts as accepted examples.
Do not supply the blind eval prompts to the teacher to create training targets.

All 100 main candidates were written directly by the in-session Codex assistant,
with varied mechanics and original geometric visuals. They were not generated by
Qwen2.5-Coder-14B. Their instructions remain exactly those in the train-only prompt
file. The original project, scene, and UI shell comes from
[`src/vibegame/seeds.py`](../src/vibegame/seeds.py); the batch game logic is authored
separately. Original project code uses the repository's Apache-2.0 license. The
180 downloaded MIT reference scripts retain their own notices and are never
loaded directly as SFT rows.

The exact candidate, authoring-source, and provenance files are:

| IDs | Candidates | Reproducible authoring source | Provenance |
| --- | --- | --- | --- |
| 001–008 | [Root batch](../data/codex_teacher_batch_root.jsonl) | [create_teacher_batch_root.py](../scripts/create_teacher_batch_root.py) | [Root provenance](../data/codex_teacher_batch_root_provenance.json) |
| 009–020, 045–064 | [Batch A](../data/codex_teacher_batch_a.jsonl) | [create_teacher_batch_a.py](../scripts/create_teacher_batch_a.py) | [Batch A provenance](../data/codex_teacher_batch_a_provenance.json) |
| 021–032, 065–084 | [Batch B](../data/codex_teacher_batch_b.jsonl) | [create_teacher_batch_b.py](../scripts/create_teacher_batch_b.py) | [Batch B provenance](../data/codex_teacher_batch_b_provenance.json) |
| 033–044, 085–100 | [Batch C](../data/codex_teacher_batch_c.jsonl) | [create_teacher_batch_c.py](../scripts/create_teacher_batch_c.py) | [Batch C provenance](../data/codex_teacher_batch_c.provenance.json) |
| Nine earlier seeds | [Seed candidates](../data/synthetic_candidates.jsonl) | [seeds.py](../src/vibegame/seeds.py) | [Seed provenance](../data/teacher_provenance.json) |

These sidecars record authorship and content fingerprints. A static contract pass
in a sidecar means the project bundle passed local schema/API checks; it does not
mean the game imported or survived a run. Engine admission evidence is produced
separately on Linux. The authoring scripts do not create fake runtime-pass records.

An optional free-T4 Qwen2.5-Coder-14B teacher pilot was also attempted through
[`scripts/synthesize.py`](../scripts/synthesize.py). The initial pilot completed
two attempts, both rejected. A revised pilot completed four attempts: two passed
the teacher admission checks and two failed, then that pilot was stopped. These
six attempts are not the source of the 100 Codex-authored candidates. The two
pilot passes are teacher-candidate validation results, not evaluation results for
the fine-tuned student and not two extra automatically admitted training rows.

The training build gives the Codex batches priority for duplicate instructions.
If a pilot output answers an instruction already represented by a Codex candidate,
it is excluded by deduplication rather than substituted for the authored answer.
Future Qwen expansion is optional; any genuinely new prompt/output pair must pass
the same license, validation, and duplicate gates. Keep the pilot attempt logs and
source references with its own provenance rather than attributing those outputs
to Codex or attributing Codex work to Qwen.

## Exact training format and admission gate

Each SFT line has exactly two keys:

```json
{"instruction":"Build a small playable game...","output":"{\"files\":{\"project.godot\":\"...\",\"main.tscn\":\"...\",\"main.gd\":\"...\"}}"}
```

`output` is a string containing strict JSON with a `files` map. Complete source
files are represented by their paths and contents. The shared project validator
enforces safe relative paths, a Godot 4 project, required scene/script files, local
resource references, size limits, and restricted APIs before executing anything.

Build the frozen prompt set, then admit candidates with:

```bash
python scripts/build_dataset.py --freeze-eval-only
python scripts/build_dataset.py \
  --input data/codex_teacher_batch_root.jsonl \
  --input data/codex_teacher_batch_a.jsonl \
  --input data/codex_teacher_batch_b.jsonl \
  --input data/codex_teacher_batch_c.jsonl \
  --input data/synthetic_candidates.jsonl \
  --godot /absolute/path/to/Godot \
  --workers 4
```

`--input` may be repeated for additional teacher-candidate JSONL files. There is
no skip-validation switch. Each accepted project must import without errors and
complete a full 60-second sandboxed headless run. Static policy failures, malformed
JSON, parse/import failures, script errors, and early exit are rejected. An OS
sandbox startup problem is also a failure, never a successful game check.

Successful evidence can be reused only for the same project content, exact Godot
binary digest, at least the requested runtime duration, intact original import
and runtime logs, and unchanged source files. Failed or mismatched attempts use
new work directories so previous diagnostics remain available.

Exact and near duplicate instructions are removed against both the blind set and
previous candidates. Project outputs are deduplicated independently using token
5-gram Jaccard similarity at 0.85. This occurs before the stable train/dev split,
so near-identical accepted outputs cannot enter both splits. The default dev split
is 10 percent, with at least one example in each split. A build with fewer than two
accepted unique projects fails rather than fabricating a development example.

## What the 500 prompts do and do not measure

The benchmark crosses 25 gameplay families with 20 explicit rule modifiers. It
has 500 unique instructions, but those instructions are related compositional
variants, not 500 independent task families. The manifest makes this limitation
explicit. It is a bootstrap engineering benchmark, not a validated measure of
fun, originality, or general game-development ability.

The frozen file refuses silent replacement when its contents differ. Never put
these 500 rows in train/dev data, use them for loss or checkpoint selection, or
generate teacher targets for them. A separate labeled dev split supplies eval
loss. Final compile and clean-run rates must use all 500 prompts, include failures
and timeouts in the denominator, and identify any incomplete run as partial.

Run data contract tests without launching an engine:

```bash
python -m unittest discover -s tests -p test_data.py -v
```