code/docs/evaluation.md

What the evaluation actually measures

The publication benchmark uses the first and only generation for each of 500 frozen prompts. Invalid JSON, rejected source, parse errors, script errors, early exits and truncated generations count as failures. There is no repair pass and no best-of-N selection. A lost generation attempt after an interrupted session is a failure; the harness never silently generates it again.

The evaluator records two fractions over all 500 prompts:

• Compile success: the original project satisfies the source policy and Godot's editor import exits successfully without a parse, script or resource error.
• 60-second clean run: import succeeds, the Godot runtime remains alive for a full wall-clock 60-second window after its startup banner, and no script or engine error is observed. The harness then terminates it. This uses elapsed time, not --quit-after 60 (which counts frames).

These are technical checks, not evidence that the requested mechanics work. No input actions are exercised. A static screen or stuck game loop can meet the headless check. A human still needs to play the samples. We do not describe these rates as a gameplay-success score.

summary.json reports final fractions only after every prompt has a persisted result and no infrastructure failure remains. An empty or incomplete evaluation reports null rates, never 0/0 or an apparent perfect score. Sandbox failures and recorded generation-backend failures stop the run and block final rates; resuming cannot silently turn a CUDA/backend failure into a model score. The summary also includes raw counts and progress for inspection.

October 4 run snapshot

The T4 experiment is stopped with 72 of 500 recorded results, comprising 64 saved generations and eight interrupted generation attempts counted as failures. Validation of the saved outputs is complete: one compile success, one clean 60-second run, and zero infrastructure failures. The partial median among measured successful launches is 1,394.69 seconds; none launched within 60 seconds. Final rates remain null because 428 prompts have no result. These partial counts do not establish the full benchmark score or gameplay quality.

Training finished on a free T4. After free GPU quota was exhausted, the owner purchased Colab Pro and continued the same T4 experiment. An A100 was selected at 14:06 UTC and then allocated with 40GB GPU memory and 83GB host RAM; the observed rate was 5.3 compute units per hour with 98.62 units remaining. These observations are not a guarantee of total cost. The predeclared alternative is a new complete 500-prompt pass in runs/teacher-v2/evaluation-a100-v1, using the same completed adapter, tokenizer, frozen prompts and decoding. Its training-prompt batch-eight capacity check completed in 271.711 seconds and passed the budget check. The first evaluation batch took about 273.5 seconds; at 14:36:58 UTC, 17 results were recorded with no compile/clean-run successes or infrastructure failures. These are partial counts, not final rates. The partial T4 directory stays intact and its results are never pooled with the new run.

The A100 entry point is scripts/evaluate_pro.py; it requires the observed compute-unit rate and balance. A completed A100 run can feed the existing recorder, demo builder and publisher by passing that evaluation directory. Keep sample indices 0, 200 and 400, and keep the finished launch video at runs/teacher-v2/VibeGame-launch-60s.mp4. The complete-500, no-infrastructure-failure and genuine-recording requirements remain unchanged.

Publication keeps the completed primary pass under evaluation/ in the model repository and the stopped T4 evidence under evaluation-t4-partial/. For A100 it also includes the original protocol declaration and numbered capacity/budget receipts, checking their model, runtime, source-manifest and report hashes before requesting a token. The predeclaration and approved budget must predate the first blind attempt. Text evidence is screened for recognizable credentials; local home prefixes are redacted with original/published hashes retained in REDACTIONS.json. The model card names the GPU from the selected evaluation manifest. These uploads preserve separate experiments and do not relax any scoring or video requirement.

Reproduce

Use Linux x86_64, including the free Colab T4 runtime. The model backend loads the saved 4-bit adapter using Unsloth. The validator itself is CPU-only. Install bubblewrap, util-linux (for prlimit), xvfb, ffmpeg, and libgl1-mesa-dri from the Ubuntu package manager. Install this repository with pip install -e ..

python scripts/install_godot.py --destination /content/godot
python scripts/evaluate.py \
  --prompts data/eval_prompts.jsonl \
  --output /content/drive/MyDrive/VibeGame-8B/evaluation \
  --backend unsloth \
  --adapter /content/drive/MyDrive/VibeGame-8B/runs/default/adapter \
  --godot /content/godot/Godot_v4.5.1-stable_linux.x86_64 \
  --expected-count 500 --run-seconds 60 --batch-size 4 --workers 4

The manifest freezes dataset SHA256, runtime SHA256, adapter files and hashes, locally cached base-model files when available, backend versions, decoding settings and the shared system-prompt hash. If base artifacts cannot be hashed, the manifest records the reason; a release must disclose this gap. Ollama stores the installed model digest and server version instead. Evaluation is greedy with a fixed seed. It does not contact a paid inference service or pull a model automatically through Ollama.

The default Unsloth configuration generates four prompts together on one free T4, then validates projects with four concurrent CPU workers while the next GPU batch runs. GPU batches execute sequentially. The evaluator limits pending validation work between GPU batches, and records time spent waiting for a worker. Ollama uses one prompt at a time. Batch size and worker count are frozen in the manifest and cannot change during resume.

Before the first blind generation, measure the completed adapter at a candidate batch size using training prompts only. For example, python scripts/benchmark_batch.py /content/drive/MyDrive/VibeGame-8B/adapter-capacity32.json --adapter /content/drive/MyDrive/VibeGame-8B/runs/default/adapter --batch-sizes 32 tests batch 32 with a short warm-up and throughput check, followed by the normal 3072-token generation cap if memory headroom remains. It loads the saved base revision and tokenizer through the evaluation loader and records adapter fingerprints. A base-only result does not establish adapter capacity. The evaluator accepts batch sizes 1, 2, 4, 8, 16 or 32; the default remains four. The benchmark's default comparison sizes remain 1, 4, 8 and 16. If an adapter capacity check fails or exceeds the 12.5 GB threshold, retain its report and measure a smaller batch in a new report before any blind generation. Outputs may reach EOS before the cap: a batch ending around 1000 tokens does not establish that a batch of 2000–3072-token outputs fits. The report records actual token lengths and does not claim forced worst-case allocation coverage.

Both the short measurement and full-cap check must also pass a conservative memory estimate. The report uses the loaded model's layer count, KV-head count, head width and element size to estimate additional cache bytes from the observed padded prompt plus longest output to the complete 4096-token context, with at least one additional 512-token cache growth block. It adds those bytes to the larger observed allocated/reserved peak. The estimate must remain below both 12.5 GB and physical GPU memory minus 2 GB; a failed short estimate skips the expensive full-cap run. Each measurement stores full_context_memory_estimate, and reports identify memory_estimate_policy: kv_full_context_v1. This is an extrapolation rather than a measured worst-case bound: workspace changes, prefill shapes and allocator fragmentation can still cause allocation failures. Previously saved reports are retained unchanged and cannot bypass the new gate.

Running all 500 clean runtime windows alone takes 8 hours 20 minutes sequentially, or an ideal minimum of 2 hours 5 minutes with four workers. Generation, imports, process startup, Drive I/O and resource contention add time; this is a lower bound, not a completion estimate. The four-token warm-up checks the inference path before blind generation, but does not establish that a full 3072-token batch fits in memory. Run the separate capacity smoke test using training prompts before starting the frozen evaluation. Do not assume training plus the full evaluation fits in a single Colab session. Persist the evaluation directory to Drive and resume across free sessions. A process interrupted in the middle of validation retains its files and receives an interruption note. Resume validates that exact immutable generation in a fresh numbered evidence directory. This does not resample or repair model output; completed validation failures are never retried. The summary discloses the number of interrupted validation attempts.

For each numbered item, the evaluator saves attempt.json, immutable generation.json, result.json, and a completed validation directory, plus complete import and runtime logs and the extracted project. result.json records validation_evidence, normally check or a check-resume-NNN directory after an interrupted check. Every new batch receives attempt records before its single generation call; persisted outputs survive a disconnect. Missing outputs from interrupted attempts are never silently resampled. Resume checks the manifest and skips completed items. Changing the data, model, runtime or settings requires a new output directory.

Each prompt in a GPU batch receives the full measured generation-call time, saved as generation_call_seconds. The clock starts before chat formatting and tokenization, includes transfers to the GPU and model generation, and stops after decoding every output. The evaluator measures the outer call and starts its queue clock at the same endpoint. Time is never divided by the batch size: all outputs become available together. Prompt-to-launch latency then adds elapsed time from call completion until its validation worker starts, extraction/hash overhead, import, and runtime startup observed by the engine log banner. Queue time includes output persistence and waiting behind other validators. The narrower generation_seconds measurement of the model generation call remains available separately. Older artifacts lacking the new timing retain an explicitly labeled legacy fallback. Timing uses the already loaded model after warm-up; it excludes model download/loading and the subsequent 60-second validation window.

Interrupted or resumed validation has no trustworthy continuous prompt-to-launch measurement, so its latency remains null and is excluded from the median and the under-60-second count. The median covers compiled projects with recorded latency; compile and clean-run fractions still use all 500 prompts. Batched throughput is not single-request latency, and none of these measurements establishes that the requested game mechanics work. The hook “prompt → playable game in 60 seconds” remains a target until an actual unedited model generation and demonstrated gameplay establish it on named hardware.

Isolation and source restrictions

Only project.godot, local .gd and .tscn text files are accepted. Each project must include main.gd and launch res://main.tscn. The parser rejects Markdown wrappers, duplicate JSON keys, traversal, symlinks, binary assets, large outputs, external paths, unsafe APIs, plugins and autoloads. This deliberately narrows the model's output vocabulary; rejected outputs count as failures.

Static filtering cannot establish that arbitrary code is safe. Godot additionally runs in a Linux bubblewrap sandbox with private network, process and mount namespaces, a private home directory, no Drive/home mounts, read-only engine/system libraries and a writable project directory. /proc is an empty private filesystem because Colab can prohibit new proc mounts. Runtime file sizes and memory are bounded. Missing isolation fails closed; there is no --unsafe bypass.

macOS installation is supported for normal developer use. Automated untrusted-project validation and capture are currently disabled on macOS: the tested restrictive macOS 26.3.1 sandbox caused the engine's dynamic loader to abort before GDScript ran. The validator raises before launching a process. Use the tested Linux runtime for evaluation, rather than broadening filesystem access to fix the loader.

Three predeclared recordings

The visible examples are indices 0, 200 and 400, selected before evaluation. If a selected generation fails, publish that failure; do not quietly swap in a successful game.

That original mode is unchanged. The optional success showcase in docs/demo.md uses a separately declared display policy: the first three clean results in frozen order after the complete benchmark, labeled as selected successes rather than representative examples. It never changes this manifest's sample indices, repairs outputs, pools hardware experiments, or removes failures from the scores.

python scripts/record_game.py \
  --evaluation /content/drive/MyDrive/VibeGame-8B/evaluation \
  --index 0 --godot /content/godot/Godot_v4.5.1-stable_linux.x86_64 \
  --output /content/drive/MyDrive/VibeGame-8B/evaluation/recordings/sample-000.mp4 \
  --seconds 15 --fps 30

Repeat for indices 200 and 400, writing sample-200.mp4 and sample-400.mp4. Capture uses Godot's Movie Maker at a fixed timestep, a private Xvfb display, software rendering and local ffmpeg. It records the untouched game with no input; it is a screen recording, not a usability test or real-time inference benchmark. Source and video hashes accompany the output; ffprobe verifies actual encoded duration and frame count before a capture is marked successful. A separately supplied --project-json is labeled as a scaffold demo and cannot be counted as a model evaluation sample.

Official engine references: command-line flags, Movie Maker, and pinned 4.5.1 release.