docs/tutorials/03-data.md

M2: build labels and splits before measuring improvement

Status: verified. The source build, deterministic splits, independent-agent review and bound integrity audit passed. The user approved P01–P12 and explicitly authorized this disclosed synthetic study; those 12 policy examples are the complete scope of the actual human review. See the human policy record and the independent review report.

M1 showed that a classification update works. It did not show what the model should learn or how to test it honestly. M2 defines those boundaries before training. Read the labeling policy first: the label depends on the whole request plus its context, and a label never authorizes command execution.

1. Decide what each label means

The policy first checks whether any positively requested action is outside the supported domain. Such a request is unsupported, including mixtures of a local step and an external step. Otherwise missing material information means clarify; enough information means suggest. Quoted or prohibited activities are not positively requested actions.

Ordinary defaults matter. A read-only display can go to standard output, and an unqualified sort defaults to ascending lexicographic order. Do not label a request ambiguous merely because an optional flag was omitted. Conversely, choosing among two conflicting target revisions cannot be replaced by an arbitrary guess.

P01–P12 in the policy form a concrete human policy-review sample. They are excluded from the dataset. The user approved all 12 examples. That establishes agreement about those policy decisions, not human review of every row. The corpus is synthetically authored; independent-agent review must remain distinct from actual human review.

2. Author scenarios before expanding views

The authored sources contain 150 scenario groups: 100 supported local-operation scenarios and 50 unsupported external-workflow scenarios. Each has a meaningful operation, authored wording and context. The builder expands controlled views, not random filename substitutions:

• Complete versus missing or conflicting material context.
• Context inside the request versus in the context field.
• An authored paraphrase, an interpretable typo, a negated external activity and a quoted keyword distraction.
• For external operations, standalone requests and compositions with a supported local action.

There are 12 related views per scenario and 1,800 rows total, balanced at 600 per class. The independent unit is the scenario group, not each sentence variation. These synthetic views share rendering conventions, so even disjoint groups may share stylistic shortcuts. This is a controlled synthetic study; a successful result would not establish real-world robustness.

From the repository root, recreate the source rows:

uv run --locked --extra ml python scripts/build_data.py \
  --output artifacts/m2-authored-reproduction.jsonl

Expected output includes status: built-not-reviewed, cases: 1800, groups: 150, human_reviewed: false, and splits_assigned: false. Use a fresh destination; the builder refuses to replace earlier evidence. It validates every row with the strict dataset schema, imports no model and obtains no predictions.

The first source review found two superficially different Git scenarios that represented the same semantic operation: unstaging a tracked path while preserving working-tree contents. The author replaced one with a distinct intent-to-add operation and preserved the earlier generated output. Exact string deduplication alone would not catch that conceptual overlap; review the scenario meaning as well as its bytes.

3. Split entire groups into four roles

The verified deterministic allocation is:

Role	Groups	Rows	Purpose
Train	84	1,008	Fit model parameters and supervised baselines.
Validation	21	252	Choose prompt, architecture, data size and training settings.
Calibration	15	180	Fit probabilities and routing/defer thresholds after model selection.
Test	30	360	Run the frozen final comparison once.

These totals allocate 56/14/10/20 percent of the 150 groups. They intentionally give the final set more room than the initial approximate 60/15/10/15 proposal. Each partition contains the same ratio of supported and unsupported scenario kinds; supported groups contain both suggest and clarify, preserving class balance.

Whole archive_compression and database_operations families form the generalization cohort: 15 groups and 180 cases, with 60 cases per label. A separate familiar-family cohort has the same number of groups, cases and class counts. This tests two particular withheld families, not all possible unseen operation types. The familiar test must also keep complete scenario groups separate from training.

The independent splitter hashes group identities with the recorded seed and assigns partitions deterministically. Moving a paraphrase or context variant across partitions would leak knowledge of the same scenario. A random row split is therefore inappropriate. Calibration needs its own partition because choosing a model and then measuring confidence on the same selection cases can overstate reliability.

Use the split configuration and a fresh output directory:

uv run --locked --extra ml python scripts/split_data.py \
  --source artifacts/m2-authored-reproduction.jsonl \
  --config configs/m2-split.json \
  --output artifacts/m2-tutorial-splits

This command was verified. The built source reproduced SHA-256 ac346712dbfda1cd5a7cc9fc3cd43d14d78c378d4fb2c9ba24f3a006f0f07b28; all four split files and the manifest reproduced byte-for-byte against data/generated/m2-v1. The manifest hash is 2e22842b5d9b34eea8209d19aeb8295376122322ad9638346e02af3b53786db4. The source JSONL's location does not alter content identity. No model should receive final-test labels while prompts or adaptation settings are being chosen.

4. Audit bytes, lineage and review coverage

The audit must check exact hashes, unique IDs, full-group isolation, cross-split duplicate inputs, expected counts and whole-family holdouts. It also binds source specifications, the builder, policy and splitting code so later changes cannot silently reuse the original dataset identity.

Review answers a different question from schema validation. A well-formed label can still be wrong. The independent ledger must identify the exact frozen source and test bytes, all reviewed scenario groups and every reviewed final case. Per-row authoring records remain honest about their initial pending status; a separately verified ledger supplies completed review coverage.

Only request and context are inference inputs. Gold labels, rationales, family names, variant IDs, tags and review metadata must not be included in the text sent to models. Otherwise the dataset can reveal its own answer through bookkeeping rather than test routing.

Run the integrity audit first:

uv run --locked --extra ml python scripts/audit_data.py \
  --dataset artifacts/m2-tutorial-splits \
  --output artifacts/m2-tutorial-initial-audit.json

This initial audit passed and reported acceptance_ready: false, because no independent review ledger was supplied. It checked 1,800 cases, 150 groups, all four partition counts and both cohorts. Do not confuse passed: true for structural integrity with completed review.

Now check the bound independent review with a different output path:

uv run --locked --extra ml python scripts/audit_data.py \
  --dataset artifacts/m2-tutorial-splits \
  --review reports/m2-label-review.json \
  --output artifacts/m2-tutorial-reviewed-audit.json

This command passed with acceptance_ready: true, a verified agent reviewer, 150 reviewed scenario groups and all 360 final-case IDs. The ledger binds the exact manifest, source hashes and final-test bytes. The reviewer examined every source scenario and renderer condition, checked all 1,800 derivations, and reviewed final-case coverage. This is not 1,800 independent human readings. The review report records the method and resolved findings; the human record separately binds the 12-example approval and authorization of a synthetic, agent-reviewed study.

To read aggregate integrity results without opening final case text:

uv run --locked --extra ml python - <<'PYTHON'
import json
from pathlib import Path

report = json.loads(Path('artifacts/m2-tutorial-reviewed-audit.json').read_text())
for name in ('train', 'validation', 'calibration', 'test'):
    part = report['partitions'][name]
    print(name, part['groups'], part['cases'], part['labels'])
print('integrity_passed:', report['passed'])
print('review:', report['review'])
print('acceptance_ready:', report['acceptance_ready'])
PYTHON

The reviewed aggregate snippet above was executed successfully. Preserve mismatches and rejected source versions. The completed M2 gate establishes this dataset's recorded construction and review boundaries; it provides no prediction accuracy or confidence-calibration result.

Alternatives and limitations

Human-authored production requests with independently adjudicated labels would provide stronger evidence of deployment behavior. They take more collection and review effort, and their permissions and provenance need recording. The current synthetic study is useful for controlled information changes, explicit scope boundaries and reproducible experiments, provided its limitations are visible.

Large random template expansion would be cheap but would not create independent evidence. More genuine scenario groups are preferable to multiplying names and wrappers. The held-out-family cohort and grouped uncertainty estimates mitigate some dependence, but they cannot remove the renderer's stylistic conventions.

Troubleshooting

• An existing output path is deliberate protection: choose a new path instead of overwriting an audited attempt.
• A duplicate or near-duplicate warning may reveal a real lineage error. Review both scenarios; moving one row to another split does not repair the concept.
• If one class disappears from a cohort, change the preregistered group allocation before freezing rather than reassigning cases after seeing predictions.
• A review hash mismatch means the review covered different bytes. Correct the underlying source and obtain a new bound review; do not bypass the check.
• Never turn an inspected final case into a new validation example while calling the remaining set the original untouched holdout.

Exercises

1. Explain why a complete request and its missing-context counterpart must share a group even though their labels differ.
2. Identify which policy inputs make a quoted command different from a positively requested external operation.
3. Explain what an independently agent-reviewed synthetic corpus can and cannot establish compared with human review of production requests.