Local perf / accuracy target validation (opt-in)

In CI, model e2e tests are warning-only for perf/accuracy: verify_perf / verify_accuracy log drift but never abort, so the run always writes complete benchmark JSON. A separate CI step then enforces the targets in models/model_targets.yaml via .github/scripts/utils/validate_perf_targets.py.

You can run that same enforcement locally as an opt-in pytest gate.

How to enable

Run your model demo/test with VALIDATE_PERF_TARGETS=1:

VALIDATE_PERF_TARGETS=1 pytest models/tt_transformers/demo/simple_text_demo.py -k "..."

A pytest_sessionfinish hook (in models/conftest.py) runs after the session — once benchmark JSON is fully written — loads the CI validator, and fails the session if any measured metric misses its target. It is off by default, so a normal pytest run is unaffected.

Requirements

The test must emit benchmark data (it calls create_benchmark_data(...).save(...), writing generated/benchmark_data/complete_run_*.json).
An entry for your model + sku must exist in models/model_targets.yaml.

Knobs

Env var	Default	Effect
`VALIDATE_PERF_TARGETS`	`0`	`1` enables the local gate.
`VALIDATE_PERF_TARGETS_STRICT_MISSING`	`0`	`1` also fails when no target entry exists (TODO/missing); otherwise missing targets only warn.
`VALIDATE_PERF_TARGETS_SKU`	(auto)	Override the SKU. If unset, it is auto-detected from the device, falling back to the `card_type` in the benchmark JSON.

Notes

Same comparison logic as CI (single source of truth) — a local pass means a CI pass for those metrics.
Validation runs once at session end, so it never truncates a test mid-run.