# Local perf / accuracy target validation (opt-in)

In CI, model e2e tests are **warning-only** for perf/accuracy: `verify_perf` /
`verify_accuracy` log drift but never abort, so the run always writes complete
benchmark JSON. A separate CI step then enforces the targets in
[`models/model_targets.yaml`](../../model_targets.yaml) via
`.github/scripts/utils/validate_perf_targets.py`.

You can run that **same** enforcement locally as an opt-in pytest gate.

## How to enable

Run your model demo/test with `VALIDATE_PERF_TARGETS=1`:

```bash
VALIDATE_PERF_TARGETS=1 pytest models/tt_transformers/demo/simple_text_demo.py -k "..."
```

A `pytest_sessionfinish` hook (in `models/conftest.py`) runs **after** the
session — once benchmark JSON is fully written — loads the CI validator, and
**fails the session** if any measured metric misses its target. It is off by
default, so a normal `pytest` run is unaffected.

## Requirements

- The test must emit benchmark data (it calls `create_benchmark_data(...).save(...)`,
  writing `generated/benchmark_data/complete_run_*.json`).
- An entry for your `model` + `sku` must exist in `models/model_targets.yaml`.

## Knobs

| Env var | Default | Effect |
| --- | --- | --- |
| `VALIDATE_PERF_TARGETS` | `0` | `1` enables the local gate. |
| `VALIDATE_PERF_TARGETS_STRICT_MISSING` | `0` | `1` also fails when no target entry exists (TODO/missing); otherwise missing targets only warn. |
| `VALIDATE_PERF_TARGETS_SKU` | _(auto)_ | Override the SKU. If unset, it is auto-detected from the device, falling back to the `card_type` in the benchmark JSON. |

## Notes

- Same comparison logic as CI (single source of truth) — a local pass means a CI pass for those metrics.
- Validation runs once at session end, so it never truncates a test mid-run.