LTX CI quality gate
The Fast T2V CI test generates five 145-frame, 1920×1088 clips. After pytest closes
the TT devices, the same worker runs vbench_bundle evaluate with up to five CPU
processes, four Torch threads each. Generation and scoring share the existing
35-minute test timeout. The commands live in tests/pipeline_reorg/models_e2e_tests.yaml.
The evaluation policy splits metrics by their input needs:
| Metrics | Input |
|---|---|
| Subject consistency, background consistency, imaging quality | Original clips, with VBench's normal preprocessing |
| Motion smoothness, dynamic degree | 960×544 copies, all 145 frames, original frame rate |
The temporal copies use PyAV in-process, with area downsampling and lossless RGB encoding; source frame timestamps are preserved and no OS command is launched. This reduces sensitivity to small spatial motion artifacts; it is not equivalent to evaluating temporal quality at full resolution. It preserves every seed, frame, metric and the existing mean thresholds. Resize fails if the frame count or frame rate changes. Missing seeds, worker failures, mismatched clip hashes, missing metrics and non-finite scores all fail the gate.
Dynamic degree uses VBench 0.1.5's original RAFT model, frame sampling, flow threshold and required positive-pair count. It stops once the remaining pairs cannot change the binary result. Unlike the spatial reduction, this early exit preserves the reference result exactly; exhaustive small cases test both passing and failing bounds.
On 2026-09-18, the five actual CI clips from run 35334111844 passed this complete gate on g15blx02 in 343 seconds using the CI entry point and the in-process resize:
| Metric | Mean | Required |
|---|---|---|
| Subject consistency | 0.9396 | 0.92 |
| Background consistency | 0.9590 | 0.93 |
| Motion smoothness | 0.9890 | 0.955 |
| Dynamic degree | 0.8 | 0.8 |
| Imaging quality | 0.6831 | 0.645 |
A separate saved passing clip also passed the reduced temporal policy (motion smoothness 0.9911, dynamic degree 1.0). Freezing its frames failed dynamic degree (0.0); alternating inverted frames failed motion smoothness (0.3110).
These measurements cover CPU scoring, not cold model startup. CI must also fit weight conversion, kernel compilation, generation and CLIP inside the total budget.
VBENCH_TEMPORAL_WIDTH=960 selects the CI policy when exporting a bundle.
The manifest records this setting and result files identify the exact manifest.
An exported width of 0, or a local pipeline run without VBENCH_EXPORT_DIR,
uses the original full-resolution evaluation. Evaluator weights come from the
worker's existing VBENCH_CACHE_DIR, TORCH_HOME and CLIP caches.