code/docs/EVALUATION.md

Inference, TVS, and scoring

Run from the repository root after installation and data preparation. The code separates model inference, view-policy composition, and official scoring.

1. Generate recent-view proposals

Create a local configuration from the public template and your own assets:

python -m scripts.configure_evaluation_protocol \
  --streaming-samples data/streamingbench/queries.jsonl \
  --ovo-samples data/ovo/queries.jsonl \
  --processor-config checkpoints/viss/locked_processor/preprocessor_config.json \
  --output configs/evaluation/recent4_full_v1.json

The generated file contains local paths and file identities. It remains private and is excluded from the source archive. Then validate coverage:

python -m scripts.validate_evaluation_protocol \
  --protocol configs/evaluation/recent4_full_v1.json

This check requires the external benchmark exports. It is not a data-free installation test.

Use the command in the main README with the prepared checkpoint and locked processor. The runner accepts --benchmark streamingbench or --benchmark ovo; annotations, query exports, and media must match.

Use --record-choice-confidence when the proposals will be passed to TVS. For a reproducibility run, supply --model-manifest with the matching model identity artifact. The same identity must be retained across all views. The runner also supports --start-index and --end-index for sharding.

The optional multi-GPU launcher is:

bash scripts/run_recent4_single_benchmark.sh \
  REPO MODEL MODEL_MANIFEST OUTPUT PYTHON BENCHMARK GPU_CSV MODEL_ID

It additionally requires the benchmark paths documented by its environment variable checks. Start with the Python entry point when preparing new assets.

2. Apply the TVS acquisition and commitment policy

The event policy is causalbudget20; the compact-history policy is causalhybrid32. Use the same checkpoint and the view encodings prescribed by your protocol. Selecting a policy name alone does not fix image resolution or the processor configuration.

Given recent-view predictions and a frozen calibration artifact:

python -m experiments.select_uncertainty_escalation_uids \
  --recent outputs/recent.json \
  --threshold data/calibration/thresholds.json \
  --samples data/benchmark/queries.jsonl \
  --output-uids outputs/event_uids.json \
  --output-samples outputs/event_queries.jsonl \
  --output-audit outputs/selection.json

Run event-view inference on the selected queries. Then compose the first two stages:

python -m experiments.materialize_uncertainty_temporal_agent \
  --recent outputs/recent.json \
  --temporal outputs/event.json \
  --selection-audit outputs/selection.json \
  --threshold-audit data/calibration/thresholds.json \
  --output outputs/two_view.json

Once the required compact-history proposals are available:

python -m experiments.materialize_confidence_gated_third_view_agent \
  --recent outputs/recent.json \
  --event outputs/event.json \
  --base-agent outputs/two_view.json \
  --third-view outputs/history.json \
  --threshold-audit data/calibration/thresholds.json \
  --output outputs/tvs.json

These commands name user-generated files; no prediction or score artifacts are shipped. Materialization combines existing proposals and does not call the VLM. materialize_tvs_view_oracle.py uses reference correctness for analysis and is not the deployed selection policy.

3. Score saved predictions

python -m scripts.score_official_benchmark_predictions \
  --benchmark streamingbench \
  --annotation data/streamingbench/annotations.json \
  --predictions outputs/tvs.json \
  --official-root external/StreamingBench \
  --output-json outputs/streamingbench_score.json

Use --benchmark ovo and the corresponding official assets for OVO-Bench. The adapter checks query coverage and invokes the original scorer. For OVO, the primary metric is the task-then-group macro average. The official scorer may have additional dependencies; install those from its own repository.

Generated predictions and scores are your run outputs. They are excluded from version control and from the source-release archive.