docs/eval.md

Evaluation

EUMU evaluation uses the packaged model/ directory and the assembled tagging checkpoint installed at model/task_a_heads.pt.

Tagging Threshold Search

After training and assembling the initial final tagging checkpoint, run:

VAL_ROOT=data/val_data \
bash training/run_search_tagging_thresholds.sh

Outputs:

training_runs/threshold_search/scene_summary.json
training_runs/threshold_search/event_summary.json

Then rebuild the final tagging checkpoint with the searched thresholds:

SCENE_SUMMARY=training_runs/threshold_search/scene_summary.json \
EVENT_SUMMARY=training_runs/threshold_search/event_summary.json \
bash training/run_assemble_tagging_heads.sh --install-to-model

The assembly script also contains the final threshold defaults, so the packaged checkpoint can be rebuilt even if the intermediate threshold summaries are not available.

Prediction Generation

Generate evaluation predictions for an image folder:

python evaluation/generate_predictions.py \
  --image-dir /path/to/eval/images \
  --ids-file /path/to/image_ids.txt \
  --out predictions.generated.json

If no --ids-file is provided, image ids are taken from image filename stems and processed in sorted order.

The output schema is:

{
  "schema_version": "1.0",
  "model_info": {
    "parameters_m": 239.169,
    "gflops_224": 23.947,
    "peak_memory_gb": 4.5
  },
  "predictions": []
}

The repository root predictions.json is the submitted reference output. Detection and captioning are reproduced through prompt-based Florence inference and config-driven post-processing; no additional detection or captioning training checkpoint is required.