| # Evaluation |
|
|
| EUMU evaluation uses the packaged `model/` directory and the assembled tagging |
| checkpoint installed at `model/task_a_heads.pt`. |
|
|
| ## Tagging Threshold Search |
|
|
| After training and assembling the initial final tagging checkpoint, run: |
|
|
| ```bash |
| VAL_ROOT=data/val_data \ |
| bash training/run_search_tagging_thresholds.sh |
| ``` |
|
|
| Outputs: |
|
|
| ```text |
| training_runs/threshold_search/scene_summary.json |
| training_runs/threshold_search/event_summary.json |
| ``` |
|
|
| Then rebuild the final tagging checkpoint with the searched thresholds: |
|
|
| ```bash |
| SCENE_SUMMARY=training_runs/threshold_search/scene_summary.json \ |
| EVENT_SUMMARY=training_runs/threshold_search/event_summary.json \ |
| bash training/run_assemble_tagging_heads.sh --install-to-model |
| ``` |
|
|
| The assembly script also contains the final threshold defaults, so the packaged |
| checkpoint can be rebuilt even if the intermediate threshold summaries are not |
| available. |
|
|
| ## Prediction Generation |
|
|
| Generate evaluation predictions for an image folder: |
|
|
| ```bash |
| python evaluation/generate_predictions.py \ |
| --image-dir /path/to/eval/images \ |
| --ids-file /path/to/image_ids.txt \ |
| --out predictions.generated.json |
| ``` |
|
|
| If no `--ids-file` is provided, image ids are taken from image filename stems |
| and processed in sorted order. |
|
|
| The output schema is: |
|
|
| ```json |
| { |
| "schema_version": "1.0", |
| "model_info": { |
| "parameters_m": 239.169, |
| "gflops_224": 23.947, |
| "peak_memory_gb": 4.5 |
| }, |
| "predictions": [] |
| } |
| ``` |
|
|
| The repository root `predictions.json` is the submitted reference output. |
| Detection and captioning are reproduced through prompt-based Florence inference |
| and config-driven post-processing; no additional detection or captioning |
| training checkpoint is required. |
|
|
