M4SEG evaluation
Evaluation defaults to PLAN-Lab/PRIMA. Download and prepare M⁴Seg with the data guide, then set the dataset root, test annotation path, and output directory in configs/evaluate.yaml:
python evaluate.py --config configs/evaluate.yaml
Use --merged_ckpt_path /path/to/downloaded_model for a local model copy, or set --hub_revision to a Hub commit ID. The checkpoint includes the base language model, visual and grounding components, merged LoRA weights, and both tokenizers.
The evaluator supports M4SEG only. It saves one prediction per example, including an explicit empty/error record when model generation fails. Use --resume to skip successful predictions from a matching run. Resume checks the annotation, model and tokenizer assets, generation settings, preprocessing setting, source code, and dependency versions against generation.json. Existing predictions without matching provenance require a new generation directory; they can still be scored separately. Shard a run with matching --n_chunks values and distinct --chunk_idx values, each writing to its own output directory. When scoring a parent directory containing completed shards, duplicate example IDs raise an error rather than choosing one output silently.
The semantic metric encoder is sentence-transformers/all-MiniLM-L6-v2, the model used by the paper's metric implementation. This release pins it to revision 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. It embeds grounded phrases; cosine similarity is computed from normalized embeddings. It is separate from PRIMA's language backbone. The evaluator downloads it automatically when text metrics are enabled, and records the selected model and resolved revision in the metric report. A custom text_model uses that model's own revision. Java is required for METEOR.
To score predictions immediately after generation:
python evaluate.py --config configs/evaluate.yaml \
--compute_metrics --text_model sentence-transformers/all-MiniLM-L6-v2
This computes all seven mask/text metrics and writes metrics.json in the output directory. For mask-only scoring without a semantic model or Java, add --no-metrics_with_text --no-metrics_with_meteor. For sharded runs, score the completed shards together with the standalone command below.
Score saved predictions separately:
python -m eval.m4seg /path/to/predictions \
--gt-json /path/to/evaluation_annotations.json \
--penalize-missing \
--with-text --text-model sentence-transformers/all-MiniLM-L6-v2 --with-meteor \
--output outputs/metrics.json
New prediction records store the native height and width of each input image. The scorer can therefore decode polygon annotations without a separate image-size cache. Older predictions can use embedded RLE sizes or the source image paths stored in their outputs. If polygon predictions have neither accessible source images nor native dimensions, supply legacy metadata with --ann-data /path/to/grounding_annotations.
Paper-compatible mIoU greedily matches masks, averages matched-pair IoU within each annotated image, then averages over annotated images. Unmatched masks do not enter that matched-pair denominator. Recall and grounded semantic metrics additionally threshold IoU and semantic similarity. SIoU measures phrase-word overlap; METEOR uses cleaned captions and COCO tokenization.
Missing or failed predictions receive zero for mask and phrase scores. METEOR uses annotated images with predicted masks, repeating an example's caption for each eligible image; meteor_evaluated_images records this coverage. Report checkpoint/annotation versions, generation settings, coverage, and failures with scores. Expected scores depend on the selected checkpoint and test annotations; no alternative test split is selected automatically.