# M4SEG evaluation

Evaluation defaults to [PLAN-Lab/PRIMA](https://huggingface.co/PLAN-Lab/PRIMA). Download and prepare [M⁴Seg](https://huggingface.co/datasets/PLAN-Lab/M4Seg) with the [data guide](datasets.md), then set the dataset root, test annotation path, and output directory in `configs/evaluate.yaml`:

```bash
python evaluate.py --config configs/evaluate.yaml
```

Use `--merged_ckpt_path /path/to/downloaded_model` for a local model copy, or set `--hub_revision` to a Hub commit ID. The checkpoint includes the base language model, visual and grounding components, merged LoRA weights, and both tokenizers.

The evaluator supports M4SEG only. It saves one prediction per example, including an explicit empty/error record when model generation fails. Use `--resume` to skip successful predictions from a matching run. Resume checks the annotation, model and tokenizer assets, generation settings, preprocessing setting, source code, and dependency versions against `generation.json`. Existing predictions without matching provenance require a new generation directory; they can still be scored separately. Shard a run with matching `--n_chunks` values and distinct `--chunk_idx` values, each writing to its own output directory. When scoring a parent directory containing completed shards, duplicate example IDs raise an error rather than choosing one output silently.

The semantic metric encoder is **[sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)**, the model used by the paper's metric implementation. This release pins it to revision `1110a243fdf4706b3f48f1d95db1a4f5529b4d41`. It embeds grounded phrases; cosine similarity is computed from normalized embeddings. It is separate from PRIMA's language backbone. The evaluator downloads it automatically when text metrics are enabled, and records the selected model and resolved revision in the metric report. A custom `text_model` uses that model's own revision. Java is required for METEOR.

To score predictions immediately after generation:

```bash
python evaluate.py --config configs/evaluate.yaml \
  --compute_metrics --text_model sentence-transformers/all-MiniLM-L6-v2
```

This computes all seven mask/text metrics and writes `metrics.json` in the output directory. For mask-only scoring without a semantic model or Java, add `--no-metrics_with_text --no-metrics_with_meteor`. For sharded runs, score the completed shards together with the standalone command below.

Score saved predictions separately:

```bash
python -m eval.m4seg /path/to/predictions \
  --gt-json /path/to/evaluation_annotations.json \
  --penalize-missing \
  --with-text --text-model sentence-transformers/all-MiniLM-L6-v2 --with-meteor \
  --output outputs/metrics.json
```

New prediction records store the native height and width of each input image. The scorer can therefore decode polygon annotations without a separate image-size cache. Older predictions can use embedded RLE sizes or the source image paths stored in their outputs. If polygon predictions have neither accessible source images nor native dimensions, supply legacy metadata with `--ann-data /path/to/grounding_annotations`.

Paper-compatible mIoU greedily matches masks, averages matched-pair IoU within each annotated image, then averages over annotated images. Unmatched masks do not enter that matched-pair denominator. Recall and grounded semantic metrics additionally threshold IoU and semantic similarity. SIoU measures phrase-word overlap; METEOR uses cleaned captions and COCO tokenization.

Missing or failed predictions receive zero for mask and phrase scores. METEOR uses annotated images with predicted masks, repeating an example's caption for each eligible image; `meteor_evaluated_images` records this coverage. Report checkpoint/annotation versions, generation settings, coverage, and failures with scores. Expected scores depend on the selected checkpoint and test annotations; no alternative test split is selected automatically.