| --- |
| license: apache-2.0 |
| base_model: allenai/Molmo2-8B |
| pipeline_tag: video-text-to-text |
| library_name: transformers |
| tags: |
| - molmo2 |
| - video |
| - object-tracking |
| - ecology |
| datasets: |
| - tidalove/cfc-track-instruction |
| - perona-lab/cfc26 |
| --- |
| |
| # Molmo2Fish |
|
|
| Molmo2Fish is [Molmo2-8B](https://github.com/allenai/molmo2) fine-tuned to **track fish in |
| sonar video, and to edit those tracks in response to natural-language feedback**. |
|
|
| Track correction is treated as a conversation: the model is shown a set of existing tracks — |
| its own earlier predictions, another tracker's output, or corrupted ground truth — is told in |
| words what is wrong with them, and returns a repaired set of tracks. |
|
|
| | | | |
| |---|---| |
| | Code | [github.com/tidalove/molmo2fish](https://github.com/tidalove/molmo2fish) | |
| | Data | [tidalove/cfc-track-instruction](https://huggingface.co/datasets/tidalove/cfc-track-instruction) | |
| | Source video | [Caltech Fish Counting](https://huggingface.co/datasets/perona-lab/cfc26) | |
|
|
| ## Files |
|
|
| | file | what it is | |
| |---|---| |
| | `*.safetensors`, `config.json`, … | HuggingFace-format weights at the repo root — what vLLM and `launch_scripts/hf_eval.py` consume | |
| | `Molmo2Fish-step420-raw.tar` | the raw training checkpoint (sharded model + optimizer state), for resuming fine-tuning | |
|
|
| ## Training |
|
|
| Rank 64 LoRA fine-tuning of Molmo2-8B on the full CFC mixture, step 420. Adapters on all |
| three components — LLM, ViT, and connector — with the base weights frozen: |
|
|
| ```bash |
| torchrun --nproc-per-node=8 launch_scripts/sft.py /path/to/Molmo2-8B cfc_correction \ |
| --lora_llm --lora_vit --lora_connector --lora_rank 64 \ |
| --save_folder=/path/to/save/folder |
| ``` |
|
|
| The `cfc_correction` mixture combines pure tracking, targeted correction, synthetically |
| corrupted correction, correction of real model predictions at two quality levels |
| (`molmo_high` / `molmo_low`), and text-only correction. See the |
| [dataset card](https://huggingface.co/datasets/tidalove/cfc-track-instruction) for what each |
| config contains. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{molmo2fish, |
| title={Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance}, |
| author={Kai van Brunt and Justin Kay and Sara Beery}, |
| year={2026}, |
| url={https://arxiv.org/abs/2608.18602} |
| } |
| ``` |
|
|
