MouseGPT

Vision-language models for mouse behavior analysis

MouseGPT combines mouse images with pose-derived kinematic context to generate natural-language descriptions of posture and movement. These descriptions support semantic retrieval, behavior classification and clustering, and conversational exploration of experimental sessions.

This is the model repository for MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis. The complete preprocessing, training, deployment, and analysis code is maintained on GitHub.

Checkpoint availability: Model uploads are pending. No checkpoint files are currently available in this repository. You can use the example data to inspect VLM inputs and explore a prepared analysis session before the model weights are uploaded.

Model family

Model	Foundation model	Purpose	Status
MouseGPT-Large	InternVL2-Llama3-76B	Detailed vision-language descriptions of mouse behavior	Upload pending
MouseGPT-Lite	MiniCPM-V 2.6, 8B	Behavior description with a smaller vision-language model	Upload pending
Kinematic estimation models	Animal detector and keypoint models	Visual measurements for pose reconstruction and kinematic preprocessing	Upload pending

TODO: model release

• Upload the MouseGPT-Large checkpoint and required configuration/tokenizer files.
• Upload the MouseGPT-Lite checkpoint and required configuration/tokenizer files.
• Upload the detector and keypoint checkpoints with their matching model configurations.
• Add checkpoint-specific download and loading instructions, revisions, checksums, and applicable model-license information.

This checklist will be updated as the files become available. Deployment requires a complete checkpoint and its matching configuration files.

Getting started

Inspect the example inputs

The small VLM bundle can be downloaded and validated without a GPU or model checkpoint:

git clone https://github.com/Ten1o/MouseGPT.git
cd MouseGPT
python -m pip install -r requirements-assets.txt
python scripts/download_assets.py vlm-demo
python 2_VLM/prompt_test/client_test.py --validate-only

The bundle contains ten images and ten prompt/response records. Each record includes a query, an images list, and an illustrative response; image paths are relative to the JSONL file. Stored responses are not supplied to the model during inference.

Run a checkpoint

Once the required checkpoint is available, follow the vision-language model guide to install the bundled SWIFT integration, start a model server, and run the inference client. The guide documents separate deployment configurations for Large and Lite. Hardware requirements depend on the checkpoint, precision, context length, and inference backend.

The keypoint guide describes detector and pose inference, camera calibration inputs, and multi-view reconstruction. The examples guide also explains how to use a prepared session for downstream analysis without rerunning pose estimation or VLM inference.

Inputs and interpretation

The standard MouseGPT workflow combines images with kinematic summaries and generates structured descriptions of the head, forelimbs, hindlimbs, torso, and overall behavior. Input preparation must preserve image/frame alignment and use the coordinate units and frame-rate conventions documented in the VLM guide.

Descriptions and downstream labels are model predictions. Evaluate their suitability for the recording conditions, animals, and behavioral definitions in your study. Cohort-level interpretation requires the relevant experimental groups, controls, and statistical analysis.

Code and data

• Code and tutorials: Ten1o/MouseGPT on GitHub.
• Example images and prepared session: MouseGPT-examples.
• Full raw data: available upon request from the corresponding authors listed in the paper.

Original MouseGPT-authored implementation material is covered by CC BY-NC-ND 4.0 with explicit component exclusions. The MMYOLO detector components and copied configurations, including MouseGPT modifications to them, are distributed under GPLv3. Other third-party code retains its applicable license notices. These code licenses do not assign terms to checkpoints; checkpoint-specific license information will accompany the model uploads.

Citation

Teng Xu et al. MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis. arXiv:2503.10212, 2025. Read the paper or copy the BibTeX citation.

Support

For installation, inference, or compatibility questions, open a GitHub issue with the model variant, checkpoint revision, environment, and a reproducible example where possible.