README.md

---
language:
- en
tags:
- neuroscience
- animal-behavior
- vision-language-model
- mousegpt
---

# MouseGPT

**Vision-language models for mouse behavior analysis**

[![Paper — arXiv](https://img.shields.io/badge/arXiv-2503.10212-B31B1B?logo=arxiv&logoColor=white)](https://arxiv.org/abs/2503.10212)
[![GitHub — Code](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/Ten1o/MouseGPT)
[![Hugging Face — Example data](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Example%20data-FFD21E)](https://huggingface.co/datasets/Ten1o/MouseGPT-examples)

MouseGPT combines mouse images with pose-derived kinematic context to generate natural-language descriptions of posture and movement. These descriptions support semantic retrieval, behavior classification and clustering, and conversational exploration of experimental sessions.

This is the model repository for **[MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis](https://arxiv.org/abs/2503.10212)**. The complete preprocessing, training, deployment, and analysis code is maintained on [GitHub](https://github.com/Ten1o/MouseGPT).

> **Checkpoint availability:** Model uploads are pending. No checkpoint files are currently available in this repository. You can use the [example data](https://huggingface.co/datasets/Ten1o/MouseGPT-examples) to inspect VLM inputs and explore a prepared analysis session before the model weights are uploaded.

## Model family

| Model | Foundation model | Purpose | Status |
| --- | --- | --- | --- |
| **MouseGPT-Large** | InternVL2-Llama3-76B | Detailed vision-language descriptions of mouse behavior | Upload pending |
| **MouseGPT-Lite** | MiniCPM-V 2.6, 8B | Behavior description with a smaller vision-language model | Upload pending |
| **Kinematic estimation models** | Animal detector and keypoint models | Visual measurements for pose reconstruction and kinematic preprocessing | Upload pending |

## TODO: model release

- [ ] Upload the MouseGPT-Large checkpoint and required configuration/tokenizer files.
- [ ] Upload the MouseGPT-Lite checkpoint and required configuration/tokenizer files.
- [ ] Upload the detector and keypoint checkpoints with their matching model configurations.
- [ ] Add checkpoint-specific download and loading instructions, revisions, checksums, and applicable model-license information.

This checklist will be updated as the files become available. Deployment requires a complete checkpoint and its matching configuration files.

## Getting started

### Inspect the example inputs

The small VLM bundle can be downloaded and validated without a GPU or model checkpoint:

```bash
git clone https://github.com/Ten1o/MouseGPT.git
cd MouseGPT
python -m pip install -r requirements-assets.txt
python scripts/download_assets.py vlm-demo
python 2_VLM/prompt_test/client_test.py --validate-only
```

The bundle contains ten images and ten prompt/response records. Each record includes a `query`, an `images` list, and an illustrative `response`; image paths are relative to the JSONL file. Stored responses are not supplied to the model during inference.

### Run a checkpoint

Once the required checkpoint is available, follow the [vision-language model guide](https://github.com/Ten1o/MouseGPT/blob/main/2_VLM/readme.md) to install the bundled SWIFT integration, start a model server, and run the inference client. The guide documents separate deployment configurations for Large and Lite. Hardware requirements depend on the checkpoint, precision, context length, and inference backend.

The [keypoint guide](https://github.com/Ten1o/MouseGPT/blob/main/1_keypoints/readme.md) describes detector and pose inference, camera calibration inputs, and multi-view reconstruction. The [examples guide](https://github.com/Ten1o/MouseGPT/blob/main/examples/README.md) also explains how to use a prepared session for downstream analysis without rerunning pose estimation or VLM inference.

## Inputs and interpretation

The standard MouseGPT workflow combines images with kinematic summaries and generates structured descriptions of the head, forelimbs, hindlimbs, torso, and overall behavior. Input preparation must preserve image/frame alignment and use the coordinate units and frame-rate conventions documented in the VLM guide.

Descriptions and downstream labels are model predictions. Evaluate their suitability for the recording conditions, animals, and behavioral definitions in your study. Cohort-level interpretation requires the relevant experimental groups, controls, and statistical analysis.

## Code and data

- **Code and tutorials:** [Ten1o/MouseGPT on GitHub](https://github.com/Ten1o/MouseGPT).
- **Example images and prepared session:** [MouseGPT-examples](https://huggingface.co/datasets/Ten1o/MouseGPT-examples).
- **Full raw data:** available upon request from the corresponding authors listed in the paper.

Original MouseGPT-authored implementation material is covered by [CC BY-NC-ND 4.0 with explicit component exclusions](https://github.com/Ten1o/MouseGPT/blob/main/LICENSE). The [MMYOLO detector components and copied configurations](https://github.com/Ten1o/MouseGPT/blob/main/LICENSES/GPL-SCOPE.md), including MouseGPT modifications to them, are distributed under GPLv3. Other third-party code retains its applicable license notices. These code licenses do not assign terms to checkpoints; checkpoint-specific license information will accompany the model uploads.

## Citation

Teng Xu et al. **MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis.** arXiv:2503.10212, 2025. [Read the paper](https://arxiv.org/abs/2503.10212) or copy the [BibTeX citation](https://github.com/Ten1o/MouseGPT#citation).

## Support

For installation, inference, or compatibility questions, open a [GitHub issue](https://github.com/Ten1o/MouseGPT/issues) with the model variant, checkpoint revision, environment, and a reproducible example where possible.