File size: 5,983 Bytes
b7e771c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
language:
- en
tags:
- neuroscience
- animal-behavior
- vision-language-model
- mousegpt
---
# MouseGPT
**Vision-language models for mouse behavior analysis**
[](https://arxiv.org/abs/2503.10212)
[](https://github.com/Ten1o/MouseGPT)
[](https://huggingface.co/datasets/Ten1o/MouseGPT-examples)
MouseGPT combines mouse images with pose-derived kinematic context to generate natural-language descriptions of posture and movement. These descriptions support semantic retrieval, behavior classification and clustering, and conversational exploration of experimental sessions.
This is the model repository for **[MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis](https://arxiv.org/abs/2503.10212)**. The complete preprocessing, training, deployment, and analysis code is maintained on [GitHub](https://github.com/Ten1o/MouseGPT).
> **Checkpoint availability:** Model uploads are pending. No checkpoint files are currently available in this repository. You can use the [example data](https://huggingface.co/datasets/Ten1o/MouseGPT-examples) to inspect VLM inputs and explore a prepared analysis session before the model weights are uploaded.
## Model family
| Model | Foundation model | Purpose | Status |
| --- | --- | --- | --- |
| **MouseGPT-Large** | InternVL2-Llama3-76B | Detailed vision-language descriptions of mouse behavior | Upload pending |
| **MouseGPT-Lite** | MiniCPM-V 2.6, 8B | Behavior description with a smaller vision-language model | Upload pending |
| **Kinematic estimation models** | Animal detector and keypoint models | Visual measurements for pose reconstruction and kinematic preprocessing | Upload pending |
## TODO: model release
- [ ] Upload the MouseGPT-Large checkpoint and required configuration/tokenizer files.
- [ ] Upload the MouseGPT-Lite checkpoint and required configuration/tokenizer files.
- [ ] Upload the detector and keypoint checkpoints with their matching model configurations.
- [ ] Add checkpoint-specific download and loading instructions, revisions, checksums, and applicable model-license information.
This checklist will be updated as the files become available. Deployment requires a complete checkpoint and its matching configuration files.
## Getting started
### Inspect the example inputs
The small VLM bundle can be downloaded and validated without a GPU or model checkpoint:
```bash
git clone https://github.com/Ten1o/MouseGPT.git
cd MouseGPT
python -m pip install -r requirements-assets.txt
python scripts/download_assets.py vlm-demo
python 2_VLM/prompt_test/client_test.py --validate-only
```
The bundle contains ten images and ten prompt/response records. Each record includes a `query`, an `images` list, and an illustrative `response`; image paths are relative to the JSONL file. Stored responses are not supplied to the model during inference.
### Run a checkpoint
Once the required checkpoint is available, follow the [vision-language model guide](https://github.com/Ten1o/MouseGPT/blob/main/2_VLM/readme.md) to install the bundled SWIFT integration, start a model server, and run the inference client. The guide documents separate deployment configurations for Large and Lite. Hardware requirements depend on the checkpoint, precision, context length, and inference backend.
The [keypoint guide](https://github.com/Ten1o/MouseGPT/blob/main/1_keypoints/readme.md) describes detector and pose inference, camera calibration inputs, and multi-view reconstruction. The [examples guide](https://github.com/Ten1o/MouseGPT/blob/main/examples/README.md) also explains how to use a prepared session for downstream analysis without rerunning pose estimation or VLM inference.
## Inputs and interpretation
The standard MouseGPT workflow combines images with kinematic summaries and generates structured descriptions of the head, forelimbs, hindlimbs, torso, and overall behavior. Input preparation must preserve image/frame alignment and use the coordinate units and frame-rate conventions documented in the VLM guide.
Descriptions and downstream labels are model predictions. Evaluate their suitability for the recording conditions, animals, and behavioral definitions in your study. Cohort-level interpretation requires the relevant experimental groups, controls, and statistical analysis.
## Code and data
- **Code and tutorials:** [Ten1o/MouseGPT on GitHub](https://github.com/Ten1o/MouseGPT).
- **Example images and prepared session:** [MouseGPT-examples](https://huggingface.co/datasets/Ten1o/MouseGPT-examples).
- **Full raw data:** available upon request from the corresponding authors listed in the paper.
Original MouseGPT-authored implementation material is covered by [CC BY-NC-ND 4.0 with explicit component exclusions](https://github.com/Ten1o/MouseGPT/blob/main/LICENSE). The [MMYOLO detector components and copied configurations](https://github.com/Ten1o/MouseGPT/blob/main/LICENSES/GPL-SCOPE.md), including MouseGPT modifications to them, are distributed under GPLv3. Other third-party code retains its applicable license notices. These code licenses do not assign terms to checkpoints; checkpoint-specific license information will accompany the model uploads.
## Citation
Teng Xu et al. **MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis.** arXiv:2503.10212, 2025. [Read the paper](https://arxiv.org/abs/2503.10212) or copy the [BibTeX citation](https://github.com/Ten1o/MouseGPT#citation).
## Support
For installation, inference, or compatibility questions, open a [GitHub issue](https://github.com/Ten1o/MouseGPT/issues) with the model variant, checkpoint revision, environment, and a reproducible example where possible.
|