tkclam/deeperfly-models

deeperfly detectors

Pretrained 2D keypoint detectors for deeperfly, markerless 3D pose estimation of Drosophila from multi-camera rigs. They include the detectors the example run configs in the deeperfly repository name, all trained in the Ramdya lab at EPFL on hand labels from two rigs.

Folder	Rig	What it is	Backbone	Trained on
261008_mvt_dinov3b_384_octacam_fly38	octacam: eight cameras around a tethered fly on a ball	the octacam detector (round 14): multiview transformer, all 38 fly38 keypoints in every view, 384x768 window per view	DINOv3 ViT-B/16	62 recordings, 4,703 labeled items
261001_mvt_dinov3b_512_hexaview_fly38	hexaview: two cameras, each seeing a freely walking fly directly and through two prisms (six views)	the hexaview detector (round 13), for one fly: multiview transformer, 38 keypoints, 512x512 window centered on the fly in each frame; run it at [pose2d] tightness = 1.2	DINOv3 ViT-B/16	17 recordings, 3,240 labeled items
260920_fast_efficientvit_b1_352_hexaview_locator	hexaview	full-frame locator that places the hexaview detector's windows	EfficientViT-B1	9 recordings, 824 labeled items
261007_mvt_dinov3b_512_hexaview_pair_fly38	hexaview, two flies	the two-fly hexaview detector (round x4): one 512x512 window per fly and view, each marking the other fly's center; needs deeperfly from 2026-10-09 on, which draws the mark (ADR-0148); run it at [pose2d] tightness = 1.2	DINOv3 ViT-B/16	102 projects (12 real two-fly recordings with 99 labeled frames, single-fly recordings, synthetic composites, pseudo-labels), 186,330 items
261007_fast_efficientvit_b1_352_hexaview_pair_locator	hexaview, two flies	full-frame locator that finds both flies in every view, for the two-fly detector	EfficientViT-B1	85 projects, 28,334 items

A real-time octacam detector is in training and will be added here.

Expected input

Each detector expects frames like these (the first two are the example clips in the deeperfly repository); the dashed boxes are its windows, and the skeleton is what it predicts. Pretrained models in the deeperfly docs describes both rigs.

octacam

hexaview, one fly

hexaview, two flies

Each folder holds the checkpoint (.pth), the skeleton it predicts (fly38.toml), the training recipe (config.toml), the training report (report.json), SHA256SUMS, and a README.md written when the model was exported. The paths in those READMEs are the lab's; use the instructions below instead.

Download

From the root of a deeperfly clone, where the example configs expect them:

uv run hf download tkclam/deeperfly-models --local-dir models

One model only, such as the octacam detector:

uv run hf download tkclam/deeperfly-models --local-dir models \
  --include "261008_mvt_dinov3b_384_octacam_fly38/*"

Check a download with sha256sum -c SHA256SUMS inside its folder.

Use

Name the checkpoint in a run config's [pose2d] table:

[pose2d]
weights = ["models/261008_mvt_dinov3b_384_octacam_fly38/mvt_dinov3b_384_octacam_fly38.pth"]

Paths resolve against the config file. See the deeperfly quickstart for a full run on the bundled example clips.

Scope and limitations

The detectors were trained on recordings from two rigs in one lab, with every label used for training (no held-out set). The octacam detector's twin, trained with four recordings held out, scores 3.25 px median and 4.49 px mean on their 9,269 labeled cells (93.7% within 10 px). On the bundled example clips, the octacam detector reprojects at 1.46 px median after bundle adjustment and the hexaview detector at 3.00 px median through the recording's calibration; that is agreement across views, not error against labels. The two-fly detector's twin from the round before, trained with recordings held out, put 4.0% of keypoints more than 15 px from the label in the direct views and 9.2% in the prism views on 35 held-out two-fly frames (5.8% and 14.7% for the single-fly detector's twin). They are not expected to transfer to other rigs, magnifications, or lighting without retraining. deeperfly train fits a detector on your own labels, and these checkpoints can serve as a starting point.

License

• The DINOv3-based detectors are derivatives of Meta's DINOv3 weights and are distributed under the DINOv3 License, including its use restrictions.
• The EfficientViT-based locators are distributed under the Apache License 2.0, the license of its ImageNet-pretrained backbone.

Citation

Please cite deeperfly (see its README) and DINOv3:

@misc{simeoni2025dinov3,
  title         = {{DINOv3}},
  author        = {Sim{\'e}oni, Oriane and Vo, Huy V. and Seitzer, Maximilian and Baldassarre, Federico and others},
  year          = {2025},
  eprint        = {2508.10104},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
}