List the round 14 octacam detector as the current one
Browse files
README.md
CHANGED
|
@@ -17,11 +17,12 @@ tags:
|
|
| 17 |
|
| 18 |
# deeperfly detectors
|
| 19 |
|
| 20 |
-
Pretrained 2D keypoint detectors for [deeperfly](https://github.com/NeLy-EPFL/deeperfly), markerless 3D pose estimation of *Drosophila* from multi-camera rigs. They
|
| 21 |
|
| 22 |
| Folder | Rig | What it is | Backbone | Trained on |
|
| 23 |
| --- | --- | --- | --- | --- |
|
| 24 |
-
| `
|
|
|
|
| 25 |
| `260927_mvt_dinov3b_512_hexaview_fly38` | hexaview: two cameras, each seeing a freely walking fly directly and through two prisms (six views) | multiview transformer, 38 keypoints, 512x512 window centered on the fly in each frame | DINOv3 ViT-B/16 | 16 recordings, 3,018 labeled items |
|
| 26 |
| `260920_fast_efficientvit_b1_352_hexaview_locator` | hexaview | full-frame locator that places the hexaview detector's windows | EfficientViT-B1 | 9 recordings, 824 labeled items |
|
| 27 |
|
|
@@ -35,11 +36,11 @@ From the root of a deeperfly clone, where the example configs expect them:
|
|
| 35 |
uv run hf download tkclam/deeperfly-models --local-dir models
|
| 36 |
```
|
| 37 |
|
| 38 |
-
One model only:
|
| 39 |
|
| 40 |
```bash
|
| 41 |
uv run hf download tkclam/deeperfly-models --local-dir models \
|
| 42 |
-
--include "
|
| 43 |
```
|
| 44 |
|
| 45 |
Check a download with `sha256sum -c SHA256SUMS` inside its folder.
|
|
@@ -50,18 +51,18 @@ Name the checkpoint in a run config's `[pose2d]` table:
|
|
| 50 |
|
| 51 |
```toml
|
| 52 |
[pose2d]
|
| 53 |
-
weights = ["models/
|
| 54 |
```
|
| 55 |
|
| 56 |
Paths resolve against the config file. See the [deeperfly quickstart](https://nely-epfl.github.io/deeperfly/stable/quickstart/) for a full run on the bundled example clips.
|
| 57 |
|
| 58 |
## Scope and limitations
|
| 59 |
|
| 60 |
-
The detectors were trained on recordings from two rigs in one lab, with every label used for training (no held-out set). On the bundled example clips, the octacam detector reprojects at 1.
|
| 61 |
|
| 62 |
## License
|
| 63 |
|
| 64 |
-
- The
|
| 65 |
- The EfficientViT-based locator is distributed under the [Apache License 2.0](LICENSE-apache-2.0), the license of its ImageNet-pretrained backbone.
|
| 66 |
|
| 67 |
## Citation
|
|
|
|
| 17 |
|
| 18 |
# deeperfly detectors
|
| 19 |
|
| 20 |
+
Pretrained 2D keypoint detectors for [deeperfly](https://github.com/NeLy-EPFL/deeperfly), markerless 3D pose estimation of *Drosophila* from multi-camera rigs. They include the detectors the example run configs in the deeperfly repository name, all trained in the [Ramdya lab](https://www.epfl.ch/labs/ramdya-lab/) at EPFL on hand labels from two rigs.
|
| 21 |
|
| 22 |
| Folder | Rig | What it is | Backbone | Trained on |
|
| 23 |
| --- | --- | --- | --- | --- |
|
| 24 |
+
| `261008_mvt_dinov3b_384_octacam_fly38` | octacam: eight cameras around a tethered fly on a ball | **the current octacam detector** (round 14): multiview transformer, all 38 `fly38` keypoints in every view, 384x768 window per view | DINOv3 ViT-B/16 | 62 recordings, 4,703 labeled items |
|
| 25 |
+
| `260927_mvt_dinov3b_384_octacam_fly38` | octacam | the previous octacam detector (round 11), same architecture; the deeperfly repository's example config names it | DINOv3 ViT-B/16 | 56 recordings, 4,488 labeled items |
|
| 26 |
| `260927_mvt_dinov3b_512_hexaview_fly38` | hexaview: two cameras, each seeing a freely walking fly directly and through two prisms (six views) | multiview transformer, 38 keypoints, 512x512 window centered on the fly in each frame | DINOv3 ViT-B/16 | 16 recordings, 3,018 labeled items |
|
| 27 |
| `260920_fast_efficientvit_b1_352_hexaview_locator` | hexaview | full-frame locator that places the hexaview detector's windows | EfficientViT-B1 | 9 recordings, 824 labeled items |
|
| 28 |
|
|
|
|
| 36 |
uv run hf download tkclam/deeperfly-models --local-dir models
|
| 37 |
```
|
| 38 |
|
| 39 |
+
One model only, such as the current octacam detector:
|
| 40 |
|
| 41 |
```bash
|
| 42 |
uv run hf download tkclam/deeperfly-models --local-dir models \
|
| 43 |
+
--include "261008_mvt_dinov3b_384_octacam_fly38/*"
|
| 44 |
```
|
| 45 |
|
| 46 |
Check a download with `sha256sum -c SHA256SUMS` inside its folder.
|
|
|
|
| 51 |
|
| 52 |
```toml
|
| 53 |
[pose2d]
|
| 54 |
+
weights = ["models/261008_mvt_dinov3b_384_octacam_fly38/mvt_dinov3b_384_octacam_fly38.pth"]
|
| 55 |
```
|
| 56 |
|
| 57 |
Paths resolve against the config file. See the [deeperfly quickstart](https://nely-epfl.github.io/deeperfly/stable/quickstart/) for a full run on the bundled example clips.
|
| 58 |
|
| 59 |
## Scope and limitations
|
| 60 |
|
| 61 |
+
The detectors were trained on recordings from two rigs in one lab, with every label used for training (no held-out set). The current octacam detector's twin, trained with four recordings held out, scores 3.25 px median and 4.49 px mean on their 9,269 labeled cells (93.7% within 10 px). On the bundled example clips, the octacam detector reprojects at 1.46 px median after bundle adjustment and the hexaview detector at 3.15 px median through the recording's calibration; that is agreement across views, not error against labels. They are not expected to transfer to other rigs, magnifications, or lighting without retraining. `deeperfly train` fits a detector on your own labels, and these checkpoints can serve as a starting point.
|
| 62 |
|
| 63 |
## License
|
| 64 |
|
| 65 |
+
- The DINOv3-based detectors are derivatives of Meta's DINOv3 weights and are distributed under the [DINOv3 License](LICENSE.md), including its use restrictions.
|
| 66 |
- The EfficientViT-based locator is distributed under the [Apache License 2.0](LICENSE-apache-2.0), the license of its ImageNet-pretrained backbone.
|
| 67 |
|
| 68 |
## Citation
|