Complete EgoRayFormer model card and include DINOv3 license

d6c8140 verified about 6 hours ago

9.19 kB

license: other
license_name: dinov3-license
license_link: https://huggingface.co/BodenAI/EgoRayFormer/blob/main/LICENSE_DINOv3.md
library_name: pytorch
pipeline_tag: keypoint-detection
base_model: timm/vit_small_plus_patch16_dinov3.lvd1689m
tags:
  - egocentric
  - 3d-human-pose-estimation
  - human-pose-estimation
  - stereo
  - fisheye
  - pytorch-lightning
metrics:
  - mpjpe
  - pa-mpjpe
model-index:
  - name: egorayformer_ue2.ckpt
    results:
      - task:
          type: keypoint-detection
          name: Egocentric 3D human pose estimation
        dataset:
          type: unrealego2
          name: UnrealEgo2
          split: test
        metrics:
          - type: mpjpe
            name: MPJPE (mm)
            value: 17.53
          - type: pa-mpjpe
            name: PA-MPJPE (mm)
            value: 16.8
  - name: egorayformer_rw.ckpt
    results:
      - task:
          type: keypoint-detection
          name: Egocentric 3D human pose estimation
        dataset:
          type: unrealego-rw
          name: UnrealEgo-RW
          split: test
        metrics:
          - type: mpjpe
            name: MPJPE (mm)
            value: 49.34
          - type: pa-mpjpe
            name: PA-MPJPE (mm)
            value: 40.59
  - name: egorayformer_ue2_to_rw.ckpt
    results:
      - task:
          type: keypoint-detection
          name: Egocentric 3D human pose estimation
        dataset:
          type: unrealego-rw
          name: UnrealEgo-RW
          split: test
        metrics:
          - type: mpjpe
            name: MPJPE (mm)
            value: 47.68
          - type: pa-mpjpe
            name: PA-MPJPE (mm)
            value: 37.73

EgoRayFormer

EgoRayFormer: Ray-Space Stereo Refinement for Egocentric 3D Human Pose Estimation

Paper · Code

Haozhe Wang, Songyuan Wu, Jihai Xu, Jie Zhao · Boden AI

Built with DINOv3

This repository holds the checkpoints of EgoRayFormer, a causal 3D human pose estimator for head-mounted stereo fisheye cameras. For each joint, the decoder projects shared 3D hypotheses along and around its viewing ray into both images. A geometric branch then corrects the projected joint locations and intersects the resulting rays, and a learned gate blends this geometric estimate with a learned pose update. Code, training and evaluation: github.com/BodenAI/EgoRayFormer.

Checkpoints

Three models of Table 1 of the paper, all with a DINOv3 ViT-S+/16 encoder at 384 × 384 (35.1 M parameters in total). Each file contains the EMA weights used for evaluation (approximately 134 MiB per checkpoint).

File	Training data	Test set	Config
`egorayformer_ue2.ckpt`	UnrealEgo2	UnrealEgo2	`configs/egorayformer_ue2.yaml`
`egorayformer_rw.ckpt`	UnrealEgo-RW	UnrealEgo-RW	`configs/egorayformer_rw.yaml`
`egorayformer_ue2_to_rw.ckpt`	UnrealEgo2, then UnrealEgo-RW	UnrealEgo-RW	`configs/egorayformer_ue2_to_rw.yaml`

SHA-256:

f4a0028421d3b9a1d8bb45fe3d336e072d8036a59afc4841fb03c5d7ce9c5100  egorayformer_ue2.ckpt
5b4612438b6204bab198f9e9d1cc7e696cd919052bc48c5cd084676e932f4c08  egorayformer_rw.ckpt
dfeb9c2867e14f4df97cda35f8fbffa37ba34f003e5a3b4133753a73ce12856e  egorayformer_ue2_to_rw.ckpt

Results

Device-relative protocol on the official test splits, mm. Every sequence is processed as one causal stream in FP32. The released files are the seed-42 models; the paper reports the mean ± s.d. over seeds 42, 1 and 2.

Model	Test set	MPJPE (this checkpoint)	PA-MPJPE (this checkpoint)	MPJPE (paper, 3 seeds)	PA-MPJPE (paper, 3 seeds)
`egorayformer_ue2`	UnrealEgo2	17.53	16.80	17.64 ± 0.12	16.86 ± 0.08
`egorayformer_rw`	UnrealEgo-RW	49.34	40.59	49.55 ± 0.40	40.59 ± 0.30
`egorayformer_ue2_to_rw`	UnrealEgo-RW	47.68	37.73	47.57 ± 0.34	37.21 ± 0.46

See the paper for the comparison with other methods.

Usage

The checkpoints are used with the code repository. Install it, prepare the UnrealEgo2 / UnrealEgo-RW records with the scripts of the repository (see its README), then download a checkpoint and evaluate it:

git clone https://github.com/BodenAI/EgoRayFormer.git && cd EgoRayFormer
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121
pip install -e .

mkdir -p checkpoints
for m in egorayformer_ue2 egorayformer_rw egorayformer_ue2_to_rw; do
  wget -P checkpoints https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/$m.ckpt
done

python run.py validate --config configs/egorayformer_rw.yaml --config configs/eval.yaml \
  --model.model.encoder.pretrained false --data.root /data/unrealego/processed \
  --ckpt_path checkpoints/egorayformer_rw.ckpt

Use egorayformer_ue2 or egorayformer_ue2_to_rw for the other two models. For the test split, run EGORAYFORMER_ALLOW_TEST=1 python run.py test with the same arguments. --model.model.encoder.pretrained false skips the download of the DINOv3 weights, which come from the checkpoint.

Input. Stereo fisheye video from a head-mounted rig: the left and right images of each frame (resized to 384 × 384) and the calibration of the rig (fisheye camera models and camera-from-device extrinsics). The model projects 3D hypotheses into both images, so the calibration is part of the input. It processes frames causally and keeps a streaming state (key–value caches, trajectory history, previous pose).

Output. The 3D positions of 16 joints (UnrealEgo skeleton: head, neck, upper arms, lower arms, hands, thighs, calves, feet, balls of the feet) relative to the device, for every frame.

Training

Each model is trained in two stages on one NVIDIA H200 GPU:

2D heatmap pre-training of the encoder at 256 × 256, starting from the DINOv3 ViT-S+/16 weights (timm/vit_small_plus_patch16_dinov3.lvd1689m);
end-to-end training on clips of 8 consecutive frames at 384 × 384, batch size 16, AdamW, bf16 mixed precision: 12 epochs on UnrealEgo-RW, 3 epochs on UnrealEgo2. The UnrealEgo2 → UnrealEgo-RW model starts from the weights of the UnrealEgo2 model and is trained for 12 epochs on UnrealEgo-RW.

The checkpoint with the lowest validation MPJPE is selected. The commands are in the README of the code repository.

Data. UnrealEgo2 and UnrealEgo-RW (Akada et al., CVPR 2024), official splits. UnrealEgo2 is rendered with a synthetic stereo fisheye rig; UnrealEgo-RW was captured with a physical head-mounted rig. For UnrealEgo-RW, the camera-from-device transform of every capture session is fitted to the training annotations only.

Limitations

The models were trained and evaluated on the two rigs of UnrealEgo2 and UnrealEgo-RW only. Other headsets need their calibration, and accuracy on other rigs has not been evaluated.
Errors increase with the distance of a joint from the cameras; the lower body has the largest errors.
One streaming step takes about 43 ms on an NVIDIA H200 (FP32, batch 1, model forward only), so the numbers above do not establish real-time operation on other hardware.
The models estimate the pose of the person wearing the headset; they are intended for research.

License

The checkpoints contain weights derived from DINOv3 and are distributed under the DINOv3 License. The code is released under the Apache License 2.0 in the code repository. UnrealEgo2 and UnrealEgo-RW are distributed by their authors under their own terms.

Citation

@article{wang2026egorayformer,
  title   = {{EgoRayFormer}: Ray-Space Stereo Refinement for Egocentric {3D} Human Pose Estimation},
  author  = {Wang, Haozhe and Wu, Songyuan and Xu, Jihai and Zhao, Jie},
  journal = {arXiv preprint arXiv:2610.26641},
  year    = {2026}
}

Contact Us

bodenai.com (中文) · boden.ai (English)

company@bodenai.com · EAI@bodenai.com