| --- |
| license: other |
| license_name: dinov3-license |
| license_link: https://huggingface.co/BodenAI/EgoRayFormer/blob/main/LICENSE_DINOv3.md |
| library_name: pytorch |
| pipeline_tag: keypoint-detection |
| base_model: timm/vit_small_plus_patch16_dinov3.lvd1689m |
| tags: |
| - egocentric |
| - 3d-human-pose-estimation |
| - human-pose-estimation |
| - stereo |
| - fisheye |
| - pytorch-lightning |
| metrics: |
| - mpjpe |
| - pa-mpjpe |
| model-index: |
| - name: egorayformer_ue2.ckpt |
| results: |
| - task: |
| type: keypoint-detection |
| name: Egocentric 3D human pose estimation |
| dataset: |
| type: unrealego2 |
| name: UnrealEgo2 |
| split: test |
| metrics: |
| - type: mpjpe |
| name: MPJPE (mm) |
| value: 17.53 |
| - type: pa-mpjpe |
| name: PA-MPJPE (mm) |
| value: 16.8 |
| - name: egorayformer_rw.ckpt |
| results: |
| - task: |
| type: keypoint-detection |
| name: Egocentric 3D human pose estimation |
| dataset: |
| type: unrealego-rw |
| name: UnrealEgo-RW |
| split: test |
| metrics: |
| - type: mpjpe |
| name: MPJPE (mm) |
| value: 49.34 |
| - type: pa-mpjpe |
| name: PA-MPJPE (mm) |
| value: 40.59 |
| - name: egorayformer_ue2_to_rw.ckpt |
| results: |
| - task: |
| type: keypoint-detection |
| name: Egocentric 3D human pose estimation |
| dataset: |
| type: unrealego-rw |
| name: UnrealEgo-RW |
| split: test |
| metrics: |
| - type: mpjpe |
| name: MPJPE (mm) |
| value: 47.68 |
| - type: pa-mpjpe |
| name: PA-MPJPE (mm) |
| value: 37.73 |
| --- |
| |
| <p align="center"> |
| <img src="https://raw.githubusercontent.com/BodenAI/EgoRayFormer/main/docs/logo/en_logo.png" alt="BODENAI" width="220"> |
| </p> |
|
|
| <h1 align="center">EgoRayFormer</h1> |
|
|
| <p align="center"><b>EgoRayFormer: Ray-Space Stereo Refinement for Egocentric 3D Human Pose Estimation</b></p> |
|
|
| <p align="center"> |
| <a href="https://arxiv.org/abs/2610.26641">Paper</a> · |
| <a href="https://github.com/BodenAI/EgoRayFormer">Code</a> |
| </p> |
|
|
| <p align="center">Haozhe Wang, Songyuan Wu, Jihai Xu, Jie Zhao · Boden AI</p> |
|
|
| <p align="center"><b>Built with DINOv3</b></p> |
|
|
| This repository holds the checkpoints of [EgoRayFormer](https://arxiv.org/abs/2610.26641), a causal 3D human pose |
| estimator for head-mounted stereo fisheye cameras. For each joint, the decoder projects shared 3D hypotheses along and |
| around its viewing ray into both images. A geometric branch then corrects the projected joint locations and intersects |
| the resulting rays, and a learned gate blends this geometric estimate with a learned pose update. Code, training and |
| evaluation: [github.com/BodenAI/EgoRayFormer](https://github.com/BodenAI/EgoRayFormer). |
|
|
|  |
|
|
| ## Checkpoints |
|
|
| Three models of Table 1 of the paper, all with a DINOv3 ViT-S+/16 encoder at 384 × 384 (35.1 M parameters in total). Each file |
| contains the EMA weights used for evaluation (approximately 134 MiB per checkpoint). |
|
|
| | File | Training data | Test set | Config | |
| |---|---|---|---| |
| | [`egorayformer_ue2.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_ue2.ckpt) | UnrealEgo2 | UnrealEgo2 | [`configs/egorayformer_ue2.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_ue2.yaml) | |
| | [`egorayformer_rw.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_rw.ckpt) | UnrealEgo-RW | UnrealEgo-RW | [`configs/egorayformer_rw.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_rw.yaml) | |
| | [`egorayformer_ue2_to_rw.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_ue2_to_rw.ckpt) | UnrealEgo2, then UnrealEgo-RW | UnrealEgo-RW | [`configs/egorayformer_ue2_to_rw.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_ue2_to_rw.yaml) | |
|
|
| SHA-256: |
|
|
| ``` |
| f4a0028421d3b9a1d8bb45fe3d336e072d8036a59afc4841fb03c5d7ce9c5100 egorayformer_ue2.ckpt |
| 5b4612438b6204bab198f9e9d1cc7e696cd919052bc48c5cd084676e932f4c08 egorayformer_rw.ckpt |
| dfeb9c2867e14f4df97cda35f8fbffa37ba34f003e5a3b4133753a73ce12856e egorayformer_ue2_to_rw.ckpt |
| ``` |
|
|
| ## Results |
|
|
| Device-relative protocol on the official test splits, mm. Every sequence is processed as one causal stream in FP32. |
| The released files are the seed-42 models; the paper reports the mean ± s.d. over seeds 42, 1 and 2. |
|
|
| | Model | Test set | MPJPE (this checkpoint) | PA-MPJPE (this checkpoint) | MPJPE (paper, 3 seeds) | PA-MPJPE (paper, 3 seeds) | |
| |---|---|---|---|---|---| |
| | `egorayformer_ue2` | UnrealEgo2 | 17.53 | 16.80 | 17.64 ± 0.12 | 16.86 ± 0.08 | |
| | `egorayformer_rw` | UnrealEgo-RW | 49.34 | 40.59 | 49.55 ± 0.40 | 40.59 ± 0.30 | |
| | `egorayformer_ue2_to_rw` | UnrealEgo-RW | 47.68 | 37.73 | 47.57 ± 0.34 | 37.21 ± 0.46 | |
|
|
| See the paper for the comparison with other methods. |
|
|
| ## Usage |
|
|
| The checkpoints are used with the code repository. Install it, prepare the UnrealEgo2 / UnrealEgo-RW records with the |
| scripts of the repository (see its README), then download a checkpoint and evaluate it: |
|
|
| ```bash |
| git clone https://github.com/BodenAI/EgoRayFormer.git && cd EgoRayFormer |
| pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121 |
| pip install -e . |
| |
| mkdir -p checkpoints |
| for m in egorayformer_ue2 egorayformer_rw egorayformer_ue2_to_rw; do |
| wget -P checkpoints https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/$m.ckpt |
| done |
| |
| python run.py validate --config configs/egorayformer_rw.yaml --config configs/eval.yaml \ |
| --model.model.encoder.pretrained false --data.root /data/unrealego/processed \ |
| --ckpt_path checkpoints/egorayformer_rw.ckpt |
| ``` |
|
|
| Use `egorayformer_ue2` or `egorayformer_ue2_to_rw` for the other two models. For the test split, run |
| `EGORAYFORMER_ALLOW_TEST=1 python run.py test` with the same arguments. `--model.model.encoder.pretrained false` skips |
| the download of the DINOv3 weights, which come from the checkpoint. |
|
|
| **Input.** Stereo fisheye video from a head-mounted rig: the left and right images of each frame (resized to |
| 384 × 384) and the calibration of the rig (fisheye camera models and camera-from-device extrinsics). The model projects |
| 3D hypotheses into both images, so the calibration is part of the input. It processes frames causally and keeps a |
| streaming state (key–value caches, trajectory history, previous pose). |
|
|
| **Output.** The 3D positions of 16 joints (UnrealEgo skeleton: head, neck, upper arms, lower arms, hands, thighs, |
| calves, feet, balls of the feet) relative to the device, for every frame. |
|
|
| ## Training |
|
|
| Each model is trained in two stages on one NVIDIA H200 GPU: |
|
|
| 1. 2D heatmap pre-training of the encoder at 256 × 256, starting from the DINOv3 ViT-S+/16 weights |
| (`timm/vit_small_plus_patch16_dinov3.lvd1689m`); |
| 2. end-to-end training on clips of 8 consecutive frames at 384 × 384, batch size 16, AdamW, bf16 mixed precision: |
| 12 epochs on UnrealEgo-RW, 3 epochs on UnrealEgo2. The UnrealEgo2 → UnrealEgo-RW model starts from the weights of the |
| UnrealEgo2 model and is trained for 12 epochs on UnrealEgo-RW. |
|
|
| The checkpoint with the lowest validation MPJPE is selected. The commands are in the README of the code repository. |
|
|
| **Data.** [UnrealEgo2 and UnrealEgo-RW](https://unrealego.mpi-inf.mpg.de/) (Akada et al., CVPR 2024), official splits. |
| UnrealEgo2 is rendered with a synthetic stereo fisheye rig; UnrealEgo-RW was captured with a physical head-mounted rig. |
| For UnrealEgo-RW, the camera-from-device transform of every capture session is fitted to the training annotations only. |
|
|
| ## Limitations |
|
|
| - The models were trained and evaluated on the two rigs of UnrealEgo2 and UnrealEgo-RW only. Other headsets need their |
| calibration, and accuracy on other rigs has not been evaluated. |
| - Errors increase with the distance of a joint from the cameras; the lower body has the largest errors. |
| - One streaming step takes about 43 ms on an NVIDIA H200 (FP32, batch 1, model forward only), so the numbers above do |
| not establish real-time operation on other hardware. |
| - The models estimate the pose of the person wearing the headset; they are intended for research. |
|
|
| ## License |
|
|
| The checkpoints contain weights derived from DINOv3 and are distributed under the |
| [DINOv3 License](LICENSE_DINOv3.md). The code is released under the Apache License 2.0 in the |
| [code repository](https://github.com/BodenAI/EgoRayFormer). UnrealEgo2 and UnrealEgo-RW are distributed by their authors |
| under their own terms. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{wang2026egorayformer, |
| title = {{EgoRayFormer}: Ray-Space Stereo Refinement for Egocentric {3D} Human Pose Estimation}, |
| author = {Wang, Haozhe and Wu, Songyuan and Xu, Jihai and Zhao, Jie}, |
| journal = {arXiv preprint arXiv:2610.26641}, |
| year = {2026} |
| } |
| ``` |
|
|
| ## Contact Us |
|
|
| <p align="center"> |
| <img src="https://raw.githubusercontent.com/BodenAI/EgoRayFormer/main/docs/logo/code.jpg" alt="WeChat QR code" width="110"> |
| </p> |
|
|
| <p align="center"> |
| <a href="https://bodenai.com">bodenai.com</a> (中文) · |
| <a href="https://www.boden.ai">boden.ai</a> (English) |
| </p> |
|
|
| <p align="center"> |
| <a href="mailto:company@bodenai.com">company@bodenai.com</a> · |
| <a href="mailto:EAI@bodenai.com">EAI@bodenai.com</a> |
| </p> |
|
|
