EgoRayFormer: Ray-Space Stereo Refinement for Egocentric 3D Human Pose Estimation
Haozhe Wang, Songyuan Wu, Jihai Xu, Jie Zhao · Boden AI
Built with DINOv3
This repository holds the checkpoints of [EgoRayFormer](https://arxiv.org/abs/2610.26641), a causal 3D human pose estimator for head-mounted stereo fisheye cameras. For each joint, the decoder projects shared 3D hypotheses along and around its viewing ray into both images. A geometric branch then corrects the projected joint locations and intersects the resulting rays, and a learned gate blends this geometric estimate with a learned pose update. Code, training and evaluation: [github.com/BodenAI/EgoRayFormer](https://github.com/BodenAI/EgoRayFormer).  ## Checkpoints Three models of Table 1 of the paper, all with a DINOv3 ViT-S+/16 encoder at 384 × 384 (35.1 M parameters in total). Each file contains the EMA weights used for evaluation (approximately 134 MiB per checkpoint). | File | Training data | Test set | Config | |---|---|---|---| | [`egorayformer_ue2.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_ue2.ckpt) | UnrealEgo2 | UnrealEgo2 | [`configs/egorayformer_ue2.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_ue2.yaml) | | [`egorayformer_rw.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_rw.ckpt) | UnrealEgo-RW | UnrealEgo-RW | [`configs/egorayformer_rw.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_rw.yaml) | | [`egorayformer_ue2_to_rw.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_ue2_to_rw.ckpt) | UnrealEgo2, then UnrealEgo-RW | UnrealEgo-RW | [`configs/egorayformer_ue2_to_rw.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_ue2_to_rw.yaml) | SHA-256: ``` f4a0028421d3b9a1d8bb45fe3d336e072d8036a59afc4841fb03c5d7ce9c5100 egorayformer_ue2.ckpt 5b4612438b6204bab198f9e9d1cc7e696cd919052bc48c5cd084676e932f4c08 egorayformer_rw.ckpt dfeb9c2867e14f4df97cda35f8fbffa37ba34f003e5a3b4133753a73ce12856e egorayformer_ue2_to_rw.ckpt ``` ## Results Device-relative protocol on the official test splits, mm. Every sequence is processed as one causal stream in FP32. The released files are the seed-42 models; the paper reports the mean ± s.d. over seeds 42, 1 and 2. | Model | Test set | MPJPE (this checkpoint) | PA-MPJPE (this checkpoint) | MPJPE (paper, 3 seeds) | PA-MPJPE (paper, 3 seeds) | |---|---|---|---|---|---| | `egorayformer_ue2` | UnrealEgo2 | 17.53 | 16.80 | 17.64 ± 0.12 | 16.86 ± 0.08 | | `egorayformer_rw` | UnrealEgo-RW | 49.34 | 40.59 | 49.55 ± 0.40 | 40.59 ± 0.30 | | `egorayformer_ue2_to_rw` | UnrealEgo-RW | 47.68 | 37.73 | 47.57 ± 0.34 | 37.21 ± 0.46 | See the paper for the comparison with other methods. ## Usage The checkpoints are used with the code repository. Install it, prepare the UnrealEgo2 / UnrealEgo-RW records with the scripts of the repository (see its README), then download a checkpoint and evaluate it: ```bash git clone https://github.com/BodenAI/EgoRayFormer.git && cd EgoRayFormer pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121 pip install -e . mkdir -p checkpoints for m in egorayformer_ue2 egorayformer_rw egorayformer_ue2_to_rw; do wget -P checkpoints https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/$m.ckpt done python run.py validate --config configs/egorayformer_rw.yaml --config configs/eval.yaml \ --model.model.encoder.pretrained false --data.root /data/unrealego/processed \ --ckpt_path checkpoints/egorayformer_rw.ckpt ``` Use `egorayformer_ue2` or `egorayformer_ue2_to_rw` for the other two models. For the test split, run `EGORAYFORMER_ALLOW_TEST=1 python run.py test` with the same arguments. `--model.model.encoder.pretrained false` skips the download of the DINOv3 weights, which come from the checkpoint. **Input.** Stereo fisheye video from a head-mounted rig: the left and right images of each frame (resized to 384 × 384) and the calibration of the rig (fisheye camera models and camera-from-device extrinsics). The model projects 3D hypotheses into both images, so the calibration is part of the input. It processes frames causally and keeps a streaming state (key–value caches, trajectory history, previous pose). **Output.** The 3D positions of 16 joints (UnrealEgo skeleton: head, neck, upper arms, lower arms, hands, thighs, calves, feet, balls of the feet) relative to the device, for every frame. ## Training Each model is trained in two stages on one NVIDIA H200 GPU: 1. 2D heatmap pre-training of the encoder at 256 × 256, starting from the DINOv3 ViT-S+/16 weights (`timm/vit_small_plus_patch16_dinov3.lvd1689m`); 2. end-to-end training on clips of 8 consecutive frames at 384 × 384, batch size 16, AdamW, bf16 mixed precision: 12 epochs on UnrealEgo-RW, 3 epochs on UnrealEgo2. The UnrealEgo2 → UnrealEgo-RW model starts from the weights of the UnrealEgo2 model and is trained for 12 epochs on UnrealEgo-RW. The checkpoint with the lowest validation MPJPE is selected. The commands are in the README of the code repository. **Data.** [UnrealEgo2 and UnrealEgo-RW](https://unrealego.mpi-inf.mpg.de/) (Akada et al., CVPR 2024), official splits. UnrealEgo2 is rendered with a synthetic stereo fisheye rig; UnrealEgo-RW was captured with a physical head-mounted rig. For UnrealEgo-RW, the camera-from-device transform of every capture session is fitted to the training annotations only. ## Limitations - The models were trained and evaluated on the two rigs of UnrealEgo2 and UnrealEgo-RW only. Other headsets need their calibration, and accuracy on other rigs has not been evaluated. - Errors increase with the distance of a joint from the cameras; the lower body has the largest errors. - One streaming step takes about 43 ms on an NVIDIA H200 (FP32, batch 1, model forward only), so the numbers above do not establish real-time operation on other hardware. - The models estimate the pose of the person wearing the headset; they are intended for research. ## License The checkpoints contain weights derived from DINOv3 and are distributed under the [DINOv3 License](LICENSE_DINOv3.md). The code is released under the Apache License 2.0 in the [code repository](https://github.com/BodenAI/EgoRayFormer). UnrealEgo2 and UnrealEgo-RW are distributed by their authors under their own terms. ## Citation ```bibtex @article{wang2026egorayformer, title = {{EgoRayFormer}: Ray-Space Stereo Refinement for Egocentric {3D} Human Pose Estimation}, author = {Wang, Haozhe and Wu, Songyuan and Xu, Jihai and Zhao, Jie}, journal = {arXiv preprint arXiv:2610.26641}, year = {2026} } ``` ## Contact Us
bodenai.com (中文) · boden.ai (English)