Blame for README.md

---
license: other
license_name: dinov3-license
license_link: https://huggingface.co/BodenAI/EgoRayFormer/blob/main/LICENSE_DINOv3.md
library_name: pytorch
pipeline_tag: keypoint-detection
base_model: timm/vit_small_plus_patch16_dinov3.lvd1689m
tags:
- egocentric
- 3d-human-pose-estimation
- human-pose-estimation
- stereo
- fisheye
- pytorch-lightning
metrics:
- mpjpe
- pa-mpjpe
model-index:
- name: egorayformer_ue2.ckpt
  results:
  - task:
      type: keypoint-detection
      name: Egocentric 3D human pose estimation
    dataset:
      type: unrealego2
      name: UnrealEgo2
      split: test
    metrics:
    - type: mpjpe
      name: MPJPE (mm)
      value: 17.53
    - type: pa-mpjpe
      name: PA-MPJPE (mm)
      value: 16.8
- name: egorayformer_rw.ckpt
  results:
  - task:
      type: keypoint-detection
      name: Egocentric 3D human pose estimation
    dataset:
      type: unrealego-rw
      name: UnrealEgo-RW
      split: test
    metrics:
    - type: mpjpe
      name: MPJPE (mm)
      value: 49.34
    - type: pa-mpjpe
      name: PA-MPJPE (mm)
      value: 40.59
- name: egorayformer_ue2_to_rw.ckpt
  results:
  - task:
      type: keypoint-detection
      name: Egocentric 3D human pose estimation
    dataset:
      type: unrealego-rw
      name: UnrealEgo-RW
      split: test
    metrics:
    - type: mpjpe
      name: MPJPE (mm)
      value: 47.68
    - type: pa-mpjpe
      name: PA-MPJPE (mm)
      value: 37.73
---

<p align="center">
  <img src="https://raw.githubusercontent.com/BodenAI/EgoRayFormer/main/docs/logo/en_logo.png" alt="BODENAI" width="220">
</p>

<h1 align="center">EgoRayFormer</h1>

<p align="center"><b>EgoRayFormer: Ray-Space Stereo Refinement for Egocentric 3D Human Pose Estimation</b></p>

<p align="center">
  <a href="https://arxiv.org/abs/2610.26641">Paper</a> ·
  <a href="https://github.com/BodenAI/EgoRayFormer">Code</a>
</p>

<p align="center">Haozhe Wang, Songyuan Wu, Jihai Xu, Jie Zhao · Boden AI</p>

<p align="center"><b>Built with DINOv3</b></p>

This repository holds the checkpoints of [EgoRayFormer](https://arxiv.org/abs/2610.26641), a causal 3D human pose
estimator for head-mounted stereo fisheye cameras. For each joint, the decoder projects shared 3D hypotheses along and
around its viewing ray into both images. A geometric branch then corrects the projected joint locations and intersects
the resulting rays, and a learned gate blends this geometric estimate with a learned pose update. Code, training and
evaluation: [github.com/BodenAI/EgoRayFormer](https://github.com/BodenAI/EgoRayFormer).

![EgoRayFormer architecture](https://raw.githubusercontent.com/BodenAI/EgoRayFormer/main/assets/fig_architecture.png)

## Checkpoints

Three models of Table 1 of the paper, all with a DINOv3 ViT-S+/16 encoder at 384 × 384 (35.1 M parameters in total). Each file
contains the EMA weights used for evaluation (approximately 134 MiB per checkpoint).

| File | Training data | Test set | Config |
|---|---|---|---|
| [`egorayformer_ue2.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_ue2.ckpt) | UnrealEgo2 | UnrealEgo2 | [`configs/egorayformer_ue2.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_ue2.yaml) |
| [`egorayformer_rw.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_rw.ckpt) | UnrealEgo-RW | UnrealEgo-RW | [`configs/egorayformer_rw.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_rw.yaml) |
| [`egorayformer_ue2_to_rw.ckpt`](https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/egorayformer_ue2_to_rw.ckpt) | UnrealEgo2, then UnrealEgo-RW | UnrealEgo-RW | [`configs/egorayformer_ue2_to_rw.yaml`](https://github.com/BodenAI/EgoRayFormer/blob/main/configs/egorayformer_ue2_to_rw.yaml) |

SHA-256:

```
f4a0028421d3b9a1d8bb45fe3d336e072d8036a59afc4841fb03c5d7ce9c5100  egorayformer_ue2.ckpt
5b4612438b6204bab198f9e9d1cc7e696cd919052bc48c5cd084676e932f4c08  egorayformer_rw.ckpt
dfeb9c2867e14f4df97cda35f8fbffa37ba34f003e5a3b4133753a73ce12856e  egorayformer_ue2_to_rw.ckpt
```

## Results

Device-relative protocol on the official test splits, mm. Every sequence is processed as one causal stream in FP32.
The released files are the seed-42 models; the paper reports the mean ± s.d. over seeds 42, 1 and 2.

| Model | Test set | MPJPE (this checkpoint) | PA-MPJPE (this checkpoint) | MPJPE (paper, 3 seeds) | PA-MPJPE (paper, 3 seeds) |
|---|---|---|---|---|---|
| `egorayformer_ue2` | UnrealEgo2 | 17.53 | 16.80 | 17.64 ± 0.12 | 16.86 ± 0.08 |
| `egorayformer_rw` | UnrealEgo-RW | 49.34 | 40.59 | 49.55 ± 0.40 | 40.59 ± 0.30 |
| `egorayformer_ue2_to_rw` | UnrealEgo-RW | 47.68 | 37.73 | 47.57 ± 0.34 | 37.21 ± 0.46 |

See the paper for the comparison with other methods.

## Usage

The checkpoints are used with the code repository. Install it, prepare the UnrealEgo2 / UnrealEgo-RW records with the
scripts of the repository (see its README), then download a checkpoint and evaluate it:

```bash
git clone https://github.com/BodenAI/EgoRayFormer.git && cd EgoRayFormer
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121
pip install -e .

mkdir -p checkpoints
for m in egorayformer_ue2 egorayformer_rw egorayformer_ue2_to_rw; do
  wget -P checkpoints https://huggingface.co/BodenAI/EgoRayFormer/resolve/main/$m.ckpt
done

python run.py validate --config configs/egorayformer_rw.yaml --config configs/eval.yaml \
  --model.model.encoder.pretrained false --data.root /data/unrealego/processed \
  --ckpt_path checkpoints/egorayformer_rw.ckpt
```

Use `egorayformer_ue2` or `egorayformer_ue2_to_rw` for the other two models. For the test split, run
`EGORAYFORMER_ALLOW_TEST=1 python run.py test` with the same arguments. `--model.model.encoder.pretrained false` skips
the download of the DINOv3 weights, which come from the checkpoint.

**Input.** Stereo fisheye video from a head-mounted rig: the left and right images of each frame (resized to
384 × 384) and the calibration of the rig (fisheye camera models and camera-from-device extrinsics). The model projects
3D hypotheses into both images, so the calibration is part of the input. It processes frames causally and keeps a
streaming state (key–value caches, trajectory history, previous pose).

**Output.** The 3D positions of 16 joints (UnrealEgo skeleton: head, neck, upper arms, lower arms, hands, thighs,
calves, feet, balls of the feet) relative to the device, for every frame.

## Training

Each model is trained in two stages on one NVIDIA H200 GPU:

1. 2D heatmap pre-training of the encoder at 256 × 256, starting from the DINOv3 ViT-S+/16 weights
   (`timm/vit_small_plus_patch16_dinov3.lvd1689m`);
2. end-to-end training on clips of 8 consecutive frames at 384 × 384, batch size 16, AdamW, bf16 mixed precision:
   12 epochs on UnrealEgo-RW, 3 epochs on UnrealEgo2. The UnrealEgo2 → UnrealEgo-RW model starts from the weights of the
   UnrealEgo2 model and is trained for 12 epochs on UnrealEgo-RW.

The checkpoint with the lowest validation MPJPE is selected. The commands are in the README of the code repository.

**Data.** [UnrealEgo2 and UnrealEgo-RW](https://unrealego.mpi-inf.mpg.de/) (Akada et al., CVPR 2024), official splits.
UnrealEgo2 is rendered with a synthetic stereo fisheye rig; UnrealEgo-RW was captured with a physical head-mounted rig.
For UnrealEgo-RW, the camera-from-device transform of every capture session is fitted to the training annotations only.

## Limitations

- The models were trained and evaluated on the two rigs of UnrealEgo2 and UnrealEgo-RW only. Other headsets need their
  calibration, and accuracy on other rigs has not been evaluated.
- Errors increase with the distance of a joint from the cameras; the lower body has the largest errors.
- One streaming step takes about 43 ms on an NVIDIA H200 (FP32, batch 1, model forward only), so the numbers above do
  not establish real-time operation on other hardware.
- The models estimate the pose of the person wearing the headset; they are intended for research.

## License

The checkpoints contain weights derived from DINOv3 and are distributed under the
[DINOv3 License](LICENSE_DINOv3.md). The code is released under the Apache License 2.0 in the
[code repository](https://github.com/BodenAI/EgoRayFormer). UnrealEgo2 and UnrealEgo-RW are distributed by their authors
under their own terms.

## Citation

```bibtex
@article{wang2026egorayformer,
  title   = {{EgoRayFormer}: Ray-Space Stereo Refinement for Egocentric {3D} Human Pose Estimation},
  author  = {Wang, Haozhe and Wu, Songyuan and Xu, Jihai and Zhao, Jie},
  journal = {arXiv preprint arXiv:2610.26641},
  year    = {2026}
}
```

## Contact Us

<p align="center">
  <img src="https://raw.githubusercontent.com/BodenAI/EgoRayFormer/main/docs/logo/code.jpg" alt="WeChat QR code" width="110">
</p>

<p align="center">
  <a href="https://bodenai.com">bodenai.com</a> (中文)&nbsp;·&nbsp;
  <a href="https://www.boden.ai">boden.ai</a> (English)
</p>

<p align="center">
  <a href="mailto:company@bodenai.com">company@bodenai.com</a>&nbsp;·&nbsp;
  <a href="mailto:EAI@bodenai.com">EAI@bodenai.com</a>
</p>