PQR3D / README.md

Document checkpoint download and nuScenes evaluation

460ab97 verified 21 days ago

9.82 kB

---
license: mit
library_name: pytorch
tags:
  - 3d-object-detection
  - autonomous-driving
  - multi-view
  - pytorch
  - arxiv:2609.32163
---
# PQR3D

Code and five released checkpoints for **PQR3D: Progressive Query Refinement over Reference-Conditioned Temporal Windows for Multi-View 3D Object Detection**. This Hub repository mirrors the [original PQR3D project](https://github.com/huiyegit/PQR3D) at commit [`5b105ba`](https://github.com/huiyegit/PQR3D/commit/5b105ba4a697b1f4979913ba4ebe91345952269c). The paper, implementation, and checkpoints are credited to the original authors.

- Paper: [arXiv:2609.32163](https://arxiv.org/abs/2609.32163)
- Code: [GitHub](https://github.com/huiyegit/PQR3D)
- Original checkpoint links and reported results: [Model Zoo](#model-zoo)

## Checkpoint contents

The five `.pth` files in [`checkpoints/`](checkpoints/) are trained PQR3D model checkpoints. Download the checkpoint matching the config you will use:

| Configuration | Checkpoint |
| --- | --- |
| [`r50_nuimg_704x256`](configs/r50_nuimg_704x256.py) | [`r50_nuimg_704x256.pth`](checkpoints/r50_nuimg_704x256.pth) |
| [`r50_nuimg_704x256_625q`](configs/r50_nuimg_704x256_625q_48ep.py) | [`r50_nuimg_704x256_625q.pth`](checkpoints/r50_nuimg_704x256_625q.pth) |
| [`r101_nuimg_1408x512`](configs/r101_nuimg_1408x512.py) | [`r101_nuimg_1408x512.pth`](checkpoints/r101_nuimg_1408x512.pth) |
| [`vov99_dd3d_1600x640`](configs/vov99_dd3d_1600x640_trainval_future.py) | [`vov99_dd3d_1600x640.pth`](checkpoints/vov99_dd3d_1600x640.pth) |
| [`vit_eva02_1600x640`](configs/vit_eva02_1600x640_trainval_future.py) | [`vit_eva02_1600x640.pth`](checkpoints/vit_eva02_1600x640.pth) |

## Download and evaluate

Evaluation uses the project's `val.py` loader. It requires a CUDA GPU, the project dependencies, and the nuScenes dataset with generated sweep info files. Follow [Environment](#environment) and [Prepare Dataset](#prepare-dataset) below before running the example.

Get the matching source version and install the Hub download helper:

```bash
git clone https://github.com/huiyegit/PQR3D.git
cd PQR3D
git checkout 5b105ba4a697b1f4979913ba4ebe91345952269c
pip install huggingface_hub
```

After preparing `data/nuscenes/`, download the first checkpoint into `checkpoints/`:

```python
from huggingface_hub import hf_hub_download
checkpoint = hf_hub_download(
    repo_id="yhhugging/PQR3D",
    filename="checkpoints/r50_nuimg_704x256.pth",
    local_dir=".",
)
print(checkpoint)
```

Run the Python snippet above from the `PQR3D` directory, then evaluate on the nuScenes validation split:

```bash
python val.py \
    --config configs/r50_nuimg_704x256.py \
    --weights checkpoints/r50_nuimg_704x256.pth
```

For another model, select the matching config and checkpoint from the table above. The validation script builds the model from the MMCV config and loads the checkpoint with `mmcv.runner.load_checkpoint`.

## Intended use and limitations

PQR3D is a research model for camera-based multi-view 3D object detection on nuScenes. The provided evaluation path processes the dataset's camera frames and temporal sweeps; it requires nuScenes files and a CUDA GPU. The repository does not provide a standalone single-image prediction interface or a `from_pretrained` loader. Results in the Model Zoo below are reported by the original project and have not been rerun for this Hub mirror.

## Original project instructions

The following setup, dataset, training, and evaluation instructions come from the original project README.

## News
* 2026-09-29: We release the arXiv version [arXiv:2609.32163](https://arxiv.org/abs/2609.32163).
* 2026-09-25: We release the source code and pretrained weights.


## Model Zoo

| Setting  | Pretrain |  epochs| Training Cost | NDS<sub>val</sub> | NDS<sub>test</sub>  | Weights | Log
|----------|:--------:|:-------------:|:-------------------:|:------------------: |:-------:|:-------:|:-------:|
| [r50_nuimg_704x256](configs/r50_nuimg_704x256.py) | [nuImg](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth) | 24| 13h (8x4090) | 58.2 | -  | [Hub checkpoint](checkpoints/r50_nuimg_704x256.pth) ([original](https://drive.google.com/file/d/1MkX2vfkRp8pD7RjJ1X9gDqLAVsoYPnMf/view?usp=share_link)) |[log](https://drive.google.com/file/d/1RhHzy9a3WT4UmpMdUT-d0Sby3A95MsfO/view?usp=share_link) |
| [r50_nuimg_704x256_625q](configs/r50_nuimg_704x256_625q_48ep.py) | [nuImg](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth) | 48| 22h (8x4090) | 58.5 | -  | [Hub checkpoint](checkpoints/r50_nuimg_704x256_625q.pth) ([original](https://drive.google.com/file/d/1DzBCKxZm3BqvM8F6UbvSBTkDhn0-QOS_/view?usp=share_link)) |[log](https://drive.google.com/file/d/1K5MFWONSF5gcl0dyopYGwG66FWqgFZuU/view?usp=share_link) |
| [r101_nuimg_1408x512](configs/r101_nuimg_1408x512.py) | [nuImg](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r101_fpn_1x_nuim/cascade_mask_rcnn_r101_fpn_1x_nuim_20201024_134804-45215b1e.pth) | 24  | 1d16h (8x4090) | 61.2 | - |  [Hub checkpoint](checkpoints/r101_nuimg_1408x512.pth) ([original](https://drive.google.com/file/d/1BlB0Qz7iTdPoI1EUga6UdqBjCQkDeiTQ/view?usp=share_link)) |[log](https://drive.google.com/file/d/1pLEmfZEy_a_oIp0VO0cwqWN0JUNuDOkP/view?usp=share_link) |
| [vov99_dd3d_1600x640](configs/vov99_dd3d_1600x640_trainval_future.py) | [DD3D](https://drive.google.com/file/d/1gQkhWERCzAosBwG5bh2BKkt1k0TJZt-A/view) | 24| 11d7h (8xA100) | - | 68.0  | [Hub checkpoint](checkpoints/vov99_dd3d_1600x640.pth) ([original](https://drive.google.com/file/d/13CWGsPbohS2JFrE43XFZoiZoeUKCZoBE/view?usp=share_link)) |[log](https://drive.google.com/file/d/1uOOjl_m-SgG0EyJBButrzD_9hVB0X_Eb/view?usp=share_link) |
|[vit_eva02_1600x640](configs/vit_eva02_1600x640_trainval_future.py) | [EVA02](https://huggingface.co/Yuxin-CV/EVA-02/blob/main/eva02/det/eva02_L_coco_seg_sys_o365.pth) | 24 | 14d16h (8xA100) | - | 71.6  | [Hub checkpoint](checkpoints/vit_eva02_1600x640.pth) ([original](https://drive.google.com/file/d/1U1fFmqRLkda5HNOFQjprCNzJm8ITvAMI/view?usp=sharing)) |[log](https://drive.google.com/file/d/1RwM9w_7-p1vEDHYCMPv04VBIX6_jQkpb/view?usp=share_link) |




## Environment

Install PyTorch 2.0 + CUDA 11.8:

```
conda create -n pqr3d python=3.8
conda activate pqr3d
conda install pytorch==2.0.0 torchvision==0.15.0 pytorch-cuda=11.8 -c pytorch -c nvidia
```


Install other dependencies:

```
pip install openmim
mim install mmcv-full==1.6.0
mim install mmdet==2.28.2
mim install mmsegmentation==0.30.0
mim install mmdet3d==1.0.0rc6
pip install setuptools==59.5.0
pip install numpy==1.23.5
```

Install turbojpeg and pillow-simd to speed up data loading (optional but important):

```
sudo apt-get update
sudo apt-get install -y libturbojpeg
pip install pyturbojpeg
pip uninstall pillow
pip install pillow-simd==9.0.0.post1
```

Compile CUDA extensions:

```
cd models/csrc
python setup.py build_ext --inplace
```

## Prepare Dataset

1. Download nuScenes from [https://www.nuscenes.org/nuscenes](https://www.nuscenes.org/nuscenes) and put it in `data/nuscenes`.
2. Download the generated info file from [Google Drive](https://drive.google.com/drive/folders/1E3OSuQiTjkVQ8b3iuG1ZyzmNHw-j2QAC?usp=share_link).
3. Folder structure:

```
data/nuscenes
├── maps
├── nuscenes_infos_test_sweep.pkl
├── nuscenes_infos_train_sweep.pkl
├── nuscenes_infos_train_mini_sweep.pkl
├── nuscenes_infos_val_sweep.pkl
├── nuscenes_infos_val_mini_sweep.pkl
├── samples
├── sweeps
├── v1.0-test
└── v1.0-trainval
```


## Training

Download pretrained weights and put it in directory `pretrain/`:

```
pretrain
├── cascade_mask_rcnn_r101_fpn_1x_nuim_20201024_134804-45215b1e.pth
├── cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth
```

Train PQR3D with 8 GPUs:

```
torchrun --nproc_per_node 8 train.py --config configs/r50_nuimg_704x256.py
```

Train PQR3D with 4 GPUs (i.e the last four GPUs):

```
export CUDA_VISIBLE_DEVICES=4,5,6,7
torchrun --nproc_per_node 4 train.py --config configs/r50_nuimg_704x256.py
```

The batch size for each GPU will be scaled automatically. So there is no need to modify the `batch_size` in config files.

## Evaluation

Single-GPU evaluation:

```
export CUDA_VISIBLE_DEVICES=0
python val.py --config configs/r50_nuimg_704x256.py --weights checkpoints/r50_nuimg_704x256.pth
```

Multi-GPU evaluation:

```
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
torchrun --nproc_per_node 8 val.py --config configs/r50_nuimg_704x256.py --weights checkpoints/r50_nuimg_704x256.pth
```

## Timing

FPS is measured with a single GPU:

```
export CUDA_VISIBLE_DEVICES=0
python timing.py --config configs/r50_nuimg_704x256.py --weights checkpoints/r50_nuimg_704x256.pth
```

## Visualization

Visualize the predicted bbox:

```
python viz_bbox_predictions.py --config configs/r50_nuimg_704x256.py --weights checkpoints/r50_nuimg_704x256.pth
```


## Acknowledgements

Our implementation is based on these excellent open-source projects: [SparseBEV](https://github.com/MCG-NJU/SparseBEV), [Sparse4D](https://github.com/HorizonRobotics/Sparse4D),  [StreamPETR](https://github.com/exiawsh/StreamPETR), [HoP](https://github.com/Sense-X/HoP), [BEVFormer](https://github.com/fundamentalvision/BEVFormer), [BEVDet](https://github.com/HuangJunJie2017/BEVDet)


## License

The source repository includes an [MIT license](LICENSE). nuScenes data and external pretrained backbones are distributed separately under their respective terms.

	---
	license: mit
	library_name: pytorch
	tags:
	- 3d-object-detection
	- autonomous-driving
	- multi-view
	- pytorch
	- arxiv:2609.32163
	---

	# PQR3D

	Code and five released checkpoints for PQR3D: Progressive Query Refinement over Reference-Conditioned Temporal Windows for Multi-View 3D Object Detection. This Hub repository mirrors the [original PQR3D project](https://github.com/huiyegit/PQR3D) at commit [`5b105ba`](https://github.com/huiyegit/PQR3D/commit/5b105ba4a697b1f4979913ba4ebe91345952269c). The paper, implementation, and checkpoints are credited to the original authors.

	- Paper: [arXiv:2609.32163](https://arxiv.org/abs/2609.32163)
	- Code: [GitHub](https://github.com/huiyegit/PQR3D)
	- Original checkpoint links and reported results: [Model Zoo](#model-zoo)

	## Checkpoint contents

	The five `.pth` files in [`checkpoints/`](checkpoints/) are trained PQR3D model checkpoints. Download the checkpoint matching the config you will use:

	\| Configuration \| Checkpoint \|
	\| --- \| --- \|
	\| [`r50_nuimg_704x256`](configs/r50_nuimg_704x256.py) \| [`r50_nuimg_704x256.pth`](checkpoints/r50_nuimg_704x256.pth) \|
	\| [`r50_nuimg_704x256_625q`](configs/r50_nuimg_704x256_625q_48ep.py) \| [`r50_nuimg_704x256_625q.pth`](checkpoints/r50_nuimg_704x256_625q.pth) \|
	\| [`r101_nuimg_1408x512`](configs/r101_nuimg_1408x512.py) \| [`r101_nuimg_1408x512.pth`](checkpoints/r101_nuimg_1408x512.pth) \|
	\| [`vov99_dd3d_1600x640`](configs/vov99_dd3d_1600x640_trainval_future.py) \| [`vov99_dd3d_1600x640.pth`](checkpoints/vov99_dd3d_1600x640.pth) \|
	\| [`vit_eva02_1600x640`](configs/vit_eva02_1600x640_trainval_future.py) \| [`vit_eva02_1600x640.pth`](checkpoints/vit_eva02_1600x640.pth) \|

	## Download and evaluate

	Evaluation uses the project's `val.py` loader. It requires a CUDA GPU, the project dependencies, and the nuScenes dataset with generated sweep info files. Follow [Environment](#environment) and [Prepare Dataset](#prepare-dataset) below before running the example.

	Get the matching source version and install the Hub download helper:

	```bash
	git clone https://github.com/huiyegit/PQR3D.git
	cd PQR3D
	git checkout 5b105ba4a697b1f4979913ba4ebe91345952269c
	pip install huggingface_hub
	```

	After preparing `data/nuscenes/`, download the first checkpoint into `checkpoints/`:

	```python
	from huggingface_hub import hf_hub_download

	checkpoint = hf_hub_download(
	repo_id="yhhugging/PQR3D",
	filename="checkpoints/r50_nuimg_704x256.pth",
	local_dir=".",
	)
	print(checkpoint)
	```

	Run the Python snippet above from the `PQR3D` directory, then evaluate on the nuScenes validation split:

	```bash
	python val.py \
	--config configs/r50_nuimg_704x256.py \
	--weights checkpoints/r50_nuimg_704x256.pth
	```

	For another model, select the matching config and checkpoint from the table above. The validation script builds the model from the MMCV config and loads the checkpoint with `mmcv.runner.load_checkpoint`.

	## Intended use and limitations

	PQR3D is a research model for camera-based multi-view 3D object detection on nuScenes. The provided evaluation path processes the dataset's camera frames and temporal sweeps; it requires nuScenes files and a CUDA GPU. The repository does not provide a standalone single-image prediction interface or a `from_pretrained` loader. Results in the Model Zoo below are reported by the original project and have not been rerun for this Hub mirror.

	## Original project instructions

	The following setup, dataset, training, and evaluation instructions come from the original project README.

	## News
	* 2026-09-29: We release the arXiv version [arXiv:2609.32163](https://arxiv.org/abs/2609.32163).
	* 2026-09-25: We release the source code and pretrained weights.


	## Model Zoo

	\| Setting \| Pretrain \| epochs\| Training Cost \| NDS<sub>val</sub> \| NDS<sub>test</sub> \| Weights \| Log
	\|----------\|:--------:\|:-------------:\|:-------------------:\|:------------------: \|:-------:\|:-------:\|:-------:\|
	\| [r50_nuimg_704x256](configs/r50_nuimg_704x256.py) \| [nuImg](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth) \| 24\| 13h (8x4090) \| 58.2 \| - \| [Hub checkpoint](checkpoints/r50_nuimg_704x256.pth) ([original](https://drive.google.com/file/d/1MkX2vfkRp8pD7RjJ1X9gDqLAVsoYPnMf/view?usp=share_link)) \|[log](https://drive.google.com/file/d/1RhHzy9a3WT4UmpMdUT-d0Sby3A95MsfO/view?usp=share_link) \|
	\| [r50_nuimg_704x256_625q](configs/r50_nuimg_704x256_625q_48ep.py) \| [nuImg](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim/cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth) \| 48\| 22h (8x4090) \| 58.5 \| - \| [Hub checkpoint](checkpoints/r50_nuimg_704x256_625q.pth) ([original](https://drive.google.com/file/d/1DzBCKxZm3BqvM8F6UbvSBTkDhn0-QOS_/view?usp=share_link)) \|[log](https://drive.google.com/file/d/1K5MFWONSF5gcl0dyopYGwG66FWqgFZuU/view?usp=share_link) \|
	\| [r101_nuimg_1408x512](configs/r101_nuimg_1408x512.py) \| [nuImg](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/nuimages_semseg/cascade_mask_rcnn_r101_fpn_1x_nuim/cascade_mask_rcnn_r101_fpn_1x_nuim_20201024_134804-45215b1e.pth) \| 24 \| 1d16h (8x4090) \| 61.2 \| - \| [Hub checkpoint](checkpoints/r101_nuimg_1408x512.pth) ([original](https://drive.google.com/file/d/1BlB0Qz7iTdPoI1EUga6UdqBjCQkDeiTQ/view?usp=share_link)) \|[log](https://drive.google.com/file/d/1pLEmfZEy_a_oIp0VO0cwqWN0JUNuDOkP/view?usp=share_link) \|
	\| [vov99_dd3d_1600x640](configs/vov99_dd3d_1600x640_trainval_future.py) \| [DD3D](https://drive.google.com/file/d/1gQkhWERCzAosBwG5bh2BKkt1k0TJZt-A/view) \| 24\| 11d7h (8xA100) \| - \| 68.0 \| [Hub checkpoint](checkpoints/vov99_dd3d_1600x640.pth) ([original](https://drive.google.com/file/d/13CWGsPbohS2JFrE43XFZoiZoeUKCZoBE/view?usp=share_link)) \|[log](https://drive.google.com/file/d/1uOOjl_m-SgG0EyJBButrzD_9hVB0X_Eb/view?usp=share_link) \|
	\|[vit_eva02_1600x640](configs/vit_eva02_1600x640_trainval_future.py) \| [EVA02](https://huggingface.co/Yuxin-CV/EVA-02/blob/main/eva02/det/eva02_L_coco_seg_sys_o365.pth) \| 24 \| 14d16h (8xA100) \| - \| 71.6 \| [Hub checkpoint](checkpoints/vit_eva02_1600x640.pth) ([original](https://drive.google.com/file/d/1U1fFmqRLkda5HNOFQjprCNzJm8ITvAMI/view?usp=sharing)) \|[log](https://drive.google.com/file/d/1RwM9w_7-p1vEDHYCMPv04VBIX6_jQkpb/view?usp=share_link) \|




	## Environment

	Install PyTorch 2.0 + CUDA 11.8:

	```
	conda create -n pqr3d python=3.8
	conda activate pqr3d
	conda install pytorch==2.0.0 torchvision==0.15.0 pytorch-cuda=11.8 -c pytorch -c nvidia
	```


	Install other dependencies:

	```
	pip install openmim
	mim install mmcv-full==1.6.0
	mim install mmdet==2.28.2
	mim install mmsegmentation==0.30.0
	mim install mmdet3d==1.0.0rc6
	pip install setuptools==59.5.0
	pip install numpy==1.23.5
	```

	Install turbojpeg and pillow-simd to speed up data loading (optional but important):

	```
	sudo apt-get update
	sudo apt-get install -y libturbojpeg
	pip install pyturbojpeg
	pip uninstall pillow
	pip install pillow-simd==9.0.0.post1
	```

	Compile CUDA extensions:

	```
	cd models/csrc
	python setup.py build_ext --inplace
	```

	## Prepare Dataset

	1. Download nuScenes from [https://www.nuscenes.org/nuscenes](https://www.nuscenes.org/nuscenes) and put it in `data/nuscenes`.
	2. Download the generated info file from [Google Drive](https://drive.google.com/drive/folders/1E3OSuQiTjkVQ8b3iuG1ZyzmNHw-j2QAC?usp=share_link).
	3. Folder structure:

	```
	data/nuscenes
	├── maps
	├── nuscenes_infos_test_sweep.pkl
	├── nuscenes_infos_train_sweep.pkl
	├── nuscenes_infos_train_mini_sweep.pkl
	├── nuscenes_infos_val_sweep.pkl
	├── nuscenes_infos_val_mini_sweep.pkl
	├── samples
	├── sweeps
	├── v1.0-test
	└── v1.0-trainval
	```


	## Training

	Download pretrained weights and put it in directory `pretrain/`:

	```
	pretrain
	├── cascade_mask_rcnn_r101_fpn_1x_nuim_20201024_134804-45215b1e.pth
	├── cascade_mask_rcnn_r50_fpn_coco-20e_20e_nuim_20201009_124951-40963960.pth
	```

	Train PQR3D with 8 GPUs:

	```
	torchrun --nproc_per_node 8 train.py --config configs/r50_nuimg_704x256.py
	```

	Train PQR3D with 4 GPUs (i.e the last four GPUs):

	```
	export CUDA_VISIBLE_DEVICES=4,5,6,7
	torchrun --nproc_per_node 4 train.py --config configs/r50_nuimg_704x256.py
	```

	The batch size for each GPU will be scaled automatically. So there is no need to modify the `batch_size` in config files.

	## Evaluation

	Single-GPU evaluation:

	```
	export CUDA_VISIBLE_DEVICES=0
	python val.py --config configs/r50_nuimg_704x256.py --weights checkpoints/r50_nuimg_704x256.pth
	```

	Multi-GPU evaluation:

	```
	export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
	torchrun --nproc_per_node 8 val.py --config configs/r50_nuimg_704x256.py --weights checkpoints/r50_nuimg_704x256.pth
	```

	## Timing

	FPS is measured with a single GPU:

	```
	export CUDA_VISIBLE_DEVICES=0
	python timing.py --config configs/r50_nuimg_704x256.py --weights checkpoints/r50_nuimg_704x256.pth
	```

	## Visualization

	Visualize the predicted bbox:

	```
	python viz_bbox_predictions.py --config configs/r50_nuimg_704x256.py --weights checkpoints/r50_nuimg_704x256.pth
	```


	## Acknowledgements

	Our implementation is based on these excellent open-source projects: [SparseBEV](https://github.com/MCG-NJU/SparseBEV), [Sparse4D](https://github.com/HorizonRobotics/Sparse4D), [StreamPETR](https://github.com/exiawsh/StreamPETR), [HoP](https://github.com/Sense-X/HoP), [BEVFormer](https://github.com/fundamentalvision/BEVFormer), [BEVDet](https://github.com/HuangJunJie2017/BEVDet)


	## License

	The source repository includes an [MIT license](LICENSE). nuScenes data and external pretrained backbones are distributed separately under their respective terms.