---
license: cc-by-nc-4.0
library_name: transformers
base_model: Pointcept/Concerto
tags:
- concerto
- point-cloud
- point-transformer-v3
pipeline_tag: feature-extraction
---

# Concerto-Base — transformers format

Unofficial transformers port of **Concerto** `concerto_base.pth` — Base (pre-release of Utonia; inputs without color / normal allowed as zeros); Point Transformer V3 encoder, channels
[48, 96, 192, 384, 512]. Loads with `trust_remote_code`; the `concerto` package is not needed. `model.safetensors` holds the
checkpoint's `state_dict` under `model.<original name>`, converted from `.pth` (every tensor checked equal).

> **Not an official release.** Concerto was created by Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang,
> Zhuotao Tian, Naiyan Wang and Hengshuang Zhao — https://github.com/Pointcept/Concerto, paper *"Concerto: Joint
> 2D-3D Self-Supervised Learning Emerges Spatial Representations"* (NeurIPS 2025, arXiv 2510.23607).

## What it is

`ConcertoModel.model` is the package's `PointTransformerV3`, built from the config stored in the checkpoint
(`config.json` → `ptv3_config`) as `concerto.load` builds it; `forward` runs `PointTransformerV3.forward` and returns a
`ConcertoModelOutput` (`last_hidden_state`: the output Point's features; `point`: the whole output Point). The code is the official inference code (Pointcept/Concerto @ 5b411171) vendored into flat files
(`concerto_model.py`, `concerto_transform.py`, `concerto_z_order.py`, `concerto_hilbert.py`); the only edits are
imports. `ConcertoProcessor` (`processing_concerto.py`, `processor_config.json`) wraps `ConcertoFeatureExtractor`,
which runs `concerto.transform.default()` on a copy of the input arrays.

Verified against the official package (same process and seeds, V100, `enable_flash=False`): preprocessing, every
tensor of the output `Point` and the features mapped back to every input point are **bit-identical**, in fp32 and
under fp16 autocast, on an indoor and an outdoor demo scene.

## Usage

```python
import numpy as np, torch
from transformers import AutoModel, AutoProcessor

# enable_flash=False without FlashAttention (pre-Ampere GPUs), as the Concerto README suggests
model = AutoModel.from_pretrained("Concerto-Base", trust_remote_code=True, enable_flash=False).cuda().eval()
processor = AutoProcessor.from_pretrained("Concerto-Base", trust_remote_code=True)
point = processor({"coord": coord, "color": color, "normal": normal})  # (N, 3) numpy arrays
point = {k: v.cuda() if torch.is_tensor(v) else v for k, v in point.items()}
with torch.inference_mode():
    outputs = model(**point)  # outputs.last_hidden_state: features of the encoder's output points; outputs.point: the Point
```

- `return_dict=False` returns a tuple.
- Per-input-point features: walk `outputs.point`'s `pooling_parent` / `pooling_inverse` back up and index with `inverse`, as in the
  Concerto README.
- The checkpoint config has `shuffle_orders=True` and `GridSample` keeps a random point per voxel: seed numpy and
  torch for reproducible outputs.
- Runtime packages beyond torch / transformers: `spconv`, `torch_scatter`, `timm`, `addict`, `scipy` (as the original);
  `flash_attn` when `enable_flash` is on.

## Licensing

- Weights (`Pointcept/Concerto`): **CC-BY-NC-4.0** (non-commercial) — [`LICENSE`](LICENSE).
- Code (vendored files and the port): **Apache-2.0** — [`LICENSE.code`](LICENSE.code).
- Attribution, sources and the changes made here: [`NOTICE`](NOTICE).

## Citation

If you use this model, please cite the original work, as its authors ask:

```bibtex
@inproceedings{zhang2025concerto,
  title={Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations},
  author={Zhang, Yujia and Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Tian, Zhuotao and Wang, Naiyan and Zhao, Hengshuang},
  booktitle={NeurIPS},
  year={2025}
}

@inproceedings{wu2025sonata,
    title={Sonata: Self-Supervised Learning of Reliable Point Representations},
    author={Wu, Xiaoyang and DeTone, Daniel and Frost, Duncan and Shen, Tianwei and Xie, Chris and Yang, Nan and Engel, Jakob and Newcombe, Richard and Zhao, Hengshuang and Straub, Julian},
    booktitle={CVPR},
    year={2025}
}

@inproceedings{wu2024ptv3,
    title={Point Transformer V3: Simpler, Faster, Stronger},
    author={Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang},
    booktitle={CVPR},
    year={2024}
}

@inproceedings{wu2024ppt,
    title={Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training},
    author={Wu, Xiaoyang and Tian, Zhuotao and Wen, Xin and Peng, Bohao and Liu, Xihui and Yu, Kaicheng and Zhao, Hengshuang},
    booktitle={CVPR},
    year={2024}
}

@inproceedings{wu2023masked,
  title={Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning},
  author={Wu, Xiaoyang and Wen, Xin and Liu, Xihui and Zhao, Hengshuang},
  journal={CVPR},
  year={2023}
}

@inproceedings{wu2022ptv2,
    title={Point transformer V2: Grouped Vector Attention and Partition-based Pooling},
    author={Wu, Xiaoyang and Lao, Yixing and Jiang, Li and Liu, Xihui and Zhao, Hengshuang},
    booktitle={NeurIPS},
    year={2022}
}

@misc{pointcept2023,
    title={Pointcept: A Codebase for Point Cloud Perception Research},
    author={Pointcept Contributors},
    howpublished={\url{https://github.com/Pointcept/Pointcept}},
    year={2023}
}
```