| --- |
| license: cc-by-nc-4.0 |
| library_name: transformers |
| base_model: Pointcept/Concerto |
| tags: |
| - concerto |
| - point-cloud |
| - point-transformer-v3 |
| pipeline_tag: feature-extraction |
| --- |
| |
| # Concerto-Base-Origin — transformers format |
|
|
| Unofficial transformers port of **Concerto** `concerto_base_origin.pth` — Base, the original Concerto; Point Transformer V3 encoder, channels |
| [48, 96, 192, 384, 512]. Loads with `trust_remote_code`; the `concerto` package is not needed. `model.safetensors` holds the |
| checkpoint's `state_dict` under `model.<original name>`, converted from `.pth` (every tensor checked equal). |
|
|
| > **Not an official release.** Concerto was created by Yujia Zhang, Xiaoyang Wu, Yixing Lao, Chengyao Wang, |
| > Zhuotao Tian, Naiyan Wang and Hengshuang Zhao — https://github.com/Pointcept/Concerto, paper *"Concerto: Joint |
| > 2D-3D Self-Supervised Learning Emerges Spatial Representations"* (NeurIPS 2025, arXiv 2510.23607). |
|
|
| ## What it is |
|
|
| `ConcertoModel.model` is the package's `PointTransformerV3`, built from the config stored in the checkpoint |
| (`config.json` → `ptv3_config`) as `concerto.load` builds it; `forward` runs `PointTransformerV3.forward` and returns a |
| `ConcertoModelOutput` (`last_hidden_state`: the output Point's features; `point`: the whole output Point). The code is the official inference code (Pointcept/Concerto @ 5b411171) vendored into flat files |
| (`concerto_model.py`, `concerto_transform.py`, `concerto_z_order.py`, `concerto_hilbert.py`); the only edits are |
| imports. `ConcertoProcessor` (`processing_concerto.py`, `processor_config.json`) wraps `ConcertoFeatureExtractor`, |
| which runs `concerto.transform.default()` on a copy of the input arrays. |
|
|
| Verified against the official package (same process and seeds, V100, `enable_flash=False`): preprocessing, every |
| tensor of the output `Point` and the features mapped back to every input point are **bit-identical**, in fp32 and |
| under fp16 autocast, on an indoor and an outdoor demo scene. |
|
|
| ## Usage |
|
|
| ```python |
| import numpy as np, torch |
| from transformers import AutoModel, AutoProcessor |
| |
| # enable_flash=False without FlashAttention (pre-Ampere GPUs), as the Concerto README suggests |
| model = AutoModel.from_pretrained("Concerto-Base-Origin", trust_remote_code=True, enable_flash=False).cuda().eval() |
| processor = AutoProcessor.from_pretrained("Concerto-Base-Origin", trust_remote_code=True) |
| point = processor({"coord": coord, "color": color, "normal": normal}) # (N, 3) numpy arrays |
| point = {k: v.cuda() if torch.is_tensor(v) else v for k, v in point.items()} |
| with torch.inference_mode(): |
| outputs = model(**point) # outputs.last_hidden_state: features of the encoder's output points; outputs.point: the Point |
| ``` |
|
|
| - `return_dict=False` returns a tuple. |
| - Per-input-point features: walk `outputs.point`'s `pooling_parent` / `pooling_inverse` back up and index with `inverse`, as in the |
| Concerto README. |
| - The checkpoint config has `shuffle_orders=True` and `GridSample` keeps a random point per voxel: seed numpy and |
| torch for reproducible outputs. |
| - Runtime packages beyond torch / transformers: `spconv`, `torch_scatter`, `timm`, `addict`, `scipy` (as the original); |
| `flash_attn` when `enable_flash` is on. |
|
|
| ## Licensing |
|
|
| - Weights (`Pointcept/Concerto`): **CC-BY-NC-4.0** (non-commercial) — [`LICENSE`](LICENSE). |
| - Code (vendored files and the port): **Apache-2.0** — [`LICENSE.code`](LICENSE.code). |
| - Attribution, sources and the changes made here: [`NOTICE`](NOTICE). |
|
|
| ## Citation |
|
|
| If you use this model, please cite the original work, as its authors ask: |
|
|
| ```bibtex |
| @inproceedings{zhang2025concerto, |
| title={Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations}, |
| author={Zhang, Yujia and Wu, Xiaoyang and Lao, Yixing and Wang, Chengyao and Tian, Zhuotao and Wang, Naiyan and Zhao, Hengshuang}, |
| booktitle={NeurIPS}, |
| year={2025} |
| } |
| |
| @inproceedings{wu2025sonata, |
| title={Sonata: Self-Supervised Learning of Reliable Point Representations}, |
| author={Wu, Xiaoyang and DeTone, Daniel and Frost, Duncan and Shen, Tianwei and Xie, Chris and Yang, Nan and Engel, Jakob and Newcombe, Richard and Zhao, Hengshuang and Straub, Julian}, |
| booktitle={CVPR}, |
| year={2025} |
| } |
| |
| @inproceedings{wu2024ptv3, |
| title={Point Transformer V3: Simpler, Faster, Stronger}, |
| author={Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang}, |
| booktitle={CVPR}, |
| year={2024} |
| } |
| |
| @inproceedings{wu2024ppt, |
| title={Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training}, |
| author={Wu, Xiaoyang and Tian, Zhuotao and Wen, Xin and Peng, Bohao and Liu, Xihui and Yu, Kaicheng and Zhao, Hengshuang}, |
| booktitle={CVPR}, |
| year={2024} |
| } |
| |
| @inproceedings{wu2023masked, |
| title={Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning}, |
| author={Wu, Xiaoyang and Wen, Xin and Liu, Xihui and Zhao, Hengshuang}, |
| journal={CVPR}, |
| year={2023} |
| } |
| |
| @inproceedings{wu2022ptv2, |
| title={Point transformer V2: Grouped Vector Attention and Partition-based Pooling}, |
| author={Wu, Xiaoyang and Lao, Yixing and Jiang, Li and Liu, Xihui and Zhao, Hengshuang}, |
| booktitle={NeurIPS}, |
| year={2022} |
| } |
| |
| @misc{pointcept2023, |
| title={Pointcept: A Codebase for Point Cloud Perception Research}, |
| author={Pointcept Contributors}, |
| howpublished={\url{https://github.com/Pointcept/Pointcept}}, |
| year={2023} |
| } |
| ``` |
|
|
