Sonata — transformers format

Unofficial transformers port of Sonata sonata.pth (facebook/sonata) — Base; Point Transformer V3 encoder, channels [48, 96, 192, 384, 512]. Loads with trust_remote_code; the sonata package is not needed. model.safetensors holds the checkpoint's state_dict under model.<original name>, converted from .pth (every tensor checked equal).

Not an official release. Sonata was created by Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang, Jakob Engel, Richard Newcombe, Hengshuang Zhao and Julian Straub (Meta) — https://github.com/facebookresearch/sonata, paper "Sonata: Self-Supervised Learning of Reliable Point Representations" (CVPR 2025).

What it is

SonataModel.model is the package's PointTransformerV3, built from the config stored in the checkpoint (config.json → ptv3_config) as sonata.load builds it; forward runs PointTransformerV3.forward and returns a SonataModelOutput (last_hidden_state: the output Point's features; point: the whole output Point). The code is the official inference code (facebookresearch/sonata @ 18c09ff8) vendored into flat files (sonata_model.py, sonata_transform.py, sonata_z_order.py, sonata_hilbert.py); the only edits are imports. SonataProcessor (processing_sonata.py, processor_config.json) wraps SonataFeatureExtractor, which runs sonata.transform.default() on a copy of the input arrays.

Verified against the official package (same process and seeds, V100, enable_flash=False): preprocessing, every tensor of the output Point and the features mapped back to every input point are bit-identical, in fp32 and under fp16 autocast, on an indoor and an outdoor demo scene.

Usage

import numpy as np, torch
from transformers import AutoModel, AutoProcessor

# enable_flash=False without FlashAttention (pre-Ampere GPUs), as the Sonata README suggests
model = AutoModel.from_pretrained("Sonata", trust_remote_code=True, enable_flash=False).cuda().eval()
processor = AutoProcessor.from_pretrained("Sonata", trust_remote_code=True)
point = processor({"coord": coord, "color": color, "normal": normal})  # (N, 3) numpy arrays
point = {k: v.cuda() if torch.is_tensor(v) else v for k, v in point.items()}
with torch.inference_mode():
    outputs = model(**point)  # outputs.last_hidden_state: features of the encoder's output points; outputs.point: the Point

• return_dict=False returns a tuple.
• Per-input-point features: walk outputs.point's pooling_parent / pooling_inverse back up and index with inverse, as in the Sonata README.
• The checkpoint config has shuffle_orders=True and GridSample keeps a random point per voxel: seed numpy and torch for reproducible outputs.
• Runtime packages beyond torch / transformers: spconv, torch_scatter, timm, addict, scipy (as the original); flash_attn when enable_flash is on.

ScanNet-20 linear probe

linear_probe_head_scannet20.safetensors / linear_probe_head_scannet20.json (from sonata_linear_prob_head_sc.pth) is the semantic segmentation head of the official demo (demo/2_sem_seg.py, SegHead): one nn.Linear applied to the features mapped back to the input points. Verified: its tensors equal the original, and its logits on the package's per-point features are bit-identical to the demo pipeline's.

import json, safetensors.torch, torch

cfg = json.load(open("linear_probe_head_scannet20.json"))  # backbone_out_channels, num_classes
head = torch.nn.Linear(cfg["backbone_out_channels"], cfg["num_classes"])
state = safetensors.torch.load_file("linear_probe_head_scannet20.safetensors")
head.load_state_dict({"weight": state["seg_head.weight"], "bias": state["seg_head.bias"]})
logits = head(per_point_features)  # (N, 20) ScanNet-20 classes

Licensing

• Weights: CC-BY-NC-4.0 (non-commercial) — LICENSE, as the official facebookresearch/sonata README releases them (restricted by the NC terms of datasets such as HM3D and ARKitScenes). facebook/sonata states no license and Pointcept/Sonata's model card says MIT; this package follows the official README.
• Code (vendored files and the port): Apache-2.0 — LICENSE.code.
• Attribution, sources and the changes made here: NOTICE.

Citation

If you use this model, please cite the original work, as its authors ask:

@inproceedings{wu2025sonata,
    title={Sonata: Self-Supervised Learning of Reliable Point Representations},
    author={Wu, Xiaoyang and DeTone, Daniel and Frost, Duncan and Shen, Tianwei and Xie, Chris and Yang, Nan and Engel, Jakob and Newcombe, Richard and Zhao, Hengshuang and Straub, Julian},
    booktitle={CVPR},
    year={2025}
}

@inproceedings{wu2024ptv3,
    title={Point Transformer V3: Simpler, Faster, Stronger},
    author={Wu, Xiaoyang and Jiang, Li and Wang, Peng-Shuai and Liu, Zhijian and Liu, Xihui and Qiao, Yu and Ouyang, Wanli and He, Tong and Zhao, Hengshuang},
    booktitle={CVPR},
    year={2024}
}

@inproceedings{wu2024ppt,
    title={Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training},
    author={Wu, Xiaoyang and Tian, Zhuotao and Wen, Xin and Peng, Bohao and Liu, Xihui and Yu, Kaicheng and Zhao, Hengshuang},
    booktitle={CVPR},
    year={2024}
}

@inproceedings{wu2023masked,
  title={Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning},
  author={Wu, Xiaoyang and Wen, Xin and Liu, Xihui and Zhao, Hengshuang},
  journal={CVPR},
  year={2023}
}

@inproceedings{wu2022ptv2,
    title={Point transformer V2: Grouped Vector Attention and Partition-based Pooling},
    author={Wu, Xiaoyang and Lao, Yixing and Jiang, Li and Liu, Xihui and Zhao, Hengshuang},
    booktitle={NeurIPS},
    year={2022}
}

@misc{pointcept2023,
    title={Pointcept: A Codebase for Point Cloud Perception Research},
    author={Pointcept Contributors},
    howpublished={\url{https://github.com/Pointcept/Pointcept}},
    year={2023}
}