OctLLM / README.md

Update README.md

f8b4027 verified 20 days ago

2.91 kB

---
tags:
- 3d
- multimodal
- text-to-3d
- image-to-3d
- 3d-understanding
- octree
license: mit
base_model:
- Qwen/Qwen2.5-VL-7B-Instruct
library_name: transformers
---
<h1 align="center">OctLLM</h1>
<h3 align="center">Octrees as an Explicit 3D Language</h3>

<p align="center">
  Ran&nbsp;Dan<sup>1</sup>,
  <a href="https://wst2001.github.io/">Si-Tong&nbsp;Wei</a><sup>1</sup>,
  Pengfei&nbsp;Xiong<sup>2</sup>,
  <a href="https://sites.google.com/site/zhangweinus/?pli=1&amp;authuser=0">Wei&nbsp;Zhang</a><sup>2</sup>,
  <a href="http://www.muyadong.com/">Yadong&nbsp;Mu</a><sup>1</sup>,
  <a href="https://wang-ps.github.io/">Peng-Shuai&nbsp;Wang</a><sup>1,†</sup>
</p>
<p align="center">
  <sup>1</sup>Peking University &nbsp; <sup>2</sup>Independent Researcher<br>
  <sup>†</sup>Corresponding author
</p>
<p align="center">
  <a href="https://arxiv.org/abs/2610.02388"><img src="https://img.shields.io/badge/arXiv-2610.02388-b31b1b?logo=arxiv&amp;logoColor=white" alt="arXiv: 2610.02388"></a>
  <a href="https://github.com/octree-nn/octllm.git"><img src="https://img.shields.io/badge/GitHub-Code-181717?logo=github&amp;logoColor=white" alt="Code"></a>
  <a href="https://plurato.github.io/OctLLM-page/"><img src="https://img.shields.io/badge/Project%20Page-Website-2e7d32?logo=googlechrome&amp;logoColor=white" alt="Project Page"></a>
</p>

Official model weights for **Octrees as an Explicit 3D Language**.

<p align="center">
  <img src="https://huggingface.co/Plurato123/OctLLM/resolve/main/assets/teaser.gif" width="100%" alt="OctLLM generated meshes, 3D descriptions, and sparse octrees">
</p>

OctLLM unifies **text-to-3D, image-to-3D, and 3D understanding** through an explicit 3D language. It represents shapes as **Sparse Octree (S-Octree)** sequences: occupancy tokens anchored to 3D coordinates and octree depth. This structured representation connects geometry generation and understanding within a single model.

To learn the 3D modality, OctLLM adds trainable branches alongside a frozen Qwen2.5-VL backbone. Mesh tokens and text/image tokens follow separate processing pathways and interact through shared self-attention. Generated octrees are completed by a 3D U-Net and decoded into textured meshes with pretrained TRELLIS decoders. The video below demonstrates generation, 3D understanding, and language interaction.

<video controls muted loop playsinline preload="metadata" poster="https://plurato.github.io/OctLLM-page/assets/images/interaction-poster.webp" src="https://plurato.github.io/OctLLM-page/assets/videos/interaction.mp4"></video>

This repository contains the OctLLM checkpoint and occupancy completion weights. Use the [official implementation](https://github.com/octree-nn/octllm.git) for installation and inference; its loading code is required for the custom 3D components. More results are available on the [project page](https://plurato.github.io/OctLLM-page/).

	---
	tags:
	- 3d
	- multimodal
	- text-to-3d
	- image-to-3d
	- 3d-understanding
	- octree
	license: mit
	base_model:
	- Qwen/Qwen2.5-VL-7B-Instruct
	library_name: transformers
	---
	<h1 align="center">OctLLM</h1>
	<h3 align="center">Octrees as an Explicit 3D Language</h3>

	<p align="center">
	Ran Dan<sup>1</sup>,
	<a href="https://wst2001.github.io/">Si-Tong Wei</a><sup>1</sup>,
	Pengfei Xiong<sup>2</sup>,
	<a href="https://sites.google.com/site/zhangweinus/?pli=1&authuser=0">Wei Zhang</a><sup>2</sup>,
	<a href="http://www.muyadong.com/">Yadong Mu</a><sup>1</sup>,
	<a href="https://wang-ps.github.io/">Peng-Shuai Wang</a><sup>1,†</sup>
	</p>
	<p align="center">
	<sup>1</sup>Peking University   <sup>2</sup>Independent Researcher<br>
	<sup>†</sup>Corresponding author
	</p>
	<p align="center">
	<a href="https://arxiv.org/abs/2610.02388"><img src="https://img.shields.io/badge/arXiv-2610.02388-b31b1b?logo=arxiv&logoColor=white" alt="arXiv: 2610.02388"></a>
	<a href="https://github.com/octree-nn/octllm.git"><img src="https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white" alt="Code"></a>
	<a href="https://plurato.github.io/OctLLM-page/"><img src="https://img.shields.io/badge/Project%20Page-Website-2e7d32?logo=googlechrome&logoColor=white" alt="Project Page"></a>
	</p>

	Official model weights for Octrees as an Explicit 3D Language.

	<p align="center">
	<img src="https://huggingface.co/Plurato123/OctLLM/resolve/main/assets/teaser.gif" width="100%" alt="OctLLM generated meshes, 3D descriptions, and sparse octrees">
	</p>

	OctLLM unifies text-to-3D, image-to-3D, and 3D understanding through an explicit 3D language. It represents shapes as Sparse Octree (S-Octree) sequences: occupancy tokens anchored to 3D coordinates and octree depth. This structured representation connects geometry generation and understanding within a single model.

	To learn the 3D modality, OctLLM adds trainable branches alongside a frozen Qwen2.5-VL backbone. Mesh tokens and text/image tokens follow separate processing pathways and interact through shared self-attention. Generated octrees are completed by a 3D U-Net and decoded into textured meshes with pretrained TRELLIS decoders. The video below demonstrates generation, 3D understanding, and language interaction.

	<video controls muted loop playsinline preload="metadata" poster="https://plurato.github.io/OctLLM-page/assets/images/interaction-poster.webp" src="https://plurato.github.io/OctLLM-page/assets/videos/interaction.mp4"></video>

	This repository contains the OctLLM checkpoint and occupancy completion weights. Use the [official implementation](https://github.com/octree-nn/octllm.git) for installation and inference; its loading code is required for the custom 3D components. More results are available on the [project page](https://plurato.github.io/OctLLM-page/).