Expand model card with authors, project links, demos, and usage

.gitattributes

@@ -1,2 +1,3 @@
 *.safetensors filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/teaser.gif filter=lfs diff=lfs merge=lfs -text

README.md

@@ -2,23 +2,110 @@
 tags:
 - octllm
 - 3d
+- multimodal
+- text-to-3d
+- image-to-3d
+- 3d-understanding
+- sparse-octree
 ---
-# OctLLM
+<h1 align="center">OctLLM</h1>
+<h3 align="center">Octrees as an Explicit 3D Language</h3>
 
-Official model weights for **Octrees as an Explicit 3D Language**.
+<p align="center">
+  Ran&nbsp;Dan<sup>1</sup>,
+  <a href="https://wst2001.github.io/">Si-Tong&nbsp;Wei</a><sup>1</sup>,
+  Pengfei&nbsp;Xiong<sup>2</sup>,
+  <a href="https://sites.google.com/site/zhangweinus/?pli=1&amp;authuser=0">Wei&nbsp;Zhang</a><sup>2</sup>,
+  <a href="http://www.muyadong.com/">Yadong&nbsp;Mu</a><sup>1</sup>,
+  <a href="https://wang-ps.github.io/">Peng-Shuai&nbsp;Wang</a><sup>1,†</sup>
+</p>
+<p align="center">
+  <sup>1</sup>Peking University &nbsp; <sup>2</sup>Independent Researcher<br>
+  <sup>†</sup>Corresponding author
+</p>
+<p align="center">
+  <img src="https://img.shields.io/badge/arXiv-Coming%20soon-b31b1b?logo=arxiv&amp;logoColor=white" alt="arXiv: Coming soon">
+  <a href="https://github.com/octree-nn/octllm.git"><img src="https://img.shields.io/badge/GitHub-Code-181717?logo=github&amp;logoColor=white" alt="Code"></a>
+  <a href="https://plurato.github.io/OctLLM-page/"><img src="https://img.shields.io/badge/Project%20Page-Website-2e7d32?logo=googlechrome&amp;logoColor=white" alt="Project Page"></a>
+</p>
+
+Official model weights for **Octrees as an Explicit 3D Language**. OctLLM represents 3D shapes as Sparse Octree (S-Octree) token sequences, supporting **text-to-3D, image-to-3D, and 3D understanding** in a single model.
+
+<p align="center">
+  <img src="https://huggingface.co/Plurato123/OctLLM/resolve/main/assets/teaser.gif" width="100%" alt="OctLLM generated meshes, 3D descriptions, and sparse octrees">
+</p>
+
+## Overview
+
+OctLLM gives a vision-language model an explicit, structured 3D representation. S-Octrees encode geometry as occupancy tokens anchored to 3D coordinates and octree depth. Trainable 3D branches process mesh tokens alongside a frozen Qwen2.5-VL backbone, with the two streams interacting through shared self-attention. For generation, occupancy completion and pretrained TRELLIS decoders turn the predicted octrees into textured meshes.
+
+- **Text-to-3D:** generate a textured 3D asset from a text description.
+- **Image-to-3D:** generate a textured 3D asset from an input image.
+- **3D understanding:** describe a 3D shape or answer questions about its geometry.
+
+<p align="center">
+  <img src="https://plurato.github.io/OctLLM-page/assets/images/method-overview.webp" width="100%" alt="OctLLM pipeline: text or image to S-Octree, occupancy completion, and mesh decoding; S-Octree to a natural-language description">
+</p>
+
+## Demo
+
+The video below demonstrates image-to-3D, text-to-3D, 3D understanding, and language interaction.
+
+<video controls muted loop playsinline preload="metadata" width="100%" poster="https://plurato.github.io/OctLLM-page/assets/images/interaction-poster.webp" src="https://plurato.github.io/OctLLM-page/assets/videos/interaction.mp4"></video>
+
+Explore more results in the [interactive 3D gallery](https://plurato.github.io/OctLLM-page/#generation), or watch [how OctLLM builds an octree and reconstructs a mesh](https://plurato.github.io/OctLLM-page/#process).
+
+## Model files
 
 The root contains the OctLLM model, tokenizer, processor, and all five model
 shards, including the 3D router, position embeddings, and mesh token weights.
 `completion/model.safetensors` contains the occupancy completion model used
-by the paper's `legacy` completion mode. TRELLIS and CLIP weights are obtained separately from
-their upstream repositories.
+by the paper's `legacy` completion mode.
+
+Keep the complete checkpoint, including all shards, `config.json`, tokenizer files, and processor files. Text-to-3D additionally uses [TRELLIS-text](https://huggingface.co/microsoft/TRELLIS-text-xlarge) and [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14); image-to-3D uses [TRELLIS-image](https://huggingface.co/microsoft/TRELLIS-image-large). These weights are obtained separately from their upstream repositories. 3D understanding and text/image chat only require the OctLLM checkpoint.
+
+## Quick start
 
-Use the accompanying OctLLM implementation; these custom 3D components
-require its loading code. Download this repository into `checkpoints/OctLLM`
-in that implementation, or set `OCTLLM_WEIGHTS_DIR` to the absolute path of
-the downloaded directory. Both OctLLM and completion then use that directory.
+Use the [official OctLLM implementation](https://github.com/octree-nn/octllm.git): the custom 3D components require its loading code. Installation requires **Linux x86_64**, **conda**, and an **NVIDIA GPU**. See the [installation guide](https://github.com/octree-nn/octllm#installation) for the full environment setup.
 
 ```bash
+git clone https://github.com/octree-nn/octllm.git
+cd octllm
+bash setup.sh
+conda activate OctLLM
+
 hf download Plurato123/OctLLM --local-dir checkpoints/OctLLM
+export OCTLLM_WEIGHTS_DIR="$(pwd)/checkpoints/OctLLM"
+
 python inference.py --prompt "Explain what an octree is." --no-mesh
-```
+```
+
+`OCTLLM_WEIGHTS_DIR` points both OctLLM and occupancy completion to the downloaded directory. To try image-to-3D with a bundled example, run:
+
+```bash
+python inference.py \
+  --prompt "Generate a 3D mesh based on this image:" \
+  --image ./assets/examples/deer.png \
+  --cuda-device 0 \
+  --output-dir outputs/demo --output-name image_asset
+```
+
+For 3D understanding:
+
+```bash
+python inference.py \
+  --mesh ./assets/examples/understanding/airplane.glb \
+  --prompt "Describe this 3D asset in detail:" \
+  --no-mesh --cuda-device 0
+```
+
+Generation saves a textured `.glb`, an S-Octree `.tokens.txt` sequence, and response metadata in `.json`. See the [inference guide](https://github.com/octree-nn/octllm#inference) for text-to-3D examples, custom image preparation, and sampling settings.
+
+## More resources
+
+- [Project page and method overview](https://plurato.github.io/OctLLM-page/)
+- [Data preparation](https://github.com/octree-nn/octllm/tree/main/dataset_toolkits)
+- [Training](https://github.com/octree-nn/octllm/tree/main/configs)
+- [Evaluation](https://github.com/octree-nn/octllm/tree/main/evaluation)
+
+The paper link and citation information will be added with the paper release.