|
Download README.md from Jack04810/SPAC-4B: direct link, hf CLI and curl.
- Browser
- Download file 7.85 kB
-
https://huggingface.co/Jack04810/SPAC-4B/resolve/main/README.md
- Command line
-
hf download hf://Jack04810/SPAC-4B/README.md
-
curl -L -o README.md https://huggingface.co/Jack04810/SPAC-4B/resolve/main/README.md
7.85 kB
| license: other | |
| base_model: Qwen/Qwen3-4B | |
| language: | |
| - en | |
| pipeline_tag: text-to-3d | |
| tags: | |
| - 3d | |
| - mesh | |
| - text-to-3d | |
| - spatial-pe | |
| - 3d-bpe | |
| - qwen3 | |
| datasets: | |
| - Jack04810/bpe-mix-dataset | |
| model_name: SPAC-4B | |
| <h1 align="center">SPAC-4B</h1> | |
| <p align="center"><b>Sparse Proximity-Aware Compression for 3D Understanding and Generation</b></p> | |
| <p align="center"> | |
| <a href="https://github.com/yangjj007/SPAC-main">Code & Demo</a> · | |
| <a href="https://huggingface.co/datasets/Jack04810/bpe-mix-dataset">Training Data</a> | |
| </p> | |
| **One language model, two directions:** describe a 3D mesh in words, or generate a mesh from a text description. SPAC combines a sparse SDF VQ-VAE, geometry-aware **3D-BPE**, and a Qwen3-4B backbone with explicit spatial positional encoding. | |
| <img src="https://huggingface.co/Jack04810/SPAC-4B/resolve/main/assets/overview.png" alt="SPAC architecture: sparse SDF encoding, 3D-BPE, Morton coordinates, and a shared language model for understanding and generation" width="100%"> | |
| ## Why sparse 3D tokens? | |
| - **Compute near the surface.** Encode a narrow SDF band instead of an entire dense voxel volume. Under the paper's fixed voxel-band assumptions, active inputs grow approximately with surface area, **O(R²)**. | |
| - **Compress in 3D.** Merge recurring face-adjacent patterns using token pairs and their relative offsets. 3D-BPE is reversible at the discrete VQ-token level. | |
| - **Keep coordinates explicit.** Serialize geometry with Morton-coordinate tokens and Spatial-PE so the language model can consume and predict 3D structure. | |
| ## From text to meshes | |
| <img src="https://huggingface.co/Jack04810/SPAC-4B/resolve/main/assets/generation_examples.png" alt="Text-to-3D qualitative comparisons reproduced from the SPAC manuscript" width="100%"> | |
| *Qualitative examples from the manuscript. SPAC predicts SDF geometry; colored appearances in the paper are visualization/postprocessing, while the standalone demo exports an untextured mesh.* | |
| <details> | |
| <summary><b>Mesh understanding examples from the manuscript</b></summary> | |
| <img src="https://huggingface.co/Jack04810/SPAC-4B/resolve/main/assets/caption_examples.png" alt="Mesh captioning case studies from the SPAC manuscript" width="100%"> | |
| </details> | |
| ## Paper highlights | |
| At an equal merge budget, 3D-BPE reduces the leaf-token sequence by approximately **24.6%**: | |
| | Tokenizer | Remaining sequence length ↓ | Bigram entropy ↓ | | |
| |:--|--:|--:| | |
| | **3D-BPE** | **0.754** | **0.972** | | |
| | 1D Morton BPE | 0.803 | 1.092 | | |
| | 1D Hilbert BPE | 0.820 | 1.220 | | |
| | 1D Random BPE | 0.998 | 2.167 | | |
| See the [merge-budget sweep](https://huggingface.co/Jack04810/SPAC-4B/resolve/main/assets/bpe_sweep.png) for additional compression and dependency measurements. | |
| <img src="https://huggingface.co/Jack04810/SPAC-4B/resolve/main/assets/encoder_efficiency.png" alt="Paper encoder comparison across resolutions: FLOPs, latency, prepared-input time, and peak GPU memory" width="100%"> | |
| *At R = 256, the manuscript reports 137.98 GFLOPs and 85.04 ms for the SPAC encoder, versus 41.38 TFLOPs and 4227.84 ms for the dense ShapeLLM encoder, under its reported protocol.* | |
| <details> | |
| <summary><b>Text-to-3D results reported in the manuscript</b></summary> | |
| | Method | CLIP ↑ | FD ↓ | KID ×100 ↓ | | |
| |:--|--:|--:|--:| | |
| | SAR3D | 23.9 | 27.2 | 0.28 | | |
| | TRELLIS | **30.8** | **18.3** | **0.10** | | |
| | ShapeLLM-Omni | 26.7 | 25.9 | 0.25 | | |
| | SPAC | 24.4 | 23.7 | 0.22 | | |
| These figures and tables describe the **paper's experimental models and protocols**. They are not fresh benchmark measurements of the checkpoint distributed on this page. In particular, the latest Toy4k training-overlap replay is a different experiment. | |
| </details> | |
| ## Run SPAC-4B | |
| Requires Linux x86_64, [CUDA Toolkit 12.4](https://developer.nvidia.com/cuda-12-4-0-download-archive) (`nvcc`), a C++ compiler, Google SparseHash headers, and a compatible NVIDIA driver. An A100 40 GB is recommended. Both the LLM and VQ-VAE run in FP32. | |
| ```bash | |
| # Ubuntu build prerequisites | |
| sudo apt-get install -y git build-essential libsparsehash-dev | |
| export CUDA_HOME=/usr/local/cuda-12.4 # use your CUDA 12.4 installation | |
| export PATH="$CUDA_HOME/bin:$PATH" | |
| git clone https://github.com/yangjj007/SPAC-main.git | |
| cd SPAC-main | |
| bash install.sh --dependencies-only | |
| .venv/bin/python download_weights.py | |
| CUDA_VISIBLE_DEVICES=0 .venv/bin/python check_model_compatibility.py --check-vae | |
| ``` | |
| TorchSparse is built from source following [Direct3D-S2](https://github.com/DreamTechAI/Direct3D-S2#installation). The installer runs the upstream clone-and-install recipe automatically, pinned to TorchSparse 2.1.0; SPAC retains its tested PyTorch 2.6.0 / torchvision 0.21.0 / CUDA 12.4 stack. | |
| ```bash | |
| git clone https://github.com/mit-han-lab/torchsparse.git .cache/torchsparse | |
| git -C .cache/torchsparse checkout 385f5ce8718fcae93540511b7f5832f4e71fd835 | |
| .venv/bin/python -m pip install --no-build-isolation .cache/torchsparse | |
| ``` | |
| The downloader pins a compatible release and installs/verifies the LLM, VQ-VAE, BPE merge table, and Spatial-PE manifest at their explicit runtime paths. It disables proxy use. To download through a mirror, append `--endpoint https://hf-mirror.com`. Repository authentication may be required when cloning the code. | |
| ```bash | |
| # Text → untextured mesh, token sequence, and runtime settings | |
| .venv/bin/python infer.py generate "A wooden barrel with metal hoops." --output outputs/barrel | |
| # Mesh → caption, using live mesh encoding | |
| .venv/bin/python infer.py understand examples/barrel/reference.glb --encode-mesh | |
| # Interactive interface | |
| CUDA_VISIBLE_DEVICES=0 bash run.sh --host 127.0.0.1 --port 7860 | |
| ``` | |
| The interface opens at `http://localhost:7860`. Packaged examples can reuse their saved input tokens; `--encode-mesh` runs the complete mesh-to-SDF-to-token path. | |
| ### Runtime file mapping | |
| | Hugging Face path | Local runtime path | | |
| |:--|:--| | |
| | Four `pytorch_model-*.bin` shards, index, config and tokenizer | `models/llm/` | | |
| | `merge_table.json` | `models/merge_table.json` | | |
| | `spatial_pe_manifest.json` | `models/spatial_pe_manifest.json` | | |
| | `vqvae/model.pt`, `vqvae/config.json` | `models/vqvae/` | | |
| The downloader records these mappings and SHA-256 checksums in `models/asset_manifest.json`. The VQ-VAE filename change preserves the checkpoint bytes. | |
| ### Inference defaults | |
| | Setting | Released demo | | |
| |:--|:--| | |
| | Precision / attention | FP32 / SDPA; TF32 and AMP disabled | | |
| | Spatial-PE / RoPE theta | 0.08 / 1,000,000 | | |
| | Generation | Greedy; temperature 0, top-k 1, top-p 1; up to 8192 new tokens | | |
| | Understanding | Greedy; up to 256 new tokens | | |
| | Geometry | SDF resolution 256; explicit coordinate grid 32³ | | |
| The release config declares FP32, RoPE theta 1,000,000, and independent input/output embeddings. The loader also handles the original export's stale tied-embedding flag without overwriting the learned `lm_head`. Geometry inference requires the SPAC wrapper and matching spatial manifest / merge table; standard text-only `AutoModel.generate()` does not add Spatial-PE. An output that reaches the token limit can be incomplete; the demo records `ended_normally` in `result.json`. | |
| ## Checkpoint provenance | |
| **SPAC-4B** uses the LLM checkpoint originally released as [`Jack04810/SPAC-4B-toy4k`](https://huggingface.co/Jack04810/SPAC-4B-toy4k), revision `de08d429d2910e784b7ca0a22731246e26649c06` (`checkpoint-20352`). It is a Toy4k-continuation checkpoint exported with FP32 weights and a 200,832-token vocabulary. Renaming the release does not change its training history. The repository also provides the matching VQ-VAE under `vqvae/`. | |
| The method is described in **“From Dense to Sparse: Morton-Ordered Sparse Mapping for Efficient 3D Mesh Generation.”** The model generates geometry without native textures or materials. License metadata is retained from the source checkpoint. | |