---
license: mit
library_name: pytorch
pipeline_tag: image-to-image
base_model: mit-han-lab/vila-u-7b-256
tags:
- vila-u
- residual-quantization
- unified-multimodal
- component-extraction
---
# VILA-U 256 complete visual codec, depth predictor and projector

The complete original [VILA-U-256 visual package](https://huggingface.co/mit-han-lab/vila-u-7b-256/tree/91f97e4b4746e6c2a27a168726ff5a2b9fe0d850): **1107 visual entries plus the separate four-entry projector**. Both official safetensors files are unchanged and match their original LFS SHA-256 values. The trained SigLIP image/text branches, residual quantizer, RGB decoder, six-layer depth predictor and language-space projector are retained. The separate Llama language model and its tokenizer are excluded.

```python
from huggingface_hub import snapshot_download
import sys, torch
folder = snapshot_download('gatilin/vila_u_256_visual_codec')
sys.path.insert(0, folder)
from component import load_model, load_projector
visual = load_model(folder)
projector = load_projector(folder)
rgb = torch.zeros(1, 3, 256, 256)  # native RGB NCHW, [-1,1]
with torch.inference_mode():
    ids, quantized = visual.rqvaesiglip.encode_image(rgb)
    # ids: [B,16,16,4]. Four residual depths use one shared codebook.
    vectors = visual.rqvaesiglip.quantizer.embed_code(ids)
    reconstruction = visual.rqvaesiglip.decode(vectors)
    language_features = projector(vectors.flatten(1,2))  # [B,256,4096]
```

Install `requirements.txt` in a dedicated environment. The native visual architecture is pinned to [mit-han-lab/vila-u revision 4ea42ea](https://github.com/mit-han-lab/vila-u/tree/4ea42ead2ce22b035ee74c7966eed245bca3e927). The SigLIP architecture config is bundled at its recorded fixed HF revision, avoiding runtime metadata or redundant pretrained-weight downloads. The native depth predictor is available as `visual.rqtransformer`; its body input must come from the correct full-model language-state interface for actual generation.

There are **16384 valid IDs**, from 0 through 16383. A 16385th stored embedding row is padding. Four depths share one codebook; they are not four independently trained vocabularies. The native straight-through arithmetic can differ slightly from direct lookup; its bound and exact canonical depth-sum/decode paths are recorded in `validation.json`.

Executed CPU float32 validation covers official file hashes, strict loading, all 1111 restored state values, shared-codebook/padding rules, fixed 256px encoding and reconstruction, language-space projection, short depth teacher-forcing and native four-depth generation/lookup, and the retained trained text tower. Synthetic fixtures do not establish paper image quality, VQA or complete language-conditioned generation. Re-run with `python validate.py`.

Portability changes are documented in `code_provenance.json`: Python 3.11+ dataclass default-factory compatibility, bundled SigLIP architecture metadata, and preservation of the input dtype after interpolation rather than hardcoded bfloat16. The original bfloat16 upsampling operation is tested exactly; a complete CUDA bfloat16 pipeline was not tested. Weight bytes are unchanged.

The official weight card and project use MIT (`LICENSE`). Apache-licensed Hugging Face/Google source notices and the original KakaoBrain RQ-VAE terms remain applicable; see `APACHE_LICENSE`, `RQ_VAE_LICENSE` and the unchanged file headers. The source model card, original configurations, extraction provenance and validation report are included.