| --- |
| license: mit |
| library_name: pytorch |
| pipeline_tag: image-to-image |
| base_model: mit-han-lab/vila-u-7b-256 |
| tags: |
| - vila-u |
| - residual-quantization |
| - unified-multimodal |
| - component-extraction |
| --- |
| # VILA-U 256 complete visual codec, depth predictor and projector |
|
|
| The complete original [VILA-U-256 visual package](https://huggingface.co/mit-han-lab/vila-u-7b-256/tree/91f97e4b4746e6c2a27a168726ff5a2b9fe0d850): **1107 visual entries plus the separate four-entry projector**. Both official safetensors files are unchanged and match their original LFS SHA-256 values. The trained SigLIP image/text branches, residual quantizer, RGB decoder, six-layer depth predictor and language-space projector are retained. The separate Llama language model and its tokenizer are excluded. |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| import sys, torch |
| folder = snapshot_download('gatilin/vila_u_256_visual_codec') |
| sys.path.insert(0, folder) |
| from component import load_model, load_projector |
| visual = load_model(folder) |
| projector = load_projector(folder) |
| rgb = torch.zeros(1, 3, 256, 256) # native RGB NCHW, [-1,1] |
| with torch.inference_mode(): |
| ids, quantized = visual.rqvaesiglip.encode_image(rgb) |
| # ids: [B,16,16,4]. Four residual depths use one shared codebook. |
| vectors = visual.rqvaesiglip.quantizer.embed_code(ids) |
| reconstruction = visual.rqvaesiglip.decode(vectors) |
| language_features = projector(vectors.flatten(1,2)) # [B,256,4096] |
| ``` |
|
|
| Install `requirements.txt` in a dedicated environment. The native visual architecture is pinned to [mit-han-lab/vila-u revision 4ea42ea](https://github.com/mit-han-lab/vila-u/tree/4ea42ead2ce22b035ee74c7966eed245bca3e927). The SigLIP architecture config is bundled at its recorded fixed HF revision, avoiding runtime metadata or redundant pretrained-weight downloads. The native depth predictor is available as `visual.rqtransformer`; its body input must come from the correct full-model language-state interface for actual generation. |
|
|
| There are **16384 valid IDs**, from 0 through 16383. A 16385th stored embedding row is padding. Four depths share one codebook; they are not four independently trained vocabularies. The native straight-through arithmetic can differ slightly from direct lookup; its bound and exact canonical depth-sum/decode paths are recorded in `validation.json`. |
|
|
| Executed CPU float32 validation covers official file hashes, strict loading, all 1111 restored state values, shared-codebook/padding rules, fixed 256px encoding and reconstruction, language-space projection, short depth teacher-forcing and native four-depth generation/lookup, and the retained trained text tower. Synthetic fixtures do not establish paper image quality, VQA or complete language-conditioned generation. Re-run with `python validate.py`. |
|
|
| Portability changes are documented in `code_provenance.json`: Python 3.11+ dataclass default-factory compatibility, bundled SigLIP architecture metadata, and preservation of the input dtype after interpolation rather than hardcoded bfloat16. The original bfloat16 upsampling operation is tested exactly; a complete CUDA bfloat16 pipeline was not tested. Weight bytes are unchanged. |
|
|
| The official weight card and project use MIT (`LICENSE`). Apache-licensed Hugging Face/Google source notices and the original KakaoBrain RQ-VAE terms remain applicable; see `APACHE_LICENSE`, `RQ_VAE_LICENSE` and the unchanged file headers. The source model card, original configurations, extraction provenance and validation report are included. |
|
|
