| --- |
| license: mit |
| library_name: pytorch |
| pipeline_tag: image-to-image |
| tags: |
| - multimodal |
| - image-tokenizer |
| - residual-vector-quantization |
| - longcat-next |
| - component-extraction |
| base_model: meituan-longcat/LongCat-Next |
| --- |
| # LongCat-Next: complete standalone visual codec and flow refiner |
|
|
| This bundle retains the native dNaViT visual encoder, bridge, RVQ, visual embedding interface, base pixel decoder, full flow-refinement Transformer, refiner VAE and conditioning projection. The language/audio backbone is excluded. It can reconstruct images from pixels or compatible RVQ IDs; it cannot synthesize images from text by itself. |
|
|
| ## Load the base codec |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| import sys, torch |
| folder=snapshot_download("gatilin/longcat_next_visual_codec") |
| sys.path.insert(0,folder) |
| from component import load_model,encode_pixels,decode_codes |
| codec=load_model(folder) |
| pixels=torch.linspace(0,1,3*112*112).reshape(1,3,112,112) |
| with torch.inference_mode(): |
| ids,grid=encode_pixels(codec,pixels) |
| reconstructed=decode_codes(codec,ids,grid) |
| ``` |
|
|
| `patchify_pixels` accepts NCHW RGB in [0,1], applies native CLIP normalization, duplicates the temporal frame, and packs the original Qwen2.5-VL patch order. Spatial dimensions must be divisible by 28; no implicit resize is performed. `decode_codes` returns native CLIP-normalized RGB tensors. Decode values back to RGB with the same mean/std when displaying. |
|
|
| RVQ uses **8 residual steps with one shared learned codebook**, 16384 valid codes per step and one padding entry. IDs are `[positions,8]` in native residual order. Do not add the language vocabulary offset before passing IDs to this codec. The original global `visual_offset=150581` is preserved in source_config.json for integration with the full LongCat backbone. |
| |
| ## Full flow refinement |
| |
| ```python |
| from component import load_refiner,refine_codes |
| refiner=load_refiner(folder,device="cuda") |
| with torch.inference_mode(): |
| refined=refine_codes(codec,refiner,ids,grid,seed=1019,num_inference_steps=28) |
| ``` |
| |
| The refiner is approximately 7.934 GB of native BF16 Transformer weights, plus the shared VAE and condition projection. Base reconstruction and flow refinement are separate stages; noise and sampling settings affect the latter. Pixel dimensions divisible by both 28 and 16 (e.g. 112,224,448) suit the two-stage image path. CPU compatibility is included; CUDA performance and numerical equivalence are not benchmarked. |
| |
| ## Exact extraction and alias preservation |
| |
| The source main-model visual subtree has 425 serialized tensor entries and 5,314,251,776 payload bytes. Its 21 duplicate codebook/EMA-buffer entries were individually downloaded and verified byte-exact by SHA-256 against their canonical shared values. We store 404 entries, 2,025,358,336 payload bytes, and restore the native shared object/storage aliases in the loader. No quantization or dtype conversion is performed during extraction. |
| |
| `decoder/model.safetensors` contains all base-decoder parameters, the refiner VAE and condition projection. `refiner/model.safetensors` contains every flow Transformer parameter. The additional outer `visual_model` copy in the upstream image-decoder container is unused by the native inference decode path and is excluded; the actual input visual encoder from the main checkpoint is retained. Tensor mapping and per-tensor source hashes for every stage are included. |
| |
| ## Validation |
| |
| Run `python validate.py` and `python validate_refiner.py` after installing requirements. The reports cover strict native loading; shared codebook object/storage; exact patch-layout inversion; 112x112 and rectangular56x84 pixels→8-stage RVQ→base reconstruction; visual embedding interface; refiner VAE and condition projection; full flow-weight loading; conditioned BF16 Transformer forward with reference-image latents; finite Euler update; and conditioning sensitivity. Tests use deterministic synthetic inputs and small spatial grids. They do not establish full-resolution perceptual quality, VQA or text-to-image benchmark performance. |
| |
| CPU uses the preserved Qwen2.5-VL window-attention implementation because newer transformers removed the original get_window_index entry point. FlashAttention calls have a segment-preserving PyTorch fallback; native data layout, attention boundaries and weights remain explicit. The input visual model retains BF16 weights and F32 codebooks; base decoder/VAE are cast to F32 at runtime on CPU. |
| |
| Sources: [official weights](https://huggingface.co/meituan-longcat/LongCat-Next), [official code](https://github.com/meituan-longcat/LongCat-Next), [paper](https://arxiv.org/abs/2603.27538). Source revisions, modifications, separate MIT/Apache notices and exact extraction records accompany the release. |
| |
| The combined CPU `refine_codes` integration smoke run was stopped on the test host because of memory/swap pressure while the encoder and full refiner were resident together. No end-to-end pipeline pass is claimed. Base-codec and complete refiner component tests above passed independently. |
| |
