Extract pinned native components; verify source bytes, strict loading and native encode/decode interfaces

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Meituan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in 
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

LICENSES.md

@@ -0,0 +1 @@
+LongCat-Next source weights and native code: MIT, retained in LICENSE. The compatibility Qwen2.5-VL window-attention implementation retains its original Apache-2.0 notices in native/qwen2_5_vl_vit.py. Other per-file upstream notices remain intact. The language and audio components are excluded.

README.md

@@ -0,0 +1,59 @@
+---
+license: mit
+library_name: pytorch
+pipeline_tag: image-to-image
+tags:
+- multimodal
+- image-tokenizer
+- residual-vector-quantization
+- longcat-next
+- component-extraction
+base_model: meituan-longcat/LongCat-Next
+---
+# LongCat-Next: complete standalone visual codec and flow refiner
+
+This bundle retains the native dNaViT visual encoder, bridge, RVQ, visual embedding interface, base pixel decoder, full flow-refinement Transformer, refiner VAE and conditioning projection. The language/audio backbone is excluded. It can reconstruct images from pixels or compatible RVQ IDs; it cannot synthesize images from text by itself.
+
+## Load the base codec
+
+```python
+from huggingface_hub import snapshot_download
+import sys, torch
+folder=snapshot_download("gatilin/longcat_next_visual_codec")
+sys.path.insert(0,folder)
+from component import load_model,encode_pixels,decode_codes
+codec=load_model(folder)
+pixels=torch.linspace(0,1,3*112*112).reshape(1,3,112,112)
+with torch.inference_mode():
+    ids,grid=encode_pixels(codec,pixels)
+    reconstructed=decode_codes(codec,ids,grid)
+```
+
+`patchify_pixels` accepts NCHW RGB in [0,1], applies native CLIP normalization, duplicates the temporal frame, and packs the original Qwen2.5-VL patch order. Spatial dimensions must be divisible by 28; no implicit resize is performed. `decode_codes` returns native CLIP-normalized RGB tensors. Decode values back to RGB with the same mean/std when displaying.
+
+RVQ uses **8 residual steps with one shared learned codebook**, 16384 valid codes per step and one padding entry. IDs are `[positions,8]` in native residual order. Do not add the language vocabulary offset before passing IDs to this codec. The original global `visual_offset=150581` is preserved in source_config.json for integration with the full LongCat backbone.
+
+## Full flow refinement
+
+```python
+from component import load_refiner,refine_codes
+refiner=load_refiner(folder,device="cuda")
+with torch.inference_mode():
+    refined=refine_codes(codec,refiner,ids,grid,seed=1019,num_inference_steps=28)
+```
+
+The refiner is approximately 7.934 GB of native BF16 Transformer weights, plus the shared VAE and condition projection. Base reconstruction and flow refinement are separate stages; noise and sampling settings affect the latter. Pixel dimensions divisible by both 28 and 16 (e.g. 112,224,448) suit the two-stage image path. CPU compatibility is included; CUDA performance and numerical equivalence are not benchmarked.
+
+## Exact extraction and alias preservation
+
+The source main-model visual subtree has 425 serialized tensor entries and 5,314,251,776 payload bytes. Its 21 duplicate codebook/EMA-buffer entries were individually downloaded and verified byte-exact by SHA-256 against their canonical shared values. We store 404 entries, 2,025,358,336 payload bytes, and restore the native shared object/storage aliases in the loader. No quantization or dtype conversion is performed during extraction.
+
+`decoder/model.safetensors` contains all base-decoder parameters, the refiner VAE and condition projection. `refiner/model.safetensors` contains every flow Transformer parameter. The additional outer `visual_model` copy in the upstream image-decoder container is unused by the native inference decode path and is excluded; the actual input visual encoder from the main checkpoint is retained. Tensor mapping and per-tensor source hashes for every stage are included.
+
+## Validation
+
+Run `python validate.py` and `python validate_refiner.py` after installing requirements. The reports cover strict native loading; shared codebook object/storage; exact patch-layout inversion; 112x112 and rectangular56x84 pixels→8-stage RVQ→base reconstruction; visual embedding interface; refiner VAE and condition projection; full flow-weight loading; conditioned BF16 Transformer forward with reference-image latents; finite Euler update; and conditioning sensitivity. Tests use deterministic synthetic inputs and small spatial grids. They do not establish full-resolution perceptual quality, VQA or text-to-image benchmark performance.
+
+CPU uses the preserved Qwen2.5-VL window-attention implementation because newer transformers removed the original get_window_index entry point. FlashAttention calls have a segment-preserving PyTorch fallback; native data layout, attention boundaries and weights remain explicit. The input visual model retains BF16 weights and F32 codebooks; base decoder/VAE are cast to F32 at runtime on CPU.
+
+Sources: [official weights](https://huggingface.co/meituan-longcat/LongCat-Next), [official code](https://github.com/meituan-longcat/LongCat-Next), [paper](https://arxiv.org/abs/2603.27538). Source revisions, modifications, separate MIT/Apache notices and exact extraction records accompany the release.

alias_verification.json

@@ -0,0 +1,86 @@
+{
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.1.embed": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+    "sha256": "fc65e921fe9fd960a9e623e2417d32b0439d1fa82fa45af99af03319e7a6eda3"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.1.embed_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+    "sha256": "b190da7cc5c132efb5dc66587dbe9fc90df832f8e5e28e8eea0c59c84c1668a1"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.2.embed": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+    "sha256": "fc65e921fe9fd960a9e623e2417d32b0439d1fa82fa45af99af03319e7a6eda3"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.2.embed_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+    "sha256": "b190da7cc5c132efb5dc66587dbe9fc90df832f8e5e28e8eea0c59c84c1668a1"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.3.embed": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+    "sha256": "fc65e921fe9fd960a9e623e2417d32b0439d1fa82fa45af99af03319e7a6eda3"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.3.embed_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+    "sha256": "b190da7cc5c132efb5dc66587dbe9fc90df832f8e5e28e8eea0c59c84c1668a1"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.4.embed": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+    "sha256": "fc65e921fe9fd960a9e623e2417d32b0439d1fa82fa45af99af03319e7a6eda3"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.4.embed_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+    "sha256": "b190da7cc5c132efb5dc66587dbe9fc90df832f8e5e28e8eea0c59c84c1668a1"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.5.embed": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+    "sha256": "fc65e921fe9fd960a9e623e2417d32b0439d1fa82fa45af99af03319e7a6eda3"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.5.embed_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+    "sha256": "b190da7cc5c132efb5dc66587dbe9fc90df832f8e5e28e8eea0c59c84c1668a1"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.6.embed": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+    "sha256": "fc65e921fe9fd960a9e623e2417d32b0439d1fa82fa45af99af03319e7a6eda3"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.6.embed_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+    "sha256": "b190da7cc5c132efb5dc66587dbe9fc90df832f8e5e28e8eea0c59c84c1668a1"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.7.embed": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+    "sha256": "fc65e921fe9fd960a9e623e2417d32b0439d1fa82fa45af99af03319e7a6eda3"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.7.embed_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+    "sha256": "b190da7cc5c132efb5dc66587dbe9fc90df832f8e5e28e8eea0c59c84c1668a1"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.1.cluster_size_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+    "sha256": "4e9a6e4b9b1e203ebe18ed20095cb95c429a537a3cf814c7c5b357b823607850"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.2.cluster_size_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+    "sha256": "4e9a6e4b9b1e203ebe18ed20095cb95c429a537a3cf814c7c5b357b823607850"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.3.cluster_size_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+    "sha256": "4e9a6e4b9b1e203ebe18ed20095cb95c429a537a3cf814c7c5b357b823607850"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.4.cluster_size_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+    "sha256": "4e9a6e4b9b1e203ebe18ed20095cb95c429a537a3cf814c7c5b357b823607850"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.5.cluster_size_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+    "sha256": "4e9a6e4b9b1e203ebe18ed20095cb95c429a537a3cf814c7c5b357b823607850"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.6.cluster_size_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+    "sha256": "4e9a6e4b9b1e203ebe18ed20095cb95c429a537a3cf814c7c5b357b823607850"
+  },
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.7.cluster_size_ema": {
+    "canonical": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+    "sha256": "4e9a6e4b9b1e203ebe18ed20095cb95c429a537a3cf814c7c5b357b823607850"
+  }
+}

aliases.json

@@ -0,0 +1,23 @@
+{
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.1.embed": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.1.embed_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.2.embed": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.2.embed_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.3.embed": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.3.embed_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.4.embed": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.4.embed_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.5.embed": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.5.embed_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.6.embed": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.6.embed_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.7.embed": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.7.embed_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.1.cluster_size_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.2.cluster_size_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.3.cluster_size_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.4.cluster_size_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.5.cluster_size_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.6.cluster_size_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.7.cluster_size_ema": "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema"
+}

code_provenance.json

@@ -0,0 +1,11 @@
+{
+  "source_repo": "meituan-longcat/LongCat-Next",
+  "source_revision": "0cf0631862402ff36366e513e4023d22e7e5c84c",
+  "compatibility_base": "Qwen2.5-VL window-attention implementation from the previously pinned/tested Lance release; original notices retained.",
+  "changes": [
+    "CPU selects eager attention instead of forced FlashAttention",
+    "Lazy import of optional image-refiner pipeline",
+    "PyTorch segmented-attention fallback for CPU",
+    "No tensor mutation during extraction; CPU decoder/VAE runtime casts are explicit."
+  ]
+}

component.py

@@ -0,0 +1,83 @@
+"""LongCat-Next dNaViT/RVQ codec and base pixel decoder. Full AR backbone excluded."""
+from pathlib import Path
+import json,torch
+from torch import nn
+from accelerate import init_empty_weights
+from safetensors.torch import load_file
+from native.configuration_longcat_next import LongcatNextConfig
+from native.visual import LongcatNextVisualTokenizer,VisionTransformerDecoder
+
+def build(config,decoder=False,decoder_state=None):
+    cfg=LongcatNextConfig(**config);m=nn.Module();m.model=nn.Module();m.model.visual_tokenizer=LongcatNextVisualTokenizer(cfg)
+    if decoder:
+        from diffusers import AutoencoderKL
+        m.image_decoder=VisionTransformerDecoder(cfg.visual_config.visual_decoder_config.image_decoder_config)
+        m.image_refiner=nn.Module();m.image_refiner.vae=AutoencoderKL.from_config(config["visual_config"]["visual_decoder_config"]["vae_config"])
+        w=decoder_state["image_refiner.cond_proj.weight"];m.image_refiner.cond_proj=nn.Linear(w.shape[1],w.shape[0],bias=True)
+    return m
+
+def load_model(path=None,device="cpu",decoder=True):
+    root=Path(path or Path(__file__).parent);c=json.loads((root/"source_config.json").read_text());s=load_file(root/"model.safetensors")
+    aliases=json.loads((root/"aliases.json").read_text())
+    for alias,canonical in aliases.items():s[alias]=s[canonical]
+    ds=load_file(root/"decoder/model.safetensors") if decoder else None
+    if ds:s.update(ds)
+    with init_empty_weights():m=build(c,decoder,ds)
+    m.load_state_dict(s,strict=True,assign=True);m=m.to(device=device).eval()
+    # Preserve F32 codebooks and BF16 input weights. Base decoder and optional VAE
+    # use F32 on CPU so quantized F32 features do not hit BF16 linear weights.
+    if decoder:m.image_decoder.float();m.image_refiner.float()
+    return m
+
+def patchify_pixels(pixels):
+    """NCHW RGB in [0,1]; no resize. Spatial dimensions must be divisible by 28."""
+    if pixels.ndim!=4 or pixels.shape[1]!=3 or any(v%28 for v in pixels.shape[-2:]):raise ValueError("Expected NCHW RGB with dimensions divisible by 28")
+    if pixels.min()<0 or pixels.max()>1:raise ValueError("Expected pixels in [0,1]")
+    mean=pixels.new_tensor([.48145466,.4578275,.40821073])[None,:,None,None];std=pixels.new_tensor([.26862954,.26130258,.27577711])[None,:,None,None]
+    x=(pixels-mean)/std;patches=[];grids=[]
+    for image in x:
+        _,h,w=image.shape;gh,gw=h//14,w//14
+        p=image.unsqueeze(0).expand(2,-1,-1,-1).reshape(1,2,3,gh//2,2,14,gw//2,2,14).permute(0,3,6,4,7,2,1,5,8).reshape(gh*gw,-1)
+        patches.append(p);grids.append([1,gh,gw])
+    return torch.cat(patches),torch.tensor(grids,device=pixels.device,dtype=torch.long)
+
+def encode_pixels(model,pixels):
+    patches,grid=patchify_pixels(pixels);ids=model.model.visual_tokenizer.encode(patches,grid)
+    return ids,grid
+
+def code_features(model,ids):
+    if ids.shape[-1]!=8 or ids.min()<0 or ids.max()>=16384:raise ValueError("Expected 8 residual-stage IDs in [0,16383]")
+    books=model.model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks
+    return sum(books[i].embed[ids[...,i].long()] for i in range(8))
+
+def decode_codes(model,ids,grid_thw):
+    features=code_features(model,ids).reshape(-1,3584).to(next(model.image_decoder.parameters()))
+    decoded=model.image_decoder(features,grid_thw)["images"]
+    return decoded # native CLIP-normalized RGB; first frame of each temporal patch
+
+
+def load_refiner(path=None,device="cpu"):
+    """Load the complete native flow Transformer, refiner VAE and conditioning projection."""
+    from native.image_refiner import ImageRefinerContainer
+    root=Path(path or Path(__file__).parent);cfg=LongcatNextConfig(**json.loads((root/"source_config.json").read_text())).visual_config.visual_decoder_config
+    with init_empty_weights():m=ImageRefinerContainer(cfg)
+    state={k.removeprefix("image_refiner."):v for k,v in load_file(root/"decoder/model.safetensors").items() if k.startswith("image_refiner.")}
+    state.update({k.removeprefix("image_refiner."):v for k,v in load_file(root/"refiner/model.safetensors").items()})
+    m.load_state_dict(state,strict=True,assign=True)
+    return m.to(device=device).eval()
+
+def refine_codes(codec,refiner,ids,grid_thw,seed=1019,num_inference_steps=28,output_type="pt"):
+    """Native flow refinement of the base reconstruction. Requires both bundles loaded."""
+    from native.image_refiner import RefinerPipeline,RefinerImageProcessor,de_transform
+    from native.refiner_modules import FlowMatchEulerDiscreteScheduler
+    device=next(refiner.parameters()).device;cfg=json.loads((Path(__file__).parent/"source_config.json").read_text())["visual_config"]["visual_decoder_config"]
+    scheduler=FlowMatchEulerDiscreteScheduler(**cfg["scheduler_config"])
+    pipeline=RefinerPipeline(vae=refiner.vae,transformer=refiner.base_transformer,cond_proj=refiner.cond_proj,scheduler=scheduler)
+    raw=decode_codes(codec,ids,grid_thw);images=[]
+    for image in raw:
+        x=de_transform(image,mean=[.48145466,.4578275,.40821073],std=[.26862954,.26130258,.27577711],rescale_factor=1/255)
+        images.append(RefinerImageProcessor.normalize(x).squeeze(0).to(device))
+    features=code_features(codec,ids)
+    if features.ndim==2:features=features.unsqueeze(0)
+    result=pipeline(encoder_hidden_states=features.to(device=device,dtype=next(refiner.cond_proj.parameters()).dtype),grid_thw_list=[tuple(map(int,g)) for g in grid_thw],image=images,generator=torch.Generator(device=device).manual_seed(seed),num_inference_steps=num_inference_steps,output_type=output_type,return_dict=True,enable_processor_bar=False)
+    return result.images

decoder/alias_verification.json

@@ -0,0 +1 @@
+{}

decoder/aliases.json

@@ -0,0 +1 @@
+{}

decoder/extraction.json

@@ -0,0 +1,11 @@
+{
+  "source_repo": "meituan-longcat/LongCat-Next",
+  "source_revision": "0cf0631862402ff36366e513e4023d22e7e5c84c",
+  "source_tensor_entries": 804,
+  "stored_tensor_entries": 804,
+  "tensor_bytes": 1049811242,
+  "source_tensor_bytes_equal": true,
+  "aliases_verified": 0,
+  "sha256": "2a0e46b2a40ed901d5b0f839df183cb8b679cd43975b59211c03999164327f71",
+  "transform": "Selection only; names, shapes, dtype and payload bytes unchanged."
+}

decoder/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a0e46b2a40ed901d5b0f839df183cb8b679cd43975b59211c03999164327f71
+size 1049912922

decoder/source_config.json

@@ -0,0 +1,286 @@
+{
+  "architectures": [
+    "LongcatNextForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_longcat_next.LongcatNextConfig",
+    "AutoModel": "modeling_longcat_next.LongcatNextModel",
+    "AutoModelForCausalLM": "modeling_longcat_next.LongcatNextForCausalLM"
+  },
+  "vocab_size": 282624,
+  "hidden_size": 3072,
+  "ffn_hidden_size": 6144,
+  "expert_ffn_hidden_size": 1024,
+  "num_layers": 14,
+  "num_attention_heads": 32,
+  "kv_lora_rank": 512,
+  "q_lora_rank": 1536,
+  "qk_rope_head_dim": 64,
+  "v_head_dim": 128,
+  "qk_nope_head_dim": 128,
+  "mla_scale_q_lora": true,
+  "mla_scale_kv_lora": true,
+  "routed_scaling_factor": 6.0,
+  "n_routed_experts": 256,
+  "rms_norm_eps": 1e-5,
+  "use_cache": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "rope_theta": 10000000,
+  "max_position_embeddings": 131072,
+  "zero_expert_num": 128,
+  "zero_expert_type": "identity",
+  "moe_topk": 12,
+  "ngram_vocab_size_ratio": 78,
+  "emb_neighbor_num": 4,
+  "emb_split_num": 4,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.57.6",
+  "model_type": "longcat_next",
+
+  "text_vocab_size": 131072,
+  "text_vocab_plus_multimodal_special_token_size": 131125,
+  "visual_embedding_layer_intermediate_size": 8192,
+  "visual_embedding_layer_hidden_act": "silu",
+  "visual_offset": 150581,
+  "audio_offset": 131125,
+
+
+  "visual_config": {
+    "image_start_token_id": 131106,
+    "image_end_token_id": 131107,
+    "image_pad_token_id": 131108,
+    "image_newline_token_id": 131109,
+
+    "_attn_implementation": "flash_attention_2",
+    "hidden_size": 1280,
+
+    "image_head_transformer_dims": 2048,
+    "image_head_transformer_ffn_scale": 16,
+    "image_head_transformer_layers": 4,
+
+    "vq_config": {
+      "codebook_dim": 3584,
+      "codebook_size": 16384,
+      "codebook_sizes": [
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384
+      ],
+      "decay": 0.99,
+      "depth": 8,
+
+      "commit_loss_ratio": 0.25,
+      "entropy_loss_ratio": 0,
+
+      "in_channels": 3584,
+      "quant_conv": true,
+      "quantizer_type": "rq",
+      "restart_unused_codes": true,
+      "shared_codebook": true,
+
+      "vq_loss_ratio": 0
+    },
+
+    "visual_decoder_config": {
+      "codebook_dim": 3584,
+
+      "image_decoder_config": {
+        "attention_dropout": 0.0,
+        "codebook_dim": 3584,
+        "distill_taps": [
+          3,
+          7,
+          15,
+          23
+        ],
+        "hidden_act": "gelu",
+        "hidden_size": 1024,
+        "intermediate_size": 2730,
+        "k_bias": false,
+        "layer_norm_eps": 1e-06,
+        "num_attention_heads": 16,
+        "num_hidden_layers": 32,
+        "patch_size": 14,
+        "q_bias": true,
+        "spatial_merge_size": 2,
+        "subln": true,
+        "swiglu": true,
+        "teacher_dims": {
+          "15": 1280,
+          "23": 1280,
+          "3": 1280,
+          "7": 1280
+        },
+        "temporal_patch_size": 2,
+        "v_bias": true
+      },
+
+      "transformer_config": {
+        "patch_size": 2,
+        "in_channels": 16,
+        "hidden_size": 2520,
+        "num_layers": 32,
+        "num_refiner_layers": 2,
+        "num_attention_heads": 21,
+        "num_kv_heads": 7,
+        "multiple_of": 256,
+        "norm_eps": 1e-5,
+        "axes_dim_rope": [40, 40, 40],
+        "axes_lens": [10000, 10000, 10000],
+        "text_feat_dim": 2048,
+        "timestep_scale": 1000.0
+      },
+
+      "vae_config": {
+        "act_fn": "silu",
+        "block_out_channels": [128, 256, 512, 512],
+        "down_block_types": [
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D"
+        ],
+        "in_channels": 3,
+        "latent_channels": 16,
+        "layers_per_block": 2,
+        "mid_block_add_attention": true,
+        "norm_num_groups": 32,
+        "out_channels": 3,
+        "sample_size": 1024,
+        "scaling_factor": 0.3611,
+        "shift_factor": 0.1159,
+        "up_block_types": [
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D"
+        ],
+        "use_post_quant_conv": false,
+        "use_quant_conv": false,
+        "force_upcast": true
+      },
+
+      "scheduler_config": {
+        "num_train_timesteps": 1000,
+        "dynamic_time_shift": true
+      },
+
+      "weight_path": "WEIGHT_PATH_TO_LONGCAT_NEXT/image_decoder/image_decoder.safetensors"
+    }
+  },
+
+
+  "audio_config": {
+    "audio_head_transformer_dims": 3072,
+    "audio_head_transformer_ffn_scale": 16,
+    "audio_head_transformer_layers": 4,
+
+    "audio_delim_token_id": 131116,
+    "audio_end_token_id": 131104,
+    "audio_pad_token_id": 131105,
+    "audio_start_token_id": 131103,
+    "audiogen_end_token_id": 131124,
+    "audiogen_start_token_id": 131123,
+    "audiotext_end_token_id": 131121,
+    "audiotext_pad_token_id": 131122,
+    "audiotext_start_token_id": 131120,
+
+    "_attn_implementation": "flash_attention_2",
+    "d_model": 1280,
+    "decoder_attention_heads": 20,
+    "decoder_ffn_dim": 5120,
+    "decoder_layers": 8,
+    "encoder_attention_heads": 20,
+    "encoder_ffn_dim": 5120,
+    "encoder_layers": 32,
+    "num_mel_bins": 128,
+
+    "avg_pooler": 4,
+    "decoder_kernel_size": 3,
+    "decoder_stride_size": 2,
+    "hop_length": 160,
+    "kernel_size": 3,
+    "max_audio_seconds": 30,
+    "n_fft": 400,
+    "num_hidden_layers": 32,
+    "sampling_rate": 16000,
+    "stride_size": 2,
+
+    "vq_config": {
+      "codebook_sizes": [
+        8192,
+        4096,
+        2048,
+        1024,
+        1024,
+        1024,
+        1024,
+        1024
+      ]
+    },
+
+    "vocoder_config": {
+      "channels": [
+        256,
+        256,
+        256,
+        256,
+        256
+      ],
+      "hop_length": 256,
+      "num_mel_bins": 80,
+      "sampling_rate": 16000
+    },
+
+
+    "flow_matching_config": {
+      "in_channels": 80,
+      "spk_emb_dim": 0,
+      "diffusion_steps": 10,
+      "cal_mel_mae": true,
+
+      "prenet_activation_function": "gelu",
+      "prenet_attention_heads": 8,
+      "prenet_d_model": 512,
+      "prenet_ffn_dim": 2048,
+      "prenet_in_dim": 1280,
+      "prenet_max_source_positions": 5000,
+      "prenet_nlayers": 12,
+      "prenet_out_dim": 80,
+      "prenet_target_mel_length_scale_ratio": 1.0,
+
+      "channels": [
+        256
+      ],
+      "dropout": 0.0,
+      "attention_head_dim": 64,
+      "n_blocks": 4,
+      "num_heads": 8,
+      "num_mid_blocks": 12,
+      "act_fn": "gelu",
+
+      "cfm_params": {
+        "inference_cfg_rate": 0.7,
+        "sigma_min": 1e-06,
+        "solver": "euler",
+        "t_scheduler": "cosine",
+        "training_cfg_rate": 0.2
+      },
+
+      "use_hidden_states_before_dconv2": true
+    },
+
+    "cosy24kvocoder_config": {
+      "weight_path": "WEIGHT_PATH_TO_LONGCAT_NEXT/cosy24k_vocoder/hift.pt"
+    }
+
+  }
+}

extraction.json

@@ -0,0 +1,11 @@
+{
+  "source_repo": "meituan-longcat/LongCat-Next",
+  "source_revision": "0cf0631862402ff36366e513e4023d22e7e5c84c",
+  "source_tensor_entries": 425,
+  "stored_tensor_entries": 404,
+  "tensor_bytes": 2025358336,
+  "source_tensor_bytes_equal": true,
+  "aliases_verified": 21,
+  "sha256": "a57489db93bc60c1167c9f565c3a3f85d4ff1505477cd3b9ff95ef390fa3fe35",
+  "transform": "Original selected tensor bytes; exact-value shared aliases stored once and restored by loader."
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a57489db93bc60c1167c9f565c3a3f85d4ff1505477cd3b9ff95ef390fa3fe35
+size 2025414272

native/compat.py

@@ -0,0 +1,16 @@
+"""Segment-preserving PyTorch fallback for CPU FlashAttention calls."""
+import torch
+from torch.nn import functional as F
+def flash_attn_varlen_func(q,k,v,cu_seqlens_q,cu_seqlens_k,max_seqlen_q,max_seqlen_k,dropout_p=0.,softmax_scale=None,causal=False,**kwargs):
+    output=torch.empty_like(q)
+    for i in range(len(cu_seqlens_q)-1):
+        a,b=int(cu_seqlens_q[i]),int(cu_seqlens_q[i+1]);c,d=int(cu_seqlens_k[i]),int(cu_seqlens_k[i+1])
+        if a==b:continue
+        qq=q[a:b].transpose(0,1).unsqueeze(0);kk=k[c:d].transpose(0,1).unsqueeze(0);vv=v[c:d].transpose(0,1).unsqueeze(0)
+        y=F.scaled_dot_product_attention(qq,kk,vv,dropout_p=dropout_p,is_causal=causal,scale=softmax_scale,enable_gqa=qq.shape[1]!=kk.shape[1]);output[a:b]=y.squeeze(0).transpose(0,1)
+    return output
+def index_first_axis(x,indices):return x[indices]
+def unpad_input(x,attention_mask):
+    lengths=attention_mask.sum(-1,dtype=torch.int32);indices=torch.nonzero(attention_mask.flatten(),as_tuple=False).flatten();cu=F.pad(lengths.cumsum(0,dtype=torch.int32),(1,0));return x.flatten(0,1)[indices],indices,cu,int(lengths.max())
+def pad_input(x,indices,batch,seqlen):
+    y=x.new_zeros((batch*seqlen,*x.shape[1:]));y[indices]=x;return y.reshape(batch,seqlen,*x.shape[1:])

native/configuration_longcat_next.py

@@ -0,0 +1,153 @@
+from transformers.configuration_utils import PretrainedConfig
+from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import Qwen2_5_VLVisionConfig
+from transformers.models.whisper.configuration_whisper import WhisperConfig
+
+from .configuration_longcat_ngram import LongcatFlashNgramConfig
+
+class LongcatNextConfig(LongcatFlashNgramConfig):
+    model_type = "longcat_next"
+    def __init__(
+        self,
+        vocab_size=131072,
+        hidden_size=6144,
+        num_hidden_layers=56,
+        num_layers=28,
+        num_attention_heads=64,
+        num_key_value_heads=None,
+        hidden_act="silu",
+        max_position_embeddings=131072,
+        initializer_range=0.02,
+        rms_norm_eps=1e-5,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=1,
+        eos_token_id=2,
+        tie_word_embeddings=False,
+        rope_theta=10000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        ffn_hidden_size=12288,
+        q_lora_rank=1536,
+        kv_lora_rank=512,
+        qk_nope_head_dim=128,
+        qk_rope_head_dim=64,
+        head_dim=64,
+        v_head_dim=128,
+        qk_head_dim=None,
+        moe_topk=12,
+        n_routed_experts=512,
+        zero_expert_num=256,
+        expert_ffn_hidden_size=2048,
+        routed_scaling_factor=6.0,
+        emb_neighbor_num=None,
+        emb_split_num=None,
+        ngram_vocab_size_ratio=None,
+        oe_ignored_token_ids=[],
+        text_vocab_size=131072, # text vocab size (vocab_size = text_vocab_size + audio_token + visual_token + multimodal_special_token_list)
+        text_vocab_plus_multimodal_special_token_size=131125,
+        visual_embedding_layer_intermediate_size=8192,
+        visual_embedding_layer_hidden_act="silu",
+        visual_offset=150581,
+        audio_offset=131125,
+        visual_config={},
+        audio_config={},
+        **kwargs,
+    ):
+        self.text_vocab_size = text_vocab_size
+        self.text_vocab_plus_multimodal_special_token_size = text_vocab_plus_multimodal_special_token_size
+        self.visual_embedding_layer_intermediate_size = visual_embedding_layer_intermediate_size
+        self.visual_embedding_layer_hidden_act = visual_embedding_layer_hidden_act
+        self.visual_offset = visual_offset
+        self.audio_offset = audio_offset
+        self.visual_config = LongcatNextVisualConfig(**visual_config)
+        self.audio_config = LongcatNextAudioConfig(**audio_config)
+        oe_ignored_token_ids = oe_ignored_token_ids or list(range(self.text_vocab_size, self.text_vocab_plus_multimodal_special_token_size))
+
+        super().__init__(
+            vocab_size=vocab_size,
+            hidden_size=hidden_size,
+            num_hidden_layers=num_hidden_layers,
+            num_layers=num_layers,
+            num_attention_heads=num_attention_heads,
+            num_key_value_heads=num_key_value_heads,
+            hidden_act=hidden_act,
+            max_position_embeddings=max_position_embeddings,
+            initializer_range=initializer_range,
+            rms_norm_eps=rms_norm_eps,
+            use_cache=use_cache,
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            rope_theta=rope_theta,
+            rope_scaling=rope_scaling,
+            attention_bias=attention_bias,
+            attention_dropout=attention_dropout,
+            ffn_hidden_size=ffn_hidden_size,
+            q_lora_rank=q_lora_rank,
+            kv_lora_rank=kv_lora_rank,
+            qk_nope_head_dim=qk_nope_head_dim,
+            qk_rope_head_dim=qk_rope_head_dim,
+            head_dim=head_dim,
+            v_head_dim=v_head_dim,
+            qk_head_dim=qk_head_dim,
+            moe_topk=moe_topk,
+            n_routed_experts=n_routed_experts,
+            zero_expert_num=zero_expert_num,
+            expert_ffn_hidden_size=expert_ffn_hidden_size,
+            routed_scaling_factor=routed_scaling_factor,
+            emb_neighbor_num=emb_neighbor_num,
+            emb_split_num=emb_split_num,
+            ngram_vocab_size_ratio=ngram_vocab_size_ratio,
+            oe_ignored_token_ids=oe_ignored_token_ids,
+            **kwargs,
+        )
+
+class LongcatNextVisualConfig(Qwen2_5_VLVisionConfig):
+    model_type = "longcat_next_visual"
+    base_config_key = ""
+
+    def __init__(
+        self,
+        image_start_token_id=131106,
+        image_end_token_id=131107,
+        image_pad_token_id=131108,
+        image_newline_token_id=131109,
+        vq_config={},
+        visual_decoder_config={},
+        **kwargs,
+    ):
+        self.image_start_token_id = image_start_token_id
+        self.image_end_token_id = image_end_token_id
+        self.image_pad_token_id = image_pad_token_id
+        self.image_newline_token_id = image_newline_token_id
+        self.vq_config = PretrainedConfig(**vq_config)
+        self.visual_decoder_config = PretrainedConfig(**visual_decoder_config)
+        self.visual_decoder_config.image_decoder_config = PretrainedConfig(**getattr(self.visual_decoder_config, "image_decoder_config", {}))
+        self.visual_decoder_config.transformer_config = PretrainedConfig(**getattr(self.visual_decoder_config, "transformer_config", {}))
+        self.visual_decoder_config.vae_config = PretrainedConfig(**getattr(self.visual_decoder_config, "vae_config", {}))
+        self.visual_decoder_config.scheduler_config = PretrainedConfig(**getattr(self.visual_decoder_config, "scheduler_config", {}))
+        super().__init__(**kwargs)
+
+class LongcatNextAudioConfig(WhisperConfig):
+    model_type = "longcat_next_audio"
+    base_config_key = ""
+
+    def __init__(
+        self,
+        vq_config={},
+        vocoder_config={},
+        flow_matching_config={},
+        cosy24kvocoder_config={},
+        **kwargs
+    ):
+        self.vq_config = PretrainedConfig(**vq_config)
+        self.vocoder_config = PretrainedConfig(**vocoder_config)
+        self.flow_matching_config = PretrainedConfig(**flow_matching_config)
+        self.flow_matching_config.cfm_params = PretrainedConfig(**getattr(self.flow_matching_config, "cfm_params", {}))
+        self.cosy24kvocoder_config = PretrainedConfig(**cosy24kvocoder_config)
+        super().__init__(**kwargs)
+
+
+__all__ = ["LongcatNextConfig", "LongcatNextVisualConfig", "LongcatNextAudioConfig"]

native/configuration_longcat_ngram.py

@@ -0,0 +1,218 @@
+from transformers.models.longcat_flash import LongcatFlashConfig
+
+
+class LongcatFlashNgramConfig(LongcatFlashConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`LongcatFlashNgramModel`]. It is used to instantiate
+    a LongCat Flash model with N-gram enhanced embeddings according to the specified arguments, defining the model architecture. 
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 131072):
+            Vocabulary size of the LongCat Flash model. Defines the number of different tokens that can be represented by the
+            `input_ids` passed when calling [`LongcatFlashNgramModel`]
+        hidden_size (`int`, *optional*, defaults to 6144):
+            Dimension of the hidden representations.
+        num_hidden_layers (`int`, *optional*, defaults to 56):
+            Number of hidden layers in the Transformer decoder.
+        num_layers (`int`, *optional*, defaults to 28):
+            Number of layers, each with 2 sublayers.
+        num_attention_heads (`int`, *optional*, defaults to 64):
+            Number of attention heads for each attention layer in the Transformer decoder.
+        num_key_value_heads (`int`, *optional*):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting from a multi-head checkpoint to a GQA checkpoint, each group key and value head should be
+            constructed by meanpooling all the original heads within that group. For more details checkout [this
+            paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
+            `num_attention_heads`.
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 131072):
+            The maximum sequence length that this model might ever be used with. Typically set this to something large
+            just in case (e.g., 512 or 1024 or 2048).
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-05):
+            The epsilon value used by the RMS normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        pad_token_id (`int`, *optional*):
+            Padding token id.
+        bos_token_id (`int`, *optional*, defaults to 1):
+            Beginning of stream token id.
+        eos_token_id (`int`, *optional*, defaults to 2):
+            End of stream token id.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether to tie input and output embeddings.
+        rope_theta (`float`, *optional*, defaults to 10000000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. Currently supports two scaling
+            strategies: linear and dynamic. Their scaling factor must be a float greater than 1. The expected format is
+            `{"type": strategy name, "factor": scaling factor}`.
+        attention_bias (`bool`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+        ffn_hidden_size (`int`, *optional*, defaults to 12288):
+            Dimension of the MLP representations.
+        q_lora_rank (`int`, *optional*, defaults to 1536):
+            The rank of the query LoRA projection in MLA (Multi-head Latent Attention).
+        kv_lora_rank (`int`, *optional*, defaults to 512):
+            The rank of the key-value LoRA projection in MLA.
+        qk_nope_head_dim (`int`, *optional*, defaults to 128):
+            The dimension of the non-position encoding part of query/key heads.
+        qk_rope_head_dim (`int`, *optional*, defaults to 64):
+            The dimension of the RoPE part of query/key heads.
+        head_dim (`int`, *optional*, defaults to 64):
+            Standard dimension of qk heads, unused except for CI.
+        v_head_dim (`int`, *optional*, defaults to 128):
+            The dimension of value heads.
+        qk_head_dim (`int`, *optional*):
+            The total dimension of query/key heads. If not specified, set to `qk_nope_head_dim + qk_rope_head_dim`.
+        moe_topk (`int`, *optional*, defaults to 12):
+            Number of experts to route to for each token in the MoE layer.
+        n_routed_experts (`int`, *optional*, defaults to 512):
+            Number of routed experts in the MoE layer.
+        zero_expert_num (`int`, *optional*, defaults to 256):
+            Number of zero experts (identity function) to add to the expert pool.
+        expert_ffn_hidden_size (`int`, *optional*, defaults to 2048):
+            Hidden size of individual expert FFN layers.
+        routed_scaling_factor (`float`, *optional*, defaults to 6.0):
+            Scaling factor applied to the routing weights.
+        emb_neighbor_num (`int`, *optional*):
+            Maximum N-gram length for N-gram embeddings. This parameter determines the context window size for N-gram computation. Higher values capture 
+            longer-range lexical patterns but increase memory usage.
+        emb_split_num (`int`, *optional*):
+            Number of hash functions (or splits) to use for N-gram embeddings. Multiple hash functions help improve the quality of N-gram representations. 
+        ngram_vocab_size_ratio (`float`, *optional*):
+            Ratio multiplier for N-gram vocabulary size relative to the base vocabulary size. The N-gram vocabulary 
+            size is calculated as `vocab_size * ngram_vocab_size_ratio`. 
+
+    Example:
+    ```python
+    >>> from transformers import LongcatFlashNgramModel, LongcatFlashNgramConfig
+
+    >>> # Initializing a LongCat Flash N-gram style configuration
+    >>> configuration = LongcatFlashNgramConfig(
+    ...     emb_neighbor_num=3,
+    ...     emb_split_num=4,
+    ...     ngram_vocab_size_ratio=1.5
+    ... )
+
+    >>> # Initializing a model from the configuration
+    >>> model = LongcatFlashNgramModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "longcat_flash_ngram"
+    keys_to_ignore_at_inference = ["past_key_values"]
+    base_model_tp_plan = {
+        "layers.*.self_attn.*.q_b_proj": "colwise",
+        "layers.*.self_attn.*.kv_b_proj": "colwise",
+        "layers.*.self_attn.*.o_proj": "rowwise",
+        "layers.*.mlps.*.gate_proj": "colwise",
+        "layers.*.mlps.*.up_proj": "colwise",
+        "layers.*.mlps.*.down_proj": "rowwise",
+        "layers.*.mlp.experts.*.gate_proj": "colwise",
+        "layers.*.mlp.experts.*.up_proj": "colwise",
+        "layers.*.mlp.experts.*.down_proj": "rowwise",
+    }
+
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+
+    def __init__(
+        self,
+        vocab_size=131072,
+        hidden_size=6144,
+        num_hidden_layers=56,
+        num_layers=28,
+        num_attention_heads=64,
+        num_key_value_heads=None,
+        hidden_act="silu",
+        max_position_embeddings=131072,
+        initializer_range=0.02,
+        rms_norm_eps=1e-5,
+        use_cache=True,
+        pad_token_id=None,
+        bos_token_id=1,
+        eos_token_id=2,
+        tie_word_embeddings=False,
+        rope_theta=10000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        ffn_hidden_size=12288,
+        q_lora_rank=1536,
+        kv_lora_rank=512,
+        qk_nope_head_dim=128,
+        qk_rope_head_dim=64,
+        head_dim=64,
+        v_head_dim=128,
+        qk_head_dim=None,
+        moe_topk=12,
+        n_routed_experts=512,
+        zero_expert_num=256,
+        expert_ffn_hidden_size=2048,
+        routed_scaling_factor=6.0,
+        emb_neighbor_num=None,
+        emb_split_num=None,
+        ngram_vocab_size_ratio=None,
+        oe_ignored_token_ids=[],
+        **kwargs,
+    ):
+        # N-gram embedding specific parameters
+        self.emb_neighbor_num = emb_neighbor_num
+        self.emb_split_num = emb_split_num
+        self.ngram_vocab_size_ratio = ngram_vocab_size_ratio
+        self.oe_ignored_token_ids = oe_ignored_token_ids
+
+        super().__init__(
+            vocab_size=vocab_size,
+            hidden_size=hidden_size,
+            num_hidden_layers=num_hidden_layers,
+            num_layers=num_layers,
+            num_attention_heads=num_attention_heads,
+            num_key_value_heads=num_key_value_heads,
+            hidden_act=hidden_act,
+            max_position_embeddings=max_position_embeddings,
+            initializer_range=initializer_range,
+            rms_norm_eps=rms_norm_eps,
+            use_cache=use_cache,
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            rope_theta=rope_theta,
+            rope_scaling=rope_scaling,
+            attention_bias=attention_bias,
+            attention_dropout=attention_dropout,
+            ffn_hidden_size=ffn_hidden_size,
+            q_lora_rank=q_lora_rank,
+            kv_lora_rank=kv_lora_rank,
+            qk_nope_head_dim=qk_nope_head_dim,
+            qk_rope_head_dim=qk_rope_head_dim,
+            head_dim=head_dim,
+            v_head_dim=v_head_dim,
+            qk_head_dim=qk_head_dim,
+            moe_topk=moe_topk,
+            n_routed_experts=n_routed_experts,
+            zero_expert_num=zero_expert_num,
+            expert_ffn_hidden_size=expert_ffn_hidden_size,
+            routed_scaling_factor=routed_scaling_factor,
+            **kwargs,
+        )
+
+
+__all__ = ["LongcatFlashNgramConfig"]

native/image_refiner.py

@@ -0,0 +1,748 @@
+"""Image refiner: refiner pipeline, refiner container, and utilities.
+
+Contains:
+- RefinerImageProcessor: Image pre/post-processing for the diffusion pipeline
+- RefinerPipeline: DiffusionPipeline for image refinement
+- ImageRefinerContainer: nn.Module container for refiner sub-modules
+- IdentityWithArgs: Placeholder module for cond_proj
+- de_transform / tensor2pil: Tensor-to-PIL conversion utilities
+"""
+
+import inspect
+import math
+import warnings
+from dataclasses import dataclass
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+import numpy as np
+from safetensors.torch import load_file
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from PIL import Image
+
+from diffusers import DiffusionPipeline
+from diffusers.configuration_utils import register_to_config
+from diffusers.image_processor import PipelineImageInput, VaeImageProcessor, is_valid_image_imagelist
+from diffusers.models.autoencoders.autoencoder_kl import AutoencoderKL
+from .refiner_modules import FlowMatchEulerDiscreteScheduler
+
+from .refiner_modules import Transformer2DModel, RotaryPosEmbed
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _clean_config_dict(cfg, cls=None) -> dict:
+    """Convert a PretrainedConfig to a clean dict for model construction.
+
+    If ``cls`` is provided, only keeps keys that match the cls.__init__ params
+    (allowlist approach). Otherwise falls back to blocklist filtering.
+    """
+    if hasattr(cfg, "to_dict"):
+        d = cfg.to_dict()
+    elif isinstance(cfg, dict):
+        d = dict(cfg)
+    else:
+        d = {k: v for k, v in vars(cfg).items()}
+
+    if cls is not None:
+        import inspect
+        sig = inspect.signature(cls.__init__)
+        valid_keys = set(sig.parameters.keys()) - {"self"}
+        if any(p.kind == inspect.Parameter.VAR_KEYWORD for p in sig.parameters.values()):
+            # Has **kwargs — can't filter by allowlist, fall through to blocklist
+            pass
+        else:
+            return {k: v for k, v in d.items() if k in valid_keys}
+
+    # Blocklist: remove HuggingFace PretrainedConfig metadata
+    _PRETRAINED_CONFIG_KEYS = {
+        "_name_or_path", "transformers_version", "model_type", "_commit_hash",
+        "_attn_implementation", "_attn_implementation_autoset", "return_dict",
+        "output_hidden_states", "output_attentions", "use_bfloat16",
+        "torchscript", "torch_dtype", "is_encoder_decoder", "is_decoder",
+        "add_cross_attention", "tie_encoder_decoder", "tie_word_embeddings",
+        "cross_attention_hidden_size", "chunk_size_feed_forward", "decoder_start_token_id",
+        "architectures", "finetuning_task", "id2label", "label2id", "prefix",
+        "problem_type", "tokenizer_class", "task_specific_params", "pruned_heads",
+        "bos_token_id", "eos_token_id", "pad_token_id", "sep_token_id",
+        "max_length", "min_length", "do_sample", "early_stopping",
+        "num_beams", "num_beam_groups", "diversity_penalty", "temperature",
+        "top_k", "top_p", "typical_p", "repetition_penalty", "length_penalty",
+        "no_repeat_ngram_size", "encoder_no_repeat_ngram_size", "bad_words_ids",
+        "num_return_sequences", "output_scores", "return_dict_in_generate",
+        "forced_bos_token_id", "forced_eos_token_id", "remove_invalid_values",
+        "exponential_decay_length_penalty", "suppress_tokens", "begin_suppress_tokens",
+        "tf_legacy_loss", "dtype",
+    }
+    return {k: v for k, v in d.items() if not k.startswith("_") and k not in _PRETRAINED_CONFIG_KEYS}
+
+
+# ---------------------------------------------------------------------------
+# Image Refiner Container (nn.Module for state_dict loading)
+# ---------------------------------------------------------------------------
+
+
+class ImageRefinerContainer(nn.Module):
+    """Container for refiner components.
+
+    Holds base_transformer, vae, cond_proj as nn.Module children so their
+    parameters appear in the parent model's state_dict and are loaded
+    automatically via from_pretrained.
+    """
+
+    def __init__(self, visual_decoder_config):
+        super().__init__()
+
+        tc = visual_decoder_config.transformer_config
+        vc = visual_decoder_config.vae_config
+
+        self.base_transformer = Transformer2DModel(**_clean_config_dict(tc))
+
+        self.vae = AutoencoderKL(**_clean_config_dict(vc))
+        self.vae.requires_grad_(False)
+
+        text_feat_dim = getattr(tc, "text_feat_dim", 3584)
+        codebook_dim = getattr(visual_decoder_config, "codebook_dim", text_feat_dim)
+        if codebook_dim != text_feat_dim:
+            self.cond_proj = nn.Linear(codebook_dim, text_feat_dim)
+        else:
+            self.cond_proj = IdentityWithArgs()
+
+    @classmethod
+    def from_pretrained(cls, config, model_path: str):
+        model = cls(config)
+        weight_dict = load_file(model_path, device="cpu")
+        model.load_state_dict({k.removeprefix("image_refiner."): v for k, v in weight_dict.items() if k.startswith("image_refiner.")}, strict=True)
+        model.eval()
+        return model
+
+    @property
+    def device(self):
+        return next(self.parameters()).device
+
+    @property
+    def dtype(self):
+        return next(self.parameters()).dtype
+
+
+class RefinerImageProcessor(VaeImageProcessor):
+    """Image processor for refiner - extends diffusers' VaeImageProcessor."""
+
+    @register_to_config
+    def __init__(
+        self,
+        do_resize: bool = True,
+        vae_scale_factor: int = 16,
+        resample: str = "lanczos",
+        max_pixels: Optional[int] = None,
+        max_side_length: Optional[int] = None,
+        do_normalize: bool = True,
+        do_binarize: bool = False,
+        do_convert_grayscale: bool = False,
+    ):
+        super().__init__(
+            do_resize=do_resize,
+            vae_scale_factor=vae_scale_factor,
+            resample=resample,
+            do_normalize=do_normalize,
+            do_binarize=do_binarize,
+            do_convert_grayscale=do_convert_grayscale,
+        )
+        self.max_pixels = max_pixels
+        self.max_side_length = max_side_length
+
+    def get_new_height_width(
+        self,
+        image: Union["PIL.Image.Image", np.ndarray, torch.Tensor],
+        height: Optional[int] = None,
+        width: Optional[int] = None,
+        max_pixels: Optional[int] = None,
+        max_side_length: Optional[int] = None,
+    ) -> Tuple[int, int]:
+        import PIL.Image
+
+        if height is None:
+            if isinstance(image, PIL.Image.Image):
+                height = image.height
+            elif isinstance(image, torch.Tensor):
+                height = image.shape[2]
+            else:
+                height = image.shape[1]
+
+        if width is None:
+            if isinstance(image, PIL.Image.Image):
+                width = image.width
+            elif isinstance(image, torch.Tensor):
+                width = image.shape[3]
+            else:
+                width = image.shape[2]
+
+        if max_side_length is None:
+            max_side_length = self.max_side_length
+        if max_pixels is None:
+            max_pixels = self.max_pixels
+
+        ratio = 1.0
+        if max_side_length is not None:
+            max_side_length_ratio = max_side_length / max(height, width)
+        else:
+            max_side_length_ratio = 1.0
+
+        cur_pixels = height * width
+        max_pixels_ratio = (max_pixels / cur_pixels) ** 0.5 if max_pixels is not None else 1.0
+        ratio = min(max_pixels_ratio, max_side_length_ratio, 1.0)
+
+        sf = self.config.vae_scale_factor
+        new_height = int(height * ratio) // sf * sf
+        new_width = int(width * ratio) // sf * sf
+        return new_height, new_width
+
+    def preprocess(
+        self,
+        image: PipelineImageInput,
+        height: Optional[int] = None,
+        width: Optional[int] = None,
+        max_pixels: Optional[int] = None,
+        max_side_length: Optional[int] = None,
+        resize_mode: str = "default",
+        crops_coords: Optional[Tuple[int, int, int, int]] = None,
+    ) -> torch.Tensor:
+        import PIL.Image
+
+        supported_formats = (PIL.Image.Image, np.ndarray, torch.Tensor)
+
+        if self.config.do_convert_grayscale and isinstance(image, (torch.Tensor, np.ndarray)) and image.ndim == 3:
+            if isinstance(image, torch.Tensor):
+                image = image.unsqueeze(1)
+            else:
+                if image.shape[-1] == 1:
+                    image = np.expand_dims(image, axis=0)
+                else:
+                    image = np.expand_dims(image, axis=-1)
+
+        if isinstance(image, list) and isinstance(image[0], np.ndarray) and image[0].ndim == 4:
+            warnings.warn(
+                "Passing `image` as a list of 4d np.ndarray is deprecated. "
+                "Please concatenate the list along the batch dimension and pass it as a single 4d np.ndarray",
+                FutureWarning,
+            )
+            image = np.concatenate(image, axis=0)
+        if isinstance(image, list) and isinstance(image[0], torch.Tensor) and image[0].ndim == 4:
+            warnings.warn(
+                "Passing `image` as a list of 4d torch.Tensor is deprecated. "
+                "Please concatenate the list along the batch dimension and pass it as a single 4d torch.Tensor",
+                FutureWarning,
+            )
+            image = torch.cat(image, axis=0)
+
+        if not is_valid_image_imagelist(image):
+            raise ValueError(
+                f"Input is in incorrect format. Currently, we only support "
+                f"{', '.join(str(x) for x in supported_formats)}"
+            )
+        if not isinstance(image, list):
+            image = [image]
+
+        if isinstance(image[0], PIL.Image.Image):
+            if crops_coords is not None:
+                image = [i.crop(crops_coords) for i in image]
+            if self.config.do_resize:
+                height, width = self.get_new_height_width(image[0], height, width, max_pixels, max_side_length)
+                image = [self.resize(i, height, width, resize_mode=resize_mode) for i in image]
+            if self.config.do_convert_grayscale:
+                image = [self.convert_to_grayscale(i) for i in image]
+            image = self.pil_to_numpy(image)
+            image = self.numpy_to_pt(image)
+        elif isinstance(image[0], np.ndarray):
+            image = np.concatenate(image, axis=0) if image[0].ndim == 4 else np.stack(image, axis=0)
+            image = self.numpy_to_pt(image)
+            height, width = self.get_new_height_width(image, height, width, max_pixels, max_side_length)
+            if self.config.do_resize:
+                image = self.resize(image, height, width)
+        elif isinstance(image[0], torch.Tensor):
+            image = torch.cat(image, axis=0) if image[0].ndim == 4 else torch.stack(image, axis=0)
+            if self.config.do_convert_grayscale and image.ndim == 3:
+                image = image.unsqueeze(1)
+            channel = image.shape[1]
+            if channel == self.config.vae_latent_channels:
+                return image
+            height, width = self.get_new_height_width(image, height, width, max_pixels, max_side_length)
+            if self.config.do_resize:
+                image = self.resize(image, height, width)
+
+        do_normalize = self.config.do_normalize
+        if do_normalize and image.min() < 0:
+            warnings.warn(
+                "Passing `image` as torch tensor with value range in [-1,1] is deprecated. "
+                f"The expected value range for image tensor is [0,1] when passing as pytorch tensor or numpy Array. "
+                f"You passed `image` with value range [{image.min()},{image.max()}]",
+                FutureWarning,
+            )
+            do_normalize = False
+        if do_normalize:
+            image = self.normalize(image)
+
+        if self.config.do_binarize:
+            image = self.binarize(image)
+
+        return image
+
+
+@dataclass
+class RefinerOutput:
+    images: Union[List[Image.Image], torch.Tensor]
+
+
+class IdentityWithArgs(nn.Module):
+    """Placeholder Identity module for cond_proj."""
+
+    def __init__(self, dtype=torch.float32, device=None):
+        super().__init__()
+        self.register_buffer("_dummy", torch.zeros((), dtype=dtype, device=device))
+
+    @property
+    def dtype(self):
+        return self._dummy.dtype
+
+    @property
+    def device(self):
+        return self._dummy.device
+
+    def forward(self, x, *args, **kwargs):
+        return x
+
+
+def _retrieve_timesteps(
+    scheduler: FlowMatchEulerDiscreteScheduler,
+    num_inference_steps: Optional[int] = None,
+    device: Optional[Union[str, torch.device]] = None,
+    timesteps: Optional[List[int]] = None,
+    **kwargs,
+):
+    # If scheduler uses dynamic shifting and caller passed num_tokens, compute mu
+    # (same as training code refiner pipeline)
+    num_tokens = kwargs.pop("num_tokens", None)
+    if num_tokens is not None and getattr(scheduler.config, "use_dynamic_shifting", False):
+        # Compute mu from num_tokens using scheduler's linear interpolation
+        base_shift = getattr(scheduler.config, "base_shift", 0.5)
+        max_shift = getattr(scheduler.config, "max_shift", 1.15)
+        base_seq_len = getattr(scheduler.config, "base_image_seq_len", 256)
+        max_seq_len = getattr(scheduler.config, "max_image_seq_len", 4096)
+        m = (max_shift - base_shift) / (max_seq_len - base_seq_len)
+        b = base_shift - m * base_seq_len
+        mu = num_tokens * m + b
+        kwargs["mu"] = mu
+
+    accepted = set(inspect.signature(scheduler.set_timesteps).parameters.keys())
+    filtered_kwargs = {k: v for k, v in kwargs.items() if k in accepted}
+
+    if timesteps is not None:
+        if "timesteps" not in accepted:
+            raise ValueError(
+                f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"
+                f" timestep schedules. Please check whether you are using the correct scheduler."
+            )
+        scheduler.set_timesteps(timesteps=timesteps, device=device, **filtered_kwargs)
+        timesteps = scheduler.timesteps
+        num_inference_steps = len(timesteps)
+    else:
+        scheduler.set_timesteps(num_inference_steps, device=device, **filtered_kwargs)
+        timesteps = scheduler.timesteps
+    return timesteps, num_inference_steps
+
+
+class RefinerPipeline(DiffusionPipeline):
+    """
+    Image refiner evaluation pipeline.
+
+    - cond comes from upstream model: encoder_hidden_states (quants / last_latent)
+    - grid_thw_list is used to split cond (consistent with training)
+    - image as ref image
+    - Supports FlowMatchEulerDiscreteScheduler + velocity model
+    """
+
+    def __init__(
+        self,
+        vae: AutoencoderKL,
+        transformer: Transformer2DModel,
+        scheduler: FlowMatchEulerDiscreteScheduler,
+        cond_proj: Optional[nn.Module] = None,
+    ):
+        super().__init__()
+
+        self.register_modules(
+            vae=vae,
+            transformer=transformer,
+            scheduler=scheduler,
+            cond_proj=cond_proj if cond_proj is not None else IdentityWithArgs(),
+        )
+
+        self.vae_scale_factor = (
+            2 ** (len(self.vae.config.block_out_channels) - 1)
+            if hasattr(self.vae.config, "block_out_channels")
+            else 8
+        )
+        self.image_processor = RefinerImageProcessor(
+            vae_scale_factor=self.vae_scale_factor * 2, do_resize=True
+        )
+        self.patch_size = int(getattr(self.transformer.config, "patch_size", 16))
+
+        self._num_timesteps: int = 0
+        self._current_timestep: Optional[torch.Tensor] = None
+        self._interrupt: bool = False
+        self._freqs_cis: Optional[torch.Tensor] = None
+        self._text_guidance_scale: float = 1.0
+        self._image_guidance_scale: float = 1.0
+        self._cfg_range: Tuple[float, float] = (0.0, 1.0)
+
+    @torch.no_grad()
+    def _get_freqs_cis(self, device, dtype):
+        if self._freqs_cis is None:
+            self._freqs_cis = RotaryPosEmbed.get_freqs_cis(
+                self.transformer.config.axes_dim_rope,
+                self.transformer.config.axes_lens,
+                theta=10000,
+            )
+        return self._freqs_cis
+
+    @staticmethod
+    def _split_tokens(
+        encoder_hidden_states: torch.Tensor,
+        grid_thw_list: List[Tuple[int, int, int]],
+    ) -> List[torch.Tensor]:
+        splits = [int(h) * int(w) // 4 for (_, h, w) in grid_thw_list]
+        return list(torch.split(encoder_hidden_states, splits, dim=1))
+
+    @staticmethod
+    def _looks_like_latents(x: Union[torch.Tensor, Image.Image], latent_ch_hint: int = 16) -> bool:
+        if not isinstance(x, torch.Tensor):
+            return False
+        if x.ndim not in (3, 4):
+            return False
+        c = int(x.shape[-3])
+        if c == 3:
+            return False
+        if c == latent_ch_hint:
+            return True
+        if c > 3 and c <= 32:
+            return True
+        return False
+
+    @torch.no_grad()
+    def _preprocess_to_vae_range(self, img: torch.Tensor) -> torch.Tensor:
+        if img.dtype not in (torch.float32, torch.float16, torch.bfloat16):
+            img = img.float()
+        if img.max() > 1.5:
+            img = img / 255.0
+        if img.min() >= 0.0 and img.max() <= 1.0:
+            img = img * 2.0 - 1.0
+        return img.clamp(-1, 1)
+
+    @torch.no_grad()
+    def _encode_image_to_latents(
+        self,
+        img_any: Union[Image.Image, torch.Tensor],
+        device,
+        dtype,
+    ) -> Tuple[torch.Tensor, int, int]:
+        latent_ch_hint = int(getattr(getattr(self.vae, "config", None), "latent_channels", 16))
+
+        if self._looks_like_latents(img_any, latent_ch_hint=latent_ch_hint):
+            z = img_any
+            if z.ndim == 3:
+                z = z.unsqueeze(0)
+            z = z.to(device=device, dtype=dtype)
+            H_lat, W_lat = z.shape[-2], z.shape[-1]
+            return z, H_lat, W_lat
+
+        if isinstance(img_any, Image.Image):
+            img = torch.from_numpy(
+                np.array(img_any).astype("float32") / 255.0
+            ).permute(2, 0, 1).unsqueeze(0)
+        elif isinstance(img_any, torch.Tensor):
+            img = img_any
+            if img.ndim == 3:
+                img = img.unsqueeze(0)
+        else:
+            raise TypeError("Unsupported image type. Use PIL.Image or torch.Tensor or latent Tensor.")
+
+        img = self._preprocess_to_vae_range(img)
+
+        H, W = img.shape[-2:]
+        base = self.patch_size * self.vae_scale_factor
+        target_H = max(base, math.ceil(H / base) * base)
+        target_W = max(base, math.ceil(W / base) * base)
+        if (H != target_H) or (W != target_W):
+            img = F.interpolate(img, size=(target_H, target_W), mode="bilinear", align_corners=False)
+
+        img = img.to(device=device, dtype=self.vae.dtype)
+
+        posterior = self.vae.encode(img).latent_dist
+        z0 = posterior.sample()
+        if getattr(self.vae.config, "shift_factor", None) is not None:
+            z0 = z0 - self.vae.config.shift_factor
+        if getattr(self.vae.config, "scaling_factor", None) is not None:
+            z0 = z0 * self.vae.config.scaling_factor
+
+        z0 = z0.to(device=device, dtype=dtype)
+        H_lat, W_lat = z0.shape[-2], z0.shape[-1]
+        return z0, H_lat, W_lat
+
+    @staticmethod
+    def _expand_to_list(x, n):
+        if x is None:
+            return [None] * n
+        if isinstance(x, (Image.Image, torch.Tensor)):
+            return [x] * n
+        assert isinstance(x, list), "`image` must be PIL / Tensor or list of them."
+        assert len(x) == n, "`len(image)` must equal number of image chunks"
+        return x
+
+    @torch.no_grad()
+    def _denoise_once(
+        self,
+        cond_tokens: torch.Tensor,
+        ref_img: Optional[Union[Image.Image, torch.Tensor]],
+        num_inference_steps: int = 28,
+        timesteps: Optional[List[int]] = None,
+        generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,
+        output_type: str = "pil",
+        text_guidance_scale: float = 1.0,
+        image_guidance_scale: float = 1.0,
+        cfg_range: Tuple[float, float] = (0.0, 1.0),
+        enable_processor_bar: bool = True,
+    ):
+        device = cond_tokens.device
+        weight_dtype = self.transformer.dtype
+
+        self._text_guidance_scale = text_guidance_scale
+        self._image_guidance_scale = image_guidance_scale
+        self._cfg_range = cfg_range
+
+        cond_tokens = cond_tokens.to(device=device, dtype=weight_dtype)
+        text_feats = self.cond_proj(cond_tokens)
+        B, L, _ = text_feats.shape
+        text_mask = torch.ones(B, L, device=device, dtype=torch.bool)
+
+        ref_image_hidden_states = None
+        H_lat: int
+        W_lat: int
+
+        if ref_img is not None:
+            if isinstance(ref_img, torch.Tensor) and ref_img.ndim == 4 and ref_img.shape[0] == B:
+                z_ref, H_lat, W_lat = self._encode_image_to_latents(ref_img, device=device, dtype=weight_dtype)
+                ref_image_hidden_states = [[z_ref[b]] for b in range(B)]
+            else:
+                z_ref, H_lat, W_lat = self._encode_image_to_latents(ref_img, device=device, dtype=weight_dtype)
+                z_single = z_ref[0]
+                ref_image_hidden_states = [[z_single] for _ in range(B)]
+        else:
+            H_lat = W_lat = 128 // self.vae_scale_factor
+
+        C_lat = getattr(self.transformer.config, "in_channels", None)
+        if C_lat is None:
+            if ref_image_hidden_states is not None:
+                C_lat = ref_image_hidden_states[0][0].shape[0]
+            else:
+                raise ValueError("transformer.config.in_channels is None and no ref_img was provided.")
+        latents_shape = (B, C_lat, H_lat, W_lat)
+
+        if isinstance(generator, list):
+            if len(generator) != B:
+                raise ValueError(
+                    f"len(generator)={len(generator)} must equal B={B} when passing list of generators."
+                )
+            latents = torch.stack(
+                [
+                    torch.randn(
+                        (1, C_lat, H_lat, W_lat),
+                        generator=generator[i],
+                        device=device,
+                        dtype=weight_dtype,
+                    ).squeeze(0)
+                    for i in range(B)
+                ],
+                dim=0,
+            )
+        else:
+            latents = torch.randn(latents_shape, generator=generator, device=device, dtype=weight_dtype)
+
+        num_tokens = H_lat * W_lat
+        timesteps_sched, num_inference_steps = _retrieve_timesteps(
+            self.scheduler,
+            num_inference_steps=num_inference_steps,
+            device=device,
+            timesteps=timesteps,
+            num_tokens=num_tokens,
+        )
+        num_warmup_steps = max(len(timesteps_sched) - num_inference_steps * self.scheduler.order, 0)
+        self._num_timesteps = len(timesteps_sched)
+
+        freqs_cis = self._get_freqs_cis(device=device, dtype=weight_dtype)
+
+        progress_bar = self.progress_bar(total=num_inference_steps) if enable_processor_bar else None
+        for i, t in enumerate(timesteps_sched):
+            if self._interrupt:
+                continue
+            self._current_timestep = t
+
+            timestep = t.expand(latents.shape[0]).to(latents.dtype)
+
+            step_frac = i / max(len(timesteps_sched) - 1, 1)
+            use_cfg = (cfg_range[0] <= step_frac <= cfg_range[1]) and (
+                text_guidance_scale > 1.0 or image_guidance_scale > 1.0
+            )
+
+            if not use_cfg:
+                optional_kwargs: Dict[str, Any] = {}
+                if "ref_image_hidden_states" in inspect.signature(self.transformer.forward).parameters:
+                    optional_kwargs["ref_image_hidden_states"] = ref_image_hidden_states
+                model_pred = self.transformer(
+                    latents, timestep, text_feats, freqs_cis, text_mask, **optional_kwargs
+                )
+            else:
+                text_uncond = torch.zeros_like(text_feats)
+
+                opt_kwargs_text: Dict[str, Any] = {}
+                if "ref_image_hidden_states" in inspect.signature(self.transformer.forward).parameters:
+                    opt_kwargs_text["ref_image_hidden_states"] = ref_image_hidden_states
+
+                model_pred_text = self.transformer(
+                    latents, timestep, text_feats, freqs_cis, text_mask, **opt_kwargs_text
+                )
+
+                opt_kwargs_ref: Dict[str, Any] = {}
+                if "ref_image_hidden_states" in inspect.signature(self.transformer.forward).parameters:
+                    opt_kwargs_ref["ref_image_hidden_states"] = ref_image_hidden_states
+
+                model_pred_ref = self.transformer(
+                    latents, timestep, text_uncond, freqs_cis, text_mask, **opt_kwargs_ref
+                )
+
+                opt_kwargs_uncond: Dict[str, Any] = {}
+                if "ref_image_hidden_states" in inspect.signature(self.transformer.forward).parameters:
+                    opt_kwargs_uncond["ref_image_hidden_states"] = None
+
+                model_pred_uncond = self.transformer(
+                    latents, timestep, text_uncond, freqs_cis, text_mask, **opt_kwargs_uncond
+                )
+
+                if text_guidance_scale > 1.0 and image_guidance_scale > 1.0:
+                    model_pred = (
+                        model_pred_uncond
+                        + image_guidance_scale * (model_pred_ref - model_pred_uncond)
+                        + text_guidance_scale * (model_pred_text - model_pred_ref)
+                    )
+                elif text_guidance_scale > 1.0:
+                    model_pred = model_pred_uncond + text_guidance_scale * (model_pred_text - model_pred_uncond)
+                elif image_guidance_scale > 1.0:
+                    model_pred = model_pred_uncond + image_guidance_scale * (model_pred_ref - model_pred_uncond)
+                else:
+                    model_pred = model_pred_text
+
+            latents = self.scheduler.step(model_pred, t, latents, return_dict=False)[0]
+            latents = latents.to(dtype=weight_dtype)
+
+            if progress_bar is not None:
+                if i == len(timesteps_sched) - 1 or (
+                    (i + 1) > num_warmup_steps and (i + 1) % self.scheduler.order == 0
+                ):
+                    progress_bar.update()
+
+        if progress_bar is not None:
+            progress_bar.close()
+
+        self._current_timestep = None
+
+        latents = latents.to(dtype=self.vae.dtype)
+        if getattr(self.vae.config, "scaling_factor", None) is not None:
+            latents = latents / self.vae.config.scaling_factor
+        if getattr(self.vae.config, "shift_factor", None) is not None:
+            latents = latents + self.vae.config.shift_factor
+        image = self.vae.decode(latents, return_dict=False)[0]
+
+        images = self.image_processor.postprocess(image, output_type=output_type)
+        return images
+
+    @torch.no_grad()
+    def __call__(
+        self,
+        *,
+        encoder_hidden_states: torch.Tensor,
+        grid_thw_list: List[Tuple[int, int, int]],
+        image: Union[Image.Image, torch.Tensor, List[Union[Image.Image, torch.Tensor]], None] = None,
+        num_inference_steps: int = 28,
+        timesteps: Optional[List[int]] = None,
+        generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,
+        output_type: str = "pil",
+        return_dict: bool = True,
+        text_guidance_scale: float = 1.5,
+        image_guidance_scale: float = 1.5,
+        cfg_range: Tuple[float, float] = (0.0, 1.0),
+        enable_processor_bar: bool = True,
+        **kwargs,
+    ) -> Union[RefinerOutput, List[Image.Image], torch.Tensor]:
+        self._interrupt = False
+
+        token_chunks = self._split_tokens(encoder_hidden_states, grid_thw_list)
+        ref_list = self._expand_to_list(image, len(token_chunks))
+
+        results_pil: List[Image.Image] = []
+        results_pt: Optional[torch.Tensor] = None
+
+        for tok, _, img_any in zip(token_chunks, grid_thw_list, ref_list):
+            imgs = self._denoise_once(
+                cond_tokens=tok,
+                ref_img=img_any,
+                num_inference_steps=num_inference_steps,
+                timesteps=timesteps,
+                generator=generator,
+                output_type=output_type,
+                text_guidance_scale=text_guidance_scale,
+                image_guidance_scale=image_guidance_scale,
+                cfg_range=cfg_range,
+                enable_processor_bar=enable_processor_bar,
+            )
+
+            if output_type == "pil":
+                results_pil += imgs
+            else:
+                results_pt = imgs if results_pt is None else torch.cat([results_pt, imgs], dim=0)
+
+        if not return_dict:
+            return results_pil if output_type == "pil" else results_pt
+        return RefinerOutput(images=results_pil if output_type == "pil" else results_pt)
+
+
+def de_transform(
+    tensor: torch.Tensor,
+    mean=(0.48145466, 0.4578275, 0.40821073),
+    std=(0.26862954, 0.26130258, 0.27577711),
+    rescale_factor: float = 1 / 255,
+) -> torch.Tensor:
+    """De-normalize and de-rescale, suitable for images processed by Qwen2VLImageProcessor."""
+    if tensor.ndim == 3:
+        tensor = tensor.unsqueeze(0)
+    mean_t = torch.tensor(mean).view(1, -1, 1, 1).to(tensor.device)
+    std_t = torch.tensor(std).view(1, -1, 1, 1).to(tensor.device)
+    tensor = tensor * std_t + mean_t
+    tensor = tensor / rescale_factor
+    tensor = torch.clamp(tensor / 255.0, 0, 1)
+    return tensor
+
+
+def tensor2pil(image_t: torch.Tensor, image_mean, image_std) -> Image.Image:
+    """Convert a tensor to a PIL Image."""
+    image_t = image_t.detach().cpu()
+    rescale_factor = 1 / 255
+    sample = de_transform(
+        image_t,
+        mean=image_mean,
+        std=image_std,
+        rescale_factor=rescale_factor,
+    )[0]
+    ndarr = sample.mul(255).add_(0.5).clamp_(0, 255).permute(1, 2, 0).to("cpu", torch.uint8).numpy()
+    return Image.fromarray(ndarr)

native/qwen2_5_vl_vit.py

@@ -0,0 +1,494 @@
+# Copyright (c) 2025 ByteDance Ltd. and/or its affiliates.
+# Copyright 2024 The Qwen Team and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# coding: utf-8
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# Copyright (c) 2025 ByteDance Ltd. and/or its affiliates.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from typing import Optional, Tuple
+from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import Qwen2_5_VLVisionConfig
+from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import Qwen2_5_VLPreTrainedModel
+from transformers.utils import is_flash_attn_2_available
+from transformers.activations import ACT2FN
+
+if is_flash_attn_2_available():
+    from flash_attn import flash_attn_varlen_func
+    from flash_attn.layers.rotary import apply_rotary_emb
+
+else:
+    flash_attn_varlen_func = None
+    apply_rotary_emb = None
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+def apply_rotary_pos_emb_vision(
+    q: torch.Tensor, k: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor
+) -> Tuple[torch.Tensor, torch.Tensor]:
+    orig_q_dtype = q.dtype
+    orig_k_dtype = k.dtype
+    q, k = q.float(), k.float()
+    cos, sin = cos.unsqueeze(-2), sin.unsqueeze(-2)
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    q_embed = q_embed.to(orig_q_dtype)
+    k_embed = k_embed.to(orig_k_dtype)
+    return q_embed, k_embed
+
+class Qwen2_5_VLVisionAttention(nn.Module):
+    def __init__(self, dim: int, num_heads: int = 16) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        self.head_dim = dim // num_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=True)
+        self.proj = nn.Linear(dim, dim)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        rotary_pos_emb: Optional[torch.Tensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        q, k, v = self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
+        if position_embeddings is None:
+            # logger.warning_once(
+            #     "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
+            #     "through `rotary_pos_emb` (2D tensor of RoPE theta values), to using externally computed "
+            #     "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.54 `rotary_pos_emb` will be "
+            #     "removed and `position_embeddings` will be mandatory."
+            # )
+            emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
+            cos = emb.cos().float()
+            sin = emb.sin().float()
+        else:
+            cos, sin = position_embeddings
+        q, k = apply_rotary_pos_emb_vision(q, k, cos, sin)
+
+        attention_mask = torch.full(
+            [1, seq_length, seq_length], torch.finfo(q.dtype).min, device=q.device, dtype=q.dtype
+        )
+        for i in range(1, len(cu_seqlens)):
+            attention_mask[..., cu_seqlens[i - 1] : cu_seqlens[i], cu_seqlens[i - 1] : cu_seqlens[i]] = 0
+
+        q = q.transpose(0, 1)
+        k = k.transpose(0, 1)
+        v = v.transpose(0, 1)
+        attn_weights = torch.matmul(q, k.transpose(1, 2)) / math.sqrt(self.head_dim)
+        attn_weights = attn_weights + attention_mask
+        attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(q.dtype)
+        attn_output = torch.matmul(attn_weights, v)
+        attn_output = attn_output.transpose(0, 1)
+        attn_output = attn_output.reshape(seq_length, -1)
+        attn_output = self.proj(attn_output)
+        return attn_output
+
+def apply_rotary_pos_emb_flashatt(
+    q: torch.Tensor, k: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor
+) -> Tuple[torch.Tensor, torch.Tensor]:
+    cos = cos.chunk(2, dim=-1)[0].contiguous()
+    sin = sin.chunk(2, dim=-1)[0].contiguous()
+    q_embed = apply_rotary_emb(q.float(), cos, sin).type_as(q)
+    k_embed = apply_rotary_emb(k.float(), cos, sin).type_as(k)
+    return q_embed, k_embed
+
+class Qwen2_5_VLVisionFlashAttention2(nn.Module):
+    def __init__(self, dim: int, num_heads: int = 16) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=True)
+        self.proj = nn.Linear(dim, dim)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        rotary_pos_emb: Optional[torch.Tensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        q, k, v = self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
+        if position_embeddings is None:
+            # logger.warning_once(
+            #     "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
+            #     "through `rotary_pos_emb` (2D tensor of RoPE theta values), to using externally computed "
+            #     "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.54 `rotary_pos_emb` will be "
+            #     "removed and `position_embeddings` will be mandatory."
+            # )
+            emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
+            cos = emb.cos().float()
+            sin = emb.sin().float()
+        else:
+            cos, sin = position_embeddings
+            cos, sin = cos.float(), sin.float()  # NOTE: BAGEL expects input and cos/sin to have the same dtype.
+        q, k = apply_rotary_pos_emb_flashatt(q.unsqueeze(0), k.unsqueeze(0), cos, sin)
+        q = q.squeeze(0)
+        k = k.squeeze(0)
+
+        max_seqlen = (cu_seqlens[1:] - cu_seqlens[:-1]).max().item()
+        attn_output = flash_attn_varlen_func(q, k, v, cu_seqlens, cu_seqlens, max_seqlen, max_seqlen).reshape(
+            seq_length, -1
+        )
+        attn_output = self.proj(attn_output)
+        return attn_output
+
+class Qwen2_5_VLVisionSdpaAttention(nn.Module):
+    def __init__(self, dim: int, num_heads: int = 16) -> None:
+        super().__init__()
+        self.num_heads = num_heads
+        self.qkv = nn.Linear(dim, dim * 3, bias=True)
+        self.proj = nn.Linear(dim, dim)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        rotary_pos_emb: Optional[torch.Tensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        q, k, v = self.qkv(hidden_states).reshape(seq_length, 3, self.num_heads, -1).permute(1, 0, 2, 3).unbind(0)
+        if position_embeddings is None:
+            # logger.warning_once(
+            #     "The attention layers in this model are transitioning from computing the RoPE embeddings internally "
+            #     "through `rotary_pos_emb` (2D tensor of RoPE theta values), to using externally computed "
+            #     "`position_embeddings` (Tuple of tensors, containing cos and sin). In v4.54 `rotary_pos_emb` will be "
+            #     "removed and `position_embeddings` will be mandatory."
+            # )
+            emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
+            cos = emb.cos().float()
+            sin = emb.sin().float()
+        else:
+            cos, sin = position_embeddings
+        q, k = apply_rotary_pos_emb_vision(q, k, cos, sin)
+
+        attention_mask = torch.zeros([1, seq_length, seq_length], device=q.device, dtype=torch.bool)
+        for i in range(1, len(cu_seqlens)):
+            attention_mask[..., cu_seqlens[i - 1] : cu_seqlens[i], cu_seqlens[i - 1] : cu_seqlens[i]] = True
+        q = q.transpose(0, 1)
+        k = k.transpose(0, 1)
+        v = v.transpose(0, 1)
+        attn_output = F.scaled_dot_product_attention(q, k, v, attention_mask, dropout_p=0.0)
+        attn_output = attn_output.transpose(0, 1)
+        attn_output = attn_output.reshape(seq_length, -1)
+        attn_output = self.proj(attn_output)
+        return attn_output
+
+QWEN2_5_VL_VISION_ATTENTION_CLASSES = {
+    "eager": Qwen2_5_VLVisionAttention,
+    "flash_attention_2": Qwen2_5_VLVisionFlashAttention2,
+    "sdpa": Qwen2_5_VLVisionSdpaAttention,
+}
+
+class Qwen2_5_VisionPatchEmbed(nn.Module):
+    def __init__(
+        self,
+        patch_size: int = 14,
+        temporal_patch_size: int = 2,
+        in_channels: int = 3,
+        embed_dim: int = 1152,
+    ) -> None:
+        super().__init__()
+        self.patch_size = patch_size
+        self.temporal_patch_size = temporal_patch_size
+        self.in_channels = in_channels
+        self.embed_dim = embed_dim
+
+        kernel_size = [temporal_patch_size, patch_size, patch_size]
+        self.proj = nn.Conv3d(in_channels, embed_dim, kernel_size=kernel_size, stride=kernel_size, bias=False)
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        target_dtype = self.proj.weight.dtype
+        hidden_states = hidden_states.view(
+            -1, self.in_channels, self.temporal_patch_size, self.patch_size, self.patch_size
+        )
+        hidden_states = self.proj(hidden_states.to(dtype=target_dtype)).view(-1, self.embed_dim)
+        return hidden_states
+
+class Qwen2_5_VisionRotaryEmbedding(nn.Module):
+    def __init__(self, dim: int, theta: float = 10000.0) -> None:
+        super().__init__()
+        inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float) / dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+    def forward(self, seqlen: int) -> torch.Tensor:
+        seq = torch.arange(seqlen, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
+        freqs = torch.outer(seq, self.inv_freq)
+        return freqs
+
+class Qwen2RMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        Qwen2RMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+class Qwen2_5_VLMLP(nn.Module):
+    def __init__(self, config, bias: bool = False):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = config.intermediate_size
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=bias)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=bias)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=bias)
+        self.act_fn = ACT2FN[config.hidden_act]
+
+    def forward(self, hidden_state):
+        return self.down_proj(self.act_fn(self.gate_proj(hidden_state)) * self.up_proj(hidden_state))
+
+class Qwen2_5_VLVisionBlock(nn.Module):
+    def __init__(self, config, attn_implementation: str = "sdpa") -> None:
+        super().__init__()
+        self.norm1 = Qwen2RMSNorm(config.hidden_size, eps=1e-6)
+        self.norm2 = Qwen2RMSNorm(config.hidden_size, eps=1e-6)
+        self.attn = QWEN2_5_VL_VISION_ATTENTION_CLASSES[attn_implementation](
+            config.hidden_size, num_heads=config.num_heads
+        )
+        self.mlp = Qwen2_5_VLMLP(config, bias=True)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        rotary_pos_emb: Optional[torch.Tensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,
+    ) -> torch.Tensor:
+        hidden_states = hidden_states + self.attn(
+            self.norm1(hidden_states),
+            cu_seqlens=cu_seqlens,
+            rotary_pos_emb=rotary_pos_emb,
+            position_embeddings=position_embeddings,
+        )
+        hidden_states = hidden_states + self.mlp(self.norm2(hidden_states))
+        return hidden_states
+
+class Qwen2_5_VLPatchMerger(nn.Module):
+    def __init__(self, dim: int, context_dim: int, spatial_merge_size: int = 2) -> None:
+        super().__init__()
+        self.hidden_size = context_dim * (spatial_merge_size**2)
+        self.ln_q = Qwen2RMSNorm(context_dim, eps=1e-6)
+        self.mlp = nn.Sequential(
+            nn.Linear(self.hidden_size, self.hidden_size),
+            nn.GELU(),
+            nn.Linear(self.hidden_size, dim),
+        )
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = self.mlp(self.ln_q(x).view(-1, self.hidden_size))
+        return x
+
+class Qwen2_5_VisionTransformerPretrainedModel(Qwen2_5_VLPreTrainedModel):
+    config_class = Qwen2_5_VLVisionConfig
+    _no_split_modules = ["Qwen2_5_VLVisionBlock"]
+
+    def __init__(self, config, *inputs, **kwargs) -> None:
+        super().__init__(config, *inputs, **kwargs)
+        self.spatial_merge_size = config.spatial_merge_size
+        self.patch_size = config.patch_size
+        self.fullatt_block_indexes = config.fullatt_block_indexes
+        self.window_size = config.window_size
+        self.spatial_merge_unit = self.spatial_merge_size * self.spatial_merge_size
+
+        self.patch_embed = Qwen2_5_VisionPatchEmbed(
+            patch_size=config.patch_size,
+            temporal_patch_size=config.temporal_patch_size,
+            in_channels=config.in_channels,
+            embed_dim=config.hidden_size,
+        )
+
+        head_dim = config.hidden_size // config.num_heads
+        self.rotary_pos_emb = Qwen2_5_VisionRotaryEmbedding(head_dim // 2)
+
+        self.blocks = nn.ModuleList(
+            [Qwen2_5_VLVisionBlock(config, config._attn_implementation) for _ in range(config.depth)]
+        )
+        self.merger = Qwen2_5_VLPatchMerger(
+            dim=config.out_hidden_size,
+            context_dim=config.hidden_size,
+            spatial_merge_size=config.spatial_merge_size,
+        )
+        # Merge each 2x2 patch group into one token via concat plus an MLP projection.
+        self.gradient_checkpointing = False
+
+    def rot_pos_emb(self, grid_thw):
+        pos_ids = []
+        for t, h, w in grid_thw:  # Generate 2D position indexes for each (t, h, w) grid.
+            hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w)  # Height-axis position indexes.
+            hpos_ids = hpos_ids.reshape(
+                h // self.spatial_merge_size,
+                self.spatial_merge_size,
+                w // self.spatial_merge_size,
+                self.spatial_merge_size,
+            )
+            hpos_ids = hpos_ids.permute(0, 2, 1, 3)
+            hpos_ids = hpos_ids.flatten()
+
+            wpos_ids = torch.arange(w).unsqueeze(0).expand(h, -1)  # Width-axis position indexes.
+            wpos_ids = wpos_ids.reshape(
+                h // self.spatial_merge_size,
+                self.spatial_merge_size,
+                w // self.spatial_merge_size,
+                self.spatial_merge_size,
+            )
+            wpos_ids = wpos_ids.permute(0, 2, 1, 3)
+            wpos_ids = wpos_ids.flatten()
+            pos_ids.append(torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1))
+        pos_ids = torch.cat(pos_ids, dim=0)
+        max_grid_size = grid_thw[:, 1:].max()
+        rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size)  # Rotary embeddings for the largest grid.
+        rotary_pos_emb = rotary_pos_emb_full[pos_ids].flatten(1)  # Gather embeddings by patch position index.
+        return rotary_pos_emb
+
+    def get_window_index(self, grid_thw):
+        window_index: list = []
+        cu_window_seqlens: list = [0]
+        window_index_id = 0
+        vit_merger_window_size = self.window_size // self.spatial_merge_size // self.patch_size
+
+        for grid_t, grid_h, grid_w in grid_thw:
+            llm_grid_h, llm_grid_w = (
+                grid_h // self.spatial_merge_size,
+                grid_w // self.spatial_merge_size,
+            )
+            index = torch.arange(grid_t * llm_grid_h * llm_grid_w).reshape(grid_t, llm_grid_h, llm_grid_w)
+            pad_h = vit_merger_window_size - llm_grid_h % vit_merger_window_size
+            pad_w = vit_merger_window_size - llm_grid_w % vit_merger_window_size
+            num_windows_h = (llm_grid_h + pad_h) // vit_merger_window_size
+            num_windows_w = (llm_grid_w + pad_w) // vit_merger_window_size
+            index_padded = F.pad(index, (0, pad_w, 0, pad_h), "constant", -100)
+            index_padded = index_padded.reshape(
+                grid_t,
+                num_windows_h,
+                vit_merger_window_size,
+                num_windows_w,
+                vit_merger_window_size,
+            )
+            index_padded = index_padded.permute(0, 1, 3, 2, 4).reshape(
+                grid_t,
+                num_windows_h * num_windows_w,
+                vit_merger_window_size,
+                vit_merger_window_size,
+            )
+            seqlens = (index_padded != -100).sum([2, 3]).reshape(-1)
+            index_padded = index_padded.reshape(-1)
+            index_new = index_padded[index_padded != -100]
+            window_index.append(index_new + window_index_id)
+            cu_seqlens_tmp = seqlens.cumsum(0) * self.spatial_merge_unit + cu_window_seqlens[-1]
+            cu_window_seqlens.extend(cu_seqlens_tmp.tolist())
+            window_index_id += (grid_t * llm_grid_h * llm_grid_w).item()
+        window_index = torch.cat(window_index, dim=0)
+
+        return window_index, cu_window_seqlens
+
+    def forward(self, hidden_states: torch.Tensor, grid_thw: torch.Tensor) -> torch.Tensor:
+        """
+        Args:
+            hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`):
+                The final hidden states of the model.
+            grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`):
+                The temporal, height and width of feature shape of each image in LLM.
+
+        Returns:
+            `torch.Tensor`: hidden_states.
+        """
+        hidden_states = self.patch_embed(hidden_states)
+        rotary_pos_emb = self.rot_pos_emb(grid_thw)  # Compute RoPE for each input video.
+        window_index, cu_window_seqlens = self.get_window_index(grid_thw)
+        cu_window_seqlens = torch.tensor(
+            cu_window_seqlens,
+            device=hidden_states.device,
+            dtype=grid_thw.dtype if torch.jit.is_tracing() else torch.int32,
+        )
+        cu_window_seqlens = torch.unique_consecutive(cu_window_seqlens)
+
+        seq_len, _ = hidden_states.size()
+        hidden_states = hidden_states.reshape(seq_len // self.spatial_merge_unit, self.spatial_merge_unit, -1)
+        hidden_states = hidden_states[window_index, :, :]
+        hidden_states = hidden_states.reshape(seq_len, -1)
+        rotary_pos_emb = rotary_pos_emb.reshape(seq_len // self.spatial_merge_unit, self.spatial_merge_unit, -1)
+        rotary_pos_emb = rotary_pos_emb[window_index, :, :]
+        rotary_pos_emb = rotary_pos_emb.reshape(seq_len, -1)
+        emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
+        position_embeddings = (emb.cos(), emb.sin())
+
+        cu_seqlens = torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).cumsum(
+            dim=0,
+            # Select dtype based on the following factors:
+            #  - FA2 requires that cu_seqlens_q must have dtype int32
+            #  - torch.onnx.export requires that cu_seqlens_q must have same dtype as grid_thw
+            # See https://github.com/huggingface/transformers/pull/34852 for more information
+            dtype=grid_thw.dtype if torch.jit.is_tracing() else torch.int32,
+        )
+        cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0)
+
+        for layer_num, blk in enumerate(self.blocks):
+            if layer_num in self.fullatt_block_indexes:
+                cu_seqlens_now = cu_seqlens
+            else:
+                cu_seqlens_now = cu_window_seqlens
+            if self.gradient_checkpointing and self.training:
+                hidden_states = self._gradient_checkpointing_func(
+                    blk.__call__, hidden_states, cu_seqlens_now, None, position_embeddings
+                )
+            else:
+                hidden_states = blk(hidden_states, cu_seqlens=cu_seqlens_now, position_embeddings=position_embeddings)
+
+        hidden_states = self.merger(hidden_states) # L x 1280 -> L//4 x 2048
+        reverse_indices = torch.argsort(window_index)
+        hidden_states = hidden_states[reverse_indices, :] # Still L//4 x 2048.
+
+        return hidden_states

native/refiner_modules.py

@@ -0,0 +1,1330 @@
+# ---------------------------------------------------------------------------
+# Standard / third-party imports shared by all sections
+# ---------------------------------------------------------------------------
+
+import itertools
+import math
+from dataclasses import dataclass
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+from .compat import flash_attn_varlen_func
+from .compat import index_first_axis, pad_input, unpad_input
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn import RMSNorm
+
+from einops import rearrange, repeat
+
+from diffusers.configuration_utils import ConfigMixin, register_to_config
+from diffusers.loaders import PeftAdapterMixin
+from diffusers.loaders.single_file_model import FromOriginalModelMixin
+from diffusers.models.activations import get_activation
+from diffusers.models.attention_processor import Attention
+from diffusers.models.embeddings import Timesteps, get_1d_rotary_pos_embed
+from diffusers.models.modeling_outputs import Transformer2DModelOutput
+from diffusers.models.modeling_utils import ModelMixin
+from diffusers.schedulers.scheduling_utils import SchedulerMixin
+from diffusers.utils import USE_PEFT_BACKEND, BaseOutput, logging, scale_lora_layers, unscale_lora_layers
+
+logger = logging.get_logger(__name__)
+
+
+def swiglu(x, y):
+    return F.silu(x.float(), inplace=False).to(x.dtype) * y
+
+
+class TimestepEmbedding(nn.Module):
+    def __init__(
+        self,
+        in_channels: int,
+        time_embed_dim: int,
+        act_fn: str = "silu",
+        out_dim: int = None,
+        post_act_fn: Optional[str] = None,
+        cond_proj_dim=None,
+        sample_proj_bias=True,
+    ):
+        super().__init__()
+
+        self.linear_1 = nn.Linear(in_channels, time_embed_dim, sample_proj_bias)
+
+        if cond_proj_dim is not None:
+            self.cond_proj = nn.Linear(cond_proj_dim, in_channels, bias=False)
+        else:
+            self.cond_proj = None
+
+        self.act = get_activation(act_fn)
+
+        if out_dim is not None:
+            time_embed_dim_out = out_dim
+        else:
+            time_embed_dim_out = time_embed_dim
+        self.linear_2 = nn.Linear(time_embed_dim, time_embed_dim_out, sample_proj_bias)
+
+        if post_act_fn is None:
+            self.post_act = None
+        else:
+            self.post_act = get_activation(post_act_fn)
+
+        self.initialize_weights()
+
+    def initialize_weights(self):
+        nn.init.normal_(self.linear_1.weight, std=0.02)
+        nn.init.zeros_(self.linear_1.bias)
+        nn.init.normal_(self.linear_2.weight, std=0.02)
+        nn.init.zeros_(self.linear_2.bias)
+
+    def forward(self, sample, condition=None):
+        if condition is not None:
+            sample = sample + self.cond_proj(condition)
+        sample = self.linear_1(sample)
+        if self.act is not None:
+            sample = self.act(sample)
+        sample = self.linear_2(sample)
+        if self.post_act is not None:
+            sample = self.post_act(sample)
+        return sample
+
+
+def apply_rotary_emb(
+    x: torch.Tensor,
+    freqs_cis: Union[torch.Tensor, Tuple[torch.Tensor]],
+    use_real: bool = True,
+    use_real_unbind_dim: int = -1,
+) -> Tuple[torch.Tensor, torch.Tensor]:
+    """
+    Apply rotary embeddings to input tensors using the given frequency tensor.
+    """
+    if use_real:
+        cos, sin = freqs_cis  # [S, D]
+        cos = cos[None, None]
+        sin = sin[None, None]
+        cos, sin = cos.to(x.device), sin.to(x.device)
+
+        if use_real_unbind_dim == -1:
+            x_real, x_imag = x.reshape(*x.shape[:-1], -1, 2).unbind(-1)
+            x_rotated = torch.stack([-x_imag, x_real], dim=-1).flatten(3)
+        elif use_real_unbind_dim == -2:
+            x_real, x_imag = x.reshape(*x.shape[:-1], 2, -1).unbind(-2)
+            x_rotated = torch.cat([-x_imag, x_real], dim=-1)
+        else:
+            raise ValueError(f"`use_real_unbind_dim={use_real_unbind_dim}` but should be -1 or -2.")
+
+        out = (x.float() * cos + x_rotated.float() * sin).to(x.dtype)
+        return out
+    else:
+        x_rotated = torch.view_as_complex(x.float().reshape(*x.shape[:-1], x.shape[-1] // 2, 2))
+        freqs_cis = freqs_cis.unsqueeze(2)
+        x_out = torch.view_as_real(x_rotated * freqs_cis).flatten(3)
+        return x_out.type_as(x)
+
+
+@dataclass
+class TeaCacheParams:
+    """
+    TeaCache parameters for Transformer2DModel.
+    See https://github.com/ali-vilab/TeaCache/ for a more comprehensive understanding.
+    """
+    previous_residual: Optional[torch.Tensor] = None
+    previous_modulated_inp: Optional[torch.Tensor] = None
+    accumulated_rel_l1_distance: float = 0
+    is_first_or_last_step: bool = False
+
+
+def derivative_approximation(*args, **kwargs):
+    pass
+
+
+def taylor_formula(*args, **kwargs):
+    pass
+
+
+def taylor_cache_init(*args, **kwargs):
+    pass
+
+
+def cache_init(*args, **kwargs):
+    pass
+
+
+def cal_type(*args, **kwargs):
+    pass
+
+
+class LuminaRMSNormZero(nn.Module):
+    """
+    Norm layer adaptive RMS normalization zero.
+    """
+
+    def __init__(
+        self,
+        embedding_dim: int,
+        norm_eps: float,
+        norm_elementwise_affine: bool,
+    ):
+        super().__init__()
+        self.silu = nn.SiLU()
+        self.linear = nn.Linear(
+            min(embedding_dim, 1024),
+            4 * embedding_dim,
+            bias=True,
+        )
+        self.norm = RMSNorm(embedding_dim, eps=norm_eps)
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        emb: Optional[torch.Tensor] = None,
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+        emb = self.linear(self.silu(emb))
+        scale_msa, gate_msa, scale_mlp, gate_mlp = emb.chunk(4, dim=1)
+        x = self.norm(x) * (1 + scale_msa[:, None])
+        return x, gate_msa, scale_mlp, gate_mlp
+
+
+class LuminaLayerNormContinuous(nn.Module):
+    def __init__(
+        self,
+        embedding_dim: int,
+        conditioning_embedding_dim: int,
+        elementwise_affine=True,
+        eps=1e-5,
+        bias=True,
+        norm_type="layer_norm",
+        out_dim: Optional[int] = None,
+    ):
+        super().__init__()
+
+        self.silu = nn.SiLU()
+        self.linear_1 = nn.Linear(conditioning_embedding_dim, embedding_dim, bias=bias)
+
+        if norm_type == "layer_norm":
+            self.norm = nn.LayerNorm(embedding_dim, eps, elementwise_affine, bias)
+        elif norm_type == "rms_norm":
+            self.norm = RMSNorm(embedding_dim, eps=eps, elementwise_affine=elementwise_affine)
+        else:
+            raise ValueError(f"unknown norm_type {norm_type}")
+
+        self.linear_2 = None
+        if out_dim is not None:
+            self.linear_2 = nn.Linear(embedding_dim, out_dim, bias=bias)
+
+    def forward(
+        self,
+        x: torch.Tensor,
+        conditioning_embedding: torch.Tensor,
+    ) -> torch.Tensor:
+        emb = self.linear_1(self.silu(conditioning_embedding).to(x.dtype))
+        scale = emb
+        x = self.norm(x) * (1 + scale)[:, None, :]
+        if self.linear_2 is not None:
+            x = self.linear_2(x)
+        return x
+
+
+class LuminaFeedForward(nn.Module):
+    def __init__(
+        self,
+        dim: int,
+        inner_dim: int,
+        multiple_of: Optional[int] = 256,
+        ffn_dim_multiplier: Optional[float] = None,
+    ):
+        super().__init__()
+
+        if ffn_dim_multiplier is not None:
+            inner_dim = int(ffn_dim_multiplier * inner_dim)
+        inner_dim = multiple_of * ((inner_dim + multiple_of - 1) // multiple_of)
+
+        self.linear_1 = nn.Linear(dim, inner_dim, bias=False)
+        self.linear_2 = nn.Linear(inner_dim, dim, bias=False)
+        self.linear_3 = nn.Linear(dim, inner_dim, bias=False)
+
+    def forward(self, x):
+        h1, h2 = self.linear_1(x), self.linear_3(x)
+        return self.linear_2(swiglu(h1, h2))
+
+
+class Lumina2CombinedTimestepCaptionEmbedding(nn.Module):
+    def __init__(
+        self,
+        hidden_size: int = 4096,
+        text_feat_dim: int = 2048,
+        frequency_embedding_size: int = 256,
+        norm_eps: float = 1e-5,
+        timestep_scale: float = 1.0,
+    ) -> None:
+        super().__init__()
+
+        self.time_proj = Timesteps(
+            num_channels=frequency_embedding_size,
+            flip_sin_to_cos=True,
+            downscale_freq_shift=0.0,
+            scale=timestep_scale,
+        )
+        self.timestep_embedder = TimestepEmbedding(
+            in_channels=frequency_embedding_size,
+            time_embed_dim=min(hidden_size, 1024),
+        )
+        self.caption_embedder = nn.Sequential(
+            RMSNorm(text_feat_dim, eps=norm_eps),
+            nn.Linear(text_feat_dim, hidden_size, bias=True),
+        )
+        self._initialize_weights()
+
+    def _initialize_weights(self):
+        nn.init.trunc_normal_(self.caption_embedder[1].weight, std=0.02)
+        nn.init.zeros_(self.caption_embedder[1].bias)
+
+    def forward(
+        self, timestep: torch.Tensor, text_hidden_states: torch.Tensor, dtype: torch.dtype
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        timestep_proj = self.time_proj(timestep).to(dtype=dtype)
+        time_embed = self.timestep_embedder(timestep_proj)
+        caption_embed = self.caption_embedder(text_hidden_states)
+        return time_embed, caption_embed
+
+
+class AttnProcessorFlash2Varlen:
+    """
+    Processor for implementing scaled dot-product attention with flash attention
+    and variable length sequences.
+    """
+
+    def __init__(self) -> None:
+        pass
+    #     if not is_flash_attn_available():
+    #         raise ImportError(
+    #             "AttnProcessorFlash2Varlen requires flash_attn. "
+    #             "Please install flash_attn."
+    #         )
+
+    def _upad_input(
+        self,
+        query_layer: torch.Tensor,
+        key_layer: torch.Tensor,
+        value_layer: torch.Tensor,
+        attention_mask: torch.Tensor,
+        query_length: int,
+        num_heads: int,
+    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, Tuple[torch.Tensor, torch.Tensor], Tuple[int, int]]:
+        def _get_unpad_data(attention_mask: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, int]:
+            seqlens_in_batch = attention_mask.sum(dim=-1, dtype=torch.int32)
+            indices = torch.nonzero(attention_mask.flatten(), as_tuple=False).flatten()
+            max_seqlen_in_batch = seqlens_in_batch.max().item()
+            cu_seqlens = F.pad(torch.cumsum(seqlens_in_batch, dim=0, dtype=torch.int32), (1, 0))
+            return indices, cu_seqlens, max_seqlen_in_batch
+
+        indices_k, cu_seqlens_k, max_seqlen_in_batch_k = _get_unpad_data(attention_mask)
+        batch_size, kv_seq_len, num_key_value_heads, head_dim = key_layer.shape
+
+        key_layer = index_first_axis(
+            key_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k,
+        )
+        value_layer = index_first_axis(
+            value_layer.reshape(batch_size * kv_seq_len, num_key_value_heads, head_dim), indices_k,
+        )
+
+        if query_length == kv_seq_len:
+            query_layer = index_first_axis(
+                query_layer.reshape(batch_size * kv_seq_len, num_heads, head_dim), indices_k,
+            )
+            cu_seqlens_q = cu_seqlens_k
+            max_seqlen_in_batch_q = max_seqlen_in_batch_k
+            indices_q = indices_k
+        elif query_length == 1:
+            max_seqlen_in_batch_q = 1
+            cu_seqlens_q = torch.arange(
+                batch_size + 1, dtype=torch.int32, device=query_layer.device
+            )
+            indices_q = cu_seqlens_q[:-1]
+            query_layer = query_layer.squeeze(1)
+        else:
+            attention_mask = attention_mask[:, -query_length:]
+            query_layer, indices_q, cu_seqlens_q, max_seqlen_in_batch_q = unpad_input(
+                query_layer, attention_mask
+            )
+
+        return (
+            query_layer, key_layer, value_layer, indices_q,
+            (cu_seqlens_q, cu_seqlens_k),
+            (max_seqlen_in_batch_q, max_seqlen_in_batch_k),
+        )
+
+    def __call__(
+        self,
+        attn: Attention,
+        hidden_states: torch.Tensor,
+        encoder_hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        image_rotary_emb: Optional[torch.Tensor] = None,
+        base_sequence_length: Optional[int] = None,
+    ) -> torch.Tensor:
+        batch_size, sequence_length, _ = hidden_states.shape
+
+        query = attn.to_q(hidden_states)
+        key = attn.to_k(encoder_hidden_states)
+        value = attn.to_v(encoder_hidden_states)
+
+        query_dim = query.shape[-1]
+        inner_dim = key.shape[-1]
+        head_dim = query_dim // attn.heads
+        dtype = query.dtype
+        kv_heads = inner_dim // head_dim
+
+        query = query.view(batch_size, -1, attn.heads, head_dim)
+        key = key.view(batch_size, -1, kv_heads, head_dim)
+        value = value.view(batch_size, -1, kv_heads, head_dim)
+
+        if attn.norm_q is not None:
+            query = attn.norm_q(query)
+        if attn.norm_k is not None:
+            key = attn.norm_k(key)
+
+        if image_rotary_emb is not None:
+            query = apply_rotary_emb(query, image_rotary_emb, use_real=False)
+            key = apply_rotary_emb(key, image_rotary_emb, use_real=False)
+
+        query, key = query.to(dtype), key.to(dtype)
+
+        if base_sequence_length is not None:
+            softmax_scale = math.sqrt(math.log(sequence_length, base_sequence_length)) * attn.scale
+        else:
+            softmax_scale = attn.scale
+
+        (
+            query_states, key_states, value_states, indices_q,
+            cu_seq_lens, max_seq_lens,
+        ) = self._upad_input(query, key, value, attention_mask, sequence_length, attn.heads)
+
+        cu_seqlens_q, cu_seqlens_k = cu_seq_lens
+        max_seqlen_in_batch_q, max_seqlen_in_batch_k = max_seq_lens
+
+        if kv_heads < attn.heads:
+            key_states = repeat(key_states, "l h c -> l (h k) c", k=attn.heads // kv_heads)
+            value_states = repeat(value_states, "l h c -> l (h k) c", k=attn.heads // kv_heads)
+
+        attn_output_unpad = flash_attn_varlen_func(
+            query_states, key_states, value_states,
+            cu_seqlens_q=cu_seqlens_q, cu_seqlens_k=cu_seqlens_k,
+            max_seqlen_q=max_seqlen_in_batch_q, max_seqlen_k=max_seqlen_in_batch_k,
+            dropout_p=0.0, causal=False, softmax_scale=softmax_scale,
+        )
+
+        hidden_states = pad_input(attn_output_unpad, indices_q, batch_size, sequence_length)
+        hidden_states = hidden_states.flatten(-2)
+        hidden_states = hidden_states.type_as(query)
+
+        hidden_states = attn.to_out[0](hidden_states)
+        hidden_states = attn.to_out[1](hidden_states)
+        return hidden_states
+
+
+class AttnProcessor:
+    """
+    Processor for implementing scaled dot-product attention (PyTorch 2.0+).
+    """
+
+    def __init__(self) -> None:
+        if not hasattr(F, "scaled_dot_product_attention"):
+            raise ImportError(
+                "AttnProcessor requires PyTorch 2.0. "
+                "Please upgrade PyTorch to version 2.0 or later."
+            )
+
+    def __call__(
+        self,
+        attn: Attention,
+        hidden_states: torch.Tensor,
+        encoder_hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        image_rotary_emb: Optional[torch.Tensor] = None,
+        base_sequence_length: Optional[int] = None,
+    ) -> torch.Tensor:
+        batch_size, sequence_length, _ = hidden_states.shape
+
+        query = attn.to_q(hidden_states)
+        key = attn.to_k(encoder_hidden_states)
+        value = attn.to_v(encoder_hidden_states)
+
+        query_dim = query.shape[-1]
+        inner_dim = key.shape[-1]
+        head_dim = query_dim // attn.heads
+        dtype = query.dtype
+        kv_heads = inner_dim // head_dim
+
+        query = query.view(batch_size, -1, attn.heads, head_dim)
+        key = key.view(batch_size, -1, kv_heads, head_dim)
+        value = value.view(batch_size, -1, kv_heads, head_dim)
+
+        if attn.norm_q is not None:
+            query = attn.norm_q(query)
+        if attn.norm_k is not None:
+            key = attn.norm_k(key)
+
+        if image_rotary_emb is not None:
+            query = apply_rotary_emb(query, image_rotary_emb, use_real=False)
+            key = apply_rotary_emb(key, image_rotary_emb, use_real=False)
+
+        query, key = query.to(dtype), key.to(dtype)
+
+        if base_sequence_length is not None:
+            softmax_scale = math.sqrt(math.log(sequence_length, base_sequence_length)) * attn.scale
+        else:
+            softmax_scale = attn.scale
+
+        if attention_mask is not None:
+            attention_mask = attention_mask.bool().view(batch_size, 1, 1, -1)
+
+        query = query.transpose(1, 2)
+        key = key.transpose(1, 2)
+        value = value.transpose(1, 2)
+
+        key = key.repeat_interleave(query.size(-3) // key.size(-3), -3)
+        value = value.repeat_interleave(query.size(-3) // value.size(-3), -3)
+
+        hidden_states = F.scaled_dot_product_attention(
+            query, key, value, attn_mask=attention_mask, scale=softmax_scale
+        )
+        hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
+        hidden_states = hidden_states.type_as(query)
+
+        hidden_states = attn.to_out[0](hidden_states)
+        hidden_states = attn.to_out[1](hidden_states)
+        return hidden_states
+
+
+
+class RotaryPosEmbed(nn.Module):
+    def __init__(
+        self,
+        theta: int,
+        axes_dim: Tuple[int, int, int],
+        axes_lens: Tuple[int, int, int] = (300, 512, 512),
+        patch_size: int = 2,
+    ):
+        super().__init__()
+        self.theta = theta
+        self.axes_dim = axes_dim
+        self.axes_lens = axes_lens
+        self.patch_size = patch_size
+
+    @staticmethod
+    def get_freqs_cis(
+        axes_dim: Tuple[int, int, int],
+        axes_lens: Tuple[int, int, int],
+        theta: int,
+    ) -> List[torch.Tensor]:
+        freqs_cis = []
+        freqs_dtype = torch.float32 if torch.backends.mps.is_available() else torch.float64
+        for i, (d, e) in enumerate(zip(axes_dim, axes_lens)):
+            emb = get_1d_rotary_pos_embed(d, e, theta=theta, freqs_dtype=freqs_dtype)
+            freqs_cis.append(emb)
+        return freqs_cis
+
+    def _get_freqs_cis(self, freqs_cis, ids: torch.Tensor) -> torch.Tensor:
+        device = ids.device
+        if ids.device.type == "mps":
+            ids = ids.to("cpu")
+
+        result = []
+        for i in range(len(self.axes_dim)):
+            freqs = freqs_cis[i].to(ids.device)
+            index = ids[:, :, i : i + 1].repeat(1, 1, freqs.shape[-1]).to(torch.int64)
+            result.append(
+                torch.gather(freqs.unsqueeze(0).repeat(index.shape[0], 1, 1), dim=1, index=index)
+            )
+        return torch.cat(result, dim=-1).to(device)
+
+    def forward(
+        self,
+        freqs_cis,
+        attention_mask,
+        l_effective_ref_img_len,
+        l_effective_img_len,
+        ref_img_sizes,
+        img_sizes,
+        device,
+    ):
+        batch_size = len(attention_mask)
+        p = self.patch_size
+
+        encoder_seq_len = attention_mask.shape[1]
+        l_effective_cap_len = attention_mask.sum(dim=1).tolist()
+
+        seq_lengths = [
+            cap_len + sum(ref_img_len) + img_len
+            for cap_len, ref_img_len, img_len in zip(
+                l_effective_cap_len, l_effective_ref_img_len, l_effective_img_len
+            )
+        ]
+
+        max_seq_len = max(seq_lengths)
+        max_ref_img_len = max([sum(ref_img_len) for ref_img_len in l_effective_ref_img_len])
+        max_img_len = max(l_effective_img_len)
+
+        position_ids = torch.zeros(batch_size, max_seq_len, 3, dtype=torch.int32, device=device)
+
+        for i, (cap_seq_len, seq_len) in enumerate(zip(l_effective_cap_len, seq_lengths)):
+            position_ids[i, :cap_seq_len] = repeat(
+                torch.arange(cap_seq_len, dtype=torch.int32, device=device), "l -> l 3"
+            )
+
+            pe_shift = cap_seq_len
+            pe_shift_len = cap_seq_len
+
+            if ref_img_sizes[i] is not None:
+                for ref_img_size, ref_img_len in zip(ref_img_sizes[i], l_effective_ref_img_len[i]):
+                    H, W = ref_img_size
+                    ref_H_tokens, ref_W_tokens = H // p, W // p
+                    assert ref_H_tokens * ref_W_tokens == ref_img_len
+
+                    row_ids = repeat(
+                        torch.arange(ref_H_tokens, dtype=torch.int32, device=device),
+                        "h -> h w", w=ref_W_tokens,
+                    ).flatten()
+                    col_ids = repeat(
+                        torch.arange(ref_W_tokens, dtype=torch.int32, device=device),
+                        "w -> h w", h=ref_H_tokens,
+                    ).flatten()
+                    position_ids[i, pe_shift_len:pe_shift_len + ref_img_len, 0] = pe_shift
+                    position_ids[i, pe_shift_len:pe_shift_len + ref_img_len, 1] = row_ids
+                    position_ids[i, pe_shift_len:pe_shift_len + ref_img_len, 2] = col_ids
+
+                    pe_shift += max(ref_H_tokens, ref_W_tokens)
+                    pe_shift_len += ref_img_len
+
+            H, W = img_sizes[i]
+            H_tokens, W_tokens = H // p, W // p
+            assert H_tokens * W_tokens == l_effective_img_len[i]
+
+            row_ids = repeat(
+                torch.arange(H_tokens, dtype=torch.int32, device=device), "h -> h w", w=W_tokens
+            ).flatten()
+            col_ids = repeat(
+                torch.arange(W_tokens, dtype=torch.int32, device=device), "w -> h w", h=H_tokens
+            ).flatten()
+
+            assert pe_shift_len + l_effective_img_len[i] == seq_len
+            position_ids[i, pe_shift_len: seq_len, 0] = pe_shift
+            position_ids[i, pe_shift_len: seq_len, 1] = row_ids
+            position_ids[i, pe_shift_len: seq_len, 2] = col_ids
+
+        freqs_cis = self._get_freqs_cis(freqs_cis, position_ids)
+
+        cap_freqs_cis = torch.zeros(
+            batch_size, encoder_seq_len, freqs_cis.shape[-1], device=device, dtype=freqs_cis.dtype
+        )
+        ref_img_freqs_cis = torch.zeros(
+            batch_size, max_ref_img_len, freqs_cis.shape[-1], device=device, dtype=freqs_cis.dtype
+        )
+        img_freqs_cis = torch.zeros(
+            batch_size, max_img_len, freqs_cis.shape[-1], device=device, dtype=freqs_cis.dtype
+        )
+
+        for i, (cap_seq_len, ref_img_len, img_len, seq_len) in enumerate(
+            zip(l_effective_cap_len, l_effective_ref_img_len, l_effective_img_len, seq_lengths)
+        ):
+            cap_freqs_cis[i, :cap_seq_len] = freqs_cis[i, :cap_seq_len]
+            ref_img_freqs_cis[i, :sum(ref_img_len)] = freqs_cis[
+                i, cap_seq_len:cap_seq_len + sum(ref_img_len)
+            ]
+            img_freqs_cis[i, :img_len] = freqs_cis[
+                i,
+                cap_seq_len + sum(ref_img_len):cap_seq_len + sum(ref_img_len) + img_len,
+            ]
+
+        return (
+            cap_freqs_cis,
+            ref_img_freqs_cis,
+            img_freqs_cis,
+            freqs_cis,
+            l_effective_cap_len,
+            seq_lengths,
+        )
+
+
+class TransformerBlock(nn.Module):
+    """
+    Transformer block for refiner model.
+    """
+
+    def __init__(
+        self,
+        dim: int,
+        num_attention_heads: int,
+        num_kv_heads: int,
+        multiple_of: int,
+        ffn_dim_multiplier: float,
+        norm_eps: float,
+        modulation: bool = True,
+    ) -> None:
+        super().__init__()
+        self.head_dim = dim // num_attention_heads
+        self.modulation = modulation
+
+        try:
+            processor = AttnProcessorFlash2Varlen()
+        except ImportError:
+            processor = AttnProcessor()
+
+        self.attn = Attention(
+            query_dim=dim,
+            cross_attention_dim=None,
+            dim_head=dim // num_attention_heads,
+            qk_norm="rms_norm",
+            heads=num_attention_heads,
+            kv_heads=num_kv_heads,
+            eps=1e-5,
+            bias=False,
+            out_bias=False,
+            processor=processor,
+        )
+
+        self.feed_forward = LuminaFeedForward(
+            dim=dim,
+            inner_dim=4 * dim,
+            multiple_of=multiple_of,
+            ffn_dim_multiplier=ffn_dim_multiplier,
+        )
+
+        if modulation:
+            self.norm1 = LuminaRMSNormZero(
+                embedding_dim=dim,
+                norm_eps=norm_eps,
+                norm_elementwise_affine=True,
+            )
+        else:
+            self.norm1 = RMSNorm(dim, eps=norm_eps)
+
+        self.ffn_norm1 = RMSNorm(dim, eps=norm_eps)
+        self.norm2 = RMSNorm(dim, eps=norm_eps)
+        self.ffn_norm2 = RMSNorm(dim, eps=norm_eps)
+
+        self.initialize_weights()
+
+    def initialize_weights(self) -> None:
+        nn.init.xavier_uniform_(self.attn.to_q.weight)
+        nn.init.xavier_uniform_(self.attn.to_k.weight)
+        nn.init.xavier_uniform_(self.attn.to_v.weight)
+        nn.init.xavier_uniform_(self.attn.to_out[0].weight)
+
+        nn.init.xavier_uniform_(self.feed_forward.linear_1.weight)
+        nn.init.xavier_uniform_(self.feed_forward.linear_2.weight)
+        nn.init.xavier_uniform_(self.feed_forward.linear_3.weight)
+
+        if self.modulation:
+            nn.init.zeros_(self.norm1.linear.weight)
+            nn.init.zeros_(self.norm1.linear.bias)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: torch.Tensor,
+        image_rotary_emb: torch.Tensor,
+        temb: Optional[torch.Tensor] = None,
+    ) -> torch.Tensor:
+        enable_taylorseer = getattr(self, 'enable_taylorseer', False)
+        if enable_taylorseer:
+            if self.modulation:
+                if temb is None:
+                    raise ValueError("temb must be provided when modulation is enabled")
+
+                if self.current['type'] == 'full':
+                    self.current['module'] = 'total'
+                    taylor_cache_init(cache_dic=self.cache_dic, current=self.current)
+
+                    norm_hidden_states, gate_msa, scale_mlp, gate_mlp = self.norm1(hidden_states, temb)
+                    attn_output = self.attn(
+                        hidden_states=norm_hidden_states,
+                        encoder_hidden_states=norm_hidden_states,
+                        attention_mask=attention_mask,
+                        image_rotary_emb=image_rotary_emb,
+                    )
+                    hidden_states = hidden_states + gate_msa.unsqueeze(1).tanh() * self.norm2(attn_output)
+                    mlp_output = self.feed_forward(self.ffn_norm1(hidden_states) * (1 + scale_mlp.unsqueeze(1)))
+                    hidden_states = hidden_states + gate_mlp.unsqueeze(1).tanh() * self.ffn_norm2(mlp_output)
+
+                    derivative_approximation(cache_dic=self.cache_dic, current=self.current, feature=hidden_states)
+
+                elif self.current['type'] == 'Taylor':
+                    self.current['module'] = 'total'
+                    hidden_states = taylor_formula(cache_dic=self.cache_dic, current=self.current)
+            else:
+                norm_hidden_states = self.norm1(hidden_states)
+                attn_output = self.attn(
+                    hidden_states=norm_hidden_states,
+                    encoder_hidden_states=norm_hidden_states,
+                    attention_mask=attention_mask,
+                    image_rotary_emb=image_rotary_emb,
+                )
+                hidden_states = hidden_states + self.norm2(attn_output)
+                mlp_output = self.feed_forward(self.ffn_norm1(hidden_states))
+                hidden_states = hidden_states + self.ffn_norm2(mlp_output)
+        else:
+            if self.modulation:
+                if temb is None:
+                    raise ValueError("temb must be provided when modulation is enabled")
+
+                norm_hidden_states, gate_msa, scale_mlp, gate_mlp = self.norm1(hidden_states, temb)
+                attn_output = self.attn(
+                    hidden_states=norm_hidden_states,
+                    encoder_hidden_states=norm_hidden_states,
+                    attention_mask=attention_mask,
+                    image_rotary_emb=image_rotary_emb,
+                )
+                hidden_states = hidden_states + gate_msa.unsqueeze(1).tanh() * self.norm2(attn_output)
+                mlp_output = self.feed_forward(self.ffn_norm1(hidden_states) * (1 + scale_mlp.unsqueeze(1)))
+                hidden_states = hidden_states + gate_mlp.unsqueeze(1).tanh() * self.ffn_norm2(mlp_output)
+            else:
+                norm_hidden_states = self.norm1(hidden_states)
+                attn_output = self.attn(
+                    hidden_states=norm_hidden_states,
+                    encoder_hidden_states=norm_hidden_states,
+                    attention_mask=attention_mask,
+                    image_rotary_emb=image_rotary_emb,
+                )
+                hidden_states = hidden_states + self.norm2(attn_output)
+                mlp_output = self.feed_forward(self.ffn_norm1(hidden_states))
+                hidden_states = hidden_states + self.ffn_norm2(mlp_output)
+
+        return hidden_states
+
+
+class Transformer2DModel(ModelMixin, ConfigMixin, PeftAdapterMixin, FromOriginalModelMixin):
+    """
+    Transformer 2D Model.
+    """
+
+    _supports_gradient_checkpointing = True
+    _no_split_modules = ["TransformerBlock"]
+    _skip_layerwise_casting_patterns = ["x_embedder", "norm"]
+
+    @register_to_config
+    def __init__(
+        self,
+        patch_size: int = 2,
+        in_channels: int = 16,
+        out_channels: Optional[int] = None,
+        hidden_size: int = 2304,
+        num_layers: int = 26,
+        num_refiner_layers: int = 2,
+        num_attention_heads: int = 24,
+        num_kv_heads: int = 8,
+        multiple_of: int = 256,
+        ffn_dim_multiplier: Optional[float] = None,
+        norm_eps: float = 1e-5,
+        axes_dim_rope: Tuple[int, int, int] = (32, 32, 32),
+        axes_lens: Tuple[int, int, int] = (300, 512, 512),
+        text_feat_dim: int = 1024,
+        timestep_scale: float = 1.0,
+    ) -> None:
+        super().__init__()
+
+        if (hidden_size // num_attention_heads) != sum(axes_dim_rope):
+            raise ValueError(
+                f"hidden_size // num_attention_heads ({hidden_size // num_attention_heads}) "
+                f"must equal sum(axes_dim_rope) ({sum(axes_dim_rope)})"
+            )
+
+        self.out_channels = out_channels or in_channels
+
+        self.rope_embedder = RotaryPosEmbed(
+            theta=10000,
+            axes_dim=axes_dim_rope,
+            axes_lens=axes_lens,
+            patch_size=patch_size,
+        )
+
+        self.x_embedder = nn.Linear(
+            in_features=patch_size * patch_size * in_channels,
+            out_features=hidden_size,
+        )
+
+        self.ref_image_patch_embedder = nn.Linear(
+            in_features=patch_size * patch_size * in_channels,
+            out_features=hidden_size,
+        )
+
+        self.time_caption_embed = Lumina2CombinedTimestepCaptionEmbedding(
+            hidden_size=hidden_size,
+            text_feat_dim=text_feat_dim,
+            norm_eps=norm_eps,
+            timestep_scale=timestep_scale,
+        )
+
+        self.noise_refiner = nn.ModuleList([
+            TransformerBlock(
+                hidden_size, num_attention_heads, num_kv_heads,
+                multiple_of, ffn_dim_multiplier, norm_eps, modulation=True,
+            )
+            for _ in range(num_refiner_layers)
+        ])
+
+        self.ref_image_refiner = nn.ModuleList([
+            TransformerBlock(
+                hidden_size, num_attention_heads, num_kv_heads,
+                multiple_of, ffn_dim_multiplier, norm_eps, modulation=True,
+            )
+            for _ in range(num_refiner_layers)
+        ])
+
+        self.context_refiner = nn.ModuleList([
+            TransformerBlock(
+                hidden_size, num_attention_heads, num_kv_heads,
+                multiple_of, ffn_dim_multiplier, norm_eps, modulation=False,
+            )
+            for _ in range(num_refiner_layers)
+        ])
+
+        self.layers = nn.ModuleList([
+            TransformerBlock(
+                hidden_size, num_attention_heads, num_kv_heads,
+                multiple_of, ffn_dim_multiplier, norm_eps, modulation=True,
+            )
+            for _ in range(num_layers)
+        ])
+
+        self.norm_out = LuminaLayerNormContinuous(
+            embedding_dim=hidden_size,
+            conditioning_embedding_dim=min(hidden_size, 1024),
+            elementwise_affine=False,
+            eps=1e-6,
+            bias=True,
+            out_dim=patch_size * patch_size * self.out_channels,
+        )
+
+        self.image_index_embedding = nn.Parameter(torch.randn(5, hidden_size))
+
+        self.gradient_checkpointing = False
+
+        self.initialize_weights()
+
+        self.enable_teacache = False
+        self.teacache_rel_l1_thresh = 0.05
+        self.teacache_params = TeaCacheParams()
+
+        coefficients = [-5.48259225, 11.48772289, -4.47407401, 2.47730926, -0.03316487]
+        self.rescale_func = np.poly1d(coefficients)
+
+    def initialize_weights(self) -> None:
+        nn.init.xavier_uniform_(self.x_embedder.weight)
+        nn.init.constant_(self.x_embedder.bias, 0.0)
+
+        nn.init.xavier_uniform_(self.ref_image_patch_embedder.weight)
+        nn.init.constant_(self.ref_image_patch_embedder.bias, 0.0)
+
+        nn.init.zeros_(self.norm_out.linear_1.weight)
+        nn.init.zeros_(self.norm_out.linear_1.bias)
+        nn.init.zeros_(self.norm_out.linear_2.weight)
+        nn.init.zeros_(self.norm_out.linear_2.bias)
+
+        nn.init.normal_(self.image_index_embedding, std=0.02)
+
+    def img_patch_embed_and_refine(
+        self,
+        hidden_states,
+        ref_image_hidden_states,
+        padded_img_mask,
+        padded_ref_img_mask,
+        noise_rotary_emb,
+        ref_img_rotary_emb,
+        l_effective_ref_img_len,
+        l_effective_img_len,
+        temb,
+    ):
+        batch_size = len(hidden_states)
+        max_combined_img_len = max([
+            img_len + sum(ref_img_len)
+            for img_len, ref_img_len in zip(l_effective_img_len, l_effective_ref_img_len)
+        ])
+
+        hidden_states = self.x_embedder(hidden_states)
+        ref_image_hidden_states = self.ref_image_patch_embedder(ref_image_hidden_states)
+
+        for i in range(batch_size):
+            shift = 0
+            for j, ref_img_len in enumerate(l_effective_ref_img_len[i]):
+                ref_image_hidden_states[i, shift:shift + ref_img_len, :] = (
+                    ref_image_hidden_states[i, shift:shift + ref_img_len, :]
+                    + self.image_index_embedding[j]
+                )
+                shift += ref_img_len
+
+        for layer in self.noise_refiner:
+            hidden_states = layer(hidden_states, padded_img_mask, noise_rotary_emb, temb)
+
+        flat_l_effective_ref_img_len = list(itertools.chain(*l_effective_ref_img_len))
+        num_ref_images = len(flat_l_effective_ref_img_len)
+        max_ref_img_len = max(flat_l_effective_ref_img_len)
+
+        batch_ref_img_mask = ref_image_hidden_states.new_zeros(num_ref_images, max_ref_img_len, dtype=torch.bool)
+        batch_ref_image_hidden_states = ref_image_hidden_states.new_zeros(
+            num_ref_images, max_ref_img_len, self.config.hidden_size
+        )
+        batch_ref_img_rotary_emb = hidden_states.new_zeros(
+            num_ref_images, max_ref_img_len, ref_img_rotary_emb.shape[-1], dtype=ref_img_rotary_emb.dtype
+        )
+        batch_temb = temb.new_zeros(num_ref_images, *temb.shape[1:], dtype=temb.dtype)
+
+        idx = 0
+        for i in range(batch_size):
+            shift = 0
+            for ref_img_len in l_effective_ref_img_len[i]:
+                batch_ref_img_mask[idx, :ref_img_len] = True
+                batch_ref_image_hidden_states[idx, :ref_img_len] = ref_image_hidden_states[i, shift:shift + ref_img_len]
+                batch_ref_img_rotary_emb[idx, :ref_img_len] = ref_img_rotary_emb[i, shift:shift + ref_img_len]
+                batch_temb[idx] = temb[i]
+                shift += ref_img_len
+                idx += 1
+
+        for layer in self.ref_image_refiner:
+            batch_ref_image_hidden_states = layer(
+                batch_ref_image_hidden_states, batch_ref_img_mask, batch_ref_img_rotary_emb, batch_temb
+            )
+
+        idx = 0
+        for i in range(batch_size):
+            shift = 0
+            for ref_img_len in l_effective_ref_img_len[i]:
+                ref_image_hidden_states[i, shift:shift + ref_img_len] = batch_ref_image_hidden_states[idx, :ref_img_len]
+                shift += ref_img_len
+                idx += 1
+
+        combined_img_hidden_states = hidden_states.new_zeros(
+            batch_size, max_combined_img_len, self.config.hidden_size
+        )
+        for i, (ref_img_len, img_len) in enumerate(zip(l_effective_ref_img_len, l_effective_img_len)):
+            combined_img_hidden_states[i, :sum(ref_img_len)] = ref_image_hidden_states[i, :sum(ref_img_len)]
+            combined_img_hidden_states[i, sum(ref_img_len):sum(ref_img_len) + img_len] = hidden_states[i, :img_len]
+
+        return combined_img_hidden_states
+
+    def flat_and_pad_to_seq(self, hidden_states, ref_image_hidden_states):
+        batch_size = len(hidden_states)
+        p = self.config.patch_size
+        device = hidden_states[0].device
+
+        img_sizes = [(img.size(1), img.size(2)) for img in hidden_states]
+        l_effective_img_len = [(H // p) * (W // p) for (H, W) in img_sizes]
+
+        if ref_image_hidden_states is not None and len(ref_image_hidden_states) > 0:
+            ref_img_sizes = [
+                [(img.size(1), img.size(2)) for img in imgs] if imgs is not None else None
+                for imgs in ref_image_hidden_states
+            ]
+            l_effective_ref_img_len = [
+                [(ref_img_size[0] // p) * (ref_img_size[1] // p) for ref_img_size in _ref_img_sizes]
+                if _ref_img_sizes is not None else [0]
+                for _ref_img_sizes in ref_img_sizes
+            ]
+        else:
+            ref_img_sizes = [None for _ in range(batch_size)]
+            l_effective_ref_img_len = [[0] for _ in range(batch_size)]
+
+        max_ref_img_len = max([sum(ref_img_len) for ref_img_len in l_effective_ref_img_len])
+        max_img_len = max(l_effective_img_len)
+
+        flat_ref_img_hidden_states = []
+        for i in range(batch_size):
+            if ref_img_sizes[i] is not None:
+                imgs = []
+                for ref_img in ref_image_hidden_states[i]:
+                    C, H, W = ref_img.size()
+                    ref_img = rearrange(ref_img, 'c (h p1) (w p2) -> (h w) (p1 p2 c)', p1=p, p2=p)
+                    imgs.append(ref_img)
+                flat_ref_img_hidden_states.append(torch.cat(imgs, dim=0))
+            else:
+                flat_ref_img_hidden_states.append(None)
+
+        flat_hidden_states = []
+        for i in range(batch_size):
+            img = hidden_states[i]
+            C, H, W = img.size()
+            img = rearrange(img, 'c (h p1) (w p2) -> (h w) (p1 p2 c)', p1=p, p2=p)
+            flat_hidden_states.append(img)
+
+        padded_ref_img_hidden_states = torch.zeros(
+            batch_size, max_ref_img_len, flat_hidden_states[0].shape[-1],
+            device=device, dtype=flat_hidden_states[0].dtype,
+        )
+        padded_ref_img_mask = torch.zeros(batch_size, max_ref_img_len, dtype=torch.bool, device=device)
+        for i in range(batch_size):
+            if ref_img_sizes[i] is not None:
+                padded_ref_img_hidden_states[i, :sum(l_effective_ref_img_len[i])] = flat_ref_img_hidden_states[i]
+                padded_ref_img_mask[i, :sum(l_effective_ref_img_len[i])] = True
+
+        padded_hidden_states = torch.zeros(
+            batch_size, max_img_len, flat_hidden_states[0].shape[-1],
+            device=device, dtype=flat_hidden_states[0].dtype,
+        )
+        padded_img_mask = torch.zeros(batch_size, max_img_len, dtype=torch.bool, device=device)
+        for i in range(batch_size):
+            padded_hidden_states[i, :l_effective_img_len[i]] = flat_hidden_states[i]
+            padded_img_mask[i, :l_effective_img_len[i]] = True
+
+        return (
+            padded_hidden_states,
+            padded_ref_img_hidden_states,
+            padded_img_mask,
+            padded_ref_img_mask,
+            l_effective_ref_img_len,
+            l_effective_img_len,
+            ref_img_sizes,
+            img_sizes,
+        )
+
+    def forward(
+        self,
+        hidden_states: Union[torch.Tensor, List[torch.Tensor]],
+        timestep: torch.Tensor,
+        text_hidden_states: torch.Tensor,
+        freqs_cis: torch.Tensor,
+        text_attention_mask: torch.Tensor,
+        ref_image_hidden_states: Optional[List[List[torch.Tensor]]] = None,
+        attention_kwargs: Optional[Dict[str, Any]] = None,
+        return_dict: bool = False,
+    ) -> Union[torch.Tensor, Transformer2DModelOutput]:
+        enable_taylorseer = getattr(self, 'enable_taylorseer', False)
+        if enable_taylorseer:
+            cal_type(self.cache_dic, self.current)
+
+        if attention_kwargs is not None:
+            attention_kwargs = attention_kwargs.copy()
+            lora_scale = attention_kwargs.pop("scale", 1.0)
+        else:
+            lora_scale = 1.0
+
+        if USE_PEFT_BACKEND:
+            scale_lora_layers(self, lora_scale)
+        else:
+            if attention_kwargs is not None and attention_kwargs.get("scale", None) is not None:
+                logger.warning(
+                    "Passing `scale` via `attention_kwargs` when not using the PEFT backend is ineffective."
+                )
+
+        batch_size = len(hidden_states)
+        is_hidden_states_tensor = isinstance(hidden_states, torch.Tensor)
+
+        if is_hidden_states_tensor:
+            assert hidden_states.ndim == 4
+            hidden_states = [_hidden_states for _hidden_states in hidden_states]
+
+        device = hidden_states[0].device
+
+        assert isinstance(text_hidden_states, torch.Tensor), \
+            f"text_hidden_states must be Tensor, got {type(text_hidden_states)}. " \
+            f"Check if freqs_cis and text_hidden_states are swapped in the caller."
+
+        temb, text_hidden_states = self.time_caption_embed(timestep, text_hidden_states, hidden_states[0].dtype)
+
+        (
+            hidden_states,
+            ref_image_hidden_states,
+            img_mask,
+            ref_img_mask,
+            l_effective_ref_img_len,
+            l_effective_img_len,
+            ref_img_sizes,
+            img_sizes,
+        ) = self.flat_and_pad_to_seq(hidden_states, ref_image_hidden_states)
+
+        (
+            context_rotary_emb,
+            ref_img_rotary_emb,
+            noise_rotary_emb,
+            rotary_emb,
+            encoder_seq_lengths,
+            seq_lengths,
+        ) = self.rope_embedder(
+            freqs_cis,
+            text_attention_mask,
+            l_effective_ref_img_len,
+            l_effective_img_len,
+            ref_img_sizes,
+            img_sizes,
+            device,
+        )
+
+        # 2. Context refinement
+        for layer in self.context_refiner:
+            text_hidden_states = layer(text_hidden_states, text_attention_mask, context_rotary_emb)
+
+        combined_img_hidden_states = self.img_patch_embed_and_refine(
+            hidden_states,
+            ref_image_hidden_states,
+            img_mask,
+            ref_img_mask,
+            noise_rotary_emb,
+            ref_img_rotary_emb,
+            l_effective_ref_img_len,
+            l_effective_img_len,
+            temb,
+        )
+
+        # 3. Joint Transformer blocks
+        max_seq_len = max(seq_lengths)
+
+        attention_mask = hidden_states.new_zeros(batch_size, max_seq_len, dtype=torch.bool)
+        joint_hidden_states = hidden_states.new_zeros(batch_size, max_seq_len, self.config.hidden_size)
+        for i, (encoder_seq_len, seq_len) in enumerate(zip(encoder_seq_lengths, seq_lengths)):
+            attention_mask[i, :seq_len] = True
+            joint_hidden_states[i, :encoder_seq_len] = text_hidden_states[i, :encoder_seq_len]
+            joint_hidden_states[i, encoder_seq_len:seq_len] = combined_img_hidden_states[i, :seq_len - encoder_seq_len]
+
+        hidden_states = joint_hidden_states
+
+        if self.enable_teacache:
+            teacache_hidden_states = hidden_states.clone()
+            teacache_temb = temb.clone()
+            modulated_inp, _, _, _ = self.layers[0].norm1(teacache_hidden_states, teacache_temb)
+            if self.teacache_params.is_first_or_last_step:
+                should_calc = True
+                self.teacache_params.accumulated_rel_l1_distance = 0
+            else:
+                self.teacache_params.accumulated_rel_l1_distance += self.rescale_func(
+                    ((modulated_inp - self.teacache_params.previous_modulated_inp).abs().mean()
+                     / self.teacache_params.previous_modulated_inp.abs().mean()).cpu().item()
+                )
+                if self.teacache_params.accumulated_rel_l1_distance < self.teacache_rel_l1_thresh:
+                    should_calc = False
+                else:
+                    should_calc = True
+                    self.teacache_params.accumulated_rel_l1_distance = 0
+            self.teacache_params.previous_modulated_inp = modulated_inp
+
+        if self.enable_teacache:
+            if not should_calc:
+                hidden_states += self.teacache_params.previous_residual
+            else:
+                ori_hidden_states = hidden_states.clone()
+                for layer_idx, layer in enumerate(self.layers):
+                    if torch.is_grad_enabled() and self.gradient_checkpointing:
+                        hidden_states = self._gradient_checkpointing_func(
+                            layer, hidden_states, attention_mask, rotary_emb, temb
+                        )
+                    else:
+                        hidden_states = layer(hidden_states, attention_mask, rotary_emb, temb)
+                self.teacache_params.previous_residual = hidden_states - ori_hidden_states
+        else:
+            if enable_taylorseer:
+                self.current['stream'] = 'layers_stream'
+
+            for layer_idx, layer in enumerate(self.layers):
+                if enable_taylorseer:
+                    layer.current = self.current
+                    layer.cache_dic = self.cache_dic
+                    layer.enable_taylorseer = True
+                    self.current['layer'] = layer_idx
+
+                if torch.is_grad_enabled() and self.gradient_checkpointing:
+                    hidden_states = self._gradient_checkpointing_func(
+                        layer, hidden_states, attention_mask, rotary_emb, temb
+                    )
+                else:
+                    hidden_states = layer(hidden_states, attention_mask, rotary_emb, temb)
+
+        hidden_states = self.norm_out(hidden_states, temb)
+
+        p = self.config.patch_size
+        output = []
+        for i, (img_size, img_len, seq_len) in enumerate(zip(img_sizes, l_effective_img_len, seq_lengths)):
+            height, width = img_size
+            output.append(rearrange(
+                hidden_states[i][seq_len - img_len:seq_len],
+                '(h w) (p1 p2 c) -> c (h p1) (w p2)',
+                h=height // p, w=width // p, p1=p, p2=p,
+            ))
+        if is_hidden_states_tensor:
+            output = torch.stack(output, dim=0)
+
+        if USE_PEFT_BACKEND:
+            unscale_lora_layers(self, lora_scale)
+
+        if enable_taylorseer:
+            self.current['step'] += 1
+
+        if not return_dict:
+            return output
+        return Transformer2DModelOutput(sample=output)
+
+
+# ---------------------------------------------------------------------------
+# FlowMatch Euler Discrete Scheduler (merged from scheduling_flow_match_euler_discrete.py)
+# ---------------------------------------------------------------------------
+
+@dataclass
+class FlowMatchEulerDiscreteSchedulerOutput(BaseOutput):
+    prev_sample: torch.FloatTensor
+
+
+class FlowMatchEulerDiscreteScheduler(SchedulerMixin, ConfigMixin):
+    _compatibles = []
+    order = 1
+
+    @register_to_config
+    def __init__(self, num_train_timesteps: int = 1000, dynamic_time_shift: bool = False):
+        timesteps = torch.linspace(0, 1, num_train_timesteps + 1, dtype=torch.float32)[:-1]
+        self.timesteps = timesteps
+        self._step_index = None
+        self._begin_index = None
+
+    @property
+    def step_index(self):
+        return self._step_index
+
+    @property
+    def begin_index(self):
+        return self._begin_index
+
+    def set_begin_index(self, begin_index: int = 0):
+        self._begin_index = begin_index
+
+    def index_for_timestep(self, timestep, schedule_timesteps=None):
+        if schedule_timesteps is None:
+            schedule_timesteps = self._timesteps
+        indices = (schedule_timesteps == timestep).nonzero()
+        pos = 1 if len(indices) > 1 else 0
+        return indices[pos].item()
+
+    def set_timesteps(self, num_inference_steps=None, device=None, timesteps=None, num_tokens=None):
+        if timesteps is None:
+            self.num_inference_steps = num_inference_steps
+            timesteps = np.linspace(0, 1, num_inference_steps + 1, dtype=np.float32)[:-1]
+            if self.config.dynamic_time_shift and num_tokens is not None:
+                m = np.sqrt(num_tokens) / 40
+                timesteps = timesteps / (m - m * timesteps + timesteps)
+        timesteps = torch.from_numpy(timesteps).to(dtype=torch.float32, device=device)
+        _timesteps = torch.cat([timesteps, torch.ones(1, device=timesteps.device)])
+        self.timesteps = timesteps
+        self._timesteps = _timesteps
+        self._step_index = None
+        self._begin_index = None
+
+    def _init_step_index(self, timestep):
+        if self.begin_index is None:
+            if isinstance(timestep, torch.Tensor):
+                timestep = timestep.to(self.timesteps.device)
+            self._step_index = self.index_for_timestep(timestep)
+        else:
+            self._step_index = self._begin_index
+
+    def step(self, model_output, timestep, sample, generator=None, return_dict=True):
+        if isinstance(timestep, (int, torch.IntTensor, torch.LongTensor)):
+            raise ValueError("Pass scheduler.timesteps values, not integer indices.")
+        if self.step_index is None:
+            self._init_step_index(timestep)
+        sample = sample.to(torch.float32)
+        t = self._timesteps[self.step_index]
+        t_next = self._timesteps[self.step_index + 1]
+        prev_sample = sample + (t_next - t) * model_output
+        prev_sample = prev_sample.to(model_output.dtype)
+        self._step_index += 1
+        if not return_dict:
+            return (prev_sample,)
+        return FlowMatchEulerDiscreteSchedulerOutput(prev_sample=prev_sample)
+
+    def __len__(self):
+        return self.config.num_train_timesteps

native/visual.py

@@ -0,0 +1,1068 @@
+from typing import Iterable, Optional, Tuple
+
+import numpy as np
+from safetensors.torch import load_file
+import torch
+import torch.utils.checkpoint
+from torch import nn
+from torch.amp import autocast
+from torch.nn import functional as F
+
+from einops import rearrange
+from .compat import flash_attn_varlen_func
+
+from transformers.activations import ACT2FN
+from transformers.modeling_outputs import BaseModelOutput
+from .qwen2_5_vl_vit import Qwen2RMSNorm, Qwen2_5_VisionTransformerPretrainedModel
+from transformers.utils import logging
+
+
+logger = logging.get_logger(__name__)
+
+
+def uniform_init(*shape):
+    t = torch.zeros(shape)
+    nn.init.kaiming_uniform_(t)
+    return t
+
+class VQEmbedding(nn.Module):
+    """VQ embedding module with ema update."""
+
+    def __init__(self, n_embed, embed_dim, ema=True, decay=0.99, restart_unused_codes=True, eps=1e-5, init_std=0.02):
+        super().__init__()
+
+        self.ema = ema
+        self.decay = decay
+        self.eps = eps
+        self.restart_unused_codes = restart_unused_codes
+        self.n_embed = n_embed
+        self.init_std = init_std
+
+        assert self.ema
+        embed = uniform_init(n_embed + 1, embed_dim).to(torch.float32)
+        self.embed = nn.Parameter(embed)
+        self.embed_ema = nn.Parameter(embed[:-1, :].clone())
+        self.cluster_size_ema = nn.Parameter(torch.ones(n_embed))
+        del embed
+        _ = [p.requires_grad_(False) for p in self.parameters()]
+
+    @torch.no_grad()
+    def compute_distances(self, inputs):
+        codebook_t = self.embed[:-1, :].t()
+
+        (embed_dim, _) = codebook_t.shape
+        inputs_shape = inputs.shape
+        assert inputs_shape[-1] == embed_dim
+
+        inputs_flat = inputs.reshape(-1, embed_dim)
+
+        inputs_norm_sq = inputs_flat.pow(2.).sum(dim=1, keepdim=True)
+        codebook_t_norm_sq = codebook_t.pow(2.).sum(dim=0, keepdim=True)
+        distances = torch.addmm(
+            inputs_norm_sq + codebook_t_norm_sq,
+            inputs_flat,
+            codebook_t,
+            alpha=-2.0,
+        )
+        distances = distances.reshape(*inputs_shape[:-1], -1)  # [B, h, w, n_embed or n_embed+1]
+        return distances
+
+    @torch.no_grad()
+    def find_nearest_embedding(self, inputs):
+        distances = self.compute_distances(inputs)  # [B, h, w, n_embed or n_embed+1]
+        embed_idxs = distances.argmin(dim=-1)  # use padding index or not
+
+        return embed_idxs
+
+    @autocast('cuda', enabled=True, dtype=torch.float32)
+    @torch.no_grad()
+    def forward(self, inputs):
+        if inputs.dtype != torch.float32:
+            inputs = inputs.to(torch.float32)
+        embed_idxs = self.find_nearest_embedding(inputs)
+        embeds = self.embed[embed_idxs]
+        return embeds, embed_idxs
+
+
+class RQBottleneck(nn.Module):
+    """
+    Quantization bottleneck via Residual Quantization.
+
+    Arguments:
+        latent_shape (Tuple[int, int, int]): the shape of latents, denoted (H, W, D)
+        code_shape (Tuple[int, int, int]): the shape of codes, denoted (h, w, d)
+        n_embed (int, List, or Tuple): the number of embeddings (i.e., the size of codebook)
+            If isinstance(n_embed, int), the sizes of all codebooks are same.
+        shared_codebook (bool): If True, codebooks are shared in all location. If False,
+            uses separate codebooks along the ``depth'' dimension. (default: False)
+        restart_unused_codes (bool): If True, it randomly assigns a feature vector in the curruent batch
+            as the new embedding of unused codes in training. (default: True)
+    """
+
+    def __init__(self,
+                 latent_shape,
+                 code_shape,
+                 n_embed,
+                 decay=0.99,
+                 shared_codebook=False,
+                 restart_unused_codes=True,
+                 commitment_loss='cumsum'
+                 ):
+        super().__init__()
+
+        if not len(code_shape) == len(latent_shape) == 3:
+            raise ValueError("incompatible code shape or latent shape")
+        if any([y % x != 0 for x, y in zip(code_shape[:2], latent_shape[:2])]):
+            raise ValueError("incompatible code shape or latent shape")
+
+        #residual quantization does not divide feature dims for quantization.
+        embed_dim = np.prod(latent_shape[:2]) // np.prod(code_shape[:2]) * latent_shape[2]
+
+        self.latent_shape = torch.Size(latent_shape)
+        self.code_shape = torch.Size(code_shape)
+        self.shape_divisor = torch.Size([latent_shape[i] // code_shape[i] for i in range(len(latent_shape))])
+
+        self.shared_codebook = shared_codebook
+        if self.shared_codebook:
+            if isinstance(n_embed, Iterable) or isinstance(decay, Iterable):
+                raise ValueError("Shared codebooks are incompatible \
+                                    with list types of momentums or sizes: Change it into int")
+
+        self.restart_unused_codes = restart_unused_codes
+        self.n_embed = n_embed if isinstance(n_embed, Iterable) else [n_embed for _ in range(self.code_shape[-1])]
+        self.decay = decay if isinstance(decay, Iterable) else [decay for _ in range(self.code_shape[-1])]
+        assert len(self.n_embed) == self.code_shape[-1]
+        assert len(self.decay) == self.code_shape[-1]
+
+        if self.shared_codebook:
+            codebook0 = VQEmbedding(self.n_embed[0],
+                                    embed_dim,
+                                    decay=self.decay[0],
+                                    restart_unused_codes=restart_unused_codes,
+                                    ).to(torch.float32)
+            self.codebooks = nn.ModuleList([codebook0 for _ in range(self.code_shape[-1])])
+        else:
+            codebooks = [VQEmbedding(self.n_embed[idx],
+                                     embed_dim,
+                                     decay=self.decay[idx],
+                                     restart_unused_codes=restart_unused_codes,
+                                     ).to(torch.float32) for idx in range(self.code_shape[-1])]
+            self.codebooks = nn.ModuleList(codebooks)
+
+        self.commitment_loss = commitment_loss
+
+    def to_code_shape(self, x):
+        (B, H, W, D) = x.shape
+        (rH, rW, _) = self.shape_divisor
+
+        x = x.reshape(B, H//rH, rH, W//rW, rW, D)
+        x = x.permute(0, 1, 3, 2, 4, 5)
+        x = x.reshape(B, H//rH, W//rW, -1)
+
+        return x
+
+    def to_latent_shape(self, x):
+        (B, h, w, _) = x.shape
+        (_, _, D) = self.latent_shape
+        (rH, rW, _) = self.shape_divisor
+
+        x = x.reshape(B, h, w, rH, rW, D)
+        x = x.permute(0, 1, 3, 2, 4, 5)
+        x = x.reshape(B, h*rH, w*rW, D)
+
+        return x
+
+    def quantize(self, x):
+        r"""
+        Return list of quantized features and the selected codewords by the residual quantization.
+        The code is selected by the residuals between x and quantized features by the previous codebooks.
+
+        Arguments:
+            x (Tensor): bottleneck feature maps to quantize.
+
+        Returns:
+            quant_list (list): list of sequentially aggregated and quantized feature maps by codebooks.
+            codes (LongTensor): codewords index, corresponding to quants.
+
+        Shape:
+            - x: (B, h, w, embed_dim)
+            - quant_list[i]: (B, h, w, embed_dim)
+            - codes: (B, h, w, d)
+        """
+        B, h, w, embed_dim = x.shape
+        ori_dtype = x.dtype
+        x = x.to(torch.float32)
+        self.codebooks = self.codebooks.to(torch.float32)
+
+        residual_feature = x.detach().clone()
+
+        quant_list = []
+        code_list = []
+        aggregated_quants = torch.zeros_like(x)
+        for i in range(self.code_shape[-1]):
+            quant, code = self.codebooks[i](residual_feature)
+            residual_feature.sub_(quant)
+            aggregated_quants.add_(quant)
+            quant_list.append(aggregated_quants.clone().to(dtype=ori_dtype))
+            code_list.append(code.unsqueeze(-1))
+
+        codes = torch.cat(code_list, dim=-1)
+        return quant_list, codes
+
+    def forward(self, x):
+        x_reshaped = self.to_code_shape(x)
+         # 强制使用float32精度来执行
+        quant_list, codes = self.quantize(x_reshaped)
+        # quant_list, codes = self.quantize(x_reshaped)
+
+        commitment_loss = self.compute_commitment_loss(x_reshaped, quant_list)
+        quants_trunc = self.to_latent_shape(quant_list[-1])
+        quants_trunc = x + (quants_trunc - x).detach()
+
+        '''
+        if self.shared_codebook:
+            cur_len = codes.view(-1).shape[0]
+            self.codebook_used[:-cur_len] = self.codebook_used[cur_len:].clone()
+            self.codebook_used[-cur_len:] = codes.view(-1)
+            codebook_usage = len(torch.unique(self.codebook_used)) / self.n_embed[0]
+        else:
+            # info|code: torch.Size([10, 16, 16, 4])
+            codebook_usage = 0
+            for idx in range(self.code_shape[-1]):
+                cur_len = codes[..., idx].view(-1).shape[0]
+                self.codebook_used[idx, :-cur_len] = self.codebook_used[idx, cur_len:].clone()
+                self.codebook_used[idx, -cur_len:] = codes[..., idx].view(-1)
+                codebook_usage += len(torch.unique(self.codebook_used[idx]))
+            codebook_usage /= (self.n_embed[0] * self.code_shape[-1])
+        '''
+        codebook_usage = 0
+        # (vq_loss, commit_loss, entropy_loss, codebook_usage) # 格式对齐
+        codebook_loss = [0, commitment_loss, 0, codebook_usage]
+
+        return quants_trunc, codebook_loss, codes
+
+    def compute_commitment_loss(self, x, quant_list):
+        r"""
+        Compute the commitment loss for the residual quantization.
+        The loss is iteratively computed by aggregating quantized features.
+        """
+        loss_list = []
+
+        for idx, quant in enumerate(quant_list):
+            partial_loss = (x-quant.detach()).pow(2.0).mean()
+            loss_list.append(partial_loss)
+
+        commitment_loss = torch.mean(torch.stack(loss_list))
+        return commitment_loss
+
+
+
+class Qwen2_5_VisionRotaryEmbedding_Modified(nn.Module):
+    def __init__(self, dim: int, theta: float = 10000.0) -> None:
+        super().__init__()
+        self.inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2, dtype=torch.float) / dim))
+        # self.register_buffer("inv_freq", inv_freq, persistent=False)
+
+    def forward(self, seqlen: int, device: torch.device) -> torch.Tensor:
+        self.inv_freq = self.inv_freq.to(device)
+        seq = torch.arange(seqlen, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
+        freqs = torch.outer(seq, self.inv_freq)
+        return freqs
+
+class VisualEncoder(Qwen2_5_VisionTransformerPretrainedModel):
+
+    def __init__(self, config):
+        config._attn_implementation = 'flash_attention_2' if torch.cuda.is_available() else 'eager'
+        super().__init__(config)
+        self.rotary_pos_emb = Qwen2_5_VisionRotaryEmbedding_Modified(config.hidden_size // config.num_heads // 2)
+        self.gradient_checkpointing = False
+        self._gradient_checkpointing_func = torch.utils.checkpoint.checkpoint
+        self.merge_size = config.merge_size if hasattr(config, 'merge_size') else 2
+        del self.merger # register visual.merger in visual_bridge_model
+
+    def get_dtype(self) -> torch.dtype:
+        return self.blocks[0].mlp.down_proj.weight.dtype
+
+    def get_device(self) -> torch.device:
+        return self.blocks[0].mlp.down_proj.weight.device
+
+    def rot_pos_emb(self, grid_thw):
+        pos_ids = []
+        for t, h, w in grid_thw:
+            hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w)
+            hpos_ids = hpos_ids.reshape(
+                h // self.spatial_merge_size,
+                self.spatial_merge_size,
+                w // self.spatial_merge_size,
+                self.spatial_merge_size,
+            )
+            hpos_ids = hpos_ids.permute(0, 2, 1, 3)
+            hpos_ids = hpos_ids.flatten()
+
+            wpos_ids = torch.arange(w).unsqueeze(0).expand(h, -1)
+            wpos_ids = wpos_ids.reshape(
+                h // self.spatial_merge_size,
+                self.spatial_merge_size,
+                w // self.spatial_merge_size,
+                self.spatial_merge_size,
+            )
+            wpos_ids = wpos_ids.permute(0, 2, 1, 3)
+            wpos_ids = wpos_ids.flatten()
+            pos_ids.append(torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1))
+        pos_ids = torch.cat(pos_ids, dim=0)
+        max_grid_size = grid_thw[:, 1:].max()
+        rotary_pos_emb_full = self.rotary_pos_emb(max_grid_size, device=grid_thw.device)
+        rotary_pos_emb = rotary_pos_emb_full[pos_ids].flatten(1)
+        return rotary_pos_emb
+
+    def forward(
+        self,
+        pixel_values: torch.Tensor,
+        grid_thw: torch.Tensor,
+        require_window_index: bool = False,
+    ):
+        '''
+        pixel_values.shape=[NumOfPatches, 1176]
+        grid_thw.shape=[NumOfSamples, 3]. [grid_t,grid_h,grid_w]
+        '''
+        hidden_states = pixel_values.to(torch.bfloat16)
+        grid_thw = grid_thw.to(pixel_values.device)
+
+        hidden_states = self.patch_embed(hidden_states)
+        rotary_pos_emb = self.rot_pos_emb(grid_thw)
+        window_index, cu_window_seqlens = self.get_window_index(grid_thw)
+        cu_window_seqlens = torch.tensor(
+            cu_window_seqlens,
+            device=hidden_states.device,
+            dtype=grid_thw.dtype if torch.jit.is_tracing() else torch.int32,
+        )
+        cu_window_seqlens = torch.unique_consecutive(cu_window_seqlens)
+
+        seq_len, _ = hidden_states.size()
+        hidden_states = hidden_states.reshape(seq_len // self.spatial_merge_unit, self.spatial_merge_unit, -1)
+        hidden_states = hidden_states[window_index, :, :]
+        hidden_states = hidden_states.reshape(seq_len, -1)
+        rotary_pos_emb = rotary_pos_emb.reshape(seq_len // self.spatial_merge_unit, self.spatial_merge_unit, -1)
+        rotary_pos_emb = rotary_pos_emb[window_index, :, :]
+        rotary_pos_emb = rotary_pos_emb.reshape(seq_len, -1)
+        emb = torch.cat((rotary_pos_emb, rotary_pos_emb), dim=-1)
+        position_embeddings = (emb.cos(), emb.sin())
+
+        cu_seqlens = torch.repeat_interleave(grid_thw[:, 1] * grid_thw[:, 2], grid_thw[:, 0]).cumsum(
+            dim=0,
+            # Select dtype based on the following factors:
+            #  - FA2 requires that cu_seqlens_q must have dtype int32
+            #  - torch.onnx.export requires that cu_seqlens_q must have same dtype as grid_thw
+            # See https://github.com/huggingface/transformers/pull/34852 for more information
+            dtype=grid_thw.dtype if torch.jit.is_tracing() else torch.int32,
+        )
+        cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0)
+
+        for layer_num, blk in enumerate(self.blocks):
+            if layer_num in self.fullatt_block_indexes:
+                cu_seqlens_now = cu_seqlens
+            else:
+                cu_seqlens_now = cu_window_seqlens
+            if self.gradient_checkpointing and self.training:
+                hidden_states = self._gradient_checkpointing_func(blk.__call__, hidden_states, cu_seqlens_now, None, position_embeddings)
+            else:
+                hidden_states = blk(
+                    hidden_states,
+                    cu_seqlens=cu_seqlens_now,
+                    position_embeddings=position_embeddings,
+                )
+
+        if require_window_index:
+            return hidden_states, window_index
+        return hidden_states
+
+
+class OmniVisualBridge(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.merge_size = self.config.merge_size if hasattr(self.config, 'merge_size') else 2
+        self.hidden_size = self.config.hidden_size * (self.merge_size**2)
+        self.window_index = self.config.window_size
+        self.ln_q = Qwen2RMSNorm(self.config.hidden_size, eps=1e-6)
+        self.mlp = nn.Sequential(
+            nn.Linear(self.hidden_size, self.hidden_size),
+            nn.GELU(),
+            nn.Linear(self.hidden_size, self.config.out_hidden_size),
+        )
+
+    def forward(self, x: torch.Tensor, window_index) -> torch.Tensor:
+        x = self.mlp(self.ln_q(x).view(-1, self.hidden_size))
+        reverse_indices = torch.argsort(window_index)
+        x = x[reverse_indices, :]
+
+        return x
+
+
+class VisualQuantizer(nn.Module):
+    def __init__(self, quantizer_config):
+        super().__init__()
+
+        self.config = quantizer_config
+        self.depth = self.config.depth
+        self.decay = self.config.decay
+        self.codebook_size = self.config.codebook_size
+        self.codebook_dim = self.config.codebook_dim
+        self.shared_codebook = self.config.shared_codebook
+        self.restart_unused_codes = self.config.restart_unused_codes
+        self.in_channels = self.config.in_channels
+
+        self.vq_loss_ratio = self.config.vq_loss_ratio
+        self.entropy_loss_ratio = self.config.entropy_loss_ratio
+        self.commit_loss_ratio = self.config.commit_loss_ratio
+
+        code_h_w = int(448 / 14)
+        latent_shape = [code_h_w, code_h_w, self.codebook_dim]
+        code_shape = [code_h_w, code_h_w, self.depth]
+
+        self.quantize = RQBottleneck(
+            latent_shape=latent_shape,
+            code_shape=code_shape,
+            n_embed=self.codebook_size,
+            decay=self.decay,
+            shared_codebook=self.shared_codebook,
+            restart_unused_codes=self.restart_unused_codes,
+        )
+
+        if self.config.quant_conv:
+            self.quant_conv = nn.Sequential(
+                nn.LayerNorm(self.in_channels),
+                nn.Linear(self.in_channels, self.in_channels),
+                nn.GELU(),
+                nn.Linear(self.in_channels, self.codebook_dim)
+            )
+        else:
+            self.quant_conv = None
+
+    def encode(self, x):
+        L, D = x.shape
+        to_qnt_feat = x.clone()
+        to_qnt_feat = to_qnt_feat.unsqueeze(0) # [L, D] -> [1, L, D]
+        N = 1
+
+        if self.quant_conv is not None:
+            to_qnt_feat = self.quant_conv(to_qnt_feat)
+
+        # quantizer needs nchw format. N,L,d -> N,1,L,d -> N,d,1,L
+        to_qnt_feat = to_qnt_feat.reshape(N, 1, L, self.codebook_dim).permute(0,3,1,2)
+        if self.config.quantizer_type == "rq":
+            to_qnt_feat = to_qnt_feat.permute(0, 2, 3, 1).contiguous() # N,d,1,L -> N,1,L,d
+            quant, emb_loss, info = self.quantize(to_qnt_feat)
+            info = info.reshape(-1, info.shape[-1]) # n,h,w,lv -> n*h*w,lv
+            info = [None, None, info]
+            quant = quant.permute(0, 3, 1, 2).contiguous() # N,1,L,d -> N,d,1,L
+        else:
+            quant, emb_loss, info = self.quantize(to_qnt_feat)
+        return quant, emb_loss, info, x.detach()
+
+    def forward(self, x):
+        quant, (vq_loss, commit_loss, entropy_loss, codebook_usage), (perplexity, min_encodings, min_encoding_indices), align_feature = \
+            self.encode(x)
+        return min_encoding_indices
+
+
+class MLP(nn.Module):
+    def __init__(
+        self,
+        hidden_size: int,
+        intermediate_size: int,
+        hidden_act: str,
+    ):
+        super().__init__()
+        self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False)
+        self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
+        self.act_fn = ACT2FN[hidden_act]
+
+    def forward(self, x):
+        return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+
+class DecoderLayer(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+        self.mlp = MLP(
+            hidden_size=self.hidden_size,
+            intermediate_size=config.visual_embedding_layer_intermediate_size,
+            hidden_act=config.visual_embedding_layer_hidden_act,
+        )
+        self.pre_layernorm = nn.LayerNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+    ):
+        residual = hidden_states
+        hidden_states = self.pre_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+
+        return hidden_states
+
+
+class VisualEmbeddingBridge(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.pre_buffer = DecoderLayer(config)
+
+    def forward(self, embeding):
+        return self.pre_buffer(embeding)
+
+
+class VisualVQBridge(nn.Module):
+    def __init__(self, visual_config):
+        super().__init__()
+        self.bridge = OmniVisualBridge(visual_config)
+        self.quantizer = VisualQuantizer(visual_config.vq_config)
+
+    def forward(
+        self,
+        visual_embed: torch.Tensor,
+        window_index: torch.Tensor,
+    ):
+        visual_embed = self.bridge(visual_embed, window_index)
+        indices = self.quantizer(visual_embed)
+        return indices
+
+
+class LongcatNextVisualTokenizer(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.visual_model = VisualEncoder(config.visual_config)
+        self.visual_bridge_model = VisualVQBridge(config.visual_config)
+        self.visual_embedding_layer = VisualEmbeddingBridge(config)
+        self.image_decoder = None
+        self._refiner_pipeline = None
+
+    @torch.no_grad()
+    def encode(self, pixel_values: torch.Tensor, visual_grid_thw: torch.Tensor):
+        visual_embed, window_index = self.visual_model(pixel_values, grid_thw=visual_grid_thw, require_window_index=True)
+        indices = self.visual_bridge_model(visual_embed, window_index)
+        return indices
+
+    @torch.no_grad()
+    def lazy_decode_and_save(self, visual_ids, tokens_h, tokens_w, save_path):
+        from .image_refiner import ImageRefinerContainer, RefinerImageProcessor, RefinerPipeline, de_transform, tensor2pil
+        from .refiner_modules import FlowMatchEulerDiscreteScheduler
+        device = next(self.parameters()).device
+        if self.image_decoder is None:
+            print("lazy load image_decoder / image_refiner / _refiner_pipeline ...")
+            vdc = self.config.visual_config.visual_decoder_config
+            self.image_decoder = VisionTransformerDecoder.from_pretrained(
+                vdc.image_decoder_config,
+                vdc.weight_path,
+            ).to(device=device, dtype=torch.bfloat16)
+            image_refiner = ImageRefinerContainer.from_pretrained(vdc, vdc.weight_path).to(device=device, dtype=torch.bfloat16)
+
+            sc = vdc.scheduler_config
+            scheduler = FlowMatchEulerDiscreteScheduler(
+                num_train_timesteps=sc.num_train_timesteps,
+                dynamic_time_shift=sc.dynamic_time_shift)
+            self._refiner_pipeline = RefinerPipeline(
+                vae=image_refiner.vae,
+                transformer=image_refiner.base_transformer,
+                scheduler=scheduler,
+                cond_proj=image_refiner.cond_proj,
+            )
+            self._refiner_pipeline.set_progress_bar_config(disable=False)
+
+        data = torch.as_tensor(visual_ids, dtype=torch.long)
+        if data.ndim == 1:
+            data = data.view(-1, len(self.config.visual_config.vq_config.codebook_sizes))
+        if data.ndim == 2:
+            data = data.unsqueeze(0)
+        batch_size = data.shape[0]
+
+        quant_features = None
+        for idx in range(len(self.config.visual_config.vq_config.codebook_sizes)):
+            embed = self.visual_bridge_model.quantizer.quantize.codebooks[idx].embed
+            feat = embed[data[..., idx].to(embed.device)]
+            quant_features = feat if quant_features is None else quant_features + feat
+        quant_features = quant_features.to(device)
+
+        # tokens_h/tokens_w are the merged grid; expand to the full (unmerged) grid
+        s = self.image_decoder.spatial_merge_size
+        grid_thw_list = [(1, tokens_h * s, tokens_w * s)]
+        grid_thw_batch = list(grid_thw_list) * batch_size
+
+        image_mean = [0.48145466, 0.4578275, 0.40821073]
+        image_std = [0.26862954, 0.26130258, 0.27577711]
+
+        emb_2d = quant_features.reshape(-1, quant_features.shape[-1]).contiguous()
+        device_type = "cuda" if str(device).startswith("cuda") else str(device)
+        with torch.amp.autocast(device_type=device_type, enabled=True, dtype=torch.float32):
+            decoder_out = self.image_decoder(emb_2d, grid_thw_batch, return_pixel_features=False)
+
+        decoded_tensors = decoder_out.get("images") or []
+        decoded_images = [tensor2pil(t, image_mean, image_std) for t in decoded_tensors]
+        decoded_path = save_path.replace(".png", "_decoded.png")
+        # decoded_images[0].save(decoded_path)
+
+
+        ref_input = []
+        for t in decoded_tensors:
+            img_01 = de_transform(t, mean=image_mean, std=image_std, rescale_factor=1 / 255)
+            img_norm = RefinerImageProcessor.normalize(img_01)
+            ref_input.append(img_norm.squeeze(0).to(device))
+
+        generators = [torch.Generator(device=device).manual_seed(42 + b) for b in range(batch_size)]
+        out = self._refiner_pipeline(
+            encoder_hidden_states=quant_features,
+            grid_thw_list=grid_thw_list,
+            image=ref_input,
+            generator=generators[0] if batch_size == 1 else generators,
+            output_type="pil",
+            return_dict=True,
+        )
+        refined_images = out.images
+        refined_path = save_path.replace(".png", "_refined.png")
+        refined_images[0].save(refined_path)
+
+        return [refined_path]
+
+
+# ---------------------------------------------------------------------------
+# Vision Transformer Decoder
+# ---------------------------------------------------------------------------
+
+def _rotate_half(x):
+    x = rearrange(x, "... (d r) -> ... d r", r=2)
+    x1, x2 = x.unbind(dim=-1)
+    x = torch.stack((-x2, x1), dim=-1)
+    return rearrange(x, "... d r -> ... (d r)")
+
+
+class VisionRoPE2D(nn.Module):
+    """2D Rotary Position Embedding for Q/K in vision decoder attention."""
+
+    def __init__(self, theta: float = 10000.0):
+        super().__init__()
+        self.theta = theta
+
+    def _rope_half(self, x_half, pos_1d, theta):
+        BH, T, d_half = x_half.shape
+        idx = torch.arange(0, d_half, 2, device=x_half.device, dtype=torch.float32)
+        inv_freq = (1.0 / (theta ** (idx / d_half))).to(x_half.dtype)
+        angles = pos_1d.to(x_half.dtype)[:, None] * inv_freq[None, :]
+        cos = torch.repeat_interleave(torch.cos(angles), 2, dim=-1).unsqueeze(0)
+        sin = torch.repeat_interleave(torch.sin(angles), 2, dim=-1).unsqueeze(0)
+        return x_half * cos + _rotate_half(x_half) * sin
+
+    def forward(self, x, positions_2d):
+        d_half = x.shape[-1] // 2
+        x_y = self._rope_half(x[:, :, :d_half], positions_2d[:, 0], self.theta)
+        x_x = self._rope_half(x[:, :, d_half:], positions_2d[:, 1], self.theta)
+        return torch.cat([x_y, x_x], dim=-1)
+
+
+class VisionAttention(nn.Module):
+    """Multi-headed attention with 2D RoPE + FlashAttention varlen."""
+
+    def __init__(self, config, rope=None, rope_shift=0):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = self.embed_dim // self.num_heads
+        if self.head_dim * self.num_heads != self.embed_dim:
+            raise ValueError(
+                f"embed_dim must be divisible by num_heads (got embed_dim={self.embed_dim}, num_heads={self.num_heads})"
+            )
+        self.scale = self.head_dim ** -0.5
+        self.dropout = config.attention_dropout
+        self.subln = config.subln
+        self.k_proj = nn.Linear(self.embed_dim, self.embed_dim, bias=getattr(config, "k_bias", True))
+        self.v_proj = nn.Linear(self.embed_dim, self.embed_dim, bias=getattr(config, "v_bias", True))
+        self.q_proj = nn.Linear(self.embed_dim, self.embed_dim, bias=getattr(config, "q_bias", True))
+        self.inner_attn_ln = Qwen2RMSNorm(self.embed_dim, eps=config.layer_norm_eps) if config.subln else nn.Identity()
+        self.out_proj = nn.Linear(self.embed_dim, self.embed_dim, bias=True)
+        self.rope = rope
+        self.rope_shift = int(rope_shift)
+
+    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):
+        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()
+
+    def _maybe_flash_attention(self, query_states, key_states, value_states, seq_lens, training):
+        if not (query_states.is_cuda and (query_states.dtype in (torch.float16, torch.bfloat16, torch.float32))):
+            return None
+        if seq_lens is None:
+            return None
+        try:
+            BxH, T, hd = query_states.shape
+            H = self.num_heads
+            assert BxH % H == 0
+            B = BxH // H
+            if int(seq_lens.sum().item()) != T:
+                return None
+            q = query_states.view(B, H, T, hd).transpose(1, 2).reshape(-1, H, hd).contiguous()
+            k = key_states.view(B, H, T, hd).transpose(1, 2).reshape(-1, H, hd).contiguous()
+            v = value_states.view(B, H, T, hd).transpose(1, 2).reshape(-1, H, hd).contiguous()
+            cu_q = torch.zeros(seq_lens.numel() + 1, dtype=torch.int32, device=seq_lens.device)
+            cu_q[1:] = torch.cumsum(seq_lens.to(torch.int32), dim=0)
+            cu_k = cu_q
+            max_seqlen = int(seq_lens.max().item())
+            orig_dtype = q.dtype
+            use_dtype = q.dtype if q.dtype in (torch.float16, torch.bfloat16) else torch.float16
+            if q.dtype != use_dtype:
+                q = q.to(use_dtype)
+                k = k.to(use_dtype)
+                v = v.to(use_dtype)
+            out = flash_attn_varlen_func(
+                q, k, v, cu_q, cu_k, max_seqlen, max_seqlen,
+                dropout_p=self.dropout if training else 0.0,
+                softmax_scale=None, causal=False, return_attn_probs=False
+            )
+            if out.dtype != orig_dtype:
+                out = out.to(orig_dtype)
+            return out.view(B, -1, H, hd).transpose(1, 2).contiguous().view(B * H, T, hd)
+        except Exception:
+            return None
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        causal_attention_mask: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = False,
+        positions_2d: Optional[torch.Tensor] = None,
+        seq_lens: Optional[torch.Tensor] = None,
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
+        bsz, tgt_len, embed_dim = hidden_states.size()
+        query_states = self.q_proj(hidden_states) * self.scale
+        key_states = self.k_proj(hidden_states)
+        value_states = self.v_proj(hidden_states)
+        query_states = self._shape(query_states, tgt_len, bsz).view(bsz * self.num_heads, tgt_len, self.head_dim)
+        key_states = self._shape(key_states, tgt_len, bsz).view(bsz * self.num_heads, tgt_len, self.head_dim)
+        value_states = self._shape(value_states, tgt_len, bsz).view(bsz * self.num_heads, tgt_len, self.head_dim)
+        if self.rope is not None and positions_2d is not None:
+            if self.rope_shift > 0:
+                q_pref = query_states[:, :self.rope_shift, :]
+                k_pref = key_states[:, :self.rope_shift, :]
+                q_rot = self.rope(query_states[:, self.rope_shift:, :], positions_2d[self.rope_shift:])
+                k_rot = self.rope(key_states[:, self.rope_shift:, :], positions_2d[self.rope_shift:])
+                query_states = torch.cat([q_pref, q_rot], dim=1).type_as(value_states)
+                key_states = torch.cat([k_pref, k_rot], dim=1).type_as(value_states)
+            else:
+                query_states = self.rope(query_states, positions_2d).type_as(value_states)
+                key_states = self.rope(key_states, positions_2d).type_as(value_states)
+        attn_output = self._maybe_flash_attention(
+            query_states, key_states, value_states, seq_lens=seq_lens, training=self.training
+        )
+        if attn_output is not None:
+            attn_weights_reshaped = None
+        else:
+            src_len = key_states.size(1)
+            attn_weights = torch.bmm(query_states, key_states.transpose(1, 2))
+            if causal_attention_mask is not None:
+                attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + causal_attention_mask
+                attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
+            if attention_mask is not None:
+                attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask
+                attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
+            attn_weights = nn.functional.softmax(attn_weights, dim=-1)
+            if output_attentions:
+                attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)
+            else:
+                attn_weights_reshaped = None
+            attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training)
+            attn_output = torch.bmm(attn_probs, value_states)
+        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)
+        attn_output = attn_output.transpose(1, 2).reshape(bsz, tgt_len, embed_dim)
+        attn_output = self.inner_attn_ln(attn_output)
+        attn_output = self.out_proj(attn_output)
+        return attn_output, attn_weights_reshaped
+
+
+class VisionSwiGLU(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size
+        self.intermediate_size = config.intermediate_size
+        self.w1 = nn.Linear(self.hidden_size, self.intermediate_size)
+        self.w2 = nn.Linear(self.hidden_size, self.intermediate_size)
+        self.w3 = nn.Linear(self.intermediate_size, self.hidden_size)
+        self.act_fn = nn.SiLU()
+        self.ffn_ln = Qwen2RMSNorm(self.intermediate_size, eps=config.layer_norm_eps) if config.subln else nn.Identity()
+
+    def forward(self, x):
+        x1 = self.w1(x)
+        x2 = self.w2(x)
+        hidden = self.act_fn(x1) * x2
+        x = self.ffn_ln(hidden)
+        x = self.w3(x)
+        return x
+
+
+class VisionMLP(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.activation_fn = ACT2FN[config.hidden_act]
+        self.fc1 = nn.Linear(config.hidden_size, config.intermediate_size)
+        self.fc2 = nn.Linear(config.intermediate_size, config.hidden_size)
+        self.ffn_ln = Qwen2RMSNorm(config.intermediate_size, eps=config.layer_norm_eps) if config.subln else nn.Identity()
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = self.fc1(hidden_states)
+        hidden_states = self.activation_fn(hidden_states)
+        hidden_states = self.ffn_ln(hidden_states)
+        hidden_states = self.fc2(hidden_states)
+        return hidden_states
+
+
+class VisionEncoderLayer(nn.Module):
+    def __init__(self, config, rope=None, rope_shift=0):
+        super().__init__()
+        self.embed_dim = config.hidden_size
+        self.self_attn = VisionAttention(config, rope=rope, rope_shift=rope_shift)
+        self.layer_norm1 = Qwen2RMSNorm(self.embed_dim, eps=config.layer_norm_eps)
+        self.mlp = VisionSwiGLU(config) if config.swiglu else VisionMLP(config)
+        self.layer_norm2 = Qwen2RMSNorm(self.embed_dim, eps=config.layer_norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor],
+        causal_attention_mask: Optional[torch.Tensor],
+        output_attentions: Optional[bool] = False,
+        positions_2d: Optional[torch.Tensor] = None,
+        seq_lens: Optional[torch.Tensor] = None,
+    ) -> Tuple[torch.FloatTensor, Optional[torch.Tensor]]:
+        residual = hidden_states
+        hidden_states = self.layer_norm1(hidden_states)
+        hidden_states, attn_weights = self.self_attn(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            causal_attention_mask=causal_attention_mask,
+            output_attentions=output_attentions,
+            positions_2d=positions_2d,
+            seq_lens=seq_lens,
+        )
+        hidden_states = residual + hidden_states
+        residual = hidden_states
+        hidden_states = self.layer_norm2(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+        outputs = (hidden_states,)
+        if output_attentions:
+            outputs += (attn_weights,)
+        return outputs
+
+
+class VisionEncoder(nn.Module):
+    def __init__(self, config, rope=None, rope_shift=0):
+        super().__init__()
+        self.config = config
+        self.layers = nn.ModuleList(
+            [VisionEncoderLayer(config, rope=rope, rope_shift=rope_shift) for _ in range(config.num_hidden_layers)]
+        )
+        self.gradient_checkpointing = False
+        self._gradient_checkpointing_func = torch.utils.checkpoint.checkpoint
+
+    def forward(
+        self,
+        inputs_embeds: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        causal_attention_mask: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        positions_2d: Optional[torch.Tensor] = None,
+        seq_lens: Optional[torch.Tensor] = None,
+    ):
+        output_attentions = output_attentions if output_attentions is not None else False
+        output_hidden_states = output_hidden_states if output_hidden_states is not None else False
+        return_dict = True if return_dict is None else return_dict
+
+        encoder_states = () if output_hidden_states else None
+        all_attentions = () if output_attentions else None
+        hidden_states = inputs_embeds
+
+        for layer in self.layers:
+            if output_hidden_states:
+                encoder_states = encoder_states + (hidden_states,)
+            if self.gradient_checkpointing and self.training:
+                def custom_forward(hs, attn, causal, pos2d, seqlens):
+                    return layer(
+                        hs,
+                        attention_mask=attn,
+                        causal_attention_mask=causal,
+                        output_attentions=False,
+                        positions_2d=pos2d,
+                        seq_lens=seqlens,
+                    )[0]
+                hidden_states = self._gradient_checkpointing_func(
+                    custom_forward,
+                    hidden_states,
+                    attention_mask if attention_mask is not None else torch.tensor(0., device=hidden_states.device),
+                    causal_attention_mask if causal_attention_mask is not None else torch.tensor(0., device=hidden_states.device),
+                    positions_2d,
+                    seq_lens if seq_lens is not None else torch.tensor([], device=hidden_states.device),
+                    use_reentrant=False,
+                )
+            else:
+                layer_outputs = layer(
+                    hidden_states,
+                    attention_mask,
+                    causal_attention_mask,
+                    output_attentions=output_attentions,
+                    positions_2d=positions_2d,
+                    seq_lens=seq_lens,
+                )
+                hidden_states = layer_outputs[0]
+                if output_attentions:
+                    all_attentions = all_attentions + (layer_outputs[1],)
+
+        if output_hidden_states:
+            encoder_states = encoder_states + (hidden_states,)
+
+        if not return_dict:
+            return tuple(v for v in [hidden_states, encoder_states, all_attentions] if v is not None)
+
+        return BaseModelOutput(
+            last_hidden_state=hidden_states,
+            hidden_states=encoder_states,
+            attentions=all_attentions,
+        )
+
+
+class PatchUnMerger(nn.Module):
+    """Learnable inverse of Qwen2_5_VLPatchMerger."""
+    def __init__(self, dim, context_dim, spatial_merge_size=2):
+        super().__init__()
+        self.spatial_merge_size = spatial_merge_size
+        self.context_dim = context_dim
+        hidden = context_dim * (spatial_merge_size ** 2)
+        self.ln_q = Qwen2RMSNorm(dim, eps=1e-6)
+        self.mlp = nn.Sequential(nn.Linear(dim, hidden), nn.GELU(), nn.Linear(hidden, hidden))
+
+    def forward(self, x):
+        x = self.mlp(self.ln_q(x))
+        return x.view(x.shape[0] * (self.spatial_merge_size ** 2), self.context_dim)
+
+
+def restore_spatial_structure_and_convert_to_images(patches, grid_thw_list, patch_size,
+                                                     channel_dim=3, temporal_patch_size=2, merge_size=2):
+    """Convert decoder pixel features back to image tensors [3, H, W]."""
+    if isinstance(patches, tuple):
+        patches = patches[0]
+    image_tensors = []
+    ptr = 0
+    for grid in grid_thw_list:
+        gt, gh, gw = (int(x) for x in (grid if not isinstance(grid, torch.Tensor) else grid.tolist()))
+        n = gt * gh * gw
+        chunk = patches[ptr:ptr + n]
+        ptr += n
+        r = chunk.reshape(gt, gh // merge_size, gw // merge_size, merge_size, merge_size,
+                          channel_dim, temporal_patch_size, patch_size, patch_size)
+        r = r.permute(0, 6, 5, 1, 3, 7, 2, 4, 8)
+        image_tensors.append(r.reshape(gt * temporal_patch_size, channel_dim, gh * patch_size, gw * patch_size)[0])
+    return image_tensors
+
+
+class VisionTransformerDecoder(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.patch_size = config.patch_size
+        self.spatial_merge_size = config.spatial_merge_size
+        self.codebook_dim = config.codebook_dim
+        self.temporal_patch_size = config.temporal_patch_size
+
+        self.rope2d = VisionRoPE2D(theta=10000.0)
+        self.post_quant_conv = nn.Linear(self.codebook_dim, self.embed_dim)
+        self.post_quant_norm = Qwen2RMSNorm(self.embed_dim, eps=config.layer_norm_eps)
+        self.patch_unmerger = PatchUnMerger(self.embed_dim, self.embed_dim, self.spatial_merge_size)
+        self.norm_in = Qwen2RMSNorm(self.embed_dim, eps=config.layer_norm_eps)
+        self.encoder = VisionEncoder(config, rope=self.rope2d, rope_shift=0)
+        self.norm_out = Qwen2RMSNorm(self.embed_dim, eps=config.layer_norm_eps)
+        self.decoder_head = nn.Sequential(
+            nn.Linear(self.embed_dim, config.intermediate_size), nn.GELU(),
+            nn.Linear(config.intermediate_size, 3 * self.patch_size * self.patch_size * self.temporal_patch_size),
+        )
+
+    @classmethod
+    def from_pretrained(cls, config, model_path: str):
+        """Load a pretrained model from a checkpoint."""
+        model = cls(config)
+        weight_dict = load_file(model_path, device="cpu")
+        model.load_state_dict({k.removeprefix("image_decoder."): v for k, v in weight_dict.items() if k.startswith("image_decoder.")}, strict=True)
+        model.eval()
+        return model
+
+    def _build_2d_positions(self, grid_thw_list):
+        pos_list = []
+        for (t, gh, gw) in grid_thw_list:
+            for _ in range(int(t)):
+                for y in range(int(gh)):
+                    for x in range(int(gw)):
+                        pos_list.append([y, x])
+        return torch.tensor(pos_list, dtype=torch.long)
+
+    def _build_attention_mask(self, grid_thw_list, device, dtype, B, num_heads):
+        counts = [int(t) * int(h) * int(w) for (t, h, w) in grid_thw_list]
+        L = sum(counts)
+        mask = torch.zeros((B, num_heads, L, L), device=device, dtype=dtype)
+        s = 0
+        for c in counts:
+            e = s + c
+            if s > 0:
+                mask[:, :, s:e, :s] = float("-inf")
+            if e < L:
+                mask[:, :, s:e, e:] = float("-inf")
+            s = e
+        return mask
+
+    def forward(self, embeddings, grid_thw, return_pixel_features=False, return_last_latent=False):
+        device = embeddings.device
+        grid_thw_list = ([(int(t), int(h), int(w)) for t, h, w in grid_thw.detach().cpu().numpy()]
+                         if isinstance(grid_thw, torch.Tensor) else list(grid_thw))
+
+        if embeddings.shape[-1] == self.codebook_dim:
+            embeddings = self.post_quant_conv(embeddings)
+            embeddings = self.post_quant_norm(embeddings)
+
+        unmerged = self.patch_unmerger(embeddings)
+        if unmerged.dim() == 2:
+            unmerged = unmerged.unsqueeze(0)
+        B, L, D = unmerged.shape
+        hidden_states = self.norm_in(unmerged)
+
+        positions_2d = self._build_2d_positions(grid_thw_list).to(device)
+        seq_lens = torch.tensor([int(t) * int(h) * int(w) for (t, h, w) in grid_thw_list],
+                                device=device, dtype=torch.int32)
+        assert positions_2d.shape[0] == L, f"positions_2d {positions_2d.shape[0]} != L {L}"
+
+        last_latent = hidden_states.detach().squeeze(0) if return_last_latent else None
+        enc_out = self.encoder(
+            inputs_embeds=hidden_states,
+            attention_mask=None,
+            causal_attention_mask=None,
+            output_attentions=False,
+            output_hidden_states=False,
+            return_dict=True,
+            positions_2d=positions_2d,
+            seq_lens=seq_lens,
+        )
+        hidden_states = enc_out.last_hidden_state
+
+        hidden_states = self.norm_out(hidden_states)
+        pixel_features = self.decoder_head(hidden_states).squeeze(0)
+
+        out_imgs = (None if return_pixel_features else
+                    restore_spatial_structure_and_convert_to_images(
+                        pixel_features, grid_thw_list, self.patch_size,
+                        temporal_patch_size=self.temporal_patch_size, merge_size=self.spatial_merge_size))
+        ret = {"images": out_imgs, "pixel_features": pixel_features}
+        if last_latent is not None:
+            ret["last_latent"] = last_latent
+        return ret

refiner/alias_verification.json

@@ -0,0 +1 @@
+{}

refiner/aliases.json

@@ -0,0 +1 @@
+{}

refiner/extraction.json

@@ -0,0 +1,11 @@
+{
+  "source_repo": "meituan-longcat/LongCat-Next",
+  "source_revision": "0cf0631862402ff36366e513e4023d22e7e5c84c",
+  "source_tensor_entries": 582,
+  "stored_tensor_entries": 582,
+  "tensor_bytes": 7934322800,
+  "source_tensor_bytes_equal": true,
+  "aliases_verified": 0,
+  "sha256": "fb94553b718c6686170f7eb73d97f59d8f3e49f3e7d2436539ad424efaef1f28",
+  "transform": "Selection only; names, shapes, dtype and payload bytes unchanged."
+}

refiner/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb94553b718c6686170f7eb73d97f59d8f3e49f3e7d2436539ad424efaef1f28
+size 7934402936

refiner/source_config.json

@@ -0,0 +1,286 @@
+{
+  "architectures": [
+    "LongcatNextForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_longcat_next.LongcatNextConfig",
+    "AutoModel": "modeling_longcat_next.LongcatNextModel",
+    "AutoModelForCausalLM": "modeling_longcat_next.LongcatNextForCausalLM"
+  },
+  "vocab_size": 282624,
+  "hidden_size": 3072,
+  "ffn_hidden_size": 6144,
+  "expert_ffn_hidden_size": 1024,
+  "num_layers": 14,
+  "num_attention_heads": 32,
+  "kv_lora_rank": 512,
+  "q_lora_rank": 1536,
+  "qk_rope_head_dim": 64,
+  "v_head_dim": 128,
+  "qk_nope_head_dim": 128,
+  "mla_scale_q_lora": true,
+  "mla_scale_kv_lora": true,
+  "routed_scaling_factor": 6.0,
+  "n_routed_experts": 256,
+  "rms_norm_eps": 1e-5,
+  "use_cache": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "rope_theta": 10000000,
+  "max_position_embeddings": 131072,
+  "zero_expert_num": 128,
+  "zero_expert_type": "identity",
+  "moe_topk": 12,
+  "ngram_vocab_size_ratio": 78,
+  "emb_neighbor_num": 4,
+  "emb_split_num": 4,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.57.6",
+  "model_type": "longcat_next",
+
+  "text_vocab_size": 131072,
+  "text_vocab_plus_multimodal_special_token_size": 131125,
+  "visual_embedding_layer_intermediate_size": 8192,
+  "visual_embedding_layer_hidden_act": "silu",
+  "visual_offset": 150581,
+  "audio_offset": 131125,
+
+
+  "visual_config": {
+    "image_start_token_id": 131106,
+    "image_end_token_id": 131107,
+    "image_pad_token_id": 131108,
+    "image_newline_token_id": 131109,
+
+    "_attn_implementation": "flash_attention_2",
+    "hidden_size": 1280,
+
+    "image_head_transformer_dims": 2048,
+    "image_head_transformer_ffn_scale": 16,
+    "image_head_transformer_layers": 4,
+
+    "vq_config": {
+      "codebook_dim": 3584,
+      "codebook_size": 16384,
+      "codebook_sizes": [
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384
+      ],
+      "decay": 0.99,
+      "depth": 8,
+
+      "commit_loss_ratio": 0.25,
+      "entropy_loss_ratio": 0,
+
+      "in_channels": 3584,
+      "quant_conv": true,
+      "quantizer_type": "rq",
+      "restart_unused_codes": true,
+      "shared_codebook": true,
+
+      "vq_loss_ratio": 0
+    },
+
+    "visual_decoder_config": {
+      "codebook_dim": 3584,
+
+      "image_decoder_config": {
+        "attention_dropout": 0.0,
+        "codebook_dim": 3584,
+        "distill_taps": [
+          3,
+          7,
+          15,
+          23
+        ],
+        "hidden_act": "gelu",
+        "hidden_size": 1024,
+        "intermediate_size": 2730,
+        "k_bias": false,
+        "layer_norm_eps": 1e-06,
+        "num_attention_heads": 16,
+        "num_hidden_layers": 32,
+        "patch_size": 14,
+        "q_bias": true,
+        "spatial_merge_size": 2,
+        "subln": true,
+        "swiglu": true,
+        "teacher_dims": {
+          "15": 1280,
+          "23": 1280,
+          "3": 1280,
+          "7": 1280
+        },
+        "temporal_patch_size": 2,
+        "v_bias": true
+      },
+
+      "transformer_config": {
+        "patch_size": 2,
+        "in_channels": 16,
+        "hidden_size": 2520,
+        "num_layers": 32,
+        "num_refiner_layers": 2,
+        "num_attention_heads": 21,
+        "num_kv_heads": 7,
+        "multiple_of": 256,
+        "norm_eps": 1e-5,
+        "axes_dim_rope": [40, 40, 40],
+        "axes_lens": [10000, 10000, 10000],
+        "text_feat_dim": 2048,
+        "timestep_scale": 1000.0
+      },
+
+      "vae_config": {
+        "act_fn": "silu",
+        "block_out_channels": [128, 256, 512, 512],
+        "down_block_types": [
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D"
+        ],
+        "in_channels": 3,
+        "latent_channels": 16,
+        "layers_per_block": 2,
+        "mid_block_add_attention": true,
+        "norm_num_groups": 32,
+        "out_channels": 3,
+        "sample_size": 1024,
+        "scaling_factor": 0.3611,
+        "shift_factor": 0.1159,
+        "up_block_types": [
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D"
+        ],
+        "use_post_quant_conv": false,
+        "use_quant_conv": false,
+        "force_upcast": true
+      },
+
+      "scheduler_config": {
+        "num_train_timesteps": 1000,
+        "dynamic_time_shift": true
+      },
+
+      "weight_path": "WEIGHT_PATH_TO_LONGCAT_NEXT/image_decoder/image_decoder.safetensors"
+    }
+  },
+
+
+  "audio_config": {
+    "audio_head_transformer_dims": 3072,
+    "audio_head_transformer_ffn_scale": 16,
+    "audio_head_transformer_layers": 4,
+
+    "audio_delim_token_id": 131116,
+    "audio_end_token_id": 131104,
+    "audio_pad_token_id": 131105,
+    "audio_start_token_id": 131103,
+    "audiogen_end_token_id": 131124,
+    "audiogen_start_token_id": 131123,
+    "audiotext_end_token_id": 131121,
+    "audiotext_pad_token_id": 131122,
+    "audiotext_start_token_id": 131120,
+
+    "_attn_implementation": "flash_attention_2",
+    "d_model": 1280,
+    "decoder_attention_heads": 20,
+    "decoder_ffn_dim": 5120,
+    "decoder_layers": 8,
+    "encoder_attention_heads": 20,
+    "encoder_ffn_dim": 5120,
+    "encoder_layers": 32,
+    "num_mel_bins": 128,
+
+    "avg_pooler": 4,
+    "decoder_kernel_size": 3,
+    "decoder_stride_size": 2,
+    "hop_length": 160,
+    "kernel_size": 3,
+    "max_audio_seconds": 30,
+    "n_fft": 400,
+    "num_hidden_layers": 32,
+    "sampling_rate": 16000,
+    "stride_size": 2,
+
+    "vq_config": {
+      "codebook_sizes": [
+        8192,
+        4096,
+        2048,
+        1024,
+        1024,
+        1024,
+        1024,
+        1024
+      ]
+    },
+
+    "vocoder_config": {
+      "channels": [
+        256,
+        256,
+        256,
+        256,
+        256
+      ],
+      "hop_length": 256,
+      "num_mel_bins": 80,
+      "sampling_rate": 16000
+    },
+
+
+    "flow_matching_config": {
+      "in_channels": 80,
+      "spk_emb_dim": 0,
+      "diffusion_steps": 10,
+      "cal_mel_mae": true,
+
+      "prenet_activation_function": "gelu",
+      "prenet_attention_heads": 8,
+      "prenet_d_model": 512,
+      "prenet_ffn_dim": 2048,
+      "prenet_in_dim": 1280,
+      "prenet_max_source_positions": 5000,
+      "prenet_nlayers": 12,
+      "prenet_out_dim": 80,
+      "prenet_target_mel_length_scale_ratio": 1.0,
+
+      "channels": [
+        256
+      ],
+      "dropout": 0.0,
+      "attention_head_dim": 64,
+      "n_blocks": 4,
+      "num_heads": 8,
+      "num_mid_blocks": 12,
+      "act_fn": "gelu",
+
+      "cfm_params": {
+        "inference_cfg_rate": 0.7,
+        "sigma_min": 1e-06,
+        "solver": "euler",
+        "t_scheduler": "cosine",
+        "training_cfg_rate": 0.2
+      },
+
+      "use_hidden_states_before_dconv2": true
+    },
+
+    "cosy24kvocoder_config": {
+      "weight_path": "WEIGHT_PATH_TO_LONGCAT_NEXT/cosy24k_vocoder/hift.pt"
+    }
+
+  }
+}

refiner/tensor_sha256.json

@@ -0,0 +1,584 @@
+{
+  "image_refiner.base_transformer.context_refiner.0.attn.norm_k.weight": "24e7c32a2ea7472ab1edbfebd5150c21af4f1b6516faf7e5b4c6330d9e4f808d",
+  "image_refiner.base_transformer.context_refiner.0.attn.norm_q.weight": "d5b63c4f1fdc9cb6a82d5544c3bc33b45c5d4b6030eb64b7d326f2a8808e1bc0",
+  "image_refiner.base_transformer.context_refiner.0.attn.to_k.weight": "e33668388ea8b63133f6cb8a5d2842a0dee7437db76ed19725b7057a0fb5d6a4",
+  "image_refiner.base_transformer.context_refiner.0.attn.to_out.0.weight": "60bebca82c37af9323d7f1fcd021c8eaadf222747f071da7c43efee021988556",
+  "image_refiner.base_transformer.context_refiner.0.attn.to_q.weight": "288724bf08ab1fa77f602d27fce10420d011c636ff0ff717f91debd6d50a49c8",
+  "image_refiner.base_transformer.context_refiner.0.attn.to_v.weight": "66190cff5d37f5a6c5217f75fa1a0391ee2daa4ec288d1f7f72044b9f6d5b799",
+  "image_refiner.base_transformer.context_refiner.0.feed_forward.linear_1.weight": "acf736f044ee2af739926e5a9a3f995b068784f8aec7d5add0a8e40cdbe0a888",
+  "image_refiner.base_transformer.context_refiner.0.feed_forward.linear_2.weight": "eae1d44166dc6ada550011739346160aa3c8052ba090c0bca1e24af683b33378",
+  "image_refiner.base_transformer.context_refiner.0.feed_forward.linear_3.weight": "7d77f4d007cfca25c36b3f5e5ed5fa0812a02034db6a711cc3d3d88916486be8",
+  "image_refiner.base_transformer.context_refiner.0.ffn_norm1.weight": "4ca2be2b63a6e4b7019419c583bb65c57cfec5a57a70673dc3b728f3ce6ebc71",
+  "image_refiner.base_transformer.context_refiner.0.ffn_norm2.weight": "f7a069ba84d9c15f5e831c6af8de8cb4847cbea9bbaed59446701d7dd2bd7485",
+  "image_refiner.base_transformer.context_refiner.0.norm1.weight": "c7f9e1c172ad7981c47264f509078dff2f3b72fcd74353507d53b6b0b574a07a",
+  "image_refiner.base_transformer.context_refiner.0.norm2.weight": "5430c1192ba4509100f93ea0da2d771199170c2dad6a905991eaddeedafe293b",
+  "image_refiner.base_transformer.context_refiner.1.attn.norm_k.weight": "54a32b28fa73a179a40d2c05f20e5564a8281e35bb622fbae74cfad41d56b06a",
+  "image_refiner.base_transformer.context_refiner.1.attn.norm_q.weight": "d2b4ed741234e1f14350a8f99a4a00d6de2f44939dc94f7c19aa4319dcac2562",
+  "image_refiner.base_transformer.context_refiner.1.attn.to_k.weight": "37c0597c8122628012721206efb09d35bdf5d9e0f1b71104777d814c7d5081af",
+  "image_refiner.base_transformer.context_refiner.1.attn.to_out.0.weight": "b6e4d0ac218f57b81d12371305218f4081aa86d62b3899b354143b8be332656f",
+  "image_refiner.base_transformer.context_refiner.1.attn.to_q.weight": "3813ce9fc1024f5564d7ecf27acfcc4a239558d4c914f512e8453e82383a943b",
+  "image_refiner.base_transformer.context_refiner.1.attn.to_v.weight": "5ce909f424097c965ee8357eceff0954bf18ad0e4d65b0aac64aecba3796ebf1",
+  "image_refiner.base_transformer.context_refiner.1.feed_forward.linear_1.weight": "fe5b56643ae5568fea587d2dab92fe5eb00d8453b086556839b78dae3ef36111",
+  "image_refiner.base_transformer.context_refiner.1.feed_forward.linear_2.weight": "0a42a95e37259148b3369d7c01b0a48aa1a7be45f98fe5a8ee8d47f502dde2e5",
+  "image_refiner.base_transformer.context_refiner.1.feed_forward.linear_3.weight": "7d95b975bb7bb6d458be3a6bcbba4e06a328943ecd36f6901d47fd8e54c983f2",
+  "image_refiner.base_transformer.context_refiner.1.ffn_norm1.weight": "8fac41c43a25eb821e0191d8e3188d23f5d6b6407384495ef4868eb900787c87",
+  "image_refiner.base_transformer.context_refiner.1.ffn_norm2.weight": "e391fb2631976b381d84a7682503ab341bb99f3b256a1e5f7a8ee4794aa14fab",
+  "image_refiner.base_transformer.context_refiner.1.norm1.weight": "fdb0cf4d95b2e4a955522613fd5e1517b087d35c738d8f32507f784132999a68",
+  "image_refiner.base_transformer.context_refiner.1.norm2.weight": "a1fd7d38eb24f4e8014e42e24492ebd417ae28fc59ddfd26785a9434fa503cc4",
+  "image_refiner.base_transformer.image_index_embedding": "6e74fb063a0fecc47c7eac0cb87880453735b183b4750a57a1f456bb34e2c7b1",
+  "image_refiner.base_transformer.layers.0.attn.norm_k.weight": "76a01e31c250bf327117c209b4dc649c49fd2bcf077a74dffccd47b901aae550",
+  "image_refiner.base_transformer.layers.0.attn.norm_q.weight": "57ce0639fc30c3bee874de8d430652d1783f3669399491d67275a9a77499f542",
+  "image_refiner.base_transformer.layers.0.attn.to_k.weight": "98426b20e2703b1c67849d0e778a0f6a76df5d28619e7926f999047243001e3f",
+  "image_refiner.base_transformer.layers.0.attn.to_out.0.weight": "9f6ade6a33ea02bb40b6fd3a2583e8fd1dc36049ade93644f5c15f04814b8453",
+  "image_refiner.base_transformer.layers.0.attn.to_q.weight": "5e541d3284274400d447a5908f4309c841f5ce3b8808c1b0771f2ee2aa58fa39",
+  "image_refiner.base_transformer.layers.0.attn.to_v.weight": "bb02aef8a20fb74c09ed69c3f1d1b0abe02e38255eb89d25efda130b33043065",
+  "image_refiner.base_transformer.layers.0.feed_forward.linear_1.weight": "60b94f53eb6819dfb3e2b0b7521a133e4bbe137a203a742a78c2d3b0fa25b65c",
+  "image_refiner.base_transformer.layers.0.feed_forward.linear_2.weight": "67e06b7ff68a44944d6af7fe22b24c9f68b65a1eae61224fc96761b6791832f6",
+  "image_refiner.base_transformer.layers.0.feed_forward.linear_3.weight": "cb985257eee0c7b3b4829aeacfbfd897f68d51c25dcf40e37941a303e33f9c16",
+  "image_refiner.base_transformer.layers.0.ffn_norm1.weight": "d7ee2b8181b5fc9e2955e90c88ce04e3363c8c1622b0f899332b678977b292c2",
+  "image_refiner.base_transformer.layers.0.ffn_norm2.weight": "1842666c992f6cf5f9552e56d0e9952e49bcd7f693128fbff541438a623f9c8e",
+  "image_refiner.base_transformer.layers.0.norm1.linear.bias": "8bbf90f309ce2c233105d7356d174a01278878bcc37ab93a1328b3bca57efc61",
+  "image_refiner.base_transformer.layers.0.norm1.linear.weight": "4680d991d2754c88cad5bf86fab0a5decb16966173eb69271ec48e3db8a1e5d9",
+  "image_refiner.base_transformer.layers.0.norm1.norm.weight": "2cb0ce846266a488b0e2cd73677948b792232ad6f43346e9cae5b9cf7f211d7d",
+  "image_refiner.base_transformer.layers.0.norm2.weight": "d908bd890d2c92da200582224e7d14990524ef9e6c6a1b98cf3f82a3cfd66233",
+  "image_refiner.base_transformer.layers.1.attn.norm_k.weight": "3fc1eb8816e0ccdd6289ab36b330cee1c268a6f21559a473a83236ed1915a069",
+  "image_refiner.base_transformer.layers.1.attn.norm_q.weight": "8e9ae0beb693ca67f56db2abbd519eabf67fa41ff402c818a384e2f9b5691de4",
+  "image_refiner.base_transformer.layers.1.attn.to_k.weight": "57afd637044e520a7c43fbbf3fb3a52b7c0f986d78298f1ff5f4f91e22a813b7",
+  "image_refiner.base_transformer.layers.1.attn.to_out.0.weight": "3aeede486592828bde4b64c4229f6a60f18a6e0ed87ec216a889e2b984f98a77",
+  "image_refiner.base_transformer.layers.1.attn.to_q.weight": "c5737e3e256ee2b75f4c6ffafc40cc218fcf289a84f7b9f4b063cd10e50e6d20",
+  "image_refiner.base_transformer.layers.1.attn.to_v.weight": "3aa6be082150c79d08e6340e3506a204e3c4a470ea8de47b88f30dbf4f79d721",
+  "image_refiner.base_transformer.layers.1.feed_forward.linear_1.weight": "b007ba57461117bbdd63917786fa169255a1b3e1196e4c1d9b954e5efaa67bc3",
+  "image_refiner.base_transformer.layers.1.feed_forward.linear_2.weight": "8c853ddb2c7a5868af0c44c0c55bea567a70c1a96981f4c196b5945f775c8305",
+  "image_refiner.base_transformer.layers.1.feed_forward.linear_3.weight": "74cad6e8de1f33039dae3676746563a8d6d9fa970d040aebf84ebfc504340d54",
+  "image_refiner.base_transformer.layers.1.ffn_norm1.weight": "f2251711274e43917f1e29ef3683189769ccdd05253024d73e700219639280b1",
+  "image_refiner.base_transformer.layers.1.ffn_norm2.weight": "ed34a3b2631db7331157ded1e5901bd27e1ab82b44f0782823d173dd8ab1b967",
+  "image_refiner.base_transformer.layers.1.norm1.linear.bias": "121ef234478e5b857143bdabe105f71686fb3bfa98f477600e62bc2c2fd2cc52",
+  "image_refiner.base_transformer.layers.1.norm1.linear.weight": "1f5130149b5d6b1e04e0bdbe2e378892d6868e153941f21adc89c966d0f6c7d2",
+  "image_refiner.base_transformer.layers.1.norm1.norm.weight": "0d6fe42bf12b62c15bac633f21d9688f8d6b7f46087b2b1fdaa013a4625038d4",
+  "image_refiner.base_transformer.layers.1.norm2.weight": "6acf10dee3d733cd0e754d3eeb8415a9927d5b19d71cb62f138959b6b49f094d",
+  "image_refiner.base_transformer.layers.10.attn.norm_k.weight": "938d8f420f7063eede040bfe45c75ea80e11307bb305bc53f1a9b56cc08dbf06",
+  "image_refiner.base_transformer.layers.10.attn.norm_q.weight": "caee930da3af4abfc31f227221c046515a5ec9978c4f522684b5b7af901a348b",
+  "image_refiner.base_transformer.layers.10.attn.to_k.weight": "9b2c092fd4ce90d4074719a88c30bcde0e510fed7547a5bf9404a8a0eed924db",
+  "image_refiner.base_transformer.layers.10.attn.to_out.0.weight": "a9f5e6306ceb198741b8300222702afc128481f9344802dbb2f8a9ecabb90156",
+  "image_refiner.base_transformer.layers.10.attn.to_q.weight": "0a4855e376aa1c6d18bccf6df5e452fa8cde35e27648c54f6c5a5304bfba567e",
+  "image_refiner.base_transformer.layers.10.attn.to_v.weight": "f05d494f1c83f446954b229e25bfc620ce6f28a833f1098cf77241180a893eff",
+  "image_refiner.base_transformer.layers.10.feed_forward.linear_1.weight": "a149d6d974738f226f1c40d7f0ec521100cf01df436456e45fff9e614ab32e1d",
+  "image_refiner.base_transformer.layers.10.feed_forward.linear_2.weight": "e9de407dcfc061b2ccbe6a7b899d8bd6597a2e5b1e2c909926595aa163c79685",
+  "image_refiner.base_transformer.layers.10.feed_forward.linear_3.weight": "63d2d8f3b04ed928f2799cb522013e15d7fac8aa2c62d505a41c6ba9b26c3d0d",
+  "image_refiner.base_transformer.layers.10.ffn_norm1.weight": "675345d319393f4ee044215ba9a4466d11ae7a1294f26fe241c179b9a294ed2f",
+  "image_refiner.base_transformer.layers.10.ffn_norm2.weight": "7c0e0a2f200513f1ba2516dcf9ca6159c89b63efaa182b6bccd502283a25ccb8",
+  "image_refiner.base_transformer.layers.10.norm1.linear.bias": "6ff48f3b8fcde3bbe85f759a958d500e7069fb1de6a1d65ae66538f0825a0510",
+  "image_refiner.base_transformer.layers.10.norm1.linear.weight": "0e22ebf6be7a5dd7fa736d8bafd4426bb9b54ba2155f69500f002597d61ef851",
+  "image_refiner.base_transformer.layers.10.norm1.norm.weight": "614970e972f54a29e91eb3132e7cabee1628eee8389a90f2d006a2f6f60f1e9e",
+  "image_refiner.base_transformer.layers.10.norm2.weight": "a56acba12382c896a2e4f7edcbfe335460aa6811b8bfce9284b0a853dffaaf8c",
+  "image_refiner.base_transformer.layers.11.attn.norm_k.weight": "29db3dd6a300e1687bbfda666238a7436322732a5f08ba374908616f3d7b06ac",
+  "image_refiner.base_transformer.layers.11.attn.norm_q.weight": "083f48a237333b14f947a68227f10014352878dd3b6d4ca336b13ded406a6907",
+  "image_refiner.base_transformer.layers.11.attn.to_k.weight": "859bd1d919b5c90ffdba53b13d19c5d19ce449d2f730ba9206dd4e5c7dd5a90c",
+  "image_refiner.base_transformer.layers.11.attn.to_out.0.weight": "0782ab838c3bfd3b7d7275c624bff56ef85b89ca51aff5323a15efd3b201b878",
+  "image_refiner.base_transformer.layers.11.attn.to_q.weight": "c34e39230bff4860a442d7d7cca17abd373f7d4f697398205a319ec775ad4665",
+  "image_refiner.base_transformer.layers.11.attn.to_v.weight": "78003ec041506ca08258d628efa31df187477be5636d3a2c84ff6d6b6cb608b1",
+  "image_refiner.base_transformer.layers.11.feed_forward.linear_1.weight": "0e19de8164998373881cfcbb6521c06180a526226307ba26a8bc90388a71fbe7",
+  "image_refiner.base_transformer.layers.11.feed_forward.linear_2.weight": "3dc6816180abcade35e91b7ac99bdcc7db66aee06efb99d6d60358f8b1e7359d",
+  "image_refiner.base_transformer.layers.11.feed_forward.linear_3.weight": "09ebf7e6efb7e2649f08a6c89633bdbffc76de78123acd662e9ba91f502aa020",
+  "image_refiner.base_transformer.layers.11.ffn_norm1.weight": "c8f0f005c1fd40a5d5420cf34e1434d84189902bd1581e4cf2d2993f5581977d",
+  "image_refiner.base_transformer.layers.11.ffn_norm2.weight": "7882c48b96d17eb1dcf53bb3638ec85b8f7cf3cefa3e3ab7b1bf783230868e64",
+  "image_refiner.base_transformer.layers.11.norm1.linear.bias": "c69b09e4ae3dc5d01e1af6bf5d5e8ecffe988d0d765bd95b6c0ebdc9bbd39e80",
+  "image_refiner.base_transformer.layers.11.norm1.linear.weight": "c129807df48e5530166201c6087f1ae2a5e5feafcc034dca903b43ecc64a41ba",
+  "image_refiner.base_transformer.layers.11.norm1.norm.weight": "3f612f25b7a04649a338bac0ea5d79c0ddb84bc1c1b20190c57ec4390eaca290",
+  "image_refiner.base_transformer.layers.11.norm2.weight": "06e05199e5c5e849e419450d698eaacc5fca1015529bb57863b12adb46fdb1c5",
+  "image_refiner.base_transformer.layers.12.attn.norm_k.weight": "8f6528b03d2737ced3c4050ba550577c33f44fdaf86aa752b33d5b1f302d3470",
+  "image_refiner.base_transformer.layers.12.attn.norm_q.weight": "b238a43cb090d4371823ce12a6ed974c55bdf28c62bb845b67a427e6f3cfe096",
+  "image_refiner.base_transformer.layers.12.attn.to_k.weight": "30a46243ea0b3c9f6838a58d6daf34b855c7df479ccb1e5bc418c001081995d6",
+  "image_refiner.base_transformer.layers.12.attn.to_out.0.weight": "85e263b417fc1bf8fe20b9bbe86d8bad64b7171c7196feb7dc6202e7632f2a22",
+  "image_refiner.base_transformer.layers.12.attn.to_q.weight": "ec7c9aa737557e58839f5f6098f0b6553519dcf46a8e80a22ac7794b1f00dbf6",
+  "image_refiner.base_transformer.layers.12.attn.to_v.weight": "c340f90c2c997832e086f0d9c47f9392a3cb27fabff5ad025500e0fa35eafa95",
+  "image_refiner.base_transformer.layers.12.feed_forward.linear_1.weight": "6fe7c5b2db25b6f214901f45bce9f3452da1d0ff0dcbb33052c67966256e156d",
+  "image_refiner.base_transformer.layers.12.feed_forward.linear_2.weight": "158a646fd77d9de1fbe94ee27334e39519ec080f2c8fd001a76f5ebf91ca9bb0",
+  "image_refiner.base_transformer.layers.12.feed_forward.linear_3.weight": "8c02455a64275226edff18a4f47fc8016cd36596bb75068e2e806371f04b7daf",
+  "image_refiner.base_transformer.layers.12.ffn_norm1.weight": "35149ac0528289cfb649d9de304e3b14ee671cc1bb49cc5205550f3b9b0c9e40",
+  "image_refiner.base_transformer.layers.12.ffn_norm2.weight": "15b6b330456ef22d0cbf0b7f5738f554ccdb022b7983b2a49eae42bbc3d2c778",
+  "image_refiner.base_transformer.layers.12.norm1.linear.bias": "3842501d9cc2716f315c046e3dc5cf965a4b0dca26e2d4e95a310180246ce2ae",
+  "image_refiner.base_transformer.layers.12.norm1.linear.weight": "2f42574b3ae7bd875177121dcd3c5360dabd0a4006591ffaf98232c1c78ddf99",
+  "image_refiner.base_transformer.layers.12.norm1.norm.weight": "e9c1c67940494810c5e57d6c4c13f40cc26d3d8f1957790775102c4e2e458071",
+  "image_refiner.base_transformer.layers.12.norm2.weight": "074c8396f33ec45810e9f798e288c3d41fb73304331ebc6c13dce017b40d65dc",
+  "image_refiner.base_transformer.layers.13.attn.norm_k.weight": "18190d6819eedef829a79985ef81bdb3ca5ec72b9ad447799b16a1cad792bc68",
+  "image_refiner.base_transformer.layers.13.attn.norm_q.weight": "dd135bf39664e05235aad2fa1e9c584243bbce6078130cfa92403a9ecf36be2b",
+  "image_refiner.base_transformer.layers.13.attn.to_k.weight": "36d243c6940282680d89e23d0583ba9c5fc0c17b2a4068d9b35facb4c88e5ff7",
+  "image_refiner.base_transformer.layers.13.attn.to_out.0.weight": "868b4b65c5d5e3e65c55753ae48fe205525448327fbedfd75a97195ba9b4744a",
+  "image_refiner.base_transformer.layers.13.attn.to_q.weight": "caebf2c67ca45f4f41c636a3aaa333795d8c8bace471f7b18cb96e5469a69f4e",
+  "image_refiner.base_transformer.layers.13.attn.to_v.weight": "3f048f3e0fae9bffdb798793697dd738e5ed732f7601e511dcda968ee4addfef",
+  "image_refiner.base_transformer.layers.13.feed_forward.linear_1.weight": "191ab65037f6a19562a435a836d4905a4553068d226ebee22764073a99f9d7be",
+  "image_refiner.base_transformer.layers.13.feed_forward.linear_2.weight": "cc1b1bd76b098cda81966528259d1eb6ce8ff55604092c1fefa940832ce1564b",
+  "image_refiner.base_transformer.layers.13.feed_forward.linear_3.weight": "420abc803f25e683b2bd12ad00c37c4f5257ddb6a972dcc5421c649d9c3de73c",
+  "image_refiner.base_transformer.layers.13.ffn_norm1.weight": "3f67017b7bbb6c5c795c2b58078aefc13f24f5a991544e86b14d075fa8f885a3",
+  "image_refiner.base_transformer.layers.13.ffn_norm2.weight": "e8b3f82b803402c9c1901bad5b9bc7a3135ab2104e8babd468e12db5ce34f147",
+  "image_refiner.base_transformer.layers.13.norm1.linear.bias": "0612f92e3664398c9f70a57614008c0600aebb85e975712800e19f63581124b0",
+  "image_refiner.base_transformer.layers.13.norm1.linear.weight": "f74c2cc26984db40f8938e4120266ce6b8bfd6be733a2edf0e5b18878cfbb725",
+  "image_refiner.base_transformer.layers.13.norm1.norm.weight": "80c0100fa95ba85bd92f0ac507aa8d7470dd8a9034a7ed03b5c50e37a8ddfe83",
+  "image_refiner.base_transformer.layers.13.norm2.weight": "6d7765a83595cb2f25d2bc449ceff9d645ad40ddf841afff3e5af8164da07311",
+  "image_refiner.base_transformer.layers.14.attn.norm_k.weight": "0db22bb1b3c9ee10bcdb1abce94f0ec18adf7e6e41416f375887e200c3dc652d",
+  "image_refiner.base_transformer.layers.14.attn.norm_q.weight": "ab0db1234bdac724488d23ff8b5ea0047788e84363bc4b69deb072d7e9abe6be",
+  "image_refiner.base_transformer.layers.14.attn.to_k.weight": "166a0850224232b81a48fc2ae096f8f0c957aaea99e21372f348443faaad9a8a",
+  "image_refiner.base_transformer.layers.14.attn.to_out.0.weight": "89f355ca749758e3a1a254eaab55b9e21102e096039f3b54c2095697cfb0cfd2",
+  "image_refiner.base_transformer.layers.14.attn.to_q.weight": "cdd01c7276d738ffaabecd57e06f5c43ea38954e1c91ea1d33baf3de4ef79055",
+  "image_refiner.base_transformer.layers.14.attn.to_v.weight": "5b45fa3a31804de272b1c9aee30f77448be68dc006fb9f2d14f6d3e87a93f39f",
+  "image_refiner.base_transformer.layers.14.feed_forward.linear_1.weight": "0739040df1d5ddcebcfbfed21d1d3e220c9936e9ebd088607e509daac4d70f8f",
+  "image_refiner.base_transformer.layers.14.feed_forward.linear_2.weight": "e0a72c79eccbb1ac5f49fc18f11b93bf2d1c0949f1bc2bcb41761f05016e92e9",
+  "image_refiner.base_transformer.layers.14.feed_forward.linear_3.weight": "419bbc41636c130ff1a21f454f66c4dfffb51c207b37d73a1ea5b09e2929aa0b",
+  "image_refiner.base_transformer.layers.14.ffn_norm1.weight": "8a2bf9c878fe929db1a03e231884aa62ed86146a20f0cb58751d5bb688a62a8a",
+  "image_refiner.base_transformer.layers.14.ffn_norm2.weight": "99e8140fe6be3f3fa04d642b43238a709ada34e2d6ae6740a83ef9839855a1e8",
+  "image_refiner.base_transformer.layers.14.norm1.linear.bias": "447317bba6b8d12bce3db9ee8565a921d5ac525d4668d241ab26f41ba74799e0",
+  "image_refiner.base_transformer.layers.14.norm1.linear.weight": "8a7a7820540ec0fe661783db766151194ba0273bbe30c0bd147853b4e913f912",
+  "image_refiner.base_transformer.layers.14.norm1.norm.weight": "965ce8a01271afdc7b4698f2f08ec11b07b57f771f39fe74dd0f2fa8ddf9ea82",
+  "image_refiner.base_transformer.layers.14.norm2.weight": "83a0ea33ddbff42e683d28878d9450fc5fcbbf2274b955d600c07a195aa5061f",
+  "image_refiner.base_transformer.layers.15.attn.norm_k.weight": "eafdd569eff3cd6338306af95617ff41326c98dc662403ae4fc0e8ea44750cd7",
+  "image_refiner.base_transformer.layers.15.attn.norm_q.weight": "4988343ff4e49eeb116e994ed7a8bd59db3a51d7d81978ada78f8f4913892709",
+  "image_refiner.base_transformer.layers.15.attn.to_k.weight": "a36513db90eefd9aff751d9b098a82bbf52e858e00bd6b9aff270afd83cb5136",
+  "image_refiner.base_transformer.layers.15.attn.to_out.0.weight": "8ee080234281195d6cc38ec6ba93b69a66105bfca73d3353f43038fdb4c4e4d7",
+  "image_refiner.base_transformer.layers.15.attn.to_q.weight": "b8b8d06b39a78fdbac1e0f7ad787132f5562003564428eafc83c07494d19e4d0",
+  "image_refiner.base_transformer.layers.15.attn.to_v.weight": "e2ad018109ab1cf57c1d7210fab2a39759296e1e777a5b103911232162435943",
+  "image_refiner.base_transformer.layers.15.feed_forward.linear_1.weight": "0201f1a070835a4c38ce0cb76dca48907799ea3309c90e854cfc9aa364b31350",
+  "image_refiner.base_transformer.layers.15.feed_forward.linear_2.weight": "6a5767b98600d8eac993165079488e0c156167ba3fb7db460e5292e51e609cec",
+  "image_refiner.base_transformer.layers.15.feed_forward.linear_3.weight": "3a7d9bc23bfa716cd2e3696abb8774507f79e162c487986767fa85e71ca71786",
+  "image_refiner.base_transformer.layers.15.ffn_norm1.weight": "2db3de7242fc99143a6356f5b20bb51c45e43bdcd6dd753b4f917f0f67844484",
+  "image_refiner.base_transformer.layers.15.ffn_norm2.weight": "f27da49a5243a2927ef75cf3ba9e660ff7ca65fdc9fb9a8480f4d4787552ebe4",
+  "image_refiner.base_transformer.layers.15.norm1.linear.bias": "e48595fe097d733644fe8ed10739b753efc111b28a7c87dda49d6bdd4f8a81c3",
+  "image_refiner.base_transformer.layers.15.norm1.linear.weight": "af2666ef01c85537e47c753ae88088f4fdf9ed06f767121b7fb1dbc461f2a3e9",
+  "image_refiner.base_transformer.layers.15.norm1.norm.weight": "c0a996e01a100f8a9cd92b2a538c538e3eb392c9fe1fad8c52a4c1e81d253a4b",
+  "image_refiner.base_transformer.layers.15.norm2.weight": "c4b52bab66f77896d0b9eab196b2ffe2e474078cdc2c38dbe178be6ed17d3a0d",
+  "image_refiner.base_transformer.layers.16.attn.norm_k.weight": "15d5e601f3a461355c335669736584f712d0ccd50c9236444e83040e6f1530b3",
+  "image_refiner.base_transformer.layers.16.attn.norm_q.weight": "77b8eea09a0a919a717ddd3ff980e2646d9e00a00b2e2f642156f425e5010245",
+  "image_refiner.base_transformer.layers.16.attn.to_k.weight": "4e251348fcbcbfd0bbf801e39f45222043426b781aa22fe584b667938ec7c9bf",
+  "image_refiner.base_transformer.layers.16.attn.to_out.0.weight": "cce702545cba2177ce187848c3889b9631ee786a92d65a497a1b16b4da4ca74c",
+  "image_refiner.base_transformer.layers.16.attn.to_q.weight": "79081121c45130b5f5bcf8d3d818a0992bfa3d2f98f1961b4ca699f8603a8fe9",
+  "image_refiner.base_transformer.layers.16.attn.to_v.weight": "9cd60c5680a29e46aa810b51d1e712a0e34c318486956a633e10a17c4771302b",
+  "image_refiner.base_transformer.layers.16.feed_forward.linear_1.weight": "35ea9e566ea13d25ed1c2d5c8d588911a400621e052e841e245a86ee79b5aaa0",
+  "image_refiner.base_transformer.layers.16.feed_forward.linear_2.weight": "ecbaf4184d8184d51469fcaeeaa32760a76b4c5caa619f3355fd3311bce200cf",
+  "image_refiner.base_transformer.layers.16.feed_forward.linear_3.weight": "91990765855498e37c720704c993167ae4ff0cea68f77cf38c17ed64e4665d65",
+  "image_refiner.base_transformer.layers.16.ffn_norm1.weight": "5b8c4f956eafe04134f9b6eddc47852f8d896f27bb7330863311025faa3d1caa",
+  "image_refiner.base_transformer.layers.16.ffn_norm2.weight": "0d1843ece2a62c79d73865e4a16057e3aa634aa3cea6a70de02289f1f39411de",
+  "image_refiner.base_transformer.layers.16.norm1.linear.bias": "b9af9bddd53081a7d291dbf2f7b389f48c4768b1f0e970b9e14f67a7656f7301",
+  "image_refiner.base_transformer.layers.16.norm1.linear.weight": "146d2e936212cd3ecdee86ae4cebbe14265f0e38c09866c2b99f22f7948da7eb",
+  "image_refiner.base_transformer.layers.16.norm1.norm.weight": "6f3bb8f3c86691cc406eb0f16ad9f8dd22fae5c19c71c499f87a0fb5cc9ecfd1",
+  "image_refiner.base_transformer.layers.16.norm2.weight": "26211d22516ebab8aa16ed64bc2af05c46dd9ca429bc16a310c6cef466187627",
+  "image_refiner.base_transformer.layers.17.attn.norm_k.weight": "336831826f0976656fbf9800c547660d0997170e0818d3e824df68e1e062c82a",
+  "image_refiner.base_transformer.layers.17.attn.norm_q.weight": "9438f9e28c043207d21121a1c869237a99a2268a659f3f5df8b09158f307e2f7",
+  "image_refiner.base_transformer.layers.17.attn.to_k.weight": "a29290f48520becf42d6f7f0498be1a30cc5e3e731a6f1fb1e19956bdd7f134b",
+  "image_refiner.base_transformer.layers.17.attn.to_out.0.weight": "fe0e96b69be8d61d45f11cfe80675f9a7c369b9a723033d876445b82619d2719",
+  "image_refiner.base_transformer.layers.17.attn.to_q.weight": "c3bb8d22b64f1d6b300655af18e8f612bdcbc13844bb5c1a4cedbdcdfee2d3d0",
+  "image_refiner.base_transformer.layers.17.attn.to_v.weight": "376651df8d782c20903bed2b97d76fd22da182138a3cf9ba8b2a4e0585f8dabb",
+  "image_refiner.base_transformer.layers.17.feed_forward.linear_1.weight": "15063d78d5bb36f65b8a877f25d1775db3ac26a8575f5223a6e2fc89af048e28",
+  "image_refiner.base_transformer.layers.17.feed_forward.linear_2.weight": "7039f21c5e4964a76a3ca72c27a68eb1c41befc2deb96e944064c0b471dbf951",
+  "image_refiner.base_transformer.layers.17.feed_forward.linear_3.weight": "a1be56a92344f6c9520d0a3f11b2d60214fb6492378ec55b13c03796c008ec3f",
+  "image_refiner.base_transformer.layers.17.ffn_norm1.weight": "a39f4b5cde4c91a55a941f185425c39ad2b036dc77af75d551425b2c2eab9b93",
+  "image_refiner.base_transformer.layers.17.ffn_norm2.weight": "a4cb6c9feb2a8569a228a82a0be4d3b3d25559dd0abee2aa1e760fa4744004cc",
+  "image_refiner.base_transformer.layers.17.norm1.linear.bias": "246568ec126f1eacd740dd9067f5f75a5cbcb2d9eba31a695c6aa52c72c29718",
+  "image_refiner.base_transformer.layers.17.norm1.linear.weight": "efb9bec1159b088112c84be663a90ff0b5f708b8c60d6e99e5eef039e33f74af",
+  "image_refiner.base_transformer.layers.17.norm1.norm.weight": "80fb747207976744e6b0bc658b9b2d23de3bce3d02360b1b1da3254a4278315d",
+  "image_refiner.base_transformer.layers.17.norm2.weight": "fa66e8e5bb68f80fdeb76aa106d7026b623cb81f3a5c0e05d2fde9b213ae662e",
+  "image_refiner.base_transformer.layers.18.attn.norm_k.weight": "e8d3af1c7f4edcb42740c50bf65d1ae52f0f6d59b4178c3976800546feb75479",
+  "image_refiner.base_transformer.layers.18.attn.norm_q.weight": "bd55558785eba80bd6643dc7cd7e5a5cc8f8b930f46fd0dc94f6b56c00c47f02",
+  "image_refiner.base_transformer.layers.18.attn.to_k.weight": "cb9f0f1259456c3fa36e55333dc958863c89f149666713cadbbb46b4fdc044f7",
+  "image_refiner.base_transformer.layers.18.attn.to_out.0.weight": "bba42efb54790ad33809090a735820adbd001ed2a6af73bc35ec4038e6af6581",
+  "image_refiner.base_transformer.layers.18.attn.to_q.weight": "6f7eb79ea2b95e521152a61990292affd47ebe5f2eb7c1180be9c658b01afb01",
+  "image_refiner.base_transformer.layers.18.attn.to_v.weight": "aa45ad682e39db5b9d4d2cb8b5017d4d749df17ab013d6c4665b207020f07205",
+  "image_refiner.base_transformer.layers.18.feed_forward.linear_1.weight": "d0617352f785ef5d3deec96e60c87afdf10e3ed81f86312a8487b3f460cefb22",
+  "image_refiner.base_transformer.layers.18.feed_forward.linear_2.weight": "5f18e3d76a105bbb9eeaff9c6292055a46d4a6770ed00198958df2891a584983",
+  "image_refiner.base_transformer.layers.18.feed_forward.linear_3.weight": "d221df2cc3069ccc244497f361640940c556a18dd0350dc14ed92e3db59affea",
+  "image_refiner.base_transformer.layers.18.ffn_norm1.weight": "b4e507c952e34611c628ea83d7d684ff014407d19def98e13e0244f0cbca8e51",
+  "image_refiner.base_transformer.layers.18.ffn_norm2.weight": "9e4bdac6ec6f4715248c44de5af109c9e7970dc2cdee24db6d9dbd065a121cd7",
+  "image_refiner.base_transformer.layers.18.norm1.linear.bias": "b839ca1861a49bc237c3adb3ff6ebc306247546e2d31c6dfcf6be0305183f381",
+  "image_refiner.base_transformer.layers.18.norm1.linear.weight": "5d8a749e5bf9f2c8bceb1fc57744442a4db9d0ddf22de75e85fd9533928fc2c4",
+  "image_refiner.base_transformer.layers.18.norm1.norm.weight": "345cdc9be17c2409d4b2b209623078aeb1bce5bf11e6748edd66c550da362131",
+  "image_refiner.base_transformer.layers.18.norm2.weight": "41ebf3b7351a9a2fd2fe13f2d01a2b277b16647a959bba8e6b61f1d1fd89236d",
+  "image_refiner.base_transformer.layers.19.attn.norm_k.weight": "067014cce02aaab466a3e7079edd40feb94b94738ade3d37c2e751782b5ab61d",
+  "image_refiner.base_transformer.layers.19.attn.norm_q.weight": "e1fbd8189709b41788af3ca6a654f45316be5f2693a7e07b24ca2ae119486e61",
+  "image_refiner.base_transformer.layers.19.attn.to_k.weight": "9793069ca5843a4f2b62752270cf2df68123c5d5a1b963c774fe563234574e9d",
+  "image_refiner.base_transformer.layers.19.attn.to_out.0.weight": "0a08e67cbe6c87d0ce79be35b7324b36c2fa3b33534d1f0f70a8357c8888541a",
+  "image_refiner.base_transformer.layers.19.attn.to_q.weight": "aa7e5744f71b837e37a7be23bf3ef4c7f74397f7d77ac3d7fb6099fa32a1f9d8",
+  "image_refiner.base_transformer.layers.19.attn.to_v.weight": "e8e97550da7317a0b170668e029b3c7967ea53d21963daf70de594d67a68d392",
+  "image_refiner.base_transformer.layers.19.feed_forward.linear_1.weight": "b126e749f960eacdac2248487e8fa1100512e2f89f525d4f3723a1fd14e75992",
+  "image_refiner.base_transformer.layers.19.feed_forward.linear_2.weight": "4785b9a6aad1b0688ba43fe666469a401edf0ed24d0ee4d277c64b4e1b66eea7",
+  "image_refiner.base_transformer.layers.19.feed_forward.linear_3.weight": "4c5edab742c10f57a32e2ced96f57f853bdcaca31537c06685900f0c6ac9204f",
+  "image_refiner.base_transformer.layers.19.ffn_norm1.weight": "04298b1cebaf61ab47704ef2c53f8a4d4013b2d9c6e12ed5f842dc706b7be79b",
+  "image_refiner.base_transformer.layers.19.ffn_norm2.weight": "07722e443d866b78aa9065c3dd06c5c87479cbe224f696c05f2e11f0c8b306ed",
+  "image_refiner.base_transformer.layers.19.norm1.linear.bias": "d7d810da143b7c84f0eae28eed504b2ace80f3c51f52383d39e3f8126806abbe",
+  "image_refiner.base_transformer.layers.19.norm1.linear.weight": "ca278c40aae2f91138dba1c72cee6571f37a475d645dbbd6573fb41fbeb99116",
+  "image_refiner.base_transformer.layers.19.norm1.norm.weight": "ed42bedc12a343d04b2e23b79dec2d9f6d298e69420ebc5f7f05429f2e23e81a",
+  "image_refiner.base_transformer.layers.19.norm2.weight": "fba6520253cac78cd99f6944dfec7b2a5a1ba27ff8f62c086171cb703e5f12ec",
+  "image_refiner.base_transformer.layers.2.attn.norm_k.weight": "5d869eb14b153b2fba71933ece3963356c2a44fe45b323918d436e6c7e53f3bf",
+  "image_refiner.base_transformer.layers.2.attn.norm_q.weight": "c0ddfc68c4e0d8d1fa48e3fd1a3fec0c7f8b5495d2e3c21263833b9f17a81e84",
+  "image_refiner.base_transformer.layers.2.attn.to_k.weight": "5e26fb8f22c56808a15277dd17fabdb964f6d90e85e9be04ac8589eec249aad2",
+  "image_refiner.base_transformer.layers.2.attn.to_out.0.weight": "41a3d27e230bb672547b812568055db11f5159740dcd991aa1d20f7ab142008b",
+  "image_refiner.base_transformer.layers.2.attn.to_q.weight": "027cc27ac5edc66991541b23609913a3cf791f9ed4117070d41235a6e69d67df",
+  "image_refiner.base_transformer.layers.2.attn.to_v.weight": "d246b6d71f988dc791dde60202ee16796fa29237cf7cdf463c288215ba886acc",
+  "image_refiner.base_transformer.layers.2.feed_forward.linear_1.weight": "f11fc4c0ed4292e0d8aa7b01a842fd9c67eec138e0113c28a3f91ad9a7f415ce",
+  "image_refiner.base_transformer.layers.2.feed_forward.linear_2.weight": "72bdf88cbebf6d941dfadbff26ed30b21f089038cdc60acdd609480e7683c713",
+  "image_refiner.base_transformer.layers.2.feed_forward.linear_3.weight": "7e960984873a0b5e084bb3e769507624cadf2792a872887ee2593bb470c351de",
+  "image_refiner.base_transformer.layers.2.ffn_norm1.weight": "bf456ca19ed588bccaf9aa47ec981a6cf8e3b99b5495d965f0f410d095d33ef2",
+  "image_refiner.base_transformer.layers.2.ffn_norm2.weight": "036f1148dfc861e5485ba633c2ea3287705a35b82cbc54ba6740eccca222e177",
+  "image_refiner.base_transformer.layers.2.norm1.linear.bias": "28bfeca559d165d012b9eec175fc3b0d37fd3afb13db6354edef3df976a9bd34",
+  "image_refiner.base_transformer.layers.2.norm1.linear.weight": "f69ab599122a219a221c246ca2eb05944c4a6731fc679057818516e77d9575a1",
+  "image_refiner.base_transformer.layers.2.norm1.norm.weight": "7e3429533590ffab7991b685430825e9e723e3374b925355191edf670c6935fb",
+  "image_refiner.base_transformer.layers.2.norm2.weight": "d3248bc86e9230794024d0366bad10932c81faddf0fb249f7547416d7daf7fc8",
+  "image_refiner.base_transformer.layers.20.attn.norm_k.weight": "5d93d20a883f3a550b6631d37bc6529fabf649501a9edda5b78d10a6931d5340",
+  "image_refiner.base_transformer.layers.20.attn.norm_q.weight": "389258347313076cd70298ed5d108b3d28b105d305cdfcc3e0ffaa2b9c7f455e",
+  "image_refiner.base_transformer.layers.20.attn.to_k.weight": "581f8e4b7dc499a335e50377b725b862171181812894764f892c6ee2e175b3cb",
+  "image_refiner.base_transformer.layers.20.attn.to_out.0.weight": "2c4f5ceb2d2cde896d3ae876c3016e81c92f2deba90bb7929cb96df97777e22e",
+  "image_refiner.base_transformer.layers.20.attn.to_q.weight": "ff976ae605989d411b1995edd538318e98f9595e4089b70c7e48e15d8fac46fa",
+  "image_refiner.base_transformer.layers.20.attn.to_v.weight": "66365532e0c3a0b66642447941bd27230018a853a73e37f12874c7747537b4fd",
+  "image_refiner.base_transformer.layers.20.feed_forward.linear_1.weight": "e36781c13ea1c896cecfaa4c89ddad62c1f57216033537932ba5072a2c086e21",
+  "image_refiner.base_transformer.layers.20.feed_forward.linear_2.weight": "30c67d87a8c9d562a52c9d4b67e2bc237e49dde790fcdb2ed7e239a121977d18",
+  "image_refiner.base_transformer.layers.20.feed_forward.linear_3.weight": "1abfee2cc5996b2829de93e80459b801aee4ba7c466efcedcfedf1c274d7356d",
+  "image_refiner.base_transformer.layers.20.ffn_norm1.weight": "d6967ff1540aad06f155dc8c16e2c7034b37fcfccc3651d7f8420b4287d4839a",
+  "image_refiner.base_transformer.layers.20.ffn_norm2.weight": "d8d1e95368aa4a65b1617d942d35677c6aeb0938e93b4df8e8e272100381594c",
+  "image_refiner.base_transformer.layers.20.norm1.linear.bias": "4c8c1c09f951ccd7f65e57bb818df1eafe760b0fb4de09fdc44503c199469aef",
+  "image_refiner.base_transformer.layers.20.norm1.linear.weight": "bc9ef0e73503371706b43859a82bab9c40e368252cc48ed3e2c4131afd4231df",
+  "image_refiner.base_transformer.layers.20.norm1.norm.weight": "28e438bdcfd30238bdb437a4bbe4906297d074a38062fdadbc163966f9c3ba24",
+  "image_refiner.base_transformer.layers.20.norm2.weight": "b14f16b999d6d836c7d85b60a9c4b71d54defa18626be64bafd049baf65602be",
+  "image_refiner.base_transformer.layers.21.attn.norm_k.weight": "ad9a3545144aaf7456e35a38510bb55a79e93befca2fd8dcb94f711cf31f4a51",
+  "image_refiner.base_transformer.layers.21.attn.norm_q.weight": "4d4ec8830f9c4a807996bca27c52631efd770735d5a66c15835d62f2428b76bb",
+  "image_refiner.base_transformer.layers.21.attn.to_k.weight": "5e90a7e5f35509b49cdac893df6c8b68ea7426265451bbbb112a3bf015f379a3",
+  "image_refiner.base_transformer.layers.21.attn.to_out.0.weight": "4abd3d578c7d0bb26591d4881b23d389731f6d358a0d4c6beaee2f9b7440db31",
+  "image_refiner.base_transformer.layers.21.attn.to_q.weight": "400d67d83cd8e500904be9190d95f8b3719c44a7d2b6c82473278280427a918d",
+  "image_refiner.base_transformer.layers.21.attn.to_v.weight": "1785ceb209c8c462edf40c9168b5a8fe77e10d632d0501a934adb99995cb9866",
+  "image_refiner.base_transformer.layers.21.feed_forward.linear_1.weight": "d3da1c21dbdd127d0b55af93a898c304bee8588b189019506586b674f481a54f",
+  "image_refiner.base_transformer.layers.21.feed_forward.linear_2.weight": "1951ef7258f5e1739fadccd7cdb73558fb47db1e164c822760cd194f32125053",
+  "image_refiner.base_transformer.layers.21.feed_forward.linear_3.weight": "95a5b8b33d102e87db21ce5c866fa2e5d247038864cfa29bfd07a7729cda95bc",
+  "image_refiner.base_transformer.layers.21.ffn_norm1.weight": "96ec48240b2c9102855a953c7dc1646ef241a0203e707450a82591e213a62a6f",
+  "image_refiner.base_transformer.layers.21.ffn_norm2.weight": "21bf27924a599ae03fa6e5bd5bbe856615fb07f6755d20f42f81c8bb776406cb",
+  "image_refiner.base_transformer.layers.21.norm1.linear.bias": "3206aa5f9464c82aafa40ebf58004723a8e145cc3604d70f0c23892b38ffb719",
+  "image_refiner.base_transformer.layers.21.norm1.linear.weight": "21550142a8e3b7cc9ccf38fb4353732e564c535f605505e85d01f2972eaa839a",
+  "image_refiner.base_transformer.layers.21.norm1.norm.weight": "1f8a5fbe5975dcc05b5de7cb2eb61202120d4419a554e0d1e5e0450a34e7badf",
+  "image_refiner.base_transformer.layers.21.norm2.weight": "1e6e6eb04e996df9441e44cc2bcb1c2434729cbeb8307a7ec7a22941fecb9cff",
+  "image_refiner.base_transformer.layers.22.attn.norm_k.weight": "2de9c985034e74672325fef74e100e81e315a55db0a967d37f85f5a75f127790",
+  "image_refiner.base_transformer.layers.22.attn.norm_q.weight": "37c84ac57dd14e83688340bcd816306f46a3bebfd2883fb5c71d278d4c5bcd61",
+  "image_refiner.base_transformer.layers.22.attn.to_k.weight": "74b25d16e1b59e1a3a1b2b01705da491d5e994ee63f3de8ed223bb7216321462",
+  "image_refiner.base_transformer.layers.22.attn.to_out.0.weight": "18d789baa4b29cf4ceaea0959f6f35ad0a62f8583bcb9c60123e5625ecb5de3d",
+  "image_refiner.base_transformer.layers.22.attn.to_q.weight": "5ef05e7e7cb547fbd68c9bfa68d1990f7323666f8e7349fde5feafad06533151",
+  "image_refiner.base_transformer.layers.22.attn.to_v.weight": "8608f5bc458e5085bb07c1c6ac18e09c500e2b4c5570b6b0b64c5ee2d0d5e43c",
+  "image_refiner.base_transformer.layers.22.feed_forward.linear_1.weight": "aa62eb5f466585d91e6c022b85ad139deedbd4019568a7fc37467f53190061be",
+  "image_refiner.base_transformer.layers.22.feed_forward.linear_2.weight": "27d540b8b9c491db53ad58411210816d2b28f3e5f6a454c65d21b5394e66202a",
+  "image_refiner.base_transformer.layers.22.feed_forward.linear_3.weight": "58b265c1dce3548646a27d372d9258f7a7e1d2814c99f52ced642a0d7c6cf9e9",
+  "image_refiner.base_transformer.layers.22.ffn_norm1.weight": "1aa3b76f5855b25d1cc4d2ada8a809c8862f1a6363c401fd59f353c5b44ae48e",
+  "image_refiner.base_transformer.layers.22.ffn_norm2.weight": "3060735bbec6292e045896951b7b64fbe3714dc05e493ab5f195dc5b84920a81",
+  "image_refiner.base_transformer.layers.22.norm1.linear.bias": "1c36d4a1d93bbe8e5c3827ca16ca1cc53f7d05b3a8d6569339fe436839b86f6b",
+  "image_refiner.base_transformer.layers.22.norm1.linear.weight": "c1b72916e15bafd7ebbc1eec965f09380b16fa8d7bd9130e6cfa9bf9e97bc5e7",
+  "image_refiner.base_transformer.layers.22.norm1.norm.weight": "e12f25b7037b07378f37d629eca55e8d3081f207e90193fa0c4524994c8f87c0",
+  "image_refiner.base_transformer.layers.22.norm2.weight": "b1bab0e644f152e14c6928ab574c6765d099b3ecae8791abaad693b03c891ce0",
+  "image_refiner.base_transformer.layers.23.attn.norm_k.weight": "0dd0814a18e0014411c68f2827b99ab7e5ea3a73f6cb11cbdfb320982c77f861",
+  "image_refiner.base_transformer.layers.23.attn.norm_q.weight": "07d7af057db8eeb287dce8263615d489a8da4928ef1bbf997775ac61a3491594",
+  "image_refiner.base_transformer.layers.23.attn.to_k.weight": "6fc6d614ea6e87e3a30200139fe28bd80172a0f626c4f439c8da492284c43b33",
+  "image_refiner.base_transformer.layers.23.attn.to_out.0.weight": "063563d4b986ae1b6c3f0384ef36d41226f6c764a575da6bfe1510f51682c292",
+  "image_refiner.base_transformer.layers.23.attn.to_q.weight": "e0ae1206dd1046fa0ef9ea9d815bfb7245b868482dc954ba53603a7b85809893",
+  "image_refiner.base_transformer.layers.23.attn.to_v.weight": "24fea0944650b80ca1ba8974bdf4325157b838d06c8310f7eff01e49145a522b",
+  "image_refiner.base_transformer.layers.23.feed_forward.linear_1.weight": "3f1a5c416429f77f6eb2f6a50e23d878d50e2551718be85b7c8102443cba09b1",
+  "image_refiner.base_transformer.layers.23.feed_forward.linear_2.weight": "9a60f6edde1f1a43fe5425912947fd3b789e7d983e2decda119753249192aa3c",
+  "image_refiner.base_transformer.layers.23.feed_forward.linear_3.weight": "398b55d6939b79ab192b6d3bcc55ac3406167a33b0d0716a734f6a8d08e70987",
+  "image_refiner.base_transformer.layers.23.ffn_norm1.weight": "4bd1020bb3ddbc7de984c9aa6e1c8e45c7db092f483045836d1f572318bbd300",
+  "image_refiner.base_transformer.layers.23.ffn_norm2.weight": "e09e7be0351f14498365ca2b4b54ed592f7499eb2f17f078a0b34bbecebbb6d5",
+  "image_refiner.base_transformer.layers.23.norm1.linear.bias": "9b6c3b22a8ef5c059dc914d24956da8b2fe561303f5b3e987af4572bb0694cc5",
+  "image_refiner.base_transformer.layers.23.norm1.linear.weight": "e4ee1f3458a0833c400316cfcc041d6e8178b7fdba834ddea09caf8339bc4a97",
+  "image_refiner.base_transformer.layers.23.norm1.norm.weight": "aa7023dad2e925c1edd8b3df03775c09e8ce2a9490331015df619e479ef04034",
+  "image_refiner.base_transformer.layers.23.norm2.weight": "dc124723656203e20791d072ca047034e605a2b63f1464129d0bf317a3399aa1",
+  "image_refiner.base_transformer.layers.24.attn.norm_k.weight": "7506cbc1d78a860952d5c47ba5bf5f11d7bfaeb10315bf812d79e007b32a0b43",
+  "image_refiner.base_transformer.layers.24.attn.norm_q.weight": "538adb54ab3eb4c239d9eb488f12a6855ef627e722dbc40e3e6e1777db2c9d37",
+  "image_refiner.base_transformer.layers.24.attn.to_k.weight": "a2f4f74c7f4d0a0466d018c6ab6fa6a4c0c97474b1e7328f929870ae92bfb783",
+  "image_refiner.base_transformer.layers.24.attn.to_out.0.weight": "e80752b8368a78d6c18aad377949c5e556b184b8452f08df775f7aaa41d18c37",
+  "image_refiner.base_transformer.layers.24.attn.to_q.weight": "41498970a961f0c345834f2bbfcc9daab28552f0b91d9e4eec83cf8be8b3c2e3",
+  "image_refiner.base_transformer.layers.24.attn.to_v.weight": "ed7f68d3929af845737b8629d4e5ae03f8acbee0e9d7e050019500feac98094d",
+  "image_refiner.base_transformer.layers.24.feed_forward.linear_1.weight": "567aa84a9c4cc56ad9f661d93055065b54a86b360c348d3e1b0d66f002a15ac6",
+  "image_refiner.base_transformer.layers.24.feed_forward.linear_2.weight": "7d1bd20f6ba714d85ccf9cb7096209283f1bb72b7076c479bbaf202c751d3d7c",
+  "image_refiner.base_transformer.layers.24.feed_forward.linear_3.weight": "ff3498815fe2793d457a55f6efab85e0a0861a7d4ebeeb21fdfdcab9ea846b33",
+  "image_refiner.base_transformer.layers.24.ffn_norm1.weight": "4ededb2150801de279cefe57f93106d47e9d65f9ac0635f45f981aa37bca39e5",
+  "image_refiner.base_transformer.layers.24.ffn_norm2.weight": "963ea939651b44718e6789a544f185676a48c92aaff4333275e7a9ba1c178182",
+  "image_refiner.base_transformer.layers.24.norm1.linear.bias": "89c0146f21bb439c2f2bd4f633f74c5fb1994c6a70da686279baeba50c741232",
+  "image_refiner.base_transformer.layers.24.norm1.linear.weight": "9d19195871e1bc2f4fc3a9c90b1640f15e465b420ac9ea4c250ab981f844c295",
+  "image_refiner.base_transformer.layers.24.norm1.norm.weight": "83c2d2bcdbc2b8296bebad0961a50dcc49393d4acc0e4a5b37116365d58763b6",
+  "image_refiner.base_transformer.layers.24.norm2.weight": "3e9b5e4778953239386e2ee1744cd3a7817bb499589d7029ab33adfcdc067657",
+  "image_refiner.base_transformer.layers.25.attn.norm_k.weight": "94350ab2b00c117367cbd7cff80ea07ba24b5f9bd8764b0e645b6944838e8c19",
+  "image_refiner.base_transformer.layers.25.attn.norm_q.weight": "79b45f3a609c0578dffa1c606c574feeac93eb200637efc74452c83b445ce463",
+  "image_refiner.base_transformer.layers.25.attn.to_k.weight": "da65f00a63856be139fa05d05adacb486b9045f763ad390b180d85b2b664e8bc",
+  "image_refiner.base_transformer.layers.25.attn.to_out.0.weight": "6d2d0b9118e45067245a59ccaf076244fc29e6c61b84df559b60324935ee3274",
+  "image_refiner.base_transformer.layers.25.attn.to_q.weight": "b199a7964e3fff8d5c12e2acaea81fe833e3c2d94013d201fa70947b1dc4135e",
+  "image_refiner.base_transformer.layers.25.attn.to_v.weight": "b4ee36b2ce16ba882c0ec08be8a69d4311025032284c9c47ecf428186accdcb4",
+  "image_refiner.base_transformer.layers.25.feed_forward.linear_1.weight": "e39dad5b652af4d041fa7ad5317a8e27ef67d13bd014a02f14909051ca440938",
+  "image_refiner.base_transformer.layers.25.feed_forward.linear_2.weight": "bb40cb0bab208f1e3029514101fe2c34e5b86c18a43ec666fac272d5bbe63ec8",
+  "image_refiner.base_transformer.layers.25.feed_forward.linear_3.weight": "47ec657d743bdf7e2af357c19de47d5d133c8e659056ac107b163a6c01604d4d",
+  "image_refiner.base_transformer.layers.25.ffn_norm1.weight": "fa04078661f183874f3fc40e44504c5c92929edf0e1817bde4115bb47e166144",
+  "image_refiner.base_transformer.layers.25.ffn_norm2.weight": "e6e88894371e81a629f5ade2256faea5bd197fa16602488fe378f00689e57a56",
+  "image_refiner.base_transformer.layers.25.norm1.linear.bias": "008de1fb501578f092ff59fa05018dde647f0c96158dc606e249cec791db2fd9",
+  "image_refiner.base_transformer.layers.25.norm1.linear.weight": "535fdef56c20597c35f761681bb9a415d6c3425c68cdde45c1f393740d7ec8fb",
+  "image_refiner.base_transformer.layers.25.norm1.norm.weight": "ba3e365ccd9b63737dba81b3c4114409a8d816439152fa20b9fdffae7f0886f8",
+  "image_refiner.base_transformer.layers.25.norm2.weight": "0336c51c4ab9af241bf9da00d863f1fb44615a87484efd48faaaf6e60955b969",
+  "image_refiner.base_transformer.layers.26.attn.norm_k.weight": "6f34e5b771ab8ed4ce29ffec715292e360654779e599bd0fe72fc2d8339c6f95",
+  "image_refiner.base_transformer.layers.26.attn.norm_q.weight": "5d1181f865ce47b5cc2ab83ab4adf3ac4c49427c7a694c0d924157c6b157a4ab",
+  "image_refiner.base_transformer.layers.26.attn.to_k.weight": "9313fda5e0e32da6774b8198963d98d5d2723e3c3dd8338cae53ecf5e0f20704",
+  "image_refiner.base_transformer.layers.26.attn.to_out.0.weight": "7a492829543401e552c92d4e0bc67fd0a959a5821b9815d988b83fc95196e208",
+  "image_refiner.base_transformer.layers.26.attn.to_q.weight": "3076a27cf297e02d067a7da2daa3ad56beb9b2dae6bb5bd2758384a28149e8b8",
+  "image_refiner.base_transformer.layers.26.attn.to_v.weight": "212357daa3eda7683e8258c85d9dc7a6648b9b47b872d0ca84b8f918d3d36373",
+  "image_refiner.base_transformer.layers.26.feed_forward.linear_1.weight": "992df65b5a5d52f867a08fb3b921f69361cb94e5cedf6af1a536c347b28063ab",
+  "image_refiner.base_transformer.layers.26.feed_forward.linear_2.weight": "f8c8e03692e1ba05e752e38f574ba44ee0014318400c43da14a552c66918be52",
+  "image_refiner.base_transformer.layers.26.feed_forward.linear_3.weight": "78a9cdb3f4cee154ba331058d78ed7a5b5343e226248dca89f1e450187c6f00b",
+  "image_refiner.base_transformer.layers.26.ffn_norm1.weight": "d6a664c52540858bc0a6eee9259846540bcceb1b92f2b3f9a51adbbaf2d5d1ae",
+  "image_refiner.base_transformer.layers.26.ffn_norm2.weight": "26a737cabd7942b0ffafe43ae37adcaf97b86da6d0972c082dcb326e6f940738",
+  "image_refiner.base_transformer.layers.26.norm1.linear.bias": "d678630ce076fcbea9942f518b7d532b7a0c9d915cbbe128354120a0f8f42d89",
+  "image_refiner.base_transformer.layers.26.norm1.linear.weight": "4182da847672593321f80d97e7a5612bb34ed6b99e28120a09057a68f8536148",
+  "image_refiner.base_transformer.layers.26.norm1.norm.weight": "641cb8d1831e0f3192f289253c8a13680b716ae1cdbf67b2e57cb832978e6dea",
+  "image_refiner.base_transformer.layers.26.norm2.weight": "4e7beca14bdfac71ddd436dd8d170573427a323ed6e0302cc8d33c4b30823254",
+  "image_refiner.base_transformer.layers.27.attn.norm_k.weight": "7e574ba5bc5fde6bbfcb8fbd078b0801cda147fd75f45f6b5738badfdfd08690",
+  "image_refiner.base_transformer.layers.27.attn.norm_q.weight": "ae9244f21b21fc41abd10dd9f8ac6c8002274078cf00fbf195bd7b295dbbbabb",
+  "image_refiner.base_transformer.layers.27.attn.to_k.weight": "cca9740228aa8e6088b50aa02c8c04012fe06fa78630ed58fe1d4c24c38b451b",
+  "image_refiner.base_transformer.layers.27.attn.to_out.0.weight": "f642bb7a1188b1a2a4818b11867a8bd879c24e1b12a3895e1ae013301f08f9e5",
+  "image_refiner.base_transformer.layers.27.attn.to_q.weight": "9551fc75bc6edcba15bf37327cd996ebc6bc9804e467e0e46e9b968c072418e6",
+  "image_refiner.base_transformer.layers.27.attn.to_v.weight": "446131d81d22c1edaf52e7d22563663326557015e7141ae820f0b395cde459f5",
+  "image_refiner.base_transformer.layers.27.feed_forward.linear_1.weight": "6765a3e46e414e3acb5dd0bec0e7ba9c02a296a4d276471c651d246aa18e631f",
+  "image_refiner.base_transformer.layers.27.feed_forward.linear_2.weight": "b807c3337f6179d4232fb63d3f61f2123c23a793f6549446b266a50288307764",
+  "image_refiner.base_transformer.layers.27.feed_forward.linear_3.weight": "4a41633f7f57a1a3330b27a2411f5dbac7a884fa5ccc83708b22c1166a482731",
+  "image_refiner.base_transformer.layers.27.ffn_norm1.weight": "a2cb7c60adf6dd6677a1b509c6ae01dadae8efdd0d4a02bcb2c6439d96eee553",
+  "image_refiner.base_transformer.layers.27.ffn_norm2.weight": "1f7eb72eb07bfafdde1bfe436837ba6eec7d20988fca01bf03b668614e508718",
+  "image_refiner.base_transformer.layers.27.norm1.linear.bias": "f5bc11e9061252fbf0947bedc65b237e600abc9f451826c0efc6bd80053701a2",
+  "image_refiner.base_transformer.layers.27.norm1.linear.weight": "732645655f43337df1b6eceec2b0a6329ad413930965c842a3dd4d00dbe275d3",
+  "image_refiner.base_transformer.layers.27.norm1.norm.weight": "bf014d4941b9684a7e48933622747f2e8fb9a6cf7ca6eff256b8936ac4eed442",
+  "image_refiner.base_transformer.layers.27.norm2.weight": "2b7c72ef9c30298f16f39e4124e8c82548b972c92fb60c0c03cf292f2cb8336b",
+  "image_refiner.base_transformer.layers.28.attn.norm_k.weight": "fc6bd6471c053f64cddfc15d757bdeab5b11203c054e46120fd92b2c126815d4",
+  "image_refiner.base_transformer.layers.28.attn.norm_q.weight": "2361eac9fb0be488b2d2144c539364246c0744e9b13a5c4428b5ba7c41923dc9",
+  "image_refiner.base_transformer.layers.28.attn.to_k.weight": "7b27c64dfee598cbabb74d0bb498fd3736dd2a33b44ed97ee02d79d0f021b510",
+  "image_refiner.base_transformer.layers.28.attn.to_out.0.weight": "e0217108560227bfcf3bf87bea19ad003c420ac2fb225f17a64d14c8673d2c0e",
+  "image_refiner.base_transformer.layers.28.attn.to_q.weight": "3e39481e090e4dbcb797d7796d632ba605b0cd24a2fd104312319c7ad72827fa",
+  "image_refiner.base_transformer.layers.28.attn.to_v.weight": "3e431f3caef8ab63538e707e19c142224771886b17d7f36feb101af51b6eacca",
+  "image_refiner.base_transformer.layers.28.feed_forward.linear_1.weight": "2e5c74366e2c6a180dde0cf4f4aa63f415792692c38221e83dca096acd6de2f5",
+  "image_refiner.base_transformer.layers.28.feed_forward.linear_2.weight": "b8e2d3ebe1a40bd4b76f330e3f7b25e240965b933266f23f8cf6276d8d6df2a5",
+  "image_refiner.base_transformer.layers.28.feed_forward.linear_3.weight": "df011ef45ecae3967ce3613d688905906748cf0bbe0be415ffa2fcb4e2e00bd6",
+  "image_refiner.base_transformer.layers.28.ffn_norm1.weight": "8cdc37774e3cf75a7fe7fa5a8f94e93eb7b73de6417d208609a2a7a66dce9da4",
+  "image_refiner.base_transformer.layers.28.ffn_norm2.weight": "b367ac22431a3d0b7dda27e1b5063ff54b5354498318c9f2605e4625277b272d",
+  "image_refiner.base_transformer.layers.28.norm1.linear.bias": "898afd30a921421c7108551ccf9049436dc3acdae7ef91d6b8ecc9565fdb640b",
+  "image_refiner.base_transformer.layers.28.norm1.linear.weight": "7f48d89759d48c71b1c213bd62d7bd7b810f339af75bdf4c06cbdb0fd7c95ae2",
+  "image_refiner.base_transformer.layers.28.norm1.norm.weight": "6defa2b9f9d8711ce4b16d50cad75c75c1e91e7325872c129175cae96f4d8aa2",
+  "image_refiner.base_transformer.layers.28.norm2.weight": "3e823ca69963d3619af1b6e226e7444932efee75bf7f00394f5ce6bcfa6b415d",
+  "image_refiner.base_transformer.layers.29.attn.norm_k.weight": "5fa3395a841197ca8ff8ff09254339cd64fc5ac7d7056a75a07663e75ef95a70",
+  "image_refiner.base_transformer.layers.29.attn.norm_q.weight": "1b08072fb3ee54b039a45032ee11df06480cc6021e08222b284347088115def9",
+  "image_refiner.base_transformer.layers.29.attn.to_k.weight": "f24da63c86be7a440d0de35e21560716b1b802cd232263f4c740f27130101dde",
+  "image_refiner.base_transformer.layers.29.attn.to_out.0.weight": "f09389f06d0dac3092a772e1f37d0bac0bdd6ff65fed87bb878b394bc62bd464",
+  "image_refiner.base_transformer.layers.29.attn.to_q.weight": "1d1eabacaabd17583e81b929e0fee3b9f6ffaf5b943fdaca9fe5ecae4dc58940",
+  "image_refiner.base_transformer.layers.29.attn.to_v.weight": "4920a7cee69cbf4ff0e15bbdb91f7eece511cd641c0d3a225bb3711ea0220738",
+  "image_refiner.base_transformer.layers.29.feed_forward.linear_1.weight": "4260cfbd9fb6272a732a8d5873cfdb3a58383175d0201289a2f8d466c1e866d3",
+  "image_refiner.base_transformer.layers.29.feed_forward.linear_2.weight": "4c3be48b7f6d3ccbdbd1748c3334bac7a02c86a34a961b5c257f49c2d3839212",
+  "image_refiner.base_transformer.layers.29.feed_forward.linear_3.weight": "65dcecee3471f932c290abd487088263623e8011939fd4795bc01a13931ab4a2",
+  "image_refiner.base_transformer.layers.29.ffn_norm1.weight": "6f9f2217afd87da7827afc403731a366aaaab162aa39ffac90dcc779edb6a0c8",
+  "image_refiner.base_transformer.layers.29.ffn_norm2.weight": "11a3f9a2a6dca48bdd26b00410a53ceb8f027ec694d88283d44ca828fbd151ae",
+  "image_refiner.base_transformer.layers.29.norm1.linear.bias": "ddc591240c707d5b31c372d921b88b0683da83ae77e634bca0dd05723b1debb1",
+  "image_refiner.base_transformer.layers.29.norm1.linear.weight": "ece7b1fa468943f94f3758785731ad3b3fa647e798da870fd2c85cb752d7a5eb",
+  "image_refiner.base_transformer.layers.29.norm1.norm.weight": "bbe967679792425090c5200d161279d91af5ec49c2de6cfd1bd43e7ffd1cb7c3",
+  "image_refiner.base_transformer.layers.29.norm2.weight": "6f5d7a2e5e01cfc38352d5901b95eca336c929acc5519884f7086b0869c35273",
+  "image_refiner.base_transformer.layers.3.attn.norm_k.weight": "c9524fe908034a4d1eb53d092f65aa9b56a424beae32fa8c7c0dad8058653eaa",
+  "image_refiner.base_transformer.layers.3.attn.norm_q.weight": "3455a614921a66e5cc2b90cc99840ded47839c2574d55c7c466ea7e802d20fe2",
+  "image_refiner.base_transformer.layers.3.attn.to_k.weight": "c3c3d57a3825d3ebcf46e33f9d794d0c7e430bdf41a08b62ab6526b7eca2e789",
+  "image_refiner.base_transformer.layers.3.attn.to_out.0.weight": "8e1d3eed94828dcce00b8b3917375a849cf22d842164b5d5304f2be590e447ee",
+  "image_refiner.base_transformer.layers.3.attn.to_q.weight": "2b577400e3a8ec3f6f5ae399200a48761174a9d723f4c8725e66c0139acafe7a",
+  "image_refiner.base_transformer.layers.3.attn.to_v.weight": "0b2aa791c8ae1a626bc0e94536533b6b877b8b941b11290fade48ab3e1798bf9",
+  "image_refiner.base_transformer.layers.3.feed_forward.linear_1.weight": "d545fc5ec185dff23ed0388eedebe6fb8408eb52bd7cdd1f367d2ea0c009fb48",
+  "image_refiner.base_transformer.layers.3.feed_forward.linear_2.weight": "29a785787dd59289a4a72972333263cb052b2fd39459fe3fc3770d9adad072a3",
+  "image_refiner.base_transformer.layers.3.feed_forward.linear_3.weight": "37a9617b24ccf8dcdc63f4dbe11a969e4c1b24dcf0ee9c2a265b1243aa57cd0e",
+  "image_refiner.base_transformer.layers.3.ffn_norm1.weight": "0b0114684e6d8ab086ebdd5745f5209fc6d27e5badb77088d1fef36e2bf066d8",
+  "image_refiner.base_transformer.layers.3.ffn_norm2.weight": "6329aaacb92f31b14e7f24c925ef12529384408847f25978e164ab909824c633",
+  "image_refiner.base_transformer.layers.3.norm1.linear.bias": "e334473fd509c09a59dde5cbcfad68bba30e3fa0a9bd3b48fdac30a6969a1285",
+  "image_refiner.base_transformer.layers.3.norm1.linear.weight": "6c8927b68bb1e1366b58cde2cef2f8fb953c2693483ddbfa9e57ffb4e3993402",
+  "image_refiner.base_transformer.layers.3.norm1.norm.weight": "ef930dcc0e1bc7d0c9c6b8f863389a33dab9cb61e115397fed86ef80e5ad1f62",
+  "image_refiner.base_transformer.layers.3.norm2.weight": "6da9543ba9d9bb2967ca221b787a1b00f1ee6eaa88ee2a60b29615edd788f02c",
+  "image_refiner.base_transformer.layers.30.attn.norm_k.weight": "395180fb5eab449a60b1b2b04fa569f43ae11857f0d02b65e1fc533755a24e6c",
+  "image_refiner.base_transformer.layers.30.attn.norm_q.weight": "d8b2638e0e380a7cf123465246e6584e3db8069d3c23414508d2d1d067c6471e",
+  "image_refiner.base_transformer.layers.30.attn.to_k.weight": "02cbda675526348c58c4c2807428d5487fdde557a3a22fc99eb04119d7c02d99",
+  "image_refiner.base_transformer.layers.30.attn.to_out.0.weight": "6fa5f774426717dfbfe28ad9189c97ebb6707d5116f51975bf0c2dc522f08bca",
+  "image_refiner.base_transformer.layers.30.attn.to_q.weight": "517b83d0163bf69832389fb730243dd59b4a64e4de411afa6c824661e4cd9134",
+  "image_refiner.base_transformer.layers.30.attn.to_v.weight": "f2be9ccf7b38b5e516f95925aa88bd45f1348f1149b92773f098e894f273fb29",
+  "image_refiner.base_transformer.layers.30.feed_forward.linear_1.weight": "76e91692a55ad5efec1c9c5b5b0d64859e9425e651a562bb74aff4cfd2e679b8",
+  "image_refiner.base_transformer.layers.30.feed_forward.linear_2.weight": "4703cd559e141a15520cbe93c443d9eed677c519edeefa1a2962170dc6a60849",
+  "image_refiner.base_transformer.layers.30.feed_forward.linear_3.weight": "edd8a7cfc8e5bbc3dc7d4d238f70854bb4c987c6ece8958a88caa2a556bc553c",
+  "image_refiner.base_transformer.layers.30.ffn_norm1.weight": "be340f00aaebd4775481942ddb041a28d9dad4dd68547dc18fbf12f94c196e40",
+  "image_refiner.base_transformer.layers.30.ffn_norm2.weight": "c8d5fb72abd21a9fca787513c6cdf95246ab836bfd61c77115ca5404c4fe5901",
+  "image_refiner.base_transformer.layers.30.norm1.linear.bias": "7c16cca95b276e6601974d490bd65c482db8634324b4f4210e30b8f61059566f",
+  "image_refiner.base_transformer.layers.30.norm1.linear.weight": "0a2e613ae19d39904d9c165fef981b5590c489e875cd73b9f3ecec782c0a6f2a",
+  "image_refiner.base_transformer.layers.30.norm1.norm.weight": "4b8fe14950cd594228b8f0cf724974308c26eaae52bdd27aa7e74b3911e49d94",
+  "image_refiner.base_transformer.layers.30.norm2.weight": "a4b62c7ccf7eadb23fcd6e4f0ee4fa20281d4df02caf7b0c598af99a38197ca8",
+  "image_refiner.base_transformer.layers.31.attn.norm_k.weight": "d700595bb2b08a72aef21921b875f587ba83ff9217540410a33a0f263bf70c0a",
+  "image_refiner.base_transformer.layers.31.attn.norm_q.weight": "58fe97f5a600e44f5800584faf84c93072e1098b449beaf20ba07eea36fa4838",
+  "image_refiner.base_transformer.layers.31.attn.to_k.weight": "b5712e3fa2b98a1286ec6719a347452b79c57936ad0475c7b6d16ca6e7ff91e8",
+  "image_refiner.base_transformer.layers.31.attn.to_out.0.weight": "c76488a0f6a690782568e9d1b44a33e3b0a67779df7ad3aa242575dd212c6504",
+  "image_refiner.base_transformer.layers.31.attn.to_q.weight": "f64a15fcf75a8b8fc5de796ac437604bb405f5e205f514c260060a74cac9c2aa",
+  "image_refiner.base_transformer.layers.31.attn.to_v.weight": "b8ea07657477052ababfa7e7652e7ca56fb07fee1eccccff86ea8c51b53c7d8b",
+  "image_refiner.base_transformer.layers.31.feed_forward.linear_1.weight": "b8969feb57c3acb46b122baea236eb046ad512abfeb09a37ebc26157b48b98f2",
+  "image_refiner.base_transformer.layers.31.feed_forward.linear_2.weight": "fedda95866ba91d00421b27aebd1b2cf2c1b23a27ed382c2b13ae670883801d2",
+  "image_refiner.base_transformer.layers.31.feed_forward.linear_3.weight": "337e2b71afbf63a9aeacfacb0cc7ccbb4f6c03e032eb628fd1ff0ba5b958f4de",
+  "image_refiner.base_transformer.layers.31.ffn_norm1.weight": "158745c91179e72b4fa1a4cc22fa194efd2dcabeb254c133eef3eb0d18b11af8",
+  "image_refiner.base_transformer.layers.31.ffn_norm2.weight": "e87d5e7ce4cb3f042254d64deedbdf40079fbe4aef8f80ce25138fc7efc9e816",
+  "image_refiner.base_transformer.layers.31.norm1.linear.bias": "3ea01d193ba813dab8145a503f21d1099f655604978ee66545d94ca80be226df",
+  "image_refiner.base_transformer.layers.31.norm1.linear.weight": "3ced36bfdf6e7d0ab4ef9f6c8e95e822f533d9cd61dea1894db9d14f4c53355b",
+  "image_refiner.base_transformer.layers.31.norm1.norm.weight": "5c17a698143d3425466f60c59ff020dd634bf0b01daa3a07b1b16175a47a8727",
+  "image_refiner.base_transformer.layers.31.norm2.weight": "f86633eb7caae43ab96f9160957ea7adbc9d69e4dc1983677e7cde2c350fa637",
+  "image_refiner.base_transformer.layers.4.attn.norm_k.weight": "0d6a688537e7aa534c8f407eca446a2c0cc4076447aee2f2b1ad05635c57b407",
+  "image_refiner.base_transformer.layers.4.attn.norm_q.weight": "5fc550a5bf088b8a4d990435ad163665b8af7c4c8a6b09a97447900463a01f93",
+  "image_refiner.base_transformer.layers.4.attn.to_k.weight": "06a0df5b8c8ac5f0deb02d8b472441fa3e19ac52339a53564f03bd9e552c9421",
+  "image_refiner.base_transformer.layers.4.attn.to_out.0.weight": "8421dbf5010c9060ded58587770cafd0055d05eb78fe31942ecc43d4d13ae071",
+  "image_refiner.base_transformer.layers.4.attn.to_q.weight": "ad7f33deef9e41feda2300467ab45ac342c7fe1dd3af70fb57a8645c2934630c",
+  "image_refiner.base_transformer.layers.4.attn.to_v.weight": "29e70b179e58270675d82523a467abea2ff5f8331275265a3dd8c5def1b1990b",
+  "image_refiner.base_transformer.layers.4.feed_forward.linear_1.weight": "530ed4c96fff6369ff95767a4a2bfab62da685d43ab0a552ba9f5f00de9b3a9c",
+  "image_refiner.base_transformer.layers.4.feed_forward.linear_2.weight": "879f57b95fceb42b92f80ce59ad000fcb4d0060a9d7e0438f02b863a09029402",
+  "image_refiner.base_transformer.layers.4.feed_forward.linear_3.weight": "187f5a60b584402871db1ebf46c0f50d42e3a1ecd010645c7794987037a53a70",
+  "image_refiner.base_transformer.layers.4.ffn_norm1.weight": "42234369208d5f51d0c6872f5e4239563d20dae1498c9dd8638c9486b46e8861",
+  "image_refiner.base_transformer.layers.4.ffn_norm2.weight": "8449eb45612580109e7453c0266a6f56199efcadab2fd22dabeb28dbb7a560ec",
+  "image_refiner.base_transformer.layers.4.norm1.linear.bias": "cc9ebfe76070edcc73fbbd559ddc5ecf10b94e9039db9622db1c0e9f16c1b7b6",
+  "image_refiner.base_transformer.layers.4.norm1.linear.weight": "5adb3f1466807aff5897f66df7f7cce2fa2d5f952033614cda9d27701086a5f5",
+  "image_refiner.base_transformer.layers.4.norm1.norm.weight": "44673af81845abb323324ab2cae60d63f7bd26652a746282f0d12d5bb46bb6b9",
+  "image_refiner.base_transformer.layers.4.norm2.weight": "78453ce56c20d5e6269c7c4137e4e894b7fb2cc61c7e3ba4460ec99427627570",
+  "image_refiner.base_transformer.layers.5.attn.norm_k.weight": "c75b3ae09d82812164d076deaac478b5dfb01dfb0cd7c601c464edd804cff5d8",
+  "image_refiner.base_transformer.layers.5.attn.norm_q.weight": "82ea3b4983393b75e8e249e031e6b849bcdf5f477dfb3046db5de6631b62d158",
+  "image_refiner.base_transformer.layers.5.attn.to_k.weight": "0d819740930fc71aef5f64e8b51e6e91a6ec273ceb46d5fdaf352f940267d2d0",
+  "image_refiner.base_transformer.layers.5.attn.to_out.0.weight": "5d9fa695e3941541b0406705d577f746c69b0594276bdcd6883aa599d5595860",
+  "image_refiner.base_transformer.layers.5.attn.to_q.weight": "39dd3bb55411f5ee20b3e6b5510f3a97be70641ad3b9c262c122a1d2ce768ba4",
+  "image_refiner.base_transformer.layers.5.attn.to_v.weight": "5a6625ae3f65e4557d6e9e99d9a3c8c656743a8120dcaf7bd3e326a83da7274c",
+  "image_refiner.base_transformer.layers.5.feed_forward.linear_1.weight": "0b110d94a212221a4471e0993eadcdfec38a6cf1de22056c7939edc6387f7dbf",
+  "image_refiner.base_transformer.layers.5.feed_forward.linear_2.weight": "a33c3d306895fca636fe8319967116efb7e99174319681a087cba51de90a2112",
+  "image_refiner.base_transformer.layers.5.feed_forward.linear_3.weight": "30a50bb2c820364eb243bd2e61a4dbb4c99cc0786749d64b16e54cc1bbcf5c9b",
+  "image_refiner.base_transformer.layers.5.ffn_norm1.weight": "c87fd7d578537c5ad2f240378f18776bb9a12798882f8a7d3c156c0885ed1354",
+  "image_refiner.base_transformer.layers.5.ffn_norm2.weight": "6022115399bcfa0139624d1aa5d358d2efd7f75971cf724e41078b5696ab2a1b",
+  "image_refiner.base_transformer.layers.5.norm1.linear.bias": "7c6e7dedcdb7039778811f2b7e7e91ea06657023af2f51adb644df80bc3deb63",
+  "image_refiner.base_transformer.layers.5.norm1.linear.weight": "fb0134588f405bb3dc4246982d48d6a2083928212e1aae921bb138bacd2e49b7",
+  "image_refiner.base_transformer.layers.5.norm1.norm.weight": "0c649b9a70be0382042d5cd412f31c58869c4cbb028746b74d39d8b62a7b2990",
+  "image_refiner.base_transformer.layers.5.norm2.weight": "d3216a95255f7ea15c8d04a02782ad599f17d123623410dac38a2b0b1e98fdc7",
+  "image_refiner.base_transformer.layers.6.attn.norm_k.weight": "ae7310dc22c0f18fe0e65f6605f2623006b4b1d48ff0f02f8a0e64072629e365",
+  "image_refiner.base_transformer.layers.6.attn.norm_q.weight": "e0dfd9c28acc60eb5f8f0a7c9a9f5faf2aa8038ee22ff4e3904d3b95e72c58c8",
+  "image_refiner.base_transformer.layers.6.attn.to_k.weight": "3cc72df34a6c2791ef116d500058949180c8ed2458304ccc4d4c6d25998357f9",
+  "image_refiner.base_transformer.layers.6.attn.to_out.0.weight": "71e3c9badd9d3f80d32ea12b91e18485a85908521145a73e2d211c5a03ae1abd",
+  "image_refiner.base_transformer.layers.6.attn.to_q.weight": "50458d86cbcfc1d2dd9bef67abb63fedf7cd13662adf65a2d200f83c0d5c9e61",
+  "image_refiner.base_transformer.layers.6.attn.to_v.weight": "4079226ac3ea8574c02e11a3ef08d8300ea942eac5d2f79f9ccbce8d3989d3fd",
+  "image_refiner.base_transformer.layers.6.feed_forward.linear_1.weight": "333c63eb96307448a7eee50ea53784b2ef29e413dd3710c0cefd5f561dd43f89",
+  "image_refiner.base_transformer.layers.6.feed_forward.linear_2.weight": "bb2470360489756d7d5400553474f0b869262967ae5f638a9f820b247f1670f0",
+  "image_refiner.base_transformer.layers.6.feed_forward.linear_3.weight": "87e5007648b0d6a821f134e6228e72423735cae123209a082c230f013867cdea",
+  "image_refiner.base_transformer.layers.6.ffn_norm1.weight": "789f0e4e6098064e876e2ff3d57f87ef67bc69fe4dc0ea91025399b62a782395",
+  "image_refiner.base_transformer.layers.6.ffn_norm2.weight": "e6f70b5a22771e84df5213747abb3a52bbb71fd5eb34b3f32eeb37716887a678",
+  "image_refiner.base_transformer.layers.6.norm1.linear.bias": "6aa83416a63eb8c092aaaaa17e7582fce3bbb6d73dbe46f138765e05fb8e0b07",
+  "image_refiner.base_transformer.layers.6.norm1.linear.weight": "087574fcac3f564a56047fed20cb93b7e965e8b6c75ea08b913e3c567f2682ef",
+  "image_refiner.base_transformer.layers.6.norm1.norm.weight": "78538ae68c76797a628b197715573a2c381a0ec129494ff24a246e28320bf62f",
+  "image_refiner.base_transformer.layers.6.norm2.weight": "10f605771665ee9322688750e1e13ba82f684c468d3c10d8f354bc48a4912253",
+  "image_refiner.base_transformer.layers.7.attn.norm_k.weight": "076a629aa63cef54c62a83241cf3c9c5d29284ece032878563b3c2ca8ed08120",
+  "image_refiner.base_transformer.layers.7.attn.norm_q.weight": "fabb10cac258aa6368be8a960146cbcd3e6a8a34f4c9912842cabf638e88cb25",
+  "image_refiner.base_transformer.layers.7.attn.to_k.weight": "1e4bbfb7109b2b07e7732ce6ea217f86844985b51289f09f5134c65bbe5ad95c",
+  "image_refiner.base_transformer.layers.7.attn.to_out.0.weight": "2204a8a7a3b7c7492cc9458f10ad900bb24ff50811741db49c73d51219b0ab6d",
+  "image_refiner.base_transformer.layers.7.attn.to_q.weight": "201b2e2f7df1e2e5bcd0dfabf988b53834d4a4f02daa2894529bf12b26e3c8bc",
+  "image_refiner.base_transformer.layers.7.attn.to_v.weight": "6ce9956227974016a4c732861c774ec18d755bd2dd0bab424dd2aeda408a6499",
+  "image_refiner.base_transformer.layers.7.feed_forward.linear_1.weight": "6c0d290d10b26971573e690220638dbb66ecdf5ad030017b029efd7dff3a45d5",
+  "image_refiner.base_transformer.layers.7.feed_forward.linear_2.weight": "2de73cd40b210c56b81b01cd432aca9e15c451681f37d105e9014373b97bd3c1",
+  "image_refiner.base_transformer.layers.7.feed_forward.linear_3.weight": "fd6c2242e13be26efaefa28e791402254efccdd7f40f25058887799c16863246",
+  "image_refiner.base_transformer.layers.7.ffn_norm1.weight": "3a8eb9e2ad311195d516b60c01cb6f5c34ccc63d56f8e8e830590ab824c72585",
+  "image_refiner.base_transformer.layers.7.ffn_norm2.weight": "6bea33167945e81f5412f23ea468e2a0f5a4a3960812dcf38cd3bca67c375910",
+  "image_refiner.base_transformer.layers.7.norm1.linear.bias": "dc6eec2c6fe8f9b4a869cfe4730f062ce03dc352291db103d256d237e53758fa",
+  "image_refiner.base_transformer.layers.7.norm1.linear.weight": "1c7b9d37673e47b3d766fe5c1b5df77bec76af410c70b0c2b53b14f0e14e9c0f",
+  "image_refiner.base_transformer.layers.7.norm1.norm.weight": "7ec3a7c32ff171cb5c463420e728f09852084a01a50251f36f7b0c4ab2ce1282",
+  "image_refiner.base_transformer.layers.7.norm2.weight": "ea241655fd2aa75961e67845ae2614659f81a2619594c559130b857817d9f73e",
+  "image_refiner.base_transformer.layers.8.attn.norm_k.weight": "1b073e18b5f86ee7dd8a78d444751f627977c229cbfe84ccf612d44a5b9b30e5",
+  "image_refiner.base_transformer.layers.8.attn.norm_q.weight": "3f99177edbfc41c8e4bc036c3451562d9787af5d4a781f42d9476f18a88e9fc8",
+  "image_refiner.base_transformer.layers.8.attn.to_k.weight": "e9313d2804f1709665fd13fd3833fcbfa5b32f21b907b281ddd99f2d3f370848",
+  "image_refiner.base_transformer.layers.8.attn.to_out.0.weight": "5ba7a8fb70ec4b5a775fc3c5be0e4b8c363ae203691f7d0082260476dd625f69",
+  "image_refiner.base_transformer.layers.8.attn.to_q.weight": "ddc8622244f5ee09fabd881d1737fa5ba76a81f88a64540ec2b10592d4903a6d",
+  "image_refiner.base_transformer.layers.8.attn.to_v.weight": "4fb4efdceb8753db3d85c755f91b3b0b6641f467326d3b50dfc39b0b3d503152",
+  "image_refiner.base_transformer.layers.8.feed_forward.linear_1.weight": "4ef819f495f1d1eeab276adea068120f851ac738a9e5ffa1bde4651d9926af86",
+  "image_refiner.base_transformer.layers.8.feed_forward.linear_2.weight": "f39a4b83611cd88306a604a3a9d20307c9a65166551af0d3f6a63a6e27c25910",
+  "image_refiner.base_transformer.layers.8.feed_forward.linear_3.weight": "7c5e0844e67b78c15b51f64cb6c694743fbcc4b205f9fa3ffc2aefde53127e9e",
+  "image_refiner.base_transformer.layers.8.ffn_norm1.weight": "21b2c231ffafa3e91088bbb6f26e5a8a731a47a54898b52a91512d1c5cce72b8",
+  "image_refiner.base_transformer.layers.8.ffn_norm2.weight": "f1ba8bc1fe2a12710f1c2ba9797e2a43dafb051cc4e6e8bec09e25f059069478",
+  "image_refiner.base_transformer.layers.8.norm1.linear.bias": "dad49dc2725d3f8644af24d4950736eb72cd571f0f327e0a2e03f27a6dfabd0c",
+  "image_refiner.base_transformer.layers.8.norm1.linear.weight": "3068974aaa1490f3baad46626351accf33ee0d9882b36f18225d8d1aec3ec168",
+  "image_refiner.base_transformer.layers.8.norm1.norm.weight": "77ee75c28066d0a6e9b7deb88c3f7a709638aed167acb8afd3c2c12a9f30250a",
+  "image_refiner.base_transformer.layers.8.norm2.weight": "c7710e0e1e851bf36cacf9bf9240a3778ee8ec46314f6ed2f931bb7b7cdd7320",
+  "image_refiner.base_transformer.layers.9.attn.norm_k.weight": "4e2b697fb381d0ff444f08383885015cb7e16e6f73af08c263c588f6f2cfb601",
+  "image_refiner.base_transformer.layers.9.attn.norm_q.weight": "806695bc14983155ccff5eae55efae322a12b1abbfae4df6faecfa4ba734a620",
+  "image_refiner.base_transformer.layers.9.attn.to_k.weight": "3b64aa2d341783e5cbdc29ecddeefd439cc5dcd87e40724e0db733e5837510bf",
+  "image_refiner.base_transformer.layers.9.attn.to_out.0.weight": "02928d71bbd6e96f886245269876279e393e0f546c9c85dbfe62cc2858b76a69",
+  "image_refiner.base_transformer.layers.9.attn.to_q.weight": "33732c7bc4ae6869cf94598cd2b104fd5a022272d487832025f9c772c92de98b",
+  "image_refiner.base_transformer.layers.9.attn.to_v.weight": "4390f7d77e570e0808c850a248ed767076b5171b0193901f334e737830a8468e",
+  "image_refiner.base_transformer.layers.9.feed_forward.linear_1.weight": "08fae091f5e2928197d2814027bbe8c6afb0c7b5cd23b41c355b60ae12642ee3",
+  "image_refiner.base_transformer.layers.9.feed_forward.linear_2.weight": "df4fcad0afbb5e70d0155498daaa77dff1c8734a72a231f09c6b252516e73bba",
+  "image_refiner.base_transformer.layers.9.feed_forward.linear_3.weight": "c4a4a2d465a5436e45af5fe33b1520438132ff4cc3cb2cee8436c45522ae9af5",
+  "image_refiner.base_transformer.layers.9.ffn_norm1.weight": "f07bca6ca52003ca8fde5a5ca866e876f61dc88beea9b1bceb46da79c92cf7c5",
+  "image_refiner.base_transformer.layers.9.ffn_norm2.weight": "949608f86801ba72b5f80a165d7f618b46088ab18d04f0ea20ece983c6a13b64",
+  "image_refiner.base_transformer.layers.9.norm1.linear.bias": "c0630e48139ee7ca91e3057caa745a866512234f7ba53dc5d6282907188c5b6f",
+  "image_refiner.base_transformer.layers.9.norm1.linear.weight": "758523130359fa33f9933a9e93f92df813568fa5fe07b5abe860d0459f30463a",
+  "image_refiner.base_transformer.layers.9.norm1.norm.weight": "8ea63b2c6bf956ba45f7d8a87a8789dc5e88a574c2765dc5ffe96cfc3a9b4beb",
+  "image_refiner.base_transformer.layers.9.norm2.weight": "8d1679bd8aab6ed749c6ed08ba63e8889985836cc78cb1e33daf4775095648b9",
+  "image_refiner.base_transformer.noise_refiner.0.attn.norm_k.weight": "70756459e0ad2776f094f6598b32fc62920a3139524ced2efbb03f2115fdc6c4",
+  "image_refiner.base_transformer.noise_refiner.0.attn.norm_q.weight": "48baefcc107520a90a7587786c718c70fd28c281e44d27d964f7b28e43b1ae21",
+  "image_refiner.base_transformer.noise_refiner.0.attn.to_k.weight": "3349a7f6b8d9e1c2c7fc4588527da51f2b1cda413d5052239e3af0a171066361",
+  "image_refiner.base_transformer.noise_refiner.0.attn.to_out.0.weight": "2f8d11d8c7e1f83076f611f09f489bb12a997f558a2b1ddb4322b8cfba65d2ac",
+  "image_refiner.base_transformer.noise_refiner.0.attn.to_q.weight": "3740f7f8d4f0869221dba3db7a4fe69644e1a4917ad0f8181f3c476bcaade3a2",
+  "image_refiner.base_transformer.noise_refiner.0.attn.to_v.weight": "3ccdccc695a1c03699d26a5de6c9232221a9682b6303c1fe0f95f16eabe93ab7",
+  "image_refiner.base_transformer.noise_refiner.0.feed_forward.linear_1.weight": "9a74c76857bb26d22330d46b3aa2932d821942e5df14bdc0b7f110829e1610c5",
+  "image_refiner.base_transformer.noise_refiner.0.feed_forward.linear_2.weight": "f9c521c2e3e662da9dd7ab029eadb49a0f018b7e4eb494c02540f18342f42121",
+  "image_refiner.base_transformer.noise_refiner.0.feed_forward.linear_3.weight": "464f27da6b4c88f4ca5fceb6478b08e4aad4bae97fcfa69eb9c8ff22588255bb",
+  "image_refiner.base_transformer.noise_refiner.0.ffn_norm1.weight": "5c40b81c5907950ff575df3d2ebb699563f3913f1f1187aeb241ccfc3137c315",
+  "image_refiner.base_transformer.noise_refiner.0.ffn_norm2.weight": "df36e799671b93d754d75470021606e5e3c9e4b4583396c050ec64385ac25b28",
+  "image_refiner.base_transformer.noise_refiner.0.norm1.linear.bias": "cb71b5decf1c2f377c9452f2f8c7eab9fd9823d39032f4cad2970c8589b7b082",
+  "image_refiner.base_transformer.noise_refiner.0.norm1.linear.weight": "69ee9dc814e8b338f371fa241e53bb8a27c8031956147cc7f54a8dccd4af4f70",
+  "image_refiner.base_transformer.noise_refiner.0.norm1.norm.weight": "318fcfbc19b30512c2fa16c28387446d8df3ddf3f516eb2dd1562dab896eca6e",
+  "image_refiner.base_transformer.noise_refiner.0.norm2.weight": "413aa00c5616bfbd68645709fea2d68f480cc8def579a866dd7fa50740e984c5",
+  "image_refiner.base_transformer.noise_refiner.1.attn.norm_k.weight": "f388658300738dfda5938edb711d0f03dec1ea81cebc735a74785d60d3f524f1",
+  "image_refiner.base_transformer.noise_refiner.1.attn.norm_q.weight": "b1d500866fde03495e5c3bc5dd03d5a5cafc60e3815018519afcddff728db7a0",
+  "image_refiner.base_transformer.noise_refiner.1.attn.to_k.weight": "e53fdc2ada57fb328728a21a5b36823392ec63e22c735997c5e47a9240dc0e85",
+  "image_refiner.base_transformer.noise_refiner.1.attn.to_out.0.weight": "0d838e434d90d0e6b8ca91f29e5f5b0832aa924d50c8d15336fb4b68902646d6",
+  "image_refiner.base_transformer.noise_refiner.1.attn.to_q.weight": "0b90556045c417f6d3bd887c73fab86cfc4915b6b0d5eeb6a4255825608d1062",
+  "image_refiner.base_transformer.noise_refiner.1.attn.to_v.weight": "62ed24ef3d92f3851eda868e2f9f358e5393bc7f4947b041403e3a135ec3b7f6",
+  "image_refiner.base_transformer.noise_refiner.1.feed_forward.linear_1.weight": "5047419bf61c7fbdd7d13ebd38fb7c8537e7d5823a3732ad8956c237c210f1f4",
+  "image_refiner.base_transformer.noise_refiner.1.feed_forward.linear_2.weight": "d2e27f803fd2f8e1bf42be1873d702adbfa37b3842c2b55c7fe9a08d484e2db5",
+  "image_refiner.base_transformer.noise_refiner.1.feed_forward.linear_3.weight": "2cc8ed6e15e09dbefb792d223752dcb9c6b2b234c4d31a89f8e416fcab3efb37",
+  "image_refiner.base_transformer.noise_refiner.1.ffn_norm1.weight": "740671eb5fbc4e748d18bcdb5f787c2a06f25b824352afe45b77b020c604bef5",
+  "image_refiner.base_transformer.noise_refiner.1.ffn_norm2.weight": "bc1ab00c46849d8a7af6bae913ac59c971d7d0492872822b6dc8c572fd36260e",
+  "image_refiner.base_transformer.noise_refiner.1.norm1.linear.bias": "e0351a142810c6531c5f6e6fc4ff9753c27b5285be6a8d5eb09f0cd0b814478d",
+  "image_refiner.base_transformer.noise_refiner.1.norm1.linear.weight": "2d2532f42259c7b22625e5df7517016f24e20ee7f1ffed5ef4562c6ac8637125",
+  "image_refiner.base_transformer.noise_refiner.1.norm1.norm.weight": "4be5518e91cd619dd936561cd5896042c46957fded12eaeb72ca23eb82bf3a31",
+  "image_refiner.base_transformer.noise_refiner.1.norm2.weight": "3cba4b25dd9660a322a704a5b4929df2a9895945e1efc7e0b3bf84597426dc28",
+  "image_refiner.base_transformer.norm_out.linear_1.bias": "a29e36af4378b96ba7ef3202532d9bf84bd266e3e5d121c4c73b4dfc9ab4dc68",
+  "image_refiner.base_transformer.norm_out.linear_1.weight": "97e92b85cbb6a0bbe0a8e523a3158e451a3f33f8655422875c6c98f2d5415481",
+  "image_refiner.base_transformer.norm_out.linear_2.bias": "f40b96812e060ba42adb6d037d93cbc1e595b3502d926895cf906852633cceb2",
+  "image_refiner.base_transformer.norm_out.linear_2.weight": "2a7e981d37daca78652c77e482985b3b906c60d73a5b92782d5bf2df0a414a57",
+  "image_refiner.base_transformer.ref_image_patch_embedder.bias": "eac9e853cf012b7b6e7ea97b978bd9b2871e6a89b88458a2753ff1e6594028da",
+  "image_refiner.base_transformer.ref_image_patch_embedder.weight": "6572f99f67b14ba0306467f0c3360c7551cc88ef3aa7b867d8f9a1130501b451",
+  "image_refiner.base_transformer.ref_image_refiner.0.attn.norm_k.weight": "48de5244b2fd1d5f8dae906d79213f5f6d1b389a9c96d5ab82d0ce06cf7aa432",
+  "image_refiner.base_transformer.ref_image_refiner.0.attn.norm_q.weight": "d420243cbc3353c80c1cec0e74091e34672a6ce28a866e75b03d436108ba7879",
+  "image_refiner.base_transformer.ref_image_refiner.0.attn.to_k.weight": "d502b90d1e2fd5fe0f6fb0f87fd4e772bff71abbf15def89b3c1361317f1deb3",
+  "image_refiner.base_transformer.ref_image_refiner.0.attn.to_out.0.weight": "0e0c3b7ae23114b2ba81875beca28fba0fe507b7edaf961e3033bf377fe13817",
+  "image_refiner.base_transformer.ref_image_refiner.0.attn.to_q.weight": "75fda07d612b61cb8d191253ad64d936cd7224756f184c25efdb53f5f810de19",
+  "image_refiner.base_transformer.ref_image_refiner.0.attn.to_v.weight": "267e3cf0876ca52c9e10add572ea0378c300b914c3af0d6d2234fc9463f37612",
+  "image_refiner.base_transformer.ref_image_refiner.0.feed_forward.linear_1.weight": "fef9d59e05594399ae49a9a37b5d0c64ac7f8c90dfd32953ab29c080cb1c42ca",
+  "image_refiner.base_transformer.ref_image_refiner.0.feed_forward.linear_2.weight": "88c9dc09699e6ec374bacb643a1e790ba6e222b618bc6da25977212968c4a85d",
+  "image_refiner.base_transformer.ref_image_refiner.0.feed_forward.linear_3.weight": "b4c993c4fe222d4e122a30b933412ed60aeed908b9ae57cfee7f6bf4b5ca178f",
+  "image_refiner.base_transformer.ref_image_refiner.0.ffn_norm1.weight": "8ccc924e25bb011eb308ba7bef1e8ce8f6051cc93f7353ab8cefd806a593b718",
+  "image_refiner.base_transformer.ref_image_refiner.0.ffn_norm2.weight": "2c8c165735d2a90b59e9ab01e48093afd1eccb5968c2d8c81eb1c5fb077b1d37",
+  "image_refiner.base_transformer.ref_image_refiner.0.norm1.linear.bias": "6d06d2e60b71d3b8a7e08217dc5e46a650a719d5ce98aba5a8e46a33dccaa2c5",
+  "image_refiner.base_transformer.ref_image_refiner.0.norm1.linear.weight": "056eb89a7dfc7237d92edc8b09b951b44f4261352356d05c6f79c3572e5650c2",
+  "image_refiner.base_transformer.ref_image_refiner.0.norm1.norm.weight": "30f58972fcf5852b78d5fee2514a6de7feaf41cee1f78fd360a5e58b1c7118ed",
+  "image_refiner.base_transformer.ref_image_refiner.0.norm2.weight": "071100c8233d6cd2cfd7cb1a1a7c1183a0e34b26a5b5d00ef6f4cd0657666104",
+  "image_refiner.base_transformer.ref_image_refiner.1.attn.norm_k.weight": "89d0d787d1ca9174501181e0ee4e66c6924560ccef5c8e41f6b221a61e693948",
+  "image_refiner.base_transformer.ref_image_refiner.1.attn.norm_q.weight": "eda15c9a547cbd34c3cba5f8727d53aad637691260808ed7e07ae6ef03172b72",
+  "image_refiner.base_transformer.ref_image_refiner.1.attn.to_k.weight": "7e57151fa3b7867ad52a7cb0273714e67e15e8d6873a7f4ab33f0237828b41dc",
+  "image_refiner.base_transformer.ref_image_refiner.1.attn.to_out.0.weight": "a3f6ce3f2edbbd66e01a438fb818817b27a7b81ebb46b3e4d6d43212a8e6b2d9",
+  "image_refiner.base_transformer.ref_image_refiner.1.attn.to_q.weight": "91e38dc152c81244f1016fdaae587cacc22b4f9865e4ae441d6e6bc4359b2317",
+  "image_refiner.base_transformer.ref_image_refiner.1.attn.to_v.weight": "71f24ce712af04ffc97e8bda62f533d01e116eda60e41ca0be3586be36ae93f2",
+  "image_refiner.base_transformer.ref_image_refiner.1.feed_forward.linear_1.weight": "9bff8b880bc58b4243acf344e2f8c25c403b91f0f7519ee2da22664c9cb4bcac",
+  "image_refiner.base_transformer.ref_image_refiner.1.feed_forward.linear_2.weight": "de00a9c14b4289a4cda051f85be05877d25136b985d3673a4f61f700f6dea37e",
+  "image_refiner.base_transformer.ref_image_refiner.1.feed_forward.linear_3.weight": "1317f4ae9afe4c25a987192003700db061ae6e72dfcbff5f3012eecc0c87ebc0",
+  "image_refiner.base_transformer.ref_image_refiner.1.ffn_norm1.weight": "537806f62e48be0a5b26903d0cdd2bac7171a5ed61afedfea58a1e93b7770d08",
+  "image_refiner.base_transformer.ref_image_refiner.1.ffn_norm2.weight": "b861a99e376ffddfe8a2856809ea93832d845642aeb7b648277628cc03d9876d",
+  "image_refiner.base_transformer.ref_image_refiner.1.norm1.linear.bias": "7aad166496f2341aa2a23d316beab6a6c1bb0abe238b7f674a2e8207dcc06365",
+  "image_refiner.base_transformer.ref_image_refiner.1.norm1.linear.weight": "da8a14c0dd3108f7ba143b6bd3e051f41af804afc2015e8cf13acb47d37d6c7e",
+  "image_refiner.base_transformer.ref_image_refiner.1.norm1.norm.weight": "da00f07ced96c3ce31b98cc7aba097a12f0b10c70acf61c738acb41feb6fab31",
+  "image_refiner.base_transformer.ref_image_refiner.1.norm2.weight": "83ee48daa1256e645ad420f87e6436f81bad1f76353544810cf6e97233204bb9",
+  "image_refiner.base_transformer.time_caption_embed.caption_embedder.0.weight": "a8de65dd55acf939375680e2d11d25771022e5d97cb6b0bea70c2aef0944745f",
+  "image_refiner.base_transformer.time_caption_embed.caption_embedder.1.bias": "0cb74303d6e988141687132fd402ddb9df4b9e519fd4c5a5a020882f019d197d",
+  "image_refiner.base_transformer.time_caption_embed.caption_embedder.1.weight": "0c993cbedc0cbddf8f9a68acd8ff2fadf4574e85c6c1a6190939045ed762e33c",
+  "image_refiner.base_transformer.time_caption_embed.timestep_embedder.linear_1.bias": "a19b10213b0e809b65eb1b107d081ba1f177befacb409eff17582ccb7392329c",
+  "image_refiner.base_transformer.time_caption_embed.timestep_embedder.linear_1.weight": "c645b5a3fce8c3f43431d3d125a339ea5205af18b841f6a14e35706086337eab",
+  "image_refiner.base_transformer.time_caption_embed.timestep_embedder.linear_2.bias": "85bbc08ec4fe4675d8754abc35bd6fb36529baeb63098d74c1c8cc583f30d490",
+  "image_refiner.base_transformer.time_caption_embed.timestep_embedder.linear_2.weight": "9c81c3e336dab8055cb145aec4b2af75581ab983b0573b5bdd9e12a70aa23dcb",
+  "image_refiner.base_transformer.x_embedder.bias": "d8e0280f1501b45654ec46e14645c0aaf3f2fb9e6de8bbb8c69679057f22648e",
+  "image_refiner.base_transformer.x_embedder.weight": "2ef643ed4bb7fb047318212f0538fdd1b04a353ea9adb72d38bb18a1dbff79d7"
+}

refiner_validation.json

@@ -0,0 +1,62 @@
+{
+  "strict_load": true,
+  "source_tensor_bytes_equal": true,
+  "checks": [
+    {
+      "name": "strict_full_flow_refiner",
+      "passed": true,
+      "tensors": 828,
+      "transformer_tensors": 582
+    },
+    {
+      "name": "native_bfloat16_weights",
+      "passed": true
+    },
+    {
+      "name": "native_refiner_conditioning",
+      "passed": true
+    },
+    {
+      "name": "conditioned_flow_forward",
+      "passed": true,
+      "shape": [
+        1,
+        16,
+        4,
+        4
+      ],
+      "dtype": "torch.bfloat16"
+    },
+    {
+      "name": "single_euler_update",
+      "passed": true,
+      "mean_abs_update": 0.07963985204696655
+    },
+    {
+      "name": "condition_affects_flow",
+      "passed": true,
+      "max_abs_difference": 0.99609375
+    },
+    {
+      "name": "refiner_source_tensor_bytes_exact",
+      "passed": true
+    }
+  ],
+  "smoke": {
+    "passed": true,
+    "device": "cpu",
+    "dtype": "torch.bfloat16",
+    "latent_shape": [
+      1,
+      16,
+      4,
+      4
+    ],
+    "reference_images": 1
+  },
+  "elapsed_seconds": 29.144361972808838,
+  "limitations": [
+    "Tiny-latent native flow forward and one Euler update, not full-resolution image refinement or perceptual-quality evaluation.",
+    "No LongCat language/audio backbone."
+  ]
+}

release_manifest.json

@@ -0,0 +1,161 @@
+{
+  "as_of": "2026-10-19",
+  "files": {
+    "model.safetensors": {
+      "size": 2025414272,
+      "sha256": "a57489db93bc60c1167c9f565c3a3f85d4ff1505477cd3b9ff95ef390fa3fe35"
+    },
+    "tensor_mapping.json": {
+      "size": 110491,
+      "sha256": "f1d2091c3200c9b1b3b20f30b98aeaaf0de60cc5761becbe739300dd420a9a35"
+    },
+    "code_provenance.json": {
+      "size": 545,
+      "sha256": "6bcb51d4848391e31fd8c35254781221f13ded239a5a49677b68fff6f012803e"
+    },
+    "LICENSE": {
+      "size": 1064,
+      "sha256": "4722341d105cec173161f93ccab6d4b49409b368ece23faa22f2f74adddef9ed"
+    },
+    "requirements.txt": {
+      "size": 114,
+      "sha256": "b87e4029730f3eb85739fdfd09267fe6717057213e494c8d741c9c7947dae625"
+    },
+    "aliases.json": {
+      "size": 3741,
+      "sha256": "34ac43464a23f2d030b2439227652925eaf8a75ff1b26d733fd9d160e4ef14ba"
+    },
+    "LICENSES.md": {
+      "size": 296,
+      "sha256": "95b0bfc0d58f7ebf65eda9324eac93d26ba9e56933c044daabec1045b8a50532"
+    },
+    "tensor_sha256.json": {
+      "size": 55010,
+      "sha256": "7215251c2ed1a54d261839f0dcf221d99f8c9c7e31f87a10b5824f1a0d58c93d"
+    },
+    "alias_verification.json": {
+      "size": 5946,
+      "sha256": "5b5c70cb3e344ecb3a421b67fd79c717976248e00735d1d3f556169fee004283"
+    },
+    "README.md": {
+      "size": 4778,
+      "sha256": "7802f4699325ff17b9641f3173231247c306eb6a897ebcd826b80c2942ad8fde"
+    },
+    "validate.py": {
+      "size": 3484,
+      "sha256": "cf9653df1d32f020005f8ae55adf201f72e4f9937357e4f2199fe980b61c92f4"
+    },
+    "validation.json": {
+      "size": 3368,
+      "sha256": "770479dc8fe793cceb169f4dbfa177f9705a49391c3d56d408b2daad83aaa620"
+    },
+    "validate_refiner.py": {
+      "size": 4033,
+      "sha256": "434b43b619a27f11841b7d884e53d4b73205a4a2f33f9d64f1d6889b693bb0b6"
+    },
+    "extraction.json": {
+      "size": 467,
+      "sha256": "88063dba8e7aadbcfebca10c74e7b31f8986181e82f246c83bbf4ff8d831f801"
+    },
+    "component.py": {
+      "size": 5631,
+      "sha256": "292040d955cbfe43278d4c2a06cc65c5401cab4ee1b07c230f1ea4087fa5d667"
+    },
+    "refiner_validation.json": {
+      "size": 1282,
+      "sha256": "292b17aba5242257005d1bcb0963c61d15e3619ae6f00f116e36b8f8cbc07561"
+    },
+    "source_config.json": {
+      "size": 6908,
+      "sha256": "9115e9785603b04382a45ebece9092235281f309f56f35eb4e43bcf53150b2a2"
+    },
+    "native/compat.py": {
+      "size": 1255,
+      "sha256": "e2ba47fa854c62ee70e507f98a7a862690e4f6d51a32326e3a6f28d878e0fddd"
+    },
+    "native/configuration_longcat_ngram.py": {
+      "size": 10729,
+      "sha256": "96a646608a90ae4d42e6b4c8f712b01f1f9033241af97c0b3f7307dc1887d191"
+    },
+    "native/refiner_modules.py": {
+      "size": 50858,
+      "sha256": "f7097d943160a42b6251440068a413399636748674711241ace8917e247d9211"
+    },
+    "native/__init__.py": {
+      "size": 0,
+      "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+    },
+    "native/visual.py": {
+      "size": 46229,
+      "sha256": "2d77ca1e774e2b2861429bf8b679f7a2a4a290659b43aa79bd1221001a40fb10"
+    },
+    "native/image_refiner.py": {
+      "size": 30146,
+      "sha256": "ef1e08c97b7275cb455d3baea84149510605e0cea44ab10142e5fac6d3c0baaf"
+    },
+    "native/qwen2_5_vl_vit.py": {
+      "size": 22477,
+      "sha256": "b8fac6524173bdf4ea2f2e9a06be2430d9b689f6915548a1392d232a5848f424"
+    },
+    "native/configuration_longcat_next.py": {
+      "size": 6337,
+      "sha256": "bce3c8fc8bc0f4e6f3d0eb39e7a3a0415b4d66a8778f90435fb6849342c41f6c"
+    },
+    "refiner/model.safetensors": {
+      "size": 7934402936,
+      "sha256": "fb94553b718c6686170f7eb73d97f59d8f3e49f3e7d2436539ad424efaef1f28"
+    },
+    "refiner/tensor_mapping.json": {
+      "size": 163223,
+      "sha256": "7cafeaf8dc303605548e96e46440abdfe362c0dbfc91f7d26334e8ae85ad393e"
+    },
+    "refiner/aliases.json": {
+      "size": 3,
+      "sha256": "ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356"
+    },
+    "refiner/tensor_sha256.json": {
+      "size": 78598,
+      "sha256": "745911a73a310edea4c4294e366d26614f8cef2d3a368ae31bc304827f119faa"
+    },
+    "refiner/alias_verification.json": {
+      "size": 3,
+      "sha256": "ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356"
+    },
+    "refiner/extraction.json": {
+      "size": 437,
+      "sha256": "056605fcdde1f46645918ae0a8c1bc6208e942cf608ac3dddc8f35cf4263d268"
+    },
+    "refiner/source_config.json": {
+      "size": 6908,
+      "sha256": "9115e9785603b04382a45ebece9092235281f309f56f35eb4e43bcf53150b2a2"
+    },
+    "decoder/model.safetensors": {
+      "size": 1049912922,
+      "sha256": "2a0e46b2a40ed901d5b0f839df183cb8b679cd43975b59211c03999164327f71"
+    },
+    "decoder/tensor_mapping.json": {
+      "size": 216637,
+      "sha256": "871954d549203d945858efc60f47c3c89ba1e0f042f2f3787b28d136774c72d8"
+    },
+    "decoder/aliases.json": {
+      "size": 3,
+      "sha256": "ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356"
+    },
+    "decoder/tensor_sha256.json": {
+      "size": 101750,
+      "sha256": "91d4e5dc236c168858701160ba372339099ebf3bfe3b8079444d90e8c8ca5a4c"
+    },
+    "decoder/alias_verification.json": {
+      "size": 3,
+      "sha256": "ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356"
+    },
+    "decoder/extraction.json": {
+      "size": 437,
+      "sha256": "776fcd26c500d7b374a67e70ae41618f1ee7f5bf234e61e63fd70e2d296e7286"
+    },
+    "decoder/source_config.json": {
+      "size": 6908,
+      "sha256": "9115e9785603b04382a45ebece9092235281f309f56f35eb4e43bcf53150b2a2"
+    }
+  }
+}

requirements.txt

@@ -0,0 +1,8 @@
+torch>=2.6
+transformers>=5.0
+diffusers>=0.35
+safetensors>=0.6
+accelerate>=1.10
+einops>=0.8
+numpy>=1.26
+Pillow>=10

source_config.json

@@ -0,0 +1,286 @@
+{
+  "architectures": [
+    "LongcatNextForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_longcat_next.LongcatNextConfig",
+    "AutoModel": "modeling_longcat_next.LongcatNextModel",
+    "AutoModelForCausalLM": "modeling_longcat_next.LongcatNextForCausalLM"
+  },
+  "vocab_size": 282624,
+  "hidden_size": 3072,
+  "ffn_hidden_size": 6144,
+  "expert_ffn_hidden_size": 1024,
+  "num_layers": 14,
+  "num_attention_heads": 32,
+  "kv_lora_rank": 512,
+  "q_lora_rank": 1536,
+  "qk_rope_head_dim": 64,
+  "v_head_dim": 128,
+  "qk_nope_head_dim": 128,
+  "mla_scale_q_lora": true,
+  "mla_scale_kv_lora": true,
+  "routed_scaling_factor": 6.0,
+  "n_routed_experts": 256,
+  "rms_norm_eps": 1e-5,
+  "use_cache": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "rope_theta": 10000000,
+  "max_position_embeddings": 131072,
+  "zero_expert_num": 128,
+  "zero_expert_type": "identity",
+  "moe_topk": 12,
+  "ngram_vocab_size_ratio": 78,
+  "emb_neighbor_num": 4,
+  "emb_split_num": 4,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.57.6",
+  "model_type": "longcat_next",
+
+  "text_vocab_size": 131072,
+  "text_vocab_plus_multimodal_special_token_size": 131125,
+  "visual_embedding_layer_intermediate_size": 8192,
+  "visual_embedding_layer_hidden_act": "silu",
+  "visual_offset": 150581,
+  "audio_offset": 131125,
+
+
+  "visual_config": {
+    "image_start_token_id": 131106,
+    "image_end_token_id": 131107,
+    "image_pad_token_id": 131108,
+    "image_newline_token_id": 131109,
+
+    "_attn_implementation": "flash_attention_2",
+    "hidden_size": 1280,
+
+    "image_head_transformer_dims": 2048,
+    "image_head_transformer_ffn_scale": 16,
+    "image_head_transformer_layers": 4,
+
+    "vq_config": {
+      "codebook_dim": 3584,
+      "codebook_size": 16384,
+      "codebook_sizes": [
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384,
+        16384
+      ],
+      "decay": 0.99,
+      "depth": 8,
+
+      "commit_loss_ratio": 0.25,
+      "entropy_loss_ratio": 0,
+
+      "in_channels": 3584,
+      "quant_conv": true,
+      "quantizer_type": "rq",
+      "restart_unused_codes": true,
+      "shared_codebook": true,
+
+      "vq_loss_ratio": 0
+    },
+
+    "visual_decoder_config": {
+      "codebook_dim": 3584,
+
+      "image_decoder_config": {
+        "attention_dropout": 0.0,
+        "codebook_dim": 3584,
+        "distill_taps": [
+          3,
+          7,
+          15,
+          23
+        ],
+        "hidden_act": "gelu",
+        "hidden_size": 1024,
+        "intermediate_size": 2730,
+        "k_bias": false,
+        "layer_norm_eps": 1e-06,
+        "num_attention_heads": 16,
+        "num_hidden_layers": 32,
+        "patch_size": 14,
+        "q_bias": true,
+        "spatial_merge_size": 2,
+        "subln": true,
+        "swiglu": true,
+        "teacher_dims": {
+          "15": 1280,
+          "23": 1280,
+          "3": 1280,
+          "7": 1280
+        },
+        "temporal_patch_size": 2,
+        "v_bias": true
+      },
+
+      "transformer_config": {
+        "patch_size": 2,
+        "in_channels": 16,
+        "hidden_size": 2520,
+        "num_layers": 32,
+        "num_refiner_layers": 2,
+        "num_attention_heads": 21,
+        "num_kv_heads": 7,
+        "multiple_of": 256,
+        "norm_eps": 1e-5,
+        "axes_dim_rope": [40, 40, 40],
+        "axes_lens": [10000, 10000, 10000],
+        "text_feat_dim": 2048,
+        "timestep_scale": 1000.0
+      },
+
+      "vae_config": {
+        "act_fn": "silu",
+        "block_out_channels": [128, 256, 512, 512],
+        "down_block_types": [
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D",
+          "DownEncoderBlock2D"
+        ],
+        "in_channels": 3,
+        "latent_channels": 16,
+        "layers_per_block": 2,
+        "mid_block_add_attention": true,
+        "norm_num_groups": 32,
+        "out_channels": 3,
+        "sample_size": 1024,
+        "scaling_factor": 0.3611,
+        "shift_factor": 0.1159,
+        "up_block_types": [
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D",
+          "UpDecoderBlock2D"
+        ],
+        "use_post_quant_conv": false,
+        "use_quant_conv": false,
+        "force_upcast": true
+      },
+
+      "scheduler_config": {
+        "num_train_timesteps": 1000,
+        "dynamic_time_shift": true
+      },
+
+      "weight_path": "WEIGHT_PATH_TO_LONGCAT_NEXT/image_decoder/image_decoder.safetensors"
+    }
+  },
+
+
+  "audio_config": {
+    "audio_head_transformer_dims": 3072,
+    "audio_head_transformer_ffn_scale": 16,
+    "audio_head_transformer_layers": 4,
+
+    "audio_delim_token_id": 131116,
+    "audio_end_token_id": 131104,
+    "audio_pad_token_id": 131105,
+    "audio_start_token_id": 131103,
+    "audiogen_end_token_id": 131124,
+    "audiogen_start_token_id": 131123,
+    "audiotext_end_token_id": 131121,
+    "audiotext_pad_token_id": 131122,
+    "audiotext_start_token_id": 131120,
+
+    "_attn_implementation": "flash_attention_2",
+    "d_model": 1280,
+    "decoder_attention_heads": 20,
+    "decoder_ffn_dim": 5120,
+    "decoder_layers": 8,
+    "encoder_attention_heads": 20,
+    "encoder_ffn_dim": 5120,
+    "encoder_layers": 32,
+    "num_mel_bins": 128,
+
+    "avg_pooler": 4,
+    "decoder_kernel_size": 3,
+    "decoder_stride_size": 2,
+    "hop_length": 160,
+    "kernel_size": 3,
+    "max_audio_seconds": 30,
+    "n_fft": 400,
+    "num_hidden_layers": 32,
+    "sampling_rate": 16000,
+    "stride_size": 2,
+
+    "vq_config": {
+      "codebook_sizes": [
+        8192,
+        4096,
+        2048,
+        1024,
+        1024,
+        1024,
+        1024,
+        1024
+      ]
+    },
+
+    "vocoder_config": {
+      "channels": [
+        256,
+        256,
+        256,
+        256,
+        256
+      ],
+      "hop_length": 256,
+      "num_mel_bins": 80,
+      "sampling_rate": 16000
+    },
+
+
+    "flow_matching_config": {
+      "in_channels": 80,
+      "spk_emb_dim": 0,
+      "diffusion_steps": 10,
+      "cal_mel_mae": true,
+
+      "prenet_activation_function": "gelu",
+      "prenet_attention_heads": 8,
+      "prenet_d_model": 512,
+      "prenet_ffn_dim": 2048,
+      "prenet_in_dim": 1280,
+      "prenet_max_source_positions": 5000,
+      "prenet_nlayers": 12,
+      "prenet_out_dim": 80,
+      "prenet_target_mel_length_scale_ratio": 1.0,
+
+      "channels": [
+        256
+      ],
+      "dropout": 0.0,
+      "attention_head_dim": 64,
+      "n_blocks": 4,
+      "num_heads": 8,
+      "num_mid_blocks": 12,
+      "act_fn": "gelu",
+
+      "cfm_params": {
+        "inference_cfg_rate": 0.7,
+        "sigma_min": 1e-06,
+        "solver": "euler",
+        "t_scheduler": "cosine",
+        "training_cfg_rate": 0.2
+      },
+
+      "use_hidden_states_before_dconv2": true
+    },
+
+    "cosy24kvocoder_config": {
+      "weight_path": "WEIGHT_PATH_TO_LONGCAT_NEXT/cosy24k_vocoder/hift.pt"
+    }
+
+  }
+}

tensor_sha256.json

@@ -0,0 +1,406 @@
+{
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed": "fc65e921fe9fd960a9e623e2417d32b0439d1fa82fa45af99af03319e7a6eda3",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.embed_ema": "b190da7cc5c132efb5dc66587dbe9fc90df832f8e5e28e8eea0c59c84c1668a1",
+  "model.visual_tokenizer.visual_bridge_model.bridge.mlp.0.weight": "73ae7da30df77353224240bcb5135153694ac55fe26f6c089ecd489c5b74b97f",
+  "model.visual_tokenizer.visual_bridge_model.bridge.mlp.2.weight": "ec63961bb0f0720eb833418d593b17706f80b1c831e1662ac5d3f29b3a0482eb",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quant_conv.1.weight": "a195b5ffc0befdb1c3f30734252c3580cb796be310f19dfb21e2cd7baeedb83a",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quant_conv.3.weight": "0fb765ce9c8c124155c7e1267257d7c6d2abe7c3f22105aa3ef8a19651948683",
+  "model.visual_tokenizer.visual_embedding_layer.pre_buffer.mlp.down_proj.weight": "f8d80c38687d0a51949780de29035f1fa5c8a2ae3e3b3b68facdc62b4ca3569a",
+  "model.visual_tokenizer.visual_embedding_layer.pre_buffer.mlp.gate_proj.weight": "0bc96ca631620d1081177a8af9d3a9db1cd607ca250334e394c02893d027fbfd",
+  "model.visual_tokenizer.visual_embedding_layer.pre_buffer.mlp.up_proj.weight": "4ee236aa5b7a04d456dab26be2491edfbad3c45db8998e3d458125e3cee907ee",
+  "model.visual_tokenizer.visual_model.blocks.0.attn.qkv.weight": "d9e147b82330139af1fcbf5bc37961d1a6fb1032738cbe99627d34906d8c6b90",
+  "model.visual_tokenizer.visual_model.blocks.0.mlp.down_proj.weight": "f64d12df314c4918037c862b6e2d4e6d6e7d94f095792f73d1facc87e20d4ad3",
+  "model.visual_tokenizer.visual_model.blocks.0.mlp.gate_proj.weight": "1dd598444ea41361e3d6b6294573ca73eeab41a6ad9149b122c0cce8ed041719",
+  "model.visual_tokenizer.visual_model.blocks.0.mlp.up_proj.weight": "9db75456af696e12e42005dd3cf27d9865061d60552a419beaa336eb93c1562d",
+  "model.visual_tokenizer.visual_model.blocks.1.attn.qkv.weight": "e79a92fc4f50e2085c0e6b9463cc27c037dfaee6d3594a1cc1bd1b682e047fa1",
+  "model.visual_tokenizer.visual_model.blocks.1.mlp.down_proj.weight": "7234b8d3beb37c0400c5f44d249440a4340f7ae3788eb3533e669c2373f7d94b",
+  "model.visual_tokenizer.visual_model.blocks.1.mlp.gate_proj.weight": "63abd9c249159d39c2bf55f364b7d487f1e077be0ecea4e0d7222b70cb34a842",
+  "model.visual_tokenizer.visual_model.blocks.1.mlp.up_proj.weight": "32b6951854995de874515971dec22191472ecedfb362abad8214714cb7a3e248",
+  "model.visual_tokenizer.visual_model.blocks.10.attn.qkv.weight": "4b83fb4dbb239625f5b9d97034765ec17c6d3def71a7215297ccfd88c1653443",
+  "model.visual_tokenizer.visual_model.blocks.10.mlp.down_proj.weight": "a006258f54c9f22d45f4cb1ba4b8c4270e026cca8a8d90d4b050d9370cdc4a99",
+  "model.visual_tokenizer.visual_model.blocks.10.mlp.gate_proj.weight": "97d957aaf7d8ef8c74b1f72fc292a20b2f1f5434e6e73ad4839173c1374e881f",
+  "model.visual_tokenizer.visual_model.blocks.10.mlp.up_proj.weight": "7d521f0d76785079a4b5414d59d7a1ad37545fc98ffcda4f0e679f39a05c3701",
+  "model.visual_tokenizer.visual_model.blocks.11.attn.qkv.weight": "170775c8954be77b680c2ce39ce62bd58dbe776253c2dc9131554c604f9d786d",
+  "model.visual_tokenizer.visual_model.blocks.11.mlp.down_proj.weight": "415a6ba0f61a0e201383dac3994bd9f5bf0c568ebf99489253631357aaf9768a",
+  "model.visual_tokenizer.visual_model.blocks.11.mlp.gate_proj.weight": "e91bcb01345aae6ed5353836e4e27b877956809ad42dc334350549e123882b82",
+  "model.visual_tokenizer.visual_model.blocks.11.mlp.up_proj.weight": "2a284d5873dc0344d418406c9ec88006b47b6009c26c0a4ec787813b95ca76b9",
+  "model.visual_tokenizer.visual_model.blocks.12.attn.qkv.weight": "131ab528e571315625eb5c2ecaa520149a0dff66254eb7822fc140558150bee9",
+  "model.visual_tokenizer.visual_model.blocks.12.mlp.down_proj.weight": "d2cd7ea18c1dea9359c235ea9eb6204fb41a3eae062cafc0263d18737b396238",
+  "model.visual_tokenizer.visual_model.blocks.12.mlp.gate_proj.weight": "54efd9e65311046b0eb9181f7d99f357ea6ef29665e44c1f3970218aae8bce21",
+  "model.visual_tokenizer.visual_model.blocks.12.mlp.up_proj.weight": "bb7eaf9eb53534afb8ee0e62ab007fcea9833c10d11c93a9d411a986a66ebf8e",
+  "model.visual_tokenizer.visual_model.blocks.13.attn.qkv.weight": "25fbaf1beb9348edda54913307778a8c7e42bf3176379f9785de4151b3c8ffec",
+  "model.visual_tokenizer.visual_model.blocks.13.mlp.down_proj.weight": "210db36046edaf2a6e7660d05d6832fea5b8360343d738be9c335f3c4af11a3b",
+  "model.visual_tokenizer.visual_model.blocks.13.mlp.gate_proj.weight": "6b39d90b861f9faab4f5ead709b1f81cec0243bb9319b8b900357dc00d0e75c5",
+  "model.visual_tokenizer.visual_model.blocks.13.mlp.up_proj.weight": "915edf910df73d4f947bb353d1a7798064711bdb7bc886c95eed78edce6192f0",
+  "model.visual_tokenizer.visual_model.blocks.14.attn.qkv.weight": "0a067491dc83fd97beb1b53082271fd233d7923a5eeb9c38fac4b936ef7b79a3",
+  "model.visual_tokenizer.visual_model.blocks.14.mlp.down_proj.weight": "cd5217f452242a3d43a90f7348ea276f0cb35400b9862f7b04c8cdc799d5148a",
+  "model.visual_tokenizer.visual_model.blocks.14.mlp.gate_proj.weight": "5e6b4b4ae1d1cc0cf6127e1d685e9fa184a830573522c01d795545f132948199",
+  "model.visual_tokenizer.visual_model.blocks.14.mlp.up_proj.weight": "b2e5ad77d0e2dfc3bb93e2ebabbbbe770503b22d4970ec2a28385982b1ff0ea7",
+  "model.visual_tokenizer.visual_model.blocks.15.attn.qkv.weight": "9bc885f306c0aff53de7f641dce65f60a0614cc2f9bf3eb14beb8b309be317b1",
+  "model.visual_tokenizer.visual_model.blocks.15.mlp.down_proj.weight": "d898e5c1e9ac4beea4dbc724c1b350bb333ea975aaf6dc82adf045b84b2559f7",
+  "model.visual_tokenizer.visual_model.blocks.15.mlp.gate_proj.weight": "118678bbc34973fab72b5e514e576c7ee65ea8d375fd337127142e33f9bc4b8d",
+  "model.visual_tokenizer.visual_model.blocks.15.mlp.up_proj.weight": "1990cfb0292a309d5d8b0d5db22ed247d1694ff35d100472ffaeda3a6abc799d",
+  "model.visual_tokenizer.visual_model.blocks.16.attn.qkv.weight": "bf11b97e7489525e8093acbb8e1605f6fff4f495101d28e144fc467b0334e26e",
+  "model.visual_tokenizer.visual_model.blocks.16.mlp.down_proj.weight": "fc7dfb10d634b831169d69a4b48e0fcfb69348ab2ad3c87649b70fd3d83234df",
+  "model.visual_tokenizer.visual_model.blocks.16.mlp.gate_proj.weight": "43f426fd8239cfb8e4167280015e68bc26498b43b2caac4f3aee0d77daac499b",
+  "model.visual_tokenizer.visual_model.blocks.16.mlp.up_proj.weight": "2c9683de7024bb73c61ba638e4b576db81bda1dc9efa1efd8943a9cb80c7b464",
+  "model.visual_tokenizer.visual_model.blocks.17.attn.qkv.weight": "eb0148df7d1be4f7c8c1a7ffd4a93e70ac01107364411f183c6aa37fa5716142",
+  "model.visual_tokenizer.visual_model.blocks.17.mlp.down_proj.weight": "0914e440613767adbdaec56cae1611de80a7c3edd3a6894a9bc1305e83f33017",
+  "model.visual_tokenizer.visual_model.blocks.17.mlp.gate_proj.weight": "2c748fe45141d43c4c222e21e65f1b3139f0eead20a82777ffa02b7f1f61e632",
+  "model.visual_tokenizer.visual_model.blocks.17.mlp.up_proj.weight": "1bd73992a109329effcbe2559ef1a3d9ded6f09b55bedff120127b0ddf1e2ecb",
+  "model.visual_tokenizer.visual_model.blocks.18.attn.qkv.weight": "eab92adb92c6476e7ea44e98e68af68da054e847cea7e5b08ad8ca4a196df3b4",
+  "model.visual_tokenizer.visual_model.blocks.18.mlp.down_proj.weight": "bebda2fc3daeb7cab182ecae4919c59fb080f5d4c83f8e05338ae650ed9a1224",
+  "model.visual_tokenizer.visual_model.blocks.18.mlp.gate_proj.weight": "28787c5112e6755d3eefcafd9f6b98c9c452bbb4e270e4e176c5c3f41c8b5a92",
+  "model.visual_tokenizer.visual_model.blocks.18.mlp.up_proj.weight": "78d58feb604e575938f5e6bf3814dce90ce4f1aeb30b3a937fb56e7f2ad3af36",
+  "model.visual_tokenizer.visual_model.blocks.19.attn.qkv.weight": "37876081ed35ca29a860d9dff20fd728796f474837be267973aae2ba6fb60030",
+  "model.visual_tokenizer.visual_model.blocks.19.mlp.down_proj.weight": "266f08276150cf08ab82cf2f3ba31d6ffddaaaf5a19776827d09e0cc10eb4e5f",
+  "model.visual_tokenizer.visual_model.blocks.19.mlp.gate_proj.weight": "5b44493f8848d401cad708b9b4a5fd92e87b70fcd102d8b40270ad09a11e4dd5",
+  "model.visual_tokenizer.visual_model.blocks.19.mlp.up_proj.weight": "fdc5a2037b86cc8c61e3686555b5ea1bc9a3c6909b942ab1754fca06f198221c",
+  "model.visual_tokenizer.visual_model.blocks.2.attn.qkv.weight": "7bdb13508d3d635e5d73ef4279cda5a7e96c4cbe9c090e875073a818b5cd8192",
+  "model.visual_tokenizer.visual_model.blocks.2.mlp.down_proj.weight": "220a842b1cc172fac7c1b08912950e60b46892c7ffe6ef93ed3f4c1f81559fca",
+  "model.visual_tokenizer.visual_model.blocks.2.mlp.gate_proj.weight": "15f07fa8cbed7a29f74331721872ca060bd9d4efd25e5544cdec65e627681c66",
+  "model.visual_tokenizer.visual_model.blocks.2.mlp.up_proj.weight": "87f49303aa7fcc58fc9728cd8eb05b53e257a28f1f5e92b6fcf5d74d31d67c9b",
+  "model.visual_tokenizer.visual_model.blocks.20.attn.qkv.weight": "b172bf48c4c126b7bef09363c3662ed3dba0d576a623f55a9628f8ceaf838519",
+  "model.visual_tokenizer.visual_model.blocks.20.mlp.down_proj.weight": "2037bd64eee813968a5d4e2cc7a745c596daf00e62ca33442115a17239ab2c20",
+  "model.visual_tokenizer.visual_model.blocks.20.mlp.gate_proj.weight": "6552fabe4567eb22359be23e9b8e2fc2689674b125b128232b9346261e50421b",
+  "model.visual_tokenizer.visual_model.blocks.20.mlp.up_proj.weight": "40ed660ad8aaec7bb7c475774f86143511ce5e5889b63889c20b45e69b9c837c",
+  "model.visual_tokenizer.visual_model.blocks.21.attn.qkv.weight": "e0797723820cd75a5f8588e7cc154840aad407a5e816c645e7e333c65bd80f5e",
+  "model.visual_tokenizer.visual_model.blocks.21.mlp.down_proj.weight": "14d93873528d916557a768ba888983f28f9619480e83d1fa19c7501f4fdeb95a",
+  "model.visual_tokenizer.visual_model.blocks.21.mlp.gate_proj.weight": "0e26a050be14b973d3f0168a897229f683648125fc4e1e68f1251b2aeca4eeb3",
+  "model.visual_tokenizer.visual_model.blocks.21.mlp.up_proj.weight": "4e7f899243cfac182dc6740be0d12c8ee4c560ba5c82f6329b4a37aec518908e",
+  "model.visual_tokenizer.visual_model.blocks.22.attn.qkv.weight": "eb1b006ce6cd1f3f00629084031017d62e3036f454809fa7151aff5a1dd52a3a",
+  "model.visual_tokenizer.visual_model.blocks.22.mlp.down_proj.weight": "1bb9bbfb00b1b36a019c14b06349b49310b92f11b1aa340acb6fbfbf41eae5c0",
+  "model.visual_tokenizer.visual_model.blocks.22.mlp.gate_proj.weight": "3e2bf99a1e2f7aacfcf40f2cf1ea1cc50f662a9e24c9250451b119cfe9901eae",
+  "model.visual_tokenizer.visual_model.blocks.22.mlp.up_proj.weight": "a347a7195ceb7f99841989a76704733c5a21c24f4f2059d74ac23eddaa474699",
+  "model.visual_tokenizer.visual_model.blocks.23.attn.qkv.weight": "fd8be80a1935e09b3176a2fdfc4c700b3d69009ea74deafc53b58712ee161869",
+  "model.visual_tokenizer.visual_model.blocks.23.mlp.down_proj.weight": "642d88b02db6a356c409a72c64e3cacdaf645743d294dbe8a6f374c04fb49c07",
+  "model.visual_tokenizer.visual_model.blocks.23.mlp.gate_proj.weight": "ec46e5687b643412673a9fc9031aa2d1874c68fe5755e5f8efa7be821c131fa5",
+  "model.visual_tokenizer.visual_model.blocks.23.mlp.up_proj.weight": "d44171b7aa623d825ee90d0293bfaea7adc76e34b283885b4d536f01ffaa5930",
+  "model.visual_tokenizer.visual_model.blocks.24.attn.qkv.weight": "92cdb23a186ef60f3a4223ff83c2a4e66391d4eaeee7839da391b9d8a7c0ad59",
+  "model.visual_tokenizer.visual_model.blocks.24.mlp.down_proj.weight": "90cc6ef221ad5123b1576a5d0222115d8ee7a5549c4dc5d2138a2411eb0b71fb",
+  "model.visual_tokenizer.visual_model.blocks.24.mlp.gate_proj.weight": "e743cbe1e71e8bdd7e8c5fe16fbcf7775a194a960d55bca658d483c291f71ebe",
+  "model.visual_tokenizer.visual_model.blocks.24.mlp.up_proj.weight": "49cd0cf129b439382140ccf5d1fb3a4bde665dcef0012e0676ceba5860978101",
+  "model.visual_tokenizer.visual_model.blocks.25.attn.qkv.weight": "a43605f82cb41069858ad76dfe12b5aca4338ed2b3fa35045452dfdf098ed85b",
+  "model.visual_tokenizer.visual_model.blocks.25.mlp.down_proj.weight": "47ff08cf93dc4a52abe0ec7907a13fd5542d6755d78005100ca54f7bb1bfccd6",
+  "model.visual_tokenizer.visual_model.blocks.25.mlp.gate_proj.weight": "04a50446c08193571458307b949a1a7bb46c8ceec8fbbf795349669b99d424e2",
+  "model.visual_tokenizer.visual_model.blocks.25.mlp.up_proj.weight": "b5b19d455fbd67acdfead45ca229f89dd4641527f7e730a9f295bdd0b686c9d1",
+  "model.visual_tokenizer.visual_model.blocks.26.attn.qkv.weight": "7ceb5f1d280b52b3267053e08ed3aaa975c91ffa655e607d4218c9b1b85fc492",
+  "model.visual_tokenizer.visual_model.blocks.26.mlp.down_proj.weight": "fd4d53d20020d7b0d05191f858908442e0fb476a8f6b4ab9ee9a3b49be3d4741",
+  "model.visual_tokenizer.visual_model.blocks.26.mlp.gate_proj.weight": "3db7a2a94693690bd9431411b4006ed2d859d1746eceabb607c3438d9ea6b447",
+  "model.visual_tokenizer.visual_model.blocks.26.mlp.up_proj.weight": "0eb292a404e231d385c860989c8f97c50eb6243c8025e352a89f0b478188f761",
+  "model.visual_tokenizer.visual_model.blocks.27.attn.qkv.weight": "2894fd6996c68687097d6b4eb71a6eb24cbf45db8f2dfcf65224fbecebf7ce02",
+  "model.visual_tokenizer.visual_model.blocks.27.mlp.down_proj.weight": "8dff26ffb4eced42f86c8f3f4812ca3310806c1030fc664c40fa705459e43b84",
+  "model.visual_tokenizer.visual_model.blocks.27.mlp.gate_proj.weight": "06308048eb1c73c022950a1519a0391d5c95bcc6eecd993f2222baa7e9a2931d",
+  "model.visual_tokenizer.visual_model.blocks.27.mlp.up_proj.weight": "fc36be0d416b30718560f7a30d3d599fad5e4ff91493eaddba1babb4572b6fbf",
+  "model.visual_tokenizer.visual_model.blocks.28.attn.qkv.weight": "0d3b7e9e5c1dd52c57876b5e80e7c0460cb63a4824c6fd224d363ebd109c9689",
+  "model.visual_tokenizer.visual_model.blocks.28.mlp.down_proj.weight": "b29e1dc4994b2fb7d059fb5383f9ce05e910aee48a2749a553d0235b3d4edbe4",
+  "model.visual_tokenizer.visual_model.blocks.28.mlp.gate_proj.weight": "93929c13cf25dcbb47031c48d7cca8bc0cecf7bc580587e9a7a91f306958c4d8",
+  "model.visual_tokenizer.visual_model.blocks.28.mlp.up_proj.weight": "afbf0debc240a89fa6909749b84f739c3117eb057f1ac8603be2d541d81a1604",
+  "model.visual_tokenizer.visual_model.blocks.29.attn.qkv.weight": "e29ca4f644f838a346e0baf5a25c0b092367590634055a0a97ac3cacf69302a7",
+  "model.visual_tokenizer.visual_model.blocks.29.mlp.down_proj.weight": "a8d04257194ce92e247268d53a3f416a1b0657041aa08575d84d34d094bba778",
+  "model.visual_tokenizer.visual_model.blocks.29.mlp.gate_proj.weight": "c7151bcd0bd7a716d357e386824a95c77e15fe4a5929bf763e679eeba73ffa23",
+  "model.visual_tokenizer.visual_model.blocks.29.mlp.up_proj.weight": "8e791e630f34b83dbc65a73fe19541af739b577ac35d08e28bb02c6f0a3e01a8",
+  "model.visual_tokenizer.visual_model.blocks.3.attn.qkv.weight": "19968141feb05eb397c86c0f365f1e977e6e147fb5dce990efd3cec96071055f",
+  "model.visual_tokenizer.visual_model.blocks.3.mlp.down_proj.weight": "014ab34664c2867f6774a45443c2d55bbf6f476488f104569bfb4d57e9d892f5",
+  "model.visual_tokenizer.visual_model.blocks.3.mlp.gate_proj.weight": "34559a0594582ed7ca5e1373a9bbe964cc5982a020d878921bff95e5837fdbe4",
+  "model.visual_tokenizer.visual_model.blocks.3.mlp.up_proj.weight": "e8250b9ac14b0285263db21dd0ecb6d3df28014345e44218ef3f664fbb78be1b",
+  "model.visual_tokenizer.visual_model.blocks.30.attn.qkv.weight": "9f6d236637f3f14c133ccc65b288e128bec3c5bfefbc8aa337053b9402fcec4a",
+  "model.visual_tokenizer.visual_model.blocks.30.mlp.down_proj.weight": "48229fde91304d2c49333979e902f6952be921edb3951a89cd99b7f123129a73",
+  "model.visual_tokenizer.visual_model.blocks.30.mlp.gate_proj.weight": "9c6a3bd4e890aa7ec3901639cfec5f9e74ad5a41d352596c1984399e4905507b",
+  "model.visual_tokenizer.visual_model.blocks.30.mlp.up_proj.weight": "0dc3fdad4f06e3efb83c507d3cfbdd4df2c8aee4a63dd39856d4d76a062a04be",
+  "model.visual_tokenizer.visual_model.blocks.31.attn.qkv.weight": "cc6adccf60977a13fe43e7f4234c69b3ebdadfd5f8867f82c2bcff9a1ea5bf5b",
+  "model.visual_tokenizer.visual_model.blocks.31.mlp.down_proj.weight": "fa539ed59b58d0d2d1e1c3fa75353a2fa75d1c236a36e846162347d44ecd20c9",
+  "model.visual_tokenizer.visual_model.blocks.31.mlp.gate_proj.weight": "bc3518bfddc5b0358997bd6b0b0467960ccab574a0be1008a6437f09c573639e",
+  "model.visual_tokenizer.visual_model.blocks.31.mlp.up_proj.weight": "3192587909697513e31a917fae2b9c1aaf37f564ac7e9890ecfd436ff02fb926",
+  "model.visual_tokenizer.visual_model.blocks.4.attn.qkv.weight": "5a54f107c5b955d2ede02c9e2611b013143dedb9e26d4ec5aa33b0a320366bba",
+  "model.visual_tokenizer.visual_model.blocks.4.mlp.down_proj.weight": "dcec10e345b489389f594cad619da520b7a714e66b78780690598b054b36402c",
+  "model.visual_tokenizer.visual_model.blocks.4.mlp.gate_proj.weight": "a7ed736b4a5e7751d264599d030bef1c304a988f9b9835294e34be824f39299a",
+  "model.visual_tokenizer.visual_model.blocks.4.mlp.up_proj.weight": "6cd86841d3f5d7e82a67317abbc6e1f35a0dfcc47ff39aca256b94a4c7cc1a4f",
+  "model.visual_tokenizer.visual_model.blocks.5.attn.qkv.weight": "6c3e7150660d5932e078b68de962210ede4bec26f7fdefca152861b418bf2047",
+  "model.visual_tokenizer.visual_model.blocks.5.mlp.down_proj.weight": "161751f0eb6a58159bc8e2bafb35c6780d7b7d4f1709e56c6c09cb64793ebfbc",
+  "model.visual_tokenizer.visual_model.blocks.5.mlp.gate_proj.weight": "da661fb3f9d6ece7670b90aa63474a7b7c11a0941c8fc952245e5698e2ee87f5",
+  "model.visual_tokenizer.visual_model.blocks.5.mlp.up_proj.weight": "bc7f7010ab97cc90b0dd53b66a202b0a58ee1992baf43b88e2ac606c643a1bd2",
+  "model.visual_tokenizer.visual_model.blocks.6.attn.qkv.weight": "ef948ec1fd5082024a8aa24549388c1edc149ac5483009793d3dedb246a7cd0c",
+  "model.visual_tokenizer.visual_model.blocks.6.mlp.down_proj.weight": "0c32507df39774ffe485bfc6235c93b7dba11a54c460629919a85227f99de9a2",
+  "model.visual_tokenizer.visual_model.blocks.6.mlp.gate_proj.weight": "f5d1da44070f177d85f7c351e97ef789779d61d9031136f4b5ba9636c0f97fc6",
+  "model.visual_tokenizer.visual_model.blocks.6.mlp.up_proj.weight": "d582af58b4c58f213e1c09f2e91d9df39b98b6c504f15175af0158bd9e73cb4a",
+  "model.visual_tokenizer.visual_model.blocks.7.attn.qkv.weight": "83d81799c55defe347f15d92942f43cba58204289e253c15b1ff437f81f91a09",
+  "model.visual_tokenizer.visual_model.blocks.7.mlp.down_proj.weight": "238d8a37b9c88520d5e5ebc9e2a31ef8d8da1216e4083c60be650438c8d420cf",
+  "model.visual_tokenizer.visual_model.blocks.7.mlp.gate_proj.weight": "29d8f56eaa7816c1b2d8f26fafe684758adddd1ec8e511f6c2cd202bda04ea3b",
+  "model.visual_tokenizer.visual_model.blocks.7.mlp.up_proj.weight": "3943f3a4676cc192fb7909e737ed9e503b6288b459bafb843b2c2bafdc297b33",
+  "model.visual_tokenizer.visual_model.blocks.8.attn.qkv.weight": "2f474d4208a1a4b90440a5cf02cd92209d3bb5c8eb018cc2776c7d7727dfb1a4",
+  "model.visual_tokenizer.visual_model.blocks.8.mlp.down_proj.weight": "55123e16b5aca2c9beb2670613bde46f1808e7b26aa080a6f463f9ad4405c9ff",
+  "model.visual_tokenizer.visual_model.blocks.8.mlp.gate_proj.weight": "cc66a33408b4115f47ae673b9d482143e0387c39fa4ccdb62e01659c7ed0883f",
+  "model.visual_tokenizer.visual_model.blocks.8.mlp.up_proj.weight": "60fd3184cb53fe06a45f1624eb5d4256a1f3f8d00fd378839fe981e1e7708057",
+  "model.visual_tokenizer.visual_model.blocks.9.attn.qkv.weight": "4a2c95d949d5ee1f1026b0ff0c55fe5049b77d7f3ac2f305d608188c2d08ff8a",
+  "model.visual_tokenizer.visual_model.blocks.9.mlp.down_proj.weight": "2ddb9e08a9c6368aefb694ecba8bd4b10302e8bbfa2d4fb6de979576ef6d4ad4",
+  "model.visual_tokenizer.visual_model.blocks.9.mlp.gate_proj.weight": "6771a413d734233eb4ff16beb9493dcf3feb6cd3ad9dd6de056a5a6212fa8d3b",
+  "model.visual_tokenizer.visual_model.blocks.9.mlp.up_proj.weight": "0b3c6bff6287f7d625e8f66605678991264510bfa264b569331b588955a8dd8a",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks.0.cluster_size_ema": "4e9a6e4b9b1e203ebe18ed20095cb95c429a537a3cf814c7c5b357b823607850",
+  "model.visual_tokenizer.visual_bridge_model.bridge.ln_q.weight": "ac06859c83f6c39df39c23d5f309780141567c3f6139d41622780f9f65b5305d",
+  "model.visual_tokenizer.visual_bridge_model.bridge.mlp.0.bias": "1c2061286760d6b19f6515b6a8078605d69083b2dd3e3e8ab1ca1cbe67b1e3ea",
+  "model.visual_tokenizer.visual_bridge_model.bridge.mlp.2.bias": "015ad851cf0b3059d3220035348c859248faf3503b9dc9e9e281b7a3a3e1b96a",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quant_conv.0.bias": "1aae2dda4016febd2765e64d20dd992319d388cc8a8690f1ae5f7984a4734dd3",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quant_conv.0.weight": "8a6b690a44605e2e52a32ae3b567443033deae0c614d00ea032807760a796e47",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quant_conv.1.bias": "85a6e0d54dc1a8e31f9facf1e49a29a0ee9ebcb5a3a63b54e04533683a917ab0",
+  "model.visual_tokenizer.visual_bridge_model.quantizer.quant_conv.3.bias": "721302203a2063d1292c0d2cabbcd4b77cdfaa606b4237b3cad4748a51ea96a9",
+  "model.visual_tokenizer.visual_embedding_layer.pre_buffer.pre_layernorm.bias": "33b9d30c3f131d7d1e00fd75981f27ae9b3a51df170bd02c94525143c8a171bf",
+  "model.visual_tokenizer.visual_embedding_layer.pre_buffer.pre_layernorm.weight": "ffc8acb3f03851b77d79e4b66f21ed52efd8d0f3b91b6b6547a826341a4f7f78",
+  "model.visual_tokenizer.visual_model.blocks.0.attn.proj.bias": "805d3e1f7a7e7d6435455cfb8c725da7ad32c33dc311aa50ba608f1adb311ed5",
+  "model.visual_tokenizer.visual_model.blocks.0.attn.proj.weight": "1d1064bcd3d1b2fd6d39e97f11d9e043d4c1874d3a57f4cfc58bdb7eadf0e052",
+  "model.visual_tokenizer.visual_model.blocks.0.attn.qkv.bias": "bf9b49bdf98361207bc4d33b822a658d0ffa1ddafc6aa8a8cf968d5a3e4db637",
+  "model.visual_tokenizer.visual_model.blocks.0.mlp.down_proj.bias": "e6b1adff93671009c243fc7bd2743276f6d25021e9d28eeabae03283ca60700a",
+  "model.visual_tokenizer.visual_model.blocks.0.mlp.gate_proj.bias": "56d9823d7558d4f8dc333586d3e3dfaac30110f8f59322097888961ad51a65d9",
+  "model.visual_tokenizer.visual_model.blocks.0.mlp.up_proj.bias": "0c9a409da124cdad9e03b8adb5104f7291c3e4e18c89c4a092c598c6122235da",
+  "model.visual_tokenizer.visual_model.blocks.0.norm1.weight": "37bdb238976923b82b3ffc2331505e2048d744986d1e2d232b80f00eb44df778",
+  "model.visual_tokenizer.visual_model.blocks.0.norm2.weight": "5494b0a55d1eb9f6fbcac9d945036f9655a03c025a57cb93a2a8b81fc3961dfe",
+  "model.visual_tokenizer.visual_model.blocks.1.attn.proj.bias": "6e8a69119206c75940b1a28b6af4b25b486c07ebbf861d5092dc6b71f92d49da",
+  "model.visual_tokenizer.visual_model.blocks.1.attn.proj.weight": "32b6c62690ad49ccfa08d6545cdd67e72d87ed5b0f11c5d0b9f517a9683c9c34",
+  "model.visual_tokenizer.visual_model.blocks.1.attn.qkv.bias": "79878eb105851f7a9e38fff89bad04275dda8dd6e8456924db7f664d99f4e759",
+  "model.visual_tokenizer.visual_model.blocks.1.mlp.down_proj.bias": "1a60617fddf254c713e966160e85c420cd0a1b93917efe13e1bde6fbac499504",
+  "model.visual_tokenizer.visual_model.blocks.1.mlp.gate_proj.bias": "aba8d299d6ce01e901b8c1d9a6b32e6f2e65584c7369c3f187a5aaeaa6a8af44",
+  "model.visual_tokenizer.visual_model.blocks.1.mlp.up_proj.bias": "fe2df42102b77f40dd8f49f361e4fc150388ffae67651010a5b2a63b17495d2b",
+  "model.visual_tokenizer.visual_model.blocks.1.norm1.weight": "af48ce0fc312770716e49355bfa61a55f13434878e7eec11ad6c3b92b6913454",
+  "model.visual_tokenizer.visual_model.blocks.1.norm2.weight": "18bf90d6d1deac998361fe8e0f179dc316df318a2b59b9e8cbb8631f24b609ee",
+  "model.visual_tokenizer.visual_model.blocks.10.attn.proj.bias": "75073d3931f9baf85f5cbdd1ace5dc26a0b96b7b8476fb8fe329a015bc3a153a",
+  "model.visual_tokenizer.visual_model.blocks.10.attn.proj.weight": "2ae37496e727d1a1a4385e59ec8029956389bf1f38560583454f8c6a642b0fb6",
+  "model.visual_tokenizer.visual_model.blocks.10.attn.qkv.bias": "0cfbd094e1d2c1f9d86eace9c8240db781874d7d57789c0976cb8344c1f185c0",
+  "model.visual_tokenizer.visual_model.blocks.10.mlp.down_proj.bias": "1f4c52098d8c9f94d4b4cef069b8825edf86bb239270d4910ed52a6cdaff733f",
+  "model.visual_tokenizer.visual_model.blocks.10.mlp.gate_proj.bias": "4f3ccdfc3f26e11e178b81241a6652a13e860e84884558e3f6e9f8954c9e0509",
+  "model.visual_tokenizer.visual_model.blocks.10.mlp.up_proj.bias": "c36395e4090beed8703a35b42918fd444aae740d4167e221b8c8b06a0b2a479a",
+  "model.visual_tokenizer.visual_model.blocks.10.norm1.weight": "1ce6859c241866bad440787516009c2f03eb3226b79f1bb04090df92c72e6573",
+  "model.visual_tokenizer.visual_model.blocks.10.norm2.weight": "f93ac720e94ee1ccf8cdfed29c80a60b71562797af79a1d5c44854de3c173987",
+  "model.visual_tokenizer.visual_model.blocks.11.attn.proj.bias": "79b1ef988558d82dfb427d474ae94ca3de0374bbf6f838b04f8c0db5bbf6fe80",
+  "model.visual_tokenizer.visual_model.blocks.11.attn.proj.weight": "3146d1cd44cbe6eb8c8febee8bb716d83ac9d194e5f1c463ffc3b498711a1703",
+  "model.visual_tokenizer.visual_model.blocks.11.attn.qkv.bias": "cf0ef3d7ed45796613aad614659dd889feffab8f84b00363a787889eb4dab4e1",
+  "model.visual_tokenizer.visual_model.blocks.11.mlp.down_proj.bias": "e2300542ce45e988577827eaab069967bd72ce0b8f4209f27f20a1e95d895e84",
+  "model.visual_tokenizer.visual_model.blocks.11.mlp.gate_proj.bias": "b8d7a0f6d85d4c6eabc29e9238b85a06a3f5bd01ab491cd4ab641c1834cd7ebf",
+  "model.visual_tokenizer.visual_model.blocks.11.mlp.up_proj.bias": "523fa9553c027177ecb46ced349c364b476e99beb5c4e1fa274efbbbdd149816",
+  "model.visual_tokenizer.visual_model.blocks.11.norm1.weight": "e515143ff7f34ed5f5070e1bb8ad7608f02235d932e55c70069fb42eabbe010a",
+  "model.visual_tokenizer.visual_model.blocks.11.norm2.weight": "b87345f708aa896c13feb373249381a9cc75e0db31a9220d096d04f5a9554fef",
+  "model.visual_tokenizer.visual_model.blocks.12.attn.proj.bias": "0fa908c394b772b159cd80c720d7a85d264aec9c416b22b36a07a75c1e445927",
+  "model.visual_tokenizer.visual_model.blocks.12.attn.proj.weight": "f2e3cfbcab93495849ea28156c922ca6aece47e7bc9e3166094bc2bb7a018139",
+  "model.visual_tokenizer.visual_model.blocks.12.attn.qkv.bias": "f5fbd4f74f2644c3f41d7d86236cf71ebb06d011210132e655c3711dd5d3716a",
+  "model.visual_tokenizer.visual_model.blocks.12.mlp.down_proj.bias": "6981d61834157c9c7bbcbace255c636d346cdc5c974790538946fd560b715d9b",
+  "model.visual_tokenizer.visual_model.blocks.12.mlp.gate_proj.bias": "7d9ee67538c8a80d8c76ca2b85a58627288ce7cbc1732ab5700fa9b0cdb1053d",
+  "model.visual_tokenizer.visual_model.blocks.12.mlp.up_proj.bias": "f24faf6d7302d40ae26cbf5690a1c2eed910543621f185f022ed16214d1dae8b",
+  "model.visual_tokenizer.visual_model.blocks.12.norm1.weight": "32544d82c6d35920b44f27f38c11511290215682c44b6f7cb2cbda19661ba90d",
+  "model.visual_tokenizer.visual_model.blocks.12.norm2.weight": "bdcd10e0c160588bde7c2b0dff87f2d2498f557d70178b990b4a628b6850990e",
+  "model.visual_tokenizer.visual_model.blocks.13.attn.proj.bias": "61f386487d9cd124d3f6838c46694a0ebc6c8c2fe202bd5c155b747c1c6b877e",
+  "model.visual_tokenizer.visual_model.blocks.13.attn.proj.weight": "72d507882d5f870b427f985a40807cd94e5788ad04512fcf121d8d3ab0df176d",
+  "model.visual_tokenizer.visual_model.blocks.13.attn.qkv.bias": "24838018b610eb1e098418ed359a6f9dde44ebe6f2b75fd3e16e99e3a16282eb",
+  "model.visual_tokenizer.visual_model.blocks.13.mlp.down_proj.bias": "c9a414dc1f75659454ab9c5caed884f0690fad2c86d4504062e0ee365f041757",
+  "model.visual_tokenizer.visual_model.blocks.13.mlp.gate_proj.bias": "46a79be2018941398bf544f67b5cb2b4515e6516bb482fa70ac72481a3b354a1",
+  "model.visual_tokenizer.visual_model.blocks.13.mlp.up_proj.bias": "e8986b416075baa2cfa0981505c93215a4642d2c6c675dfb78ba57af4abc532d",
+  "model.visual_tokenizer.visual_model.blocks.13.norm1.weight": "c681093b60ffc00811659142243746cbd32e2f57cd3a530ca1e688dad7038d3f",
+  "model.visual_tokenizer.visual_model.blocks.13.norm2.weight": "a79d148291180528faed52656b1d3c710fa7052d68dd7e7a775673428079598b",
+  "model.visual_tokenizer.visual_model.blocks.14.attn.proj.bias": "f66dcc415719cdf2e01338bc07d83761d7539682610482fefbbe2940beb7dfc1",
+  "model.visual_tokenizer.visual_model.blocks.14.attn.proj.weight": "90a4743a5c8f2876fc45c240a8224dc91ba157376cc24af69513cdd45b10a2ce",
+  "model.visual_tokenizer.visual_model.blocks.14.attn.qkv.bias": "45fcf18461d98ce44cf559dcfb9c3fee51899f08a7dee19d5ef5a721441e0885",
+  "model.visual_tokenizer.visual_model.blocks.14.mlp.down_proj.bias": "3b418d40c0c566386bcc40e0c2f19da68c560ccb00f90e80309a69d9e219edbf",
+  "model.visual_tokenizer.visual_model.blocks.14.mlp.gate_proj.bias": "0ca6e135ee577dfd1ce5429030507df2e069017d903e92535c70c03fd83644d1",
+  "model.visual_tokenizer.visual_model.blocks.14.mlp.up_proj.bias": "128ca9719664233b52a13f21a5e762ec9531bf66af6ce21cabe1cee7f82d03db",
+  "model.visual_tokenizer.visual_model.blocks.14.norm1.weight": "d9bcd46f9ac22327e07ffe424020cc975d863d9e66abf0bcfac42fce4d7b5989",
+  "model.visual_tokenizer.visual_model.blocks.14.norm2.weight": "1b38f516859c2aa265beea086e0ed9165ee39e2f955d9b84cac9db3163b400e8",
+  "model.visual_tokenizer.visual_model.blocks.15.attn.proj.bias": "ce7860ceebdcaa2389d379a1268e8fe2f63a25d9cb48b9891cae7208a97b15b6",
+  "model.visual_tokenizer.visual_model.blocks.15.attn.proj.weight": "a76fa437974204d45bb891cc9c7cf488bbabf0cba249cb53fbae2187e9dcf4c7",
+  "model.visual_tokenizer.visual_model.blocks.15.attn.qkv.bias": "4bdb939249fc44378c7021ba51714e9a036937ef1ccf6e9a3865f03d4845529a",
+  "model.visual_tokenizer.visual_model.blocks.15.mlp.down_proj.bias": "c1956150291221c171bd6e37bbd9a19101ac3d8523df3b4bdd738926fb361e65",
+  "model.visual_tokenizer.visual_model.blocks.15.mlp.gate_proj.bias": "3fc53de8cd573a9727ee2bd09c0814dd4f62caefdd516884eb323c1d70334825",
+  "model.visual_tokenizer.visual_model.blocks.15.mlp.up_proj.bias": "82e31055a75f4ee825302fa3658ca4412e5c3c86b3c7b490c72991b5d30f9525",
+  "model.visual_tokenizer.visual_model.blocks.15.norm1.weight": "854a4f95c2897d2c56cf23981b64d352f2f966c2cd47f5d9d09b3cbc10f24dda",
+  "model.visual_tokenizer.visual_model.blocks.15.norm2.weight": "396834a1608e91bfb625d591a5ae35b4058e796f211aebc2f962177a099fb05c",
+  "model.visual_tokenizer.visual_model.blocks.16.attn.proj.bias": "be0742fac52248e6a2c38f20098fe6362a6c1ed85ca55b8acfae09cfe13099e3",
+  "model.visual_tokenizer.visual_model.blocks.16.attn.proj.weight": "9038a6384419c66b5bf6a1291fea6884ea4deff18439fd931059b8d5dc19181c",
+  "model.visual_tokenizer.visual_model.blocks.16.attn.qkv.bias": "ef93f2c9cf873c58aa220ee4f07aa9d74fd3bff2dcfc68f1ba3235303994fd10",
+  "model.visual_tokenizer.visual_model.blocks.16.mlp.down_proj.bias": "c1723050077196d86ac28bb7116a36feb3dfa0448716ca0aeebae411afaf3fd0",
+  "model.visual_tokenizer.visual_model.blocks.16.mlp.gate_proj.bias": "81970e916ea3c3f36eff77c36bc1d86689da7b8beb30d422350ec7bae5d92b85",
+  "model.visual_tokenizer.visual_model.blocks.16.mlp.up_proj.bias": "ce0ae8808843a2eb931dc0fd9758a518b0433b962b87e6026c3bb31b92a701ed",
+  "model.visual_tokenizer.visual_model.blocks.16.norm1.weight": "0e8eadcb318bb2355fce16327e13375594f354455d763788ef1482297853e288",
+  "model.visual_tokenizer.visual_model.blocks.16.norm2.weight": "9575769ecca0fc94d8baf0fada6b0a59727490a36807e1aa1d11ce62b8f70103",
+  "model.visual_tokenizer.visual_model.blocks.17.attn.proj.bias": "f0c8759beedfe7928e6964350c0cf1d63512178e2d35c87bc03bfb8753d5aa33",
+  "model.visual_tokenizer.visual_model.blocks.17.attn.proj.weight": "53799ccebe1979521642439f5c9335f28a58b3ef767c9c8b1ec0b1c395221fca",
+  "model.visual_tokenizer.visual_model.blocks.17.attn.qkv.bias": "99a35b72130ce0d3cf293faa679f99a7c098cb52761b65ec6ddc012bfd0617f0",
+  "model.visual_tokenizer.visual_model.blocks.17.mlp.down_proj.bias": "b4026e00675220f91d4865893fea35d55189e9229eebe8a59302f057d0ec3cd2",
+  "model.visual_tokenizer.visual_model.blocks.17.mlp.gate_proj.bias": "a0c57e9d40f2022f614f0413cb3bff490772fb38f1d94b759161a73a241a377a",
+  "model.visual_tokenizer.visual_model.blocks.17.mlp.up_proj.bias": "813ce4b9c1ccbe46e06b5644354ff039d1d0519db302478e29d2055d7844e3fd",
+  "model.visual_tokenizer.visual_model.blocks.17.norm1.weight": "0bedb261347e1f6a0e379b4d45073cdc89c06775d0d58bf443ed3acc9c2de084",
+  "model.visual_tokenizer.visual_model.blocks.17.norm2.weight": "bb6239127441f4c5e0d4a9ca8f43d3ffd57aff892391beea49945ccc0982a6d9",
+  "model.visual_tokenizer.visual_model.blocks.18.attn.proj.bias": "dc6f5e3c90c04ada09ee3962d03d09657aad82de45b5ea71350828502155f813",
+  "model.visual_tokenizer.visual_model.blocks.18.attn.proj.weight": "ebd11e5fe3e67632d17b33ce19c1a4d9d010070ba80596e9b18ec8dd557b5c87",
+  "model.visual_tokenizer.visual_model.blocks.18.attn.qkv.bias": "459f81f5b78192ddb0b7045aba9230d97e9c4e8715d3e3782867b90fd6d24d0c",
+  "model.visual_tokenizer.visual_model.blocks.18.mlp.down_proj.bias": "571d7e078e2692fa47b72de6fc7716788112fe2d0986c116e68c556ea0d2da06",
+  "model.visual_tokenizer.visual_model.blocks.18.mlp.gate_proj.bias": "ec7a00f14d8f30a6ecbc66e0c8267d934fffe8404c329665fe0814dd61d6bf67",
+  "model.visual_tokenizer.visual_model.blocks.18.mlp.up_proj.bias": "2244a78c23e0251cce1ab93b8c54b7346362cf5db0f35e7097ac42db075a40bf",
+  "model.visual_tokenizer.visual_model.blocks.18.norm1.weight": "c2a5ad2a350b0619fd406758b5131532979289000116b32329634707de4e2609",
+  "model.visual_tokenizer.visual_model.blocks.18.norm2.weight": "04ed9a499c933ef96885b67f5707daceea80c192b596a48338e23c230be02167",
+  "model.visual_tokenizer.visual_model.blocks.19.attn.proj.bias": "993bb84d205abe7af609295e8228be2b2514bb586b679693b09a1137e527bfd1",
+  "model.visual_tokenizer.visual_model.blocks.19.attn.proj.weight": "fbf37493f4703b5785c118b2206a5f9fddb2db93ef81b2f8a5286054f167be3a",
+  "model.visual_tokenizer.visual_model.blocks.19.attn.qkv.bias": "e39fb1e39a17e5d7ebebd00fff9224ba6734eae2baa894e6fb37175509506dc9",
+  "model.visual_tokenizer.visual_model.blocks.19.mlp.down_proj.bias": "4c37402ab040e47505351487dd70d5d5e20ce1b92844e1a5143386e2a509c102",
+  "model.visual_tokenizer.visual_model.blocks.19.mlp.gate_proj.bias": "406c6b1089c8a9189f211313970993da646dba933b0f8787da095260fc1c75e6",
+  "model.visual_tokenizer.visual_model.blocks.19.mlp.up_proj.bias": "7515e1b9ca05e22aa66ff32c18a73053076f5443efa94e78b96b3cd0d1c9361f",
+  "model.visual_tokenizer.visual_model.blocks.19.norm1.weight": "ac6e5d48bf0b7ddd5490f5ac4a7013a1e9e4d8409087b7eba0ddb95b877f5d23",
+  "model.visual_tokenizer.visual_model.blocks.19.norm2.weight": "ee350e8f73dd6178cc9fbf50918f0515787d757b19110b7289da95dcf127cc94",
+  "model.visual_tokenizer.visual_model.blocks.2.attn.proj.bias": "d63fba4c79aa197602f5d4d73494e1078be2a07b060103e24785753894db5e79",
+  "model.visual_tokenizer.visual_model.blocks.2.attn.proj.weight": "adbded99ed29b426e39706f68e72ed769d0d56f8c0050160123946d57d4d6871",
+  "model.visual_tokenizer.visual_model.blocks.2.attn.qkv.bias": "aa1369dd9eba31b4217ca24daf4b764587466ef80b3e377a6999104407681548",
+  "model.visual_tokenizer.visual_model.blocks.2.mlp.down_proj.bias": "bfc95d790de6a8ff2246fa8faa852631199d0e9ffa754dc0756c81f3855dfe55",
+  "model.visual_tokenizer.visual_model.blocks.2.mlp.gate_proj.bias": "53d3ed6a4fb7b5e89972ab2d5c1a9e0ec13b3e0dda1fbb839cdc86a547bb3e70",
+  "model.visual_tokenizer.visual_model.blocks.2.mlp.up_proj.bias": "077327176d6c711bb64ea64ed188bc2f783c79e90451abab40381682c31a78ca",
+  "model.visual_tokenizer.visual_model.blocks.2.norm1.weight": "090f41ffce0f4bc1e6f311664646488df7cbf239f59f2c82dcad18923db78830",
+  "model.visual_tokenizer.visual_model.blocks.2.norm2.weight": "aec692beea96ff8724c5270bb5ce1aaadcdb9c3ed5db5ffd559439b38098ae72",
+  "model.visual_tokenizer.visual_model.blocks.20.attn.proj.bias": "7dd8916402d4e79f207a4edbd97a9d9b1e5f4ed9232c55537fb8a4ffa9939a9a",
+  "model.visual_tokenizer.visual_model.blocks.20.attn.proj.weight": "f98d96f5abb4cbc7c795a71381e21b54a3609490a1c64e7a0ae5a99968b5120a",
+  "model.visual_tokenizer.visual_model.blocks.20.attn.qkv.bias": "1f3ce28ea45079605d62e1a930df76fef6c2e21c554b9dfa00820e5e415c916a",
+  "model.visual_tokenizer.visual_model.blocks.20.mlp.down_proj.bias": "b0ec78db47a0152a54c86cbe1224d7fba4d6f27329192a6f6325cc3f78dfe4fb",
+  "model.visual_tokenizer.visual_model.blocks.20.mlp.gate_proj.bias": "9a0e6b955206470765ffbc8485b29ffac0e68708e58bf3fb400c2de6e9d01b4f",
+  "model.visual_tokenizer.visual_model.blocks.20.mlp.up_proj.bias": "0c3238a7449bf1ec5dee3e2e2b967ee63e37dc7a644778e275741fb2cce2c89a",
+  "model.visual_tokenizer.visual_model.blocks.20.norm1.weight": "259174b9b357bb0de8b215f68090567e3eaacfc17d3e081811b6f1aca4c89300",
+  "model.visual_tokenizer.visual_model.blocks.20.norm2.weight": "1758ee89e1696461c756d8d3a101a2f8474d4e2daed9baf91c6d6c824375a545",
+  "model.visual_tokenizer.visual_model.blocks.21.attn.proj.bias": "7289531b3b225f7f8e02e2347242b4e4414f8a352dd61df923bf43eb7c97efa0",
+  "model.visual_tokenizer.visual_model.blocks.21.attn.proj.weight": "d97972b99e8a7fc88b3896ba867a20977dd39686606ff0bce948170bec0a00c1",
+  "model.visual_tokenizer.visual_model.blocks.21.attn.qkv.bias": "314864073fc74cad288c006b38cabc74f97ddf241a09d4dcdcc289d6e76149ae",
+  "model.visual_tokenizer.visual_model.blocks.21.mlp.down_proj.bias": "eaa6d9886e12e97bf3d59d5968dab5689111c3dce643208932350f880945548f",
+  "model.visual_tokenizer.visual_model.blocks.21.mlp.gate_proj.bias": "83ae37b3000f6c7ca3dbb2557dce28636f08b3fae1f49ef307d3a9e59f610f6e",
+  "model.visual_tokenizer.visual_model.blocks.21.mlp.up_proj.bias": "b0cd5a73912aa34e0783d0da7cb9ae1fdc7a241ea7b44acb41c91196cf951021",
+  "model.visual_tokenizer.visual_model.blocks.21.norm1.weight": "d09c53011e76d00e74a6db06b284358f021331004112c9524d10606a7f660f0e",
+  "model.visual_tokenizer.visual_model.blocks.21.norm2.weight": "171e74c5e89ef2d1b7d6a11dd80da997ff210f0d8bd9f12aed1e84efb2d80abe",
+  "model.visual_tokenizer.visual_model.blocks.22.attn.proj.bias": "163f5d0831f07ec6706f20eae45bfe4525e1be80a44e622d056fe50fac92adf0",
+  "model.visual_tokenizer.visual_model.blocks.22.attn.proj.weight": "14df80711c0ad423aeb110defc1307f92bd55e7c5a947d10745109243be61b6d",
+  "model.visual_tokenizer.visual_model.blocks.22.attn.qkv.bias": "4780d09b95c774c4d54bfd0ef5f2d6190adfad7bc8ebb0382600820f5e789008",
+  "model.visual_tokenizer.visual_model.blocks.22.mlp.down_proj.bias": "02baef1446c3d074f80049d89dbb6d1570cecd3f807a12531f170ae4ca35103f",
+  "model.visual_tokenizer.visual_model.blocks.22.mlp.gate_proj.bias": "674d2576035209de49ecced1119b637ec15f3763995d0004ca50e4a414fd9599",
+  "model.visual_tokenizer.visual_model.blocks.22.mlp.up_proj.bias": "3e7dd489f4bc629b936fd03831057720c9b177c4bd465b3a974f5b3040fc7794",
+  "model.visual_tokenizer.visual_model.blocks.22.norm1.weight": "122fe1b26062854c1154dfdaf0d282db60387abf3a3df7f0818a692306026fcc",
+  "model.visual_tokenizer.visual_model.blocks.22.norm2.weight": "a53a61401b3905fb6cc6ec739ec8978434b5d4a179cd010d7c830b06f833f4e7",
+  "model.visual_tokenizer.visual_model.blocks.23.attn.proj.bias": "b45d0eb3b02a59ca64cd96143ad1ab09e18ffb7fe923e1f94470303958ee9451",
+  "model.visual_tokenizer.visual_model.blocks.23.attn.proj.weight": "1069068a4f277d898c4f45f369a974b70525f3111709b579ca9f8680462680b3",
+  "model.visual_tokenizer.visual_model.blocks.23.attn.qkv.bias": "6d1af895c6a310d076259990d7d80fb65d39a79eff83a10c89ef481806d7dbb0",
+  "model.visual_tokenizer.visual_model.blocks.23.mlp.down_proj.bias": "c50a0428c25f24eb47641ece99d6485531207d7a3cceb14cb7da01e0884e27f4",
+  "model.visual_tokenizer.visual_model.blocks.23.mlp.gate_proj.bias": "45f8e7c83bb813eee73b06bdc5745d50eeec0d38d93cd7805fe3305083a02b93",
+  "model.visual_tokenizer.visual_model.blocks.23.mlp.up_proj.bias": "9d1ee8458a0a73e8dc4353fac48e08ca3ef4caa6e99147884742c2d993881b31",
+  "model.visual_tokenizer.visual_model.blocks.23.norm1.weight": "34383d4558dda560ff4929849b14f06277f0273a1f76283a950ceabfd124520e",
+  "model.visual_tokenizer.visual_model.blocks.23.norm2.weight": "a69d5d2ed30e8a12df8171a7300faa7e3c3aa285ac70b0b2322683abf50ae6a6",
+  "model.visual_tokenizer.visual_model.blocks.24.attn.proj.bias": "472bfc593b5c6f9e8811c835b4ec15fa3116da37f8e38dc36b8348bddbc82b02",
+  "model.visual_tokenizer.visual_model.blocks.24.attn.proj.weight": "637550f8d7d9db269ed81d4bb7428293ca412a98191df5d22396fdfa2e094467",
+  "model.visual_tokenizer.visual_model.blocks.24.attn.qkv.bias": "ac6f5edb704b2064f02ee7b87456b60a11bb449f6d9c1ba51c3ef23c26885321",
+  "model.visual_tokenizer.visual_model.blocks.24.mlp.down_proj.bias": "f8fa835aa2287073a26071b7fcb2a82a6bfdf6ff07d0f929d789bc1b9c52bfed",
+  "model.visual_tokenizer.visual_model.blocks.24.mlp.gate_proj.bias": "589a63b15f498452480a26d50589471157554852ece4240fed1ae5895100f95a",
+  "model.visual_tokenizer.visual_model.blocks.24.mlp.up_proj.bias": "e2a17b0c2e1080c57afa17cc45a85b53021c8317fe160c8a5ac9ef738d770bb8",
+  "model.visual_tokenizer.visual_model.blocks.24.norm1.weight": "23179f5e5bbc4d21127793bdd3d735d176733769a4c4b25b0adf33a6b1624b21",
+  "model.visual_tokenizer.visual_model.blocks.24.norm2.weight": "e30e9330be6fe18565f391df61152736ba3cc92533e6240f64109849f9b5b14b",
+  "model.visual_tokenizer.visual_model.blocks.25.attn.proj.bias": "ecfe7eab944d41cfef31fca9ca4110d9b5a522023194ad21b5b8909186eaba3a",
+  "model.visual_tokenizer.visual_model.blocks.25.attn.proj.weight": "6f9fbaa6fb9240cdc5064ea2a10befd28901e9c6e10f0e05733c2276c694cf6f",
+  "model.visual_tokenizer.visual_model.blocks.25.attn.qkv.bias": "8392af005375329d12c5f0a8ef9f23424545515d4e576181a0709488192e00b9",
+  "model.visual_tokenizer.visual_model.blocks.25.mlp.down_proj.bias": "adfc87d95812363760beb1471356ae7ec03abfdf8c5269b12fe06008a0720be1",
+  "model.visual_tokenizer.visual_model.blocks.25.mlp.gate_proj.bias": "34fbcf4ec68faf97c7702901ffec61f477da03567d79328ce1e1ab443ac20750",
+  "model.visual_tokenizer.visual_model.blocks.25.mlp.up_proj.bias": "a96e98c0fa45e96095e6cd1c92ed4082b428d30c4f83ba1f82e1782719e9db5c",
+  "model.visual_tokenizer.visual_model.blocks.25.norm1.weight": "1238c42972ed951e9a6c490975c53df6c970f78a159b12402dda13c249b4ed9e",
+  "model.visual_tokenizer.visual_model.blocks.25.norm2.weight": "28830dca38ade790d4f7700d4d9b426b3e42d3d1eab25fa0b93d28f779a876a5",
+  "model.visual_tokenizer.visual_model.blocks.26.attn.proj.bias": "14cc70e6092a5e97d8af683b8a6bb738e439baf12f779b32b9b253736d629562",
+  "model.visual_tokenizer.visual_model.blocks.26.attn.proj.weight": "8030ea4a9e51a6bfbba398700991e8cd52a3b5bef56dbf4145a3fd80e89d68b4",
+  "model.visual_tokenizer.visual_model.blocks.26.attn.qkv.bias": "ac8f9ade0b0a6df2a925788c13ad70a5434770d635228647ba049b0a1397c99f",
+  "model.visual_tokenizer.visual_model.blocks.26.mlp.down_proj.bias": "ac5765e4b18ce143539ca20d7145ae4547c49006f96f5056cd9b9dd256f6b5c1",
+  "model.visual_tokenizer.visual_model.blocks.26.mlp.gate_proj.bias": "494da881cf4154fd673b7a6f03ff2c6aeec91447a3588452cd313669f5ee022e",
+  "model.visual_tokenizer.visual_model.blocks.26.mlp.up_proj.bias": "dc7f1de6eacdcd3afcad4ced5b1999f38a4a50d15185cb061fe37648cb73f88a",
+  "model.visual_tokenizer.visual_model.blocks.26.norm1.weight": "e09e079069dbb9b0ef27843d720c431ceac74013afb8c4901c6bf2757b2f2f12",
+  "model.visual_tokenizer.visual_model.blocks.26.norm2.weight": "16eec9d5a80666d67d93d1e99d558033be05cc11e8015836c5f5e2e32d8afaf4",
+  "model.visual_tokenizer.visual_model.blocks.27.attn.proj.bias": "87659eba91e619724e3577223a1d4b859f7779467c3c1715a6ee317cc2546ad8",
+  "model.visual_tokenizer.visual_model.blocks.27.attn.proj.weight": "df6d5bf5a386a7bdad098ff7e23c9256c47d3e43c19c4e3b2d002434ae7acbef",
+  "model.visual_tokenizer.visual_model.blocks.27.attn.qkv.bias": "16fde0f1e34f297cf606e26e8d7e36eaae05c3aa66175450a4fdd99fa16f7dda",
+  "model.visual_tokenizer.visual_model.blocks.27.mlp.down_proj.bias": "b005232e490bc6754e99e5e5e6c612214908c2d8a871e2b329768aab8fa1444c",
+  "model.visual_tokenizer.visual_model.blocks.27.mlp.gate_proj.bias": "88fddef01bcbaa00a4c02252e07464775163f18a77ddcdd0c0af8c896746277c",
+  "model.visual_tokenizer.visual_model.blocks.27.mlp.up_proj.bias": "51d9cb4ecb8b5ddd159d3175ad96e46e06433087059b6dea732888b2375efb20",
+  "model.visual_tokenizer.visual_model.blocks.27.norm1.weight": "723baf0bcc3f289def7d9647a6c13cd8502a3c71b8bb57174fdbe2a9e4c59b40",
+  "model.visual_tokenizer.visual_model.blocks.27.norm2.weight": "bce58f40d05b97958514be89518ed160fc4e28ba1d4e71242dea756972e07f33",
+  "model.visual_tokenizer.visual_model.blocks.28.attn.proj.bias": "ce974ab595eb1a080e2901853aa4de4a1542df34b10d85a939e9f15ed5bac983",
+  "model.visual_tokenizer.visual_model.blocks.28.attn.proj.weight": "9ea8a5d94cf7b45177edb5d57b702d88910c8b1e637538e4354161279220cbff",
+  "model.visual_tokenizer.visual_model.blocks.28.attn.qkv.bias": "48cc30133ecd736eab3ff550927eff08040e49a4996d976b864b1d28a713d156",
+  "model.visual_tokenizer.visual_model.blocks.28.mlp.down_proj.bias": "0c9b76781a43e267c2ea7303f681c7d5c6831349addf385d9cfad52949aa5338",
+  "model.visual_tokenizer.visual_model.blocks.28.mlp.gate_proj.bias": "14b55fc77551a0bec1b856d681767e4f49fd437704c2aaf9ad6b445f34442782",
+  "model.visual_tokenizer.visual_model.blocks.28.mlp.up_proj.bias": "6656ca4833087156699756658126d94533764c911bfc6a318fc672e31004fa15",
+  "model.visual_tokenizer.visual_model.blocks.28.norm1.weight": "479bf308614dbbd6287f7b17c779458a40dde89e8e6da2e97bcbd8c521109e0f",
+  "model.visual_tokenizer.visual_model.blocks.28.norm2.weight": "e35519e9f57a951572354215720c4f75a26a1c36c57cf5351c367923444fe407",
+  "model.visual_tokenizer.visual_model.blocks.29.attn.proj.bias": "12e2922d2bfcabd3fcb6a56504c0b261bdb6bae3c3adf6cb3f55c4967ec5c4ee",
+  "model.visual_tokenizer.visual_model.blocks.29.attn.proj.weight": "cf748c3365cb411d0d6bb4a570d939dcb4eeea338e09179bb6a6fd839dc6dbeb",
+  "model.visual_tokenizer.visual_model.blocks.29.attn.qkv.bias": "fafbe5692c1b5925050dc71e6c013373137557d45464d1a62a7c2d22fa0c3184",
+  "model.visual_tokenizer.visual_model.blocks.29.mlp.down_proj.bias": "ec72a2039095cc921ab073417eb3bdea3e81b8f58b02e533a91b2857a101b33b",
+  "model.visual_tokenizer.visual_model.blocks.29.mlp.gate_proj.bias": "d21211071cbfc42b2dad0b2664bdf702095f1f77962c09993267ea264bbfe8c8",
+  "model.visual_tokenizer.visual_model.blocks.29.mlp.up_proj.bias": "d542f42f315224e4af6180d5943f6c435604ad357d8a64252327af3379979cc4",
+  "model.visual_tokenizer.visual_model.blocks.29.norm1.weight": "c8a28b1c0f6c942fccb09686cd4525a6c0230ddf0ef3447cf88c721720b59f5e",
+  "model.visual_tokenizer.visual_model.blocks.29.norm2.weight": "24be473ac046dec0e5cdd2c2033804157ff4c8549a6593c716f14bff889c14a6",
+  "model.visual_tokenizer.visual_model.blocks.3.attn.proj.bias": "1a2cbba62b6542ef40354babf73976a6e2a5315bda8e1cd91dd63bada8f699af",
+  "model.visual_tokenizer.visual_model.blocks.3.attn.proj.weight": "b14238838f71f2aeac0c401b2ce35c9625ffb5708ae545a6afa98032e8218aac",
+  "model.visual_tokenizer.visual_model.blocks.3.attn.qkv.bias": "5258cfee50f11f7d354c27431c9328f750ae1ebd0785fb143861a765854a1217",
+  "model.visual_tokenizer.visual_model.blocks.3.mlp.down_proj.bias": "6aaa308a4c1ca2320ca615a6e20387d08509cd26a831a7607baa41c1bfcb2bc7",
+  "model.visual_tokenizer.visual_model.blocks.3.mlp.gate_proj.bias": "123cb642d7860264a32e3b3244f6fe0a21150eddc028475adf4fae91b8136025",
+  "model.visual_tokenizer.visual_model.blocks.3.mlp.up_proj.bias": "2caacf7bb3814b765a672b394b172d8578e2d5e9a5444c31f04ce475390bb2af",
+  "model.visual_tokenizer.visual_model.blocks.3.norm1.weight": "d5e93baea2f04cd5381701f248c34957e170d311ef67165c72a55d8e1e8c57d8",
+  "model.visual_tokenizer.visual_model.blocks.3.norm2.weight": "504be3de5013a42bbac342c3663ea4dadac6ac2b58d6bc5ea75021d2f563a179",
+  "model.visual_tokenizer.visual_model.blocks.30.attn.proj.bias": "6d48fccb29833a4e8e0b9d1f70c2258c7be573d921164b04ab302a2174035f1d",
+  "model.visual_tokenizer.visual_model.blocks.30.attn.proj.weight": "f6ba9628c38f8005faaf84e5357b9a8c21eee8e575a5333f7e683de792240e8d",
+  "model.visual_tokenizer.visual_model.blocks.30.attn.qkv.bias": "a525da18c53385d394077f52ae0958ac92ad4e22ed6a51788a322b8123d76075",
+  "model.visual_tokenizer.visual_model.blocks.30.mlp.down_proj.bias": "43a14ebce9b5201d15953e51e5841d3e8822b29d522f11f7ff3b6ba3234a248c",
+  "model.visual_tokenizer.visual_model.blocks.30.mlp.gate_proj.bias": "728c6d857e1aa918cccb97b45f022a1674783ef37a8c5a75e79398ca52656718",
+  "model.visual_tokenizer.visual_model.blocks.30.mlp.up_proj.bias": "03b033d7a356e0ad451c41b64ca49eaa7c75f7b8c25e542e11e96ae1d6368695",
+  "model.visual_tokenizer.visual_model.blocks.30.norm1.weight": "2450e34112c9974c9ca43b8bd0c7d70a8224465e971f4347a29416bb0fc611d0",
+  "model.visual_tokenizer.visual_model.blocks.30.norm2.weight": "8092849acf030475a9be8cb913b217474a4a5e3948cb5337d7d70b2180d18fc8",
+  "model.visual_tokenizer.visual_model.blocks.31.attn.proj.bias": "d1aeb8f8034cf6644ce1c05a83cee98d0482f2692072d152a075562b76d1cf82",
+  "model.visual_tokenizer.visual_model.blocks.31.attn.proj.weight": "113dabb9d29b5a9ec92bacd330f5d16e81fc8f489384d1ccfbc512089da88678",
+  "model.visual_tokenizer.visual_model.blocks.31.attn.qkv.bias": "6908faf0ac0d8da7030bca10b2c00020f5cf724272717bf4662bb734a8600836",
+  "model.visual_tokenizer.visual_model.blocks.31.mlp.down_proj.bias": "8e131e89eec3bb0ad3bbb07907fbe8aedcf4fc4315069393bfabf5ca0b1bc704",
+  "model.visual_tokenizer.visual_model.blocks.31.mlp.gate_proj.bias": "4f0cabf06362bb2dd4ac1c04591e9cae954dcb306cb2709e26f655f62574b278",
+  "model.visual_tokenizer.visual_model.blocks.31.mlp.up_proj.bias": "295910b3b39491d6861409d21529fe92b96c262f1b1d4f5a1539c47395143c42",
+  "model.visual_tokenizer.visual_model.blocks.31.norm1.weight": "88db8f78729960e6cd4af347bdfabf9301076f641ac278b03b1843f48b50f083",
+  "model.visual_tokenizer.visual_model.blocks.31.norm2.weight": "cbdb90111859c42ca97e36a1bcb9b4aa7f791e85bb9df274729c2f57daa7661f",
+  "model.visual_tokenizer.visual_model.blocks.4.attn.proj.bias": "14a69f1017874ce98cbacaa9d39d1ea0f9ac205ac850145ebbbcc6ed0a857d1b",
+  "model.visual_tokenizer.visual_model.blocks.4.attn.proj.weight": "ee49ba61fe1dada3e26632c75a0123eaf5cfc1aa32467f46bb3ca7672d2405d6",
+  "model.visual_tokenizer.visual_model.blocks.4.attn.qkv.bias": "c780ecf8feba2c9c87971215177b20ac70be1b40e4fe2dc3aa94074d258a9272",
+  "model.visual_tokenizer.visual_model.blocks.4.mlp.down_proj.bias": "3938e21d6b388f89f44db65dad03c82e535a21c5738a15a2b530dedf4de6ce46",
+  "model.visual_tokenizer.visual_model.blocks.4.mlp.gate_proj.bias": "5c5090a6d6fd907090be331a485f79c6bb4075f76a879f098ebc58189c9bde44",
+  "model.visual_tokenizer.visual_model.blocks.4.mlp.up_proj.bias": "e9fe07ee31d002974d75c64ecd04e0dd214b183dffda36c90a5cba9ebd31b4cf",
+  "model.visual_tokenizer.visual_model.blocks.4.norm1.weight": "88f2ea107e7a5f67168cab076217ac843dc10965ef69de44e12046e6ad50b234",
+  "model.visual_tokenizer.visual_model.blocks.4.norm2.weight": "68b2aab99c7e886378d0e83df023fa26be7a3fa2b302de0a39c10adbd732a3f6",
+  "model.visual_tokenizer.visual_model.blocks.5.attn.proj.bias": "19a73d008c09c5e8b0178f42dedc1e3f952de1d1efa73fa5f887f65c13f98a2e",
+  "model.visual_tokenizer.visual_model.blocks.5.attn.proj.weight": "67552d68ef95b058fb3c56e62b8d245d9b6f5406393692a89873ee4bd5e0e81e",
+  "model.visual_tokenizer.visual_model.blocks.5.attn.qkv.bias": "202503b3a046d98577210eac6495be8af1973109a74c62c269c4dc385a9e6524",
+  "model.visual_tokenizer.visual_model.blocks.5.mlp.down_proj.bias": "49838a6e84bfd653cc5182104603dcfda2f67a9c4d60a1682966e4d2ec0df7c2",
+  "model.visual_tokenizer.visual_model.blocks.5.mlp.gate_proj.bias": "36bcd99299fbf07678bda609f94208661797f1e263879b9f46ff3708049da51e",
+  "model.visual_tokenizer.visual_model.blocks.5.mlp.up_proj.bias": "7a1aecc076b5de2e9c37faffd1928f99b385ff29e24179227dc326021193b5a4",
+  "model.visual_tokenizer.visual_model.blocks.5.norm1.weight": "f248742b796649f19bc971dc76e015d5bd1fc3960fd628e9392bd9ffaf1d2339",
+  "model.visual_tokenizer.visual_model.blocks.5.norm2.weight": "dbdec9592470bb9deb753970e8faf80bef29b45f7ca155307da873900b58d5af",
+  "model.visual_tokenizer.visual_model.blocks.6.attn.proj.bias": "40c795e7dfcc6a66235c5c2233435355e9c495e023f09d4e2818d41820c2aca3",
+  "model.visual_tokenizer.visual_model.blocks.6.attn.proj.weight": "d3a07cce69d833e5226721e188e3bccdc3a74cb150e99cdf804dcc03dc8c31c6",
+  "model.visual_tokenizer.visual_model.blocks.6.attn.qkv.bias": "95dec5616b8e2801bb5b0ce83181dc8966391d671e2418ff7d8ef4bb27a884a7",
+  "model.visual_tokenizer.visual_model.blocks.6.mlp.down_proj.bias": "f637b1bfe77b5778b56ba6530e4653f2d308592b16ef6596d40b73286923b688",
+  "model.visual_tokenizer.visual_model.blocks.6.mlp.gate_proj.bias": "ea67711f2d71daca1b478b71b3adffd0a0e0288729a4f774eb322d5fe61ccf4d",
+  "model.visual_tokenizer.visual_model.blocks.6.mlp.up_proj.bias": "8067bbd53f1a3083e7b01437f46367f476487fcbda26a01b970065262eb06bbf",
+  "model.visual_tokenizer.visual_model.blocks.6.norm1.weight": "2ea4e24ddf066b93690d3061137fefa987249a15fb2265978d9d3f2fb20471bd",
+  "model.visual_tokenizer.visual_model.blocks.6.norm2.weight": "da4edcf16a448236bd9da06c4e96ebde67984bd387bdf5fcfeb2aa7403acf485",
+  "model.visual_tokenizer.visual_model.blocks.7.attn.proj.bias": "e1c3c24eeb191794bb5a81fed3b542b0e9be7719b05ff4c8ca86f0eb139829b8",
+  "model.visual_tokenizer.visual_model.blocks.7.attn.proj.weight": "9cd705241ca0420372a1e2c0d27619d77f33000f8bb0388e529ec6202d604cf2",
+  "model.visual_tokenizer.visual_model.blocks.7.attn.qkv.bias": "2388a918340d8023a8e847720d940831de0c925641d8f2ab234612bbc34d791b",
+  "model.visual_tokenizer.visual_model.blocks.7.mlp.down_proj.bias": "f7550cb325354c346921a4a0798b0c2f4b6571d518314e1cc16bc2ffe39ec34e",
+  "model.visual_tokenizer.visual_model.blocks.7.mlp.gate_proj.bias": "64c32c1e9ea101b8dfb3c129506610c3b59c6be45bf18c6ca50c09756f5ac357",
+  "model.visual_tokenizer.visual_model.blocks.7.mlp.up_proj.bias": "4077df962861c5d9ab06a8dc8ff870c88fdc68039197588c93444335b806fbdc",
+  "model.visual_tokenizer.visual_model.blocks.7.norm1.weight": "41920a9e26417b1abcccd2da2b8ddc70bec04a5ca2e53a54c6ee75744ce129d0",
+  "model.visual_tokenizer.visual_model.blocks.7.norm2.weight": "0b1760afe44e670a65daa2a97a17b1a31b7a2c6c9c567e33c493d7ed34dee94f",
+  "model.visual_tokenizer.visual_model.blocks.8.attn.proj.bias": "d6e99ecbe2d51e7e8cc5e78db872556bd5b677e2e66c9a07696a901e52fa2a67",
+  "model.visual_tokenizer.visual_model.blocks.8.attn.proj.weight": "58beb51ce824c65232a3abcc8a2fb004bc1ad1fca741d5bffa5b1b2ae820f40f",
+  "model.visual_tokenizer.visual_model.blocks.8.attn.qkv.bias": "425e7ad7e1d4e653c13345ceac1e744c60517a6df679a788f237ddc8987ad9dc",
+  "model.visual_tokenizer.visual_model.blocks.8.mlp.down_proj.bias": "081aa5ba330845bbe5cbc39d65756f540df4f92121848ea7d1f37da357940ab5",
+  "model.visual_tokenizer.visual_model.blocks.8.mlp.gate_proj.bias": "aa4fe0c97219fd63688acf9b107f17062ea3c1222604fbcca31c53f99d86bd86",
+  "model.visual_tokenizer.visual_model.blocks.8.mlp.up_proj.bias": "1a0b9aed1abb2fa598c30fb35d09d1b01d374d5784de51207f14fdad8fcbdffd",
+  "model.visual_tokenizer.visual_model.blocks.8.norm1.weight": "015bba8059c443bff5bf9469be7f6b4f56bfdc9fb8940e3d6f79a694e7aed2c0",
+  "model.visual_tokenizer.visual_model.blocks.8.norm2.weight": "109161cc7df0bc191e7ea693dff54cb44a3050164a8771006742f36ba39a6a09",
+  "model.visual_tokenizer.visual_model.blocks.9.attn.proj.bias": "e0a207205ca4e489fd89215a79dc9c23555630be2f44b68551fdbc5957eb03fd",
+  "model.visual_tokenizer.visual_model.blocks.9.attn.proj.weight": "7badd89250518964688d21452edc7511c2a487bdee7c89ef3695efd990cdc8f6",
+  "model.visual_tokenizer.visual_model.blocks.9.attn.qkv.bias": "0369f90a6d478d60dab371b8c39657990766d93bb3c7235aa93b05f59efe3ada",
+  "model.visual_tokenizer.visual_model.blocks.9.mlp.down_proj.bias": "86400f15dcacee61ea33a0b039f56456f9960a90d313a28c7f30b1ae36c3e3b2",
+  "model.visual_tokenizer.visual_model.blocks.9.mlp.gate_proj.bias": "a209bab8085a9ad0351b9853307149d3075204b25bd4043b160845f94127561d",
+  "model.visual_tokenizer.visual_model.blocks.9.mlp.up_proj.bias": "8d23d2f0ec741a37a661e65fd248028ba3640229fa32e47d58d813e75002f563",
+  "model.visual_tokenizer.visual_model.blocks.9.norm1.weight": "5ac92cda5a888e7efe77f1ff7b72675db289055047be02949bfc0dd85794b8df",
+  "model.visual_tokenizer.visual_model.blocks.9.norm2.weight": "414826d87991062d8c053aac6dd9cfe0c7d45a6768d088d88bc7c15062b7fec4",
+  "model.visual_tokenizer.visual_model.patch_embed.proj.weight": "445267f7f85623e04fa7676956de78a26c64aabc259e4c533cd01b5a54589802"
+}

validate.py

@@ -0,0 +1,30 @@
+from pathlib import Path
+import json,sys,time,torch
+root=Path(__file__).resolve().parent;sys.path.insert(0,str(root.resolve()))
+from component import load_model,patchify_pixels,encode_pixels,decode_codes,code_features
+from native.visual import restore_spatial_structure_and_convert_to_images
+torch.set_num_threads(4);checks=[];t=time.time()
+def check(name,ok,**details):
+ assert ok,(name,details)
+ checks.append(dict(name=name,passed=True,**details));print(name,details,flush=True)
+m=load_model(root);check('strict_base_codec_load',True,tensors=len(m.state_dict()))
+books=m.model.visual_tokenizer.visual_bridge_model.quantizer.quantize.codebooks
+check('native_shared_codebook_object',all(book is books[0] for book in books))
+check('shared_codebook_storage',len({book.embed.data_ptr() for book in books})==1)
+with torch.inference_mode():
+ for h,w in [(112,112),(56,84)]:
+  pixels=torch.linspace(0,1,3*h*w).reshape(1,3,h,w);p,grid=patchify_pixels(pixels)
+  recovered=restore_spatial_structure_and_convert_to_images(p,grid,14)[0]
+  mean=pixels.new_tensor([.48145466,.4578275,.40821073])[:,None,None];std=pixels.new_tensor([.26862954,.26130258,.27577711])[:,None,None]
+  check(f'patch_layout_inverse_{h}x{w}',torch.equal(recovered,(pixels[0]-mean)/std),patches=list(p.shape))
+  ids,g=encode_pixels(m,pixels);check(f'rvq_ids_{h}x{w}',ids.shape==(h//28*w//28,8) and ids.min()>=0 and ids.max()<16384,shape=list(ids.shape),minimum=int(ids.min()),maximum=int(ids.max()))
+  features=code_features(m,ids);check(f'lookup_sum_{h}x{w}',features.shape==(h//28*w//28,3584) and torch.isfinite(features).all().item())
+  images=decode_codes(m,ids,g);y=images[0];check(f'base_reconstruction_{h}x{w}',y.shape==(3,h,w) and torch.isfinite(y).all().item(),shape=list(y.shape),mse=(y-((pixels[0]-mean)/std)).square().mean().item())
+ hidden=m.model.visual_tokenizer.visual_embedding_layer(torch.zeros(3,3072,dtype=torch.bfloat16));check('language_visual_embedding_interface',hidden.shape==(3,3072) and torch.isfinite(hidden).all().item())
+ pixels=torch.linspace(-1,1,3*64*64).reshape(1,3,64,64);z=m.image_refiner.vae.encode(pixels).latent_dist.mode();y=m.image_refiner.vae.decode(z).sample
+ check('refiner_vae_reconstruction',z.shape==(1,16,8,8) and y.shape==pixels.shape and torch.isfinite(y).all().item(),latent_shape=list(z.shape))
+ c=m.image_refiner.cond_proj(torch.randn(4,3584));check('refiner_condition_projection',c.shape[0]==4 and torch.isfinite(c).all().item(),shape=list(c.shape))
+aliases=json.loads((root/'alias_verification.json').read_text());check('source_shared_aliases_exact',len(aliases)==21)
+check('base_source_bytes_exact',json.loads((root/'extraction.json').read_text())['source_tensor_bytes_equal'] and json.loads((root/'decoder/extraction.json').read_text())['source_tensor_bytes_equal'])
+result=dict(strict_load=True,source_tensor_bytes_equal=True,checks=checks,smoke=dict(passed=True,device='cpu',visual_dtype='bfloat16',base_decoder_dtype='float32',quantizer_dtype='float32'),elapsed_seconds=time.time()-t,limitations=['This report covers the native dNaViT/RVQ/base decoder and refiner VAE/condition projection. Full flow refiner has a separate validation report.','No full LongCat text/audio backbone or VQA/T2I benchmark.','CPU compatibility uses a vendored Qwen2.5-VL window-attention implementation and preserves native decoder behavior.'])
+(root/'validation.json').write_text(json.dumps(result,indent=2)+'\n');print('LONGCAT BASE VALIDATED',result['elapsed_seconds'],flush=True)

validate_refiner.py

@@ -0,0 +1,38 @@
+from pathlib import Path
+import json,sys,time,torch,shutil
+root=Path(__file__).resolve().parent
+sys.path.insert(0,str(root.resolve()))
+from accelerate import init_empty_weights
+from safetensors.torch import load_file
+from native.configuration_longcat_next import LongcatNextConfig
+from native.image_refiner import ImageRefinerContainer
+from native.refiner_modules import RotaryPosEmbed
+torch.set_num_threads(4);checks=[];t=time.time()
+def check(name,ok,**details):
+ assert ok,(name,details)
+ checks.append(dict(name=name,passed=True,**details));print(name,details,flush=True)
+cfg=LongcatNextConfig(**json.loads((root/'source_config.json').read_text())).visual_config.visual_decoder_config
+with init_empty_weights():m=ImageRefinerContainer(cfg)
+state={k.removeprefix('image_refiner.'):v for k,v in load_file(root/'decoder/model.safetensors').items() if k.startswith('image_refiner.')}
+state.update({k.removeprefix('image_refiner.'):v for k,v in load_file(root/'refiner/model.safetensors').items()})
+m.load_state_dict(state,strict=True,assign=True);m.eval();del state
+check('strict_full_flow_refiner',True,tensors=len(m.state_dict()),transformer_tensors=len(m.base_transformer.state_dict()))
+dtype=next(m.base_transformer.parameters()).dtype;check('native_bfloat16_weights',dtype==torch.bfloat16)
+torch.manual_seed(1019);latent=torch.randn(1,16,4,4,dtype=dtype);reference=[[torch.randn(16,4,4,dtype=dtype)]];features=torch.randn(1,4,3584,dtype=dtype);mask=torch.ones(1,4,dtype=torch.bool);tc=cfg.transformer_config
+freqs=RotaryPosEmbed.get_freqs_cis(tc.axes_dim_rope,tc.axes_lens,theta=10000)
+with torch.inference_mode():
+ cond=m.cond_proj(features);check('native_refiner_conditioning',cond.shape==(1,4,2048) and torch.isfinite(cond).all().item())
+ result=m.base_transformer(latent,torch.tensor([.5],dtype=dtype),cond,freqs,mask,ref_image_hidden_states=reference,return_dict=False)
+ pred=result if torch.is_tensor(result) else (result[0] if isinstance(result,(tuple,list)) else result.sample)
+ if isinstance(pred,list):pred=pred[0].unsqueeze(0)
+ check('conditioned_flow_forward',pred.shape==latent.shape and torch.isfinite(pred).all().item(),shape=list(pred.shape),dtype=str(pred.dtype))
+ update=latent.float()-.1*pred.float();check('single_euler_update',torch.isfinite(update).all().item() and not torch.equal(update,latent.float()),mean_abs_update=(update-latent.float()).abs().mean().item())
+ altered=m.base_transformer(latent,torch.tensor([.5],dtype=dtype),cond*0,freqs,mask,ref_image_hidden_states=reference,return_dict=False)
+ other=altered if torch.is_tensor(altered) else (altered[0] if isinstance(altered,(tuple,list)) else altered.sample)
+ if isinstance(other,list):other=other[0].unsqueeze(0)
+ check('condition_affects_flow',not torch.equal(pred,other),max_abs_difference=(pred-other).abs().max().item())
+ex=json.loads((root/'refiner/extraction.json').read_text());check('refiner_source_tensor_bytes_exact',ex['source_tensor_bytes_equal'])
+report=dict(strict_load=True,source_tensor_bytes_equal=True,checks=checks,smoke=dict(passed=True,device='cpu',dtype=str(dtype),latent_shape=[1,16,4,4],reference_images=1),elapsed_seconds=time.time()-t,limitations=['Tiny-latent native flow forward and one Euler update, not full-resolution image refinement or perceptual-quality evaluation.','No LongCat language/audio backbone.'])
+(root/'refiner_validation.json').write_text(json.dumps(report,indent=2)+'\n')
+base=json.loads((root/'validation.json').read_text());base['checks'].extend([{'name':'refiner_'+c['name'],**{k:v for k,v in c.items() if k!='name'}} for c in checks]);base['full_flow_refiner_validation']='refiner_validation.json';base['limitations']=[x for x in base['limitations'] if 'separate validation report' not in x];base['limitations'].append('Full flow-refiner weights and tiny-latent conditioned forward are validated separately; no full-resolution quality benchmark.');(root/'validation.json').write_text(json.dumps(base,indent=2)+'\n')
+print('LONGCAT FULL REFINER VALIDATED',report['elapsed_seconds'],flush=True)

validation.json

@@ -0,0 +1,166 @@
+{
+  "strict_load": true,
+  "source_tensor_bytes_equal": true,
+  "checks": [
+    {
+      "name": "strict_base_codec_load",
+      "passed": true,
+      "tensors": 1229
+    },
+    {
+      "name": "native_shared_codebook_object",
+      "passed": true
+    },
+    {
+      "name": "shared_codebook_storage",
+      "passed": true
+    },
+    {
+      "name": "patch_layout_inverse_112x112",
+      "passed": true,
+      "patches": [
+        64,
+        1176
+      ]
+    },
+    {
+      "name": "rvq_ids_112x112",
+      "passed": true,
+      "shape": [
+        16,
+        8
+      ],
+      "minimum": 33,
+      "maximum": 16208
+    },
+    {
+      "name": "lookup_sum_112x112",
+      "passed": true
+    },
+    {
+      "name": "base_reconstruction_112x112",
+      "passed": true,
+      "shape": [
+        3,
+        112,
+        112
+      ],
+      "mse": 0.19658847153186798
+    },
+    {
+      "name": "patch_layout_inverse_56x84",
+      "passed": true,
+      "patches": [
+        24,
+        1176
+      ]
+    },
+    {
+      "name": "rvq_ids_56x84",
+      "passed": true,
+      "shape": [
+        6,
+        8
+      ],
+      "minimum": 33,
+      "maximum": 16305
+    },
+    {
+      "name": "lookup_sum_56x84",
+      "passed": true
+    },
+    {
+      "name": "base_reconstruction_56x84",
+      "passed": true,
+      "shape": [
+        3,
+        56,
+        84
+      ],
+      "mse": 0.4664306044578552
+    },
+    {
+      "name": "language_visual_embedding_interface",
+      "passed": true
+    },
+    {
+      "name": "refiner_vae_reconstruction",
+      "passed": true,
+      "latent_shape": [
+        1,
+        16,
+        8,
+        8
+      ]
+    },
+    {
+      "name": "refiner_condition_projection",
+      "passed": true,
+      "shape": [
+        4,
+        2048
+      ]
+    },
+    {
+      "name": "source_shared_aliases_exact",
+      "passed": true
+    },
+    {
+      "name": "base_source_bytes_exact",
+      "passed": true
+    },
+    {
+      "name": "refiner_strict_full_flow_refiner",
+      "passed": true,
+      "tensors": 828,
+      "transformer_tensors": 582
+    },
+    {
+      "name": "refiner_native_bfloat16_weights",
+      "passed": true
+    },
+    {
+      "name": "refiner_native_refiner_conditioning",
+      "passed": true
+    },
+    {
+      "name": "refiner_conditioned_flow_forward",
+      "passed": true,
+      "shape": [
+        1,
+        16,
+        4,
+        4
+      ],
+      "dtype": "torch.bfloat16"
+    },
+    {
+      "name": "refiner_single_euler_update",
+      "passed": true,
+      "mean_abs_update": 0.07963985204696655
+    },
+    {
+      "name": "refiner_condition_affects_flow",
+      "passed": true,
+      "max_abs_difference": 0.99609375
+    },
+    {
+      "name": "refiner_refiner_source_tensor_bytes_exact",
+      "passed": true
+    }
+  ],
+  "smoke": {
+    "passed": true,
+    "device": "cpu",
+    "visual_dtype": "bfloat16",
+    "base_decoder_dtype": "float32",
+    "quantizer_dtype": "float32"
+  },
+  "elapsed_seconds": 11.928704977035522,
+  "limitations": [
+    "No full LongCat text/audio backbone or VQA/T2I benchmark.",
+    "CPU compatibility uses a vendored Qwen2.5-VL window-attention implementation and preserves native decoder behavior.",
+    "Full flow-refiner weights and tiny-latent conditioned forward are validated separately; no full-resolution quality benchmark."
+  ],
+  "full_flow_refiner_validation": "refiner_validation.json"
+}