Publish model weights, paper figures, and complete model card

.gitattributes

@@ -33,3 +33,13 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/decoding.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/demo.mp4 filter=lfs diff=lfs merge=lfs -text
+assets/fig1-teaser.png filter=lfs diff=lfs merge=lfs -text
+assets/fig2-architecture.png filter=lfs diff=lfs merge=lfs -text
+assets/fig4-attention-mask.png filter=lfs diff=lfs merge=lfs -text
+assets/fig6-self-speculative-decoding.png filter=lfs diff=lfs merge=lfs -text
+assets/fig7-grounding-performance.png filter=lfs diff=lfs merge=lfs -text
+assets/logo.png filter=lfs diff=lfs merge=lfs -text
+assets/demo-poster.jpg filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,52 @@
+Kimi K3 License
+
+Copyright (c) 2026 Moonshot AI
+
+Permission is hereby granted, free of charge, to any person (the "Licensee")
+obtaining a copy of this software — including the model weights, parameters,
+configuration files, inference and training code, and associated documentation
+(collectively, the "Software") — to deal in the Software without restriction.
+This includes, without limitation, the rights to use, copy, modify, merge,
+publish, distribute, sublicense, and/or sell copies of the Software; to run,
+deploy, fine-tune, or otherwise modify the Software and create derivative works
+from it; and to permit persons to whom the Software is furnished to do so, in
+each case subject to the following conditions:
+
+1. The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software. Licensee's use of the
+Software must comply with applicable laws and regulations.
+
+2. "Model as a Service" means giving a third party access to language model
+inference or fine-tuning (e.g., via API) in a manner that allows such third
+party to exercise meaningful control over the inputs, parameters, or training
+data. This does not include (a) end-user products with model capabilities solely
+embedded within specific features or harnesses, or (b) mere relaying of requests
+to models hosted by others.
+
+If the Licensee or any of its affiliates operates a Model as a Service business,
+and the aggregate revenue of the Licensee and its affiliates exceeds 20 million
+US dollars (or the equivalent in other currencies) in total over any consecutive
+12 months, the Licensee must enter into a separate agreement with Moonshot AI
+before using the Software or its derivative works for any commercial purpose. 
+
+3. If the Software (or any derivative works thereof) is used for any of the
+Licensee's commercial products or services that have more than 100 million
+monthly active users, or more than 20 million US dollars (or equivalent in other
+currencies) in monthly revenue, "Kimi K3" must be prominently displayed on the
+user interface of such product or service.
+
+4. The requirements set forth in Sections 2 and 3 do not apply to: (a) internal
+use of the Software, defined as any use that does not make the Software, its
+outputs, or its underlying capabilities available to third parties; or (b) any
+use of the Software accessed through Moonshot AI's official products or
+certified inference partners.
+
+5. THE SOFTWARE AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN “AS IS”
+BASIS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
+LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE
+AND NONINFRINGEMENT. IN NO EVENT SHALL MOONSHOT AI OR ITS AFFILIATES OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+For any questions regarding this license, please contact <license@moonshot.ai>.

README.md

@@ -1,3 +1,381 @@
 ---
-license: apache-2.0
+license: other
+license_name: kimi-k3
+license_link: https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/LICENSE
+language:
+  - en
+  - zh
+library_name: transformers
+pipeline_tag: image-text-to-text
+tags:
+  - visual-grounding
+  - object-detection
+  - referring-expression-comprehension
+  - pointing
+  - ocr
+  - document-layout
+  - custom-code
+  - diffusion-language-model
+inference: false
 ---
+
+<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/logo.png" width="150" alt="GroundAnything logo" /></p>
+
+# GroundAnything: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed
+
+**Model family:** [GroundAnything — DLM / parallel decoding](https://huggingface.co/GroundingPI/GroundAnything) · [GroundAnything-VLM — autoregressive](https://huggingface.co/GroundingPI/GroundAnything-VLM).
+
+**This repository contains the GroundAnything DLM checkpoint.** Use entropy-guided decoding for the main benchmark setting, or select optional self-speculative decoding.
+
+<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/fig1-teaser.png" width="100%" alt="GroundAnything Figure 1: broad visual grounding and parallel visual evidence extraction" /></p>
+
+## 🔗 Quick Links
+
+- 🚀 **Online Demo:** Coming soon — XXX.
+- 💻 **GitHub Code:** Coming soon — XXX.
+- 📄 **Paper:** [arXiv:2609.39600](https://arxiv.org/abs/2609.39600).
+- 🧪 **Evaluation data:** Coming soon — XXX.
+
+# Model Overview
+
+### Description:
+
+Autoregressive (AR) grounding models serialize spatial predictions, introducing sequential latency and imposing a causal order on output tokens. We view grounding as visual evidence extraction: objects, locations, and spatial relations are jointly constrained by the image and query, yet their dependencies do not imply an intrinsic left-to-right generation order. This distinction makes bidirectional diffusion a natural fit, allowing spatial hypotheses to emerge in parallel and be jointly refined through iterative denoising.
+
+We introduce **GroundAnything**, a 4B-parameter grounding foundation model that reconciles fast parallel decoding with precise localization through blockwise denoising. Across 30 grounding benchmarks, the autoregressive variant, **GroundAnything-VLM**, establishes a new overall state of the art among similarly sized models at **72.42%**, remaining competitive with GPT-6 Astra (**71.35%**). With **entropy-guided decoding**, GroundAnything also surpasses the prior state of the art at this scale, averaging **61.75%** versus **53.32%** for the fast MTP-based LocateAnything model.
+
+An optional **self-speculative mode** achieves a **4.51× speedup** over the AR counterpart with a **0.74 percentage-point drop in COCO F1mIoU** at the paper's reported operating point. Infrastructure experiments show that progressive inference optimizations translate parallel decoding into practical speedups. These support efficient visual grounding in latency-sensitive real-world systems.
+
+### Demo Videos
+
+<video controls playsinline preload="none" width="100%" poster="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/demo-poster.jpg" src="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/demo.mp4"></video>
+
+**Parallel decoding in action**
+
+<video controls playsinline preload="none" width="100%" src="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/decoding.mp4"></video>
+
+### License/Terms of Use:
+
+This checkpoint is released under the **Kimi K3 License**; see the repository's [LICENSE](LICENSE) and the [upstream license](https://huggingface.co/moonshotai/Kimi-K3/blob/f831ab66814297da540d832a5235f8e904f29d06/LICENSE). The model incorporates Kimi K3-derived vision components and implementation code. The license includes additional conditions for certain commercial uses. Third-party components retain their respective licenses and copyright notices.
+
+### Deployment Geography:
+
+Global.
+
+### Use Case:
+
+- Open-vocabulary object localization and dense-scene grounding.
+- Referring-expression comprehension and visual-prompt-based localization.
+- Point-based localization, spatial reasoning, and GUI element grounding.
+- OCR with text localization and document layout understanding.
+- Perception research for robotics, embodied agents, and autonomous systems.
+
+### Release Date:
+
+- **Paper [09/30/2026]:** [GroundAnything](https://arxiv.org/abs/2609.39600).
+
+## References(s):
+
+- [GroundAnything paper and supplementary material](https://arxiv.org/abs/2609.39600).
+- [GroundingPI: grounding with visual primitives](https://arxiv.org/abs/2609.39601).
+
+<details>
+<summary>Citation</summary>
+
+```bibtex
+@misc{yu2026groundanything,
+  title = {GroundAnything: Reconciling Parallel Decoding with Precise Visual Grounding at Flash Speed},
+  author = {Qize Yu and Lianrui Fan and Bowen Ping and Xini Ding and Zetian Song and Junbo Niu and Kaixuan Wang and Tianxing Chen and Yue Chen and Minghua He and Yuran Wang and Jie Huang and Haojun Zhang and Min Chen and Hao Li and Wenxuan Song and Ruihai Wu and Xianming Liu and Shilong Liu and Shuchang Zhou and Ping Luo and Shiyu Huang},
+  year = {2026},
+  eprint = {2609.39600},
+  archivePrefix = {arXiv},
+  primaryClass = {cs.CV},
+  url = {https://arxiv.org/abs/2609.39600},
+}
+```
+
+</details>
+
+## Model Architecture:
+
+**Architecture type:** a shared vision-language backbone supporting an autoregressive checkpoint and a blockwise diffusion checkpoint.
+
+- **Vision encoder:** MoonViT-V2 / Kimi-K3 vision backbone.
+- **Language backbone:** Qwen3-4B-Instruct-2507.
+- **Multimodal projector:** 2 × 2 spatial aggregation and a two-layer MLP.
+- **Spatial vocabulary:** 1,000 coordinate tokens shared with semantic labels and protocol markers.
+- **DLM conversion:** the shared decoder and vocabulary head support both causal prediction and bidirectional response-block denoising. A mask token is added for diffusion generation.
+
+<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/fig2-architecture.png" width="100%" alt="GroundAnything Figure 2: vision-language architecture and autoregressive-to-diffusion conversion" /></p>
+
+*Figure 2. Shared model architecture and the conversion to parallel grounding.*
+
+<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/fig4-attention-mask.png" width="640" alt="GroundAnything Figure 4: clean-stream causal and noisy-response block attention" /></p>
+
+*Figure 4. The attention mask used during diffusion conversion. The clean stream uses causal attention. A noisy response block attends bidirectionally within its block and reads the clean conditioning context and strictly preceding clean response blocks. Conversion uses B=32; the illustration uses two-token blocks.*
+
+## Input(s):
+
+**Input types:** image and text.
+
+- **Image:** one RGB image per request. The Python client accepts JPEG, PNG, and WebP files; the supplied processor handles image preparation.
+- **Text:** a natural-language instruction, category list, referring expression, OCR/layout query, or a prompt containing example boxes.
+- **Image encoding for the HTTP API:** an `image_url` content part containing a base64 data URI, alongside a `text` content part.
+
+Use the checkpoint's own tokenizer, processor, and chat template. Multiple categories are separated by `</c>`. Reference boxes use the same 0–999 spatial-token vocabulary as outputs.
+
+## Output(s):
+
+**Output type:** text containing semantic labels and quantized spatial coordinates.
+
+All three released models use the **GAM protocol**: integer coordinate tokens `<0>` through `<999>`, object-reference delimiters, and box delimiters. A bounding box contains `(x1, y1, x2, y2)`; a point contains `(x, y)`. Adjacent coordinate tokens have no intervening spaces. Multiple instances of the same label are comma-separated inside one box wrapper. A missing target is represented by `None`.
+
+Illustrative syntax:
+
+```text
+<|object_ref_start|>car<|object_ref_end|><|box_start|><100><200><500><650><|box_end|>
+<|object_ref_start|>car center<|object_ref_end|><|box_start|><300><425><|box_end|>
+<|object_ref_start|>absent object<|object_ref_end|><|box_start|>None<|box_end|>
+```
+
+The client returns parsed predictions together with `raw_output`, `finish_reason`, `usage`, `parse_error`, and `valid`. It maps coordinates back to image pixels for visualization. A truncated or malformed response is marked invalid. For custom API clients, preserve spatial tokens with `skip_special_tokens=false` and avoid inserting spaces between them.
+
+## Software Integration:
+
+**Runtime engines:** the repository's custom **SGLang** integration for the DLM and VLM services, plus a native Transformers reference route for the DLM.
+
+**Default serving environment:** Linux and Python 3.12 with the source package's serving profile. Use `python3 run.py setup serve` to install the bundled custom engine and its pinned dependencies; installing upstream SGLang alone does not provide the same model and decoding integration.
+
+The serving profile pins Torch **2.9.1**, Transformers **5.5.4**, Triton **3.5.1**, sgl-kernel **0.3.20**, and FlashInfer **0.5.3**. Serving and evaluation use separate environments.
+
+**Tested hardware:** NVIDIA **B300, B200, H200, H800**, and **PPU**. The provided SGLang serving installer is a CUDA/GPU recipe; PPU requires its matching platform runtime and is not selected by this GPU installer.
+
+The default DLM service uses **BF16, Triton attention, eager execution, one GPU, one active request, and two queued requests**. Client concurrency queues requests; it does not imply a multi-request model batch. CUDA Graph and selective FP8 are discussed below as separate infrastructure experiments.
+
+## Model Version(s):
+
+| Checkpoint | Generation | Evaluation mode |
+|:---|:---|:---|
+| [GroundAnything](https://huggingface.co/GroundingPI/GroundAnything) | Entropy-guided blockwise diffusion; optional self-speculation | **GAM** |
+| [GroundAnything-VLM](https://huggingface.co/GroundingPI/GroundAnything-VLM) | Autoregressive generation | **GAM** |
+
+The main GroundAnything benchmark results use **entropy-guided diffusion without autoregressive verification**. GroundAnything-VLM uses autoregressive decoding. The self-speculative speed–quality operating point is a separate experiment and should not be substituted for the main diffusion benchmark setting.
+
+## Testing and Evaluation Datasets:
+
+### Data Modality:
+
+Image and text, with task-specific box, point, text-region, or interaction annotations.
+
+## Evaluation Dataset:
+
+**Evaluation data: XXX — download URL coming soon.**
+
+The shared evaluation toolkit provides **42 task recipes across eight task families**: Grounding, Referring, Dense, OCR, Layout, GUI, Pointing, and VisualPrompt. These are executable task/split recipes, not a count of distinct datasets. Dataset paths are registered in `configs/datasets.yaml`; task IDs are listed in `configs/eval/tasks.json`.
+
+### Evaluation Modes
+
+The evaluator accepts seven mode identifiers. A mode selects the **prompt and output parser** independently of the inference engine and decoding algorithm.
+
+| Mode | Supported evaluation interface | Output / coordinates |
+|:---|:---|:---|
+| **`GAM`** | **GroundingPI, GroundAnything, and GroundAnything-VLM** | Native spatial tokens, **0–999** |
+| `DLM` | Legacy compatible diffusion-checkpoint alias | Same GAM spatial-token protocol |
+| `RLV2` | Compatible RL-checkpoint alias | Same GAM spatial-token protocol |
+| `VLM` | Generic VLM baselines | Explicit pixel, 0–1000, or 0–1 coordinate mode |
+| `REXOMNI` | Rex-Omni adapter | Model-specific spatial-token parser |
+| `LOCATEANYTHING` | LocateAnything adapter | Its own output parser and generation-mode setting |
+| `GROUNDINGDINO` | External compatible GroundingDINO bridge | JSON coordinate responses |
+
+**Use `mode: GAM` for all three of our checkpoints.** The `-VLM` suffix identifies an autoregressive checkpoint; it does not select the evaluator's generic `VLM` mode. The DLM's entropy-guided or self-speculative algorithm is selected separately through `decoder`.
+
+External baseline weights and model services are separate from the evaluation toolkit. Metrics remain task-specific: box localization quality, point accuracy, OCR text-and-region matching, GUI grounding accuracy, and counting error. Full protocols and benchmark tables are provided in the paper and supplementary material.
+
+## Quantitative Evaluation Benchmarks
+
+<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/fig7-grounding-performance.png" width="100%" alt="GroundAnything Figure 7: GroundAnything and GroundAnything-VLM benchmark overview" /></p>
+
+*Figure 7. Paper-reported capability overview. Detailed task-level scores, baseline settings, and evaluation protocols are provided in the paper and supplementary material.*
+
+## Inference:
+
+### Installation
+
+Run the commands from the **GroundAnything source repository root** after obtaining the code package, using **Linux x86_64 and Python 3.12**. The code link above is reserved for the public release. Use the serving profile in the supplied source bundle so that the custom model adapter, decoding implementation, and dependencies remain aligned.
+
+```bash
+python3 -m pip install -r requirements.txt huggingface_hub
+python3 run.py setup serve
+```
+
+The installer verifies and extracts its bundled frameworks, creates `.venv-serve`, and records resolved packages. It prepares the custom SGLang implementation and applies the serving profile's dependency order. The profile includes the required cuDNN compatibility selection; its dependency-check report records the known Torch/cuDNN metadata exception.
+
+**Choose the checkpoint-specific recipe below.** GroundAnything-VLM users need only the VLM download and launch commands; the DLM commands require the separate GroundAnything weights.
+
+### GroundAnything: Entropy-guided Service
+
+Download the **complete DLM model bundle**, including its custom code and tokenizer, then launch:
+
+```bash
+hf download GroundingPI/GroundAnything --local-dir weights/dlm_bundle
+python3 run.py serve --decoder denoise
+```
+
+The published model package is already a DLM bundle, so `prepare-model` is unnecessary for this download. The endpoint is `http://127.0.0.1:8101/v1`, with model ID `groundinganything`.
+
+### GroundAnything: Self-speculative Service
+
+Stop the existing DLM service before switching its decoder:
+
+```bash
+python3 run.py serve --decoder speculative
+```
+
+This reuses the same DLM weights and endpoint, with diffusion proposals and greedy causal verification.
+
+### GroundAnything-VLM: Autoregressive Service
+
+Use the separate autoregressive checkpoint and its service recipe:
+
+```bash
+hf download GroundingPI/GroundAnything-VLM --local-dir weights/vlm
+python3 run.py serve --config configs/release/vlm_sglang.yaml
+```
+
+This service uses `http://127.0.0.1:8102/v1`, with model ID `groundinganything-vlm`. The two repositories share the spatial interface but have different loading and generation paths.
+
+### Worker (recommended)
+
+Start the appropriate service once, then reuse the client:
+
+```python
+from grounding_anything import GroundingAnything, visualize
+
+client = GroundingAnything(
+    base_url="http://127.0.0.1:8101/v1",
+    model="groundinganything",
+)
+result = client.predict("example.jpg", "the red car", task="bbox")
+print(result.to_dict())
+if result.valid:
+    visualize("example.jpg", result).save("prediction.png")
+
+point_result = client.predict(
+    "example.jpg", "the center of the red car", task="point"
+)
+```
+
+The HTTP client retains no model weights. A generic interactive request has no benchmark task identity; it does not automatically receive the benchmark's task-specific sampling and stopping policy. Use the evaluator below to reproduce the reported protocol.
+
+### Supported Tasks & Prompt Templates
+
+| Task | Prompt example |
+|:---|:---|
+| Category / dense grounding | `Locate all the instances that match the following categories: car</c>person.` |
+| Referring boxes | `Locate the target referred to by the following description: the red car.` |
+| Category points | `Point to: car</c>person.` |
+| Referring points | `Point to the target referred to by the following description: the red car.` |
+| OCR | `OCR task detect all the text in box format.` |
+| Layout | `Detect all document layout elements that match the following categories: title</c>text.` |
+| GUI | `Point to the UI element to click for the following instruction: open the settings menu.` |
+| Visual prompting | Provide reference boxes in the native spatial-token format, then request similar objects. |
+
+```text
+Given reference boxes <|box_start|><100><200><500><650><|box_end|> indicating one or more objects, find all similar objects in the image and output their bounding boxes.
+```
+
+The convenience client's `predict()` method wraps referring-box and referring-point prompts. Use an OpenAI-compatible request to `/chat/completions` for the other task templates; keep the image and prompt in the same user message.
+
+### Generation Modes
+
+#### Entropy-guided decoding
+
+Image/query prefill creates a causal prefix cache and a known anchor. The release recipe uses **block size 32, sub-block size 4, and entropy threshold 0.8**. A physical block contains the known anchor and 31 masked positions; sub-blocks are completed from left to right while the forward pass evaluates the physical block.
+
+For each still-masked position in the active sub-block, the decoder measures entropy from the unmodified token distribution over the generatable vocabulary, excluding the mask token. Positions at or below the threshold are committed together. If no position qualifies, the lowest-entropy position is committed so decoding makes progress. Committed tokens remain fixed.
+
+After a block is complete, a causal forward reconstructs its authoritative KV cache and supplies the next anchor. This cache-building pass **does not verify or reject the generated block**. A block requiring D denoising passes therefore uses D + 1 model forwards, excluding the initial prefill.
+
+#### Self-speculative decoding
+
+The model uses its own shared weights to draft tokens with bidirectional attention and verify them with causal attention. Verification accepts the **longest consecutive matching prefix**, stops at the first mismatch, applies the causal correction, and discards the rejected suffix cache states. The shipped speculative route uses greedy verification; it is not a general stochastic speculative sampler.
+
+<p align="center"><img src="https://huggingface.co/GroundingPI/GroundAnything/resolve/0f8e30894c3ca86378d01ae51ec69c217c78151b/assets/fig6-self-speculative-decoding.png" width="100%" alt="GroundAnything Figure 6: linear and quadratic self-speculative schedules with shared model weights" /></p>
+
+*Figure 6. The paper studies two schedules: linear drafting and verification use two model forwards and 2B query tokens per round; quadratic fusion uses one forward after initialization with B(B + 1) query tokens. These counts describe queries and model calls, not total Transformer FLOPs.*
+
+The documented `--decoder speculative` service is the linear shared-weight route. Exact greedy verification is relative to the converted model's causal branch; it does not imply identical outputs to the separately trained GroundAnything-VLM checkpoint.
+
+### Benchmark Sampling Policy
+
+Entropy-guided evaluation applies the five task profiles from `infer/decode/configs/task_profiles.json`:
+
+| Task profile | Recipes | Temperature | Top-p | Output-token budget |
+|:---|---:|---:|---:|---:|
+| Strict single target | 14 | 0 | 1 | 512 |
+| Medium, non-OCR | 12 | 0.1 | 0.95 | 4,096 |
+| Dense, non-OCR | 8 | 0.1 | 0.95 | 8,192 |
+| Medium OCR | 4 | 0.3 | 0.95 | 4,096 |
+| Dense OCR | 4 | 0 | 1 | 4,096 |
+
+The evaluator sends outer sampling parameters and the nested `custom_params.gam_dlm_decode` contract, adds the strict-single-target policy when required, and checks the server's actual decoder before sending requests. Unknown task profiles and mismatched decoders fail early. The entropy-guided benchmark policy rejects a global `max_tokens` override because it would replace the task budgets.
+
+Self-speculative evaluation uses greedy sampling (`temperature=0`, `top_p=1`, repetition penalty 1). GroundAnything-VLM retains its own autoregressive task recipes. All of these routes use **GAM** prompts and spatial-token parsing.
+
+### Run Evaluation
+
+```bash
+python3 run.py setup eval
+
+# Match the command to the service that is already running.
+python3 run.py eval --decoder denoise
+python3 run.py eval --decoder speculative
+python3 run.py eval --config configs/eval/vlm.yaml
+```
+
+Choose one evaluation command for the intended checkpoint/decoder:
+
+| Checkpoint / decoder | Recipe | Mode | Endpoint model ID |
+|:---|:---|:---|:---|
+| GroundAnything / entropy-guided | `configs/eval/dlm.yaml` | **GAM** | `groundinganything` |
+| GroundAnything / self-speculative | `configs/eval/dlm_speculative.yaml` | **GAM** | `groundinganything` |
+| GroundAnything-VLM / autoregressive | `configs/eval/vlm.yaml` | **GAM** | `groundinganything-vlm` |
+
+Use `service_contract: openai` and the corresponding port (8101 for DLM, 8102 for VLM). For a preflight that checks configured inputs without sending inference requests:
+
+```bash
+.venv-eval/bin/python scripts/evaluate.py configs/eval/dlm.yaml --dry-run
+```
+
+Configure the running service, `data_root`, dataset registry, task list, and a fresh `run_id` before launching. The default `limit: 8` is a smoke test; set **`limit: null`** for full evaluation. Evaluation connects to an existing service and does not start or change the decoder.
+
+Each run saves `run.json` (configuration and provenance), `responses.jsonl` (raw responses, finish reasons, and token usage), task logs, and `summary.json` (metrics and completion state). Record the checkpoint revision and task configuration when comparing results.
+
+## Inference Infrastructure
+
+### SGLang Execution
+
+The custom SGLang integration coordinates model loading, request scheduling, attention kernels, and KV-cache ownership for blockwise generation. Denoising uses bidirectional attention inside the active block; completed history is retained as causal KV. Self-speculation additionally verifies proposals and removes rejected suffix states. These cache semantics must be preserved when optimizing execution.
+
+The supplied recipe selects **BF16 + Triton attention + eager execution**. DLM recipes record the engine source, decoder, effective settings, profile digest, and package versions in `outputs/sglang/<decoder>/engine_runtime.json`; DLM evaluation checks decoder alignment before starting workers. The VLM service records its causal model/runtime configuration separately in `outputs/sglang/vlm/engine_runtime.json`. For higher service concurrency, use independent replicas with distinct devices, ports, and output directories; the default queue is not continuous multi-request batching.
+
+### CUDA Graph and Selective FP8
+
+The paper evaluates a progressive infrastructure sequence:
+
+| Layer | Purpose | Status in the supplied default recipe |
+|:---|:---|:---|
+| Native PyTorch eager | Reference model execution | Reference implementation |
+| SGLang eager | Integrated scheduling, attention, and cache execution | **Default serving path** |
+| CUDA Graph replay | Replay compatible captured GPU work to reduce repeated launch overhead | Evaluated in the paper; **disabled by the default launcher** |
+| Selective FP8 | Reduce arithmetic cost in eligible language-model linear operations while retaining other components in BF16 | Evaluated in the paper; **default serving remains BF16** |
+
+CUDA Graph changes how compatible GPU work is submitted; it does not define a new token-commitment or verification rule. Captured shapes and state updates must remain compatible with the active decoding path. Selective FP8 can change logits and subsequent decoding decisions, so it is a distinct numerical configuration.
+
+The optional graph implementation captures fixed-shape block work. Its FlashInfer path uses persistent attention masks and device-resident buffer updates for bidirectional drafting and causal verification. The Triton verifier path separates draft/verification metadata and input buffers while sharing parameters and the real KV pool. Its supported capture case is B32 with one request and no tensor/pipeline/data parallel expansion; prefill and unsupported shapes use eager execution. Optional shadow checks compare cache states, logits, and token choices against eager execution. These implementation paths are not exposed as a public `run.py serve --cuda-graph` switch in this release.
+
+The paper's cumulative infrastructure speedups compare execution implementations **within the same decoding mode**. They are separate from the headline comparison of self-speculation against the autoregressive checkpoint. The default release commands above do not enable Graph replay or FP8, and no unsupported activation flags are implied.
+
+## Ethical Considerations:
+
+Grounding predictions can miss small or occluded objects, repeat instances, or produce inaccurate text and coordinates. Validate localization quality on the intended task and inspect incomplete responses. GUI points describe image locations; the model does not execute interface actions. Perception outputs require task-specific validation before integration into physical systems.

added_tokens.json

@@ -0,0 +1,1030 @@
+{
+  "</c>": 152669,
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<0>": 151669,
+  "<100>": 151769,
+  "<101>": 151770,
+  "<102>": 151771,
+  "<103>": 151772,
+  "<104>": 151773,
+  "<105>": 151774,
+  "<106>": 151775,
+  "<107>": 151776,
+  "<108>": 151777,
+  "<109>": 151778,
+  "<10>": 151679,
+  "<110>": 151779,
+  "<111>": 151780,
+  "<112>": 151781,
+  "<113>": 151782,
+  "<114>": 151783,
+  "<115>": 151784,
+  "<116>": 151785,
+  "<117>": 151786,
+  "<118>": 151787,
+  "<119>": 151788,
+  "<11>": 151680,
+  "<120>": 151789,
+  "<121>": 151790,
+  "<122>": 151791,
+  "<123>": 151792,
+  "<124>": 151793,
+  "<125>": 151794,
+  "<126>": 151795,
+  "<127>": 151796,
+  "<128>": 151797,
+  "<129>": 151798,
+  "<12>": 151681,
+  "<130>": 151799,
+  "<131>": 151800,
+  "<132>": 151801,
+  "<133>": 151802,
+  "<134>": 151803,
+  "<135>": 151804,
+  "<136>": 151805,
+  "<137>": 151806,
+  "<138>": 151807,
+  "<139>": 151808,
+  "<13>": 151682,
+  "<140>": 151809,
+  "<141>": 151810,
+  "<142>": 151811,
+  "<143>": 151812,
+  "<144>": 151813,
+  "<145>": 151814,
+  "<146>": 151815,
+  "<147>": 151816,
+  "<148>": 151817,
+  "<149>": 151818,
+  "<14>": 151683,
+  "<150>": 151819,
+  "<151>": 151820,
+  "<152>": 151821,
+  "<153>": 151822,
+  "<154>": 151823,
+  "<155>": 151824,
+  "<156>": 151825,
+  "<157>": 151826,
+  "<158>": 151827,
+  "<159>": 151828,
+  "<15>": 151684,
+  "<160>": 151829,
+  "<161>": 151830,
+  "<162>": 151831,
+  "<163>": 151832,
+  "<164>": 151833,
+  "<165>": 151834,
+  "<166>": 151835,
+  "<167>": 151836,
+  "<168>": 151837,
+  "<169>": 151838,
+  "<16>": 151685,
+  "<170>": 151839,
+  "<171>": 151840,
+  "<172>": 151841,
+  "<173>": 151842,
+  "<174>": 151843,
+  "<175>": 151844,
+  "<176>": 151845,
+  "<177>": 151846,
+  "<178>": 151847,
+  "<179>": 151848,
+  "<17>": 151686,
+  "<180>": 151849,
+  "<181>": 151850,
+  "<182>": 151851,
+  "<183>": 151852,
+  "<184>": 151853,
+  "<185>": 151854,
+  "<186>": 151855,
+  "<187>": 151856,
+  "<188>": 151857,
+  "<189>": 151858,
+  "<18>": 151687,
+  "<190>": 151859,
+  "<191>": 151860,
+  "<192>": 151861,
+  "<193>": 151862,
+  "<194>": 151863,
+  "<195>": 151864,
+  "<196>": 151865,
+  "<197>": 151866,
+  "<198>": 151867,
+  "<199>": 151868,
+  "<19>": 151688,
+  "<1>": 151670,
+  "<200>": 151869,
+  "<201>": 151870,
+  "<202>": 151871,
+  "<203>": 151872,
+  "<204>": 151873,
+  "<205>": 151874,
+  "<206>": 151875,
+  "<207>": 151876,
+  "<208>": 151877,
+  "<209>": 151878,
+  "<20>": 151689,
+  "<210>": 151879,
+  "<211>": 151880,
+  "<212>": 151881,
+  "<213>": 151882,
+  "<214>": 151883,
+  "<215>": 151884,
+  "<216>": 151885,
+  "<217>": 151886,
+  "<218>": 151887,
+  "<219>": 151888,
+  "<21>": 151690,
+  "<220>": 151889,
+  "<221>": 151890,
+  "<222>": 151891,
+  "<223>": 151892,
+  "<224>": 151893,
+  "<225>": 151894,
+  "<226>": 151895,
+  "<227>": 151896,
+  "<228>": 151897,
+  "<229>": 151898,
+  "<22>": 151691,
+  "<230>": 151899,
+  "<231>": 151900,
+  "<232>": 151901,
+  "<233>": 151902,
+  "<234>": 151903,
+  "<235>": 151904,
+  "<236>": 151905,
+  "<237>": 151906,
+  "<238>": 151907,
+  "<239>": 151908,
+  "<23>": 151692,
+  "<240>": 151909,
+  "<241>": 151910,
+  "<242>": 151911,
+  "<243>": 151912,
+  "<244>": 151913,
+  "<245>": 151914,
+  "<246>": 151915,
+  "<247>": 151916,
+  "<248>": 151917,
+  "<249>": 151918,
+  "<24>": 151693,
+  "<250>": 151919,
+  "<251>": 151920,
+  "<252>": 151921,
+  "<253>": 151922,
+  "<254>": 151923,
+  "<255>": 151924,
+  "<256>": 151925,
+  "<257>": 151926,
+  "<258>": 151927,
+  "<259>": 151928,
+  "<25>": 151694,
+  "<260>": 151929,
+  "<261>": 151930,
+  "<262>": 151931,
+  "<263>": 151932,
+  "<264>": 151933,
+  "<265>": 151934,
+  "<266>": 151935,
+  "<267>": 151936,
+  "<268>": 151937,
+  "<269>": 151938,
+  "<26>": 151695,
+  "<270>": 151939,
+  "<271>": 151940,
+  "<272>": 151941,
+  "<273>": 151942,
+  "<274>": 151943,
+  "<275>": 151944,
+  "<276>": 151945,
+  "<277>": 151946,
+  "<278>": 151947,
+  "<279>": 151948,
+  "<27>": 151696,
+  "<280>": 151949,
+  "<281>": 151950,
+  "<282>": 151951,
+  "<283>": 151952,
+  "<284>": 151953,
+  "<285>": 151954,
+  "<286>": 151955,
+  "<287>": 151956,
+  "<288>": 151957,
+  "<289>": 151958,
+  "<28>": 151697,
+  "<290>": 151959,
+  "<291>": 151960,
+  "<292>": 151961,
+  "<293>": 151962,
+  "<294>": 151963,
+  "<295>": 151964,
+  "<296>": 151965,
+  "<297>": 151966,
+  "<298>": 151967,
+  "<299>": 151968,
+  "<29>": 151698,
+  "<2>": 151671,
+  "<300>": 151969,
+  "<301>": 151970,
+  "<302>": 151971,
+  "<303>": 151972,
+  "<304>": 151973,
+  "<305>": 151974,
+  "<306>": 151975,
+  "<307>": 151976,
+  "<308>": 151977,
+  "<309>": 151978,
+  "<30>": 151699,
+  "<310>": 151979,
+  "<311>": 151980,
+  "<312>": 151981,
+  "<313>": 151982,
+  "<314>": 151983,
+  "<315>": 151984,
+  "<316>": 151985,
+  "<317>": 151986,
+  "<318>": 151987,
+  "<319>": 151988,
+  "<31>": 151700,
+  "<320>": 151989,
+  "<321>": 151990,
+  "<322>": 151991,
+  "<323>": 151992,
+  "<324>": 151993,
+  "<325>": 151994,
+  "<326>": 151995,
+  "<327>": 151996,
+  "<328>": 151997,
+  "<329>": 151998,
+  "<32>": 151701,
+  "<330>": 151999,
+  "<331>": 152000,
+  "<332>": 152001,
+  "<333>": 152002,
+  "<334>": 152003,
+  "<335>": 152004,
+  "<336>": 152005,
+  "<337>": 152006,
+  "<338>": 152007,
+  "<339>": 152008,
+  "<33>": 151702,
+  "<340>": 152009,
+  "<341>": 152010,
+  "<342>": 152011,
+  "<343>": 152012,
+  "<344>": 152013,
+  "<345>": 152014,
+  "<346>": 152015,
+  "<347>": 152016,
+  "<348>": 152017,
+  "<349>": 152018,
+  "<34>": 151703,
+  "<350>": 152019,
+  "<351>": 152020,
+  "<352>": 152021,
+  "<353>": 152022,
+  "<354>": 152023,
+  "<355>": 152024,
+  "<356>": 152025,
+  "<357>": 152026,
+  "<358>": 152027,
+  "<359>": 152028,
+  "<35>": 151704,
+  "<360>": 152029,
+  "<361>": 152030,
+  "<362>": 152031,
+  "<363>": 152032,
+  "<364>": 152033,
+  "<365>": 152034,
+  "<366>": 152035,
+  "<367>": 152036,
+  "<368>": 152037,
+  "<369>": 152038,
+  "<36>": 151705,
+  "<370>": 152039,
+  "<371>": 152040,
+  "<372>": 152041,
+  "<373>": 152042,
+  "<374>": 152043,
+  "<375>": 152044,
+  "<376>": 152045,
+  "<377>": 152046,
+  "<378>": 152047,
+  "<379>": 152048,
+  "<37>": 151706,
+  "<380>": 152049,
+  "<381>": 152050,
+  "<382>": 152051,
+  "<383>": 152052,
+  "<384>": 152053,
+  "<385>": 152054,
+  "<386>": 152055,
+  "<387>": 152056,
+  "<388>": 152057,
+  "<389>": 152058,
+  "<38>": 151707,
+  "<390>": 152059,
+  "<391>": 152060,
+  "<392>": 152061,
+  "<393>": 152062,
+  "<394>": 152063,
+  "<395>": 152064,
+  "<396>": 152065,
+  "<397>": 152066,
+  "<398>": 152067,
+  "<399>": 152068,
+  "<39>": 151708,
+  "<3>": 151672,
+  "<400>": 152069,
+  "<401>": 152070,
+  "<402>": 152071,
+  "<403>": 152072,
+  "<404>": 152073,
+  "<405>": 152074,
+  "<406>": 152075,
+  "<407>": 152076,
+  "<408>": 152077,
+  "<409>": 152078,
+  "<40>": 151709,
+  "<410>": 152079,
+  "<411>": 152080,
+  "<412>": 152081,
+  "<413>": 152082,
+  "<414>": 152083,
+  "<415>": 152084,
+  "<416>": 152085,
+  "<417>": 152086,
+  "<418>": 152087,
+  "<419>": 152088,
+  "<41>": 151710,
+  "<420>": 152089,
+  "<421>": 152090,
+  "<422>": 152091,
+  "<423>": 152092,
+  "<424>": 152093,
+  "<425>": 152094,
+  "<426>": 152095,
+  "<427>": 152096,
+  "<428>": 152097,
+  "<429>": 152098,
+  "<42>": 151711,
+  "<430>": 152099,
+  "<431>": 152100,
+  "<432>": 152101,
+  "<433>": 152102,
+  "<434>": 152103,
+  "<435>": 152104,
+  "<436>": 152105,
+  "<437>": 152106,
+  "<438>": 152107,
+  "<439>": 152108,
+  "<43>": 151712,
+  "<440>": 152109,
+  "<441>": 152110,
+  "<442>": 152111,
+  "<443>": 152112,
+  "<444>": 152113,
+  "<445>": 152114,
+  "<446>": 152115,
+  "<447>": 152116,
+  "<448>": 152117,
+  "<449>": 152118,
+  "<44>": 151713,
+  "<450>": 152119,
+  "<451>": 152120,
+  "<452>": 152121,
+  "<453>": 152122,
+  "<454>": 152123,
+  "<455>": 152124,
+  "<456>": 152125,
+  "<457>": 152126,
+  "<458>": 152127,
+  "<459>": 152128,
+  "<45>": 151714,
+  "<460>": 152129,
+  "<461>": 152130,
+  "<462>": 152131,
+  "<463>": 152132,
+  "<464>": 152133,
+  "<465>": 152134,
+  "<466>": 152135,
+  "<467>": 152136,
+  "<468>": 152137,
+  "<469>": 152138,
+  "<46>": 151715,
+  "<470>": 152139,
+  "<471>": 152140,
+  "<472>": 152141,
+  "<473>": 152142,
+  "<474>": 152143,
+  "<475>": 152144,
+  "<476>": 152145,
+  "<477>": 152146,
+  "<478>": 152147,
+  "<479>": 152148,
+  "<47>": 151716,
+  "<480>": 152149,
+  "<481>": 152150,
+  "<482>": 152151,
+  "<483>": 152152,
+  "<484>": 152153,
+  "<485>": 152154,
+  "<486>": 152155,
+  "<487>": 152156,
+  "<488>": 152157,
+  "<489>": 152158,
+  "<48>": 151717,
+  "<490>": 152159,
+  "<491>": 152160,
+  "<492>": 152161,
+  "<493>": 152162,
+  "<494>": 152163,
+  "<495>": 152164,
+  "<496>": 152165,
+  "<497>": 152166,
+  "<498>": 152167,
+  "<499>": 152168,
+  "<49>": 151718,
+  "<4>": 151673,
+  "<500>": 152169,
+  "<501>": 152170,
+  "<502>": 152171,
+  "<503>": 152172,
+  "<504>": 152173,
+  "<505>": 152174,
+  "<506>": 152175,
+  "<507>": 152176,
+  "<508>": 152177,
+  "<509>": 152178,
+  "<50>": 151719,
+  "<510>": 152179,
+  "<511>": 152180,
+  "<512>": 152181,
+  "<513>": 152182,
+  "<514>": 152183,
+  "<515>": 152184,
+  "<516>": 152185,
+  "<517>": 152186,
+  "<518>": 152187,
+  "<519>": 152188,
+  "<51>": 151720,
+  "<520>": 152189,
+  "<521>": 152190,
+  "<522>": 152191,
+  "<523>": 152192,
+  "<524>": 152193,
+  "<525>": 152194,
+  "<526>": 152195,
+  "<527>": 152196,
+  "<528>": 152197,
+  "<529>": 152198,
+  "<52>": 151721,
+  "<530>": 152199,
+  "<531>": 152200,
+  "<532>": 152201,
+  "<533>": 152202,
+  "<534>": 152203,
+  "<535>": 152204,
+  "<536>": 152205,
+  "<537>": 152206,
+  "<538>": 152207,
+  "<539>": 152208,
+  "<53>": 151722,
+  "<540>": 152209,
+  "<541>": 152210,
+  "<542>": 152211,
+  "<543>": 152212,
+  "<544>": 152213,
+  "<545>": 152214,
+  "<546>": 152215,
+  "<547>": 152216,
+  "<548>": 152217,
+  "<549>": 152218,
+  "<54>": 151723,
+  "<550>": 152219,
+  "<551>": 152220,
+  "<552>": 152221,
+  "<553>": 152222,
+  "<554>": 152223,
+  "<555>": 152224,
+  "<556>": 152225,
+  "<557>": 152226,
+  "<558>": 152227,
+  "<559>": 152228,
+  "<55>": 151724,
+  "<560>": 152229,
+  "<561>": 152230,
+  "<562>": 152231,
+  "<563>": 152232,
+  "<564>": 152233,
+  "<565>": 152234,
+  "<566>": 152235,
+  "<567>": 152236,
+  "<568>": 152237,
+  "<569>": 152238,
+  "<56>": 151725,
+  "<570>": 152239,
+  "<571>": 152240,
+  "<572>": 152241,
+  "<573>": 152242,
+  "<574>": 152243,
+  "<575>": 152244,
+  "<576>": 152245,
+  "<577>": 152246,
+  "<578>": 152247,
+  "<579>": 152248,
+  "<57>": 151726,
+  "<580>": 152249,
+  "<581>": 152250,
+  "<582>": 152251,
+  "<583>": 152252,
+  "<584>": 152253,
+  "<585>": 152254,
+  "<586>": 152255,
+  "<587>": 152256,
+  "<588>": 152257,
+  "<589>": 152258,
+  "<58>": 151727,
+  "<590>": 152259,
+  "<591>": 152260,
+  "<592>": 152261,
+  "<593>": 152262,
+  "<594>": 152263,
+  "<595>": 152264,
+  "<596>": 152265,
+  "<597>": 152266,
+  "<598>": 152267,
+  "<599>": 152268,
+  "<59>": 151728,
+  "<5>": 151674,
+  "<600>": 152269,
+  "<601>": 152270,
+  "<602>": 152271,
+  "<603>": 152272,
+  "<604>": 152273,
+  "<605>": 152274,
+  "<606>": 152275,
+  "<607>": 152276,
+  "<608>": 152277,
+  "<609>": 152278,
+  "<60>": 151729,
+  "<610>": 152279,
+  "<611>": 152280,
+  "<612>": 152281,
+  "<613>": 152282,
+  "<614>": 152283,
+  "<615>": 152284,
+  "<616>": 152285,
+  "<617>": 152286,
+  "<618>": 152287,
+  "<619>": 152288,
+  "<61>": 151730,
+  "<620>": 152289,
+  "<621>": 152290,
+  "<622>": 152291,
+  "<623>": 152292,
+  "<624>": 152293,
+  "<625>": 152294,
+  "<626>": 152295,
+  "<627>": 152296,
+  "<628>": 152297,
+  "<629>": 152298,
+  "<62>": 151731,
+  "<630>": 152299,
+  "<631>": 152300,
+  "<632>": 152301,
+  "<633>": 152302,
+  "<634>": 152303,
+  "<635>": 152304,
+  "<636>": 152305,
+  "<637>": 152306,
+  "<638>": 152307,
+  "<639>": 152308,
+  "<63>": 151732,
+  "<640>": 152309,
+  "<641>": 152310,
+  "<642>": 152311,
+  "<643>": 152312,
+  "<644>": 152313,
+  "<645>": 152314,
+  "<646>": 152315,
+  "<647>": 152316,
+  "<648>": 152317,
+  "<649>": 152318,
+  "<64>": 151733,
+  "<650>": 152319,
+  "<651>": 152320,
+  "<652>": 152321,
+  "<653>": 152322,
+  "<654>": 152323,
+  "<655>": 152324,
+  "<656>": 152325,
+  "<657>": 152326,
+  "<658>": 152327,
+  "<659>": 152328,
+  "<65>": 151734,
+  "<660>": 152329,
+  "<661>": 152330,
+  "<662>": 152331,
+  "<663>": 152332,
+  "<664>": 152333,
+  "<665>": 152334,
+  "<666>": 152335,
+  "<667>": 152336,
+  "<668>": 152337,
+  "<669>": 152338,
+  "<66>": 151735,
+  "<670>": 152339,
+  "<671>": 152340,
+  "<672>": 152341,
+  "<673>": 152342,
+  "<674>": 152343,
+  "<675>": 152344,
+  "<676>": 152345,
+  "<677>": 152346,
+  "<678>": 152347,
+  "<679>": 152348,
+  "<67>": 151736,
+  "<680>": 152349,
+  "<681>": 152350,
+  "<682>": 152351,
+  "<683>": 152352,
+  "<684>": 152353,
+  "<685>": 152354,
+  "<686>": 152355,
+  "<687>": 152356,
+  "<688>": 152357,
+  "<689>": 152358,
+  "<68>": 151737,
+  "<690>": 152359,
+  "<691>": 152360,
+  "<692>": 152361,
+  "<693>": 152362,
+  "<694>": 152363,
+  "<695>": 152364,
+  "<696>": 152365,
+  "<697>": 152366,
+  "<698>": 152367,
+  "<699>": 152368,
+  "<69>": 151738,
+  "<6>": 151675,
+  "<700>": 152369,
+  "<701>": 152370,
+  "<702>": 152371,
+  "<703>": 152372,
+  "<704>": 152373,
+  "<705>": 152374,
+  "<706>": 152375,
+  "<707>": 152376,
+  "<708>": 152377,
+  "<709>": 152378,
+  "<70>": 151739,
+  "<710>": 152379,
+  "<711>": 152380,
+  "<712>": 152381,
+  "<713>": 152382,
+  "<714>": 152383,
+  "<715>": 152384,
+  "<716>": 152385,
+  "<717>": 152386,
+  "<718>": 152387,
+  "<719>": 152388,
+  "<71>": 151740,
+  "<720>": 152389,
+  "<721>": 152390,
+  "<722>": 152391,
+  "<723>": 152392,
+  "<724>": 152393,
+  "<725>": 152394,
+  "<726>": 152395,
+  "<727>": 152396,
+  "<728>": 152397,
+  "<729>": 152398,
+  "<72>": 151741,
+  "<730>": 152399,
+  "<731>": 152400,
+  "<732>": 152401,
+  "<733>": 152402,
+  "<734>": 152403,
+  "<735>": 152404,
+  "<736>": 152405,
+  "<737>": 152406,
+  "<738>": 152407,
+  "<739>": 152408,
+  "<73>": 151742,
+  "<740>": 152409,
+  "<741>": 152410,
+  "<742>": 152411,
+  "<743>": 152412,
+  "<744>": 152413,
+  "<745>": 152414,
+  "<746>": 152415,
+  "<747>": 152416,
+  "<748>": 152417,
+  "<749>": 152418,
+  "<74>": 151743,
+  "<750>": 152419,
+  "<751>": 152420,
+  "<752>": 152421,
+  "<753>": 152422,
+  "<754>": 152423,
+  "<755>": 152424,
+  "<756>": 152425,
+  "<757>": 152426,
+  "<758>": 152427,
+  "<759>": 152428,
+  "<75>": 151744,
+  "<760>": 152429,
+  "<761>": 152430,
+  "<762>": 152431,
+  "<763>": 152432,
+  "<764>": 152433,
+  "<765>": 152434,
+  "<766>": 152435,
+  "<767>": 152436,
+  "<768>": 152437,
+  "<769>": 152438,
+  "<76>": 151745,
+  "<770>": 152439,
+  "<771>": 152440,
+  "<772>": 152441,
+  "<773>": 152442,
+  "<774>": 152443,
+  "<775>": 152444,
+  "<776>": 152445,
+  "<777>": 152446,
+  "<778>": 152447,
+  "<779>": 152448,
+  "<77>": 151746,
+  "<780>": 152449,
+  "<781>": 152450,
+  "<782>": 152451,
+  "<783>": 152452,
+  "<784>": 152453,
+  "<785>": 152454,
+  "<786>": 152455,
+  "<787>": 152456,
+  "<788>": 152457,
+  "<789>": 152458,
+  "<78>": 151747,
+  "<790>": 152459,
+  "<791>": 152460,
+  "<792>": 152461,
+  "<793>": 152462,
+  "<794>": 152463,
+  "<795>": 152464,
+  "<796>": 152465,
+  "<797>": 152466,
+  "<798>": 152467,
+  "<799>": 152468,
+  "<79>": 151748,
+  "<7>": 151676,
+  "<800>": 152469,
+  "<801>": 152470,
+  "<802>": 152471,
+  "<803>": 152472,
+  "<804>": 152473,
+  "<805>": 152474,
+  "<806>": 152475,
+  "<807>": 152476,
+  "<808>": 152477,
+  "<809>": 152478,
+  "<80>": 151749,
+  "<810>": 152479,
+  "<811>": 152480,
+  "<812>": 152481,
+  "<813>": 152482,
+  "<814>": 152483,
+  "<815>": 152484,
+  "<816>": 152485,
+  "<817>": 152486,
+  "<818>": 152487,
+  "<819>": 152488,
+  "<81>": 151750,
+  "<820>": 152489,
+  "<821>": 152490,
+  "<822>": 152491,
+  "<823>": 152492,
+  "<824>": 152493,
+  "<825>": 152494,
+  "<826>": 152495,
+  "<827>": 152496,
+  "<828>": 152497,
+  "<829>": 152498,
+  "<82>": 151751,
+  "<830>": 152499,
+  "<831>": 152500,
+  "<832>": 152501,
+  "<833>": 152502,
+  "<834>": 152503,
+  "<835>": 152504,
+  "<836>": 152505,
+  "<837>": 152506,
+  "<838>": 152507,
+  "<839>": 152508,
+  "<83>": 151752,
+  "<840>": 152509,
+  "<841>": 152510,
+  "<842>": 152511,
+  "<843>": 152512,
+  "<844>": 152513,
+  "<845>": 152514,
+  "<846>": 152515,
+  "<847>": 152516,
+  "<848>": 152517,
+  "<849>": 152518,
+  "<84>": 151753,
+  "<850>": 152519,
+  "<851>": 152520,
+  "<852>": 152521,
+  "<853>": 152522,
+  "<854>": 152523,
+  "<855>": 152524,
+  "<856>": 152525,
+  "<857>": 152526,
+  "<858>": 152527,
+  "<859>": 152528,
+  "<85>": 151754,
+  "<860>": 152529,
+  "<861>": 152530,
+  "<862>": 152531,
+  "<863>": 152532,
+  "<864>": 152533,
+  "<865>": 152534,
+  "<866>": 152535,
+  "<867>": 152536,
+  "<868>": 152537,
+  "<869>": 152538,
+  "<86>": 151755,
+  "<870>": 152539,
+  "<871>": 152540,
+  "<872>": 152541,
+  "<873>": 152542,
+  "<874>": 152543,
+  "<875>": 152544,
+  "<876>": 152545,
+  "<877>": 152546,
+  "<878>": 152547,
+  "<879>": 152548,
+  "<87>": 151756,
+  "<880>": 152549,
+  "<881>": 152550,
+  "<882>": 152551,
+  "<883>": 152552,
+  "<884>": 152553,
+  "<885>": 152554,
+  "<886>": 152555,
+  "<887>": 152556,
+  "<888>": 152557,
+  "<889>": 152558,
+  "<88>": 151757,
+  "<890>": 152559,
+  "<891>": 152560,
+  "<892>": 152561,
+  "<893>": 152562,
+  "<894>": 152563,
+  "<895>": 152564,
+  "<896>": 152565,
+  "<897>": 152566,
+  "<898>": 152567,
+  "<899>": 152568,
+  "<89>": 151758,
+  "<8>": 151677,
+  "<900>": 152569,
+  "<901>": 152570,
+  "<902>": 152571,
+  "<903>": 152572,
+  "<904>": 152573,
+  "<905>": 152574,
+  "<906>": 152575,
+  "<907>": 152576,
+  "<908>": 152577,
+  "<909>": 152578,
+  "<90>": 151759,
+  "<910>": 152579,
+  "<911>": 152580,
+  "<912>": 152581,
+  "<913>": 152582,
+  "<914>": 152583,
+  "<915>": 152584,
+  "<916>": 152585,
+  "<917>": 152586,
+  "<918>": 152587,
+  "<919>": 152588,
+  "<91>": 151760,
+  "<920>": 152589,
+  "<921>": 152590,
+  "<922>": 152591,
+  "<923>": 152592,
+  "<924>": 152593,
+  "<925>": 152594,
+  "<926>": 152595,
+  "<927>": 152596,
+  "<928>": 152597,
+  "<929>": 152598,
+  "<92>": 151761,
+  "<930>": 152599,
+  "<931>": 152600,
+  "<932>": 152601,
+  "<933>": 152602,
+  "<934>": 152603,
+  "<935>": 152604,
+  "<936>": 152605,
+  "<937>": 152606,
+  "<938>": 152607,
+  "<939>": 152608,
+  "<93>": 151762,
+  "<940>": 152609,
+  "<941>": 152610,
+  "<942>": 152611,
+  "<943>": 152612,
+  "<944>": 152613,
+  "<945>": 152614,
+  "<946>": 152615,
+  "<947>": 152616,
+  "<948>": 152617,
+  "<949>": 152618,
+  "<94>": 151763,
+  "<950>": 152619,
+  "<951>": 152620,
+  "<952>": 152621,
+  "<953>": 152622,
+  "<954>": 152623,
+  "<955>": 152624,
+  "<956>": 152625,
+  "<957>": 152626,
+  "<958>": 152627,
+  "<959>": 152628,
+  "<95>": 151764,
+  "<960>": 152629,
+  "<961>": 152630,
+  "<962>": 152631,
+  "<963>": 152632,
+  "<964>": 152633,
+  "<965>": 152634,
+  "<966>": 152635,
+  "<967>": 152636,
+  "<968>": 152637,
+  "<969>": 152638,
+  "<96>": 151765,
+  "<970>": 152639,
+  "<971>": 152640,
+  "<972>": 152641,
+  "<973>": 152642,
+  "<974>": 152643,
+  "<975>": 152644,
+  "<976>": 152645,
+  "<977>": 152646,
+  "<978>": 152647,
+  "<979>": 152648,
+  "<97>": 151766,
+  "<980>": 152649,
+  "<981>": 152650,
+  "<982>": 152651,
+  "<983>": 152652,
+  "<984>": 152653,
+  "<985>": 152654,
+  "<986>": 152655,
+  "<987>": 152656,
+  "<988>": 152657,
+  "<989>": 152658,
+  "<98>": 151767,
+  "<990>": 152659,
+  "<991>": 152660,
+  "<992>": 152661,
+  "<993>": 152662,
+  "<994>": 152663,
+  "<995>": 152664,
+  "<996>": 152665,
+  "<997>": 152666,
+  "<998>": 152667,
+  "<999>": 152668,
+  "<99>": 151768,
+  "<9>": 151678,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652,
+  "|<MASK>|": 152670
+}

assets/decoding.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f6fb9d57b88af79cbe78ee9f66cbba22519732daa2068484fc1e449d8fb138a
+size 8080597

assets/demo.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d987aac49b390c0411faecc33892873fb0b8b811ff1f961d8a87cb39c406285
+size 22955151

chat_template.jinja

@@ -0,0 +1,7 @@
+{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system
+You are a helpful assistant.<|im_end|>
+{% endif %}<|im_start|>{{ message['role'] }}
+{% if message['content'] is string %}{{ message['content'] }}<|im_end|>
+{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>
+{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
+{% endif %}

checksums.sha256

@@ -0,0 +1,22 @@
+20c797ce19af0c17de52c6afb144644768a591c521655f5ebf5712c9850f2887  LICENSE
+235136e6f31b16cd2012fa1279cf97141b5eef6916c775ba075cdc7f42c043ba  README.md
+669bb095cca2e86ddc821926f4c3b42dd7389f2ad3ec49c8edca8fe6294aae7f  added_tokens.json
+a0bc6f6fc7a29a80017a433e8f03a1cc1236e838a944a2d034295a60c4f2fddb  chat_template.jinja
+22c369e2bdac19723fe7db7e4c64b224462744b1f1aae7bd5daf05307db3b9fd  config.json
+7ad33532c846fdd81dd41a6ba54db54e3c47cada50472d732ba91eaede725323  configuration_groundinganything.py
+6b255ff08426effe581833b3809bf2232cac9eb93d1e5cf83fff76ce62791e38  configuration_groundinganything_vision.py
+42e0cb3d5a6a9e5c93c20555739f1038753ba1adb139c5c82a3a9f3b39ed5eab  generation_config.json
+33cb222671104ec4f0d3d234a3db7561b94d11543089d5f250c14a2f1b8651c3  image_processing_groundinganything.py
+78403540328f9847d6b7ebc5c44eb2e6a752863de0afb7d0710728bb161dc60d  media_utils.py
+8831e4f1a044471340f7c0a83d7bd71306a5b867e95fd870f74d0c5308a904d5  merges.txt
+3cf09355cde8cf4877cdc76e0a72301d572c1d6263b2831f8a03368285bb65e2  model.safetensors
+86ba0522e32bcf5d512f8e7dcc529abd22a39f10d55fa8327efcb65cbd758a67  modeling_groundinganything.py
+a839e10631620ae5fe70ec3aa9d009af521f7ea03faa66b32adce287105003be  modeling_groundinganything_vision.py
+11ddb518de0bcafe3f46f49310887bf7424b4f533d300c5f4749052c9d6465d1  preprocessor_config.json
+90e533272ff79d068ce922e9156e5d83f3cf2cc9e565af37b135c09e9fe57b81  processing_groundinganything.py
+534931ef99997dec0735da46a570a45696cfee6dace291cc9fa9a2da24420685  requirements.txt
+d948513463b339ae85d6be6c99ecee787cc9c7a96be09180aa521a32a216eff4  special_tokens_map.json
+5d9a9d7525aeecc0360ffd43f891ce9d2220297df5092a293a98e2fd31e9d99d  streammind_gate.py
+7e0398aca93659140fd6345deb335db2330dee89a2aad1228d8604c1952f6dc8  tokenizer.json
+2aa39a017b159c2ac86edb3baf6a78a77549fc7b26ea3cee9c37128b31fec589  tokenizer_config.json
+ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910  vocab.json

config.json

@@ -0,0 +1,141 @@
+{
+  "architectures": [
+    "GroundAnythingForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_groundinganything.GroundAnythingConfig",
+    "AutoModel": "modeling_groundinganything.GroundAnythingModel",
+    "AutoModelForCausalLM": "modeling_groundinganything.GroundAnythingForConditionalGeneration",
+    "AutoModelForImageTextToText": "modeling_groundinganything.GroundAnythingForConditionalGeneration",
+    "AutoProcessor": "processing_groundinganything.GroundAnythingProcessor"
+  },
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 151645,
+  "hidden_size": 2560,
+  "image_token_id": 151655,
+  "model_type": "groundinganything",
+  "pad_token_id": 151643,
+  "text_config": {
+    "_name_or_path": "Qwen3-4B-Instruct-2507",
+    "architectures": [
+      "Qwen3ForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "dtype": "bfloat16",
+    "eos_token_id": 151645,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 2560,
+    "initializer_range": 0.02,
+    "intermediate_size": 9728,
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 262144,
+    "max_window_layers": 36,
+    "model_type": "qwen3",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 36,
+    "num_key_value_heads": 8,
+    "pad_token_id": 151643,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "rope_theta": 5000000,
+      "rope_type": "default"
+    },
+    "sliding_window": null,
+    "tie_word_embeddings": false,
+    "use_cache": false,
+    "use_sliding_window": false,
+    "vocab_size": 152670
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.7.0",
+  "use_cache": false,
+  "video_token_id": 151656,
+  "vision_config": {
+    "activation_func": "gelu_pytorch_tanh",
+    "attention_dropout": 0.0,
+    "attn_bias": false,
+    "dtype": "bfloat16",
+    "frame_windows_size": 4,
+    "hidden_act": "gelu",
+    "hidden_size": 1024,
+    "image_size": 448,
+    "init_pos_emb_height": 64,
+    "init_pos_emb_time": 4,
+    "init_pos_emb_width": 64,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "layer_norm_eps": 1e-06,
+    "layer_norm_type": "layer_norm",
+    "linear_bias": false,
+    "max_position_embeddings": 8192,
+    "merge_kernel_size": [
+      2,
+      2
+    ],
+    "merge_type": "sd2_tpool",
+    "mlp_type": "mlp2",
+    "model_type": "groundinganything_vision",
+    "norm_type": "rmsnorm",
+    "num_attention_heads": 12,
+    "num_channels": 3,
+    "num_hidden_layers": 27,
+    "out_hidden_size": 2560,
+    "patch_embed_proj_bias": false,
+    "patch_position_encoding_type": "absolute",
+    "patch_size": 14,
+    "pos_emb_interpolation_mode": "bilinear",
+    "pos_emb_type": "divided_fixed",
+    "projector_hidden_act": "gelu",
+    "projector_hidden_size": 4096,
+    "projector_ln_eps": 1e-05,
+    "qkv_hidden_size": 1536,
+    "rope_theta": 10000.0,
+    "spatial_merge_size": 2,
+    "tokens_per_second": 1,
+    "use_head": false,
+    "use_patch_position_encoding": false
+  },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652
+}

configuration_groundinganything.py

@@ -0,0 +1,171 @@
+from dataclasses import dataclass
+
+from transformers.models.qwen3.configuration_qwen3 import Qwen3Config
+try:
+    from transformers.configuration_utils import PreTrainedConfig
+except ImportError:
+    from transformers.configuration_utils import PretrainedConfig as PreTrainedConfig
+
+
+@dataclass(init=False)
+class GroundAnythingVLMVisionConfig(PreTrainedConfig):
+    model_type = "groundinganything_vision"
+    base_config_key = "vision_config"
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+
+    hidden_size: int = 1024
+    intermediate_size: int = 4096
+    num_hidden_layers: int = 24
+    num_attention_heads: int = 16
+    num_channels: int = 3
+    image_size: int = 448
+    patch_size: int = 14
+    hidden_act: str = "gelu"
+    layer_norm_eps: float = 1e-6
+    layer_norm_type: str = "layer_norm"
+    attention_dropout: float = 0.0
+    initializer_range: float = 0.02
+    rope_theta: float = 10000.0
+    use_head: bool = False
+    out_hidden_size: int = 1024
+    spatial_merge_size: int = 2
+    tokens_per_second: int = 1
+    frame_windows_size: int = 4
+    use_patch_position_encoding: bool = False
+    patch_position_encoding_type: str = "absolute"
+    max_position_embeddings: int = 8192
+    init_pos_emb_height: int = 64
+    init_pos_emb_width: int = 64
+    init_pos_emb_time: int = 4
+    pos_emb_type: str = "divided_fixed"
+    merge_kernel_size: list | None = None
+    merge_type: str = "sd2_tpool"
+    qkv_hidden_size: int = 1536
+    norm_type: str = "rmsnorm"
+    attn_bias: bool = False
+    patch_embed_proj_bias: bool = False
+    mlp_type: str = "mlp2"
+    linear_bias: bool = False
+    activation_func: str = "gelu_pytorch_tanh"
+    pos_emb_interpolation_mode: str = "bilinear"
+    projector_hidden_size: int = 4096
+    projector_hidden_act: str = "gelu"
+    projector_ln_eps: float = 1e-5
+
+
+@dataclass(init=False)
+class GroundAnythingVLMConfig(PreTrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`GroundAnythingVLMBaseModel`]. It is used to instantiate a
+    GroundAnythingVLMBaseModel model according to the specified arguments, defining the model architecture. Instantiating a configuration
+    with the defaults will yield a GroundAnything-VLM configuration.
+
+    Configuration objects inherit from [`PreTrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PreTrainedConfig`] for more information.
+
+    Args:
+        text_config (`Union[PreTrainedConfig, dict]`, *optional*, defaults to `Qwen3Config`):
+            The config object or dictionary of the text backbone.
+        vision_config (`Union[PreTrainedConfig, dict]`, *optional*, defaults to `GroundAnythingVLMVisionConfig`):
+            The config object or dictionary of the vision backbone.
+        image_token_id (`int`, *optional*, defaults to 151655):
+            The image token index to encode the image prompt.
+        video_token_id (`int`, *optional*, defaults to 151656):
+            The video token index to encode the image prompt.
+        vision_start_token_id (`int`, *optional*, defaults to 151652):
+            The token index to denote start of vision input.
+        vision_end_token_id (`int`, *optional*, defaults to 151653):
+            The token index to denote end of vision input.
+    """
+
+    model_type = "groundinganything_vlm"
+    # `text_config` is resolved dynamically based on its `model_type` (defaults to `qwen3`),
+    # so we use `AutoConfig` here as a placeholder; `__post_init__` swaps it for the
+    # concrete config class via `CONFIG_MAPPING`.
+    sub_configs = {"vision_config": GroundAnythingVLMVisionConfig, "text_config": Qwen3Config}
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+
+    def __init__(self, **kwargs):
+        self.text_config = kwargs.pop("text_config", None)
+        self.vision_config = kwargs.pop("vision_config", None)
+        self.image_token_id = kwargs.pop("image_token_id", 151655)
+        self.video_token_id = kwargs.pop("video_token_id", 151656)
+        self.vision_start_token_id = kwargs.pop("vision_start_token_id", 151652)
+        self.vision_end_token_id = kwargs.pop("vision_end_token_id", 151653)
+        self.tie_word_embeddings = kwargs.pop("tie_word_embeddings", False)
+        self.bos_token_id = kwargs.pop("bos_token_id", None)
+        self.eos_token_id = kwargs.pop("eos_token_id", None)
+        self.pad_token_id = kwargs.pop("pad_token_id", None)
+        if isinstance(self.vision_config, dict):
+            self.vision_config = GroundAnythingVLMVisionConfig(**self.vision_config)
+        if isinstance(self.text_config, dict):
+            text_model_type = self.text_config.get("model_type", "qwen3")
+            if text_model_type != "qwen3":
+                raise ValueError(f"unsupported text model type: {text_model_type}")
+            text_config_cls = Qwen3Config
+            self.sub_configs["text_config"] = text_config_cls
+            self.text_config = text_config_cls(**self.text_config)
+        # Transformers 5.x uses a generated dataclass __init__ that dispatches
+        # to this subclass' __post_init__. Call the base hook explicitly so its
+        # private attention/config state is initialized. Keep 4.x compatible.
+        if hasattr(PreTrainedConfig, "__post_init__"):
+            PreTrainedConfig.__post_init__(self, **kwargs)
+        else:
+            super().__init__(**kwargs)
+        self.__post_init__()
+
+    text_config: dict | PreTrainedConfig | None = None
+    vision_config: dict | PreTrainedConfig | None = None
+    image_token_id: int = 151655
+    video_token_id: int = 151656
+    vision_start_token_id: int = 151652
+    vision_end_token_id: int = 151653
+    tie_word_embeddings: bool = False
+    # Generation-related token ids are mirrored from `text_config` in `__post_init__`
+    # so downstream tools (e.g. `generate`, vLLM) that read them at the top level keep working.
+    bos_token_id: int | None = None
+    eos_token_id: int | list[int] | None = None
+    pad_token_id: int | None = None
+
+    def __post_init__(self, **kwargs):
+        # Resolve vision_config
+        if isinstance(self.vision_config, dict):
+            self.vision_config = self.sub_configs["vision_config"](**self.vision_config)
+        elif self.vision_config is None:
+            self.vision_config = self.sub_configs["vision_config"]()
+
+        # Resolve text_config dynamically via CONFIG_MAPPING (defaults to qwen3)
+        if isinstance(self.text_config, dict):
+            text_model_type = self.text_config.get("model_type", "qwen3")
+            self.text_config["model_type"] = text_model_type
+            if text_model_type != "qwen3":
+                raise ValueError(f"unsupported text model type: {text_model_type}")
+            text_config_cls = Qwen3Config
+            self.sub_configs["text_config"] = text_config_cls
+            self.text_config = text_config_cls(**self.text_config)
+        elif self.text_config is None:
+            text_config_cls = Qwen3Config
+            self.sub_configs["text_config"] = text_config_cls
+            self.text_config = text_config_cls()
+
+        # Mirror generation-related token ids from text_config to the top level so
+        # downstream tools (e.g. `generate`, chat templates, vLLM) that read them
+        # from the top-level config keep working.
+        for tok_key in ("bos_token_id", "eos_token_id", "pad_token_id"):
+            text_val = getattr(self.text_config, tok_key, None)
+            if text_val is not None and getattr(self, tok_key, None) is None:
+                setattr(self, tok_key, text_val)
+
+        del kwargs
+
+
+__all__ = ["GroundAnythingVLMConfig", "GroundAnythingVLMVisionConfig", "GroundAnythingConfig"]
+
+
+class GroundAnythingConfig(GroundAnythingVLMConfig):
+    """DLM release identity for the shared Qwen3 VLM configuration."""
+
+    model_type = "groundinganything"

configuration_groundinganything_vision.py

@@ -0,0 +1,285 @@
+from typing import Optional
+
+from transformers.configuration_utils import PretrainedConfig
+
+
+class GroundAnythingBackboneLinearConfig(PretrainedConfig):
+    model_type = "kimi_linear"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        model_type="kimi_linear",
+        vocab_size=163840,
+        hidden_size=4096,
+        head_dim=None,
+        intermediate_size=11008,
+        num_hidden_layers=32,
+        num_attention_heads=32,
+        num_key_value_heads=None,
+        hidden_act="silu",
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        pad_token_id=0,
+        bos_token_id=1,
+        eos_token_id=2,
+        rope_theta=10000.0,
+        rope_scaling=None,
+        tie_word_embeddings=False,
+        moe_intermediate_size: Optional[int] = None,
+        moe_renormalize: bool = True,
+        moe_router_activation_func: str = "sigmoid",
+        num_experts: Optional[int] = None,
+        num_experts_per_token: Optional[int] = None,
+        num_shared_experts: int = 0,
+        routed_scaling_factor: float = 1.0,
+        first_k_dense_replace: int = 0,
+        moe_layer_freq: int = 1,
+        use_grouped_topk: bool = True,
+        num_expert_group: int = 1,
+        topk_group: int = 1,
+        q_lora_rank: Optional[int] = None,
+        kv_lora_rank: Optional[int] = None,
+        qk_nope_head_dim: Optional[int] = None,
+        qk_rope_head_dim: Optional[int] = None,
+        v_head_dim: Optional[int] = None,
+        mla_use_nope: Optional[bool] = False,
+        mla_use_output_gate: Optional[bool] = False,
+        num_nextn_predict_layers: int = 0,
+        linear_attn_config: Optional[dict] = None,
+        attn_res_block_size: Optional[int] = None,
+        latent_moe_use_norm: bool = False,
+        activation_situ_beta: Optional[float] = None,
+        activation_situ_linear_beta: Optional[float] = None,
+        max_position_embeddings: int = 4096,
+        routed_expert_hidden_size: Optional[int] = None,
+        topk_method: str = "noaux_tc",
+        **kwargs,
+    ):
+        self.model_type = model_type
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.head_dim = (
+            head_dim if head_dim is not None else hidden_size // num_attention_heads
+        )
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+
+        # for backward compatibility
+        if num_key_value_heads is None:
+            num_key_value_heads = num_attention_heads
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+
+        self.q_lora_rank = q_lora_rank
+        self.kv_lora_rank = kv_lora_rank
+        self.qk_nope_head_dim = qk_nope_head_dim
+        self.qk_rope_head_dim = qk_rope_head_dim
+        self.v_head_dim = v_head_dim
+        self.mla_use_nope = mla_use_nope
+        self.mla_use_output_gate = mla_use_output_gate
+        # moe config
+        self.num_experts = num_experts
+        self.num_experts_per_token = num_experts_per_token
+        self.moe_renormalize = moe_renormalize
+        self.num_shared_experts = num_shared_experts
+        self.routed_scaling_factor = routed_scaling_factor
+        self.moe_router_activation_func = moe_router_activation_func
+        assert self.moe_router_activation_func in ("softmax", "sigmoid")
+        self.moe_intermediate_size = moe_intermediate_size
+        self.first_k_dense_replace = first_k_dense_replace
+        self.moe_layer_freq = moe_layer_freq
+        self.use_grouped_topk = use_grouped_topk
+        self.num_expert_group = num_expert_group
+        self.topk_group = topk_group
+        self.num_nextn_predict_layers = num_nextn_predict_layers
+
+        self.attn_res_block_size = attn_res_block_size
+        self.latent_moe_use_norm = latent_moe_use_norm
+        self.activation_situ_beta = activation_situ_beta
+        self.activation_situ_linear_beta = activation_situ_linear_beta
+        self.max_position_embeddings = max_position_embeddings
+        self.routed_expert_hidden_size = routed_expert_hidden_size
+        self.topk_method = topk_method
+
+        if linear_attn_config is not None:
+            assert linear_attn_config["kda_layers"] is not None
+            assert linear_attn_config["full_attn_layers"] is not None
+        self.linear_attn_config = linear_attn_config
+
+        super().__init__(
+            pad_token_id=pad_token_id,
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+    @property
+    def is_mla(self):
+        return (
+            self.q_lora_rank is not None
+            or self.kv_lora_rank is not None
+            or self.qk_nope_head_dim is not None
+            or self.qk_rope_head_dim is not None
+            or self.v_head_dim is not None
+            or self.mla_use_nope is True
+        )
+
+    @property
+    def is_moe(self):
+        return self.num_experts is not None
+
+    @property
+    def is_linear_attn(self) -> bool:
+        return not (
+            self.linear_attn_config is None
+            or (
+                isinstance(self.linear_attn_config, dict)
+                and self.linear_attn_config["kda_layers"] is not None
+                and len(self.linear_attn_config["kda_layers"]) == 0
+            )
+        )
+
+    def is_kda_layer(self, layer_idx: int):
+        return (
+            self.linear_attn_config is not None
+            and (layer_idx + 1) in self.linear_attn_config["kda_layers"]
+        )
+
+
+class GroundAnythingBackboneVisionConfig(PretrainedConfig):
+
+    def __init__(
+            self,
+            patch_size: int = 14,
+            init_pos_emb_height: int = 64,
+            init_pos_emb_width: int = 64,
+            init_pos_emb_time: int = 4,
+            pos_emb_type: str = 'divided_fixed',
+            vt_num_attention_heads: int = 12,
+            vt_num_hidden_layers: int = 27,
+            vt_hidden_size: int = 1024,
+            vt_intermediate_size: int = 4096,
+            merge_kernel_size: tuple = (2, 2),
+            merge_type: str = 'sd2_tpool',
+            _attn_implementation: str = 'flash_attention_2',
+            # MM Projector parameters
+            mm_projector_type: str = 'patchmergerv2',
+            mm_hidden_size: int | None = None,
+            projector_hidden_act: str = "gelu",
+            projector_ln_eps: float = 1e-5,
+            # vision tower parameters
+            qkv_hidden_size: int = 1536,
+            norm_type: str = 'rmsnorm',
+            attn_bias: bool = False,
+            patch_embed_proj_bias: bool = False,
+            mlp_type: str = 'mlp2',
+            linear_bias: bool = False,
+            activation_func: str = 'gelu_pytorch_tanh',
+            pos_emb_interpolation_mode: str = 'bilinear',
+            # Other parameters
+            ignore_index: int = -100,
+            media_placeholder_token_id: int = 163605,
+            pad_token_id: int = 0,
+            text_hidden_size=7168,
+            **kwargs):
+
+        self.patch_size = patch_size
+        self.init_pos_emb_height = init_pos_emb_height
+        self.init_pos_emb_width = init_pos_emb_width
+        self.init_pos_emb_time = init_pos_emb_time
+        self.pos_emb_type = pos_emb_type
+        self.vt_num_attention_heads = vt_num_attention_heads
+        self.vt_num_hidden_layers = vt_num_hidden_layers
+        self.vt_hidden_size = vt_hidden_size
+        self.vt_intermediate_size = vt_intermediate_size
+        self.merge_kernel_size = merge_kernel_size
+        self.merge_type = merge_type
+        self._attn_implementation = _attn_implementation
+
+        # MM Projector config
+        self.mm_projector_type = mm_projector_type
+        self.mm_hidden_size = mm_hidden_size if mm_hidden_size is not None else vt_hidden_size
+        self.projector_hidden_act = projector_hidden_act
+        self.projector_ln_eps = projector_ln_eps
+        self.text_hidden_size = text_hidden_size
+
+        # vision tower parameters
+        self.qkv_hidden_size = qkv_hidden_size
+        self.norm_type = norm_type
+        self.attn_bias = attn_bias
+        self.patch_embed_proj_bias = patch_embed_proj_bias
+        self.mlp_type = mlp_type
+        self.linear_bias = linear_bias
+        self.activation_func = activation_func
+        self.pos_emb_interpolation_mode = pos_emb_interpolation_mode
+
+        super().__init__(**kwargs)
+
+
+class GroundAnythingBackboneConfig(PretrainedConfig):
+    """Kimi-K3 model configuration.
+
+    Args:
+        text_config (dict | GroundAnythingBackboneLinearConfig): Configuration for the text model.
+        
+        Vision Tower Parameters (from MoonViT3dConfig):
+            patch_size (int): Patch size for vision tower.
+            init_pos_emb_height (int): Initial position embedding height.
+            init_pos_emb_width (int): Initial position embedding width.
+            init_pos_emb_time (int): Initial position embedding time dimension.
+            pos_emb_type (str): Type of position embedding.
+            vt_num_attention_heads (int): Number of attention heads in vision tower.
+            vt_num_hidden_layers (int): Number of hidden layers in vision tower.
+            vt_hidden_size (int): Hidden size of vision tower.
+            vt_intermediate_size (int): Intermediate size in vision tower FFN.
+            merge_kernel_size (tuple): Kernel size for patch merging.
+            merge_type (str): Type of merge operation.
+            _attn_implementation (str): Attention implementation type.
+        
+        MM Projector Parameters (from MultiModalProjectorConfig):
+            mm_projector_type (str): Type of multimodal projector.
+            mm_hidden_size (int): Hidden size from vision tower (should match vt_hidden_size).
+            projector_hidden_act (str): Activation function for projector.
+            projector_ln_eps (float): Layer norm epsilon for projector.
+        
+        Other Parameters:
+            ignore_index (int): The ignore index for the loss function.
+            media_placeholder_token_id (int): The token ID to use for media placeholders.
+            pad_token_id (int): The token ID to use for padding.
+    """
+
+    model_type = "groundinganything_backbone"
+
+    def __init__(
+        self,
+        text_config: dict | GroundAnythingBackboneLinearConfig = None,
+        vision_config: dict | GroundAnythingBackboneVisionConfig = None,
+        # Other parameters
+        ignore_index: int = -100,
+        media_placeholder_token_id: int = 163605,
+        pad_token_id: int = 0,
+        **kwargs,
+    ):
+        if isinstance(text_config, dict):
+            text_config = GroundAnythingBackboneLinearConfig(**text_config)
+        if isinstance(vision_config, dict):
+            vision_config = GroundAnythingBackboneVisionConfig(**vision_config)
+        self.text_config = text_config
+        self.vision_config = vision_config
+        # Other config
+        self.ignore_index = ignore_index
+        self.media_placeholder_token_id = media_placeholder_token_id
+        if getattr(self.text_config, "quantization_config", None) is not None:
+            self.quantization_config = self.text_config.quantization_config
+
+        super().__init__(pad_token_id=pad_token_id, **kwargs)

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.7.0",
+  "use_cache": true
+}

image_processing_groundinganything.py

@@ -0,0 +1,208 @@
+"""Image processor class for Kimi-K3.
+"""
+
+import json
+import os
+from typing import Any, Dict, Optional, Union
+
+import numpy as np
+import torch
+from PIL import Image
+from transformers.image_processing_utils import (BaseImageProcessor,
+                                                 BatchFeature)
+from transformers.utils import TensorType
+
+from .media_utils import (MediaInput, TransparentBgConfig, _to_tensor,
+                          ensure_media_type, image_to_np, navit_patchify,
+                          navit_resize_image, normalize)
+
+
+class GroundAnythingVLMImageProcessor(BaseImageProcessor):
+    model_type = "groundinganything_backbone"
+
+    def __init__(
+        self,
+        media_proc_cfg: dict,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.media_proc_cfg = dict(media_proc_cfg)
+        self.patch_size = int(self.media_proc_cfg.get("patch_size", 14))
+        self.merge_size = int(self.media_proc_cfg.get("merge_kernel_size", 2))
+        max_output_tokens = os.getenv("IMAGE_MAX_TOKEN_NUM")
+        if max_output_tokens:
+            max_input_patches = int(max_output_tokens) * self.merge_size**2
+            self.media_proc_cfg["in_patch_limit"] = min(
+                int(self.media_proc_cfg["in_patch_limit"]), max_input_patches
+            )
+        self.max_output_tokens = (
+            int(self.media_proc_cfg["in_patch_limit"]) // self.merge_size**2
+        )
+
+    @property
+    def _transparent_bg_config(self) -> Optional[TransparentBgConfig]:
+        cfg = self.media_proc_cfg.get("transparent_bg_config")
+        if cfg is None:
+            return None
+        if isinstance(cfg, TransparentBgConfig):
+            return cfg
+        return TransparentBgConfig(**cfg)
+
+    @property
+    def _transparent_bg_fill_stage(self) -> str:
+        return self.media_proc_cfg.get("transparent_bg_fill_stage",
+                                       "before_resize")
+
+    def media_tokens_calculator(self, media: MediaInput):
+        media = ensure_media_type(
+            media,
+            transparent_bg_config=self._transparent_bg_config,
+            transparent_bg_fill_stage=self._transparent_bg_fill_stage,
+        )
+        ret = self.get_resize_config(media)
+        return ret['num_tokens']
+
+    @classmethod
+    def make_image_prompt(cls, width: int, height: int) -> str:
+        """Build the K3 image placeholder with resolution info."""
+        return (f"<|media_begin|>image {width}x{height}"
+                f"<|media_content|><|media_pad|><|media_end|>")
+
+    def get_resize_config(self, media_input: MediaInput) -> dict:
+        if media_input['type'] == 'image':
+            w, h = media_input['image'].size
+            input_patch_limit = int(self.media_proc_cfg['in_patch_limit'])
+            while True:
+                ret = navit_resize_image(
+                    w, h, self.media_proc_cfg['patch_size'],
+                    self.media_proc_cfg['merge_kernel_size'],
+                    input_patch_limit,
+                    self.media_proc_cfg['patch_limit_on_one_side'],
+                    self.media_proc_cfg['fixed_output_tokens'])
+                if ret['num_tokens'] <= self.max_output_tokens:
+                    return ret
+                reduced_limit = int(
+                    input_patch_limit * self.max_output_tokens /
+                    ret['num_tokens'])
+                input_patch_limit = min(input_patch_limit - 1,
+                                        reduced_limit)
+        else:
+            raise ValueError("Unsupported type: {}".format(
+                media_input['type']))
+
+    def resize_image(self, image: Image.Image, new_width: int, new_height: int,
+                     pad_width: int, pad_height: int) -> np.ndarray:
+        image_np = image_to_np(
+            image,
+            (new_width, new_height),
+            "resize",
+            transparent_bg_config=self._transparent_bg_config,
+            transparent_bg_fill_stage=self._transparent_bg_fill_stage,
+        )
+        image_np = np.pad(
+            image_np,
+            ((0, pad_height), (0, pad_width), (0, 0)),
+            mode="constant",
+            constant_values=0,
+        )
+        return image_np
+
+    def preprocess(
+        self,
+        medias: Optional[list[MediaInput]] = None,
+        return_tensors: Optional[Union[str, TensorType]] = None,
+        images=None,
+        do_resize: Optional[bool] = None,
+        **kwargs,
+    ) -> BatchFeature:
+        """
+        Preprocess a atom vision input (images) into model-ready tensors.
+
+        Args:
+            medias: List of MediaInput.
+            return_tensors: Desired output format ('pt', 'np', 'tf', or None).
+
+        Returns:
+            BatchFeature containing 'pixel_values' and 'grid_thws' tensors.
+        """
+        del do_resize, kwargs
+        if medias is None:
+            medias = images
+        if medias is None:
+            medias = []
+        if not isinstance(medias, list):
+            medias = [medias]
+        medias = [item if isinstance(item, dict) else {"type": "image", "image": item} for item in medias]
+        if medias:
+            pixel_values = []
+            for item in medias:
+                item = ensure_media_type(
+                    item,
+                    transparent_bg_config=self._transparent_bg_config,
+                    transparent_bg_fill_stage=self._transparent_bg_fill_stage,
+                )
+                resize_config = self.get_resize_config(item)
+                new_width, new_height, pad_width, pad_height = resize_config[
+                    'new_width'], resize_config['new_height'], resize_config[
+                        'pad_width'], resize_config['pad_height']
+                if item['type'] == 'image':
+                    image = item['image']
+                    image_np = self.resize_image(image, new_width, new_height,
+                                                 pad_width, pad_height)
+                    pixel_values.append(np.expand_dims(image_np, axis=0))
+                else:
+                    raise ValueError("Unsupported type: {}".format(
+                        item['type']))
+            normalized_pixel_values = []
+            image_std_inv = 1.0 / np.array(self.media_proc_cfg['image_std'])
+            image_mean = np.array(self.media_proc_cfg['image_mean'])
+            for pixels in pixel_values:
+                pixels = normalize(pixels, image_mean, image_std_inv)
+                pixels_and_thw = navit_patchify(
+                    pixels,
+                    self.media_proc_cfg['patch_size'],
+                )
+                normalized_pixel_values.append(pixels_and_thw)
+
+            pixel_values = torch.cat([
+                _to_tensor(pixel_value['pixel_values'])
+                for pixel_value in normalized_pixel_values
+            ])
+            grid_thws = torch.cat([
+                _to_tensor(pixel_value['grid_thw'],
+                           dtype=torch.int64).unsqueeze(0)
+                for pixel_value in normalized_pixel_values
+            ])
+
+            data = {
+                'pixel_values': pixel_values,
+                'image_grid_thw': grid_thws,
+            }
+
+        else:
+            data = {}
+
+        return BatchFeature(data=data, tensor_type=return_tensors)
+
+    def __repr__(self):
+        return f"GroundAnythingVLMImageProcessor(media_proc_cfg={self.media_proc_cfg})"
+
+    def to_dict(self) -> Dict[str, Any]:
+        output = super().to_dict()
+        output["media_proc_cfg"] = self.media_proc_cfg
+        if "media_processor" in output:
+            del output["media_processor"]
+        return output
+
+    @classmethod
+    def from_dict(cls, config_dict: Dict[str, Any], **kwargs):
+        config = config_dict.copy()
+        media_proc_cfg = config.pop("media_proc_cfg", {})
+        return cls(media_proc_cfg=media_proc_cfg, **config, **kwargs)
+
+    def to_json_string(self):
+        dictionary = self.to_dict()
+        for key, value in dictionary.items():
+            if hasattr(value, 'tolist'):
+                dictionary[key] = value.tolist()
+        return json.dumps(dictionary, indent=2, sort_keys=True) + "\n"

media_utils.py

@@ -0,0 +1,376 @@
+import base64
+import functools
+import io
+import math
+from dataclasses import dataclass
+from typing import Literal, TypedDict
+
+import numpy as np
+from PIL import Image
+
+
+class ImageInput(TypedDict):
+    type: Literal['image']
+    image: Image.Image
+
+
+MediaInput = ImageInput
+
+
+@dataclass
+class TransparentBgConfig:
+    """The config of the transparent background."""
+
+    pattern: Literal["white", "black", "gray", "chessboard"] = "black"
+    """The pattern of the transparent background."""
+
+    chessboard_square_size: int = 16
+    """The size of the squares in the chessboard background."""
+
+    chessboard_square_on_top_left: bool = True
+    """Whether to start the chessboard with a white square on the top left."""
+
+    chessboard_white_value: int = 255
+    """The value of the white pixels in the background."""
+
+    chessboard_gray_value: int = 200
+    """The value of the gray pixels in the background."""
+
+
+@functools.lru_cache(maxsize=256)
+def _create_chessboard_background(
+    height: int,
+    width: int,
+    square_size: int,
+    square_on_top_left: bool,
+    white_value: int,
+    gray_value: int,
+) -> np.ndarray:
+    """Create a chessboard background."""
+    bg = np.ones((height, width, 3), dtype=np.uint8) * white_value
+    for y in range(0, height, square_size):
+        for x in range(0, width, square_size):
+            if (y // square_size + x // square_size) % 2 == (
+                    1 if square_on_top_left else 0):
+                bg[y:y + square_size, x:x + square_size] = gray_value
+    return bg
+
+
+def fill_transparent_bg_with(
+    image: Image.Image,
+    transparent_bg_config: TransparentBgConfig | None = None,
+) -> Image.Image:
+    """Composite a (possibly) transparent image onto a configured background.
+
+    When ``transparent_bg_config`` is ``None``, the image is simply converted
+    to RGB (preserving the historical behavior). Otherwise the alpha channel
+    is alpha-composited over a background generated according to the config.
+    """
+    if transparent_bg_config is None:
+        return image.convert("RGB")
+
+    if image.mode == "RGB":
+        return image
+
+    has_alpha = "A" in image.getbands() or "transparency" in image.info
+    if not has_alpha:
+        return image.convert("RGB")
+
+    img = np.array(image.convert("RGBA"))
+    height, width = img.shape[:2]
+    bg_pattern = transparent_bg_config.pattern
+    if bg_pattern == "white":
+        bg = np.full((height, width, 3), 255, dtype=np.uint8)
+    elif bg_pattern == "black":
+        bg = np.zeros((height, width, 3), dtype=np.uint8)
+    elif bg_pattern == "gray":
+        bg = np.full((height, width, 3), 128, dtype=np.uint8)
+    elif bg_pattern == "chessboard":
+        bg = _create_chessboard_background(
+            height,
+            width,
+            transparent_bg_config.chessboard_square_size,
+            transparent_bg_config.chessboard_square_on_top_left,
+            transparent_bg_config.chessboard_white_value,
+            transparent_bg_config.chessboard_gray_value,
+        )
+    else:
+        raise ValueError(f"Invalid background pattern: {bg_pattern}")
+
+    alpha = img[:, :, 3]
+    img_rgb = img[:, :, :3]
+    alpha_normalized = alpha.astype(np.float32) / 255.0
+    alpha_3d = np.stack([alpha_normalized] * 3, axis=2)
+    result = alpha_3d * img_rgb + (1 - alpha_3d) * bg
+    result = result.astype(np.uint8)
+    return Image.fromarray(result)
+
+
+def navit_resize_image(
+    width: int,
+    height: int,
+    patch_size: int,
+    merge_kernel_size: int,
+    in_patch_limit: int,
+    patch_limit_on_one_side: int,
+    fixed_output_tokens: int | None,
+):
+    # Apply the patch limits.
+    s1 = math.sqrt(
+        in_patch_limit /
+        (max(1.0, width // patch_size) * max(1.0, height // patch_size)))
+    s2 = patch_limit_on_one_side * patch_size / width
+    s3 = patch_limit_on_one_side * patch_size / height
+    scale = min(1.0, s1, s2, s3)
+    new_w, new_h = max(1, int(width * scale)), max(1, int(height * scale))
+    new_w = min(new_w, patch_limit_on_one_side * patch_size)
+    new_h = min(new_h, patch_limit_on_one_side * patch_size)
+
+    # Calculate the padding to make the height and width divisible by the merge kernel size and patch size.
+    factor = merge_kernel_size * patch_size
+
+    pad_height = (factor - new_h % factor) % factor
+    pad_width = (factor - new_w % factor) % factor
+
+    if fixed_output_tokens is not None:
+        num_tokens = fixed_output_tokens
+    else:
+        # Calculate new dimensions after padding and patching
+        token_height = (new_h + pad_height) // factor
+        token_width = (new_w + pad_width) // factor
+
+        assert token_height * merge_kernel_size <= patch_limit_on_one_side, (
+            f"token_height {token_height} * merge_kernel_size {merge_kernel_size} > patch_limit_on_one_side {patch_limit_on_one_side}"
+        )
+        assert token_width * merge_kernel_size <= patch_limit_on_one_side, (
+            f"token_width {token_width} * merge_kernel_size {merge_kernel_size} > patch_limit_on_one_side {patch_limit_on_one_side}"
+        )
+
+        num_tokens = token_height * token_width
+    return {
+        "num_tokens": num_tokens,
+        "new_width": new_w,
+        "new_height": new_h,
+        "pad_width": pad_width,
+        "pad_height": pad_height,
+        "sampled_nframes": 1,
+    }
+
+
+def _to_pil(
+    data: str | bytes | Image.Image,
+    transparent_bg_config: TransparentBgConfig | None = None,
+    to_rgb: bool = True,
+) -> Image.Image:
+    """Load an image and (optionally) composite its transparent background.
+
+    Args:
+        data: A PIL Image, a base64 ``data:`` URL, a file path, or raw bytes.
+        transparent_bg_config: The config used to fill the transparent
+            background. ``None`` keeps the historical behavior of converting
+            to RGB without compositing.
+        to_rgb: If ``False`` the image is returned as-is (the
+            ``transparent_bg_config`` is ignored). The caller is then
+            expected to call :func:`fill_transparent_bg_with` later — e.g.
+            after a resize.
+    """
+    if isinstance(data, Image.Image):
+        image = data
+    elif isinstance(data, str):
+        if data.startswith("data:"):
+            raw_base64 = data.split(",")[1]
+            image = Image.open(io.BytesIO(base64.b64decode(raw_base64)))
+        else:
+            image = Image.open(data)
+    elif isinstance(data, bytes):
+        image = Image.open(io.BytesIO(data))
+    else:
+        raise ValueError(f"Unsupported data type: {type(data)}")
+
+    if not to_rgb:
+        return image
+
+    return fill_transparent_bg_with(image, transparent_bg_config)
+
+
+def ensure_media_type(
+    media: MediaInput,
+    transparent_bg_config: TransparentBgConfig | None = None,
+    transparent_bg_fill_stage: Literal["before_resize",
+                                       "after_resize"] = "before_resize",
+) -> MediaInput:
+    if media['type'] == 'image':
+        media['image'] = _to_pil(
+            media['image'],
+            transparent_bg_config=transparent_bg_config,
+            to_rgb=transparent_bg_fill_stage == "before_resize",
+        )
+        return media
+    else:
+        raise ValueError(f"Unsupported media type: {media['type']}")
+
+
+def image_to_np(
+    image: Image.Image,
+    resize_to: tuple[int, int] | None = None,
+    mode: str = "resize",
+    raise_error_for_ill_resize: bool = True,
+    transparent_bg_config: TransparentBgConfig | None = None,
+    transparent_bg_fill_stage: Literal["before_resize",
+                                       "after_resize"] = "before_resize",
+) -> np.ndarray:
+    """Convert an image to a numpy array.
+
+    Args:
+        content: The image to convert.
+        resize_to: The size to resize the image to.
+        mode: The mode to resize the image to.
+        raise_error_for_ill_resize: Whether to raise an error for ill-sized resize.
+        transparent_bg_config: The config of the transparent background. Only
+            used when ``transparent_bg_fill_stage == "after_resize"`` (the
+            caller is responsible for filling before resize otherwise).
+        transparent_bg_fill_stage: When to composite the transparent
+            background — before or after the resize step.
+
+    Returns:
+        A numpy array.
+    """
+    assert isinstance(image, Image.Image), "image must be a PIL Image"
+    if resize_to is not None:
+        if mode == "resize":
+            image = image.resize(resize_to, resample=Image.Resampling.BICUBIC)
+            if transparent_bg_fill_stage == "after_resize":
+                image = fill_transparent_bg_with(image, transparent_bg_config)
+
+        elif mode == "rescale_and_pad_to_center":
+            scale = min(resize_to[0] / image.width,
+                        resize_to[1] / image.height, 1.0)
+            new_width = round(image.width * scale)
+            new_height = round(image.height * scale)
+            if new_width == 0 or new_height == 0:
+                if raise_error_for_ill_resize:
+                    raise ValueError(
+                        f"Invalid resize to: {resize_to}, from image size: {image.size}"
+                    )
+                else:
+                    return np.zeros((resize_to[1], resize_to[0], 3),
+                                    dtype=np.uint8)
+
+            image = image.resize((new_width, new_height),
+                                 resample=Image.Resampling.BICUBIC)
+            if transparent_bg_fill_stage == "after_resize":
+                image = fill_transparent_bg_with(image, transparent_bg_config)
+            padding_left = (resize_to[0] - new_width) // 2
+            padding_right = resize_to[0] - new_width - padding_left
+            padding_top = (resize_to[1] - new_height) // 2
+            padding_bottom = resize_to[1] - new_height - padding_top
+            image = np.asarray(image)
+            image = np.pad(
+                image,
+                ((padding_top, padding_bottom), (padding_left, padding_right),
+                 (0, 0)),
+                mode="constant",
+                constant_values=0,
+            )
+            assert image.shape == (resize_to[1], resize_to[0], 3)
+
+        elif mode == "rescale_and_pad_to_rightbottom":
+            scale = min(resize_to[0] / image.width,
+                        resize_to[1] / image.height, 1.0)
+            new_width = round(image.width * scale)
+            new_height = round(image.height * scale)
+            if new_width == 0 or new_height == 0:
+                if raise_error_for_ill_resize:
+                    raise ValueError(
+                        f"Invalid resize to: {resize_to}, from image size: {image.size}"
+                    )
+                else:
+                    return np.zeros((resize_to[1], resize_to[0], 3),
+                                    dtype=np.uint8)
+
+            image = image.resize((new_width, new_height),
+                                 resample=Image.Resampling.BICUBIC)
+            if transparent_bg_fill_stage == "after_resize":
+                image = fill_transparent_bg_with(image, transparent_bg_config)
+            padding_right = resize_to[0] - new_width
+            padding_bottom = resize_to[1] - new_height
+            image = np.asarray(image)
+            image = np.pad(
+                image,
+                ((0, padding_bottom), (0, padding_right), (0, 0)),
+                mode="constant",
+                constant_values=0,
+            )
+            assert image.shape == (resize_to[1], resize_to[0], 3)
+
+        else:
+            raise ValueError(f"Invalid mode: {mode}")
+
+    if isinstance(image, Image.Image):
+        return np.asarray(image)
+    else:
+        return image
+
+
+def navit_patchify(pixel_values: np.ndarray,
+                   patch_size: int) -> dict[str, np.ndarray]:
+    """Reshape the pixel values to a navit shape.
+
+    Args:
+        pixel_values: np.ndarray, shape (t, h, w, c)
+        patch_size: int
+
+    Returns:
+        dict[str, np.ndarray]
+        - patches: np.ndarray, shape (t * h//patch_size * w//patch_size, c, patch_size, patch_size)
+        - grid_thw: np.ndarray, (t, h//patch_size, w//patch_size)
+    """
+    T, H, W, C = pixel_values.shape
+    assert C == 3, "pixel_values must have 3 channels"
+
+    patches = pixel_values.reshape(T, H // patch_size, patch_size,
+                                   W // patch_size, patch_size, C)
+    # (T, H//patch_size, W//patch_size, C, patch_size, patch_size)
+    patches = patches.transpose(0, 1, 3, 5, 2, 4)
+    patches = patches.reshape(-1, C, patch_size, patch_size)
+    grid_thw = np.array([T, H // patch_size, W // patch_size])
+    return {"pixel_values": patches, "grid_thw": grid_thw}
+
+
+def normalize(x: np.ndarray,
+              mean,
+              std_inv,
+              pixels_dtype: np.dtype = np.float32) -> np.ndarray:
+    """Normalize the image.
+
+    Args:
+        x: The image to normalize. The shape is (..., 3). The dtype is uint8. The range is [0, 255].
+        mean: The mean of the image.
+        std_inv: The inverse of the std of the image.
+        pixels_dtype: The dtype of the image.
+    Returns:
+        The normalized image. The shape is (..., 3). The dtype is determined by the pixels_dtype.
+    """
+    x = (x / 255.0).astype(pixels_dtype)
+    x -= mean
+    x *= std_inv
+    return x
+
+
+def _to_tensor(data, **kwargs):
+    import torch
+
+    if isinstance(data, np.ndarray):
+        return torch.from_numpy(data).to(**kwargs)
+    elif isinstance(data, torch.Tensor):
+        return data.to(**kwargs)
+    elif isinstance(data, list):
+        return [_to_tensor(item, **kwargs) for item in data]
+    elif isinstance(data, tuple):
+        return tuple(_to_tensor(item, **kwargs) for item in data)
+    elif isinstance(data, dict):
+        return {k: _to_tensor(v, **kwargs) for k, v in data.items()}
+    elif data is None:
+        return None
+    else:
+        raise ValueError(f"Unsupported data type: {type(data)}")

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3cf09355cde8cf4877cdc76e0a72301d572c1d6263b2831f8a03368285bb65e2
+size 9687419736

modeling_groundinganything.py

@@ -0,0 +1,1799 @@
+import math
+from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Any, Optional, Union
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+from torch.nn import LayerNorm
+
+from transformers import AutoModel
+from transformers.cache_utils import Cache
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import BaseModelOutput, BaseModelOutputWithPooling, ModelOutput
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from transformers.models.siglip.modeling_siglip import SiglipMLP
+from transformers.processing_utils import Unpack
+from transformers.utils import (
+    TransformersKwargs,
+    auto_docstring,
+    can_return_tuple,
+    replace_return_docstrings,
+)
+try:
+    from transformers.utils.generic import is_flash_attention_requested
+except ImportError:
+    def is_flash_attention_requested(config):
+        return getattr(config, "_attn_implementation", None) == "flash_attention_2"
+
+from .configuration_groundinganything import GroundAnythingVLMConfig, GroundAnythingVLMVisionConfig, GroundAnythingConfig
+from .modeling_groundinganything_vision import MoonViT3dPretrainedModel
+
+
+@dataclass
+@auto_docstring(
+    custom_intro="""
+    Base class for GroundAnything-VLM outputs, with hidden states and attentions.
+    """
+)
+class GroundAnythingVLMBaseModelOutputWithPast(ModelOutput):
+    r"""
+    past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
+        It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache).
+
+        Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
+        `past_key_values` input) to speed up sequential decoding.
+    """
+
+    last_hidden_state: Optional[torch.FloatTensor] = None
+    past_key_values: Optional[Cache] = None
+    hidden_states: Optional[tuple[torch.FloatTensor]] = None
+    attentions: Optional[tuple[torch.FloatTensor]] = None
+
+
+@dataclass
+@auto_docstring(
+    custom_intro="""
+    Base class for GroundAnything-VLM causal language model (or autoregressive) outputs.
+    """
+)
+class GroundAnythingVLMCausalLMOutputWithPast(ModelOutput):
+    r"""
+    loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided):
+        Language modeling loss (for next-token prediction).
+    logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`):
+        Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
+    past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
+        It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache).
+
+        Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
+        `past_key_values` input) to speed up sequential decoding.
+    """
+
+    loss: Optional[torch.FloatTensor] = None
+    logits: Optional[torch.FloatTensor] = None
+    past_key_values: Optional[Cache] = None
+    hidden_states: Optional[tuple[torch.FloatTensor]] = None
+    attentions: Optional[tuple[torch.FloatTensor]] = None
+
+
+# ---------------------------------------------------------------------------
+# Vision Rotary Embedding
+# ---------------------------------------------------------------------------
+
+
+class VisionRotaryEmbedding(nn.Module):
+    """
+    3D (T,H,W) Rotary frequency constructor with 4:6:6 split.
+    Supports both grid_thw-based and explicit position-based RoPE computation.
+    """
+
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__()
+        head_dim = config.hidden_size // config.num_attention_heads
+        base = config.rope_theta
+
+        assert head_dim % 2 == 0, "head_dim must be even for rotary."
+        assert head_dim % 16 == 0, "head_dim must be divisible by 16."
+        half = head_dim // 2
+        assert half % 16 == 0, "head_dim//2 must also be divisible by 16 to split into 4:6:6."
+
+        self.head_dim = head_dim
+        self.half = half
+        self.base = base
+
+        # 4:6:6 split for T:H:W
+        unit = half // 16
+        self.t_size = 4 * unit
+        self.h_size = 6 * unit
+        self.w_size = 6 * unit
+
+        self.register_buffer(
+            "inv_freq_t",
+            1.0 / (base ** (torch.arange(self.t_size, dtype=torch.float32) / self.t_size)),
+            persistent=False,
+        )
+        self.register_buffer(
+            "inv_freq_h",
+            1.0 / (base ** (torch.arange(self.h_size, dtype=torch.float32) / self.h_size)),
+            persistent=False,
+        )
+        self.register_buffer(
+            "inv_freq_w",
+            1.0 / (base ** (torch.arange(self.w_size, dtype=torch.float32) / self.w_size)),
+            persistent=False,
+        )
+
+    def forward(self, grid_thw: torch.Tensor) -> torch.Tensor:
+        """
+        Compute rotary position embeddings from grid_thw (Qwen2VL style).
+
+        Args:
+            grid_thw: [num_samples, 3] tensor with [t, h, w] for each sample
+
+        Returns:
+            freqs: [total_seq_len, half] tensor of position frequencies
+        """
+        device = grid_thw.device
+        inv_t = self.inv_freq_t.to(device=device)
+        inv_h = self.inv_freq_h.to(device=device)
+        inv_w = self.inv_freq_w.to(device=device)
+
+        all_freqs = []
+        for sample_thw in grid_thw:
+            t, h, w = sample_thw[0].item(), sample_thw[1].item(), sample_thw[2].item()
+
+            # Compute frequency tables
+            ft = torch.outer(torch.arange(t, device=device, dtype=torch.float32), inv_t)
+            fh = torch.outer(torch.arange(h, device=device, dtype=torch.float32), inv_h)
+            fw = torch.outer(torch.arange(w, device=device, dtype=torch.float32), inv_w)
+
+            # Build position indices for this sample
+            t_ids = torch.arange(t, device=device).repeat_interleave(h * w)
+            h_ids = torch.arange(h, device=device).repeat_interleave(w).repeat(t)
+            w_ids = torch.arange(w, device=device).repeat(h).repeat(t)
+
+            # Concatenate frequencies: [seq_len, half]
+            sample_freqs = torch.cat([ft[t_ids], fh[h_ids], fw[w_ids]], dim=-1)
+            all_freqs.append(sample_freqs)
+
+        return torch.cat(all_freqs, dim=0)
+
+    def forward_from_positions(self, patch_positions: torch.Tensor) -> torch.Tensor:
+        """
+        Compute rotary position embeddings from explicit patch positions.
+
+        Args:
+            patch_positions: [seq_len, 3] tensor with [t, h, w] positions for each patch
+
+        Returns:
+            freqs: [seq_len, half] tensor of position frequencies
+        """
+        device = patch_positions.device
+        inv_t = self.inv_freq_t.to(device=device)
+        inv_h = self.inv_freq_h.to(device=device)
+        inv_w = self.inv_freq_w.to(device=device)
+
+        t_pos = patch_positions[:, 0].float()
+        h_pos = patch_positions[:, 1].float()
+        w_pos = patch_positions[:, 2].float()
+
+        ft = torch.outer(t_pos, inv_t)
+        fh = torch.outer(h_pos, inv_h)
+        fw = torch.outer(w_pos, inv_w)
+
+        return torch.cat([ft, fh, fw], dim=-1)
+
+    def forward_with_thw(self, t: int, h: int, w: int, device=None) -> torch.Tensor:
+        """
+        Compute rotary position embeddings from explicit t, h, w dimensions.
+
+        Args:
+            t: Number of temporal frames
+            h: Number of height patches
+            w: Number of width patches
+            device: Target device
+
+        Returns:
+            freqs: [t*h*w, half] tensor of position frequencies
+        """
+        if device is None:
+            device = self.inv_freq_t.device
+
+        inv_t = self.inv_freq_t.to(device=device)
+        inv_h = self.inv_freq_h.to(device=device)
+        inv_w = self.inv_freq_w.to(device=device)
+
+        ft = torch.outer(torch.arange(t, device=device, dtype=torch.float32), inv_t)
+        fh = torch.outer(torch.arange(h, device=device, dtype=torch.float32), inv_h)
+        fw = torch.outer(torch.arange(w, device=device, dtype=torch.float32), inv_w)
+
+        t_ids = torch.arange(t, device=device).repeat_interleave(h * w)
+        h_ids = torch.arange(h, device=device).repeat_interleave(w).repeat(t)
+        w_ids = torch.arange(w, device=device).repeat(h).repeat(t)
+
+        freqs = torch.cat([ft[t_ids], fh[h_ids], fw[w_ids]], dim=-1)
+        return freqs
+
+
+# ---------------------------------------------------------------------------
+# Patch Embedding
+# ---------------------------------------------------------------------------
+
+
+class GroundAnythingVLMVisionEmbeddings(nn.Module):
+    """
+    Patch embedding layer that converts pre-processed patches to embeddings.
+
+    This module is designed to receive patches that have already been extracted
+    and arranged by the Qwen2VL image processor in 2x2 block spatial order.
+
+    Input format: [total_patches, num_channels, patch_size, patch_size]
+    Output format: [total_patches, embed_dim]
+    """
+
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.image_size = config.image_size
+        self.patch_size = config.patch_size
+        self.in_channels = config.num_channels
+
+        self.patch_embedding = nn.Conv2d(
+            in_channels=config.num_channels,
+            out_channels=self.embed_dim,
+            kernel_size=self.patch_size,
+            stride=self.patch_size,
+            bias=False,
+        )
+
+    def forward(self, hidden_states: torch.FloatTensor) -> torch.Tensor:
+        target_dtype = self.patch_embedding.weight.dtype
+        hidden_states = hidden_states.view(-1, self.in_channels, self.patch_size, self.patch_size)
+        hidden_states = self.patch_embedding(hidden_states.to(dtype=target_dtype)).view(-1, self.embed_dim)
+
+        return hidden_states
+
+
+# ---------------------------------------------------------------------------
+# Patch Merger
+# ---------------------------------------------------------------------------
+
+
+class GroundAnythingVLMVisionPatchMerger(nn.Module):
+    """
+    Patch merger that merges spatial_merge_size x spatial_merge_size patches into one.
+
+    This module is designed to work with Qwen2VL-style patch processing where patches
+    are already arranged in 2x2 block order by the image processor.
+    """
+
+    def __init__(
+        self,
+        dim: int,
+        context_dim: int,
+        spatial_merge_size: int = 2,
+        layer_norm_eps: float = 1e-05,
+        use_patch_position_encoding: bool = False,
+        patch_position_encoding_type: str = "absolute",
+        max_position_embeddings: int = 8192,
+    ) -> None:
+        super().__init__()
+        self.hidden_size = context_dim * (spatial_merge_size**2)
+        self.ln_q = LayerNorm(context_dim, eps=layer_norm_eps)
+        self.mlp = nn.Sequential(
+            nn.Linear(self.hidden_size, self.hidden_size),
+            nn.GELU(),
+            nn.Linear(self.hidden_size, dim),
+        )
+        self.spatial_merge_size = spatial_merge_size
+        self.use_patch_position_encoding = use_patch_position_encoding
+        self.patch_position_encoding_type = patch_position_encoding_type
+
+        if self.use_patch_position_encoding:
+            if self.patch_position_encoding_type != "absolute":
+                raise ValueError(
+                    f"Unknown patch_position_encoding_type: {self.patch_position_encoding_type}. "
+                    "Only 'absolute' is supported."
+                )
+            self.pos_emb_h = nn.Embedding(max_position_embeddings, dim)
+            self.pos_emb_w = nn.Embedding(max_position_embeddings, dim)
+
+    def forward(self, x: torch.Tensor, patch_positions: Optional[torch.Tensor] = None) -> torch.Tensor:
+        """
+        Merge patches from Qwen2VL-style input.
+
+        The input patches are already arranged in 2x2 block order by the image processor,
+        so we simply need to apply LayerNorm, reshape, and project through MLP.
+
+        Args:
+            x: Input tensor of shape [batch_size, seq_len, hidden_size] or [seq_len, hidden_size]
+               where seq_len = t * h * w (patches in 2x2 block order)
+
+        Returns:
+            Merged tensor of shape [batch_size, seq_len // spatial_merge_size^2, dim]
+            or [seq_len // spatial_merge_size^2, dim]
+        """
+        if patch_positions is not None and patch_positions.dim() == 3:
+            patch_positions = patch_positions.squeeze(0)
+
+        x = self.ln_q(x).view(-1, self.hidden_size)
+        x = self.mlp(x)
+
+        if self.use_patch_position_encoding and patch_positions is not None:
+            pp = patch_positions.view(-1, self.spatial_merge_size**2, 3)
+            pp = pp[:, 0, :]
+            pp = (pp // self.spatial_merge_size).long()
+
+            x = x + self.pos_emb_h(pp[:, 1]) + self.pos_emb_w(pp[:, 2])
+
+        return x
+
+
+def rotate_half(x):
+    """
+    Interleaved rotation to match Source model's implementation.
+    (x1, x2, x3, x4) -> (-x2, x1, -x4, x3)
+    """
+    x_even = x[..., ::2]
+    x_odd = x[..., 1::2]
+    return torch.stack((-x_odd, x_even), dim=-1).flatten(-2)
+
+
+def get_norm_layer(config):
+    if config.layer_norm_type == "rms_norm":
+        return nn.RMSNorm(config.hidden_size, eps=config.layer_norm_eps)
+    else:
+        return nn.LayerNorm(config.hidden_size, eps=config.layer_norm_eps)
+
+
+def apply_rotary_pos_emb(q, k, freqs):
+    # q, k: (B, H, L, D)
+    # freqs: (B, L, D)
+    orig_q_dtype = q.dtype
+    orig_k_dtype = k.dtype
+    q, k = q.float(), k.float()
+    # We need to broadcast freqs to match heads
+    # (B, L, D) -> (B, 1, L, D)
+    # Keep the same dtype as q, k to avoid memory doubling from float32 promotion
+    cos = freqs.cos().unsqueeze(1).float()
+    sin = freqs.sin().unsqueeze(1).float()
+
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    q_embed = q_embed.to(orig_q_dtype)
+    k_embed = k_embed.to(orig_k_dtype)
+    return q_embed, k_embed
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: Optional[torch.Tensor],
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs,
+):
+    """Eager attention; query/key/value are expected as ``(B, H, L, D)``."""
+    attn_weights = torch.matmul(query, key.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        attn_weights = attn_weights + attention_mask
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+    attn_output = torch.matmul(attn_weights, value)
+    attn_output = attn_output.transpose(1, 2).contiguous()  # (B, L, H, D)
+    return attn_output, attn_weights
+
+
+class GroundAnythingVLMVisionAttention(nn.Module):
+    """
+    Multi-headed attention with RoPE support, dispatched through
+    :data:`ALL_ATTENTION_FUNCTIONS` (``eager`` / ``sdpa`` / ``flash_attention_2``)
+    based on ``config._attn_implementation``.
+    """
+
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = self.embed_dim // self.num_heads
+        if self.head_dim * self.num_heads != self.embed_dim:
+            raise ValueError(
+                f"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim} and `num_heads`: {self.num_heads})."
+            )
+
+        self.num_key_value_groups = 1  # required by repeat_kv-aware eager paths
+        self.scale = self.head_dim**-0.5
+        self.scaling = self.scale  # alias expected by some attention interfaces
+        self.attention_dropout = config.attention_dropout
+        self.is_causal = False
+        self.qkv = nn.Linear(self.embed_dim, self.embed_dim * 3)
+        self.proj = nn.Linear(self.embed_dim, self.embed_dim)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        rotary_pos_emb: Optional[torch.Tensor] = None,
+        output_attentions: bool = False,
+        cu_seqlens: Optional[torch.Tensor] = None,
+        max_seqlen: Optional[int] = None,
+        **kwargs,
+    ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
+        batch_size, q_len, _ = hidden_states.size()
+        # (B, L, 3*H*D) -> (B, L, 3, H, D) -> 3 x (B, L, H, D) -> 3 x (B, H, L, D)
+        q, k, v = (
+            self.qkv(hidden_states)
+            .reshape(batch_size, q_len, 3, self.num_heads, self.head_dim)
+            .permute(2, 0, 1, 3, 4)
+            .unbind(0)
+        )
+        query_states = q.transpose(1, 2)
+        key_states = k.transpose(1, 2)
+        value_states = v.transpose(1, 2)
+
+        if rotary_pos_emb is not None:
+            query_states, key_states = apply_rotary_pos_emb(query_states, key_states, rotary_pos_emb)
+
+        attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
+            self.config._attn_implementation, eager_attention_forward
+        )
+        dropout = 0.0 if not self.training else self.attention_dropout
+
+        if cu_seqlens is not None and is_flash_attention_requested(self.config):
+            # Flash Attention varlen path: pass cu_seq_lens / max_length kwargs.
+            if max_seqlen is None:
+                max_seqlen = (cu_seqlens[1:] - cu_seqlens[:-1]).max()
+            attn_output, _ = attention_interface(
+                self,
+                query_states,
+                key_states,
+                value_states,
+                attention_mask=None,
+                scaling=self.scale,
+                dropout=dropout,
+                cu_seq_lens_q=cu_seqlens,
+                cu_seq_lens_k=cu_seqlens,
+                max_length_q=max_seqlen,
+                max_length_k=max_seqlen,
+                is_causal=False,
+                **kwargs,
+            )
+        elif cu_seqlens is not None:
+            # Non-FA implementations do not understand cu_seqlens directly; mirror
+            # Qwen3-VL by splitting the packed sequence into per-sample chunks
+            # along the L dim of (B, H, L, D) and running attention per chunk.
+            lengths = (cu_seqlens[1:] - cu_seqlens[:-1]).tolist()
+            splits = [torch.split(t, lengths, dim=2) for t in (query_states, key_states, value_states)]
+            attn_outputs = [
+                attention_interface(
+                    self,
+                    q_chunk,
+                    k_chunk,
+                    v_chunk,
+                    attention_mask=None,
+                    scaling=self.scale,
+                    dropout=dropout,
+                    is_causal=False,
+                    **kwargs,
+                )[0]
+                for q_chunk, k_chunk, v_chunk in zip(*splits)
+            ]
+            # interface output is (B, l_i, H, D); concat along the L axis
+            attn_output = torch.cat(attn_outputs, dim=1)
+        else:
+            attn_mask = None
+            if attention_mask is not None:
+                attn_mask = attention_mask
+                if attn_mask.dim() == 2:
+                    attn_mask = attn_mask.unsqueeze(0)
+                if attn_mask.shape[0] == 1 and batch_size > 1:
+                    attn_mask = attn_mask.expand(batch_size, -1, -1)
+                attn_mask = attn_mask.unsqueeze(1)  # (B, 1, L, L)
+            attn_output, _ = attention_interface(
+                self,
+                query_states,
+                key_states,
+                value_states,
+                attention_mask=attn_mask,
+                scaling=self.scale,
+                dropout=dropout,
+                is_causal=False,
+                **kwargs,
+            )
+
+        attn_output = attn_output.reshape(batch_size, q_len, self.embed_dim)
+        attn_output = self.proj(attn_output)
+
+        return attn_output, None
+
+
+class GroundAnythingVLMVisionEncoderLayer(nn.Module):
+    """Vision encoder layer with pre-norm and Flash Attention 2."""
+
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__()
+        self.embed_dim = config.hidden_size
+        self.self_attn = GroundAnythingVLMVisionAttention(config)
+        self.layer_norm1 = get_norm_layer(config)
+        self.mlp = SiglipMLP(config)
+        self.layer_norm2 = get_norm_layer(config)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        rotary_pos_emb: Optional[torch.Tensor] = None,
+        output_attentions: bool = False,
+        cu_seqlens: Optional[torch.Tensor] = None,
+        max_seqlen: Optional[int] = None,
+    ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
+        residual = hidden_states
+        hidden_states = self.layer_norm1(hidden_states)
+
+        hidden_states, attn_weights = self.self_attn(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            rotary_pos_emb=rotary_pos_emb,
+            output_attentions=output_attentions,
+            cu_seqlens=cu_seqlens,
+            max_seqlen=max_seqlen,
+        )
+        hidden_states = residual + hidden_states
+
+        residual = hidden_states
+        hidden_states = self.layer_norm2(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+
+        outputs = (hidden_states, attn_weights) if output_attentions else (hidden_states,)
+        return outputs
+
+
+class GroundAnythingVLMVisionEncoder(nn.Module):
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__()
+        self.config = config
+        self.layers = nn.ModuleList([GroundAnythingVLMVisionEncoderLayer(config) for _ in range(config.num_hidden_layers)])
+        # Gradient checkpointing support
+        self.gradient_checkpointing = False
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        rotary_pos_emb: Optional[torch.Tensor] = None,
+        output_attentions: bool = False,
+        output_hidden_states: bool = False,
+        return_dict: bool = True,
+        cu_seqlens: Optional[torch.Tensor] = None,
+        max_seqlen: Optional[int] = None,
+    ) -> Union[tuple, BaseModelOutput]:
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attentions = () if output_attentions else None
+
+        for layer in self.layers:
+            if output_hidden_states:
+                all_hidden_states = all_hidden_states + (hidden_states,)
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    layer.__call__,
+                    hidden_states,
+                    attention_mask,
+                    rotary_pos_emb,
+                    output_attentions,
+                    cu_seqlens,
+                    max_seqlen,
+                )
+            else:
+                layer_outputs = layer(
+                    hidden_states,
+                    attention_mask=attention_mask,
+                    rotary_pos_emb=rotary_pos_emb,
+                    output_attentions=output_attentions,
+                    cu_seqlens=cu_seqlens,
+                    max_seqlen=max_seqlen,
+                )
+
+            hidden_states = layer_outputs[0]
+
+            if output_attentions:
+                all_self_attentions = all_self_attentions + (layer_outputs[1],)
+
+        if output_hidden_states:
+            all_hidden_states = all_hidden_states + (hidden_states,)
+
+        if not return_dict:
+            return tuple(v for v in [hidden_states, all_hidden_states, all_self_attentions] if v is not None)
+
+        return BaseModelOutput(
+            last_hidden_state=hidden_states,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attentions,
+        )
+
+
+class GroundAnythingVLMPreTrainedModel(PreTrainedModel):
+    _supports_attention_backend = True
+    config_class = GroundAnythingVLMConfig
+    base_model_prefix = "model"
+    input_modalities = ("image", "video", "text")
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["MoonViTEncoderLayer", "Qwen3DecoderLayer"]
+    _skip_keys_device_placement = "past_key_values"
+    _supports_flash_attn = True
+    _supports_sdpa = True
+
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        # Re-initialize VisionRotaryEmbedding inv_freq buffers.
+        # These are registered with persistent=False, so they are not in the checkpoint
+        # state_dict. When ``from_pretrained`` materializes the model from meta tensors,
+        # the values in these buffers end up uninitialized. Mirror Qwen3-VL by explicitly
+        # filling them here so RoPE produces the correct frequencies post-load.
+        if isinstance(module, VisionRotaryEmbedding):
+            base = module.base
+            with torch.no_grad():
+                inv_t = 1.0 / (base ** (torch.arange(module.t_size, dtype=torch.float32) / module.t_size))
+                inv_h = 1.0 / (base ** (torch.arange(module.h_size, dtype=torch.float32) / module.h_size))
+                inv_w = 1.0 / (base ** (torch.arange(module.w_size, dtype=torch.float32) / module.w_size))
+                module.inv_freq_t.copy_(inv_t.to(module.inv_freq_t.device))
+                module.inv_freq_h.copy_(inv_h.to(module.inv_freq_h.device))
+                module.inv_freq_w.copy_(inv_w.to(module.inv_freq_w.device))
+
+
+class Siglip2MultiheadAttentionPoolingHead(nn.Module):
+    """
+    Multi-Head Attention Pooling with a learned probe (PMA-style).
+    """
+
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__()
+        self.embed_dim = config.hidden_size
+        self.probe = nn.Parameter(torch.randn(1, 1, config.hidden_size))
+        self.attention = nn.MultiheadAttention(config.hidden_size, config.num_attention_heads, batch_first=True)
+        self.norm = nn.RMSNorm(config.hidden_size, eps=config.layer_norm_eps)
+        self.mlp = SiglipMLP(config)
+
+    def forward(self, hidden_states):
+        batch_size = hidden_states.shape[0]
+        probe = self.probe.repeat(batch_size, 1, 1)
+
+        attn_output, _ = self.attention(probe, hidden_states, hidden_states)
+
+        residual = attn_output
+        attn_output = self.norm(attn_output)
+        attn_output = residual + self.mlp(attn_output)
+
+        return attn_output[:, 0]
+
+
+# ---------------------------------------------------------------------------
+# Vision Model
+# ---------------------------------------------------------------------------
+
+
+class GroundAnythingVLMVisionPretrainedModel(GroundAnythingVLMPreTrainedModel):
+    """
+    GroundAnything-VLM Vision Model.
+
+    This vision model is designed to work with Qwen2VL-style image processing:
+        - Receives pre-processed patches in 2x2 block spatial order
+        - Applies RoPE with matching 2x2 block layout conversion
+        - Accepts explicit patch_positions for RoPE computation
+
+    Input format:
+        hidden_state: [total_patches, num_channels, patch_size, patch_size]
+        grid_thw: [num_samples, 3] with [t, h, w] for each sample
+    """
+
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__(config)
+        self.config = config
+        self.spatial_merge_size = config.spatial_merge_size
+
+        # Vision components
+        self.embeddings = GroundAnythingVLMVisionEmbeddings(config)
+        self.layernorm_pre = get_norm_layer(config)
+        self.encoder = GroundAnythingVLMVisionEncoder(config)
+        self.video_rope = VisionRotaryEmbedding(config)
+
+        if config.use_head:
+            self.layernorm_post = get_norm_layer(config)
+            self.head = Siglip2MultiheadAttentionPoolingHead(config)
+        else:
+            self.layernorm_post = None
+            self.head = None
+
+        self.merger = GroundAnythingVLMVisionPatchMerger(
+            dim=config.out_hidden_size,
+            context_dim=config.hidden_size,
+            spatial_merge_size=config.spatial_merge_size,
+            layer_norm_eps=config.layer_norm_eps,
+            use_patch_position_encoding=getattr(config, "use_patch_position_encoding", False),
+            patch_position_encoding_type=getattr(config, "patch_position_encoding_type", "absolute"),
+            max_position_embeddings=getattr(config, "max_position_embeddings", 8192),
+        )
+
+        self.post_init()
+
+    def _build_cu_seqlens(
+        self,
+        grid_thw: torch.Tensor,
+        total_patches: int,
+        fixed_t: Optional[int] = 4,
+        device: Optional[torch.device] = None,
+    ) -> tuple[torch.Tensor, int]:
+        if grid_thw is None or grid_thw.numel() == 0:
+            # Fallback for no grid_thw: treat as single sequence
+            return torch.tensor([0, total_patches], dtype=torch.int32, device=device), total_patches
+
+        if device is None:
+            device = grid_thw.device
+
+        cu_seqlens = [0]
+        max_seqlen = 0
+        total_entries = grid_thw.shape[0]
+        current_len = 0
+
+        # Calculate cumulative lengths: split sequences based on fixed_t if provided
+        for idx in range(total_entries):
+            t_val = grid_thw[idx, 0].item()
+            h_val = grid_thw[idx, 1].item()
+            w_val = grid_thw[idx, 2].item()
+
+            if fixed_t is not None and fixed_t > 0 and t_val > fixed_t:
+                # Split large t into chunks of fixed_t
+                num_full_windows = t_val // fixed_t
+                remainder = t_val % fixed_t
+
+                # Add full windows
+                for _ in range(num_full_windows):
+                    chunk_patches = fixed_t * int(h_val) * int(w_val)
+                    current_len += chunk_patches
+                    max_seqlen = max(max_seqlen, chunk_patches)
+                    cu_seqlens.append(current_len)
+
+                # Add remainder if any
+                if remainder > 0:
+                    chunk_patches = remainder * int(h_val) * int(w_val)
+                    current_len += chunk_patches
+                    max_seqlen = max(max_seqlen, chunk_patches)
+                    cu_seqlens.append(current_len)
+            else:
+                # Standard case: add as one chunk
+                chunk_patches = t_val * int(h_val) * int(w_val)
+                current_len += chunk_patches
+                max_seqlen = max(max_seqlen, chunk_patches)
+                cu_seqlens.append(current_len)
+
+        last_len = cu_seqlens[-1]
+        if last_len != total_patches:
+            raise ValueError(
+                "cu_seqlens calculation mismatch:\n"
+                f"- total_patches: {total_patches}\n"
+                f"- calculated total: {last_len}\n"
+                f"- grid_thw: {grid_thw}"
+            )
+
+        return torch.tensor(cu_seqlens, dtype=torch.int32, device=device), max_seqlen
+
+    def _build_block_attention_mask(
+        self,
+        grid_thw: torch.Tensor,
+        total_patches: int,
+        fixed_t: Optional[int] = 4,
+        device: Optional[torch.device] = None,
+    ) -> Optional[torch.Tensor]:
+        if grid_thw is None or grid_thw.numel() == 0:
+            return None
+
+        if device is None:
+            device = grid_thw.device
+
+        lengths = []
+        total_entries = grid_thw.shape[0]
+
+        for idx in range(total_entries):
+            t_val = grid_thw[idx, 0].item()
+            h_val = grid_thw[idx, 1].item()
+            w_val = grid_thw[idx, 2].item()
+
+            if fixed_t is not None and fixed_t > 0 and t_val > fixed_t:
+                # Split large t into chunks of fixed_t
+                num_full_windows = t_val // fixed_t
+                remainder = t_val % fixed_t
+
+                # Add full windows
+                for _ in range(num_full_windows):
+                    lengths.append(fixed_t * int(h_val) * int(w_val))
+
+                # Add remainder if any
+                if remainder > 0:
+                    lengths.append(remainder * int(h_val) * int(w_val))
+            else:
+                lengths.append(t_val * int(h_val) * int(w_val))
+
+        total_len = sum(lengths)
+        if total_len != total_patches:
+            raise ValueError(
+                "Block attention mask length mismatch:\n"
+                f"- total_patches: {total_patches}\n"
+                f"- total_len: {total_len}\n"
+                f"- grid_thw: {grid_thw}"
+            )
+
+        attn_mask = torch.ones((total_len, total_len), dtype=torch.bool, device=device)
+        start = 0
+        for size in lengths:
+            end = start + size
+            attn_mask[start:end, start:end] = False
+            start = end
+
+        return attn_mask
+
+    @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=GroundAnythingVLMVisionConfig)
+    def forward(
+        self,
+        hidden_state: torch.Tensor,
+        grid_thw: Optional[torch.Tensor] = None,
+        patch_positions: Optional[torch.Tensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        skip_merger: Optional[bool] = False,
+    ) -> Union[tuple, BaseModelOutputWithPooling]:
+        r"""
+        Forward pass for vision model.
+
+        This method accepts pre-processed patches from Qwen2VL image processor and applies
+        RoPE (Rotary Position Embedding) in 2x2 block layout to match the spatial arrangement
+        of patches.
+
+        Args:
+            hidden_state: Pre-processed patches from Qwen2VL processor.
+                Shape: [total_patches, num_channels, patch_size, patch_size]
+            grid_thw: Grid sizes tensor of shape [num_samples, 3] with [t, h, w] for each sample.
+                Required for computing RoPE and handling visible indices.
+            patch_positions: Optional explicit patch positions for RoPE computation.
+            output_attentions: Whether to return attention weights.
+            output_hidden_states: Whether to return all hidden states.
+            return_dict: Whether to return a ModelOutput instead of tuple.
+            skip_merger: If True, skip patch merger (useful for consistency checking).
+
+        Returns:
+            BaseModelOutputWithPooling with last_hidden_state containing merged features.
+        """
+        output_attentions = (
+            output_attentions if output_attentions is not None else getattr(self.config, "output_attentions", False)
+        )
+        output_hidden_states = (
+            output_hidden_states
+            if output_hidden_states is not None
+            else getattr(self.config, "output_hidden_states", False)
+        )
+        return_dict = True if return_dict is None else return_dict
+
+        # 1. Embeddings
+        # Note: embeddings returns [total_patches, embed_dim], we need to add batch dimension
+        hidden_states = self.embeddings(hidden_state)
+        if hidden_states.dim() == 2:
+            hidden_states = hidden_states.unsqueeze(0)  # [1, total_patches, embed_dim]
+        batch_size, total_patches, _ = hidden_states.shape
+
+        # 2. RoPE Construction
+        if patch_positions is not None and patch_positions.dim() == 3:
+            patch_positions = patch_positions.squeeze(0)
+        freqs_visible = self.video_rope.forward_from_positions(patch_positions)
+
+        # Concatenate D/2 + D/2 -> D for applying rope
+        freqs_visible = torch.cat([freqs_visible, freqs_visible], dim=-1)
+        if freqs_visible.dim() == 2:
+            freqs_visible = freqs_visible.unsqueeze(0)
+
+        # 3. Pre-Norm & Encoder
+        hidden_states = self.layernorm_pre(hidden_states)
+
+        cu_seqlens, max_seqlen = self._build_cu_seqlens(
+            grid_thw=grid_thw,
+            total_patches=total_patches,
+            fixed_t=getattr(self.config, "frame_windows_size", 4),
+            device=hidden_states.device,
+        )
+
+        encoder_outputs = self.encoder(
+            hidden_states,
+            attention_mask=None,
+            rotary_pos_emb=freqs_visible,
+            output_attentions=output_attentions,
+            output_hidden_states=True,  # Always get hidden states to use -2 layer
+            return_dict=True,
+            cu_seqlens=cu_seqlens,
+            max_seqlen=max_seqlen,
+        )
+
+        # Use second-to-last layer output for better feature representation
+        if encoder_outputs.hidden_states is not None and len(encoder_outputs.hidden_states) >= 2 and not skip_merger:
+            sequence_output = encoder_outputs.hidden_states[-1]
+        else:
+            sequence_output = encoder_outputs[0]
+
+        # Post-Norm
+        if self.layernorm_post is not None:
+            sequence_output = self.layernorm_post(sequence_output)
+
+        # Skip merger for consistency check with original ViT
+        if skip_merger:
+            pooled_output = None
+            if self.head is not None:
+                pooled_output = self.head(sequence_output)
+
+            if not return_dict:
+                return (sequence_output, pooled_output) + (
+                    encoder_outputs.hidden_states if output_hidden_states else None,
+                )
+            return BaseModelOutputWithPooling(
+                last_hidden_state=sequence_output,
+                pooler_output=pooled_output,
+                hidden_states=encoder_outputs.hidden_states if output_hidden_states else None,
+                attentions=encoder_outputs.attentions if output_attentions else None,
+            )
+
+        # Patch merger: input patches are already in 2x2 block order from Qwen2VL processor
+        merged_output = self.merger(sequence_output, patch_positions=patch_positions)
+
+        if not return_dict:
+            return (merged_output,) + (encoder_outputs.hidden_states if output_hidden_states else None,)
+
+        return BaseModelOutputWithPooling(
+            last_hidden_state=merged_output,
+            pooler_output=None,
+            hidden_states=encoder_outputs.hidden_states if output_hidden_states else None,
+            attentions=encoder_outputs.attentions if output_attentions else None,
+        )
+
+
+class GroundAnythingVLMTwoLayerProjector(nn.Module):
+    """K3-native 2x2 patch merger followed by a two-layer MLP."""
+
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__()
+        merge_area = int(config.spatial_merge_size) ** 2
+        input_size = int(config.hidden_size) * merge_area
+        hidden_size = int(config.projector_hidden_size)
+        output_size = int(config.out_hidden_size)
+        self.input_size = input_size
+        self.pre_norm = nn.LayerNorm(config.hidden_size, eps=config.projector_ln_eps)
+        self.fc1 = nn.Linear(input_size, hidden_size, bias=False)
+        self.act = nn.GELU()
+        self.fc2 = nn.Linear(hidden_size, output_size, bias=False)
+        self.post_norm = nn.RMSNorm(output_size, eps=config.projector_ln_eps)
+        for layer in (self.fc1, self.fc2):
+            nn.init.trunc_normal_(layer.weight, std=math.sqrt(2.0 / layer.in_features))
+
+    def forward(self, features):
+        outputs = []
+        for item in features:
+            if item.ndim != 3 or item.shape[1] * item.shape[2] != self.input_size:
+                raise ValueError(
+                    f"Expected K3 merged features [tokens, 4, 1024], got {tuple(item.shape)}"
+                )
+            item = self.pre_norm(item).reshape(item.shape[0], self.input_size)
+            outputs.append(self.post_norm(self.fc2(self.act(self.fc1(item)))))
+        return outputs
+
+
+class GroundAnythingVLMVisionModel(nn.Module):
+    """MoonViT3D plus the freshly initialized GroundAnything projection connector."""
+
+    def __init__(self, config: GroundAnythingVLMVisionConfig):
+        super().__init__()
+        self.config = config
+        self.spatial_merge_size = int(config.spatial_merge_size)
+        self.vision_tower = MoonViT3dPretrainedModel(config)
+        self.projector = GroundAnythingVLMTwoLayerProjector(config)
+
+    def forward(self, pixel_values, grid_thw=None, patch_positions=None, **kwargs):
+        del patch_positions, kwargs
+        if grid_thw is None:
+            raise ValueError("image_grid_thw is required for K3 MoonViT3D")
+        target_dtype = self.vision_tower.patch_embed.proj.weight.dtype
+        features = self.vision_tower(pixel_values.to(dtype=target_dtype), grid_thw)
+        projected = self.projector(features)
+        merged = torch.cat(projected, dim=0)
+        return BaseModelOutputWithPooling(last_hidden_state=merged)
+
+
+@auto_docstring
+class GroundAnythingVLMBaseModel(GroundAnythingVLMPreTrainedModel):
+    base_model_prefix = ""
+    # Reference: fix gemma3 grad acc #37208
+    accepts_loss_kwargs = False
+    config: GroundAnythingVLMConfig
+    _no_split_modules = ["MoonViTEncoderLayer", "Qwen3DecoderLayer"]
+
+    def __init__(self, config: GroundAnythingVLMConfig):
+        super().__init__(config)
+        self.visual = GroundAnythingVLMVisionModel(config.vision_config)
+        self.language_model = AutoModel.from_config(config.text_config)
+        self.streammind_gate = None
+        self._streammind_model_path = None
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def _load_streammind_gate(self):
+        if self.streammind_gate is not None:
+            return self.streammind_gate
+        from safetensors.torch import load_file
+        from .streammind_gate import StreamMindGate
+
+        if not self._streammind_model_path:
+            raise RuntimeError(
+                "StreamMind gate path is unavailable. Load the model with "
+                "GroundAnythingVLMStreamMindForConditionalGeneration.from_pretrained()."
+            )
+        gate = StreamMindGate(self.config.text_config.hidden_size)
+        gate_path = Path(self._streammind_model_path) / "streammind_gate.safetensors"
+        if not gate_path.is_file():
+            from huggingface_hub import hf_hub_download
+
+            gate_path = Path(
+                hf_hub_download(
+                    repo_id=self._streammind_model_path,
+                    filename="streammind_gate.safetensors",
+                )
+            )
+        state = load_file(str(gate_path))
+        gate.load_state_dict(state, strict=True)
+        gate.to(
+            device=next(self.visual.parameters()).device,
+            dtype=next(self.visual.parameters()).dtype,
+        ).eval()
+        self.streammind_gate = gate
+        return gate
+
+    def _streammind_vision_tokens(self, pixel_values, image_grid_thw, patch_positions=None):
+        pixel_values = pixel_values.type(self.visual.vision_tower.patch_embed.proj.weight.dtype)
+        rope = self.visual.video_rope
+        saved_rope = (rope.inv_freq_t, rope.inv_freq_h, rope.inv_freq_w)
+        try:
+            # The StreamMind checkpoint was trained after the whole OV model,
+            # including non-persistent RoPE buffers, was cast to BF16.
+            rope.inv_freq_t = rope.inv_freq_t.to(pixel_values.dtype)
+            rope.inv_freq_h = rope.inv_freq_h.to(pixel_values.dtype)
+            rope.inv_freq_w = rope.inv_freq_w.to(pixel_values.dtype)
+            vision_output = self.visual(
+                pixel_values,
+                grid_thw=image_grid_thw,
+                patch_positions=patch_positions,
+            )
+        finally:
+            rope.inv_freq_t, rope.inv_freq_h, rope.inv_freq_w = saved_rope
+        merged = vision_output.last_hidden_state.reshape(-1, vision_output.last_hidden_state.shape[-1])
+        merge = self.visual.spatial_merge_size
+        time = int(image_grid_thw[:, 0].sum().item())
+        patches_per_time = int(
+            (image_grid_thw[0, 1].item() // merge)
+            * (image_grid_thw[0, 2].item() // merge)
+        )
+        vision_tokens = merged.reshape(1, time, patches_per_time, -1)
+        return vision_tokens
+
+    def streammind_gate_forward(self, pixel_values, image_grid_thw, patch_positions=None):
+        """Run the gate on one segment without changing the base LLM visual path."""
+        vision_tokens = self._streammind_vision_tokens(
+            pixel_values, image_grid_thw, patch_positions=patch_positions
+        )
+        return self._load_streammind_gate()(vision_tokens)
+
+    def streammind_gate_forward_segments(self, segments):
+        """Run EPFE continuously over a list of codec segments from one stream."""
+        tokens = [
+            self._streammind_vision_tokens(
+                segment["pixel_values"],
+                segment["image_grid_thw"],
+                patch_positions=segment.get("patch_positions"),
+            )
+            for segment in segments
+        ]
+        lengths = [token.shape[1] for token in tokens]
+        boundaries = torch.tensor(lengths).cumsum(0).tolist()
+        return self._load_streammind_gate()(
+            torch.cat(tokens, dim=1), response_positions=boundaries
+        )
+
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+
+    def set_input_embeddings(self, value):
+        self.language_model.set_input_embeddings(value)
+
+    def set_decoder(self, decoder):
+        self.language_model = decoder
+
+    def get_decoder(self):
+        return self.language_model
+
+    def get_video_features(
+        self,
+        pixel_values_videos: torch.FloatTensor,
+        video_grid_thw: Optional[torch.LongTensor] = None,
+        patch_positions=None,
+    ):
+        """
+        Encodes videos into continuous embeddings that can be forwarded to the language model.
+
+        Args:
+            pixel_values_videos: Pre-processed patches from Qwen2VL processor.
+                `torch.FloatTensor` of shape `(total_patches, num_channels, patch_size, patch_size)`
+            video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
+                The temporal, height and width of feature shape of each video in LLM.
+        """
+        # Convert to correct dtype
+        pixel_values_videos = pixel_values_videos.type(self.visual.vision_tower.patch_embed.proj.weight.dtype)
+
+        # Forward through vision model with grid_thw
+        vision_output = self.visual(pixel_values_videos, grid_thw=video_grid_thw, patch_positions=patch_positions)
+
+        # Extract the actual tensor from BaseModelOutputWithPooling
+        if hasattr(vision_output, "last_hidden_state"):
+            video_embeds = vision_output.last_hidden_state
+        else:
+            video_embeds = vision_output[0]  # Fallback for tuple output
+
+        # Compute split sizes from video_grid_thw or from input shape
+        if video_grid_thw is not None:
+            split_sizes = (video_grid_thw.prod(-1) // self.visual.spatial_merge_size**2).tolist()
+        else:
+            # Compute from input shape
+            batch_size = pixel_values_videos.shape[0]
+            split_sizes = [video_embeds.shape[1]] * batch_size
+
+        # Split embeddings per video
+        if len(split_sizes) > 1:
+            video_embeds = torch.split(video_embeds.view(-1, video_embeds.shape[-1]), split_sizes)
+        else:
+            video_embeds = [video_embeds.view(-1, video_embeds.shape[-1])]
+
+        return video_embeds
+
+    def get_image_features(
+        self, pixel_values, image_grid_thw: Optional[torch.LongTensor] = None, patch_positions=None
+    ):
+        """
+        Encodes images into continuous embeddings that can be forwarded to the language model.
+
+        Args:
+            pixel_values: Pre-processed patches from Qwen2VL processor.
+                - `torch.FloatTensor` of shape `(total_patches, num_channels, patch_size, patch_size)`
+            image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
+                The temporal, height and width of feature shape of each image in LLM.
+        """
+        # Kimi-K3 processor emits already-unfolded image patches as
+        # [total_patches, channels, patch_height, patch_width].
+        if pixel_values.dim() == 4:
+            # Convert to correct dtype
+            pixel_values = pixel_values.type(self.visual.vision_tower.patch_embed.proj.weight.dtype)
+
+            # Forward through vision model with grid_thw
+            vision_output = self.visual(pixel_values, grid_thw=image_grid_thw, patch_positions=patch_positions)
+
+            # Extract the actual tensor from BaseModelOutputWithPooling
+            if hasattr(vision_output, "last_hidden_state"):
+                image_embeds = vision_output.last_hidden_state
+            else:
+                image_embeds = vision_output[0]
+
+            # Compute split sizes from grid_thw
+            if image_grid_thw is not None:
+                split_sizes = (image_grid_thw.prod(-1) // self.visual.spatial_merge_size**2).tolist()
+            else:
+                # Fallback: assume single image
+                split_sizes = [image_embeds.shape[0] if image_embeds.dim() == 2 else image_embeds.shape[1]]
+
+            # Split embeddings per image
+            image_embeds_flat = image_embeds.view(-1, image_embeds.shape[-1])
+            if len(split_sizes) > 1:
+                image_embeds = list(torch.split(image_embeds_flat, split_sizes))
+            else:
+                image_embeds = [image_embeds_flat]
+
+            return image_embeds
+        else:
+            raise ValueError(
+                f"Unsupported pixel_values shape: expected 4D tensor [total_patches, C, H, W], "
+                f"got {pixel_values.shape if hasattr(pixel_values, 'shape') else type(pixel_values)}"
+            )
+
+    def get_placeholder_mask(
+        self,
+        input_ids: torch.LongTensor,
+        inputs_embeds: torch.FloatTensor,
+        image_features: Optional[torch.FloatTensor] = None,
+        video_features: Optional[torch.FloatTensor] = None,
+    ):
+        """
+        Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is
+        equal to the length of multimodal features. If the lengths are different, an error is raised.
+        """
+        if input_ids is None:
+            special_image_mask = inputs_embeds == self.get_input_embeddings()(
+                torch.tensor(self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
+            )
+            special_image_mask = special_image_mask.all(-1)
+            special_video_mask = inputs_embeds == self.get_input_embeddings()(
+                torch.tensor(self.config.video_token_id, dtype=torch.long, device=inputs_embeds.device)
+            )
+            special_video_mask = special_video_mask.all(-1)
+        else:
+            special_image_mask = input_ids == self.config.image_token_id
+            special_video_mask = input_ids == self.config.video_token_id
+
+        n_image_tokens = special_image_mask.sum()
+        special_image_mask = special_image_mask.unsqueeze(-1).expand_as(inputs_embeds).to(inputs_embeds.device)
+        if image_features is not None and inputs_embeds[special_image_mask].numel() != image_features.numel():
+            raise ValueError(
+                f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {image_features.shape[0]}"
+            )
+
+        n_video_tokens = special_video_mask.sum()
+        special_video_mask = special_video_mask.unsqueeze(-1).expand_as(inputs_embeds).to(inputs_embeds.device)
+        if video_features is not None and inputs_embeds[special_video_mask].numel() != video_features.numel():
+            raise ValueError(
+                f"Videos features and video tokens do not match: tokens: {n_video_tokens}, features {video_features.shape[0]}"
+            )
+
+        return special_image_mask, special_video_mask
+
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        pixel_values: Optional[torch.Tensor] = None,
+        pixel_values_videos: Optional[torch.FloatTensor] = None,
+        image_grid_thw: Optional[torch.LongTensor] = None,
+        patch_positions: Optional[torch.LongTensor] = None,
+        video_grid_thw: Optional[torch.LongTensor] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        second_per_grid_ts: Optional[torch.Tensor] = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> Union[tuple, GroundAnythingVLMBaseModelOutputWithPast]:
+        r"""
+        image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
+            The temporal, height and width of feature shape of each image in LLM.
+        video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
+            The temporal, height and width of feature shape of each video in LLM.
+        patch_positions (`torch.LongTensor` of shape `(total_patches, 3)` or `(1, total_patches, 3)`, *optional*):
+            Explicit per-patch `(t, h, w)` position indices used by the vision tower to compute 3D rotary
+            position embeddings (and the optional absolute position embedding inside the patch merger).
+            `total_patches` is the sum of `t * h * w` across all images and videos in the batch, matching
+            the layout produced by the Qwen2VL-style image processor.
+        second_per_grid_ts (`torch.Tensor` of shape `(num_videos)`, *optional*):
+            The time interval (in seconds) for each grid along the temporal dimension in the 3D position IDs.
+        cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*):
+              Indices depicting the position of the input sequence tokens in the sequence. Contrarily to
+              `position_ids`, this tensor is not affected by padding.
+
+        Note: see the top-level ``GroundAnythingVLMBaseForConditionalGeneration.forward``
+        docstring; currently video flows in via the ``image_grid_thw`` / ``pixel_values``
+        alias, so ``pixel_values_videos`` / ``video_grid_thw`` /
+        ``second_per_grid_ts`` are unused at this layer.
+        """
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        return_dict = True if return_dict is None else return_dict
+
+        if inputs_embeds is None:
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+
+        image_embeds = None
+
+        if pixel_values is not None:
+            image_embeds = self.get_image_features(pixel_values, image_grid_thw, patch_positions=patch_positions)
+
+        if image_embeds is not None:
+            image_embeds = torch.cat(image_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
+            image_mask, _ = self.get_placeholder_mask(
+                input_ids, inputs_embeds=inputs_embeds, image_features=image_embeds
+            )
+            inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
+
+        if pixel_values_videos is not None:
+            video_embeds = self.get_video_features(
+                pixel_values_videos, video_grid_thw, patch_positions=patch_positions
+            )
+            video_embeds = torch.cat(video_embeds, dim=0).to(inputs_embeds.device, inputs_embeds.dtype)
+            _, video_mask = self.get_placeholder_mask(
+                input_ids, inputs_embeds=inputs_embeds, video_features=video_embeds
+            )
+            inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_embeds)
+
+        # Use simple 1D position_ids
+        if position_ids is None:
+            batch_size, seq_length, _ = inputs_embeds.shape
+            if attention_mask is not None:
+                position_ids = attention_mask.long().cumsum(-1) - 1
+                position_ids.masked_fill_(attention_mask == 0, 1)
+            else:
+                position_ids = (
+                    torch.arange(seq_length, device=inputs_embeds.device).unsqueeze(0).expand(batch_size, -1)
+                )
+
+            # Handle cache_position for generation
+            if cache_position is not None and cache_position[0] != 0:
+                position_ids = position_ids + cache_position[0]
+
+        outputs = self.language_model(
+            input_ids=None,
+            position_ids=position_ids,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=True,
+            cache_position=cache_position,
+            **kwargs,
+        )
+
+        output = GroundAnythingVLMBaseModelOutputWithPast(
+            last_hidden_state=outputs.last_hidden_state,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+        return output if return_dict else output.to_tuple()
+
+
+@auto_docstring
+class GroundAnythingVLMBaseForConditionalGeneration(GroundAnythingVLMPreTrainedModel, GenerationMixin):
+    _tied_weights_keys = {"lm_head.weight": "model.language_model.embed_tokens.weight"}
+    # Reference: fix gemma3 grad acc #37208
+    accepts_loss_kwargs = False
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = GroundAnythingVLMBaseModel(config)
+        self.lm_head = nn.Linear(config.text_config.hidden_size, config.text_config.vocab_size, bias=False)
+        self.post_init()
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, *args, **kwargs):
+        model = super().from_pretrained(pretrained_model_name_or_path, *args, **kwargs)
+        model.model._streammind_model_path = str(pretrained_model_name_or_path)
+        return model
+
+    def get_input_embeddings(self):
+        return self.model.get_input_embeddings()
+
+    def set_input_embeddings(self, value):
+        self.model.set_input_embeddings(value)
+
+    def set_decoder(self, decoder):
+        self.model.set_decoder(decoder)
+
+    def get_decoder(self):
+        return self.model.get_decoder()
+
+    def get_video_features(
+        self,
+        pixel_values_videos: torch.FloatTensor,
+        video_grid_thw: Optional[torch.LongTensor] = None,
+        patch_positions=None,
+    ):
+        return self.model.get_video_features(pixel_values_videos, video_grid_thw, patch_positions=patch_positions)
+
+    def get_image_features(self, pixel_values: torch.FloatTensor, image_grid_thw: Optional[torch.LongTensor] = None):
+        return self.model.get_image_features(pixel_values, image_grid_thw)
+
+    # Make modules available through conditional class for BC
+    @property
+    def language_model(self):
+        return self.model.language_model
+
+    @property
+    def visual(self):
+        return self.model.visual
+
+    def streammind_gate_forward(self, pixel_values, image_grid_thw, patch_positions=None):
+        return self.model.streammind_gate_forward(
+            pixel_values,
+            image_grid_thw,
+            patch_positions=patch_positions,
+        )
+
+    def streammind_gate_forward_segments(self, segments):
+        return self.model.streammind_gate_forward_segments(segments)
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        pixel_values: Optional[torch.Tensor] = None,
+        pixel_values_videos: Optional[torch.FloatTensor] = None,
+        image_grid_thw: Optional[torch.LongTensor] = None,
+        patch_positions: Optional[torch.LongTensor] = None,
+        video_grid_thw: Optional[torch.LongTensor] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        second_per_grid_ts: Optional[torch.Tensor] = None,
+        logits_to_keep: Union[int, torch.Tensor] = 0,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> Union[tuple, GroundAnythingVLMCausalLMOutputWithPast]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
+            config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
+            (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
+        image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
+            The temporal, height and width of feature shape of each image in LLM.
+        video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
+            The temporal, height and width of feature shape of each video in LLM.
+        patch_positions (`torch.LongTensor` of shape `(total_patches, 3)` or `(1, total_patches, 3)`, *optional*):
+            Explicit per-patch `(t, h, w)` position indices used by the vision tower to compute 3D rotary
+            position embeddings (and the optional absolute position embedding inside the patch merger).
+            `total_patches` is the sum of `t * h * w` across all images and videos in the batch, matching
+            the layout produced by the Qwen2VL-style image processor.
+        second_per_grid_ts (`torch.Tensor` of shape `(num_videos)`, *optional*):
+            The time interval (in seconds) for each grid along the temporal dimension in the 3D position IDs.
+        cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*):
+              Indices depicting the position of the input sequence tokens in the sequence. Contrarily to
+              `position_ids`, this tensor is not affected by padding.
+
+        Note (native-video alias):
+            The companion ``GroundAnythingVLMProcessor.__call__(videos=...)`` does NOT
+            pass ``pixel_values_videos`` / ``video_grid_thw`` / ``second_per_grid_ts``
+            to this forward. Instead it aliases the video patch tensor as
+            ``pixel_values=`` and ``image_grid_thw=``, so video inputs share the
+            same code path as multi-image inputs (vision is purely
+            spatial; temporal information is carried by per-frame ``<X.X seconds>``
+            text tags emitted by the processor). The ``*_videos`` and
+            ``second_per_grid_ts`` kwargs are kept declared here only for API
+            completeness and future use (e.g. 3D mRoPE / ``get_rope_index``); they
+            are NOT consumed by the current vision encoder.
+
+        Example:
+
+        ```python
+        >>> from PIL import Image
+        >>> import requests
+        >>> from transformers import AutoProcessor, GroundAnythingVLMBaseForConditionalGeneration
+
+        >>> model = GroundAnythingVLMBaseForConditionalGeneration.from_pretrained("/path/to/GroundAnything-VLM", trust_remote_code=True)
+        >>> processor = AutoProcessor.from_pretrained("/path/to/GroundAnything-VLM", trust_remote_code=True)
+
+        >>> messages = [
+            {
+                "role": "user",
+                "content": [
+                    {"type": "image"},
+                    {"type": "text", "text": "What is shown in this image?"},
+                ],
+            },
+        ]
+        >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg"
+        >>> image = Image.open(requests.get(url, stream=True).raw)
+
+        >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+        >>> inputs = processor(text=[text], images=[image], return_tensors="pt")
+
+        >>> # Generate
+        >>> generate_ids = model.generate(inputs.input_ids, max_length=30)
+        >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+        "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..."
+        ```"""
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        outputs = self.model(
+            input_ids=input_ids,
+            pixel_values=pixel_values,
+            pixel_values_videos=pixel_values_videos,
+            image_grid_thw=image_grid_thw,
+            patch_positions=patch_positions,
+            video_grid_thw=video_grid_thw,
+            second_per_grid_ts=second_per_grid_ts,
+            position_ids=position_ids,
+            attention_mask=attention_mask,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=True,
+            cache_position=cache_position,
+            **kwargs,
+        )
+
+        hidden_states = outputs[0]
+
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(
+                logits=logits, labels=labels, vocab_size=self.config.text_config.vocab_size, **kwargs
+            )
+
+            # A packed batch can be text-only on one distributed rank. Keep every
+            # trainable projector tensor in that rank's graph so ZeRO launches the
+            # same gradient collectives as ranks that received visual samples.
+            projector_zero_anchor = None
+            for parameter in self.model.visual.projector.parameters():
+                if parameter.requires_grad:
+                    term = parameter.reshape(-1)[0] * 0.0
+                    projector_zero_anchor = term if projector_zero_anchor is None else projector_zero_anchor + term
+            if projector_zero_anchor is not None:
+                loss = loss + projector_zero_anchor
+
+        return GroundAnythingVLMCausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        attention_mask=None,
+        inputs_embeds=None,
+        cache_position=None,
+        position_ids=None,
+        use_cache=True,
+        pixel_values=None,
+        pixel_values_videos=None,
+        image_grid_thw=None,
+        patch_positions=None,
+        video_grid_thw=None,
+        second_per_grid_ts=None,
+        is_first_iteration=False,
+        **kwargs,
+    ):
+        # Overwritten -- in specific circumstances we don't want to forward image inputs to the model
+        model_inputs = super().prepare_inputs_for_generation(
+            input_ids,
+            past_key_values=past_key_values,
+            attention_mask=attention_mask,
+            inputs_embeds=inputs_embeds,
+            cache_position=cache_position,
+            position_ids=position_ids,
+            pixel_values=pixel_values,
+            pixel_values_videos=pixel_values_videos,
+            image_grid_thw=image_grid_thw,
+            video_grid_thw=video_grid_thw,
+            second_per_grid_ts=second_per_grid_ts,
+            patch_positions=patch_positions,
+            use_cache=use_cache,
+            is_first_iteration=is_first_iteration,
+            **kwargs,
+        )
+
+        # After the prefill iteration, drop image inputs so the vision tower
+        # isn't re-run on decode steps. Gating on `is_first_iteration` (the
+        # Qwen3-VL convention) is the only reliable signal in transformers
+        # 5.x: `past_key_values` is non-None even on the first call (an empty
+        # DynamicCache is created up-front by `generate`), and `cache_position`
+        # may be `None` for remote-code models.
+        if not is_first_iteration and use_cache:
+            model_inputs["pixel_values"] = None
+            model_inputs["pixel_values_videos"] = None
+
+        return model_inputs
+
+    def _get_image_nums_and_video_nums(
+        self,
+        input_ids: Optional[torch.LongTensor],
+        inputs_embeds: Optional[torch.Tensor] = None,
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        """
+        Get the number of images and videos for each sample to calculate the separation length of the sample tensor.
+        These parameters are not passed through the processor to avoid unpredictable impacts from interface modifications.
+
+        Args:
+            input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
+                Indices of input sequence tokens in the vocabulary.
+
+        Returns:
+            image_nums (`torch.LongTensor` of shape `(batch_size, num_images_sample)`)
+            video_nums (`torch.LongTensor` of shape `(batch_size, num_videos_sample)`)
+        """
+        image_token_id = self.config.image_token_id
+        video_token_id = self.config.video_token_id
+        vision_start_token_id = self.config.vision_start_token_id
+
+        if inputs_embeds is not None:
+            vision_start_mask = (
+                inputs_embeds
+                == self.get_input_embeddings()(
+                    torch.tensor(vision_start_token_id, dtype=torch.long, device=inputs_embeds.device)
+                )
+            )[..., 0]
+            image_mask = (
+                inputs_embeds
+                == self.get_input_embeddings()(
+                    torch.tensor(image_token_id, dtype=torch.long, device=inputs_embeds.device)
+                )
+            )[..., 0]
+            video_mask = (
+                inputs_embeds
+                == self.get_input_embeddings()(
+                    torch.tensor(video_token_id, dtype=torch.long, device=inputs_embeds.device)
+                )
+            )[..., 0]
+        else:
+            vision_start_mask = input_ids == vision_start_token_id
+            image_mask = input_ids == image_token_id
+            video_mask = input_ids == video_token_id
+
+        vision_first_mask = torch.roll(vision_start_mask, shifts=1, dims=1)
+        image_nums = torch.sum(vision_first_mask & image_mask, dim=1)
+        video_nums = torch.sum(vision_first_mask & video_mask, dim=1)
+
+        return image_nums, video_nums
+
+    def _expand_inputs_for_generation(
+        self,
+        expand_size: int = 1,
+        is_encoder_decoder: bool = False,
+        input_ids: Optional[torch.LongTensor] = None,
+        **model_kwargs,
+    ) -> tuple[torch.LongTensor, dict[str, Any]]:
+        # Overwritten -- Support for expanding tensors without a batch size dimension
+        # e.g., pixel_values, image_grid_thw, pixel_values_videos, video_grid_thw, second_per_grid_t
+        # pixel_values.shape[0] is sum(seqlen_images for samples)
+        # image_grid_thw.shape[0] is sum(num_images for samples)
+
+        if expand_size == 1:
+            return input_ids, model_kwargs
+
+        visual_keys = [
+            "pixel_values",
+            "image_grid_thw",
+            "pixel_values_videos",
+            "video_grid_thw",
+            "second_per_grid_ts",
+            "patch_positions",
+        ]
+
+        def _expand_dict_for_generation_visual(dict_to_expand):
+            image_grid_thw = model_kwargs.get("image_grid_thw", None)
+            video_grid_thw = model_kwargs.get("video_grid_thw", None)
+            image_nums, video_nums = self._get_image_nums_and_video_nums(
+                input_ids, inputs_embeds=model_kwargs.get("inputs_embeds", None)
+            )
+
+            def _repeat_interleave_samples(x, lengths, repeat_times):
+                samples = torch.split(x, lengths)
+                repeat_args = [repeat_times] + [1] * (x.dim() - 1)
+                result = torch.cat([sample.repeat(*repeat_args) for sample in samples], dim=0)
+                return result
+
+            for key in dict_to_expand:
+                if key == "pixel_values":
+                    # split images into samples
+                    samples = torch.split(image_grid_thw, list(image_nums))
+                    # compute the sequence length of images for each sample
+                    lengths = [torch.prod(sample, dim=1).sum() for sample in samples]
+                    dict_to_expand[key] = _repeat_interleave_samples(
+                        dict_to_expand[key], lengths=lengths, repeat_times=expand_size
+                    )
+                elif key == "image_grid_thw":
+                    # get the num of images for each sample
+                    lengths = list(image_nums)
+                    dict_to_expand[key] = _repeat_interleave_samples(
+                        dict_to_expand[key], lengths=lengths, repeat_times=expand_size
+                    )
+                elif key == "pixel_values_videos":
+                    samples = torch.split(video_grid_thw, list(video_nums))
+                    lengths = [torch.prod(sample, dim=1).sum() for sample in samples]
+                    dict_to_expand[key] = _repeat_interleave_samples(
+                        dict_to_expand[key], lengths=lengths, repeat_times=expand_size
+                    )
+                elif key == "video_grid_thw":
+                    lengths = list(video_nums)
+                    dict_to_expand[key] = _repeat_interleave_samples(
+                        dict_to_expand[key], lengths=lengths, repeat_times=expand_size
+                    )
+                elif key == "second_per_grid_ts":
+                    dict_to_expand[key] = _repeat_interleave_samples(
+                        dict_to_expand[key], lengths=list(video_nums), repeat_times=expand_size
+                    )
+                elif key == "patch_positions":
+                    if image_grid_thw is not None and image_grid_thw.numel() > 0 and image_nums.sum() > 0:
+                        samples = torch.split(image_grid_thw, list(image_nums))
+                        lengths = [torch.prod(sample, dim=1).sum() for sample in samples]
+                    elif video_grid_thw is not None and video_grid_thw.numel() > 0 and video_nums.sum() > 0:
+                        samples = torch.split(video_grid_thw, list(video_nums))
+                        lengths = [torch.prod(sample, dim=1).sum() for sample in samples]
+                    else:
+                        continue
+                    dict_to_expand[key] = _repeat_interleave_samples(
+                        dict_to_expand[key], lengths=lengths, repeat_times=expand_size
+                    )
+            return dict_to_expand
+
+        def _expand_dict_for_generation(dict_to_expand):
+            for key in dict_to_expand:
+                if (
+                    key != "cache_position"
+                    and dict_to_expand[key] is not None
+                    and isinstance(dict_to_expand[key], torch.Tensor)
+                    and key not in visual_keys
+                ):
+                    dict_to_expand[key] = dict_to_expand[key].repeat_interleave(expand_size, dim=0)
+            return dict_to_expand
+
+        model_kwargs = _expand_dict_for_generation_visual(model_kwargs)
+
+        if input_ids is not None:
+            input_ids = input_ids.repeat_interleave(expand_size, dim=0)
+
+        model_kwargs = _expand_dict_for_generation(model_kwargs)
+
+        if is_encoder_decoder:
+            if model_kwargs.get("encoder_outputs") is None:
+                raise ValueError("If `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.")
+            model_kwargs["encoder_outputs"] = _expand_dict_for_generation(model_kwargs["encoder_outputs"])
+
+        return input_ids, model_kwargs
+
+
+class GroundAnythingVLMModel(GroundAnythingVLMBaseModel):
+    """Named GroundAnything-VLM base-model entry point."""
+
+
+class GroundAnythingVLMForConditionalGeneration(GroundAnythingVLMBaseForConditionalGeneration):
+    """GroundAnything Qwen3-4B with the Kimi-K3 MoonViT3D visual encoder."""
+
+
+__all__ = [
+    "GroundAnythingVLMForConditionalGeneration",
+    "GroundAnythingVLMModel",
+    "GroundAnythingVLMBaseForConditionalGeneration",
+    "GroundAnythingVLMBaseModel",
+    "GroundAnythingVLMPreTrainedModel",
+]
+
+
+class GroundAnythingModel(GroundAnythingVLMModel):
+    """DLM release model identity; SGLang loads wrapper weights separately."""
+
+    config_class = GroundAnythingConfig
+
+
+class GroundAnythingForConditionalGeneration(GroundAnythingVLMForConditionalGeneration):
+    config_class = GroundAnythingConfig

modeling_groundinganything_vision.py

@@ -0,0 +1,1345 @@
+# coding=utf-8
+# Copyright 2025-2026 The Moonshot AI Team and HuggingFace Inc. team. All rights reserved.
+#
+# The code is based on llava (llava/modeling_llava.py), but modified for Kimi-K3.
+#
+# Licensing Information:
+# - Code derived from llava (llava/modeling_llava.py) is licensed under the Apache License, Version 2.0.
+# - Other parts of the code are licensed under the Kimi K3 License (see the LICENSE file in this repository).
+#
+# Apache License, Version 2.0:
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# NOTE: Reference implementation for model architecture; see the model card for production deployment.
+import math
+from collections.abc import Sequence
+from copy import deepcopy
+from typing import Optional
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import activations
+
+try:
+    from transformers.activations import PytorchGELUTanh
+except ImportError:
+    from transformers.activations import GELUTanh
+    activations.PytorchGELUTanh = GELUTanh
+    PytorchGELUTanh = GELUTanh
+from transformers.activations import PytorchGELUTanh
+from transformers.configuration_utils import PretrainedConfig
+from transformers.modeling_utils import PreTrainedModel
+from transformers.models.llava.modeling_llava import \
+    LlavaCausalLMOutputWithPast
+from transformers.utils import is_flash_attn_2_available
+
+GroundAnythingBackboneConfig = PretrainedConfig
+# GroundAnything-VLM imports only MoonViT3D from this reference module.  Keep the
+# Kimi text implementation optional so loading the vision tower does not
+# require fla-core.
+GroundAnythingBackboneLinearForCausalLM = None
+
+# Flash attention imports
+if is_flash_attn_2_available():
+    from flash_attn import flash_attn_varlen_func
+else:
+    flash_attn_varlen_func = None
+
+
+def multihead_attention(
+    q: torch.Tensor,
+    k: torch.Tensor,
+    v: torch.Tensor,
+    q_cu_seqlens: torch.Tensor | None = None,
+    k_cu_seqlens: torch.Tensor | None = None,
+    max_seqlen_q: int | None = None,
+    max_seqlen_k: int | None = None,
+    deterministic: bool = False,
+):
+    """Multi-head attention using flash attention 2.
+
+    Args:
+        q, k, v: tensor of shape (batch_size, seqlen, num_heads, head_dim),
+            or (tot_seqlens, num_heads, head_dim) if packing.
+        q_cu_seqlens (torch.Tensor): cumulative sequence lengths of q.
+            The first element should be 0 and the last element should be q.shape[0].
+        k_cu_seqlens (torch.Tensor): cumulative sequence lengths of k.
+            The first element should be 0 and the last element should be k.shape[0].
+
+    Returns:
+        output: shape (batch_size, seqlen, dim) or (tot_seqlens, dim) if packing,
+            where dim = num_heads * head_dim
+    """
+    attn_out = flash_attn_varlen_func(
+        q,
+        k,
+        v,
+        q_cu_seqlens,
+        k_cu_seqlens,
+        max_seqlen_q,
+        max_seqlen_k,
+        causal=False,
+        deterministic=deterministic,
+    )
+    if isinstance(attn_out, tuple):
+        attn_out = attn_out[0]
+
+    attn_out = attn_out.flatten(start_dim=-2)
+
+    return attn_out
+
+
+def eager_attention(
+    q: torch.Tensor,
+    k: torch.Tensor,
+    v: torch.Tensor,
+    q_cu_seqlens: Optional[torch.Tensor] = None,
+    k_cu_seqlens: Optional[torch.Tensor] = None,
+    **kwargs,
+) -> torch.Tensor:
+    seq_length = q.shape[0]
+    attention_mask = torch.zeros([1, seq_length, seq_length],
+                                 device=q.device,
+                                 dtype=torch.bool)
+    for i in range(1, len(q_cu_seqlens)):
+        attention_mask[
+            ...,
+            q_cu_seqlens[i - 1]:q_cu_seqlens[i],
+            q_cu_seqlens[i - 1]:q_cu_seqlens[i],
+        ] = True
+    q = q.transpose(0, 1)
+    k = k.transpose(0, 1)
+    v = v.transpose(0, 1)
+
+    attn_weight = q @ k.transpose(-2, -1) / math.sqrt(q.shape[-1])
+    attn_weight = attn_weight.masked_fill(
+        ~attention_mask, torch.finfo(attn_weight.dtype).min)
+    attn_weight = torch.softmax(attn_weight, dim=-1,
+                                dtype=torch.float32).to(q.dtype)
+
+    attn_output = attn_weight @ v
+    attn_output = attn_output.transpose(0, 1)
+    attn_output = attn_output.reshape(seq_length, -1)
+    return attn_output
+
+
+def sdpa_attention(
+    q: torch.Tensor,
+    k: torch.Tensor,
+    v: torch.Tensor,
+    q_cu_seqlens: torch.Tensor | None = None,
+    k_cu_seqlens: torch.Tensor | None = None,
+    **kwargs,
+) -> torch.Tensor:
+    del k_cu_seqlens, kwargs
+    outputs = []
+    for index in range(1, len(q_cu_seqlens)):
+        start = int(q_cu_seqlens[index - 1])
+        end = int(q_cu_seqlens[index])
+        query = q[start:end].transpose(0, 1)
+        key = k[start:end].transpose(0, 1)
+        value = v[start:end].transpose(0, 1)
+        output = F.scaled_dot_product_attention(
+            query, key, value, dropout_p=0.0, is_causal=False
+        )
+        outputs.append(output.transpose(0, 1))
+    return torch.cat(outputs, dim=0).flatten(start_dim=-2)
+
+
+VL_VISION_ATTENTION_FUNCTIONS = {
+    "flash_attention_2": multihead_attention,
+    "eager": eager_attention,
+    "sdpa": sdpa_attention,
+}
+
+
+def _apply_rope_input_validation(x, freqs_cis):
+    assert x.ndim == freqs_cis.ndim + 1, (x.shape, freqs_cis.shape)
+    assert x.shape[:-2] == freqs_cis.shape[:-1], (x.shape, freqs_cis.shape)
+    assert x.shape[-1] == 2 * freqs_cis.shape[-1], (x.shape, freqs_cis.shape)
+    assert freqs_cis.dtype == torch.complex64, freqs_cis.dtype
+
+
+def get_rope_shape_decorate(func):
+    _get_rope_shape_first_call_flag = set()
+
+    def wrapper(org, interpolation_mode, shape):
+        key = (org.requires_grad, torch.is_grad_enabled(), interpolation_mode)
+        if key not in _get_rope_shape_first_call_flag:
+            _get_rope_shape_first_call_flag.add(key)
+            _ = func(org, interpolation_mode, shape=(64, 64))
+        return func(org, interpolation_mode, shape)
+
+    return wrapper
+
+
+@get_rope_shape_decorate
+@torch.compile(dynamic=True)
+def get_rope_shape(org, interpolation_mode, shape):
+    return (F.interpolate(
+        org.permute((2, 0, 1)).unsqueeze(0),
+        size=shape,
+        mode=interpolation_mode,
+    ).squeeze(0).permute((1, 2, 0)).flatten(end_dim=1))
+
+
+def apply_rope(xq: torch.Tensor, xk: torch.Tensor,
+               freqs_cis: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+    """
+    Args: (The leading dimensions of all inputs should be the same)
+        xq: query, tensor of shape (..., num_heads, head_dim)
+        xk: key, tensor of shape (..., num_heads, head_dim)
+        freqs_cis: tensor of shape (..., head_dim/2), dtype=torch.complex64. It contains the precomputed cis(freqs) for each position in the 2D grid.
+    Returns:
+        xq_out, xk_out: tensors of shape (..., num_heads, head_dim)
+    """
+    _apply_rope_input_validation(xq, freqs_cis)
+    _apply_rope_input_validation(xk, freqs_cis)
+
+    freqs_cis = freqs_cis.unsqueeze(-2)  # ..., 1, head_dim/2
+    # ..., num_heads, head_dim/2
+    xq_ = torch.view_as_complex(xq.float().view(*xq.shape[:-1], -1, 2))
+    xk_ = torch.view_as_complex(xk.float().view(*xq.shape[:-1], -1, 2))
+    xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(
+        -2)  # ..., num_heads, head_dim
+    xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(
+        -2)  # ..., num_heads, head_dim
+    return xq_out.type_as(xq), xk_out.type_as(xk)
+
+
+def get_1d_sincos_pos_embed_from_grid(embed_dim, pos):
+    """
+    From:
+    https://github.com/OpenGVLab/InternVideo/blob/421f6d2361fc8f61a3394244571f2601a4e99e29/InternVideo2/multi_modality/models/backbones/internvideo2/pos_embed.py#L86
+    embed_dim: output dimension for each position
+    pos: a list of positions to be encoded: size (M,)
+    out: (M, D)
+    """
+    assert embed_dim % 2 == 0
+    omega = np.arange(embed_dim // 2, dtype=np.float32)
+    omega /= embed_dim / 2.0
+    omega = 1.0 / 10000**omega  # (D/2,)
+
+    pos = pos.reshape(-1)  # (M,)
+    out = np.einsum('m,d->md', pos, omega)  # (M, D/2), outer product
+
+    emb_sin = np.sin(out)  # (M, D/2)
+    emb_cos = np.cos(out)  # (M, D/2)
+
+    emb = np.concatenate([emb_sin, emb_cos], axis=1)  # (M, D)
+    return emb
+
+
+def get_1d_sincos_pos_embed(embed_dim, t_size, cls_token=False):
+    """
+    t_size: int of the temporal size
+    return:
+    pos_embed: [t_size, embed_dim] or [1+t_size, embed_dim] (w/ or w/o cls_token)
+    """
+    grid_t = np.arange(t_size, dtype=np.float32)
+    pos_embed = get_1d_sincos_pos_embed_from_grid(embed_dim, grid_t)
+    if cls_token:
+        pos_embed = np.concatenate([np.zeros([1, embed_dim]), pos_embed],
+                                   axis=0)
+    return pos_embed
+
+
+class Learnable2DInterpPosEmbDivided_fixed(nn.Module):
+
+    def __init__(self,
+                 height: int,
+                 width: int,
+                 num_frames: int,
+                 dim: int,
+                 interpolation_mode: str = 'bicubic') -> None:
+        super().__init__()
+        self.height = height
+        self.width = width
+        self.num_frames = num_frames
+        self.dim = dim
+        self.interpolation_mode = interpolation_mode
+        self.weight = nn.Parameter(torch.empty(height, width, dim))
+        self.register_buffer('time_weight',
+                             torch.from_numpy(
+                                 get_1d_sincos_pos_embed(
+                                     self.dim,
+                                     self.num_frames)).float().unsqueeze(1),
+                             persistent=False)
+
+        self.reset_parameters()
+
+    def reset_parameters(self):
+        nn.init.normal_(self.weight)
+
+    def forward(self, x: torch.Tensor,
+                grid_thws: torch.Tensor) -> torch.Tensor:
+        pos_embs = []
+        for t, h, w in grid_thws.tolist():
+            assert t <= self.num_frames, f't:{t} > self.num_frames:{self.num_frames}'
+            if (h, w) == self.weight.shape[:-1]:
+                pos_emb_2d = self.weight.flatten(end_dim=1)
+            else:
+                pos_emb_2d = get_rope_shape(
+                    self.weight,
+                    interpolation_mode=self.interpolation_mode,
+                    shape=(h, w),
+                )
+
+            if t == 1:
+                pos_emb_3d = pos_emb_2d
+            else:
+                pos_emb_3d = pos_emb_2d.unsqueeze(0).repeat(
+                    t, 1, 1) + self.time_weight[0:t]
+
+            pos_embs.append(pos_emb_3d.reshape(-1, pos_emb_3d.shape[-1]))
+
+        out = x + torch.cat(pos_embs)
+        return out
+
+
+class MoonVision3dPatchEmbed(nn.Module):
+
+    def __init__(self,
+                 out_dim: int,
+                 in_dim: int = 3,
+                 patch_size: int | tuple[int, int] = (14, 14),
+                 pos_emb_height: int = 14,
+                 pos_emb_width: int = 14,
+                 pos_emb_time: int = 4,
+                 pos_emb_type: str = 'divided_fixed',
+                 patch_embed_proj_bias: bool = True,
+                 pos_emb_interpolation_mode: str = 'bicubic'):
+        super().__init__()
+        assert isinstance(
+            patch_size,
+            int | Sequence), f'Invalid patch_size type: {type(patch_size)}'
+        if isinstance(patch_size, int):
+            patch_size = (patch_size, patch_size)
+        assert (len(patch_size) == 2
+                ), f'Expected patch_size to be a tuple of 2, got {patch_size}'
+        self.patch_size = patch_size
+
+        self.proj = nn.Conv2d(in_dim,
+                              out_dim,
+                              kernel_size=patch_size,
+                              stride=patch_size,
+                              bias=patch_embed_proj_bias)
+
+        if pos_emb_type == 'divided_fixed':
+            self.pos_emb = Learnable2DInterpPosEmbDivided_fixed(
+                height=pos_emb_height,
+                width=pos_emb_width,
+                num_frames=pos_emb_time,
+                dim=out_dim,
+                interpolation_mode=pos_emb_interpolation_mode)
+        else:
+            raise NotImplementedError(
+                f'Not support pos_emb_type: {pos_emb_type}')
+
+    def forward(self, x: torch.Tensor,
+                grid_thws: torch.Tensor) -> torch.Tensor:
+        """
+        Args:
+            x (L, Channels): input tensor
+            grid_hws (N, 3): temporal, height and width
+
+        Returns:
+            (L, Cout) tensor
+        """
+        x = self.proj(x).view(x.size(0), -1)
+        # apply positional embedding
+        x = self.pos_emb(x, grid_thws)
+        return x
+
+
+class Rope2DPosEmbRepeated(nn.Module):
+    """2D rotary position embedding with multi-resolution support.
+
+    This class is intended to be used in the following way:
+    1. Before training, create an instance of Rope2DPosEmb. This instance will hold the precomputed cis.
+    2. Before each forward pass, call `get_freqs_cis_by_*` to get the `freqs_cis` tensor for this iteration.
+    3. During the forward pass, pass the `freqs_cis` tensor to each attention layer, and call `apply` just before each attention operation.
+        The rope is shared across all attention layers and all heads.
+
+    Refs:
+    - RoFormer: https://arxiv.org/abs/2104.09864
+    - VisionLLaMA: https://arxiv.org/abs/2403.00522
+    - https://github.com/Meituan-AutoML/VisionLLaMA/blob/main/dit/models.py
+
+    Args:
+        dim (int): usually the multi-head attention dimension, should be divisible by 4 (TODO: relax this constraint if needed)
+        max_height (int): the maximum height of the 2D grid
+        max_width (int): the maximum width of the 2D grid
+        theta_base (float): the base of the theta
+        device (str): the device to store the precomputed cis
+    """
+
+    def __init__(self,
+                 dim: int,
+                 max_height: int,
+                 max_width: int,
+                 theta_base=10000):
+        super().__init__()
+        self.dim = dim
+        assert self.dim % 4 == 0, 'dim must be divisible by 4'
+        self.max_height = max_height
+        self.max_width = max_width
+        self.theta_base = theta_base
+
+    def extra_repr(self):
+        return f'dim={self.dim}, max_height={self.max_height}, max_width={self.max_width}, theta_base={self.theta_base}'
+
+    def _precompute_freqs_cis(self, device: torch.device) -> torch.Tensor:
+        """Calculate the cis(freqs) for each position in the 2D grid.
+
+        Return: complex tensor of shape (max_height, max_width, dim//2) and value:
+            height axis: ret[h, w, 2*i] = cis(h * theta_base**(-4*i/dim))
+            weight axis: ret[h, w, 2*i+1] = cis(w * theta_base**(-4*i/dim))   with (i in [0, dim//4))
+            note: `cis` is a mathematical notation defined by cis x = cos x + i sin x,
+        """
+        N = self.max_height * self.max_width
+        flat_pos = torch.arange(0, N).float().to(device)
+        x_pos = flat_pos % self.max_width
+        y_pos = flat_pos // self.max_width
+        dim_range = (torch.arange(0, self.dim,
+                                  4)[:(self.dim // 4)].float().to(device)
+                     )  # C/4
+        freqs = 1.0 / (self.theta_base**(dim_range / self.dim))
+        x_freqs = torch.outer(x_pos, freqs).float()  # N, C/4
+        y_freqs = torch.outer(y_pos, freqs).float()  # N, C/4
+        x_cis = torch.polar(torch.ones_like(x_freqs), x_freqs)  # N, C/4
+        y_cis = torch.polar(torch.ones_like(y_freqs), y_freqs)  # N, C/4
+        # N, C/4, 2
+        freqs_cis = torch.cat(
+            [x_cis.unsqueeze(dim=-1),
+             y_cis.unsqueeze(dim=-1)], dim=-1)
+        # max_height, max_width, C/2
+        freqs_cis = freqs_cis.reshape(self.max_height, self.max_width, -1)
+        return freqs_cis
+
+    def get_freqs_cis(self, grid_thws: torch.Tensor,
+                      device: torch.device) -> torch.Tensor:
+        """
+        Args:
+            grid_thws (torch.Tensor): grid time, height and width
+
+        Returns:
+            freqs_cis: tensor of shape (sum(t * height * width), dim//2)
+        """
+        if not hasattr(self, 'freqs_cis'):
+            self.register_buffer('freqs_cis',
+                                 self._precompute_freqs_cis(device),
+                                 persistent=False)
+
+        shapes = grid_thws.tolist()
+        assert all(1 <= h <= self.max_height and 1 <= w <= self.max_width
+                   for t, h, w in shapes), (
+                       shapes,
+                       self.max_height,
+                       self.max_width,
+                   )
+        freqs_cis = torch.cat(
+            [
+                self.freqs_cis[:h, :w].reshape(-1, self.dim // 2).repeat(t, 1)
+                for t, h, w in shapes
+            ],
+            dim=0,
+        )
+        return freqs_cis
+
+
+class MLP2(nn.Module):
+    """
+    Args:
+        dims: [in_dim, hidden_dim, out_dim]
+        bias: whether to use bias in linear layer.
+    """
+
+    def __init__(self, dims: list[int], activation, bias=True):
+        super().__init__()
+        assert len(dims) == 3
+        self.fc0 = nn.Linear(dims[0], dims[1], bias=bias)
+        self.fc1 = nn.Linear(dims[1], dims[2], bias=bias)
+        self.activation = activation
+        for m in [self.fc0, self.fc1]:
+            nn.init.trunc_normal_(m.weight, std=math.sqrt(2 / m.in_features))
+            if m.bias is not None:
+                nn.init.zeros_(m.bias)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x = self.fc0(x)
+        x = self.activation(x)
+        return self.fc1(x)
+
+
+class MoonViTEncoderLayer(nn.Module):
+
+    def __init__(
+        self,
+        num_heads: int,
+        hidden_dim: int,
+        mlp_dim: int,
+        qkv_hidden_size: int | None = None,
+        norm_type: str = 'layernorm',
+        mlp_type: str = 'mlp2',
+        *,
+        attn_implementation: str = 'flash_attention_2',
+        activation=F.gelu,
+        attn_bias: bool = False,
+        linear_bias: bool = True,
+        use_deterministic_attn: bool = False,
+    ):
+        super().__init__()
+        self.num_heads = num_heads
+        self.hidden_dim = hidden_dim
+        self.qkv_hidden_size = hidden_dim if qkv_hidden_size is None else qkv_hidden_size
+        self.hidden_size_per_attention_head = self.qkv_hidden_size // self.num_heads
+        self.attn_implementation = attn_implementation
+        self.use_deterministic_attn = use_deterministic_attn
+
+        if norm_type == "layernorm":
+            self.norm0 = nn.LayerNorm(hidden_dim)
+            self.norm1 = nn.LayerNorm(hidden_dim)
+        elif norm_type == "rmsnorm":
+            self.norm0 = nn.RMSNorm(hidden_dim, eps=1e-6)
+            self.norm1 = nn.RMSNorm(hidden_dim, eps=1e-6)
+        else:
+            raise NotImplementedError(f"Not support norm_type: {norm_type}")
+
+        if mlp_type == "mlp2":
+            self.mlp = MLP2([hidden_dim, mlp_dim, hidden_dim],
+                            activation,
+                            bias=linear_bias)
+        else:
+            raise NotImplementedError(f"Not support mlp_type: {mlp_type}")
+
+        self.wqkv = nn.Linear(hidden_dim,
+                              self.qkv_hidden_size * 3,
+                              bias=attn_bias)
+        self.wo = nn.Linear(self.qkv_hidden_size, hidden_dim, bias=attn_bias)
+
+    def attention_qkvpacked(
+        self,
+        x: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        max_seqlen: torch.Tensor,
+        rope_freqs_cis: torch.Tensor | None = None,
+    ):
+        """
+        Args:
+            x (torch.Tensor): (batch_size, seqlen, hidden_dim)
+            cu_seqlens (torch.Tensor):
+        """
+        xqkv = self.wqkv(x)
+
+        qkv_shape = xqkv.size()[:-1] + (
+            3,
+            self.num_heads,
+            self.hidden_size_per_attention_head,
+        )
+        # xqkv: (batch_size, seqlen, 3, nheads, headdim)
+        xqkv = xqkv.view(*qkv_shape)
+        xq, xk, xv = torch.unbind(xqkv, dim=-3)
+
+        xq, xk = apply_rope(xq, xk, rope_freqs_cis)
+
+        attn_func = VL_VISION_ATTENTION_FUNCTIONS[self.attn_implementation]
+        attn_out = attn_func(xq,
+                             xk,
+                             xv,
+                             q_cu_seqlens=cu_seqlens,
+                             k_cu_seqlens=cu_seqlens,
+                             max_seqlen_k=max_seqlen,
+                             max_seqlen_q=max_seqlen,
+                             deterministic=self.use_deterministic_attn)
+
+        attn_out = self.wo(attn_out)
+        return attn_out
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        max_seqlen: int,
+        rope_freqs_cis: torch.Tensor | None = None,
+    ):
+        residual = hidden_states
+        hidden_states = self.norm0(hidden_states)
+
+        hidden_states = self.attention_qkvpacked(hidden_states, cu_seqlens,
+                                                 max_seqlen, rope_freqs_cis)
+        hidden_states = residual + hidden_states
+
+        residual = hidden_states
+        hidden_states = self.norm1(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+
+        return hidden_states
+
+
+class MoonViT3dEncoder(nn.Module):
+
+    def __init__(self,
+                 hidden_dim: int,
+                 num_layers: int,
+                 block_cfg: dict,
+                 use_deterministic_attn: bool = False) -> None:
+        super().__init__()
+        self.use_deterministic_attn = use_deterministic_attn
+
+        qkv_hidden_size = block_cfg['hidden_dim'] if block_cfg.get(
+            'qkv_hidden_size') is None else block_cfg['qkv_hidden_size']
+        self.rope_2d = Rope2DPosEmbRepeated(
+            qkv_hidden_size // block_cfg['num_heads'], 512, 512)
+        self.blocks = nn.ModuleList([
+            MoonViTEncoderLayer(
+                **block_cfg,
+                use_deterministic_attn=self.use_deterministic_attn)
+            for _ in range(num_layers)
+        ])
+        norm_type = block_cfg.get('norm_type', 'layernorm')
+        if norm_type == "layernorm":
+            self.final_layernorm = nn.LayerNorm(hidden_dim)
+        elif norm_type == "rmsnorm":
+            self.final_layernorm = nn.RMSNorm(hidden_dim, eps=1e-6)
+        else:
+            raise NotImplementedError(f"Not support norm_type: {norm_type}")
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        grid_thws: torch.Tensor,
+    ) -> torch.Tensor:
+        rope_freqs_cis = self.rope_2d.get_freqs_cis(
+            grid_thws=grid_thws, device=hidden_states.device)
+
+        lengths = torch.cat((
+            torch.zeros(1, dtype=grid_thws.dtype, device=grid_thws.device),
+            grid_thws[:, 0] * grid_thws[:, 1] * grid_thws[:, 2],
+        ))
+
+        max_seqlen = lengths.max()
+        cu_seqlens = lengths.to(hidden_states.device).cumsum(dim=0,
+                                                             dtype=torch.int32)
+        for block in self.blocks:
+            hidden_states = block(hidden_states,
+                                  cu_seqlens,
+                                  max_seqlen,
+                                  rope_freqs_cis=rope_freqs_cis)
+
+        hidden_states = self.final_layernorm(hidden_states)
+        return hidden_states
+
+
+def tpool_patch_merger(
+        x: torch.Tensor,
+        grid_thws: torch.Tensor,
+        merge_kernel_size: tuple[int, int] = (2, 2),
+) -> list[torch.Tensor]:
+    d_model = x.size(-1)
+
+    outputs = []
+    pre_sum = 0
+    for t, h, w in grid_thws.tolist():
+        # Get the current sequence
+        seq = x[pre_sum:pre_sum + t * h * w]
+        # Reshape along self.merge_kernel_size and concat to the last dimension
+        kernel_height, kernel_width = merge_kernel_size
+        new_height, new_width = h // kernel_height, w // kernel_width
+        reshaped_seq = seq.view(t, new_height, kernel_height, new_width,
+                                kernel_width, d_model)
+        reshaped_seq = reshaped_seq.permute(0, 1,
+                                            3, 2, 4, 5).contiguous().mean(
+                                                dim=0)  # temporal pooling
+        padded_seq = reshaped_seq.view(new_height * new_width,
+                                       kernel_height * kernel_width, -1)
+        outputs.append(padded_seq)
+        pre_sum += t * h * w
+
+    return outputs
+
+
+class MoonViT3dPretrainedModel(PreTrainedModel):
+    config_class = None
+    model_type = 'moonvit3d'
+    _no_split_modules = ['MoonViTEncoderLayer']
+    _supports_flash_attn = True
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+
+    def __init__(self, config, *inputs, **kwargs):
+        super().__init__(config, *inputs, **kwargs)
+        config = deepcopy(config)
+        self.merge_kernel_size = config.merge_kernel_size
+        self.patch_size = config.patch_size
+        self.merge_type = config.merge_type
+
+        self.patch_embed = MoonVision3dPatchEmbed(
+            out_dim=config.hidden_size,
+            patch_size=config.patch_size,
+            pos_emb_height=config.init_pos_emb_height,
+            pos_emb_width=config.init_pos_emb_width,
+            pos_emb_time=config.init_pos_emb_time,
+            pos_emb_type=config.pos_emb_type,
+            patch_embed_proj_bias=getattr(config, 'patch_embed_proj_bias',
+                                          True),
+            pos_emb_interpolation_mode=getattr(
+                config, 'pos_emb_interpolation_mode', 'bicubic'),
+        )
+
+        self.encoder = MoonViT3dEncoder(
+            hidden_dim=config.hidden_size,
+            num_layers=config.num_hidden_layers,
+            block_cfg={
+                'num_heads': config.num_attention_heads,
+                'hidden_dim': config.hidden_size,
+                'qkv_hidden_size': getattr(config, 'qkv_hidden_size', None),
+                'mlp_dim': config.intermediate_size,
+                'norm_type': getattr(config, 'norm_type', 'layernorm'),
+                'mlp_type': getattr(config, 'mlp_type', 'mlp2'),
+                'activation': PytorchGELUTanh(),
+                'attn_bias': getattr(config, 'attn_bias', True),
+                'linear_bias': getattr(config, 'linear_bias', True),
+                'attn_implementation': config._attn_implementation,
+            },
+            use_deterministic_attn=getattr(self, 'use_deterministic_attn',
+                                           False))
+
+    def forward(self, pixel_values: torch.Tensor,
+                grid_thws: torch.Tensor) -> torch.Tensor:
+        """
+        Args:
+            pixel_values (torch.Tensor): The input pixel values.
+            grid_thws (torch.Tensor): Temporal, height and width.
+
+        Returns:
+            torch.Tensor: The output tokens.
+        """
+        # grid_thws = grid_thws.to('cpu')
+        assert grid_thws.ndim == 2, f'grid_thws should be 2D, got {grid_thws.ndim}'
+        assert grid_thws.size(1) == 3, f'No support for thw: {grid_thws}'
+        hidden_states = self.patch_embed(pixel_values, grid_thws)
+        hidden_states = self.encoder(hidden_states, grid_thws)
+        if self.merge_type == 'sd2_tpool':  # spatial downsampling 2x with temporal pooling all
+            hidden_states = tpool_patch_merger(
+                hidden_states,
+                grid_thws,
+                merge_kernel_size=self.merge_kernel_size)
+        else:
+            raise NotImplementedError(f'Not support {self.merge_type}')
+
+        return hidden_states
+
+
+# ============================================================================
+# MM Projector Helper Classes (from mm_projector/modeling_mm_projectors.py)
+# ============================================================================
+
+
+class IdentityMap(nn.Module):
+
+    def __init__(self):
+        super().__init__()
+
+    def forward(self, x, *args, **kwargs):
+        return x
+
+
+class MLP(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        # TODO, use faster LayerNorm
+        self.pre_norm = nn.LayerNorm(config.mm_hidden_size)
+        self.proj = nn.Sequential(
+            nn.Linear(config.mm_hidden_size, config.hidden_size), nn.GELU(),
+            nn.Linear(config.hidden_size, config.hidden_size))
+
+    def forward(self, x, *args, **kwargs):
+        assert isinstance(x,
+                          list | tuple), f'x is not a list or tuple: {type(x)}'
+        lengths = [item.shape[0] for item in x]
+        x = torch.cat(x, dim=0)
+        x = self.pre_norm(x)
+        x = self.proj(x)
+        x = torch.split(x, lengths, dim=0)
+
+        return x
+
+
+class PatchMergerMLP(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        eps = config.projector_ln_eps
+        self.hidden_size = config.mm_hidden_size * (
+            config.merge_kernel_size[0] * config.merge_kernel_size[1])
+        self.pre_norm = nn.LayerNorm(config.mm_hidden_size, eps=eps)
+        self.proj = nn.Sequential(
+            nn.Linear(self.hidden_size, self.hidden_size),
+            nn.GELU(),
+            nn.Linear(self.hidden_size, config.hidden_size),
+        )
+
+    def forward(self, x, *args, **kwargs):
+        if isinstance(x, list) or isinstance(x, tuple):
+            x = [
+                self.proj(self.pre_norm(item).view(item.shape[0], -1))
+                for item in x
+            ]
+        else:
+            # B, N, N_k, C = x.shape
+            B = x.shape[0]
+            x = self.proj(self.pre_norm(x).view(B, -1, self.hidden_size))
+        return x
+
+
+class PatchMergerMLPV2(nn.Module):
+
+    def __init__(self, config):
+        super().__init__()
+        eps = config.projector_ln_eps
+        self.hidden_size = config.mm_hidden_size * (
+            config.merge_kernel_size[0] * config.merge_kernel_size[1])
+        self.proj = nn.Sequential(
+            nn.Linear(self.hidden_size, self.hidden_size, bias=False),
+            nn.GELU(),
+            nn.Linear(self.hidden_size, config.hidden_size, bias=False),
+        )
+        self.post_norm = nn.RMSNorm(config.hidden_size, eps=eps)
+        for m in self.proj.modules():
+            if isinstance(m, nn.Linear):
+                nn.init.trunc_normal_(m.weight,
+                                      std=math.sqrt(2 / m.in_features))
+                if m.bias is not None:
+                    nn.init.zeros_(m.bias)
+
+    def forward(self, x, *args, **kwargs):
+        if isinstance(x, list) or isinstance(x, tuple):
+            lengths = [item.shape[0] for item in x]
+            x = torch.concat([item.view(item.shape[0], -1) for item in x],
+                             dim=0)
+            x = self.post_norm(self.proj(x))
+            x = torch.split(x, lengths, dim=0)
+        else:
+            # B, N, N_k, C = x.shape
+            B = x.shape[0]
+            x = self.proj(x.view(B, -1, self.hidden_size))
+            x = self.post_norm(x)
+        return x
+
+
+class GroundAnythingBackbonePreTrainedModel(PreTrainedModel):
+    config_class = GroundAnythingBackboneConfig
+    base_model_prefix = "model"
+    _no_split_modules = [
+        "MoonViT3dPretrainedModel",
+        "MoonViTEncoderLayer",
+        "KimiDecoderLayer",
+        "PatchMergerMLP",
+        "PatchMergerMLPV2",
+    ]
+    _skip_keys_device_placement = "past_key_values"
+    _supports_flash_attn_2 = True
+    _supports_sdpa = False
+
+    def _init_weights(self, module):
+        # important: this ported version of Llava isn't meant for training from scratch - only
+        # inference and fine-tuning - so the proper init weights code has been removed - the original codebase
+        # https://github.com/haotian-liu/LLaVA/tree/main/llava should serve for that purpose
+        std = (self.config.initializer_range if hasattr(
+            self.config, "initializer_range") else
+               self.config.text_config.initializer_range)
+
+        if hasattr(module, "class_embedding"):
+            module.class_embedding.data.normal_(mean=0.0, std=std)
+
+        if isinstance(module, (nn.Linear, nn.Conv2d)):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=std)
+            if module.padding_idx is not None:
+                module.weight.data[module.padding_idx].zero_()
+
+
+class VisionTowerConfig(PretrainedConfig):
+    model_type = 'moonvit3d'
+
+    def __init__(self, config: GroundAnythingBackboneConfig, **kwargs):
+        super().__init__(**kwargs)
+        self.patch_size = config.patch_size
+        self.init_pos_emb_height = config.init_pos_emb_height
+        self.init_pos_emb_width = config.init_pos_emb_width
+        self.init_pos_emb_time = config.init_pos_emb_time
+        self.pos_emb_type = config.pos_emb_type
+        self.num_attention_heads = config.vt_num_attention_heads
+        self.num_hidden_layers = config.vt_num_hidden_layers
+        self.hidden_size = config.vt_hidden_size
+        self.intermediate_size = config.vt_intermediate_size
+        self.merge_kernel_size = config.merge_kernel_size
+        self.merge_type = config.merge_type
+        self._attn_implementation = config._attn_implementation
+        self.qkv_hidden_size = getattr(config, 'qkv_hidden_size', None)
+        self.norm_type = getattr(config, 'norm_type', 'layernorm')
+        self.attn_bias = getattr(config, 'attn_bias', True)
+        self.patch_embed_proj_bias = getattr(config, 'patch_embed_proj_bias',
+                                             True)
+        self.mlp_type = getattr(config, 'mlp_type', 'mlp2')
+        self.linear_bias = getattr(config, 'linear_bias', True)
+        self.pos_emb_interpolation_mode = getattr(
+            config, 'pos_emb_interpolation_mode', 'bilinear')
+
+
+class ProjectorConfig:
+
+    def __init__(self, config: GroundAnythingBackboneConfig):
+        self.mm_projector_type = config.mm_projector_type
+        self.mm_hidden_size = config.mm_hidden_size
+        self.hidden_size = config.text_hidden_size
+        self.merge_kernel_size = config.merge_kernel_size
+        self.projector_hidden_act = config.projector_hidden_act
+        self.projector_ln_eps = config.projector_ln_eps
+
+
+# ref https://github.com/huggingface/transformers/blob/78b2929c0554b79e0489b451ce4ece14d265ead2/src/transformers/models/llava/modeling_llava.py#L240
+class GroundAnythingBackboneForConditionalGeneration(GroundAnythingBackbonePreTrainedModel):
+
+    @classmethod
+    def _supports_default_dynamic_cache(cls) -> bool:
+        return False
+
+    def __init__(self, config: GroundAnythingBackboneConfig):
+        super().__init__(config)
+
+        vt_config = VisionTowerConfig(config.vision_config)
+        self.vision_tower = MoonViT3dPretrainedModel(vt_config)
+
+        proj_config = ProjectorConfig(config.vision_config)
+        if proj_config.mm_projector_type == 'identity':
+            self.mm_projector = IdentityMap()
+        elif proj_config.mm_projector_type == 'mlp':
+            self.mm_projector = MLP(proj_config)
+        elif proj_config.mm_projector_type == 'patchmerger':
+            self.mm_projector = PatchMergerMLP(proj_config)
+        elif proj_config.mm_projector_type == 'patchmergerv2':
+            self.mm_projector = PatchMergerMLPV2(proj_config)
+        else:
+            raise ValueError(
+                f"Unsupported mm_projector_type: {proj_config.mm_projector_type}"
+            )
+
+        self.language_model = GroundAnythingBackboneLinearForCausalLM(config.text_config)
+        self.post_init()
+
+        if hasattr(self.language_model, 'dtype'):
+            target_dtype = self.language_model.dtype
+            self.vision_tower = self.vision_tower.to(dtype=target_dtype)
+            self.mm_projector = self.mm_projector.to(dtype=target_dtype)
+
+    def get_input_embeddings(self):
+        return self.language_model.get_input_embeddings()
+
+    def set_input_embeddings(self, value):
+        self.language_model.set_input_embeddings(value)
+
+    def get_output_embeddings(self):
+        return self.language_model.get_output_embeddings()
+
+    def set_output_embeddings(self, new_embeddings):
+        self.language_model.set_output_embeddings(new_embeddings)
+
+    def set_decoder(self, decoder):
+        self.language_model.set_decoder(decoder)
+
+    def get_decoder(self):
+        return self.language_model.get_decoder()
+
+    def tie_weights(self):
+        return self.language_model.tie_weights()
+
+    def resize_token_embeddings(self,
+                                new_num_tokens: int | None = None,
+                                pad_to_multiple_of=None) -> nn.Embedding:
+        model_embeds = self.language_model.resize_token_embeddings(
+            new_num_tokens, pad_to_multiple_of)
+        # update vocab size
+        self.config.text_config.vocab_size = model_embeds.num_embeddings
+        self.vocab_size = model_embeds.num_embeddings
+        return model_embeds
+
+    def _merge_input_ids_with_image_features(
+        self,
+        image_features: list[torch.Tensor],
+        inputs_embeds: torch.Tensor,
+        input_ids: torch.Tensor,
+        attention_mask: torch.Tensor,
+        labels: torch.Tensor | None = None,
+    ):
+        """
+        Args:
+            image_features (:obj:`torch.Tensor` of shape :obj:`(num_image_tokens, embed_dim)`):
+                The image features to merge with the input embeddings.
+            inputs_embeds (:obj:`torch.Tensor` of shape :obj:`(batch_size, sequence_length, embed_dim)`):
+                The input embeddings.
+            input_ids (:obj:`torch.Tensor` of shape :obj:`(batch_size, sequence_length)`):
+                The input ids.
+            attention_mask (:obj:`torch.Tensor` of shape :obj:`(batch_size, sequence_length)`):
+                The attention mask.
+            labels (:obj:`torch.Tensor` of shape :obj:`(batch_size, sequence_length)`, *optional*):
+                The labels.
+        """
+        _, embed_dim = image_features[0].shape
+        feature_lengths = [x.shape[0] for x in image_features]
+        image_features = torch.cat(image_features, dim=0)
+
+        image_token_index: int = self.config.media_placeholder_token_id
+        pad_token_id: int = self.config.pad_token_id
+        ignore_index: int = self.config.ignore_index
+
+        batch_size, sequence_length = input_ids.shape
+        left_padding = not torch.sum(
+            input_ids[:, -1] == torch.tensor(pad_token_id))
+
+        # 1. Create a mask to know where special image tokens are
+        _token_occupation_table = torch.ones_like(input_ids.flatten())
+        _token_occupation_table[input_ids.flatten() ==
+                                image_token_index] = torch.tensor(
+                                    feature_lengths,
+                                    dtype=torch.long,
+                                    device=input_ids.device)
+        _token_occupation_table = _token_occupation_table.reshape(
+            input_ids.shape)
+
+        max_embed_dim = _token_occupation_table.sum(-1).max().item()
+        assert (
+            max_embed_dim >= sequence_length
+        ), f"The maximum embedding dimension ({max_embed_dim}) is less than the sequence length ({sequence_length})"
+        batch_indices, non_image_indices = torch.where(
+            input_ids != image_token_index)
+
+        # 2. Compute the positions where text should be written
+        # Calculate new positions for text tokens in merged image-text sequence.
+        new_token_positions = torch.cumsum(_token_occupation_table, -1) - 1
+        nb_image_pad = max_embed_dim - 1 - new_token_positions[:, -1]
+        if left_padding:
+            new_token_positions += nb_image_pad[:,
+                                                None]  # offset for left padding
+        text_to_overwrite = new_token_positions[batch_indices,
+                                                non_image_indices]
+
+        # 3. Create the full embedding, already padded to the maximum position
+        final_embedding = torch.zeros(
+            batch_size,
+            max_embed_dim,
+            embed_dim,
+            dtype=inputs_embeds.dtype,
+            device=inputs_embeds.device,
+        )
+        final_attention_mask = torch.zeros(batch_size,
+                                           max_embed_dim,
+                                           dtype=attention_mask.dtype,
+                                           device=inputs_embeds.device)
+        if labels is not None:
+            final_labels = torch.full(
+                (batch_size, max_embed_dim),
+                ignore_index,
+                dtype=input_ids.dtype,
+                device=input_ids.device,
+            )
+        # In case the Vision model or the Language model has been offloaded to CPU, we need to manually
+        # set the corresponding tensors into their correct target device.
+        target_device = inputs_embeds.device
+        batch_indices, non_image_indices, text_to_overwrite = (
+            batch_indices.to(target_device),
+            non_image_indices.to(target_device),
+            text_to_overwrite.to(target_device),
+        )
+        attention_mask = attention_mask.to(target_device)
+
+        # 4. Fill the embeddings based on the mask.
+        final_embedding[batch_indices,
+                        text_to_overwrite] = inputs_embeds[batch_indices,
+                                                           non_image_indices]
+        final_attention_mask[batch_indices,
+                             text_to_overwrite] = attention_mask[
+                                 batch_indices, non_image_indices]
+        if labels is not None:
+            final_labels[batch_indices,
+                         text_to_overwrite] = labels[batch_indices,
+                                                     non_image_indices]
+
+        # 5. Fill the embeddings corresponding to the images. Anything that is not `text_positions` needs filling (#29835)
+        image_to_overwrite = torch.full((batch_size, max_embed_dim),
+                                        True,
+                                        dtype=torch.bool,
+                                        device=inputs_embeds.device)
+        image_to_overwrite[batch_indices, text_to_overwrite] = False
+        image_to_overwrite &= image_to_overwrite.cumsum(
+            -1) - 1 >= nb_image_pad[:, None].to(target_device)
+
+        if image_to_overwrite.sum() != image_features.shape[:-1].numel():
+            raise ValueError(
+                f"The input provided to the model are wrong. The number of image tokens is {image_to_overwrite.sum()} while"
+                f" the number of image features given to the model is {image_features.shape[:-1].numel()}. "
+                "This prevents correct indexing and breaks batch generation.")
+
+        final_embedding[image_to_overwrite] = (
+            image_features.contiguous().reshape(-1,
+                                                embed_dim).to(target_device))
+        final_attention_mask |= image_to_overwrite
+        position_ids = (final_attention_mask.cumsum(-1) - 1).masked_fill_(
+            (final_attention_mask == 0), 1)
+
+        # 6. Mask out the embedding at padding positions, as we later use the past_key_value value to determine the non-attended tokens.
+        batch_indices, pad_indices = torch.where(input_ids == pad_token_id)
+        indices_to_mask = new_token_positions[batch_indices, pad_indices]
+
+        final_embedding[batch_indices, indices_to_mask] = 0
+
+        if labels is None:
+            final_labels = None
+
+        return final_embedding, final_attention_mask, final_labels, position_ids
+
+    def _extract_image_features(self, pixel_values: torch.Tensor,
+                                grid_thws: torch.Tensor) -> list[torch.Tensor]:
+        """
+        Args:
+            pixel_values (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, num_channels, height, width)`):
+                The pixel values of the images processed by image processor.
+            grid_thws (:obj:`torch.Tensor` of shape :obj:`(batch_size, 3)`):
+                The grid, height, width of the images.
+
+        Returns:
+            selected_image_feature (:obj:`torch.FloatTensor` of shape :obj:`(num_image_tokens, embed_dim)`):
+                The selected image features to use as input to the projector head.
+
+        """
+
+        target_dtype = self.vision_tower.patch_embed.proj.weight.dtype
+        pixel_values = pixel_values.to(target_dtype)
+
+        image_features = self.vision_tower(pixel_values, grid_thws)
+        return image_features
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        pixel_values: torch.FloatTensor | list[torch.FloatTensor]
+        | None = None,
+        grid_thws: torch.Tensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: list[torch.FloatTensor] | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        use_cache: bool | None = None,
+        output_attentions: bool | None = None,
+        output_hidden_states: bool | None = None,
+        return_dict: bool | None = None,
+    ) -> tuple | LlavaCausalLMOutputWithPast:
+        r"""
+        Args:
+            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+                Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
+                config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
+                (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
+
+        ```"""
+        assert self.vision_tower is not None, "vision_tower is not loaded"
+        output_attentions = (output_attentions if output_attentions is not None
+                             else self.config.output_attentions)
+        output_hidden_states = (output_hidden_states
+                                if output_hidden_states is not None else
+                                self.config.output_hidden_states)
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        if inputs_embeds is None:
+            # 1. Extra the input embeddings
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+
+            # 2. Merge text and images
+            if pixel_values is not None and len(
+                    pixel_values) > 0 and input_ids.shape[1] != 1:
+                image_features = self._extract_image_features(
+                    pixel_values, grid_thws)
+                if self.mm_projector:
+                    image_features = self.mm_projector(image_features)
+
+                inputs_embeds = inputs_embeds.to(
+                    image_features[0].dtype)  # num_tokens, embed_dim
+                inputs_embeds, attention_mask, labels, position_ids = (
+                    self._merge_input_ids_with_image_features(
+                        image_features,
+                        inputs_embeds,
+                        input_ids,
+                        attention_mask,
+                        labels,
+                    ))
+
+            # In case input_ids.shape[1] == 1 & pixel_values==None & past_key_values != None, we are in the case of
+            # generation with cache
+            elif (past_key_values is not None and pixel_values is not None
+                  and input_ids.shape[1] == 1):
+                # Retrieve the first layer to inspect the logits and mask out the hidden states
+                # that are set to 0
+                first_layer_past_key_value = past_key_values[0][0][:, :, :, 0]
+
+                # Sum all dimensions of head_dim (-2) to avoid random errors such as: https://github.com/huggingface/transformers/pull/28032#issuecomment-1863691941
+                batch_index, non_attended_tokens = torch.where(
+                    first_layer_past_key_value.float().sum(-2) == 0)
+
+                # Get the target length
+                target_length = input_ids.shape[1]
+                past_length = first_layer_past_key_value.shape[-1]
+
+                extended_attention_mask = torch.ones(
+                    (attention_mask.shape[0], past_length),
+                    dtype=attention_mask.dtype,
+                    device=attention_mask.device,
+                )
+
+                # Filter out only the tokens that can be un-attended, this can happen
+                # if one uses Llava + Fused modules where the cache on the
+                # first iteration is already big enough, or if one passes custom cache
+                valid_indices = non_attended_tokens < extended_attention_mask.size(
+                    -1)
+                new_batch_index = batch_index[valid_indices]
+                new_non_attended_tokens = non_attended_tokens[valid_indices]
+
+                # Zero-out the places where we don't need to attend
+                extended_attention_mask[new_batch_index,
+                                        new_non_attended_tokens] = 0
+
+                attention_mask = torch.cat(
+                    (extended_attention_mask, attention_mask[:,
+                                                             -target_length:]),
+                    dim=1)
+                position_ids = torch.sum(attention_mask,
+                                         dim=1).unsqueeze(-1) - 1
+
+        outputs = self.language_model(
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+
+        logits = outputs[0]
+
+        loss = None
+        if labels is not None:
+            # Shift so that tokens < n predict n
+            if attention_mask is not None:
+                shift_attention_mask = attention_mask[..., 1:]
+                shift_logits = logits[..., :-1, :][shift_attention_mask.to(
+                    logits.device) != 0].contiguous()
+                shift_labels = labels[..., 1:][shift_attention_mask.to(
+                    labels.device) != 0].contiguous()
+            else:
+                shift_logits = logits[..., :-1, :].contiguous()
+                shift_labels = labels[..., 1:].contiguous()
+            # Flatten the tokens
+            loss_fct = nn.CrossEntropyLoss()
+            loss = loss_fct(
+                shift_logits.view(-1, shift_logits.size(-1)),
+                shift_labels.view(-1).to(shift_logits.device),
+            )
+
+        if not return_dict:
+            output = (logits, ) + outputs[1:]
+            return (loss, ) + output if loss is not None else output
+
+        return LlavaCausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def prepare_inputs_for_generation(
+        self,
+        input_ids,
+        past_key_values=None,
+        inputs_embeds=None,
+        pixel_values=None,
+        grid_thws=None,
+        attention_mask=None,
+        **kwargs,
+    ):
+        if past_key_values is not None:
+            if hasattr(past_key_values, "get_seq_length"):
+                cache_length = past_key_values.get_seq_length()
+                past_length = getattr(past_key_values, 'seen_tokens',
+                                      cache_length)
+            else:
+                cache_length = past_length = past_key_values[0][0].shape[2]
+
+            # Keep only the unprocessed tokens:
+            # 1 - If the length of the attention_mask exceeds the length of input_ids, then we are in a setting where
+            # some of the inputs are exclusively passed as part of the cache (e.g. when passing input_embeds as
+            # input)
+            if attention_mask is not None and attention_mask.shape[
+                    1] > input_ids.shape[1]:
+                input_ids = input_ids[:, -(attention_mask.shape[1] -
+                                           past_length):]
+            # 2 - If the past_length is smaller than input_ids', then input_ids holds all input tokens. We can discard
+            # input_ids based on the past_length.
+            elif past_length < input_ids.shape[1]:
+                input_ids = input_ids[:, past_length:]
+            # 3 - Otherwise (past_length >= input_ids.shape[1]), let's assume input_ids only has unprocessed tokens.
+            elif self.config.media_placeholder_token_id in input_ids:
+                input_ids = input_ids[:, input_ids.shape[1] - 1:]
+            # If the cache has seen more tokens than it can hold, then the cache has a size limit. Let's discard the
+            # older attention values, as their corresponding values are not part of the input.
+            if cache_length < past_length and attention_mask is not None:
+                attention_mask = attention_mask[:, -(cache_length +
+                                                     input_ids.shape[1]):]
+
+        position_ids = kwargs.get("position_ids", None)
+        if attention_mask is not None and position_ids is None:
+            # create position_ids on the fly for batch generation
+            position_ids = attention_mask.long().cumsum(-1) - 1
+            position_ids.masked_fill_(attention_mask == 0, 1)
+            if past_key_values:
+                position_ids = position_ids[:, -input_ids.shape[1]:]
+
+        # if `inputs_embeds` are passed, we only want to use them in the 1st generation step
+        if inputs_embeds is not None and past_key_values is None:
+            model_inputs = {"inputs_embeds": inputs_embeds}
+        else:
+            model_inputs = {"input_ids": input_ids}
+
+        model_inputs.update({
+            "position_ids": position_ids,
+            "past_key_values": past_key_values,
+            "use_cache": kwargs.get("use_cache"),
+            "attention_mask": attention_mask,
+            "pixel_values": pixel_values,
+            "grid_thws": grid_thws,
+        })
+        return model_inputs
+
+    def _reorder_cache(self, *args, **kwargs):
+        return self.language_model._reorder_cache(*args, **kwargs)

preprocessor_config.json

@@ -0,0 +1,37 @@
+{
+  "media_proc_cfg": {
+    "in_patch_limit": 4096,
+    "patch_size": 14,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_kernel_size": 2,
+    "fixed_output_tokens": null,
+    "patch_limit_on_one_side": 512,
+    "in_patch_limit_each_frame": 16384,
+    "in_patch_limit_video": 655360,
+    "sample_fps": 8.0,
+    "max_num_frames_each_video": null,
+    "temporal_merge_kernel_size": 4,
+    "timestamp_mode": "hh:mm:ss.fff",
+    "transparent_bg_config": {
+      "pattern": "chessboard",
+      "chessboard_square_size": 8,
+      "chessboard_square_on_top_left": true,
+      "chessboard_white_value": 255,
+      "chessboard_gray_value": 180
+    },
+    "transparent_bg_fill_stage": "after_resize",
+    "config_type": "media_proc.processors.moonvit.MoonViTMediaProcessorConfig"
+  },
+  "auto_map": {
+    "AutoProcessor": "processing_groundinganything.GroundAnythingProcessor"
+  }
+}

processing_groundinganything.py

@@ -0,0 +1,144 @@
+"""Processor glue for GroundAnything-VLM with Kimi-K3 MoonViT preprocessing."""
+
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.processing_utils import ProcessorMixin
+
+from .media_utils import MediaInput
+from .image_processing_groundinganything import GroundAnythingVLMImageProcessor
+
+
+class GroundAnythingVLMProcessor(ProcessorMixin):
+    attributes = ["image_processor", "tokenizer"]
+    image_processor_class = "AutoImageProcessor"
+    tokenizer_class = "AutoTokenizer"
+
+    def __init__(self, image_processor=None, tokenizer=None, chat_template=None, **kwargs):
+        del kwargs
+        super().__init__(
+            image_processor=image_processor,
+            tokenizer=tokenizer,
+            chat_template=chat_template or getattr(tokenizer, "chat_template", None),
+        )
+
+    @property
+    def image_token(self):
+        return "<|image_pad|>"
+
+    @property
+    def image_token_id(self):
+        return self.tokenizer.convert_tokens_to_ids(self.image_token)
+
+    def _get_num_multimodal_tokens(self, image_sizes=None, **kwargs):
+        del kwargs
+        num_image_tokens = []
+        num_image_patches = []
+        for height, width in image_sizes or ():
+            image_stub = type("ImageSize", (), {"size": (width, height)})()
+            resize = self.image_processor.get_resize_config(
+                {"type": "image", "image": image_stub}
+            )
+            tokens = int(resize["num_tokens"])
+            num_image_tokens.append(tokens)
+            num_image_patches.append(tokens * self.image_processor.merge_size**2)
+        return {
+            "num_image_tokens": num_image_tokens,
+            "num_image_patches": num_image_patches,
+        }
+
+    @classmethod
+    def register_for_auto_class(cls, auto_class="AutoProcessor"):
+        cls._auto_class = auto_class
+
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
+        import json
+        import os
+        from transformers import AutoTokenizer
+
+        kwargs.pop("_from_auto", None)
+        kwargs.pop("trust_remote_code", None)
+        kwargs.pop("code_revision", None)
+        with open(os.path.join(pretrained_model_name_or_path, "preprocessor_config.json"), encoding="utf-8") as f:
+            processor_config = json.load(f)
+        image_processor = GroundAnythingVLMImageProcessor(
+            media_proc_cfg=processor_config["media_proc_cfg"]
+        )
+        tokenizer = AutoTokenizer.from_pretrained(
+            pretrained_model_name_or_path, trust_remote_code=True, **kwargs
+        )
+        return cls(image_processor=image_processor, tokenizer=tokenizer)
+
+    def apply_chat_template(self, messages, **kwargs):
+        if self.chat_template and "chat_template" not in kwargs:
+            kwargs["chat_template"] = self.chat_template
+        return self.tokenizer.apply_chat_template(messages, **kwargs)
+
+    def __call__(
+        self,
+        text=None,
+        images=None,
+        return_tensors="pt",
+        padding=False,
+        **kwargs,
+    ):
+        return_mm_token_type_ids = kwargs.pop("return_mm_token_type_ids", False)
+        if isinstance(text, str):
+            text = [text]
+
+        image_inputs = {}
+        if images is not None:
+            image_inputs = self.image_processor(
+                images=images,
+                return_tensors=return_tensors,
+            )
+            text = list(text)
+            image_index = 0
+            merge_length = self.image_processor.merge_size**2
+            for batch_index, prompt in enumerate(text):
+                while self.image_token in prompt:
+                    grid = image_inputs["image_grid_thw"][image_index]
+                    num_tokens = int(grid.prod().item()) // merge_length
+                    prompt = prompt.replace(
+                        self.image_token, "<|image_placeholder|>" * num_tokens, 1
+                    )
+                    image_index += 1
+                text[batch_index] = prompt.replace(
+                    "<|image_placeholder|>", self.image_token
+                )
+            if image_index != len(image_inputs["image_grid_thw"]):
+                raise ValueError(
+                    "number of image placeholders does not match image inputs"
+                )
+
+        text_inputs = self.tokenizer(
+            text,
+            return_tensors=return_tensors,
+            padding=padding,
+            **kwargs,
+        )
+        if return_mm_token_type_ids:
+            input_ids = text_inputs["input_ids"]
+            if hasattr(input_ids, "new_zeros"):
+                mm_token_type_ids = input_ids.new_zeros(input_ids.shape)
+                mm_token_type_ids[input_ids == self.image_token_id] = 1
+            else:
+                mm_token_type_ids = [
+                    [int(token == self.image_token_id) for token in row]
+                    for row in input_ids
+                ]
+            text_inputs["mm_token_type_ids"] = mm_token_type_ids
+
+        return BatchFeature(data={**text_inputs, **image_inputs})
+
+    def batch_decode(self, *args, **kwargs):
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        return self.tokenizer.decode(*args, **kwargs)
+
+
+__all__ = ["GroundAnythingVLMProcessor"]
+
+
+class GroundAnythingProcessor(GroundAnythingVLMProcessor):
+    """DLM release processor identity."""

requirements.txt

@@ -0,0 +1,4 @@
+transformers>=5.3,<5.8
+huggingface-hub>=1.3,<2
+safetensors>=0.6
+torch>=2.8

streammind_gate.py

@@ -0,0 +1,132 @@
+from dataclasses import dataclass
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from mamba_ssm.models.mixer_seq_simple import create_block
+from transformers import Qwen3Config
+from transformers.models.qwen3 import Qwen3ForCausalLM
+
+
+class PreNet(nn.Module):
+    def __init__(self, d_code, d_model):
+        super().__init__()
+        self.fc3 = nn.Linear(d_code, d_model)
+
+    def forward(self, x):
+        return F.leaky_relu(self.fc3(x))
+
+
+class PostNet(nn.Module):
+    def __init__(self, d_model, n_class):
+        super().__init__()
+        self.fc3 = nn.Linear(d_model, n_class)
+
+    def forward(self, x):
+        return self.fc3(F.leaky_relu(x))
+
+
+@dataclass
+class SSMConfig:
+    d_model: int = 2560
+    n_ssm: int = 1
+
+
+class VideoMamba(nn.Module):
+    def __init__(self, config):
+        super().__init__()
+        self.ssms = nn.ModuleList(
+            [create_block(config.d_model, d_intermediate=0, layer_idx=i) for i in range(config.n_ssm)]
+        )
+        self.norm_fn = nn.LayerNorm(config.d_model)
+
+    def forward(self, embeds, inference_params=None):
+        hidden_states = embeds
+        residual = None
+        for ssm in self.ssms:
+            hidden_states, residual = ssm(
+                hidden_states, residual, inference_params=inference_params
+            )
+        residual = hidden_states + residual if residual is not None else hidden_states
+        return self.norm_fn(residual.to(dtype=self.norm_fn.weight.dtype))
+
+
+class Qwen3ForCausalLMCls(Qwen3ForCausalLM):
+    def forward(self, inputs_embeds=None, labels=None, attention_mask=None, **kwargs):
+        outputs = self.model(inputs_embeds=inputs_embeds, attention_mask=attention_mask)
+        logits = self.lm_head(outputs.last_hidden_state).float()
+        loss = None
+        if labels is not None:
+            shift_logits = logits[..., :-1, :].contiguous().view(-1, self.config.vocab_size)
+            shift_labels = labels[..., 1:].contiguous().view(-1).to(shift_logits.device)
+            loss = nn.CrossEntropyLoss(
+                weight=torch.tensor([0.15, 0.85], device=shift_logits.device)
+            )(shift_logits, shift_labels)
+        return {"loss": loss, "logits": logits}
+
+
+class ClsNet(nn.Module):
+    def __init__(self, hidden_size=2560, num_layers=4):
+        super().__init__()
+        config = Qwen3Config(
+            vocab_size=2,
+            hidden_size=hidden_size,
+            num_hidden_layers=num_layers,
+            num_attention_heads=32,
+            num_key_value_heads=8,
+            intermediate_size=12288,
+            head_dim=128,
+            max_position_embeddings=8192,
+            rms_norm_eps=1e-6,
+            tie_word_embeddings=False,
+            attention_bias=False,
+        )
+        self.cls_model = Qwen3ForCausalLMCls(config)
+
+    def forward(self, x, labels=None, attention_mask=None):
+        return self.cls_model(inputs_embeds=x, labels=labels, attention_mask=attention_mask)
+
+
+class StreamMindGate(nn.Module):
+    def __init__(self, hidden_size=2560):
+        super().__init__()
+        self.pre_net = PreNet(hidden_size, hidden_size)
+        self.mamba_model = VideoMamba(SSMConfig(d_model=hidden_size))
+        self.post_net = PostNet(hidden_size, hidden_size)
+        self.cls_net = ClsNet(hidden_size=hidden_size, num_layers=4)
+
+    def perception_tokens(self, vision_tokens):
+        """Convert [B,T,P,D] visual patches to one EPFE token per time step."""
+        x = vision_tokens.mean(dim=2)
+        batch, time, dim = x.shape
+        x = self.pre_net(x.reshape(batch * time, dim)).reshape(batch, time, dim)
+        x = self.mamba_model(x)
+        x = self.post_net(x.reshape(batch * time, dim)).reshape(batch, time, dim)
+        return x
+
+    def forward(self, vision_tokens, response_positions=None):
+        """Return [B,T,2] silent/speak logits for every EPFE time step."""
+        tokens = self.perception_tokens(vision_tokens)
+        batch, time, dim = tokens.shape
+        target_ids = torch.zeros(batch, time, dtype=torch.long, device=tokens.device)
+        if response_positions is not None:
+            target_ids[:, torch.as_tensor(response_positions, device=tokens.device) - 1] = 1
+        targets = self.cls_net.cls_model.model.embed_tokens(
+            target_ids.reshape(batch * time)
+        )
+        pair = torch.stack((tokens.reshape(batch * time, dim), targets), dim=1)
+        rotary = self.cls_net.cls_model.model.rotary_emb
+        saved_inv_freq = rotary.inv_freq
+        try:
+            # Match the training checkpoint, where the full model (including
+            # non-persistent Qwen3 RoPE buffers) was cast to BF16.
+            rotary.inv_freq = rotary.inv_freq.to(pair.dtype)
+            output = self.cls_net(
+                pair,
+                attention_mask=torch.ones(pair.shape[:2], device=pair.device),
+            )
+        finally:
+            rotary.inv_freq = saved_inv_freq
+        # Autoregressive shift: position 0 predicts the target token at
+        # position 1, matching StreamMind's logits[..., :-1, :] evaluation.
+        return output["logits"][:, 0].reshape(batch, time, 2)

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e0398aca93659140fd6345deb335db2330dee89a2aad1228d8604c1952f6dc8
+size 11604906