SkyContact/MOSS-VL-Realtime-GGUF

Moss-VL-Realtime GGUF (Q4_K)

GGUF quantization of OpenMOSS-Team/MOSS-VL-Realtime-NF4 (11B cross-attention VLM: Qwen3 text trunk with 12 pure cross-attn layers + SigLIP tower with deepstack), repackaged for the SkyVision.cpp inference engine.

Built for SkyVision.cpp only. The GGUF declares general.architecture = moss_vl; llama.cpp / ollama / LM Studio and other generic GGUF runtimes cannot load it. This repository is not part of the generic GGUF ecosystem - do not use it as a signal of whether the model "works".

Files

File	Size	Notes
Moss-VL-Realtime-q4k.gguf	8.4 GB	NF4 → Q4_K (linears) + Q6_K (norm-adjacent) + F16 (vision), fits a 12 GB GPU

Quantization scheme

• NF4 weights are dequantized bit-exactly against bitsandbytes CUDA first (check_moss_vl_nf4.py);
• then requantized: cross-attn / norm-adjacent linears → Q6_K, the rest → Q4_K, vision tower + 1D tensors kept F16/F32;
• the fused cross-KV projection weight is built at load time (engine-side);
• verified against the official stack: image prefill token stream 24/24 and video 78/78 token-exact, layer-2 cross-attention internals corr ≥ 0.999, KV-cache readback vs official roped K corr 0.9997.

Usage

vqa-cli --model Moss-VL-Realtime-q4k.gguf --image photo.jpg --ask "Describe this image."
vqa-cli --model Moss-VL-Realtime-q4k.gguf --video clip.mp4 --max-frames 8 --ask "What happens?"

12 GB GPU deployments: --n-ctx 4096 (and MOSS_XCTX=6144 for long videos). Full chat, multi-turn sessions, realtime streaming (proactive narration with <|silence|>) and a web UI are available via --serve.

License

Follows the base model's license (see the OpenMOSS-Team/MOSS-VL-Realtime-NF4 page).