qijithu/Quicksviewer-8B

Quicksviewer: An LMM for Efficient Video Understanding via Reinforced Compression of Video Cubes

arXiv / Code / Models / Project

Quicksviewer is a Large Multimodal Model (LMM) that can efficiently understand videos with extensive frames as input. Quicksviewer adopts a new perceiving paradigm that partitions a video of nonuniform density into varying cubes using Gumbel Softmax, followed by a unified resampling for each cube to achieve dynamic online compression. This directory is a self-contained Hugging Face repository for the QuickSViewer model.

Default model contract

• Temporal resampler: 64 queries, 4096 output width, additive absolute 0.1-second temporal embeddings.
• Gumbel cubing: full-depth SigLIP boundary features (forward_n_layers=-1), FPQ 5, deterministic inference (lr_gumbel=0), thumbnail last.

Files

• configuration_quicksviewer.py: composite Qwen/vision configuration and Transformers auto registration.
• modeling_quicksviewer.py: QuickSViewerForConditionalGeneration, standard generate() and KV cache, visual embedding replacement, timestamp patching, and encode_vision() deployment boundary.
• modeling_siglip.py: standalone SigLIP ViT implementation adapted from the verified training path.
• vision_quicksviewer.py, cubing.py, and video_processing_quicksviewer.py: the complete QSV visual path. cubing.py owns cube-layout math; video_processing_quicksviewer.py owns video sampling, timestamp-slot utilities, and the temporal Perceiver resampler.
• processing_quicksviewer.py, image/video processors, tokenizer resources, and chat_template.jinja: external media/chat preprocessing.
• model-*.safetensors, model.safetensors.index.json: converted iter50 weights.

from transformers import AutoModelForImageTextToText, AutoProcessor
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo_id, trust_remote_code=True, torch_dtype="auto", device_map="auto")

Transformers loading

Requires Python, PyTorch, Transformers, safetensors, NumPy, Pillow, and Decord for video paths. Validation was performed with Transformers 4.57.1. Use trust_remote_code=True because the architecture is custom.

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

path = "/path/to/quicksviewer_8b_hf"
processor = AutoProcessor.from_pretrained(path, trust_remote_code=True)
model, info = AutoModelForCausalLM.from_pretrained(
    path,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",       # flash_attention_2 can be selected if available
    device_map="cuda:0",
    output_loading_info=True,
)
assert not info["missing_keys"] and not info["unexpected_keys"]
model.eval()

Text

messages = [{"role": "user", "content": "你好，请只回答：你好"}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    add_generation_prompt=True,
    enable_thinking=False,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=32, do_sample=False, use_cache=True)
print(processor.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Image

from PIL import Image

messages = [{"role": "user", "content": [
    {"type": "image", "image": Image.open("example.jpg")},
    {"type": "text", "text": "描述图片。"},
]}]
inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True, enable_thinking=False,
)
# Pixel values intentionally remain uint8. For very long videos, leave them on CPU;
# the model moves/normalizes frames in bounded chunks. Move tensor metadata separately.
for key, value in list(inputs.items()):
    if key != "pixel_values" and hasattr(value, "to"):
        inputs[key] = value.to(model.device)
output = model.generate(**inputs, max_new_tokens=128, do_sample=False)

Video

messages = [{"role": "user", "content": [
    {"type": "video", "video": "/path/to/video.mp4"},
    {"type": "text", "text": "总结视频内容。"},
]}]
inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True, enable_thinking=False,
    max_frames=360,
)
for key, value in list(inputs.items()):
    if key != "pixel_values" and hasattr(value, "to"):
        inputs[key] = value.to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False, use_cache=True)

model.chat(processor, messages, ...) is provided as a compact convenience API. The explicit processor/generate path is recommended for serving integrations.

Processing behavior

• Supports PIL, NumPy, Torch images, local paths, and HTTP(S) image URLs.
• Supports local/HTTP video paths and already decoded frame sequences.
• Video path decoding uses Decord resize-to-980 and uniform linspace frame sampling, matching the current serving path.
• Pixels remain CPU uint8 until bounded model-side frame chunks are normalized.
• Truncation is intentionally rejected because it can split visual/timestamp spans.
• Multimodal batching greater than one is intentionally rejected; SGLang batching should concatenate request-level vision features in its native adapter rather than changing the HF checkpoint format.