Instructions to use SamChen567/phase1_qbase_internvid with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SamChen567/phase1_qbase_internvid with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="SamChen567/phase1_qbase_internvid", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("SamChen567/phase1_qbase_internvid", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use SamChen567/phase1_qbase_internvid with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SamChen567/phase1_qbase_internvid"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase1_qbase_internvid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SamChen567/phase1_qbase_internvid

SGLang

How to use SamChen567/phase1_qbase_internvid with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SamChen567/phase1_qbase_internvid" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase1_qbase_internvid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SamChen567/phase1_qbase_internvid" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase1_qbase_internvid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use SamChen567/phase1_qbase_internvid with Docker Model Runner:
```
docker model run hf.co/SamChen567/phase1_qbase_internvid
```

phase1_qbase_internvid / README.md

SamChen567

Add model card

c3b18fc verified 22 days ago

preview code

Raw History Blame Contribute Delete

4.91 kB

metadata

library_name: transformers
base_model: DAMO-NLP-SG/VideoLLaMA3-7B
tags:
  - video
  - token-compression
  - videollama3

phase1_qbase_internvid (native-res, video-only, 2 epochs)

Phase 1 of the qbase video-token compressor curriculum: a learned-query compressor ("qbase") inserted between the frozen SigLIP-NaViT vision encoder and the frozen Qwen2-7B LLM of VideoLLaMA3-7B, trained with caption cross-entropy.

This repo was re-used. The repo name previously referred to an older Phase-1 run (16384-token video budget, 40 % still images, 3 epochs / 3,441 steps); no weights from that run were ever uploaded here. Everything in this repo is the run described below.

This checkpoint needs the custom code in the training repository (videollama3/model, Videollama3Qwen2ForCausalLM with token_compressor); it does not load with stock transformers alone.

What is trained

module	status
vision encoder (SigLIP-NaViT)	frozen
`token_compressor` (qbase, 170M)	trained
`mm_projector` (mlp2x_gelu, 17M)	trained
`<\|compression_start\|>` / `<\|compression_end\|>` embedding rows	trained (all other rows frozen)
LLM (Qwen2-7B) + `lm_head`	frozen

Warm start: SamChen567/compressor_pretrain_video_norm (qbase + projector + the two embedding rows).

Compressor

transformer_decoder_flat: 8 layers, 8 heads, hidden 1152, K = 64 learned queries (+ sin/cos position encoding); cross-attention onto the segment's encoder tokens with (t, h, w) RoPE.
Adaptive segmentation (non-learned): N = ⌊T/4⌋ + 1 segments per video, a forced cut every 8 frames, remaining cuts at the largest consecutive-frame cosine distance of the per-frame mean encoder feature. Each segment is 1–8 frames. Deterministic top-k (segment_sample_tau = 0).
Option A (match_encoder_scale): output rescaled to the encoder tokens' per-dim mean/std, then a learnable per-channel γ/β.
Option B (distr_loss_weight = 0.05): CORAL-style distribution-match auxiliary loss (see the note on its effective weight below).

Input layout seen by the LLM

Each segment carries its own time range (segment_time_marks = True); the per-frame Time X.0s: text of the uncompressed template is removed:

<|im_start|>user
Time:0s-3s:<|compression_start|>[64]<|compression_end|>Time:4s-4s:<|compression_start|>[64]<|compression_end|> ... 
{prompt}<|im_end|>
<|im_start|>assistant
{caption}<|im_end|>

Seconds are integers (first / last frame of the segment). Example: a 120 s clip at 1 fps → 31 segments → 1,984 compressed tokens (vs. 35,880 uncompressed at 640×360). Checkpoints trained before 2026-10-12 used one whole-video range instead; build the layout from this checkpoint's config.json.

Data

InternVid clips < 180 s, 264,186 samples, video only (no still images).
Captions re-generated by Qwen3-VL-8B-Instruct at 1 fps, native resolution, with several prompt styles (baseline_default, scene_shift, motion_scene, ...).
One turn per sample: <video>\n{prompt} → caption. Loss on the assistant caption only.

Vision geometry

fps = 1, max_frames = 180.
Native resolution: per-video budget 53,820 merged tokens (180 × 299), per-frame cap 2048, per-frame floor 64. 640×360 → 299 tokens/frame (patch 14, merge 2).

Training


epochs / steps	2 / 1,376
global batch	384 (2 per device × 8 GPUs × 24 grad-acc, flattened B = 1 sequences)
LR	qbase 2e-5, projector 5e-6; 10 % warmup, cosine; weight decay 0
precision / parallel	bf16, DeepSpeed ZeRO-1, gradient checkpointing
wall time	11.5 h

The exact launcher is included as launcher.sh (env overrides used for this run: EPOCHS=2 COMPRESSOR_LR=2e-5 PER_DEVICE_BATCH=2, data anno_data/internvid_qwen3vl_lt180.json). The full loss curve is in trainer_state.json.

Loss

steps	logged loss	≈ per-token CE (×8, see note)
1–25	0.2312	1.85
664–688 (end of epoch 1)	0.1790	1.43
689–713 (start of epoch 2)	0.1779	1.42
1352–1376 (end)	0.1733	1.39

No NaN, no skipped samples; grad norm stayed in 0.15–0.33. These are training losses only — no held-out evaluation is reported here.

Notes on loss scaling (known quirk of the training code)

The custom compute_loss divides the CE sum by the token count gathered across all 8 GPUs but does not multiply back by the number of processes, so the logged loss is ≈ per-token CE / 8. AdamW is nearly scale-invariant, so this mostly affects the readout; with these small gradients max_grad_norm = 1.0 never clips.
The Option-B term is divided only by grad-accum steps, not by 8, so relative to CE its effective weight is ≈ 0.05 × 8 = 0.4, not the configured 0.05. Earlier Phase-1/2 runs from the same code share this behaviour.