Instructions to use SamChen567/phase1_qbase_internvid with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SamChen567/phase1_qbase_internvid with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="SamChen567/phase1_qbase_internvid", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("SamChen567/phase1_qbase_internvid", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use SamChen567/phase1_qbase_internvid with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SamChen567/phase1_qbase_internvid"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase1_qbase_internvid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SamChen567/phase1_qbase_internvid

SGLang

How to use SamChen567/phase1_qbase_internvid with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SamChen567/phase1_qbase_internvid" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase1_qbase_internvid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SamChen567/phase1_qbase_internvid" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase1_qbase_internvid",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use SamChen567/phase1_qbase_internvid with Docker Model Runner:
```
docker model run hf.co/SamChen567/phase1_qbase_internvid
```

phase1_qbase_internvid / README.md

SamChen567

Add model card

c3b18fc verified 23 days ago

preview code

Raw History Blame Contribute Delete

4.91 kB

---
library_name: transformers
base_model: DAMO-NLP-SG/VideoLLaMA3-7B
tags:
  - video
  - token-compression
  - videollama3
---
# phase1_qbase_internvid (native-res, video-only, 2 epochs)

Phase 1 of the qbase video-token compressor curriculum: a learned-query compressor
("qbase") inserted between the frozen SigLIP-NaViT vision encoder and the frozen
Qwen2-7B LLM of VideoLLaMA3-7B, trained with caption cross-entropy.

> **This repo was re-used.** The repo name previously referred to an older Phase-1 run
> (16384-token video budget, 40 % still images, 3 epochs / 3,441 steps); no weights from
> that run were ever uploaded here. Everything in this repo is the run described below.

This checkpoint needs the custom code in the training repository (`videollama3/model`,
`Videollama3Qwen2ForCausalLM` with `token_compressor`); it does not load with stock
`transformers` alone.

## What is trained

| module | status |
|---|---|
| vision encoder (SigLIP-NaViT) | frozen |
| `token_compressor` (qbase, 170M) | trained |
| `mm_projector` (mlp2x_gelu, 17M) | trained |
| `<\|compression_start\|>` / `<\|compression_end\|>` embedding rows | trained (all other rows frozen) |
| LLM (Qwen2-7B) + `lm_head` | frozen |

Warm start: `SamChen567/compressor_pretrain_video_norm` (qbase + projector + the two embedding rows).

## Compressor

- `transformer_decoder_flat`: 8 layers, 8 heads, hidden 1152, **K = 64** learned queries
  (+ sin/cos position encoding); cross-attention onto the segment's encoder tokens with
  (t, h, w) RoPE.
- **Adaptive segmentation** (non-learned): `N = ⌊T/4⌋ + 1` segments per video, a forced
  cut every 8 frames, remaining cuts at the largest consecutive-frame cosine distance of
  the per-frame mean encoder feature. Each segment is 1–8 frames. Deterministic top-k
  (`segment_sample_tau = 0`).
- **Option A** (`match_encoder_scale`): output rescaled to the encoder tokens' per-dim
  mean/std, then a learnable per-channel γ/β.
- **Option B** (`distr_loss_weight = 0.05`): CORAL-style distribution-match auxiliary loss
  (see the note on its effective weight below).

## Input layout seen by the LLM

Each segment carries its own time range (`segment_time_marks = True`); the per-frame
`Time X.0s:` text of the uncompressed template is removed:

```
<|im_start|>user
Time:0s-3s:<|compression_start|>[64]<|compression_end|>Time:4s-4s:<|compression_start|>[64]<|compression_end|> ... 
{prompt}<|im_end|>
<|im_start|>assistant
{caption}<|im_end|>
```

Seconds are integers (first / last frame of the segment). Example: a 120 s clip at
1 fps → 31 segments → 1,984 compressed tokens (vs. 35,880 uncompressed at 640×360).
Checkpoints trained before 2026-10-12 used one whole-video range instead; build the
layout from this checkpoint's `config.json`.

## Data

- InternVid clips **< 180 s**, 264,186 samples, **video only** (no still images).
- Captions re-generated by **Qwen3-VL-8B-Instruct at 1 fps, native resolution**, with
  several prompt styles (`baseline_default`, `scene_shift`, `motion_scene`, ...).
- One turn per sample: `<video>\n{prompt}` → caption. Loss on the assistant caption only.

## Vision geometry

- fps = 1, `max_frames = 180`.
- Native resolution: per-video budget 53,820 merged tokens (180 × 299), per-frame cap
  2048, per-frame floor 64. 640×360 → 299 tokens/frame (patch 14, merge 2).

## Training

| | |
|---|---|
| epochs / steps | 2 / 1,376 |
| global batch | 384 (2 per device × 8 GPUs × 24 grad-acc, flattened B = 1 sequences) |
| LR | qbase 2e-5, projector 5e-6; 10 % warmup, cosine; weight decay 0 |
| precision / parallel | bf16, DeepSpeed ZeRO-1, gradient checkpointing |
| wall time | 11.5 h |

The exact launcher is included as `launcher.sh` (env overrides used for this run:
`EPOCHS=2 COMPRESSOR_LR=2e-5 PER_DEVICE_BATCH=2`, data
`anno_data/internvid_qwen3vl_lt180.json`). The full loss curve is in `trainer_state.json`.

### Loss

| steps | logged loss | ≈ per-token CE (×8, see note) |
|---|---|---|
| 1–25 | 0.2312 | 1.85 |
| 664–688 (end of epoch 1) | 0.1790 | 1.43 |
| 689–713 (start of epoch 2) | 0.1779 | 1.42 |
| 1352–1376 (end) | 0.1733 | 1.39 |

No NaN, no skipped samples; grad norm stayed in 0.15–0.33. These are training losses
only — no held-out evaluation is reported here.

### Notes on loss scaling (known quirk of the training code)

- The custom `compute_loss` divides the CE sum by the token count **gathered across all
  8 GPUs** but does not multiply back by the number of processes, so the **logged loss is
  ≈ per-token CE / 8**. AdamW is nearly scale-invariant, so this mostly affects the
  readout; with these small gradients `max_grad_norm = 1.0` never clips.
- The Option-B term is divided only by grad-accum steps, not by 8, so **relative to CE its
  effective weight is ≈ 0.05 × 8 = 0.4**, not the configured 0.05. Earlier Phase-1/2 runs
  from the same code share this behaviour.

	---
	library_name: transformers
	base_model: DAMO-NLP-SG/VideoLLaMA3-7B
	tags:
	- video
	- token-compression
	- videollama3
	---

	# phase1_qbase_internvid (native-res, video-only, 2 epochs)

	Phase 1 of the qbase video-token compressor curriculum: a learned-query compressor
	("qbase") inserted between the frozen SigLIP-NaViT vision encoder and the frozen
	Qwen2-7B LLM of VideoLLaMA3-7B, trained with caption cross-entropy.

	> This repo was re-used. The repo name previously referred to an older Phase-1 run
	> (16384-token video budget, 40 % still images, 3 epochs / 3,441 steps); no weights from
	> that run were ever uploaded here. Everything in this repo is the run described below.

	This checkpoint needs the custom code in the training repository (`videollama3/model`,
	`Videollama3Qwen2ForCausalLM` with `token_compressor`); it does not load with stock
	`transformers` alone.

	## What is trained

	\| module \| status \|
	\|---\|---\|
	\| vision encoder (SigLIP-NaViT) \| frozen \|
	\| `token_compressor` (qbase, 170M) \| trained \|
	\| `mm_projector` (mlp2x_gelu, 17M) \| trained \|
	\| `<\\|compression_start\\|>` / `<\\|compression_end\\|>` embedding rows \| trained (all other rows frozen) \|
	\| LLM (Qwen2-7B) + `lm_head` \| frozen \|

	Warm start: `SamChen567/compressor_pretrain_video_norm` (qbase + projector + the two embedding rows).

	## Compressor

	- `transformer_decoder_flat`: 8 layers, 8 heads, hidden 1152, K = 64 learned queries
	(+ sin/cos position encoding); cross-attention onto the segment's encoder tokens with
	(t, h, w) RoPE.
	- Adaptive segmentation (non-learned): `N = ⌊T/4⌋ + 1` segments per video, a forced
	cut every 8 frames, remaining cuts at the largest consecutive-frame cosine distance of
	the per-frame mean encoder feature. Each segment is 1–8 frames. Deterministic top-k
	(`segment_sample_tau = 0`).
	- Option A (`match_encoder_scale`): output rescaled to the encoder tokens' per-dim
	mean/std, then a learnable per-channel γ/β.
	- Option B (`distr_loss_weight = 0.05`): CORAL-style distribution-match auxiliary loss
	(see the note on its effective weight below).

	## Input layout seen by the LLM

	Each segment carries its own time range (`segment_time_marks = True`); the per-frame
	`Time X.0s:` text of the uncompressed template is removed:

	```
	<\|im_start\|>user
	Time:0s-3s:<\|compression_start\|>[64]<\|compression_end\|>Time:4s-4s:<\|compression_start\|>[64]<\|compression_end\|> ...
	{prompt}<\|im_end\|>
	<\|im_start\|>assistant
	{caption}<\|im_end\|>
	```

	Seconds are integers (first / last frame of the segment). Example: a 120 s clip at
	1 fps → 31 segments → 1,984 compressed tokens (vs. 35,880 uncompressed at 640×360).
	Checkpoints trained before 2026-10-12 used one whole-video range instead; build the
	layout from this checkpoint's `config.json`.

	## Data

	- InternVid clips < 180 s, 264,186 samples, video only (no still images).
	- Captions re-generated by Qwen3-VL-8B-Instruct at 1 fps, native resolution, with
	several prompt styles (`baseline_default`, `scene_shift`, `motion_scene`, ...).
	- One turn per sample: `<video>\n{prompt}` → caption. Loss on the assistant caption only.

	## Vision geometry

	- fps = 1, `max_frames = 180`.
	- Native resolution: per-video budget 53,820 merged tokens (180 × 299), per-frame cap
	2048, per-frame floor 64. 640×360 → 299 tokens/frame (patch 14, merge 2).

	## Training

	\| \| \|
	\|---\|---\|
	\| epochs / steps \| 2 / 1,376 \|
	\| global batch \| 384 (2 per device × 8 GPUs × 24 grad-acc, flattened B = 1 sequences) \|
	\| LR \| qbase 2e-5, projector 5e-6; 10 % warmup, cosine; weight decay 0 \|
	\| precision / parallel \| bf16, DeepSpeed ZeRO-1, gradient checkpointing \|
	\| wall time \| 11.5 h \|

	The exact launcher is included as `launcher.sh` (env overrides used for this run:
	`EPOCHS=2 COMPRESSOR_LR=2e-5 PER_DEVICE_BATCH=2`, data
	`anno_data/internvid_qwen3vl_lt180.json`). The full loss curve is in `trainer_state.json`.

	### Loss

	\| steps \| logged loss \| ≈ per-token CE (×8, see note) \|
	\|---\|---\|---\|
	\| 1–25 \| 0.2312 \| 1.85 \|
	\| 664–688 (end of epoch 1) \| 0.1790 \| 1.43 \|
	\| 689–713 (start of epoch 2) \| 0.1779 \| 1.42 \|
	\| 1352–1376 (end) \| 0.1733 \| 1.39 \|

	No NaN, no skipped samples; grad norm stayed in 0.15–0.33. These are training losses
	only — no held-out evaluation is reported here.

	### Notes on loss scaling (known quirk of the training code)

	- The custom `compute_loss` divides the CE sum by the token count **gathered across all
	8 GPUs but does not multiply back by the number of processes, so the logged loss is
	≈ per-token CE / 8**. AdamW is nearly scale-invariant, so this mostly affects the
	readout; with these small gradients `max_grad_norm = 1.0` never clips.
	- The Option-B term is divided only by grad-accum steps, not by 8, so **relative to CE its
	effective weight is ≈ 0.05 × 8 = 0.4**, not the configured 0.05. Earlier Phase-1/2 runs
	from the same code share this behaviour.