Instructions to use SamChen567/phase2_qbase_enhance with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SamChen567/phase2_qbase_enhance with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="SamChen567/phase2_qbase_enhance", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("SamChen567/phase2_qbase_enhance", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use SamChen567/phase2_qbase_enhance with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SamChen567/phase2_qbase_enhance"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase2_qbase_enhance",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SamChen567/phase2_qbase_enhance

SGLang

How to use SamChen567/phase2_qbase_enhance with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SamChen567/phase2_qbase_enhance" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase2_qbase_enhance",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SamChen567/phase2_qbase_enhance" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase2_qbase_enhance",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use SamChen567/phase2_qbase_enhance with Docker Model Runner:
```
docker model run hf.co/SamChen567/phase2_qbase_enhance
```

phase2_qbase_enhance / README.md

SamChen567

Add model card

c474b5e verified 24 days ago

preview code

Raw History Blame Contribute Delete

4.86 kB

---
library_name: transformers
base_model: SamChen567/phase1_qbase_internvid
tags:
  - video
  - token-compression
  - videollama3
---
# phase2_qbase_enhance (native-res, video + images, 1 epoch)

Phase 2 of the qbase video-token compressor curriculum, built on VideoLLaMA3-7B
(`DAMO-NLP-SG/VideoLLaMA3-7B`). It continues training the learned-query compressor
("qbase") that sits between the frozen SigLIP-NaViT vision encoder and the frozen
Qwen2-7B LLM. The training signal is still answer cross-entropy, but the data is now a
broad video-and-image SFT mix instead of captions only.

Warm start: [`SamChen567/phase1_qbase_internvid`](https://huggingface.co/SamChen567/phase1_qbase_internvid)
(qbase + projector + the two compression-token embedding rows).

This checkpoint needs the custom code in the training repository (`videollama3/model`,
`Videollama3Qwen2ForCausalLM` with `token_compressor`). It does not load with stock
`transformers` alone.

## What is trained

| module | status |
|---|---|
| vision encoder (SigLIP-NaViT) | frozen |
| `token_compressor` (qbase, 170M) | trained, LR 2e-5 |
| `mm_projector` (mlp2x_gelu, 17M) | trained, LR 1e-6 |
| `<\|compression_start\|>` / `<\|compression_end\|>` embedding rows | trained (all other rows frozen) |
| LLM (Qwen2-7B) + `lm_head` | frozen |

## Compressor

The compressor architecture is the same as in Phase 1.

- `transformer_decoder_flat`: 8 layers, 8 heads, hidden 1152, **K = 64** learned queries
  per segment, with cross-attention onto the segment's encoder tokens using (t, h, w) RoPE.
- **Adaptive segmentation** (not learned): a video gets `N = ⌊T/4⌋ + 1` segments. There is
  a forced cut every 8 frames, and the remaining cuts go where the per-frame mean encoder
  feature changes most (largest cosine distance between consecutive frames). Each segment
  is 1–8 frames long. Segmentation is deterministic (`segment_sample_tau = 0`), and the
  target frames per segment is fixed at 4 (`TF_CHOICES = 4`, not mixed 1–4).
- A still image is a single segment of 64 tokens.
- **Option A** (`match_encoder_scale`): the output is rescaled to the encoder tokens'
  per-dim mean/std, followed by a learnable per-channel γ/β.
- **Option B** (`distr_loss_weight = 0.05`): a CORAL-style distribution-matching auxiliary
  loss. The note at the end explains why its effective weight is higher than 0.05.

## Input layout seen by the LLM

Each segment carries its own time range (`segment_time_marks = True`), the same as in
Phase 1:

```
Time:0s-3s:<|compression_start|>[64]<|compression_end|>Time:4s-4s:<|compression_start|>[64]<|compression_end|> ...
```

## Data

- `phase2_all`: **755,269 samples**, a reorganised union of the Video-XL / Video-XL-Pro
  SFT data covering 16 tasks. It includes **every still image** in that data, so images
  are 50 % of the samples.
- Videos are decoded at 1 fps with `max_frames = 180`, at native resolution. The per-video
  budget is 65,536 merged tokens, with at most 2,048 and at least 64 tokens per frame.
- Samples are grouped by estimated encoder cost (`group_by_encode_cost`) to balance the
  load across GPUs.
- 14 samples failed to decode during training (empty image lists). Each was replaced with
  a randomly drawn sample.

## Training

| | |
|---|---|
| epochs / steps | 1 / 1,574 |
| global batch | 480 (2 per device × 8 GPUs × 30 grad-acc, flattened sequences) |
| LR | qbase 2e-5, projector 1e-6; 3 % warmup, cosine; weight decay 0 |
| precision / parallel | bf16, DeepSpeed ZeRO-1, gradient checkpointing (LLM + compressor) |
| wall time | 15.1 h (54,210 s) |

The exact launcher is included as `launcher.sh`; this run used its defaults. The full
loss curve is in `trainer_state.json`.

### Loss

| steps | logged loss (mean) |
|---|---|
| 1–25 | 0.1711 |
| 400–424 | 0.1590 |
| 775–799 | 0.1567 |
| 1200–1224 | 0.1534 |
| 1550–1574 (end) | 0.1538 |

The reported `train_loss` is 0.1566. The run had no NaN losses, and the gradient norm
stayed between 0.075 and 0.94.

`trainer_state.json` also contains per-micro-batch diagnostics logged on rank 0. These
are per-token CE values and are **not** divided by 8:

| steps | `loss_img` (still-image tokens) | `loss_vid` (video tokens) |
|---|---|---|
| 1–100 | 1.074 | 0.931 |
| 700–800 | 0.951 | 0.864 |
| 1475–1574 | 0.929 | 0.847 |

These are training losses only. No held-out evaluation is reported here.

### Notes on loss scaling (known quirk of the training code)

- The custom `compute_loss` divides the CE sum by the token count **gathered across all
  8 GPUs** but does not multiply back by the number of processes. As a result, the
  **logged `loss` is ≈ per-token CE / 8**.
- The Option-B term is divided only by the number of grad-accum steps, not by 8, so
  **relative to CE its effective weight is ≈ 0.05 × 8 = 0.4**.
- Phase 1 has the same behaviour.

	---
	library_name: transformers
	base_model: SamChen567/phase1_qbase_internvid
	tags:
	- video
	- token-compression
	- videollama3
	---

	# phase2_qbase_enhance (native-res, video + images, 1 epoch)

	Phase 2 of the qbase video-token compressor curriculum, built on VideoLLaMA3-7B
	(`DAMO-NLP-SG/VideoLLaMA3-7B`). It continues training the learned-query compressor
	("qbase") that sits between the frozen SigLIP-NaViT vision encoder and the frozen
	Qwen2-7B LLM. The training signal is still answer cross-entropy, but the data is now a
	broad video-and-image SFT mix instead of captions only.

	Warm start: [`SamChen567/phase1_qbase_internvid`](https://huggingface.co/SamChen567/phase1_qbase_internvid)
	(qbase + projector + the two compression-token embedding rows).

	This checkpoint needs the custom code in the training repository (`videollama3/model`,
	`Videollama3Qwen2ForCausalLM` with `token_compressor`). It does not load with stock
	`transformers` alone.

	## What is trained

	\| module \| status \|
	\|---\|---\|
	\| vision encoder (SigLIP-NaViT) \| frozen \|
	\| `token_compressor` (qbase, 170M) \| trained, LR 2e-5 \|
	\| `mm_projector` (mlp2x_gelu, 17M) \| trained, LR 1e-6 \|
	\| `<\\|compression_start\\|>` / `<\\|compression_end\\|>` embedding rows \| trained (all other rows frozen) \|
	\| LLM (Qwen2-7B) + `lm_head` \| frozen \|

	## Compressor

	The compressor architecture is the same as in Phase 1.

	- `transformer_decoder_flat`: 8 layers, 8 heads, hidden 1152, K = 64 learned queries
	per segment, with cross-attention onto the segment's encoder tokens using (t, h, w) RoPE.
	- Adaptive segmentation (not learned): a video gets `N = ⌊T/4⌋ + 1` segments. There is
	a forced cut every 8 frames, and the remaining cuts go where the per-frame mean encoder
	feature changes most (largest cosine distance between consecutive frames). Each segment
	is 1–8 frames long. Segmentation is deterministic (`segment_sample_tau = 0`), and the
	target frames per segment is fixed at 4 (`TF_CHOICES = 4`, not mixed 1–4).
	- A still image is a single segment of 64 tokens.
	- Option A (`match_encoder_scale`): the output is rescaled to the encoder tokens'
	per-dim mean/std, followed by a learnable per-channel γ/β.
	- Option B (`distr_loss_weight = 0.05`): a CORAL-style distribution-matching auxiliary
	loss. The note at the end explains why its effective weight is higher than 0.05.

	## Input layout seen by the LLM

	Each segment carries its own time range (`segment_time_marks = True`), the same as in
	Phase 1:

	```
	Time:0s-3s:<\|compression_start\|>[64]<\|compression_end\|>Time:4s-4s:<\|compression_start\|>[64]<\|compression_end\|> ...
	```

	## Data

	- `phase2_all`: 755,269 samples, a reorganised union of the Video-XL / Video-XL-Pro
	SFT data covering 16 tasks. It includes every still image in that data, so images
	are 50 % of the samples.
	- Videos are decoded at 1 fps with `max_frames = 180`, at native resolution. The per-video
	budget is 65,536 merged tokens, with at most 2,048 and at least 64 tokens per frame.
	- Samples are grouped by estimated encoder cost (`group_by_encode_cost`) to balance the
	load across GPUs.
	- 14 samples failed to decode during training (empty image lists). Each was replaced with
	a randomly drawn sample.

	## Training

	\| \| \|
	\|---\|---\|
	\| epochs / steps \| 1 / 1,574 \|
	\| global batch \| 480 (2 per device × 8 GPUs × 30 grad-acc, flattened sequences) \|
	\| LR \| qbase 2e-5, projector 1e-6; 3 % warmup, cosine; weight decay 0 \|
	\| precision / parallel \| bf16, DeepSpeed ZeRO-1, gradient checkpointing (LLM + compressor) \|
	\| wall time \| 15.1 h (54,210 s) \|

	The exact launcher is included as `launcher.sh`; this run used its defaults. The full
	loss curve is in `trainer_state.json`.

	### Loss

	\| steps \| logged loss (mean) \|
	\|---\|---\|
	\| 1–25 \| 0.1711 \|
	\| 400–424 \| 0.1590 \|
	\| 775–799 \| 0.1567 \|
	\| 1200–1224 \| 0.1534 \|
	\| 1550–1574 (end) \| 0.1538 \|

	The reported `train_loss` is 0.1566. The run had no NaN losses, and the gradient norm
	stayed between 0.075 and 0.94.

	`trainer_state.json` also contains per-micro-batch diagnostics logged on rank 0. These
	are per-token CE values and are not divided by 8:

	\| steps \| `loss_img` (still-image tokens) \| `loss_vid` (video tokens) \|
	\|---\|---\|---\|
	\| 1–100 \| 1.074 \| 0.931 \|
	\| 700–800 \| 0.951 \| 0.864 \|
	\| 1475–1574 \| 0.929 \| 0.847 \|

	These are training losses only. No held-out evaluation is reported here.

	### Notes on loss scaling (known quirk of the training code)

	- The custom `compute_loss` divides the CE sum by the token count **gathered across all
	8 GPUs** but does not multiply back by the number of processes. As a result, the
	logged `loss` is ≈ per-token CE / 8.
	- The Option-B term is divided only by the number of grad-accum steps, not by 8, so
	relative to CE its effective weight is ≈ 0.05 × 8 = 0.4.
	- Phase 1 has the same behaviour.