---
library_name: transformers
base_model: DAMO-NLP-SG/VideoLLaMA3-7B
tags:
  - video
  - token-compression
  - videollama3
---

# phase1_qbase_internvid (native-res, video-only, 2 epochs)

Phase 1 of the qbase video-token compressor curriculum: a learned-query compressor
("qbase") inserted between the frozen SigLIP-NaViT vision encoder and the frozen
Qwen2-7B LLM of VideoLLaMA3-7B, trained with caption cross-entropy.

> **This repo was re-used.** The repo name previously referred to an older Phase-1 run
> (16384-token video budget, 40 % still images, 3 epochs / 3,441 steps); no weights from
> that run were ever uploaded here. Everything in this repo is the run described below.

This checkpoint needs the custom code in the training repository (`videollama3/model`,
`Videollama3Qwen2ForCausalLM` with `token_compressor`); it does not load with stock
`transformers` alone.

## What is trained

| module | status |
|---|---|
| vision encoder (SigLIP-NaViT) | frozen |
| `token_compressor` (qbase, 170M) | trained |
| `mm_projector` (mlp2x_gelu, 17M) | trained |
| `<\|compression_start\|>` / `<\|compression_end\|>` embedding rows | trained (all other rows frozen) |
| LLM (Qwen2-7B) + `lm_head` | frozen |

Warm start: `SamChen567/compressor_pretrain_video_norm` (qbase + projector + the two embedding rows).

## Compressor

- `transformer_decoder_flat`: 8 layers, 8 heads, hidden 1152, **K = 64** learned queries
  (+ sin/cos position encoding); cross-attention onto the segment's encoder tokens with
  (t, h, w) RoPE.
- **Adaptive segmentation** (non-learned): `N = ⌊T/4⌋ + 1` segments per video, a forced
  cut every 8 frames, remaining cuts at the largest consecutive-frame cosine distance of
  the per-frame mean encoder feature. Each segment is 1–8 frames. Deterministic top-k
  (`segment_sample_tau = 0`).
- **Option A** (`match_encoder_scale`): output rescaled to the encoder tokens' per-dim
  mean/std, then a learnable per-channel γ/β.
- **Option B** (`distr_loss_weight = 0.05`): CORAL-style distribution-match auxiliary loss
  (see the note on its effective weight below).

## Input layout seen by the LLM

Each segment carries its own time range (`segment_time_marks = True`); the per-frame
`Time X.0s:` text of the uncompressed template is removed:

```
<|im_start|>user
Time:0s-3s:<|compression_start|>[64]<|compression_end|>Time:4s-4s:<|compression_start|>[64]<|compression_end|> ... 
{prompt}<|im_end|>
<|im_start|>assistant
{caption}<|im_end|>
```

Seconds are integers (first / last frame of the segment). Example: a 120 s clip at
1 fps → 31 segments → 1,984 compressed tokens (vs. 35,880 uncompressed at 640×360).
Checkpoints trained before 2026-10-12 used one whole-video range instead; build the
layout from this checkpoint's `config.json`.

## Data

- InternVid clips **< 180 s**, 264,186 samples, **video only** (no still images).
- Captions re-generated by **Qwen3-VL-8B-Instruct at 1 fps, native resolution**, with
  several prompt styles (`baseline_default`, `scene_shift`, `motion_scene`, ...).
- One turn per sample: `<video>\n{prompt}` → caption. Loss on the assistant caption only.

## Vision geometry

- fps = 1, `max_frames = 180`.
- Native resolution: per-video budget 53,820 merged tokens (180 × 299), per-frame cap
  2048, per-frame floor 64. 640×360 → 299 tokens/frame (patch 14, merge 2).

## Training

| | |
|---|---|
| epochs / steps | 2 / 1,376 |
| global batch | 384 (2 per device × 8 GPUs × 24 grad-acc, flattened B = 1 sequences) |
| LR | qbase 2e-5, projector 5e-6; 10 % warmup, cosine; weight decay 0 |
| precision / parallel | bf16, DeepSpeed ZeRO-1, gradient checkpointing |
| wall time | 11.5 h |

The exact launcher is included as `launcher.sh` (env overrides used for this run:
`EPOCHS=2 COMPRESSOR_LR=2e-5 PER_DEVICE_BATCH=2`, data
`anno_data/internvid_qwen3vl_lt180.json`). The full loss curve is in `trainer_state.json`.

### Loss

| steps | logged loss | ≈ per-token CE (×8, see note) |
|---|---|---|
| 1–25 | 0.2312 | 1.85 |
| 664–688 (end of epoch 1) | 0.1790 | 1.43 |
| 689–713 (start of epoch 2) | 0.1779 | 1.42 |
| 1352–1376 (end) | 0.1733 | 1.39 |

No NaN, no skipped samples; grad norm stayed in 0.15–0.33. These are training losses
only — no held-out evaluation is reported here.

### Notes on loss scaling (known quirk of the training code)

- The custom `compute_loss` divides the CE sum by the token count **gathered across all
  8 GPUs** but does not multiply back by the number of processes, so the **logged loss is
  ≈ per-token CE / 8**. AdamW is nearly scale-invariant, so this mostly affects the
  readout; with these small gradients `max_grad_norm = 1.0` never clips.
- The Option-B term is divided only by grad-accum steps, not by 8, so **relative to CE its
  effective weight is ≈ 0.05 × 8 = 0.4**, not the configured 0.05. Earlier Phase-1/2 runs
  from the same code share this behaviour.