---
library_name: transformers
base_model: SamChen567/phase1_qbase_internvid
tags:
  - video
  - token-compression
  - videollama3
---

# phase2_qbase_enhance (native-res, video + images, 1 epoch)

Phase 2 of the qbase video-token compressor curriculum, built on VideoLLaMA3-7B
(`DAMO-NLP-SG/VideoLLaMA3-7B`). It continues training the learned-query compressor
("qbase") that sits between the frozen SigLIP-NaViT vision encoder and the frozen
Qwen2-7B LLM. The training signal is still answer cross-entropy, but the data is now a
broad video-and-image SFT mix instead of captions only.

Warm start: [`SamChen567/phase1_qbase_internvid`](https://huggingface.co/SamChen567/phase1_qbase_internvid)
(qbase + projector + the two compression-token embedding rows).

This checkpoint needs the custom code in the training repository (`videollama3/model`,
`Videollama3Qwen2ForCausalLM` with `token_compressor`). It does not load with stock
`transformers` alone.

## What is trained

| module | status |
|---|---|
| vision encoder (SigLIP-NaViT) | frozen |
| `token_compressor` (qbase, 170M) | trained, LR 2e-5 |
| `mm_projector` (mlp2x_gelu, 17M) | trained, LR 1e-6 |
| `<\|compression_start\|>` / `<\|compression_end\|>` embedding rows | trained (all other rows frozen) |
| LLM (Qwen2-7B) + `lm_head` | frozen |

## Compressor

The compressor architecture is the same as in Phase 1.

- `transformer_decoder_flat`: 8 layers, 8 heads, hidden 1152, **K = 64** learned queries
  per segment, with cross-attention onto the segment's encoder tokens using (t, h, w) RoPE.
- **Adaptive segmentation** (not learned): a video gets `N = ⌊T/4⌋ + 1` segments. There is
  a forced cut every 8 frames, and the remaining cuts go where the per-frame mean encoder
  feature changes most (largest cosine distance between consecutive frames). Each segment
  is 1–8 frames long. Segmentation is deterministic (`segment_sample_tau = 0`), and the
  target frames per segment is fixed at 4 (`TF_CHOICES = 4`, not mixed 1–4).
- A still image is a single segment of 64 tokens.
- **Option A** (`match_encoder_scale`): the output is rescaled to the encoder tokens'
  per-dim mean/std, followed by a learnable per-channel γ/β.
- **Option B** (`distr_loss_weight = 0.05`): a CORAL-style distribution-matching auxiliary
  loss. The note at the end explains why its effective weight is higher than 0.05.

## Input layout seen by the LLM

Each segment carries its own time range (`segment_time_marks = True`), the same as in
Phase 1:

```
Time:0s-3s:<|compression_start|>[64]<|compression_end|>Time:4s-4s:<|compression_start|>[64]<|compression_end|> ...
```

## Data

- `phase2_all`: **755,269 samples**, a reorganised union of the Video-XL / Video-XL-Pro
  SFT data covering 16 tasks. It includes **every still image** in that data, so images
  are 50 % of the samples.
- Videos are decoded at 1 fps with `max_frames = 180`, at native resolution. The per-video
  budget is 65,536 merged tokens, with at most 2,048 and at least 64 tokens per frame.
- Samples are grouped by estimated encoder cost (`group_by_encode_cost`) to balance the
  load across GPUs.
- 14 samples failed to decode during training (empty image lists). Each was replaced with
  a randomly drawn sample.

## Training

| | |
|---|---|
| epochs / steps | 1 / 1,574 |
| global batch | 480 (2 per device × 8 GPUs × 30 grad-acc, flattened sequences) |
| LR | qbase 2e-5, projector 1e-6; 3 % warmup, cosine; weight decay 0 |
| precision / parallel | bf16, DeepSpeed ZeRO-1, gradient checkpointing (LLM + compressor) |
| wall time | 15.1 h (54,210 s) |

The exact launcher is included as `launcher.sh`; this run used its defaults. The full
loss curve is in `trainer_state.json`.

### Loss

| steps | logged loss (mean) |
|---|---|
| 1–25 | 0.1711 |
| 400–424 | 0.1590 |
| 775–799 | 0.1567 |
| 1200–1224 | 0.1534 |
| 1550–1574 (end) | 0.1538 |

The reported `train_loss` is 0.1566. The run had no NaN losses, and the gradient norm
stayed between 0.075 and 0.94.

`trainer_state.json` also contains per-micro-batch diagnostics logged on rank 0. These
are per-token CE values and are **not** divided by 8:

| steps | `loss_img` (still-image tokens) | `loss_vid` (video tokens) |
|---|---|---|
| 1–100 | 1.074 | 0.931 |
| 700–800 | 0.951 | 0.864 |
| 1475–1574 | 0.929 | 0.847 |

These are training losses only. No held-out evaluation is reported here.

### Notes on loss scaling (known quirk of the training code)

- The custom `compute_loss` divides the CE sum by the token count **gathered across all
  8 GPUs** but does not multiply back by the number of processes. As a result, the
  **logged `loss` is ≈ per-token CE / 8**.
- The Option-B term is divided only by the number of grad-accum steps, not by 8, so
  **relative to CE its effective weight is ≈ 0.05 × 8 = 0.4**.
- Phase 1 has the same behaviour.