Add model card

README.md

@@ -0,0 +1,115 @@
+---
+library_name: transformers
+base_model: DAMO-NLP-SG/VideoLLaMA3-7B
+tags:
+  - video
+  - token-compression
+  - videollama3
+---
+
+# phase1_qbase_internvid (native-res, video-only, 2 epochs)
+
+Phase 1 of the qbase video-token compressor curriculum: a learned-query compressor
+("qbase") inserted between the frozen SigLIP-NaViT vision encoder and the frozen
+Qwen2-7B LLM of VideoLLaMA3-7B, trained with caption cross-entropy.
+
+> **This repo was re-used.** The repo name previously referred to an older Phase-1 run
+> (16384-token video budget, 40 % still images, 3 epochs / 3,441 steps); no weights from
+> that run were ever uploaded here. Everything in this repo is the run described below.
+
+This checkpoint needs the custom code in the training repository (`videollama3/model`,
+`Videollama3Qwen2ForCausalLM` with `token_compressor`); it does not load with stock
+`transformers` alone.
+
+## What is trained
+
+| module | status |
+|---|---|
+| vision encoder (SigLIP-NaViT) | frozen |
+| `token_compressor` (qbase, 170M) | trained |
+| `mm_projector` (mlp2x_gelu, 17M) | trained |
+| `<\|compression_start\|>` / `<\|compression_end\|>` embedding rows | trained (all other rows frozen) |
+| LLM (Qwen2-7B) + `lm_head` | frozen |
+
+Warm start: `SamChen567/compressor_pretrain_video_norm` (qbase + projector + the two embedding rows).
+
+## Compressor
+
+- `transformer_decoder_flat`: 8 layers, 8 heads, hidden 1152, **K = 64** learned queries
+  (+ sin/cos position encoding); cross-attention onto the segment's encoder tokens with
+  (t, h, w) RoPE.
+- **Adaptive segmentation** (non-learned): `N = ⌊T/4⌋ + 1` segments per video, a forced
+  cut every 8 frames, remaining cuts at the largest consecutive-frame cosine distance of
+  the per-frame mean encoder feature. Each segment is 1–8 frames. Deterministic top-k
+  (`segment_sample_tau = 0`).
+- **Option A** (`match_encoder_scale`): output rescaled to the encoder tokens' per-dim
+  mean/std, then a learnable per-channel γ/β.
+- **Option B** (`distr_loss_weight = 0.05`): CORAL-style distribution-match auxiliary loss
+  (see the note on its effective weight below).
+
+## Input layout seen by the LLM
+
+Each segment carries its own time range (`segment_time_marks = True`); the per-frame
+`Time X.0s:` text of the uncompressed template is removed:
+
+```
+<|im_start|>user
+Time:0s-3s:<|compression_start|>[64]<|compression_end|>Time:4s-4s:<|compression_start|>[64]<|compression_end|> ... 
+{prompt}<|im_end|>
+<|im_start|>assistant
+{caption}<|im_end|>
+```
+
+Seconds are integers (first / last frame of the segment). Example: a 120 s clip at
+1 fps → 31 segments → 1,984 compressed tokens (vs. 35,880 uncompressed at 640×360).
+Checkpoints trained before 2026-10-12 used one whole-video range instead; build the
+layout from this checkpoint's `config.json`.
+
+## Data
+
+- InternVid clips **< 180 s**, 264,186 samples, **video only** (no still images).
+- Captions re-generated by **Qwen3-VL-8B-Instruct at 1 fps, native resolution**, with
+  several prompt styles (`baseline_default`, `scene_shift`, `motion_scene`, ...).
+- One turn per sample: `<video>\n{prompt}` → caption. Loss on the assistant caption only.
+
+## Vision geometry
+
+- fps = 1, `max_frames = 180`.
+- Native resolution: per-video budget 53,820 merged tokens (180 × 299), per-frame cap
+  2048, per-frame floor 64. 640×360 → 299 tokens/frame (patch 14, merge 2).
+
+## Training
+
+| | |
+|---|---|
+| epochs / steps | 2 / 1,376 |
+| global batch | 384 (2 per device × 8 GPUs × 24 grad-acc, flattened B = 1 sequences) |
+| LR | qbase 2e-5, projector 5e-6; 10 % warmup, cosine; weight decay 0 |
+| precision / parallel | bf16, DeepSpeed ZeRO-1, gradient checkpointing |
+| wall time | 11.5 h |
+
+The exact launcher is included as `launcher.sh` (env overrides used for this run:
+`EPOCHS=2 COMPRESSOR_LR=2e-5 PER_DEVICE_BATCH=2`, data
+`anno_data/internvid_qwen3vl_lt180.json`). The full loss curve is in `trainer_state.json`.
+
+### Loss
+
+| steps | logged loss | ≈ per-token CE (×8, see note) |
+|---|---|---|
+| 1–25 | 0.2312 | 1.85 |
+| 664–688 (end of epoch 1) | 0.1790 | 1.43 |
+| 689–713 (start of epoch 2) | 0.1779 | 1.42 |
+| 1352–1376 (end) | 0.1733 | 1.39 |
+
+No NaN, no skipped samples; grad norm stayed in 0.15–0.33. These are training losses
+only — no held-out evaluation is reported here.
+
+### Notes on loss scaling (known quirk of the training code)
+
+- The custom `compute_loss` divides the CE sum by the token count **gathered across all
+  8 GPUs** but does not multiply back by the number of processes, so the **logged loss is
+  ≈ per-token CE / 8**. AdamW is nearly scale-invariant, so this mostly affects the
+  readout; with these small gradients `max_grad_norm = 1.0` never clips.
+- The Option-B term is divided only by grad-accum steps, not by 8, so **relative to CE its
+  effective weight is ≈ 0.05 × 8 = 0.4**, not the configured 0.05. Earlier Phase-1/2 runs
+  from the same code share this behaviour.