ShuitsuKoyama commited on
Commit
fe3293a
·
verified ·
1 Parent(s): efc0834

Release Rigel checkpoint

Browse files
Files changed (3) hide show
  1. README.md +67 -0
  2. config.yaml +22 -0
  3. model.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: bsd-3-clause
3
+ base_model: Qwen/Qwen3-VL-2B-Instruct
4
+ library_name: peft
5
+ pipeline_tag: video-text-to-text
6
+ tags:
7
+ - caption-evaluation
8
+ - video-captioning
9
+ - image-captioning
10
+ - lora
11
+ - qwen3-vl
12
+ ---
13
+
14
+ # Rigel
15
+
16
+ Rigel is a caption quality scorer for videos and images. Given a video or image, a candidate
17
+ caption, and optionally reference captions, it outputs a continuous score in [0, 1].
18
+
19
+ - **Base model:** [Qwen/Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) (frozen)
20
+ - **Trainable parts (this repository):** LoRA adapters (r=8, alpha=16, `q_proj` / `v_proj`) and a
21
+ bottleneck classification head (2048 → 256 → 512 → 5)
22
+ - **Score:** softmax expected value over 5 classes mapped to {0, 0.25, 0.5, 0.75, 1}
23
+
24
+ Code: https://github.com/smilab-newton/self-distillation-smoothing
25
+
26
+ ## Files
27
+
28
+ | file | content |
29
+ |---|---|
30
+ | `model.safetensors` | LoRA adapters + bottleneck head (~9 MB). The base Qwen3-VL weights are downloaded separately. |
31
+ | `config.yaml` | Model hyperparameters matching the weights |
32
+
33
+ ## Usage
34
+
35
+ ```bash
36
+ git clone https://github.com/smilab-newton/self-distillation-smoothing
37
+ cd self-distillation-smoothing
38
+ uv sync
39
+ uv run rigel-score --media path/to/video.mp4 \
40
+ --cand "a man is playing guitar on stage" \
41
+ --ref "a musician performs a guitar solo"
42
+ ```
43
+
44
+ ```python
45
+ from rigel import from_pretrained
46
+
47
+ model = from_pretrained(modality="video") # downloads this repository
48
+ scores = model(
49
+ media_path=["path/to/video.mp4"],
50
+ cand=["a man is playing guitar on stage"],
51
+ refs=[["a musician performs a guitar solo"]], # [[]] for reference-free scoring
52
+ )
53
+ ```
54
+
55
+ A CUDA GPU is required.
56
+
57
+ ## Results
58
+
59
+ | benchmark | metric | value |
60
+ |---|---|---|
61
+ | VATEX-EVAL 0 / 1 / 9 refs | Kendall τ / Spearman ρ | 42.1 / 53.7, 45.6 / 57.8, 50.8 / 63.9 |
62
+ | ActivityNet-FOIL | video-level accuracy | 96.2 |
63
+ | Composite (no ref / ref) | τ_b, τ_c | 59.5, 64.3 / 61.2, 66.1 |
64
+ | Flickr8K-Expert (no ref / ref) | τ_b, τ_c | 59.5, 59.9 / 58.6, 59.0 |
65
+ | Flickr8K-CF (no ref / ref) | τ_b, τ_c | 40.4, 20.9 / 40.4, 20.9 |
66
+ | Nebula (no ref / ref) | τ_b, τ_c | 58.1, 55.1 / 60.6, 57.5 |
67
+ | FOIL 1-ref / 4-ref | accuracy | 99.1 / 99.2 |
config.yaml ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Inference configuration for the Rigel scorer (Qwen3-VL-2B + LoRA + bottleneck head).
2
+ # Values must match the checkpoint (model.safetensors on the Hugging Face Hub).
3
+
4
+ model_class: QwenLoRAScoreSmoothing
5
+ model_name: Qwen/Qwen3-VL-2B-Instruct
6
+ attn_implementation: sdpa
7
+
8
+ # Video / image preprocessing
9
+ num_frames: 8
10
+ max_pixels: 12845056
11
+
12
+ # Bottleneck head (hidden 2048 -> 256 -> 512 -> num_classes)
13
+ bottleneck_dim: 256
14
+ num_classes: 5
15
+
16
+ # LoRA
17
+ lora_r: 8
18
+ lora_alpha: 16
19
+ lora_dropout: 0.1
20
+ lora_target_modules:
21
+ - q_proj
22
+ - v_proj
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0ddf9841fe2828f393ddd5ea947c920cebca9dae9ecc001ba7a8d88788bfe3ea
3
+ size 9080756