Video-Text-to-Text
PEFT
Safetensors
caption-evaluation
video-captioning
image-captioning
lora
qwen3-vl
Instructions to use ShuitsuKoyama/Rigel with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use ShuitsuKoyama/Rigel with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Release Rigel checkpoint
Browse files- README.md +67 -0
- config.yaml +22 -0
- model.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,67 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: bsd-3-clause
|
| 3 |
+
base_model: Qwen/Qwen3-VL-2B-Instruct
|
| 4 |
+
library_name: peft
|
| 5 |
+
pipeline_tag: video-text-to-text
|
| 6 |
+
tags:
|
| 7 |
+
- caption-evaluation
|
| 8 |
+
- video-captioning
|
| 9 |
+
- image-captioning
|
| 10 |
+
- lora
|
| 11 |
+
- qwen3-vl
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# Rigel
|
| 15 |
+
|
| 16 |
+
Rigel is a caption quality scorer for videos and images. Given a video or image, a candidate
|
| 17 |
+
caption, and optionally reference captions, it outputs a continuous score in [0, 1].
|
| 18 |
+
|
| 19 |
+
- **Base model:** [Qwen/Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) (frozen)
|
| 20 |
+
- **Trainable parts (this repository):** LoRA adapters (r=8, alpha=16, `q_proj` / `v_proj`) and a
|
| 21 |
+
bottleneck classification head (2048 → 256 → 512 → 5)
|
| 22 |
+
- **Score:** softmax expected value over 5 classes mapped to {0, 0.25, 0.5, 0.75, 1}
|
| 23 |
+
|
| 24 |
+
Code: https://github.com/smilab-newton/self-distillation-smoothing
|
| 25 |
+
|
| 26 |
+
## Files
|
| 27 |
+
|
| 28 |
+
| file | content |
|
| 29 |
+
|---|---|
|
| 30 |
+
| `model.safetensors` | LoRA adapters + bottleneck head (~9 MB). The base Qwen3-VL weights are downloaded separately. |
|
| 31 |
+
| `config.yaml` | Model hyperparameters matching the weights |
|
| 32 |
+
|
| 33 |
+
## Usage
|
| 34 |
+
|
| 35 |
+
```bash
|
| 36 |
+
git clone https://github.com/smilab-newton/self-distillation-smoothing
|
| 37 |
+
cd self-distillation-smoothing
|
| 38 |
+
uv sync
|
| 39 |
+
uv run rigel-score --media path/to/video.mp4 \
|
| 40 |
+
--cand "a man is playing guitar on stage" \
|
| 41 |
+
--ref "a musician performs a guitar solo"
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
```python
|
| 45 |
+
from rigel import from_pretrained
|
| 46 |
+
|
| 47 |
+
model = from_pretrained(modality="video") # downloads this repository
|
| 48 |
+
scores = model(
|
| 49 |
+
media_path=["path/to/video.mp4"],
|
| 50 |
+
cand=["a man is playing guitar on stage"],
|
| 51 |
+
refs=[["a musician performs a guitar solo"]], # [[]] for reference-free scoring
|
| 52 |
+
)
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
A CUDA GPU is required.
|
| 56 |
+
|
| 57 |
+
## Results
|
| 58 |
+
|
| 59 |
+
| benchmark | metric | value |
|
| 60 |
+
|---|---|---|
|
| 61 |
+
| VATEX-EVAL 0 / 1 / 9 refs | Kendall τ / Spearman ρ | 42.1 / 53.7, 45.6 / 57.8, 50.8 / 63.9 |
|
| 62 |
+
| ActivityNet-FOIL | video-level accuracy | 96.2 |
|
| 63 |
+
| Composite (no ref / ref) | τ_b, τ_c | 59.5, 64.3 / 61.2, 66.1 |
|
| 64 |
+
| Flickr8K-Expert (no ref / ref) | τ_b, τ_c | 59.5, 59.9 / 58.6, 59.0 |
|
| 65 |
+
| Flickr8K-CF (no ref / ref) | τ_b, τ_c | 40.4, 20.9 / 40.4, 20.9 |
|
| 66 |
+
| Nebula (no ref / ref) | τ_b, τ_c | 58.1, 55.1 / 60.6, 57.5 |
|
| 67 |
+
| FOIL 1-ref / 4-ref | accuracy | 99.1 / 99.2 |
|
config.yaml
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Inference configuration for the Rigel scorer (Qwen3-VL-2B + LoRA + bottleneck head).
|
| 2 |
+
# Values must match the checkpoint (model.safetensors on the Hugging Face Hub).
|
| 3 |
+
|
| 4 |
+
model_class: QwenLoRAScoreSmoothing
|
| 5 |
+
model_name: Qwen/Qwen3-VL-2B-Instruct
|
| 6 |
+
attn_implementation: sdpa
|
| 7 |
+
|
| 8 |
+
# Video / image preprocessing
|
| 9 |
+
num_frames: 8
|
| 10 |
+
max_pixels: 12845056
|
| 11 |
+
|
| 12 |
+
# Bottleneck head (hidden 2048 -> 256 -> 512 -> num_classes)
|
| 13 |
+
bottleneck_dim: 256
|
| 14 |
+
num_classes: 5
|
| 15 |
+
|
| 16 |
+
# LoRA
|
| 17 |
+
lora_r: 8
|
| 18 |
+
lora_alpha: 16
|
| 19 |
+
lora_dropout: 0.1
|
| 20 |
+
lora_target_modules:
|
| 21 |
+
- q_proj
|
| 22 |
+
- v_proj
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0ddf9841fe2828f393ddd5ea947c920cebca9dae9ecc001ba7a8d88788bfe3ea
|
| 3 |
+
size 9080756
|