Publish EventLens-4B model card and configuration

d0215d4 verified 12 days ago

6.52 kB

---
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: video-text-to-text
base_model: Qwen/Qwen3-VL-4B-Instruct
base_model_relation: finetune
datasets:
- Lostgreen/EventLens-Train
tags:
- eventlens
- qwen3_vl
- video-understanding
- temporal-grounding
- reinforcement-learning
- on-policy-distillation
- safetensors
---
![EventLens-4B — event-structure learning for video understanding](assets/model-cover.png)

**[Results](#results) · [Quick start](#quick-start) · [Training](#training) · [Dataset](https://huggingface.co/datasets/Lostgreen/EventLens-Train)**

**EventLens-4B** is a video-language model developed through event-structure reinforcement learning and **multi-teacher on-policy distillation (MT-OPD)**. Built on [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct), it learns to recognize **event boundaries, progression, and relations**, alongside temporal grounding and video question answering.

The model is part of **EventLens: Event-Structure Reinforcement Learning for Video Understanding**, developed by **AIM3 Lab, Renmin University of China**, and **Kling AI**.

## Results

Selected results reported in the EventLens paper, compared with the corresponding pretrained backbone under the shared **256-frame** protocol.

| Benchmark | Qwen3-VL-4B | EventLens-4B | Gain |
| --- | ---: | ---: | ---: |
| QVHighlights | 54.31 | **58.50** | **+4.19** |
| ActivityNet | 41.99 | **45.76** | **+3.77** |
| Charades-STA | 43.64 | **47.99** | **+4.35** |
| E.T. Bench | 34.55 | **41.47** | **+6.92** |
| VinoGround | 37.00 | **45.00** | **+8.00** |

QVHighlights, ActivityNet, and Charades-STA use **mIoU**. E.T. Bench and VinoGround use their official benchmark scores. Gains are absolute score differences; higher is better. These are paper-reported results.

## Model details

| Property | Value |
| --- | --- |
| Base model | `Qwen/Qwen3-VL-4B-Instruct` |
| Architecture | `Qwen3VLForConditionalGeneration` |
| Checkpoint format | Full BF16 safetensors weights |
| Weight files | 2 shards · 9.65 GB |
| Stored tensor parameters¹ | 4,826,771,968 |
| Student training | Multi-teacher on-policy distillation |
| Specialist teachers used in training | Three 4B models: Boundary, Progression, Relation |
| Paper prompt mode | Direct task answers (`no_cot`) |
| Training view | `Lostgreen/EventLens-Train` / `paper_mt_opd` |

¹ The 4B name follows the backbone series. The stored tensor count includes both the language and vision components. Weight file size is distinct from runtime memory use.

## Quick start

Use a Transformers version that supports Qwen3-VL. The example follows the [official Qwen3-VL integration](https://huggingface.co/docs/transformers/model_doc/qwen3_vl).

```bash
pip install -U transformers accelerate torch torchcodec
```

```python
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
MODEL_PATH = "Lostgreen/EventLens-4B"
processor = AutoProcessor.from_pretrained(MODEL_PATH)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    MODEL_PATH,
    dtype="auto",
    device_map="auto",
    attn_implementation="sdpa",
).eval()
messages = [{
    "role": "user",
    "content": [
        {"type": "video", "path": "/path/to/video.mp4"},
        {"type": "text", "text": "Describe the main events in this video in chronological order."},
    ],
}]
inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
    num_frames=32,
).to(model.device)
inputs.pop("token_type_ids", None)
output_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
answer_ids = output_ids[:, inputs.input_ids.shape[1]:]
print(processor.batch_decode(answer_ids, skip_special_tokens=True)[0])
```

This short example samples **32 frames**. Reproducing the paper's scores requires the benchmark-specific prompts, preprocessing, and 256-frame protocol. For event recovery tasks, follow the task's answer format in the released dataset; frame order and clip grouping are part of the input.

## Training

![Specialist reinforcement learning followed by multi-teacher on-policy distillation](assets/training-pipeline.png)

**Stage I — Learn event specialists.** Three Qwen3-VL-4B teachers are trained on event boundaries, single-event progression, and multi-event relations using reinforcement learning with verifiable rewards.

**Stage II — Distill a unified student.** The 4B student generates its own responses. Task routing selects the matching specialist, which provides token distributions for on-policy distillation. Both EventLens student sizes use the same three 4B teachers.

The released model is the **unified 4B student**. Specialist teacher checkpoints are not included in this release.

### Training data

The public [EventLens training dataset](https://huggingface.co/datasets/Lostgreen/EventLens-Train) provides frozen task views, original hierarchy annotations, and shared sampled JPEG frames. The default student view contains **37,428 training examples**:

| Task family | Training examples |
| --- | ---: |
| Event boundaries | 10,000 |
| Event progression | 10,000 |
| Event relations | 10,000 |
| Temporal grounding | 4,791 |
| Video QA | 2,637 |

The historical `monitor_val` split contains 1,950 records and overlaps training. Benchmark results above use the corresponding external evaluation benchmarks.

## Intended use

EventLens-4B supports research on video event understanding, temporal grounding, event ordering, and video question answering. Event timing and ordering depend on the supplied visual evidence: sparse sampling can omit brief actions, and generated descriptions or timestamps may be inaccurate. Review outputs when exact event boundaries matter.

## License and credits

**EventLens model weights are released under [CC-BY-NC-SA-4.0](LICENSE) for non-commercial use.** The [Qwen3-VL-4B-Instruct base model](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) is distributed under Apache-2.0. The EventLens dataset uses CC-BY-NC-SA-4.0 for EventLens-derived annotations and labels, with source-specific terms for third-party data and media. The original Apache-2.0 license is retained in [LICENSE-BASE-QWEN-APACHE2](LICENSE-BASE-QWEN-APACHE2), with attribution in [NOTICE](NOTICE).

EventLens builds on **Qwen3-VL**, **EasyR1**, and **verl**. The training dataset acknowledges its upstream sources in its [dataset card](https://huggingface.co/datasets/Lostgreen/EventLens-Train).

	---
	license: cc-by-nc-sa-4.0
	library_name: transformers
	pipeline_tag: video-text-to-text
	base_model: Qwen/Qwen3-VL-4B-Instruct
	base_model_relation: finetune
	datasets:
	- Lostgreen/EventLens-Train
	tags:
	- eventlens
	- qwen3_vl
	- video-understanding
	- temporal-grounding
	- reinforcement-learning
	- on-policy-distillation
	- safetensors
	---

	![EventLens-4B — event-structure learning for video understanding](assets/model-cover.png)

	[Results](#results) · [Quick start](#quick-start) · [Training](#training) · [Dataset](https://huggingface.co/datasets/Lostgreen/EventLens-Train)

	EventLens-4B is a video-language model developed through event-structure reinforcement learning and multi-teacher on-policy distillation (MT-OPD). Built on [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct), it learns to recognize event boundaries, progression, and relations, alongside temporal grounding and video question answering.

	The model is part of EventLens: Event-Structure Reinforcement Learning for Video Understanding, developed by AIM3 Lab, Renmin University of China, and Kling AI.

	## Results

	Selected results reported in the EventLens paper, compared with the corresponding pretrained backbone under the shared 256-frame protocol.

	\| Benchmark \| Qwen3-VL-4B \| EventLens-4B \| Gain \|
	\| --- \| ---: \| ---: \| ---: \|
	\| QVHighlights \| 54.31 \| 58.50 \| +4.19 \|
	\| ActivityNet \| 41.99 \| 45.76 \| +3.77 \|
	\| Charades-STA \| 43.64 \| 47.99 \| +4.35 \|
	\| E.T. Bench \| 34.55 \| 41.47 \| +6.92 \|
	\| VinoGround \| 37.00 \| 45.00 \| +8.00 \|

	QVHighlights, ActivityNet, and Charades-STA use mIoU. E.T. Bench and VinoGround use their official benchmark scores. Gains are absolute score differences; higher is better. These are paper-reported results.

	## Model details

	\| Property \| Value \|
	\| --- \| --- \|
	\| Base model \| `Qwen/Qwen3-VL-4B-Instruct` \|
	\| Architecture \| `Qwen3VLForConditionalGeneration` \|
	\| Checkpoint format \| Full BF16 safetensors weights \|
	\| Weight files \| 2 shards · 9.65 GB \|
	\| Stored tensor parameters¹ \| 4,826,771,968 \|
	\| Student training \| Multi-teacher on-policy distillation \|
	\| Specialist teachers used in training \| Three 4B models: Boundary, Progression, Relation \|
	\| Paper prompt mode \| Direct task answers (`no_cot`) \|
	\| Training view \| `Lostgreen/EventLens-Train` / `paper_mt_opd` \|

	¹ The 4B name follows the backbone series. The stored tensor count includes both the language and vision components. Weight file size is distinct from runtime memory use.

	## Quick start

	Use a Transformers version that supports Qwen3-VL. The example follows the [official Qwen3-VL integration](https://huggingface.co/docs/transformers/model_doc/qwen3_vl).

	```bash
	pip install -U transformers accelerate torch torchcodec
	```

	```python
	from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

	MODEL_PATH = "Lostgreen/EventLens-4B"

	processor = AutoProcessor.from_pretrained(MODEL_PATH)
	model = Qwen3VLForConditionalGeneration.from_pretrained(
	MODEL_PATH,
	dtype="auto",
	device_map="auto",
	attn_implementation="sdpa",
	).eval()

	messages = [{
	"role": "user",
	"content": [
	{"type": "video", "path": "/path/to/video.mp4"},
	{"type": "text", "text": "Describe the main events in this video in chronological order."},
	],
	}]

	inputs = processor.apply_chat_template(
	messages,
	tokenize=True,
	add_generation_prompt=True,
	return_dict=True,
	return_tensors="pt",
	num_frames=32,
	).to(model.device)
	inputs.pop("token_type_ids", None)

	output_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
	answer_ids = output_ids[:, inputs.input_ids.shape[1]:]
	print(processor.batch_decode(answer_ids, skip_special_tokens=True)[0])
	```

	This short example samples 32 frames. Reproducing the paper's scores requires the benchmark-specific prompts, preprocessing, and 256-frame protocol. For event recovery tasks, follow the task's answer format in the released dataset; frame order and clip grouping are part of the input.

	## Training

	![Specialist reinforcement learning followed by multi-teacher on-policy distillation](assets/training-pipeline.png)

	Stage I — Learn event specialists. Three Qwen3-VL-4B teachers are trained on event boundaries, single-event progression, and multi-event relations using reinforcement learning with verifiable rewards.

	Stage II — Distill a unified student. The 4B student generates its own responses. Task routing selects the matching specialist, which provides token distributions for on-policy distillation. Both EventLens student sizes use the same three 4B teachers.

	The released model is the unified 4B student. Specialist teacher checkpoints are not included in this release.

	### Training data

	The public [EventLens training dataset](https://huggingface.co/datasets/Lostgreen/EventLens-Train) provides frozen task views, original hierarchy annotations, and shared sampled JPEG frames. The default student view contains 37,428 training examples:

	\| Task family \| Training examples \|
	\| --- \| ---: \|
	\| Event boundaries \| 10,000 \|
	\| Event progression \| 10,000 \|
	\| Event relations \| 10,000 \|
	\| Temporal grounding \| 4,791 \|
	\| Video QA \| 2,637 \|

	The historical `monitor_val` split contains 1,950 records and overlaps training. Benchmark results above use the corresponding external evaluation benchmarks.

	## Intended use

	EventLens-4B supports research on video event understanding, temporal grounding, event ordering, and video question answering. Event timing and ordering depend on the supplied visual evidence: sparse sampling can omit brief actions, and generated descriptions or timestamps may be inaccurate. Review outputs when exact event boundaries matter.

	## License and credits

	EventLens model weights are released under [CC-BY-NC-SA-4.0](LICENSE) for non-commercial use. The [Qwen3-VL-4B-Instruct base model](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) is distributed under Apache-2.0. The EventLens dataset uses CC-BY-NC-SA-4.0 for EventLens-derived annotations and labels, with source-specific terms for third-party data and media. The original Apache-2.0 license is retained in [LICENSE-BASE-QWEN-APACHE2](LICENSE-BASE-QWEN-APACHE2), with attribution in [NOTICE](NOTICE).

	EventLens builds on Qwen3-VL, EasyR1, and verl. The training dataset acknowledges its upstream sources in its [dataset card](https://huggingface.co/datasets/Lostgreen/EventLens-Train).