| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-VL-8B-Instruct |
| pipeline_tag: video-text-to-text |
| library_name: transformers |
| tags: |
| - video-understanding |
| - reinforcement-learning |
| - adaptive-reasoning |
| - qwen3-vl |
| --- |
| |
| # EVAThink-Stage1 |
|
|
| <!-- TODO(author): one-line expansion of the name, paper title / arXiv link, code repo link --> |
|
|
| **EVAThink-Stage1** is the intermediate checkpoint after the first of the two reinforcement learning stages behind |
| [EVAThink-Qwen3](https://huggingface.co/MaplesWCT/EVAThink-Qwen3). It is fine-tuned from |
| [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) and already chooses per question whether to answer |
| directly or to write an evidence-grounded reasoning trace first, but it still reasons on most questions. |
|
|
| **For general use, prefer [EVAThink-Qwen3](https://huggingface.co/MaplesWCT/EVAThink-Qwen3).** It reaches about the same |
| accuracy with roughly a third fewer output tokens. This checkpoint is released for studying the training stages. |
|
|
| ## Highlights |
|
|
| - **Accuracy is already close to the final model.** Averaged over six benchmarks it scores 65.4, against 65.6 for |
| EVAThink-Qwen3 and 64.3 for the base model. |
| - **It thinks much more often than it needs to.** It reasons on 77% of Video-MME questions and 83% of LongVideoBench |
| questions (EVAThink-Qwen3: 34% and 46%). Stage 2 mainly teaches the model to skip reasoning when it isn't needed. |
| - **It still adapts to the task.** On purely perceptual spatial questions (VSI-Bench) it almost never reasons (0.2%), and on |
| MVBench it reasons on about a third of questions. |
|
|
| ## Results |
|
|
| Same protocol as EVAThink-Qwen3, all models evaluated by us: |
| - 64 frames sampled uniformly (first and last frame included) |
| - each frame resized to a fixed 200,704 pixels (factor 28) |
| - greedy decoding with presence penalty 1.5, up to 4,096 new tokens |
| - vLLM |
|
|
| EVAThink models get the bare question and options with no instruction. Qwen3-VL-8B-Instruct gets each benchmark's official prompt. |
|
|
| | Benchmark (split, #Q) | Qwen3-VL-8B-Instruct | **EVAThink-Stage1** | EVAThink-Qwen3 | Stage1 think-rate | Stage1 mean tokens | |
| |---|---|---|---|---|---| |
| | Video-MME w/o subs (2,700) | 64.9 | 65.4 | 66.3 | 76.7% | 203 | |
| | MVBench (4,000) | 65.8 | 65.2 | 66.5 | 31.9% | 90 | |
| | MMVU, multiple-choice (625) | 70.4 | 73.4 | 72.8 | 96.0% | 413 | |
| | Video-MMMU (900) | 65.6 | 66.7 | 67.4 | 98.3% | 1,038 | |
| | LongVideoBench val (1,337) | 60.7 | 61.3 | 60.7 | 83.2% | 237 | |
| | VSI-Bench (5,130) | 58.3 | 60.3 | 59.9 | 0.2% | 6 | |
|
|
| Notes: |
| - These numbers come from our own fixed-budget protocol and are not directly comparable with numbers in other papers. |
| - Mean output length over the six benchmarks is 331 tokens for EVAThink-Stage1 and 213 for EVAThink-Qwen3. |
| - A response counts as "thinking" (think-rate) when it doesn't start directly with the answer. |
| - See the [EVAThink-Qwen3 card](https://huggingface.co/MaplesWCT/EVAThink-Qwen3) for per-benchmark scoring details. |
|
|
| ## Usage |
|
|
| EVAThink-Stage1 uses the standard Qwen3-VL architecture and chat template, so any Qwen3-VL-compatible stack works. |
| Give it just the question and options. |
|
|
| ### Transformers |
|
|
| ```python |
| from transformers import AutoProcessor, Qwen3VLForConditionalGeneration |
| |
| model_id = "MaplesWCT/EVAThink-Stage1" |
| model = Qwen3VLForConditionalGeneration.from_pretrained(model_id, dtype="auto", device_map="auto") |
| processor = AutoProcessor.from_pretrained(model_id) |
| |
| question = "What does the person do right after opening the fridge?\nA. Takes out milk\nB. Closes it\nC. Pours water\nD. Leaves the room" |
| messages = [{ |
| "role": "user", |
| "content": [ |
| {"type": "video", "video": "path/to/video.mp4"}, |
| {"type": "text", "text": question}, |
| ], |
| }] |
| inputs = processor.apply_chat_template( |
| messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", |
| fps=2, |
| ).to(model.device) |
| out = model.generate(**inputs, max_new_tokens=4096, do_sample=False) |
| print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]) |
| ``` |
|
|
| ### vLLM |
|
|
| ```bash |
| vllm serve MaplesWCT/EVAThink-Stage1 --max-model-len 64000 --limit-mm-per-prompt '{"video": 1}' |
| ``` |
|
|
| ### Output format |
|
|
| Same as EVAThink-Qwen3: either the answer alone (e.g. `B`), or a short reasoning passage followed by the final answer. |
| Take the letter at the start of the response; if there isn't one, take the last letter that follows "answer is". |
|
|
| ## Training |
|
|
| <!-- TODO(author): check this outline before release. --> |
|
|
| Stage 1 starts from Qwen3-VL-8B-Instruct and runs GRPO on |
| [EVAThink-Dataset-Stage1](https://huggingface.co/datasets/MaplesWCT/EVAThink-Dataset-Stage1): 1,766 multiple-choice video |
| questions, each shown in a **dense** view (the evidence fills the clip) and a **sparse** view (the evidence is diluted in a |
| longer video), 32 frames per view. |
|
|
| - The reward depends on the view. On the dense view a correct direct answer earns the most; on the sparse view a correct |
| reasoned answer does. Malformed responses are penalized. |
| - A regularizer on the first response token keeps the model from collapsing onto a single mode. |
| - Both views of a question go into the same update: 32 prompts (16 questions × 2 views) × 8 rollouts per step. |
| - One epoch (110 steps), learning rate 1e-6, no KL penalty. |
|
|
| EVAThink-Qwen3 continues from this checkpoint with a stage-2 reward that no longer looks at the view. |
|
|
| ## Limitations |
|
|
| - It reasons on most questions, so it is slower than EVAThink-Qwen3 at about the same accuracy. |
| - It was evaluated on multiple-choice video QA, plus numerical estimation in VSI-Bench. Behavior on open-ended tasks has not been studied. |
| - Evaluation uses a fixed 64-frame budget. Results under other frame budgets may differ. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{evathink2026, |
| title = {TODO}, |
| author = {TODO}, |
| year = {2026}, |
| note = {TODO: arXiv link} |
| } |
| ``` |
|
|
| ## Acknowledgements |
|
|
| Built on [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL). |
|
|
