MaplesWCT commited on
Commit
664c429
·
verified ·
1 Parent(s): 68efbcf

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -18
README.md CHANGED
@@ -14,7 +14,7 @@ tags:
14
 
15
  <!-- TODO(author): one-line expansion of the name, paper title / arXiv link, code repo link -->
16
 
17
- **EVAThink-Qwen3** is a video-language model fine-tuned from [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) with reinforcement learning to **decide per question whether to think**. When the visual evidence plainly supports an answer, it replies directly. When the evidence is sparse, discontinuous, or scattered across frames, it writes a short, evidence-grounded reasoning trace first.
18
 
19
  You don't need to tell it which mode to use. Give it the plain question and options, and it picks the mode itself.
20
 
@@ -43,7 +43,7 @@ Each model uses its intended prompt:
43
  | MVBench (4,000) | 65.8 | **69.0** | 66.5 | 5.1% |
44
  | MMVU, multiple-choice (625) | 70.4 | 69.6 | **72.8** | 59.4% |
45
  | Video-MMMU (900) | 65.6 | 64.3 | **67.4** | 85.7% |
46
- | LongVideoBench val (1,337) | **60.7** | 54.6 | **60.7** | 46.1% |
47
  | VSI-Bench (5,130) | 58.3 | 54.6 | **59.9** | 0.0% |
48
 
49
  Notes:
@@ -97,21 +97,6 @@ vllm serve MaplesWCT/EVAThink-Qwen3 --max-model-len 64000 --limit-mm-per-prompt
97
 
98
  A simple parser covers both: take the letter at the start of the response. If there isn't one, take the last letter that follows "answer is" (ignoring `**` and `:`). If that also fails, fall back to the last standalone option letter in the response.
99
 
100
- ## Training
101
-
102
- <!-- TODO(author): fill in data sources, sizes, and the exact recipe. The outline below reflects our internal setup; please check it before release. -->
103
-
104
- EVAThink starts from Qwen3-VL-8B-Instruct and is trained in two RL (GRPO) stages on multiple-choice video QA with verifiable answers:
105
-
106
- 1. **Stage 1: learning when to think.** Training uses paired samples of questions whose evidence is dense vs. sparse, together with a regularizer on the first token that shapes the choice between answering directly and reasoning.
107
- 2. **Stage 2: calibrating the mode choice.** Training continues on held-out questions with a reward that slightly favors correct direct answers over correct reasoned answers (1.1 vs. 1.0; wrong answers get 0), which discourages thinking when it isn't needed.
108
-
109
- ## Limitations
110
-
111
- - Because the model mostly thinks on the questions it's unsure of, questions it thinks on have lower accuracy than questions it answers directly. That is a selection effect: it doesn't mean thinking hurts.
112
- - The model was evaluated on multiple-choice video QA, plus numerical estimation in VSI-Bench. Behavior on open-ended captioning or grounding tasks has not been studied.
113
- - Evaluation uses a fixed 64-frame budget. Results under other frame budgets may differ.
114
-
115
  ## Citation
116
 
117
  ```bibtex
@@ -125,4 +110,4 @@ EVAThink starts from Qwen3-VL-8B-Instruct and is trained in two RL (GRPO) stages
125
 
126
  ## Acknowledgements
127
 
128
- Built on [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL). We compare against [VideoAuto-R1](https://github.com/IVUL-KAUST/VideoAuto-R1).
 
14
 
15
  <!-- TODO(author): one-line expansion of the name, paper title / arXiv link, code repo link -->
16
 
17
+ **EVAThink-Qwen3** is a vision-language model fine-tuned from [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) with reinforcement learning to **decide per question whether to think**. When the visual evidence plainly supports an answer, it replies directly. When the evidence is sparse, discontinuous, or scattered across frames, it writes a short, evidence-grounded reasoning trace first.
18
 
19
  You don't need to tell it which mode to use. Give it the plain question and options, and it picks the mode itself.
20
 
 
43
  | MVBench (4,000) | 65.8 | **69.0** | 66.5 | 5.1% |
44
  | MMVU, multiple-choice (625) | 70.4 | 69.6 | **72.8** | 59.4% |
45
  | Video-MMMU (900) | 65.6 | 64.3 | **67.4** | 85.7% |
46
+ | LongVideoBench val (1,337) | 60.4 | 54.6 | **60.7** | 46.1% |
47
  | VSI-Bench (5,130) | 58.3 | 54.6 | **59.9** | 0.0% |
48
 
49
  Notes:
 
97
 
98
  A simple parser covers both: take the letter at the start of the response. If there isn't one, take the last letter that follows "answer is" (ignoring `**` and `:`). If that also fails, fall back to the last standalone option letter in the response.
99
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
100
  ## Citation
101
 
102
  ```bibtex
 
110
 
111
  ## Acknowledgements
112
 
113
+ Built on [Qwen3-VL](https://github.com/QwenLM/Qwen3-VL).