yiyexy commited on
Commit
bf93051
·
verified ·
1 Parent(s): 099bdbd

TimeLens ActivityNet: Qwen score at the aligned setting

Browse files
Files changed (2) hide show
  1. README.md +2 -2
  2. README_CN.md +2 -2
README.md CHANGED
@@ -115,7 +115,7 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
115
  | LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
116
  | VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
117
  | **Temporal grounding (mIoU)** | | | | | | |
118
- | TimeLens ActivityNet | 46.83 | 128f × 153,664 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
119
  | TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
120
  | TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
121
 
@@ -124,7 +124,7 @@ OV2.5-4B (stage 4, iteration 11750) against **Qwen3.5-4B with thinking disabled*
124
  \* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
125
  \* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
126
  \* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
127
- \* Settings are the ones that produced each reported score. Most rows are matched, but three are not: on **TimeLens ActivityNet** we ran 2048 frames × 409,600 px against Qwen's 128 × 153,664, so that +12.01 is not like-for-like (TimeLens Charades, at identical settings, is +12.53). On **TOMATO** Qwen ran 16 frames at fps=2 while we used the model defaults, and on **MMOU** only we have a score.
128
  \* Where a score is the best over several settings, the search was uneven and mostly favoured Qwen: VideoMME long no-subtitles 28 settings vs our 6, LVBench 27 vs 16, LongVideoBench 28 vs 2, VideoEval-Pro 28 vs 2.
129
  \* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
130
 
 
115
  | LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
116
  | VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
117
  | **Temporal grounding (mIoU)** | | | | | | |
118
+ | TimeLens ActivityNet | 53.84 | 2048f × 409,600, fps=4 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
119
  | TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
120
  | TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
121
 
 
124
  \* Video-Index 512f: frames are supplied by the task as images, so both sides take multi-image input. The codec number comes from the official-budget run (TC256 cap); the matching frame-sampling control on the video files scores 19.05.
125
  \* EgoSchema: the 500-question public subset. Answers to the full 5031 questions are not public.
126
  \* VideoMME with subtitles and MMOU: Qwen has no comparable result on the official path. Qwen's VideoMME long with subtitles run was invalid (judge recorded 0/900) and is left out rather than reported.
127
+ \* Settings are the ones that produced each reported score. Most rows are matched; two are not: on **TOMATO** Qwen ran 16 frames at fps=2 while we used the model defaults, and on **MMOU** only we have a score.
128
  \* Where a score is the best over several settings, the search was uneven and mostly favoured Qwen: VideoMME long no-subtitles 28 settings vs our 6, LVBench 27 vs 16, LongVideoBench 28 vs 2, VideoEval-Pro 28 vs 2.
129
  \* The codec column is scored by rule, since the answers are option letters and the judge is equivalent there.
130
 
README_CN.md CHANGED
@@ -115,7 +115,7 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
115
  | LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
116
  | VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
117
  | **时间定位(mIoU)** | | | | | | |
118
- | TimeLens ActivityNet | 46.83 | 128f × 153,664 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
119
  | TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
120
  | TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
121
 
@@ -124,7 +124,7 @@ OV2.5-4B(stage 4,iteration 11750)对 **Qwen3.5-4B(关闭 thinking)**
124
  \* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
125
  \* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
126
  \* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
127
- \* 设置列是各自取得该分数时的实际设置。多数行两边一致,但有三行不是:**TimeLens ActivityNet** 我方跑的是 2048 帧 × 409,600 px,Qwen 是 128 × 153,664,因此那 +12.01 并非同条件对比(设置完全相同的 TimeLens Charades 为 +12.53);**TOMATO** Qwen 为 16 帧 fps=2 而我方用模型默认值;**MMOU** 只有我方有分。
128
  \* 对「取多设置最高分」的行,两边搜索量并不对等,且多数对 Qwen 更有利:VideoMME long 无字幕 28 个设置对我方 6 个,LVBench 27 对 16,LongVideoBench 28 对 2,VideoEval-Pro 28 对 2。
129
  \* codec 一列为规则分——回答均为选项字母,与 judge 等价。
130
 
 
115
  | LongVideoBench val | 62.68 | 700f × 300,000 | 68.96 | 1280f × 80,000 | 71.06 | TC768 × 235,200 |
116
  | VideoEval-Pro | 62.84 | 2048f × 100,000 | 64.39 | 950f × 200,000 | 64.62 | TC768 × 235,200 |
117
  | **时间定位(mIoU)** | | | | | | |
118
+ | TimeLens ActivityNet | 53.84 | 2048f × 409,600, fps=4 | 58.84 | 2048f × 409,600, fps=4 | 59.11 | TC128 × 313,600 |
119
  | TimeLens Charades | 48.05 | 2048f × 409,600, fps=4 | 60.58 | 2048f × 409,600, fps=4 | 61.25 | TC128 × 313,600 |
120
  | TimeLens QVHighlights | 67.51 | 2048f × 409,600, fps=4 | 68.39 | 2048f × 409,600, fps=4 | 70.31 | TC128 × 313,600 |
121
 
 
124
  \* Video-Index 512f:帧由任务以图片给出,两边都是多图输入。codec 一列取自官方预算测试(TC256 上限);同一批的抽帧对照(视频文件输入)为 19.05。
125
  \* EgoSchema:公开答案的 500 题子集。全量 5031 题答案不公开。
126
  \* VideoMME 有字幕三项与 MMOU:Qwen 在官方路径下没有可比结果。Qwen 的 VideoMME long 有字幕那次运行无效(judge 记录为 0/900),故留空而非报出。
127
+ \* 设置列是各自取得该分数时的实际设置。多数行两边一致,有两行不是:**TOMATO** Qwen 为 16 帧 fps=2 而我方用模型默认值;**MMOU** 只有我方有分。
128
  \* 对「取多设置最高分」的行,两边搜索量并不对等,且多数对 Qwen 更有利:VideoMME long 无字幕 28 个设置对我方 6 个,LVBench 27 对 16,LongVideoBench 28 对 2,VideoEval-Pro 28 对 2。
129
  \* codec 一列为规则分——回答均为选项字母,与 judge 等价。
130