Align VEFX-Bench v1.0 protocol and metadata

Defines item-first GeoAgg Overall, pins reproducibility identifiers, aligns the nine task categories, and removes inaccessible links.

README.md

@@ -11,13 +11,15 @@ short_description: VEFX-Bench reference code & inference utils
 
 <div align="center">
 
-# VEFX-Bench
+# VEFX-Bench v1.0
 
 ### Benchmarking Generic Video Editing and Visual Effects
 
 </div>
 
-**VEFX-Bench** is a comprehensive benchmark for evaluating text-driven video editing and visual effects. It includes **5,049 annotated examples** spanning **9 categories** and **32 subcategories**, evaluated by **VEFX-Reward** — a VLM-based reward model that scores edits across three dimensions on a 1–4 scale:
+> **Frozen release:** VEFX-Bench v1.0 — frozen on 2026-08-03. See the immutable dataset, manifest, and checkpoint identifiers under [Reproducibility](#-reproducibility).
+
+**VEFX-Bench** is a comprehensive benchmark for evaluating text-driven video editing and visual effects. It includes **5,049 annotated examples** spanning **9 categories** and **32 subcategories**, with a frozen **300-item v1.0 test set** evaluated by **VEFX-Reward** — a VLM-based reward model that scores edits across three dimensions on a 1–4 scale:
 
 | Dimension | What it measures |
 |---|---|
@@ -29,9 +31,9 @@ short_description: VEFX-Bench reference code & inference utils
 
 ## 🏆 Model Leaderboard
 
-VEFX-Reward scores on 1–4 scale. Ranked by **GeoAgg** (α=2 for IF, β=1 for RQ, γ=1 for EE). Higher is better.
+VEFX-Reward scores each dimension on a 1–4 scale. **Overall is the primary GeoAgg metric; it is not the arithmetic average of IF, RQ, and EE.** For item `j`, define `i_j=(IF_j-1)/3`, `r_j=(RQ_j-1)/3`, and `e_j=(EE_j-1)/3`, then compute `g_j=1+3*(i_j^2*r_j*e_j)^(1/4)`. The leaderboard computes **Overall = (1/300)·Σ_j g_j**: GeoAgg is computed per item first, then the 300 values are averaged. Missing or failed items receive `(1,1,1)`. The arithmetic Mean is diagnostic only and is not used for ranking.
 
-> **📅 Updated: May 2, 2026** — For the latest results & submissions, visit the **[live leaderboard →](https://vefx-leaderboard.com/)**
+> **📅 Updated: August 3, 2026** — For the latest results & submissions, visit the **[live leaderboard →](https://vefx-leaderboard.com/)**
 
 | Rank | Model | Type | IF ↑ | RQ ↑ | EE ↑ | GeoAgg ↑ |
 |:---:|---|---|:---:|:---:|:---:|:---:|
@@ -93,14 +95,23 @@ Each demo shows the **original video** (left) alongside the **edited video** (ri
 
 ## 📦 VEFX-Bench Dataset
 
-The benchmark dataset is hosted on HuggingFace at **[VEFX-Reward/VEFX-Bench](https://huggingface.co/datasets/VEFX-Reward/VEFX-Bench)**.
+The canonical frozen benchmark dataset is hosted on Hugging Face at **[xiangbog/VEFX-Bench](https://huggingface.co/datasets/xiangbog/VEFX-Bench/tree/3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235)**. Submit and inspect results on the public **[VEFX-Leaderboard](https://vefx-leaderboard.com/)**.
 
 | | |
 |---|---|
 | 🎬 **300** Source Videos (720p) | 📝 `prompts.json` with editing instructions |
 | 📂 **9** Task Categories | 🗂️ `benchmark_meta.json` with category labels |
 
-**Task Categories:** Style Transfer · Object Manipulation · Background Change · Color/Lighting · Motion/Animation · Text/Overlay · Composition · Removal/Inpainting · Complex/Multi-step
+**Task Categories:** Attribute Editing · Camera Angle Editing · Camera Motion Editing · Creative Edit · Instance Editing · Instance Motion Editing · Quantity Editing · Style Editing · Visual Effect Editing
+
+### 🔒 Reproducibility
+
+| Artifact | Frozen identifier |
+|---|---|
+| Dataset revision | `3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235` |
+| `benchmark_meta.json` SHA-256 | `277d89f5cf23af4163fe6ee120654210f51d19c6dee2d3aa76a3248471d1f333` |
+| Reward model revision | `a15a8dbe1b3eb07ee0919e8de059f170436ec9ff` |
+| `model.safetensors` SHA-256 | `c3c0d03f770f0a73631206821922213de75413bb517f7d7a2fd9ab1f2c38f59d` |
 
 ### Download and Evaluate
 
@@ -108,7 +119,12 @@ The benchmark dataset is hosted on HuggingFace at **[VEFX-Reward/VEFX-Bench](htt
 from huggingface_hub import snapshot_download
 
 # Download the benchmark dataset
-snapshot_download(repo_id="VEFX-Reward/VEFX-Bench", repo_type="dataset", local_dir="./vefx_bench")
+snapshot_download(
+    repo_id="xiangbog/VEFX-Bench",
+    repo_type="dataset",
+    revision="3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235",
+    local_dir="./vefx_bench",
+)
 ```
 
 **Evaluation workflow:**
@@ -171,7 +187,7 @@ scores = model.score(
     instruction="Remove the woman with the grey backpack walking on the right side of the frame.",
 )
 print(scores)
-# {'IF': 2.34, 'RQ': 1.93, 'EE': 1.82, 'Overall': 6.09}
+# {'IF': 2.34, 'RQ': 1.93, 'EE': 1.82, 'Overall': 2.082, 'GeoAgg': 2.082, 'Mean': 2.03}
 ```
 
 ### CLI Usage
@@ -239,7 +255,7 @@ VEFXReward(
 
 #### `model.score(original_video, edited_video, instruction) → dict`
 
-Score a single video edit. Returns `{'IF': float, 'RQ': float, 'EE': float, 'Overall': float}`.
+Score a single video edit. Returns `{'IF': float, 'RQ': float, 'EE': float, 'Overall': float, 'GeoAgg': float, 'Mean': float}`. `Overall` and `GeoAgg` are identical item-level v1.0 scores; `Mean` is diagnostic only.
 
 #### `model.score_batch(original_videos, edited_videos, instructions) → list[dict]`
 

examples/batch_scoring.py

@@ -39,7 +39,7 @@ def main():
         results.append({**row, **scores})
         print(f"[{i+1}/{len(rows)}] IF={scores['IF']:.2f}  RQ={scores['RQ']:.2f}  EE={scores['EE']:.2f}  Overall={scores['Overall']:.2f}")
 
-    fieldnames = list(rows[0].keys()) + ["IF", "RQ", "EE", "Overall"]
+    fieldnames = list(rows[0].keys()) + ["IF", "RQ", "EE", "Overall", "GeoAgg", "Mean"]
     with open(args.output, "w", newline="") as f:
         writer = csv.DictWriter(f, fieldnames=fieldnames)
         writer.writeheader()

examples/multi_gpu_scoring.py

@@ -38,7 +38,7 @@ def worker_main(args):
                   f"IF={scores['IF']:.2f} RQ={scores['RQ']:.2f} EE={scores['EE']:.2f}", flush=True)
         except Exception as e:
             print(f"[GPU {args.gpu_id}] [{i+1}/{len(shard)}] ERROR: {e}", flush=True)
-            results.append({**item, "IF": None, "RQ": None, "EE": None, "Overall": None, "error": str(e)})
+            results.append({**item, "IF": None, "RQ": None, "EE": None, "Overall": None, "GeoAgg": None, "Mean": None, "error": str(e)})
 
     with open(args.output_file, "w") as f:
         json.dump(results, f)
@@ -104,7 +104,7 @@ def main():
             with open(of) as f:
                 all_results.extend(json.load(f))
 
-    fieldnames = list(rows[0].keys()) + ["IF", "RQ", "EE", "Overall"]
+    fieldnames = list(rows[0].keys()) + ["IF", "RQ", "EE", "Overall", "GeoAgg", "Mean"]
     with open(args.output, "w", newline="") as f:
         writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
         writer.writeheader()

examples/quick_start.py

@@ -31,7 +31,8 @@ def main():
     print(f"  Instructional Following (IF): {scores['IF']:.2f}")
     print(f"  Render Quality          (RQ): {scores['RQ']:.2f}")
     print(f"  Edit Exclusivity        (EE): {scores['EE']:.2f}")
-    print(f"  Overall                     : {scores['Overall']:.2f}")
+    print(f"  Overall (GeoAgg, primary)   : {scores['Overall']:.2f}")
+    print(f"  Mean (diagnostic)           : {scores['Mean']:.2f}")
     print("=" * 50)
 
 

index.html

@@ -3,7 +3,7 @@
 <head>
 <meta charset="UTF-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
-<title>VEFX-Code · Reference code for VEFX-Bench</title>
+<title>VEFX-Code · Reference code for VEFX-Bench v1.0</title>
 <style>
   :root { --fg:#1a1a1a; --muted:#666; --bg:#fff; --card:#f7f7f9; --accent:#5b6cff; --border:#e5e7eb; }
   * { box-sizing:border-box; }
@@ -22,6 +22,7 @@
   th { background:var(--card); font-weight:600; }
   td.num, th.num { text-align:center; }
   code { background:var(--card); padding:.15em .4em; border-radius:4px; font-size:.9em; }
+  td code { overflow-wrap:anywhere; word-break:break-all; }
   pre { background:var(--card); padding:1em; border-radius:6px; overflow-x:auto; }
   pre code { background:none; padding:0; }
   .grid { display:grid; grid-template-columns:1fr 1fr; gap:1em; margin:1em 0; }
@@ -35,13 +36,14 @@
 <body>
 
 <h1>🎬 VEFX-Code</h1>
-<p class="tagline">Reference code &amp; inference utilities for the <strong>VEFX-Bench</strong> benchmark — a comprehensive benchmark for evaluating text-driven video editing and visual effects.</p>
+<p class="tagline">Reference code &amp; inference utilities for <strong>VEFX-Bench v1.0</strong> — frozen on 2026-08-03.</p>
 
 <div class="pill-row">
   <a class="pill" href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/tree/main" target="_blank">📂 Browse Files</a>
   <a class="pill" href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/blob/main/README.md" target="_blank">📖 Full README</a>
-  <a class="pill" href="https://huggingface.co/datasets/VEFX-Reward/VEFX-Bench" target="_blank">📦 VEFX-Bench Dataset</a>
-  <a class="pill" href="https://huggingface.co/VEFX-Reward/VEFX-Reward-4B" target="_blank">🤖 VEFX-Reward-4B Model</a>
+  <a class="pill" href="https://huggingface.co/datasets/xiangbog/VEFX-Bench/tree/3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235" target="_blank">📦 Frozen v1.0 Dataset</a>
+  <a class="pill" href="https://huggingface.co/VEFX-Reward/VEFX-Reward-4B/tree/a15a8dbe1b3eb07ee0919e8de059f170436ec9ff" target="_blank">🤖 Frozen Reward Model</a>
+  <a class="pill" href="https://vefx-leaderboard.com/" target="_blank">🏆 Live VEFX-Leaderboard</a>
 </div>
 
 <h2>📊 What's in VEFX-Bench</h2>
@@ -53,8 +55,21 @@
   <div class="card"><h3>Edit Exclusivity (EE)</h3><p>Were only the intended regions modified, without side-effects?</p></div>
 </div>
 
+<h2>🗂️ Frozen v1.0 Task Taxonomy</h2>
+<p>Attribute Editing · Camera Angle Editing · Camera Motion Editing · Creative Edit · Instance Editing · Instance Motion Editing · Quantity Editing · Style Editing · Visual Effect Editing</p>
+
+<h2>🔒 Reproducibility</h2>
+<table>
+  <tbody>
+    <tr><th>Dataset revision</th><td><code>3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235</code></td></tr>
+    <tr><th>benchmark_meta.json SHA-256</th><td><code>277d89f5cf23af4163fe6ee120654210f51d19c6dee2d3aa76a3248471d1f333</code></td></tr>
+    <tr><th>Reward model revision</th><td><code>a15a8dbe1b3eb07ee0919e8de059f170436ec9ff</code></td></tr>
+    <tr><th>model.safetensors SHA-256</th><td><code>c3c0d03f770f0a73631206821922213de75413bb517f7d7a2fd9ab1f2c38f59d</code></td></tr>
+  </tbody>
+</table>
+
 <h2>🏆 Model Leaderboard</h2>
-<p>VEFX-Reward scores on 1–4 scale. Ranked by <strong>GeoAgg</strong> (α=2 for IF, β=1 for RQ, γ=1 for EE). Higher is better.</p>
+<p><strong>Overall is the primary GeoAgg metric, not the arithmetic average.</strong> For each item <code>j</code>, first compute <code>i=(IF-1)/3</code>, <code>r=(RQ-1)/3</code>, and <code>e=(EE-1)/3</code>, then <code>g_j=1+3·(i²·r·e)^(1/4)</code>. The leaderboard uses <code>Overall=(1/300)·Σg_j</code>: item-level GeoAgg first, then the mean over 300 items. Missing or failed items receive scores <code>(1,1,1)</code>. The arithmetic Mean is diagnostic only.</p>
 <table>
   <thead><tr><th class="num">Rank</th><th>Model</th><th>Type</th><th class="num">IF ↑</th><th class="num">RQ ↑</th><th class="num">EE ↑</th><th class="num">GeoAgg ↑</th></tr></thead>
   <tbody>
@@ -89,7 +104,7 @@ python examples/quick_start.py \
 <p>See <a href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/tree/main/examples" target="_blank">examples/</a> for batch &amp; multi-GPU scoring scripts.</p>
 
 <footer>
-  Released under Apache-2.0. Code lives in the <a href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/tree/main" target="_blank">Files tab</a>.
+  Released under Apache-2.0. Code lives in the <a href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/tree/main" target="_blank">Files tab</a>; results and submissions are on the <a href="https://vefx-leaderboard.com/" target="_blank">live leaderboard</a>.
 </footer>
 
 </body>

vefx_reward/inference.py

@@ -6,7 +6,7 @@ Usage:
 
     model = VEFXReward("VEFX-Reward/VEFX-Reward-4B", device="cuda")
     scores = model.score("original.mp4", "edited.mp4", "add a hat to the person")
-    # {'IF': 3.21, 'RQ': 2.85, 'EE': 3.54, 'Overall': 9.60}
+    # {'IF': 3.21, 'RQ': 2.85, 'EE': 3.54, 'Overall': 3.189, 'GeoAgg': 3.189, 'Mean': 3.20}
 """
 
 import json
@@ -56,7 +56,7 @@ class VEFXReward:
         model = VEFXReward("VEFX-Reward/VEFX-Reward-4B")
         scores = model.score("original.mp4", "edited.mp4", "make it snowy")
         print(scores)
-        # {'IF': 3.2, 'RQ': 2.9, 'EE': 3.5, 'Overall': 9.6}
+        # {'IF': 3.2, 'RQ': 2.9, 'EE': 3.5, 'Overall': 3.19, 'GeoAgg': 3.19, 'Mean': 3.2}
     """
 
     def __init__(
@@ -177,7 +177,17 @@ class VEFXReward:
             scores = {DIMS[j]: round(float(soft[0, j]), 3) for j in range(num_dims)}
         else:
             scores = {DIMS[j]: round(float(logits_np[0, j]), 3) for j in range(self.output_dim)}
-        scores["Overall"] = round(sum(scores[d] for d in DIMS), 3)
+        # VEFX-Bench v1.0 item-level Overall. Normalize scores to [0,1],
+        # apply the weighted geometric aggregate (IF weight 2), then map
+        # back to [1,4]. Dataset Overall is the mean of these item values.
+        i = max(0.0, min(1.0, (scores["IF"] - 1.0) / 3.0))
+        r = max(0.0, min(1.0, (scores["RQ"] - 1.0) / 3.0))
+        e = max(0.0, min(1.0, (scores["EE"] - 1.0) / 3.0))
+        overall = 1.0 + 3.0 * ((i ** 2) * r * e) ** 0.25
+
+        scores["Overall"] = round(overall, 3)
+        scores["GeoAgg"] = scores["Overall"]
+        scores["Mean"] = round(sum(scores[d] for d in DIMS) / len(DIMS), 3)
         return scores
 
     @torch.no_grad()
@@ -195,8 +205,10 @@ class VEFXReward:
             instruction: The editing instruction text.
 
         Returns:
-            Dictionary with keys ``'IF'``, ``'RQ'``, ``'EE'``, ``'Overall'``.
-            Each dimension is scored on a continuous 1–4 scale.
+            Dictionary with keys ``'IF'``, ``'RQ'``, ``'EE'``, ``'Overall'``,
+            ``'GeoAgg'``, and ``'Mean'``. ``Overall`` and ``GeoAgg`` are the
+            same primary item-level v1.0 score; ``Mean`` is diagnostic only.
+            All values use the continuous 1–4 scale.
         """
         batch = self._build_batch(original_video, edited_video, instruction)
         logits = self.model(**batch, return_dict=True)["logits"]