ifire commited on
Commit
def2839
·
verified ·
1 Parent(s): da50cbf

Rewrite the card in the house style with durable facts only

Browse files
Files changed (1) hide show
  1. README.md +16 -201
README.md CHANGED
@@ -1,215 +1,30 @@
1
- <p align="center">
2
- <img src="https://raw.githubusercontent.com/VectorSpaceLab/EditScore/refs/heads/main/assets/logo.png" width="65%">
3
- </p>
4
-
5
- <p align="center">
6
- <a href="https://vectorspacelab.github.io/EditScore"><img src="https://img.shields.io/badge/Project%20Page-EditScore-yellow" alt="project page"></a>
7
- <a href="https://arxiv.org/abs/2509.23909"><img src="https://img.shields.io/badge/arXiv%20paper-2509.23909-b31b1b.svg" alt="arxiv"></a>
8
- <a href="https://huggingface.co/collections/EditScore/editscore-68d8e27ee676981221db3cfe"><img src="https://img.shields.io/badge/EditScore-🤗-yellow" alt="model"></a>
9
- <a href="https://huggingface.co/datasets/EditScore/EditReward-Bench"><img src="https://img.shields.io/badge/EditReward--Bench-🤗-yellow" alt="dataset"></a>
10
- <a href="https://huggingface.co/datasets/EditScore/EditScore-Reward-Data"><img src="https://img.shields.io/badge/EditScore--Reward--Data-🤗-yellow" alt="dataset"></a>
11
- <a href="https://huggingface.co/datasets/EditScore/EditScore-RL-Data"><img src="https://img.shields.io/badge/EditScore--RL--Data-🤗-yellow" alt="dataset"></a>
12
- </p>
13
-
14
- <h4 align="center">
15
- <p>
16
- <a href=#-news>News</a> |
17
- <a href=#-quick-start>Quick Start</a> |
18
- <a href=#-benchmark-your-image-editing-reward-model usage>Benchmark Usage</a> |
19
- <a href=#%EF%B8%8F-citing-us>Citation</a>
20
- <p>
21
- </h4>
22
-
23
- **EditScore** is a series of state-of-the-art open-source reward models (7B–72B) designed to evaluate and enhance instruction-guided image editing.
24
- ## ✨ Highlights
25
- - **State-of-the-Art Performance**: Effectively matches the performance of leading proprietary VLMs. With a self-ensembling strategy, **our largest model surpasses even GPT-5** on our comprehensive benchmark, **EditReward-Bench**.
26
- - **A Reliable Evaluation Standard**: We introduce **EditReward-Bench**, the first public benchmark specifically designed for evaluating reward models in image editing, featuring 13 subtasks, 11 state-of-the-art editing models (*including proprietary models*) and expert human annotations.
27
- - **Simple and Easy-to-Use**: Get an accurate quality score for your image edits with just a few lines of code.
28
- - **Versatile Applications**: Ready to use as a best-in-class reranker to improve editing outputs, or as a high-fidelity reward signal for **stable and effective Reinforcement Learning (RL) fine-tuning**.
29
-
30
- ## 🔥 News
31
- - **2026-02-01**: Our work has been accepted to ICLR 2026 🎉
32
- - **2025-11-21**: We're excited to release the training configs for EditScore reward models! 🎯 Built on LLaMA-Factory, we trained EditScore ranging from 4B to 72B parameters with simple YAML configurations. Check out the [EditScore-train guide](examples/EditScore-train/README.md) to get started with training your own reward models.
33
- - **2025-10-31**: We’re thrilled to announce the **Qwen3-VL** variants of **EditScore**! 🚀 Powered by Qwen3-VL, the new 4B and 8B models achieve outstanding efficiency and performance. Impressively, the 4B model already matches the performance of the original 32B version, while the 8B model delivers results comparable to the original 72B model. The models are now available on [huggingface](https://huggingface.co/EditScore/models), see [Usage Example](#-usage-example) for how to use. Detailed comparisons with Qwen2.5-VL variants are in the [performance table](https://raw.githubusercontent.com/VectorSpaceLab/EditScore/refs/heads/main/assets/table_editscore_qwen3_vl.png).
34
- - **2025-10-27**: Released [OmniGen2-EditScore7B-v1.1](https://huggingface.co/OmniGen2/OmniGen2-EditScore7B-v1.1), achieving a **7.01 (+0.73) GEdit score** within **700 steps**, by incorporating the **reweighting strategy** from [TempFlow](https://arxiv.org/abs/2508.04324). Additionally, the **JSON repair process** has been enhanced using [json_repair](https://github.com/mangiucugna/json_repair), improving **EditScore’s stability** under various conditions. Upgrade via `pip install -U editscore`.
35
- - **2025-10-22**: **Introducing Our Reinforcement Learning Training Framework!**
36
- We're excited to release our complete RL pipeline, the result of a massive effort to simplify fine-tuning for image editing models. Key features include:
37
- - **Ready-to-Use RL Dataset**: Includes the complete dataset used in the EditScore project, along with clear usage guidelines and preparation scripts.
38
- - **An Easy-to-Use Reward Model**: Seamlessly integrate **EditScore** as a reward signal.
39
- - **A Scalable Reward Server**: Built with native multi-node support for high-throughput training.
40
- - **Flexible Training Code**: Supports distributed training, variable image resolutions and mixed tasks (t2i, edit, in-context generation) out-of-the-box.
41
- Dive into our comprehensive guide on [RL Fine-Tuning](examples/OmniGen2-RL#application-2-reinforcement-fine-tuning) to get started.
42
-
43
- - 2025-10-16: Training datasets [EditScore-Reward-Data](https://huggingface.co/datasets/EditScore/EditScore-Reward-Data) and [EditScore-RL-Data](https://huggingface.co/datasets/EditScore/EditScore-RL-Data) are available.
44
- - 2025-10-15: **EditScore** is now available on PyPI — install it easily with `pip install editscore`.
45
- - 2025-10-15: Best-of-N inference scripts for OmniGen2, Flux-dev-Kontext, and Qwen-Image-Edit are now available! See [this](#apply-editscore-to-image-editing) for details.
46
- - 2025-09-30: We release **OmniGen2-EditScore7B**, unlocking online RL For Image Editing via high-fidelity EditScore. LoRA weights are available at [Hugging Face](https://huggingface.co/OmniGen2/OmniGen2-EditScore7B) and [ModelScope](https://www.modelscope.cn/models/OmniGen2/OmniGen2-EditScore7B).
47
- - 2025-09-30: We are excited to release **EditScore** and **EditReward-Bench**! Model weights and the benchmark dataset are now publicly available. You can access them on Hugging Face: [Models Collection](https://huggingface.co/collections/EditScore/editscore-68d8e27ee676981221db3cfe) and [Benchmark Dataset](https://huggingface.co/datasets/EditScore/EditReward-Bench), and on ModelScope: [Models Collection](https://www.modelscope.cn/collections/EditScore-8b0d53aa945d4e) and [Benchmark Dataset](https://www.modelscope.cn/datasets/EditScore/EditReward-Bench).
48
-
49
- ## 📖 Introduction
50
- While Reinforcement Learning (RL) holds immense potential for this domain, its progress has been severely hindered by the absence of a high-fidelity, efficient reward signal.
51
-
52
- To overcome this barrier, we provide a systematic, two-part solution:
53
-
54
- - **A Rigorous Evaluation Standard**: We first introduce **EditReward-Bench**, a new public benchmark for the direct and reliable evaluation of reward models. It features 13 diverse subtasks and expert human annotations, establishing a gold standard for measuring reward signal quality.
55
-
56
- - **A Powerful & Versatile Tool**: Guided by our benchmark, we developed the **EditScore** model series. Through meticulous data curation and an effective self-ensembling strategy, EditScore sets a new state of the art for open-source reward models, even surpassing the accuracy of leading proprietary VLMs.
57
-
58
- <p align="center">
59
- <img src="https://raw.githubusercontent.com/VectorSpaceLab/EditScore/refs/heads/main/assets/table_reward_model_results.png" width="95%">
60
- <br>
61
- <em>Benchmark results on EditReward-Bench.</em>
62
- </p>
63
-
64
- We demonstrate the practical utility of EditScore through two key applications:
65
-
66
- - **As a State-of-the-Art Reranker**: Use EditScore to perform Best-of-*N* selection and instantly improve the output quality of diverse editing models.
67
- - **As a High-Fidelity Reward for RL**: Use EditScore as a robust reward signal to fine-tune models via RL, enabling stable training and unlocking significant performance gains where general-purpose VLMs fail.
68
-
69
- This repository releases both the **EditScore** models and the **EditReward-Bench** dataset to facilitate future research in reward modeling, policy optimization, and AI-driven model improvement.
70
-
71
- <p align="center">
72
- <img src="https://raw.githubusercontent.com/VectorSpaceLab/EditScore/refs/heads/main/assets/figure_edit_results.png" width="95%">
73
- <br>
74
- <em>EditScore as a superior reward signal for image editing.</em>
75
- </p>
76
-
77
-
78
- ## 📌 TODO
79
- We are actively working on improving EditScore and expanding its capabilities. Here's what's next:
80
-
81
- - [x] Qwen3-VL variants of EditScore.
82
- - [x] Release training data for reward model and online RL.
83
- - [x] Release RL training code applying EditScore to OmniGen2.
84
- - [x] Provide Best-of-N inference scripts for OmniGen2, Flux-dev-Kontext, and Qwen-Image-Edit.
85
 
86
- ## 🚀 Quick Start
87
 
88
- ### 🛠️ Environment Setup
89
- We offer two ways to install EditScore. Choose the one that best fits your needs.
90
- **Method 1: Install from PyPI (Recommended for Users)**: If you want to use EditScore as a library in your own project.
91
- **Method 2: Install from Source (For Developers)**: If you plan to contribute to the code, modify it, or run the examples in this repository
92
 
93
- #### Prerequisites: Installing PyTorch
94
- Both installation methods require PyTorch to be installed first, as its version is dependent on your system's CUDA setup.
95
- ```bash
96
- # (Optional) Create a clean Python environment
97
- conda create -n editscore python=3.12
98
- conda activate editscore
99
 
100
- # Choose the command that matches your CUDA version.
101
- # This example is for CUDA 12.6.
102
- pip install torch==2.7.1 torchvision --extra-index-url https://download.pytorch.org/whl/cu126
103
- ````
104
 
105
- <details>
106
- <summary>🌏 For users in Mainland China</summary>
107
- ```bash
108
- # Install PyTorch from a domestic mirror
109
- pip install torch==2.7.1 torchvision --index-url https://mirror.sjtu.edu.cn/pytorch-wheels/cu126
110
- ```
111
- </details>
112
 
113
- #### Method 1: Install from PyPI (Recommended for Users)
114
- ```bash
115
- pip install -U editscore
116
- ```
117
 
118
- #### Method 2: Install from Source (For Developers)
119
- This method gives you a local, editable version of the project.
120
- 1. Clone the repository
121
- ```bash
122
- git clone https://github.com/VectorSpaceLab/EditScore.git
123
  cd EditScore
124
- ```
125
-
126
- 2. Install EditScore in editable mode
127
- ```bash
128
  pip install -e .
129
- ```
130
-
131
- #### ✅ (Recommended) Install Optional High-Performance Dependencies
132
- For the best performance, especially during inference, we highly recommend installing vllm.
133
- ```bash
134
- pip install -U vllm
135
- ```
136
-
137
- ---
138
-
139
- ### 🧪 Usage Example
140
- Using EditScore is straightforward. The model will be automatically downloaded from the Hugging Face Hub on its first run.
141
- ```python
142
- from PIL import Image
143
- from editscore import EditScore
144
-
145
- # Load the EditScore model. It will be downloaded automatically.
146
- # Replace with the specific model version you want to use.
147
- model_path = "Qwen/Qwen3-VL-4B-Instruct"
148
- lora_path = "EditScore/EditScore-Qwen3-VL-4B-Instruct"
149
-
150
- scorer = EditScore(
151
- backbone="qwen3vl", # set to "qwen3vl_vllm" for faster inference
152
- model_name_or_path=model_path,
153
- lora_path=lora_path,
154
- score_range=25,
155
- num_pass=1, # Increase for better performance via self-ensembling
156
- )
157
-
158
- # Below is Qwen2.5-VL version
159
-
160
- # model_path = "Qwen/Qwen2.5-VL-7B-Instruct"
161
- # lora_path = "EditScore/EditScore-7B"
162
-
163
- # scorer = EditScore(
164
- # backbone="qwen25vl", # set to "qwen25vl_vllm" for faster inference
165
- # model_name_or_path=model_path,
166
- # lora_path=lora_path,
167
- # score_range=25,
168
- # num_pass=1, # Increase for better performance via self-ensembling
169
- # )
170
-
171
- input_image = Image.open("example_images/input.png")
172
- output_image = Image.open("example_images/output.png")
173
- instruction = "Adjust the background to a glass wall."
174
-
175
- result = scorer.evaluate([input_image, output_image], instruction)
176
- print(f"Edit Score: {result['overall']}")
177
- # Expected output: A dictionary containing the final score and other details.
178
- ```
179
-
180
- ---
181
-
182
- ## 📊 Benchmark Your Image-Editing Reward Model
183
- #### Install benchmark dependencies
184
- To use example code for benchmark, run following
185
- ```bash
186
- pip install -r requirements.txt
187
- ```
188
-
189
- We provide an evaluation script to benchmark reward models on **EditReward-Bench**. To evaluate your own custom reward model, simply create a scorer class with a similar interface and update the script.
190
- ```bash
191
- # This script will evaluate the default EditScore model on the benchmark
192
  bash evaluate.sh
193
-
194
- # Or speed up inference with VLLM
195
- bash evaluate_vllm.sh
196
  ```
197
 
198
- ## Apply EditScore to Image Editing
199
- We offer two example use cases for your exploration:
200
- - **Best-of-N selection**: Use EditScore to automatically pick the most preferred image among multiple candidates.
201
- - **Reinforcement fine-tuning**: Use EditScore as a reward model to guide RL-based optimization.
202
 
203
- For detailed instructions and examples, please refer to the [documentation](examples/OmniGen2-RL/README.md).
204
 
205
- ## ❤️ Citing Us
206
- If you find this repository or our work useful, please consider giving a star ⭐ and citation 🦖, which would be greatly appreciated:
207
-
208
- ```bibtex
209
- @article{luo2025editscore,
210
- title={EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling},
211
- author={Xin Luo and Jiahao Wang and Chenyuan Wu and Shitao Xiao and Xiyan Jiang and Defu Lian and Jiajun Zhang and Dong Liu and Zheng Liu},
212
- journal={arXiv preprint arXiv:2509.23909},
213
- year={2025}
214
- }
215
- ```
 
1
+ ---
2
+ license: other
3
+ ---
4
+ # EditScore-code
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
 
6
+ Mirror of the EditScore source code: the `editscore` Python package, evaluation scripts and training examples for image-edit reward models.
7
 
8
+ ## What it is for
 
 
 
9
 
10
+ The repo copies the EditScore code repo. It holds the `editscore` package, which loads an EditScore reward model and scores an image edit against its instruction. It also holds evaluation scripts that benchmark reward models on EditReward-Bench, and examples for training EditScore reward models, best-of-N selection and reinforcement fine-tuning of image editing models. The model weights live in separate repos.
 
 
 
 
 
11
 
12
+ ## Provenance
 
 
 
13
 
14
+ - Upstream source: [VectorSpaceLab/EditScore](https://github.com/VectorSpaceLab/EditScore). Paper: [arXiv 2509.23909](https://arxiv.org/abs/2509.23909). Project page: [EditScore](https://vectorspacelab.github.io/EditScore).
15
+ - Related upstream data: [EditReward-Bench](https://huggingface.co/datasets/EditScore/EditReward-Bench), [EditScore-Reward-Data](https://huggingface.co/datasets/EditScore/EditScore-Reward-Data), [EditScore-RL-Data](https://huggingface.co/datasets/EditScore/EditScore-RL-Data). Models: the [EditScore collection](https://huggingface.co/collections/EditScore/editscore-68d8e27ee676981221db3cfe).
 
 
 
 
 
16
 
17
+ ## Use
 
 
 
18
 
19
+ ```sh
20
+ hf download chibifire/EditScore-code --local-dir EditScore
 
 
 
21
  cd EditScore
 
 
 
 
22
  pip install -e .
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
  bash evaluate.sh
 
 
 
24
  ```
25
 
26
+ Install PyTorch for your CUDA version first. `vllm` is optional and speeds up inference. See the [upstream README](https://github.com/VectorSpaceLab/EditScore) for full setup, usage and the training and RL guides under `examples/`.
 
 
 
27
 
28
+ ## Licence
29
 
30
+ The original card names no licence, so the header says `other`. The terms are in the `LICENSE` file shipped with the code, copied from [upstream](https://github.com/VectorSpaceLab/EditScore).