File size: 7,341 Bytes
4b3655c cacf9d2 4b3655c cacf9d2 a0188b2 cacf9d2 2992ea0 cacf9d2 2992ea0 cacf9d2 2992ea0 cacf9d2 2992ea0 cacf9d2 2992ea0 cacf9d2 2992ea0 cacf9d2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | ---
license: cc-by-4.0
base_model: black-forest-labs/FLUX.1-dev
library_name: diffusers
tags:
- diffusion
- flow-matching
- flux
- text-to-image
- lora
- policy-distillation
- reinforcement-learning
- academic-research
pipeline_tag: text-to-image
---
# CapField-OPD
### Learning Continuous Capability Fields via Joint-Anchored Multi-Teacher On-Policy Distillation for Flow Models
[](https://arxiv.org/abs/2609.34658v2)
[](PROJECT_PAGE_URL)
[](https://github.com/LPengYang/CapField-OPD)
## Model Card
This Hugging Face repository hosts the released LoRA checkpoints for **CapField-OPD**, an on-policy distillation (OPD) framework that consolidates multiple reward-specialized teachers into a single student through a **continuous capability field**.
Existing multi-teacher OPD methods route each prompt to a single teacher according to its semantic category, implicitly binding the desired capability to prompt content. CapField-OPD instead conditions the student on an **explicit capability coordinate** $\boldsymbol{\lambda} \in [0,1]^K$, whose axes control the strength of each capability. The base model, single-capability teachers, and joint-capability teachers serve as **anchors** of the field, and their outputs are combined through coordinate-dependent activation weights, so that every capability configuration receives a unique supervision target and capability control no longer depends on prompt semantics.
The learned field supports:
- **Continuous capability control** — smoothly adjust the strength of each capability (e.g., text rendering, compositional fidelity, visual aesthetics) at inference time, including capability combinations.
- **Robust capability invocation** — the requested capability stays active under semantics-preserving prompt rewriting and on out-of-distribution prompts where semantic routing fails.
- **Capability extrapolation** — coordinates can extend beyond the training anchors, which act as reference states rather than hard performance limits, sometimes yielding higher rewards than the anchors themselves.
- **Coordinate-based test-time scaling** — a small calibration set is used to profile the capability landscape; the coordinate with the highest mean reward serves as the recommended default, and frequently optimal coordinates form a candidate set for test-time search without resampling random seeds.
The released checkpoints are intended to support reproducible academic research on flow model alignment, reward optimization, and on-policy distillation.
## Released Checkpoints
The repository contains LoRA checkpoints trained from `black-forest-labs/FLUX.1-dev` (LoRA rank $r=64$, $\alpha=128$):
| Path | Description |
| --- | --- |
| `student_ckpt/` | Unified CapField-OPD student distilled over the continuous capability field. Contains `adapter_model.safetensors` (LoRA weights), `adapter_config.json`, and `task_conditioner.pt` (the capability-coordinate conditioner; all three files are required for coordinate-conditioned inference). |
| `teachers_ckpt/geneval_teacher/` | Single-capability teacher optimized for GenEval-style compositional generation. |
| `teachers_ckpt/ocr_teacher/` | Single-capability teacher optimized for OCR/text rendering accuracy. |
| `teachers_ckpt/aesthetic_teacher/` | Single-capability teacher optimized for visual aesthetics (HPSv3, CLIP, and PickScore combined with equal weights). |
| `teachers_ckpt/geneval_aesthetic_teacher/` | Joint teacher optimized for compositional generation and aesthetics on GenEval prompts. |
| `teachers_ckpt/ocr_aesthetic_teacher/` | Joint teacher optimized for text rendering and aesthetics on OCR prompts. |
All teachers are trained with Flow-GRPO-Fast and then frozen; they supervise the student only at student-visited states along its own rollout trajectories.
## Base Model
These checkpoints are LoRA adapters for:
`black-forest-labs/FLUX.1-dev`
## Usage
Capability control uses the official CapField-OPD codebase, which loads the
student LoRA together with `student_ckpt/task_conditioner.pt` (the
capability-coordinate conditioner):
```bash
git clone GITHUB_REPO_URL
cd CapField-OPD
# environment (conda env: capfieldopd) - see the repo README "Installation"
conda create -n capfieldopd python=3.10 -y && conda activate capfieldopd
pip install torch==2.7.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt
```
Pass a prompt and one or more capability coordinates; every coordinate yields one
image, and all images share the same initial noise so only the capability
changes. The coordinate is 3-axis, `lambda = (geneval, ocr, aesthetics)`:
```bash
export PROMPT='A photo of a red apple and a blue cup on a table'
python inference.py \
--ckpt Yang18/CapField-OPD --ckpt_subfolder student_ckpt \
--prompt "${PROMPT}" \
--coords "0.0 0.0 0.0" "1.0 0.0 0.0" "1.0 0.0 1.0"
```
The three coordinates correspond to:
- `(0,0,0)` — base model (capabilities off);
- `(1,0,0)` — GenEval teacher (strong compositional fidelity);
- `(1,0,1)` — joint GenEval + aesthetics teacher.
Intermediate values give continuous control, and values beyond `1` extrapolate
the learned response (e.g. `(1.2, 0, 0)`). The remaining axes work the same way:
`(0,1,0)` / `(0,0,1)` are the OCR and aesthetics anchors, and `(0,1,1)` is their
joint anchor.
Images are written under `outputs/inference/`. For reward-based scoring (e.g.
`geneval`, `ocr`, `pickscore`, HPSv3) start the optional service in
`reward_server.py` in a separate environment.
## Intended Use
This project is released for academic research only. It is intended for studying flow model alignment, reward-guided optimization, on-policy distillation, explicit capability control, and evaluation of text-to-image generation systems.
Users are responsible for ensuring that their use of these checkpoints complies with all applicable laws, research ethics requirements, the Creative Commons Attribution license for this release, and the license terms of the underlying FLUX.1-dev model.
## License
The CapField-OPD released checkpoints and model card are provided under the **Creative Commons Attribution 4.0 International (CC BY 4.0)** license.
Because these checkpoints are based on `black-forest-labs/FLUX.1-dev`, use of the base model is also governed by the FLUX.1 [dev] Non-Commercial License:
https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
## Citation
If you find this project useful, please cite:
```bibtex
@article{ling2026capfield,
title={CapField-OPD: Learning Continuous Capability Fields via Joint-Anchored Multi-Teacher On-Policy Distillation for Flow Models},
author={Ling, Pengyang and Zhou, Yujie and Bu, Jiazi and Wang, Yibin and Ma, Xiaoxiao and Jin, Yi and Chen, Huaian and Zang, Yuhang},
journal={arXiv preprint arXiv:2609.34658},
year={2026}
}
```
## Acknowledgements
We thank the [Flow-GRPO](https://github.com/yifan123/flow_grpo), [DiffusionOPD](https://github.com/ali-vilab/DiffusionOPD), and [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) projects.
|