Instructions to use Wenwu2000/ReSeed with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use Wenwu2000/ReSeed with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("Wenwu2000/ReSeed", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
Simplify README and clarify DMD checkpoint sources
Browse files
README.md
CHANGED
|
@@ -16,170 +16,89 @@ base_model:
|
|
| 16 |
- Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
| 17 |
---
|
| 18 |
|
| 19 |
-
# ReSeed
|
| 20 |
|
| 21 |
**A First-Step Bias Restores Lost Diversity in Diffusion Model Distillation**
|
| 22 |
|
| 23 |
[Paper](https://arxiv.org/abs/2610.37110) · [Project page](https://twwinde.github.io/reseed-project-page/) · [Code](https://github.com/TWWinde/ReSeed)
|
| 24 |
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
retained as a release snapshot.
|
| 30 |
-
|
| 31 |
-
The SD3-Medium and SD3.5-Medium students were trained by the ReSeed authors.
|
| 32 |
-
The FLUX and Wan students come from public releases by their respective
|
| 33 |
-
authors. Public teacher models and shared components are referenced below,
|
| 34 |
-
not copied into this repository.
|
| 35 |
-
|
| 36 |
-
## Contents
|
| 37 |
-
|
| 38 |
-
- [What ReSeed changes](#what-reseed-changes)
|
| 39 |
-
- [Released checkpoints and model sources](#released-checkpoints-and-model-sources)
|
| 40 |
-
- [Repository layout](#repository-layout)
|
| 41 |
-
- [Environment and authentication](#environment-and-authentication)
|
| 42 |
-
- [Download the release](#download-the-release)
|
| 43 |
-
- [SD3-Medium and SD3.5-Medium inference](#sd3-medium-and-sd35-medium-inference)
|
| 44 |
-
- [SenseFlow-FLUX inference](#senseflow-flux-inference)
|
| 45 |
-
- [Wan2.1 text-to-video inference](#wan21-text-to-video-inference)
|
| 46 |
-
- [Reproducibility and file verification](#reproducibility-and-file-verification)
|
| 47 |
-
- [Troubleshooting](#troubleshooting)
|
| 48 |
-
- [Limitations and licensing](#limitations-and-licensing)
|
| 49 |
-
- [Citation](#citation)
|
| 50 |
-
|
| 51 |
-
## What ReSeed changes
|
| 52 |
-
|
| 53 |
-
ReSeed trains small, per-channel residual biases inside a frozen distilled
|
| 54 |
-
student. At inference, the biases are active **only during the first of four
|
| 55 |
-
denoising steps**. The original student weights remain unchanged, and the
|
| 56 |
-
final three steps run without the biases. No teacher forward pass is needed
|
| 57 |
-
when generating with the trained adapter.
|
| 58 |
-
|
| 59 |
-
During adapter training, the student's first-step velocity is matched to
|
| 60 |
-
the teacher's average velocity over the same time interval. This is a
|
| 61 |
-
post-distillation correction; it does not repeat student distillation.
|
| 62 |
-
|
| 63 |
-
The adapter is **not** a LoRA and is **not** a complete Diffusers pipeline.
|
| 64 |
-
It must be loaded using the ReSeed adapter code and applied to the exact
|
| 65 |
-
student it was fitted on. Do not load it with `load_lora_weights`, apply it
|
| 66 |
-
to the teacher, or leave it active for all four steps.
|
| 67 |
-
|
| 68 |
-
## Released checkpoints and model sources
|
| 69 |
-
|
| 70 |
-
### Weights included in this branch
|
| 71 |
-
|
| 72 |
-
| Model | Adapter training seed | Adapter update | Bias parameters | Matching student |
|
| 73 |
-
| --- | ---: | ---: | ---: | --- |
|
| 74 |
-
| SD3-Medium | 42 | **1900** | 72,192 | Our DMD, training seed 42, update 2000; included |
|
| 75 |
-
| SD3.5-Medium | 42 | **2100** | 72,192 | Our DMD, training seed 42, update 2000; included |
|
| 76 |
-
| SenseFlow-FLUX | 42 | **2000** | 233,472 | Public SenseFlow-FLUX; download separately |
|
| 77 |
-
| Wan2.1-T2V-1.3B | 42 | **300** | 46,080 | Public Wan DMD2, iteration 1000; download separately |
|
| 78 |
-
|
| 79 |
-
Adapter updates and student updates are different counters. For example,
|
| 80 |
-
the SD3.5 adapter at update 2100 must be used with the included student at
|
| 81 |
-
update 2000, not with an unrelated student checkpoint named `2100`.
|
| 82 |
-
|
| 83 |
-
Three different seeds may appear in a workflow:
|
| 84 |
-
|
| 85 |
-
1. **Student training seed:** selects our DMD student run; it is 42 here.
|
| 86 |
-
2. **Adapter training seed:** selects the ReSeed fitting run; it is 42 here.
|
| 87 |
-
3. **Generation seed:** selects the initial noise for an image or video.
|
| 88 |
-
This can be any supported seed, such as 0 or 1; it need not be 42.
|
| 89 |
-
|
| 90 |
-
These are individual checkpoints, not averaged weights. One selected run
|
| 91 |
-
does not by itself reproduce a three-training-seed aggregate in the paper.
|
| 92 |
-
GAD and DP-DMD student weights are not included in this release.
|
| 93 |
-
|
| 94 |
-
### Public teachers and external students
|
| 95 |
-
|
| 96 |
-
| Component | Original source | Revision or checkpoint |
|
| 97 |
-
| --- | --- | --- |
|
| 98 |
-
| SD3-Medium teacher and shared components | [stabilityai/stable-diffusion-3-medium-diffusers](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) | `ea42f8cef0f178587cf766dc8129abd379c90671` |
|
| 99 |
-
| SD3.5-Medium teacher and shared components | [stabilityai/stable-diffusion-3.5-medium](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium) | `b940f670f0eda2d07fbb75229e779da1ad11eb80` |
|
| 100 |
-
| FLUX teacher and shared components | [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) | `3de623fc3c33e44ffbe2bad470d0f45bccf2eb21` |
|
| 101 |
-
| SenseFlow-FLUX student transformer | [domiso/SenseFlow](https://huggingface.co/domiso/SenseFlow) | Revision `1bbd81546d590acf24470b39ab28f9cd918c3927`, subfolder **`SenseFlow-FLUX`** |
|
| 102 |
-
| Wan teacher and shared components | [Wan-AI/Wan2.1-T2V-1.3B-Diffusers](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers) | Original public teacher; see the video runtime setup |
|
| 103 |
-
| Wan DMD2 student | [csusupergear/wan_t2v_checkpoints_release](https://huggingface.co/csusupergear/wan_t2v_checkpoints_release) | Prefix **`dmd2_t2v_vipe/0001000`**; retain its `.pth` and `.net_model.zip` files |
|
| 104 |
-
|
| 105 |
-
The SenseFlow transformer is directly in `SenseFlow-FLUX`, not
|
| 106 |
-
`SenseFlow-FLUX/transformer`. The Wan checkpoint uses the DFD/fastgen format;
|
| 107 |
-
it is not a standalone Diffusers `transformer/` folder.
|
| 108 |
-
|
| 109 |
-
The original SD3-M manifest retains the historical internal objective label
|
| 110 |
-
`tdm`; its recorded implementation is `sd_distill_classic_v1`. Original
|
| 111 |
-
manifests are preserved without silently changing their labels or paths.
|
| 112 |
-
|
| 113 |
-
## Repository layout
|
| 114 |
-
|
| 115 |
-
```text
|
| 116 |
-
Wenwu2000/ReSeed (main)/
|
| 117 |
-
├── README.md
|
| 118 |
-
├── requirements.txt
|
| 119 |
-
├── inference.py # Single-prompt SD3-M / SD3.5-M example
|
| 120 |
-
├── release_manifest.json # Pairings, sources, sizes and SHA-256 hashes
|
| 121 |
-
├── adapters/
|
| 122 |
-
│ ├── sd3m/seed_42/checkpoint-1900/
|
| 123 |
-
│ ├── sd35m/seed_42/checkpoint-2100/
|
| 124 |
-
│ ├── flux/seed_42/checkpoint-2000/
|
| 125 |
-
│ └── wan21/seed_42/checkpoint-300/
|
| 126 |
-
│ # Each adapter directory contains residual_biases.pt and manifest.json
|
| 127 |
-
└── students/
|
| 128 |
-
├── sd3m/seed_42/checkpoint-2000/
|
| 129 |
-
└── sd35m/seed_42/checkpoint-2000/
|
| 130 |
-
├── manifest.json
|
| 131 |
-
└── student/
|
| 132 |
-
├── config.json
|
| 133 |
-
├── diffusion_pytorch_model-00001-of-00002.safetensors
|
| 134 |
-
├── diffusion_pytorch_model-00002-of-00002.safetensors
|
| 135 |
-
└── diffusion_pytorch_model.safetensors.index.json
|
| 136 |
-
```
|
| 137 |
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
|
| 142 |
-
|
| 143 |
-
weights, text encoders, VAEs, datasets, prompt caches and generated media.
|
| 144 |
-
It is an inference release, not a resumable training checkpoint archive.
|
| 145 |
-
Absolute paths in original manifests are provenance records from our
|
| 146 |
-
training system; users do not need to recreate those server paths.
|
| 147 |
|
| 148 |
-
##
|
| 149 |
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
example generator does not automatically quantize or offload them.
|
| 154 |
|
| 155 |
-
|
|
|
|
|
|
|
|
|
|
| 156 |
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
| Transformers | 4.45.2 |
|
| 162 |
-
| Hugging Face Hub | 0.36.2 |
|
| 163 |
|
| 164 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 165 |
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 171 |
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
|
| 177 |
-
## Download
|
| 178 |
|
| 179 |
-
|
| 180 |
-
|
| 181 |
|
| 182 |
```bash
|
|
|
|
| 183 |
hf download Wenwu2000/ReSeed --revision main --local-dir reseed-weights
|
| 184 |
|
| 185 |
git clone https://github.com/TWWinde/ReSeed.git
|
|
@@ -188,10 +107,8 @@ pip install -r reseed-weights/requirements.txt
|
|
| 188 |
pip install -e ReSeed/image --no-deps
|
| 189 |
```
|
| 190 |
|
| 191 |
-
The
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
For SD3-Medium only, avoid downloading the other student:
|
| 195 |
|
| 196 |
```bash
|
| 197 |
hf download Wenwu2000/ReSeed --revision main --local-dir reseed-weights \
|
|
@@ -199,26 +116,13 @@ hf download Wenwu2000/ReSeed --revision main --local-dir reseed-weights \
|
|
| 199 |
"adapters/sd3m/*" "students/sd3m/*"
|
| 200 |
```
|
| 201 |
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
their
|
| 205 |
-
|
| 206 |
-
Python can download the same branch without cloning a large Git repository:
|
| 207 |
-
|
| 208 |
-
```python
|
| 209 |
-
from huggingface_hub import snapshot_download
|
| 210 |
-
|
| 211 |
-
weights = snapshot_download(
|
| 212 |
-
repo_id="Wenwu2000/ReSeed",
|
| 213 |
-
revision="main",
|
| 214 |
-
local_dir="reseed-weights",
|
| 215 |
-
)
|
| 216 |
-
print(weights)
|
| 217 |
-
```
|
| 218 |
|
| 219 |
-
## SD3
|
| 220 |
|
| 221 |
-
|
| 222 |
|
| 223 |
```bash
|
| 224 |
hf download stabilityai/stable-diffusion-3-medium-diffusers \
|
|
@@ -228,93 +132,38 @@ hf download stabilityai/stable-diffusion-3.5-medium \
|
|
| 228 |
--revision b940f670f0eda2d07fbb75229e779da1ad11eb80 --local-dir models/sd35m
|
| 229 |
```
|
| 230 |
|
| 231 |
-
|
| 232 |
-
teacher transformer. It reuses the public text encoders, tokenizers, VAE
|
| 233 |
-
and scheduler. It does not run the teacher during student inference.
|
| 234 |
-
|
| 235 |
-
### Generate with ReSeed
|
| 236 |
|
| 237 |
```bash
|
| 238 |
python reseed-weights/inference.py --weights reseed-weights --model sd3m \
|
| 239 |
--base models/sd3m --prompt "a photo of a cat" --seed 0 \
|
| 240 |
-
--output samples/
|
| 241 |
|
| 242 |
python reseed-weights/inference.py --weights reseed-weights --model sd35m \
|
| 243 |
--base models/sd35m --prompt "a photo of a cat" --seed 0 \
|
| 244 |
-
--output samples/
|
| 245 |
```
|
| 246 |
|
| 247 |
-
`--
|
| 248 |
-
|
| 249 |
-
student and adapter from `release_manifest.json` automatically.
|
| 250 |
|
| 251 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 252 |
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
```bash
|
| 256 |
-
python reseed-weights/inference.py --weights reseed-weights --model sd3m \
|
| 257 |
-
--base models/sd3m --prompt "a photo of a cat" --seed 0 \
|
| 258 |
-
--without-reseed --output samples/sd3m_dmd_seed0.png
|
| 259 |
-
|
| 260 |
-
python reseed-weights/inference.py --weights reseed-weights --model sd35m \
|
| 261 |
-
--base models/sd35m --prompt "a photo of a cat" --seed 0 \
|
| 262 |
-
--without-reseed --output samples/sd35m_dmd_seed0.png
|
| 263 |
-
```
|
| 264 |
-
|
| 265 |
-
The baseline keeps the same DMD weights and initial noise; only the adapter
|
| 266 |
-
is omitted. It is not a comparison against the public many-step teacher.
|
| 267 |
-
The script refuses to overwrite an existing image, so use separate output names.
|
| 268 |
-
|
| 269 |
-
### Fixed sampler and adapter behavior
|
| 270 |
-
|
| 271 |
-
| Setting | Both SD3 student examples |
|
| 272 |
-
| --- | --- |
|
| 273 |
-
| Resolution | 1024×1024 |
|
| 274 |
-
| Solver | Native FlowMatch Euler |
|
| 275 |
-
| Number of steps | 4 |
|
| 276 |
-
| Student CFG | **1** |
|
| 277 |
-
| Scheduler | Static shift 3, dynamic shifting disabled |
|
| 278 |
-
| Explicit unshifted sigmas | `[1.0, 0.75, 0.5, 0.25]` |
|
| 279 |
-
| Text maximum sequence length | 256 |
|
| 280 |
-
| Inference dtype | BF16 |
|
| 281 |
-
| Initial noise | CPU FP32 Gaussian draw, then BF16 cast |
|
| 282 |
-
| ReSeed gate | Active for step 1 only; disabled for steps 2–4 |
|
| 283 |
-
|
| 284 |
-
The `sigmas` argument above is the **unshifted** schedule supplied to the
|
| 285 |
-
pipeline; the scheduler applies its shift. Do not replace it with the
|
| 286 |
-
already shifted values from an adapter manifest. Those manifests may also
|
| 287 |
-
contain BF16-rounded training boundaries.
|
| 288 |
-
|
| 289 |
-
The teacher references use **28 steps / CFG 7** for SD3-Medium and **40
|
| 290 |
-
steps / CFG 4.5** for SD3.5-Medium. Those guidance settings must not replace
|
| 291 |
-
the student's CFG 1. The adapter manifests record the exact first interval
|
| 292 |
-
and teacher microstep settings used for fitting.
|
| 293 |
-
|
| 294 |
-
For multiple samples, change `--seed` and the output filename. The example
|
| 295 |
-
handles one prompt/sample per invocation and resets its gate for each run.
|
| 296 |
-
The commands are process-level examples, not single-GPU Slurm submissions.
|
| 297 |
-
Our MUSICA jobs request a whole four-H100 node and parallelize independent
|
| 298 |
-
samples across its four GPUs.
|
| 299 |
-
|
| 300 |
-
## SenseFlow-FLUX inference
|
| 301 |
-
|
| 302 |
-
### Obtain FLUX components and the released student
|
| 303 |
|
| 304 |
```bash
|
| 305 |
hf download black-forest-labs/FLUX.1-dev \
|
| 306 |
--revision 3de623fc3c33e44ffbe2bad470d0f45bccf2eb21 --local-dir models/flux-dev
|
| 307 |
-
|
| 308 |
hf download domiso/SenseFlow --revision 1bbd81546d590acf24470b39ab28f9cd918c3927 \
|
| 309 |
--include "SenseFlow-FLUX/*" --local-dir models/senseflow
|
| 310 |
```
|
| 311 |
|
| 312 |
-
|
| 313 |
-
adapter. The public FLUX snapshot provides shared components and the scheduler.
|
| 314 |
-
|
| 315 |
-
### Prepare a prompt CSV
|
| 316 |
-
|
| 317 |
-
Save the following as `prompts.csv` in the working directory:
|
| 318 |
|
| 319 |
```csv
|
| 320 |
case_number,evaluation_seed,prompt,prompt_id,sample_id
|
|
@@ -322,11 +171,7 @@ case_number,evaluation_seed,prompt,prompt_id,sample_id
|
|
| 322 |
1,1,a photo of a cat,0,1
|
| 323 |
```
|
| 324 |
|
| 325 |
-
|
| 326 |
-
seed. Rows for the same prompt share `prompt_id`; `sample_id` identifies
|
| 327 |
-
the sample within that prompt. Quote the prompt field if it contains commas.
|
| 328 |
-
|
| 329 |
-
### Generate SenseFlow + ReSeed
|
| 330 |
|
| 331 |
```bash
|
| 332 |
cd ReSeed/image
|
|
@@ -335,165 +180,57 @@ python tools/flux_generate.py --model senseflow \
|
|
| 335 |
--senseflow-transformer ../../models/senseflow/SenseFlow-FLUX \
|
| 336 |
--first-interval-residual-bias-checkpoint ../../reseed-weights/adapters/flux/seed_42/checkpoint-2000 \
|
| 337 |
--prompts ../../prompts.csv --expected-count 2 --samples-per-prompt 2 \
|
| 338 |
-
--evaluation-protocol reseed-
|
| 339 |
--steps 4 --resolution 1024 --max-sequence-length 512 \
|
| 340 |
--output ../../samples/flux_reseed
|
| 341 |
cd ../..
|
| 342 |
```
|
| 343 |
|
| 344 |
-
Omit
|
| 345 |
-
|
| 346 |
-
|
| 347 |
-
|
| 348 |
|
| 349 |
-
|
| 350 |
-
`[1, .904531, .759511, .512844, 0]`, with BF16 inference and text length 512.
|
| 351 |
-
Its guidance embedding is **disabled**: do not pass `--guidance 3.5` for
|
| 352 |
-
SenseFlow. FLUX.1-dev guidance 3.5 belongs to the teacher target, not this
|
| 353 |
-
student. The generator checks the schedule against Diffusers and activates
|
| 354 |
-
the bias only during the first step.
|
| 355 |
|
| 356 |
-
|
| 357 |
-
|
| 358 |
-
|
| 359 |
-
|
| 360 |
-
The Wan adapter uses the DMD2 student in the
|
| 361 |
-
[DFD checkpoint release](https://huggingface.co/csusupergear/wan_t2v_checkpoints_release)
|
| 362 |
-
and runs through [DFD / fastgen](https://github.com/csy2077/data-forcing-distillation).
|
| 363 |
-
Follow the [ReSeed video installation guide](https://github.com/TWWinde/ReSeed/blob/eed9a03bd85a0af8a7dc36e8da5b48d2a31f58ca/video/README.md)
|
| 364 |
-
to install the ReSeed module and apply its inference patch to a configured
|
| 365 |
-
DFD checkout. The image package alone is not sufficient for video generation.
|
| 366 |
-
Use DFD's environment and public Wan component setup; do not assume that
|
| 367 |
-
the image requirements file provides the complete video environment.
|
| 368 |
-
|
| 369 |
-
### Download the public DMD2 checkpoint
|
| 370 |
|
| 371 |
```bash
|
| 372 |
hf download csusupergear/wan_t2v_checkpoints_release \
|
| 373 |
--include "dmd2_t2v_vipe/0001000*" --local-dir wan-checkpoints
|
| 374 |
```
|
| 375 |
|
| 376 |
-
Keep the
|
| 377 |
-
|
| 378 |
-
file or a Diffusers pipeline directory.
|
| 379 |
-
|
| 380 |
-
### Generate matched DMD2 and DMD2 + ReSeed videos
|
| 381 |
-
|
| 382 |
-
Create a UTF-8 text file with one prompt per line, then run the following
|
| 383 |
-
from the configured, patched DFD checkout:
|
| 384 |
|
| 385 |
```bash
|
| 386 |
export DFD_CKPT_ROOT=/absolute/path/to/wan-checkpoints
|
| 387 |
export DFD_RESEED_ADAPTER=/absolute/path/to/reseed-weights/adapters/wan21/seed_42/checkpoint-300
|
| 388 |
export DFD_PROMPT_FILE=/absolute/path/to/video-prompts.txt
|
| 389 |
-
|
| 390 |
python evaluation/generate_multiseed.py --config evaluation/config.yaml \
|
| 391 |
--models dmd2 dmd2_reseed --seeds 0 1 \
|
| 392 |
--output-root evaluation/reseed-example
|
| 393 |
```
|
| 394 |
|
| 395 |
-
|
| 396 |
-
|
| 397 |
-
|
| 398 |
-
|
| 399 |
-
|
| 400 |
-
|
| 401 |
-
| Student solver | Native four-step ODE sampler |
|
| 402 |
-
| Student time boundaries | `[.999, .937, .833, .624, 0]` |
|
| 403 |
-
| Student CFG | None; no student CFG branch is evaluated |
|
| 404 |
-
| Video size | 832×480 |
|
| 405 |
-
| Frames / FPS | 81 / 16 |
|
| 406 |
-
| Inference dtype | BF16 |
|
| 407 |
-
| ReSeed gate | First step only |
|
| 408 |
-
| Teacher reference | 50 steps, CFG 5, shift 5 |
|
| 409 |
-
|
| 410 |
-
Do not substitute the public DFD post-trained checkpoint for DMD2: the
|
| 411 |
-
included ReSeed adapter was fitted on **DMD2 iteration 1000**. The DFD
|
| 412 |
-
post-trained model is a different comparison method, not this adapter's base.
|
| 413 |
-
|
| 414 |
-
## Reproducibility and file verification
|
| 415 |
-
|
| 416 |
-
`release_manifest.json` records each model's teacher source, student origin,
|
| 417 |
-
selected adapter update, matching student path, training seed provenance,
|
| 418 |
-
original checkpoint location and per-file SHA-256 hashes. Original
|
| 419 |
-
checkpoint manifests remain beside their weights.
|
| 420 |
-
The release manifest retains `revision: single-seed`, identifying the
|
| 421 |
-
original release branch. The same checkpoint files are also available on
|
| 422 |
-
`main`; their hashes and model pairings are unchanged.
|
| 423 |
-
|
| 424 |
-
After a **full** download, verify all inference files listed in the manifest:
|
| 425 |
-
|
| 426 |
-
```python
|
| 427 |
-
import hashlib
|
| 428 |
-
import json
|
| 429 |
-
from pathlib import Path
|
| 430 |
-
|
| 431 |
-
root = Path("reseed-weights")
|
| 432 |
-
manifest = json.loads((root / "release_manifest.json").read_text())
|
| 433 |
-
for entry in manifest["files"]:
|
| 434 |
-
path = root / entry["path"]
|
| 435 |
-
assert path.is_file(), f"Missing: {path}"
|
| 436 |
-
assert path.stat().st_size == entry["bytes"], f"Wrong size: {path}"
|
| 437 |
-
digest = hashlib.sha256()
|
| 438 |
-
with path.open("rb") as handle:
|
| 439 |
-
for block in iter(lambda: handle.read(8 * 1024 * 1024), b""):
|
| 440 |
-
digest.update(block)
|
| 441 |
-
assert digest.hexdigest() == entry["sha256"], f"Hash mismatch: {path}"
|
| 442 |
-
print("All listed inference files verified.")
|
| 443 |
-
```
|
| 444 |
|
| 445 |
-
|
| 446 |
-
|
| 447 |
-
|
| 448 |
-
|
| 449 |
-
|
| 450 |
-
|
| 451 |
-
|
| 452 |
-
|
| 453 |
-
The
|
| 454 |
-
|
| 455 |
-
an explicitly separate experiment.
|
| 456 |
-
|
| 457 |
-
Exact pixel equality across hardware, attention kernels or library versions
|
| 458 |
-
is not guaranteed. BF16/FP32 changes can also change generated samples.
|
| 459 |
-
Use the reference environment and pinned public snapshots when checking a
|
| 460 |
-
reported comparison. The example scripts are inference entry points; they
|
| 461 |
-
do not by themselves run COCO quality scoring, diversity evaluation or VBench.
|
| 462 |
-
|
| 463 |
-
## Troubleshooting
|
| 464 |
-
|
| 465 |
-
| Symptom | Check |
|
| 466 |
-
| --- | --- |
|
| 467 |
-
| `401`, `403` or a private repository appears missing | Log in with an authorized account; check private-repo access and upstream gated-model approval separately. |
|
| 468 |
-
| Missing weights in an old local download | Refresh the snapshot with `--revision main`; check any selective-download patterns. |
|
| 469 |
-
| `No module named reseed` | Install `ReSeed/image` in the same Python environment used for inference. |
|
| 470 |
-
| A LoRA loader rejects `residual_biases.pt` | Use the ReSeed residual-bias loader, not PEFT or `load_lora_weights`. |
|
| 471 |
-
| Missing shard or incomplete student | Download both safetensor shards, their index and `config.json`; do not copy a single shard alone. |
|
| 472 |
-
| SD3 sampler contract error | Use the matching pinned base, static shift 3, the supplied unshifted sigmas and student CFG 1. |
|
| 473 |
-
| Unexpected FLUX guidance error | Use `--model senseflow`, the `SenseFlow-FLUX` folder, and no guidance embedding. |
|
| 474 |
-
| Wan adapter is ignored | Install the ReSeed inference patch and use the `dmd2_reseed` arm. |
|
| 475 |
-
| Wan checkpoint loader fails | Preserve the public `.pth` / `.net_model.zip` pair and pass the correct checkpoint prefix. |
|
| 476 |
-
| CUDA out of memory | Check which model is loaded and available GPU memory. The example does not silently change resolution, precision, quantization or offload settings. |
|
| 477 |
-
| Output image already exists | Select a new output filename; the SD3 example intentionally refuses overwrites. |
|
| 478 |
-
|
| 479 |
-
## Limitations and licensing
|
| 480 |
-
|
| 481 |
-
These adapters are tied to their listed students and native sampling
|
| 482 |
-
settings. Applying one to another backbone, another distillation run, a
|
| 483 |
-
different step count or a different resolution is outside this release's
|
| 484 |
-
documented setup. A diversity gain on an evaluation bank does not guarantee
|
| 485 |
-
that every prompt, quality metric or generated sample improves.
|
| 486 |
-
|
| 487 |
-
This branch provides one selected run per model, not training data, complete
|
| 488 |
-
training state or all experimental variants. It does not include weights
|
| 489 |
-
for GAD, DP-DMD, SDXL or MiniMax experiments.
|
| 490 |
-
|
| 491 |
-
The ReSeed implementation is MIT-licensed in the code repository. That does
|
| 492 |
-
**not** replace the licenses of model weights or dependencies. Consult each
|
| 493 |
-
linked upstream model's terms and the code repository's
|
| 494 |
-
[NOTICE](https://github.com/TWWinde/ReSeed/blob/eed9a03bd85a0af8a7dc36e8da5b48d2a31f58ca/NOTICE.md).
|
| 495 |
-
Keeping this repository private does not grant additional redistribution or
|
| 496 |
-
commercial-use rights. This documentation update does not make the repository public.
|
| 497 |
|
| 498 |
## Citation
|
| 499 |
|
|
@@ -508,6 +245,3 @@ commercial-use rights. This documentation update does not make the repository pu
|
|
| 508 |
url={https://arxiv.org/abs/2610.37110}
|
| 509 |
}
|
| 510 |
```
|
| 511 |
-
|
| 512 |
-
Please also acknowledge the teacher models, external distilled students
|
| 513 |
-
and DFD / fastgen software when using them in a new study.
|
|
|
|
| 16 |
- Wan-AI/Wan2.1-T2V-1.3B-Diffusers
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# ReSeed
|
| 20 |
|
| 21 |
**A First-Step Bias Restores Lost Diversity in Diffusion Model Distillation**
|
| 22 |
|
| 23 |
[Paper](https://arxiv.org/abs/2610.37110) · [Project page](https://twwinde.github.io/reseed-project-page/) · [Code](https://github.com/TWWinde/ReSeed)
|
| 24 |
|
| 25 |
+
ReSeed improves diversity in a frozen four-step student by fitting small
|
| 26 |
+
residual biases to the teacher's first-interval velocity. The biases are
|
| 27 |
+
enabled only in the first denoising step; the original weights and final
|
| 28 |
+
three steps remain unchanged.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
+
This repository provides four ReSeed adapters and two self-trained SD3 DMD
|
| 31 |
+
students. The selected adapter runs and our two student runs use training
|
| 32 |
+
seed 42. Generation seeds can be chosen independently.
|
| 33 |
|
| 34 |
+
## Student models and their sources
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
+
### SD3-Medium and SD3.5-Medium: trained by us
|
| 37 |
|
| 38 |
+
We trained these four-step students in our CRRA `unified_distill` codebase
|
| 39 |
+
from the corresponding Stability AI models. Both released students are
|
| 40 |
+
**seed 42, checkpoint-2000**. Their full transformer weights are included:
|
|
|
|
| 41 |
|
| 42 |
+
| Student | Released weights | Teacher | Teacher sampling reference |
|
| 43 |
+
| --- | --- | --- | --- |
|
| 44 |
+
| SD3-Medium DMD | [checkpoint-2000](https://huggingface.co/Wenwu2000/ReSeed/tree/main/students/sd3m/seed_42/checkpoint-2000) | [SD3-Medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers) | 28 steps, CFG 7 |
|
| 45 |
+
| SD3.5-Medium DMD | [checkpoint-2000](https://huggingface.co/Wenwu2000/ReSeed/tree/main/students/sd35m/seed_42/checkpoint-2000) | [SD3.5-Medium](https://huggingface.co/stabilityai/stable-diffusion-3.5-medium) | 40 steps, CFG 4.5 |
|
| 46 |
|
| 47 |
+
These are our experimental students, **not checkpoints from the official
|
| 48 |
+
DMD2 release**. They use four-step FlowMatch Euler inference at CFG 1.
|
| 49 |
+
For the underlying methods, see [DMD](https://arxiv.org/abs/2311.18828) and
|
| 50 |
+
[DMD2](https://github.com/tianweiy/DMD2).
|
|
|
|
|
|
|
| 51 |
|
| 52 |
+
The original training manifests are included beside each student. The
|
| 53 |
+
SD3-Medium manifest retains the historical objective label `tdm` and records
|
| 54 |
+
the implementation `sd_distill_classic_v1`; the SD3.5-Medium manifest records
|
| 55 |
+
`dmd`. These records have not been relabelled. The release's exact student
|
| 56 |
+
and adapter pairings are listed in [release_manifest.json](release_manifest.json).
|
| 57 |
|
| 58 |
+
### Wan2.1 DMD2: public checkpoint from the DFD authors
|
| 59 |
+
|
| 60 |
+
The video student is the Wan2.1-T2V-1.3B DMD2 checkpoint released by the
|
| 61 |
+
[Data-Forcing Distillation authors](https://github.com/csy2077/data-forcing-distillation).
|
| 62 |
+
We use **`dmd2_t2v_vipe/0001000`**, trained on the ViPE `long` set:
|
| 63 |
+
|
| 64 |
+
- [Checkpoint files](https://huggingface.co/csusupergear/wan_t2v_checkpoints_release/tree/main/dmd2_t2v_vipe)
|
| 65 |
+
- [Teacher: Wan2.1-T2V-1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers)
|
| 66 |
+
- [DFD / fastgen code and setup](https://github.com/csy2077/data-forcing-distillation)
|
| 67 |
+
|
| 68 |
+
We did not train this DMD2 student. We trained the ReSeed adapter on top of
|
| 69 |
+
it and provide only that adapter here. The separately released DFD
|
| 70 |
+
post-trained model (`ours_wan_t2v_vipe/0000040`) is a different baseline,
|
| 71 |
+
not the student used by this adapter.
|
| 72 |
+
|
| 73 |
+
### FLUX: public SenseFlow student
|
| 74 |
+
|
| 75 |
+
We use the four-step **SenseFlow-FLUX** transformer from
|
| 76 |
+
[domiso/SenseFlow](https://huggingface.co/domiso/SenseFlow/tree/main/SenseFlow-FLUX),
|
| 77 |
+
with shared components from [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev).
|
| 78 |
+
This student was released by the SenseFlow authors, not trained by us.
|
| 79 |
+
Download it separately; this repository contains its ReSeed adapter only.
|
| 80 |
+
|
| 81 |
+
## Released adapters
|
| 82 |
+
|
| 83 |
+
| Student | Adapter directory | Adapter updates | Bias parameters |
|
| 84 |
+
| --- | --- | ---: | ---: |
|
| 85 |
+
| SD3-Medium DMD | `adapters/sd3m/seed_42/checkpoint-1900` | 1900 | 72,192 |
|
| 86 |
+
| SD3.5-Medium DMD | `adapters/sd35m/seed_42/checkpoint-2100` | 2100 | 72,192 |
|
| 87 |
+
| SenseFlow-FLUX | `adapters/flux/seed_42/checkpoint-2000` | 2000 | 233,472 |
|
| 88 |
+
| Wan2.1 DMD2 | `adapters/wan21/seed_42/checkpoint-300` | 300 | 46,080 |
|
| 89 |
|
| 90 |
+
Each directory contains `residual_biases.pt` and `manifest.json`. Adapter
|
| 91 |
+
updates count ReSeed fitting, not student distillation: both included SD3
|
| 92 |
+
students remain at update 2000. Use each adapter with its listed student.
|
| 93 |
+
These are residual-bias adapters, not LoRA weights.
|
| 94 |
|
| 95 |
+
## Download and install
|
| 96 |
|
| 97 |
+
Use Python 3.10+ and a CUDA/BF16-capable PyTorch installation. Our image
|
| 98 |
+
runtime used PyTorch 2.6.0, Diffusers 0.34.0 and Transformers 4.45.2.
|
| 99 |
|
| 100 |
```bash
|
| 101 |
+
pip install huggingface_hub==0.36.2
|
| 102 |
hf download Wenwu2000/ReSeed --revision main --local-dir reseed-weights
|
| 103 |
|
| 104 |
git clone https://github.com/TWWinde/ReSeed.git
|
|
|
|
| 107 |
pip install -e ReSeed/image --no-deps
|
| 108 |
```
|
| 109 |
|
| 110 |
+
The full download is about 18.22 GB, mostly the two student transformers.
|
| 111 |
+
To download only SD3-Medium, use:
|
|
|
|
|
|
|
| 112 |
|
| 113 |
```bash
|
| 114 |
hf download Wenwu2000/ReSeed --revision main --local-dir reseed-weights \
|
|
|
|
| 116 |
"adapters/sd3m/*" "students/sd3m/*"
|
| 117 |
```
|
| 118 |
|
| 119 |
+
Use `sd35m` instead of `sd3m` for SD3.5-Medium. For FLUX or Wan, only the
|
| 120 |
+
corresponding adapter is needed from this repository. Upstream gated models
|
| 121 |
+
require accepting their terms and logging in with `hf auth login`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
|
| 123 |
+
## SD3 image generation
|
| 124 |
|
| 125 |
+
Download the public components for the model you want to run:
|
| 126 |
|
| 127 |
```bash
|
| 128 |
hf download stabilityai/stable-diffusion-3-medium-diffusers \
|
|
|
|
| 132 |
--revision b940f670f0eda2d07fbb75229e779da1ad11eb80 --local-dir models/sd35m
|
| 133 |
```
|
| 134 |
|
| 135 |
+
Generate an image with the matching DMD student and ReSeed:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
|
| 137 |
```bash
|
| 138 |
python reseed-weights/inference.py --weights reseed-weights --model sd3m \
|
| 139 |
--base models/sd3m --prompt "a photo of a cat" --seed 0 \
|
| 140 |
+
--output samples/sd3m_reseed.png
|
| 141 |
|
| 142 |
python reseed-weights/inference.py --weights reseed-weights --model sd35m \
|
| 143 |
--base models/sd35m --prompt "a photo of a cat" --seed 0 \
|
| 144 |
+
--output samples/sd35m_reseed.png
|
| 145 |
```
|
| 146 |
|
| 147 |
+
Add `--without-reseed` and choose a different output filename to generate
|
| 148 |
+
the plain DMD baseline with the same prompt and initial noise.
|
|
|
|
| 149 |
|
| 150 |
+
The script loads our student transformer and the public model's text
|
| 151 |
+
encoders, tokenizers, VAE and scheduler. It uses 1024×1024 BF16 inference,
|
| 152 |
+
four Euler steps, CFG 1 and static shift 3. The pipeline receives unshifted
|
| 153 |
+
sigmas `[1, .75, .5, .25]`; do not substitute the shifted boundaries from
|
| 154 |
+
the training manifest. Initial noise is drawn in FP32 on CPU, then cast to
|
| 155 |
+
BF16. The adapter is disabled after the first step.
|
| 156 |
|
| 157 |
+
## FLUX image generation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 158 |
|
| 159 |
```bash
|
| 160 |
hf download black-forest-labs/FLUX.1-dev \
|
| 161 |
--revision 3de623fc3c33e44ffbe2bad470d0f45bccf2eb21 --local-dir models/flux-dev
|
|
|
|
| 162 |
hf download domiso/SenseFlow --revision 1bbd81546d590acf24470b39ab28f9cd918c3927 \
|
| 163 |
--include "SenseFlow-FLUX/*" --local-dir models/senseflow
|
| 164 |
```
|
| 165 |
|
| 166 |
+
Create `prompts.csv`:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 167 |
|
| 168 |
```csv
|
| 169 |
case_number,evaluation_seed,prompt,prompt_id,sample_id
|
|
|
|
| 171 |
1,1,a photo of a cat,0,1
|
| 172 |
```
|
| 173 |
|
| 174 |
+
Then run:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 175 |
|
| 176 |
```bash
|
| 177 |
cd ReSeed/image
|
|
|
|
| 180 |
--senseflow-transformer ../../models/senseflow/SenseFlow-FLUX \
|
| 181 |
--first-interval-residual-bias-checkpoint ../../reseed-weights/adapters/flux/seed_42/checkpoint-2000 \
|
| 182 |
--prompts ../../prompts.csv --expected-count 2 --samples-per-prompt 2 \
|
| 183 |
+
--evaluation-protocol reseed-example \
|
| 184 |
--steps 4 --resolution 1024 --max-sequence-length 512 \
|
| 185 |
--output ../../samples/flux_reseed
|
| 186 |
cd ../..
|
| 187 |
```
|
| 188 |
|
| 189 |
+
Omit the adapter argument for the plain SenseFlow baseline. This student
|
| 190 |
+
uses its native four-step dynamically shifted Euler schedule with the
|
| 191 |
+
guidance embedding disabled. Do not pass the teacher's guidance 3.5 to the
|
| 192 |
+
student. The transformer is in `SenseFlow-FLUX`, not a nested `transformer/` folder.
|
| 193 |
|
| 194 |
+
## Wan video generation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 195 |
|
| 196 |
+
First follow the [ReSeed video setup](https://github.com/TWWinde/ReSeed/blob/eed9a03bd85a0af8a7dc36e8da5b48d2a31f58ca/video/README.md)
|
| 197 |
+
to install the video code and apply the ReSeed inference patch to DFD.
|
| 198 |
+
The video runtime has separate dependencies from the image package.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 199 |
|
| 200 |
```bash
|
| 201 |
hf download csusupergear/wan_t2v_checkpoints_release \
|
| 202 |
--include "dmd2_t2v_vipe/0001000*" --local-dir wan-checkpoints
|
| 203 |
```
|
| 204 |
|
| 205 |
+
Keep the `.pth` and `.net_model.zip` files under their original names.
|
| 206 |
+
From the configured DFD checkout, use a text file with one prompt per line:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 207 |
|
| 208 |
```bash
|
| 209 |
export DFD_CKPT_ROOT=/absolute/path/to/wan-checkpoints
|
| 210 |
export DFD_RESEED_ADAPTER=/absolute/path/to/reseed-weights/adapters/wan21/seed_42/checkpoint-300
|
| 211 |
export DFD_PROMPT_FILE=/absolute/path/to/video-prompts.txt
|
|
|
|
| 212 |
python evaluation/generate_multiseed.py --config evaluation/config.yaml \
|
| 213 |
--models dmd2 dmd2_reseed --seeds 0 1 \
|
| 214 |
--output-root evaluation/reseed-example
|
| 215 |
```
|
| 216 |
|
| 217 |
+
This generates matched DMD2 and DMD2 + ReSeed videos. The student uses four
|
| 218 |
+
ODE steps with boundaries `[.999, .937, .833, .624, 0]`, no student CFG,
|
| 219 |
+
81 frames at 832×480, 16 FPS and BF16. The teacher reference uses 50 UniPC
|
| 220 |
+
steps, CFG 5 and shift 5. ReSeed is active only in the first student step.
|
| 221 |
+
|
| 222 |
+
## Files and licenses
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 223 |
|
| 224 |
+
The release contains inference weights, not optimizer state, fake-score
|
| 225 |
+
models, public teacher weights, datasets or generated samples. Text encoders
|
| 226 |
+
and VAEs come from the linked base models. File hashes, pinned source
|
| 227 |
+
revisions and checkpoint pairings are recorded in `release_manifest.json`;
|
| 228 |
+
its branch and visibility fields describe the original private release.
|
| 229 |
+
The repository is now public, with current instructions on `main` and the
|
| 230 |
+
original `single-seed` branch retained as an archive.
|
| 231 |
+
|
| 232 |
+
The ReSeed code is MIT-licensed. Model weights retain their upstream
|
| 233 |
+
licenses and access conditions; see [NOTICE](https://github.com/TWWinde/ReSeed/blob/main/NOTICE.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 234 |
|
| 235 |
## Citation
|
| 236 |
|
|
|
|
| 245 |
url={https://arxiv.org/abs/2610.37110}
|
| 246 |
}
|
| 247 |
```
|
|
|
|
|
|
|
|
|