datnguyentien204 commited on
Commit
0261f7a
·
verified ·
1 Parent(s): 40440d6

Update RotEx documentation, Apache license, and inference checkpoint paths

Browse files
.gitignore ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Model weights are distributed through Hugging Face.
2
+ *.pt
3
+ *.pth
4
+ *.ckpt
5
+ *.safetensors
6
+
7
+ # Python and download caches.
8
+ __pycache__/
9
+ *.py[cod]
10
+ .cache/
11
+
12
+ # Generated inference outputs.
13
+ outputs/
LICENSE ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright [yyyy] [name of copyright owner]
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
README.md CHANGED
@@ -1,4 +1,4 @@
1
- # [Official] RotEx: A Rotational Expert Network for Realistic Scene Synthesis Under Illumination Variation
2
 
3
  <p align="center">
4
  📄 <b>Paper: coming soon</b>&nbsp;&nbsp; | &nbsp;&nbsp;
@@ -7,84 +7,49 @@
7
  </p>
8
 
9
  <p align="center">
10
- <img src="demo/RotEx_Proposed.png" alt="Overview of RotEx and its double-stream and single-stream transformer blocks" width="1100"/>
11
  </p>
12
- <p align="center">
13
- <em>Architecture of RotEx: content and style streams, depth conditioning, and illumination-dependent expert routing.</em>
14
- </p>
15
-
16
- > The Hugging Face commands below describe the intended weight layout; they become usable once the checkpoints are uploaded.
17
 
18
  ## Introduction
19
 
20
  **RotEx** is a transformer-based image-to-image framework for synthesizing scenes under varying illumination conditions. It combines content latents, depth information, style features, and a continuous time-of-day variable to generate illumination variations of an input scene.
21
 
22
- The model operates in the latent space of a FLUX autoencoder. DINOv2 supplies style features, Depth Anything V2 supplies depth maps, and time-conditioned mixture-of-experts modules adapt the synthesis process to the requested illumination. The release includes both multi-step inference and a distilled one-step inference path.
23
-
24
  ### Key Features
25
 
26
- - **Continuous illumination control:** Specify a time-of-day value and map it to an angle through `phi = hour * 2π / 24`.
27
- - **Illumination-dependent experts:** Route style and image features through experts conditioned on the illumination angle.
28
- - **Content and depth conditioning:** Use an encoded content image and its estimated depth map during synthesis.
29
- - **Two inference modes:** Use a full checkpoint with a multi-step Heun solver, or a distilled checkpoint with a single model evaluation.
30
- - **Image and video processing:** Generate image sequences across hours or vary the target hour across the frames of a video.
31
- - **Reference-style conditioning:** Use a separate style image to condition illumination synthesis.
32
 
33
  ## Demo
34
 
35
  ### Illumination Variation
36
 
37
  <p align="center">
38
- <img src="demo/illumination_variation_space.jpg" alt="Scene synthesis across the illumination variation space" width="1000"/>
39
- </p>
40
- <p align="center">
41
- <em>Illustration of the illumination variation space.</em>
42
  </p>
43
 
44
  ### Downstream Trajectory Comparison
45
 
46
  <p align="center">
47
- <img src="demo/VLA_Traj.png" alt="AutoVLA trajectory comparison under different image translation methods" width="1100"/>
48
- </p>
49
- <p align="center">
50
- <em>Qualitative comparison of AutoVLA rollouts using daylight reference images and images produced by different translation methods.</em>
51
  </p>
52
 
53
- ## Installation and Requirements
54
-
55
- The supplied inference code uses CUDA and imports FlashAttention directly. Use a compatible NVIDIA GPU and a Linux environment with the CUDA toolkit required by your FlashAttention build.
56
-
57
- | Component | Requirement |
58
- | --- | --- |
59
- | Python | 3.10 or later |
60
- | PyTorch | 2.6.0 or later, as specified in `requirements.txt` |
61
- | torchvision | A version compatible with the installed PyTorch release; `requirements.txt` specifies 0.19.0 or later |
62
- | FlashAttention | 2.0.0 or later, with a build compatible with PyTorch, CUDA, and the GPU |
63
- | Default inference precision | `torch.bfloat16` |
64
- | Default image resolution | 384 × 384 |
65
 
66
- ### 1. Clone the Repository
67
 
68
  ```shell
69
  git clone https://github.com/val-utehy/RotEx.git
70
  cd RotEx
71
- ```
72
-
73
- All commands below run from the directory containing `config.py`, `flux/`, and the inference scripts. If using this code inside the BenchRotEx workspace, that directory is `B/RotEx`.
74
-
75
- ### 2. Create an Environment
76
 
77
- ```shell
78
  conda create -n rotex python=3.10 -y
79
  conda activate rotex
80
  python -m pip install --upgrade pip
81
  ```
82
 
83
- ### 3. Install PyTorch and Dependencies
84
-
85
- Install a matching CUDA-enabled PyTorch and torchvision pair using the [official PyTorch installation instructions](https://pytorch.org/get-started/locally/). Ensure the selected versions satisfy the requirements above.
86
-
87
- After installing PyTorch:
88
 
89
  ```shell
90
  python -m pip install packaging ninja psutil
@@ -92,92 +57,44 @@ python -m pip install "flash-attn>=2.0.0" --no-build-isolation
92
  python -m pip install -r requirements.txt
93
  ```
94
 
95
- See the [FlashAttention installation instructions](https://github.com/Dao-AILab/flash-attention#installation-and-features) for supported CUDA/GPU combinations and build requirements.
96
-
97
- Verify the environment:
98
-
99
- ```shell
100
- python -c "import torch, flash_attn; print('PyTorch:', torch.__version__); print('CUDA:', torch.cuda.is_available()); print('BF16:', torch.cuda.is_bf16_supported())"
101
- ```
102
-
103
- ### 4. Prepare the Auxiliary Models
104
-
105
- The inference scripts load these pretrained components:
106
-
107
- | Component | Source | Local behavior |
108
- | --- | --- | --- |
109
- | FLUX autoencoder | [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev), `ae.safetensors` | Loaded from `checkpoints/ae.safetensors`; downloaded there if missing |
110
- | DINOv2 Base | [facebook/dinov2-base](https://huggingface.co/facebook/dinov2-base) | Downloaded through Transformers and stored in the Hugging Face cache |
111
- | Depth Anything V2 Small | [depth-anything/Depth-Anything-V2-Small-hf](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf) | Downloaded through Transformers and stored in the Hugging Face cache |
112
-
113
- An internet connection is needed for the first download. If access to an auxiliary model requires authentication or approval, complete the access process on its model page and log in with `hf auth login`.
114
-
115
- To download the autoencoder explicitly:
116
-
117
- ```shell
118
- hf download black-forest-labs/FLUX.1-dev ae.safetensors --local-dir checkpoints
119
- ```
120
-
121
  ## Pretrained Models
122
 
123
- RotEx weights will be distributed through [Hugging Face](https://huggingface.co/val-utehy/RotEx). Download the checkpoint matching the inference mode you intend to run.
124
 
125
- | Checkpoint | Purpose | Expected local path |
126
  | --- | --- | --- |
127
- | Full model, best checkpoint | Multi-step image and video inference | `checkpoints/full/best.pt` |
128
- | Full model, final checkpoint | Alternative full-model checkpoint | `checkpoints/full/last.pt` |
129
- | Distilled model, best checkpoint | One-step inference | `distill_checkpoints/best_distill.pt` |
130
-
131
- Once the model repository is published with this directory structure, download the required weights from the project root:
132
 
133
  ```shell
134
  hf download val-utehy/RotEx checkpoints/full/best.pt --local-dir .
135
  hf download val-utehy/RotEx distill_checkpoints/best_distill.pt --local-dir .
 
136
  ```
137
 
138
- The final full-model checkpoint is optional:
139
-
140
- ```shell
141
- hf download val-utehy/RotEx checkpoints/full/last.pt --local-dir .
142
- ```
143
-
144
- The `hf` command is provided by `huggingface_hub`. See the [Hugging Face CLI guide](https://huggingface.co/docs/huggingface_hub/guides/cli) for download and authentication options.
145
-
146
- Additional placement instructions are available in [checkpoints/README.md](checkpoints/README.md), [checkpoints/full/README.md](checkpoints/full/README.md), and [distill_checkpoints/README.md](distill_checkpoints/README.md).
147
-
148
- > [!NOTE]
149
- > The scripts still contain paths from the original development environment. Pass explicit paths to the multi-step scripts and update the configuration constants in `infer_1step_style.py`. The model configuration is defined in `config.py`; these scripts do not use an `opt.pkl` file.
150
 
151
- ## Usage Guide
152
 
153
- ### 1. Model Training
154
-
155
- This release contains model components and inference scripts. A standalone training entry point and a distillation training entry point are not included in this directory.
156
-
157
- ### 2. Choosing an Inference Script
 
 
 
 
 
158
 
159
- | Script | Checkpoint | Style input | Output |
160
- | --- | --- | --- | --- |
161
- | `infer_1step_style.py` | Distilled | One fixed reference image | One PNG per image and hour for each configured input/output pair; hours 1–24 by default |
162
- | `infer_multistep.py` | Full | Every image in a style directory | One labeled strip per content/style pair; hours 0–24 by default |
163
- | `infer_multistep_vid.py` | Full | One fixed reference image | One MP4 with the target hour interpolated across frames |
164
 
165
- All image-directory scripts read `.jpg`, `.jpeg`, and `.png` files directly inside the configured folders. They do not search subdirectories recursively.
166
 
167
- ### 3. Multi-Step Image Inference
168
 
169
- Prepare separate content and style directories, for example:
170
-
171
- ```text
172
- inputs/
173
- ├── content/
174
- │ ├── scene_001.png
175
- │ └── scene_002.jpg
176
- └── style/
177
- └── night_reference.jpg
178
- ```
179
-
180
- Run:
181
 
182
  ```shell
183
  python infer_multistep.py \
@@ -185,30 +102,12 @@ python infer_multistep.py \
185
  --content_dir inputs/content \
186
  --style_dir inputs/style \
187
  --output_dir outputs/multistep \
188
- --denoise_steps 50 \
189
- --image_size 384 \
190
- --depth_size 48 \
191
- --num_seeds 1 \
192
- --device cuda \
193
- --dtype bfloat16
194
  ```
195
 
196
- Each content image is combined with every style image. Results are saved as `<content_stem>__sty_<style_stem>.png`. Each result is a horizontal strip containing the content image, style reference, depth visualization, and generated images for hours 0 through 24. Existing output files are skipped.
197
-
198
- | Argument | Default | Description |
199
- | --- | --- | --- |
200
- | `--denoise_steps` | `50` | Number of Heun integration steps; each step evaluates the model twice |
201
- | `--image_size` | `384` | Square image resolution used for content and style preprocessing |
202
- | `--depth_size` | `48` | Depth-map resolution supplied to the model |
203
- | `--num_seeds` | `1` | Number of independently sampled outputs to average per hour, in decoded image space |
204
- | `--device` | `cuda` | Device used for inference |
205
- | `--dtype` | `bfloat16` | Use `bfloat16` for the supplied default GPU path |
206
 
207
- The hour list is set inside `run_inference()` and is not exposed as a command-line argument. The legacy `--noise_alpha` argument is unused; initialization is sampled from Gaussian noise. The image solver clamps latents to `[-20, 20]` during integration and `[-15, 15]` before decoding.
208
-
209
- ### 4. One-Step Image Inference with a Fixed Style
210
-
211
- Open `infer_1step_style.py` and configure the distilled checkpoint, style image, and dataset paths:
212
 
213
  ```python
214
  CHECKPOINT = "distill_checkpoints/best_distill.pt"
@@ -219,15 +118,11 @@ DATA_PAIRS = [
219
  HOURS = list(range(1, 25))
220
  ```
221
 
222
- Run:
223
-
224
  ```shell
225
  python infer_1step_style.py
226
  ```
227
 
228
- The reference style is encoded once and reused for all configured image folders. Outputs follow the `<image_stem>_hXX.png` naming convention. The script writes to the configured output paths and can overwrite existing images with the same names.
229
-
230
- ### 5. Multi-Step Video Inference
231
 
232
  ```shell
233
  python infer_multistep_vid.py \
@@ -236,63 +131,14 @@ python infer_multistep_vid.py \
236
  --style_path inputs/style/night_reference.jpg \
237
  --output_dir outputs/video \
238
  --denoise_steps 50 \
239
- --image_size 384 \
240
- --depth_size 48 \
241
- --max_height 384 \
242
  --hour_start 0 \
243
  --hour_end 24 \
244
- --device cuda \
245
- --dtype bfloat16 \
246
  --no_audio
247
  ```
248
 
249
- The script encodes the style reference once, reuses a base noise tensor across frames, and interpolates the target hour from `--hour_start` to `--hour_end` over the input video. The output is named `<video_stem>_chronflux.mp4` and uses the input video's reported frame rate.
250
-
251
- `--image_size` controls the square resolution used for model inference. `--max_height` controls output resizing, with output dimensions aligned to multiples of 16. The video solver uses latent clamps of `[-8, 8]` during integration and `[-6, 6]` before decoding, which differ from the image solver.
252
-
253
- The example produces a silent video. Audio transfer is optional and requires MoviePy, which is not included in `requirements.txt`; the script falls back to saving the video without audio if transfer fails.
254
-
255
- ### 6. Illumination Control and Reproducibility
256
-
257
- The scripts map hours to the model's illumination angle as follows:
258
-
259
- ```python
260
- phi = hour * (2 * math.pi / 24.0)
261
- ```
262
-
263
- Hours 0 and 24 correspond to the endpoints of one full cycle. The image scripts use discrete hour lists, while the video script supplies intermediate hour values across frames. To render fractional hours in an image script, edit its hour list; the one-step filename formatting must also be adapted if the values are not integers.
264
-
265
- The image scripts sample new Gaussian noise for each output and do not expose a seed argument. Repeated runs can therefore produce different results. Increasing `--num_seeds` in the multi-step image script averages decoded outputs and increases computation; it is separate from increasing the number of solver steps.
266
-
267
- ## Repository Structure
268
-
269
- ```text
270
- RotEx/
271
- ��── README.md
272
- ├── config.py
273
- ├── requirements.txt
274
- ├── infer_1step_style.py
275
- ├── infer_multistep.py
276
- ├── infer_multistep_vid.py
277
- ├── flux/ # Model, autoencoder, encoders, and utilities
278
- ├── demo/ # Architecture and qualitative figures
279
- ├── checkpoints/
280
- │ ├── README.md
281
- │ ├── ae.safetensors # Download separately
282
- │ └── full/
283
- │ ├── README.md
284
- │ ├── best.pt # Download from the RotEx model repository
285
- │ └── last.pt # Optional alternative full checkpoint
286
- └── distill_checkpoints/
287
- ├── README.md
288
- └── best_distill.pt # Download from the RotEx model repository
289
- ```
290
-
291
- The Python implementation retains internal names such as `ChronFlux` and the `q1_paper` configuration preset. These are the identifiers used by the current inference scripts.
292
-
293
  ## Citation
294
 
295
- Paper details and the official BibTeX entry will be added when available.
296
 
297
  ## References
298
 
@@ -303,4 +149,4 @@ Paper details and the official BibTeX entry will be added when available.
303
 
304
  ## License
305
 
306
- A project license has not yet been included in this release. The license will be specified when the repository is published. Consult the respective upstream repositories and model cards for the terms governing pretrained components.
 
1
+ # [Official] RotEx: A Rotational Expert Network for Realistic Scene Illumination Variation Synthesis
2
 
3
  <p align="center">
4
  📄 <b>Paper: coming soon</b>&nbsp;&nbsp; | &nbsp;&nbsp;
 
7
  </p>
8
 
9
  <p align="center">
10
+ <img src="demo/RotEx_Proposed.png" alt="RotEx architecture" width="1100"/>
11
  </p>
 
 
 
 
 
12
 
13
  ## Introduction
14
 
15
  **RotEx** is a transformer-based image-to-image framework for synthesizing scenes under varying illumination conditions. It combines content latents, depth information, style features, and a continuous time-of-day variable to generate illumination variations of an input scene.
16
 
 
 
17
  ### Key Features
18
 
19
+ - **Continuous illumination control:** Generate scene variations across different times of the day.
20
+ - **Illumination-dependent experts:** Adapt style and image features through time-conditioned expert routing.
21
+ - **Content and depth conditioning:** Guide synthesis using content latents and estimated scene depth.
22
+ - **Multi-step and one-step inference:** Support full-model sampling and distilled single-step generation.
23
+ - **Image and video processing:** Process image directories and videos with reference-style conditioning.
 
24
 
25
  ## Demo
26
 
27
  ### Illumination Variation
28
 
29
  <p align="center">
30
+ <img src="demo/illumination_variation_space.jpg" alt="Illumination variation" width="1000"/>
 
 
 
31
  </p>
32
 
33
  ### Downstream Trajectory Comparison
34
 
35
  <p align="center">
36
+ <img src="demo/VLA_Traj.png" alt="AutoVLA trajectory comparison" width="1100"/>
 
 
 
37
  </p>
38
 
39
+ ## Installation
 
 
 
 
 
 
 
 
 
 
 
40
 
41
+ **Requirements:** Python ≥ 3.10, PyTorch ≥ 2.6, a compatible torchvision version, FlashAttention ≥ 2.0, and an NVIDIA GPU with CUDA.
42
 
43
  ```shell
44
  git clone https://github.com/val-utehy/RotEx.git
45
  cd RotEx
 
 
 
 
 
46
 
 
47
  conda create -n rotex python=3.10 -y
48
  conda activate rotex
49
  python -m pip install --upgrade pip
50
  ```
51
 
52
+ Install CUDA-enabled PyTorch and torchvision following the [PyTorch instructions](https://pytorch.org/get-started/locally/), then run:
 
 
 
 
53
 
54
  ```shell
55
  python -m pip install packaging ninja psutil
 
57
  python -m pip install -r requirements.txt
58
  ```
59
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
60
  ## Pretrained Models
61
 
62
+ Download weights from [Hugging Face](https://huggingface.co/val-utehy/RotEx/tree/main):
63
 
64
+ | Model | Download | Save to |
65
  | --- | --- | --- |
66
+ | Multi-step | [best.pt](https://huggingface.co/val-utehy/RotEx/resolve/main/checkpoints/full/best.pt?download=true) | `checkpoints/full/best.pt` |
67
+ | Multi-step, final | [last.pt](https://huggingface.co/val-utehy/RotEx/resolve/main/checkpoints/full/last.pt?download=true) | `checkpoints/full/last.pt` |
68
+ | One-step | [best_distill.pt](https://huggingface.co/val-utehy/RotEx/resolve/main/distill_checkpoints/best_distill.pt?download=true) | `distill_checkpoints/best_distill.pt` |
 
 
69
 
70
  ```shell
71
  hf download val-utehy/RotEx checkpoints/full/best.pt --local-dir .
72
  hf download val-utehy/RotEx distill_checkpoints/best_distill.pt --local-dir .
73
+ hf download black-forest-labs/FLUX.1-dev ae.safetensors --local-dir checkpoints
74
  ```
75
 
76
+ DINOv2 and Depth Anything V2 are downloaded automatically. Run `hf auth login` if model access requires authentication.
 
 
 
 
 
 
 
 
 
 
 
77
 
78
+ ### Expected Weights Structure
79
 
80
+ ```text
81
+ RotEx/
82
+ ├── checkpoints/
83
+ │ ├── ae.safetensors
84
+ │ └── full/
85
+ │ ├── best.pt
86
+ │ └── last.pt # Optional
87
+ └── distill_checkpoints/
88
+ └── best_distill.pt
89
+ ```
90
 
91
+ ## Usage
 
 
 
 
92
 
93
+ ### 1. Training
94
 
95
+ Will be updated soon.
96
 
97
+ ### 2. Multi-Step Image Inference
 
 
 
 
 
 
 
 
 
 
 
98
 
99
  ```shell
100
  python infer_multistep.py \
 
102
  --content_dir inputs/content \
103
  --style_dir inputs/style \
104
  --output_dir outputs/multistep \
105
+ --denoise_steps 50
 
 
 
 
 
106
  ```
107
 
108
+ ### 3. One-Step Image Inference
 
 
 
 
 
 
 
 
 
109
 
110
+ Set these paths in `infer_1step_style.py`:
 
 
 
 
111
 
112
  ```python
113
  CHECKPOINT = "distill_checkpoints/best_distill.pt"
 
118
  HOURS = list(range(1, 25))
119
  ```
120
 
 
 
121
  ```shell
122
  python infer_1step_style.py
123
  ```
124
 
125
+ ### 4. Multi-Step Video Inference
 
 
126
 
127
  ```shell
128
  python infer_multistep_vid.py \
 
131
  --style_path inputs/style/night_reference.jpg \
132
  --output_dir outputs/video \
133
  --denoise_steps 50 \
 
 
 
134
  --hour_start 0 \
135
  --hour_end 24 \
 
 
136
  --no_audio
137
  ```
138
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
139
  ## Citation
140
 
141
+ <!-- Paper details and the official BibTeX entry will be added when available. -->
142
 
143
  ## References
144
 
 
149
 
150
  ## License
151
 
152
+ This project is licensed under the [Apache License 2.0](LICENSE). Third-party components retain their respective licenses.
checkpoints/README.md CHANGED
@@ -1,53 +1,19 @@
1
  # Pretrained Checkpoints
2
 
3
- RotEx model weights will be hosted on **Hugging Face**:
4
 
5
- [RotEx model repository](https://huggingface.co/val-utehy/RotEx)
 
 
 
6
 
7
- The commands below assume the uploaded files preserve the directory structure shown here.
8
-
9
- ## Directory Layout
10
-
11
- ```text
12
- checkpoints/
13
- ├── README.md
14
- ├── ae.safetensors
15
- └── full/
16
- ├── README.md
17
- ├── best.pt
18
- └── last.pt
19
- ```
20
-
21
- `best.pt` and `last.pt` are full-model checkpoints for `infer_multistep.py` and `infer_multistep_vid.py`. Use the distilled model in [`../distill_checkpoints/`](../distill_checkpoints/README.md) for `infer_1step_style.py`.
22
-
23
- ## Download the Full Model
24
-
25
- Run from the project root, the directory containing `config.py`:
26
 
27
  ```shell
28
  hf download val-utehy/RotEx checkpoints/full/best.pt --local-dir .
29
- ```
30
-
31
- Optionally download the final full-model checkpoint:
32
-
33
- ```shell
34
- hf download val-utehy/RotEx checkpoints/full/last.pt --local-dir .
35
- ```
36
-
37
- Pass `--checkpoint checkpoints/full/best.pt` to either multi-step inference script. See [`full/README.md`](full/README.md) for examples.
38
-
39
- ## Download the Autoencoder
40
-
41
- The FLUX autoencoder is distributed separately by its original provider:
42
-
43
- [black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev)
44
-
45
- ```shell
46
  hf download black-forest-labs/FLUX.1-dev ae.safetensors --local-dir checkpoints
47
  ```
48
 
49
- If authentication is required, complete the model's access process and run `hf auth login` first. The inference loader also attempts to download `ae.safetensors` automatically when it is absent from this directory.
50
-
51
- DINOv2 and Depth Anything V2 are loaded through Transformers into the Hugging Face cache; their weights do not need to be copied into this folder.
52
 
53
- See the [main README](../README.md) for installation and inference instructions.
 
1
  # Pretrained Checkpoints
2
 
3
+ Weights: [Hugging Face](https://huggingface.co/val-utehy/RotEx/tree/main).
4
 
5
+ | File | Download | Save to |
6
+ | --- | --- | --- |
7
+ | Full model, best | [best.pt](https://huggingface.co/val-utehy/RotEx/resolve/main/checkpoints/full/best.pt?download=true) | `checkpoints/full/best.pt` |
8
+ | Full model, final | [last.pt](https://huggingface.co/val-utehy/RotEx/resolve/main/checkpoints/full/last.pt?download=true) | `checkpoints/full/last.pt` |
9
 
10
+ Run from the project root:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
 
12
  ```shell
13
  hf download val-utehy/RotEx checkpoints/full/best.pt --local-dir .
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
  hf download black-forest-labs/FLUX.1-dev ae.safetensors --local-dir checkpoints
15
  ```
16
 
17
+ One-step weights: [distill_checkpoints](../distill_checkpoints/README.md).
 
 
18
 
19
+ Usage: [main README](../README.md).
checkpoints/full/README.md CHANGED
@@ -1,13 +1,9 @@
1
- # Full-Model Weights
2
 
3
- Download full-model checkpoints from the [RotEx Hugging Face repository](https://huggingface.co/val-utehy/RotEx).
4
-
5
- Downloads become available after the files are uploaded.
6
-
7
- | File | Use |
8
  | --- | --- |
9
- | `best.pt` | Best full-model checkpoint |
10
- | `last.pt` | Final full-model checkpoint; optional alternative |
11
 
12
  Run from the project root:
13
 
@@ -15,35 +11,12 @@ Run from the project root:
15
  hf download val-utehy/RotEx checkpoints/full/best.pt --local-dir .
16
  ```
17
 
18
- To download the alternative checkpoint:
19
 
20
  ```shell
21
  hf download val-utehy/RotEx checkpoints/full/last.pt --local-dir .
22
  ```
23
 
24
- The expected local paths are `checkpoints/full/best.pt` and `checkpoints/full/last.pt`.
25
-
26
- ## Image Inference
27
-
28
- ```shell
29
- python infer_multistep.py \
30
- --checkpoint checkpoints/full/best.pt \
31
- --content_dir inputs/content \
32
- --style_dir inputs/style \
33
- --output_dir outputs/multistep
34
- ```
35
-
36
- ## Video Inference
37
-
38
- ```shell
39
- python infer_multistep_vid.py \
40
- --checkpoint checkpoints/full/best.pt \
41
- --video_path inputs/video.mp4 \
42
- --style_path inputs/style/night_reference.jpg \
43
- --output_dir outputs/video \
44
- --no_audio
45
- ```
46
-
47
- These checkpoints are intended for the multi-step scripts. For one-step inference, download `distill_checkpoints/best_distill.pt` and follow the [distilled-checkpoint instructions](../../distill_checkpoints/README.md).
48
 
49
- See the [main README](../../README.md) for dependencies, auxiliary models, and inference options.
 
1
+ # Multi-Step Weights
2
 
3
+ | File | Download |
 
 
 
 
4
  | --- | --- |
5
+ | Best checkpoint | [best.pt](https://huggingface.co/val-utehy/RotEx/resolve/main/checkpoints/full/best.pt?download=true) |
6
+ | Final checkpoint | [last.pt](https://huggingface.co/val-utehy/RotEx/resolve/main/checkpoints/full/last.pt?download=true) |
7
 
8
  Run from the project root:
9
 
 
11
  hf download val-utehy/RotEx checkpoints/full/best.pt --local-dir .
12
  ```
13
 
14
+ Optional final checkpoint:
15
 
16
  ```shell
17
  hf download val-utehy/RotEx checkpoints/full/last.pt --local-dir .
18
  ```
19
 
20
+ Use with `infer_multistep.py` or `infer_multistep_vid.py`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
21
 
22
+ Usage: [main README](../../README.md).
distill_checkpoints/README.md CHANGED
@@ -1,35 +1,17 @@
1
- # Distilled One-Step Weights
2
 
3
- Download the distilled checkpoint from the [RotEx Hugging Face repository](https://huggingface.co/val-utehy/RotEx).
4
 
5
- Downloads become available after the checkpoint is uploaded.
6
-
7
- Run from the project root, the directory containing `config.py`:
8
 
9
  ```shell
10
  hf download val-utehy/RotEx distill_checkpoints/best_distill.pt --local-dir .
11
  ```
12
 
13
- The expected local path is:
14
-
15
- ```text
16
- distill_checkpoints/best_distill.pt
17
- ```
18
-
19
- ## Usage
20
-
21
- Set the checkpoint constant in `infer_1step_style.py`:
22
 
23
  ```python
24
  CHECKPOINT = "distill_checkpoints/best_distill.pt"
25
  ```
26
 
27
- Configure the input/output paths in the script, then run:
28
-
29
- ```shell
30
- python infer_1step_style.py
31
- ```
32
-
33
- `infer_1step_style.py` requires a fixed reference image configured through `FIXED_STYLE_PATH` and input/output folders configured through `DATA_PAIRS`.
34
-
35
- The script also needs the FLUX autoencoder, DINOv2, and Depth Anything V2 described in the [main README](../README.md). Full-model weights for multi-step inference belong in [`../checkpoints/full/`](../checkpoints/full/README.md).
 
1
+ # One-Step Weights
2
 
3
+ Download: [best_distill.pt](https://huggingface.co/val-utehy/RotEx/resolve/main/distill_checkpoints/best_distill.pt?download=true).
4
 
5
+ Run from the project root:
 
 
6
 
7
  ```shell
8
  hf download val-utehy/RotEx distill_checkpoints/best_distill.pt --local-dir .
9
  ```
10
 
11
+ In `infer_1step_style.py`:
 
 
 
 
 
 
 
 
12
 
13
  ```python
14
  CHECKPOINT = "distill_checkpoints/best_distill.pt"
15
  ```
16
 
17
+ Usage: [main README](../README.md).
 
 
 
 
 
 
 
 
infer_1step_style.py CHANGED
@@ -18,7 +18,7 @@ from config import get_config
18
  PI = math.pi
19
 
20
  # ===== MODEL =====
21
- CHECKPOINT = "/workspace/ChronFLUX/ChronFlux-v1/distill_checkpoints/rift_1step_epoch_5.pt"
22
 
23
  # ===== DATA (FIXED MAPPING) =====
24
  DATA_PAIRS = [
@@ -207,4 +207,4 @@ def main():
207
 
208
 
209
  if __name__ == "__main__":
210
- main()
 
18
  PI = math.pi
19
 
20
  # ===== MODEL =====
21
+ CHECKPOINT = "distill_checkpoints/best_distill.pt"
22
 
23
  # ===== DATA (FIXED MAPPING) =====
24
  DATA_PAIRS = [
 
207
 
208
 
209
  if __name__ == "__main__":
210
+ main()
infer_multistep.py CHANGED
@@ -194,7 +194,7 @@ def run_inference(
194
 
195
  def main():
196
  parser = argparse.ArgumentParser(description="ChronFlux 24-hour inference")
197
- parser.add_argument("--checkpoint", type=str, default="/workspace/RIFT-9226/checkpoints/rift_epoch_30.pt")
198
  parser.add_argument("--content_dir", type=str, default="/workspace/RIFT/rafael_ds_ver2/img_test/cnt")
199
  parser.add_argument("--style_dir", type=str, default="/workspace/RIFT/rafael_ds_ver2/img_test/sty")
200
  parser.add_argument("--output_dir", type=str, default="/workspace/RIFT-9226/inference_outputs")
 
194
 
195
  def main():
196
  parser = argparse.ArgumentParser(description="ChronFlux 24-hour inference")
197
+ parser.add_argument("--checkpoint", type=str, default="checkpoints/full/best.pt")
198
  parser.add_argument("--content_dir", type=str, default="/workspace/RIFT/rafael_ds_ver2/img_test/cnt")
199
  parser.add_argument("--style_dir", type=str, default="/workspace/RIFT/rafael_ds_ver2/img_test/sty")
200
  parser.add_argument("--output_dir", type=str, default="/workspace/RIFT-9226/inference_outputs")