| --- |
| license: cc-by-nc-4.0 |
| pipeline_tag: image-to-image |
| --- |
| |
| # RAESR |
|
|
| Release package for the paper's single-pass super-resolution models: |
|
|
| - **RAESR**: frozen DINOv3-L/16 fused latent → released ViT-XL RoPE decoder. |
| - **RAESR-D**: frozen DINOv3-S/16+ fused latent → learned frontend (projector) → the same released decoder. |
|
|
| Both models follow the paper protocol: RGB image → bicubic resize to **512×512** → one forward pass → **mean-only color matching** in the float domain, before PNG quantization: per RGB channel, `clamp(sr - mean(sr,H,W) + mean(lr,H,W), 0, 1)`, where `lr` is the same 512×512 bicubic-resized tensor the model consumed. The formula is recorded in every run manifest. There is no hidden color alignment, post-processing, normalization, or implicit fallback model. |
|
|
| The released RAESR decoder is the paper's **Stage 3, seed 42, step-600 EMA** checkpoint: |
|
|
| - Decoder SHA-256: `2e4de5df246172f067e0437f2346d17ca719a8dd12146624df0042983c580cb3` |
|
|
| The encoder layers are named 0-indexed in code and manifests (`layers 1..23` for RAESR, `layers 1..11` for RAESR-D); these are layers 2–24 and 2–12 in the paper's 1-indexed notation. |
|
|
| ## Repository contents |
|
|
| ```text |
| ema_decoder.pt released decoder checkpoint (ViT-XL, 2D RoPE) |
| weights/raesr_d_frontend.pt released RAESR-D frontend weights |
| raesr/ model runtime (DINOv3 encoders, decoder, RAESR-D frontend) |
| scripts/ infer.py, download_artifacts.py, verify_artifacts.py, smoke_test.py, test_infer.py |
| benchmark/ run_raesr_benchmark.py, score.py, summarize.py, validate_outputs.py, verify_paper_reproduction.py, tests/ |
| manifests/ artifact identities, paper reference metrics, runtime provenance |
| environment/ pinned runtime and metric requirements |
| docs/REPRODUCIBILITY.md exact protocol and validation behavior |
| ``` |
|
|
| ## 1. Create the environment |
|
|
| The environment file installs Python 3.10, PyTorch 2.7.1 and torchvision 0.22.1 (CUDA 12.8 wheels from the official PyTorch index) and the pinned dependencies: |
|
|
| ```bash |
| conda env create -f environment/environment.yml |
| conda activate raesr |
| ``` |
|
|
| See `environment/requirements-runtime.txt` and `environment/requirements-metrics.txt` for pinned non-PyTorch packages. |
|
|
| ## 2. Download and verify model weights |
|
|
| Both released checkpoints are fetched from this repository and verified locally. `download_artifacts.py` writes them to `artifacts/ema_decoder.pt` and `artifacts/weights/raesr_d_frontend.pt`: |
|
|
| ```bash |
| python scripts/download_artifacts.py --output-dir artifacts |
| python scripts/verify_artifacts.py --decoder artifacts/ema_decoder.pt --frontend artifacts/weights/raesr_d_frontend.pt |
| ``` |
|
|
| `download_artifacts.py` downloads from the `main` revision by default (override with `--revision`) and always checks the recorded SHA-256. |
|
|
| Both models need the **native DINOv3** source checkout and the canonical Meta weights, which are not redistributed: |
|
|
| ```text |
| <DINOV3_ROOT>/hubconf.py |
| <DINOV3_WEIGHTS>/dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth (RAESR, DINOv3-L/16) |
| <DINOV3_WEIGHTS>/dinov3_vits16plus_pretrain_lvd1689m-4057cbaa.pth (RAESR-D, DINOv3-S/16+) |
| ``` |
|
|
| Obtain them from https://github.com/facebookresearch/dinov3 under Meta's DINOv3 License. Verify the weights you intend to use before inference (`--model` selects the DINOv3 checkpoint): |
|
|
| ```bash |
| python scripts/verify_artifacts.py \ |
| --decoder artifacts/ema_decoder.pt \ |
| --frontend artifacts/weights/raesr_d_frontend.pt \ |
| --model raesr-d \ |
| --dinov3-root <DINOV3_ROOT> \ |
| --dinov3-weights <DINOV3_WEIGHTS> |
| ``` |
|
|
| ## 3. Run folder inference |
|
|
| ```bash |
| python scripts/infer.py \ |
| --input-dir <INPUT_IMAGES> \ |
| --output-dir <OUTPUT_PNGS> \ |
| --model raesr \ |
| --decoder artifacts/ema_decoder.pt \ |
| --dinov3-root <DINOV3_ROOT> \ |
| --dinov3-weights <DINOV3_WEIGHTS> \ |
| --device cuda |
| ``` |
|
|
| Pass `--model raesr-d --frontend artifacts/weights/raesr_d_frontend.pt` to run RAESR-D, with `--dinov3-weights` pointing to the folder that holds the DINOv3-S/16+ weight. Without `--frontend`, the frontend is read from `weights/raesr_d_frontend.pt` in a full clone of this repository. Both models consume the same input protocol and produce 512×512 outputs. |
|
|
| Supported inputs are PNG, JPEG, BMP, TIFF, and WebP. Each input stem must be unique. Outputs use the same stem with `.png`; existing valid outputs resume by default. Use `--overwrite` to rerun selected images, or `--num-shards N --shard-index I` to launch disjoint GPU workers. Each worker writes `run_manifest.shard_XX.json` with hashes, runtime versions, inputs, outputs, the model choice, and the fixed protocol. |
|
|
| ## 4. Smoke test |
|
|
| The default smoke test validates image, output, and package contracts without downloading assets or loading a GPU model: |
|
|
| ```bash |
| python scripts/smoke_test.py |
| ``` |
|
|
| To verify the released checkpoint hash, add `--decoder artifacts/ema_decoder.pt`. To execute one real forward pass, add `--with-model` plus the DINO paths. |
|
|
| ## 5. Evaluate on paired benchmark data |
|
|
| Use this layout: |
|
|
| ```text |
| <DATA_ROOT>/ |
| lsdir/LR/ lsdir/GT/ |
| realsr/LR/ realsr/GT/ |
| drealsr/LR/drealsr/GT/ |
| div2k/LR/ div2k/GT/ |
| ``` |
|
|
| Run inference and the paper metric set in one command: |
|
|
| ```bash |
| python benchmark/run_raesr_benchmark.py \ |
| --data-root <DATA_ROOT> \ |
| --output-root <RESULTS_ROOT> \ |
| --model raesr \ |
| --dinov3-root <DINOV3_ROOT> \ |
| --dinov3-weights <DINOV3_WEIGHTS> \ |
| --gpu-ids <GPU_ID_1,GPU_ID_2,...> \ |
| --metric-cache <METRIC_CACHE> \ |
| --offline-metrics \ |
| --expected-counts |
| ``` |
|
|
| Pass `--model raesr-d --frontend artifacts/weights/raesr_d_frontend.pt` for the RAESR-D benchmark. Outputs are written under `<RESULTS_ROOT>/sr/<model>/<dataset>` and metrics under `<RESULTS_ROOT>/metrics/<model>/<dataset>.csv`. |
|
|
| `--gpu-ids` is optional. When supplied, the runner launches one deterministic input shard per listed physical GPU with `CUDA_VISIBLE_DEVICES` set for that shard, records the assignment in its manifest, waits for every shard, then scores once on the first listed GPU. Without it, inference remains single-process. Each shard log is stored under `<RESULTS_ROOT>/logs/<model>/<dataset>/`. |
|
|
| `--metric-cache` is an explicit root holding the pyiqa, Torchvision, CLIP, and Hugging Face metric weights. It sets `TORCH_HOME` and `HUGGINGFACE_HUB_CACHE` for the scorer and checks the selected metric assets before any image is scored. Use `--offline-metrics` with a complete cache to forbid network access. |
|
|
| The scorer writes per-image rows and one `MEAN` row with: PSNR-Y, SSIM-Y, LPIPS, DISTS, NIQE, MUSIQ, MANIQA-PIPAL, TOPIQ-NR, and CLIP-IQA. It rejects missing pairs, duplicate stems, mismatched image dimensions, metric failures, and non-finite metric values. |
|
|
| ### Verify the paper rows |
|
|
| `manifests/paper_reference_metrics.json` (RAESR) and `manifests/paper_reference_metrics_raesr_d.json` (RAESR-D) contain the paper-protocol four-split means, unweighted macro means, split counts, artifact hashes, the accepted protocol, and an absolute tolerance of `1e-4`. Expected paper macro means: RAESR PSNR 24.0825 / SSIM 0.6269; RAESR-D PSNR 23.39 / SSIM 0.6245. After a four-split run, verify every `MEAN` row and the macro result: |
|
|
| ```bash |
| python benchmark/verify_paper_reproduction.py \ |
| --metrics-root <RESULTS_ROOT>/metrics/raesr \ |
| --benchmark-provenance <RESULTS_ROOT>/raesr_benchmark.provenance.json \ |
| --out <RESULTS_ROOT>/paper_reproduction.json |
| ``` |
|
|
| For RAESR-D, point `--metrics-root` at `<RESULTS_ROOT>/metrics/raesr-d` and add `--reference manifests/paper_reference_metrics_raesr_d.json`. |
|
|
| The command checks the benchmark provenance (artifact hashes, per-shard manifests, output inventories, counts, score sidecars) and then compares every per-dataset and macro metric against the reference manifest. It writes every observed/reference/delta triplet and exits nonzero for a provenance, schema, count, finite-value, or tolerance failure. |
|
|
| ## Reproducibility files |
|
|
| - `manifests/model_artifacts.json`: artifact identities (decoder, RAESR-D frontend, DINOv3 checkpoints) and the architecture contract. |
| - `manifests/paper_reference_metrics.json` / `manifests/paper_reference_metrics_raesr_d.json`: paper-protocol reference means, protocol, and tolerance. |
| - `manifests/runtime_provenance.json`: source hashes of the runtime and the only changes made for this package. |
| - `docs/REPRODUCIBILITY.md`: exact protocol, expected outputs, and validation behavior. |
|
|
| ## License and attribution |
|
|
| This package is distributed under CC BY-NC 4.0 for non-commercial research use (see `LICENSE`). Several runtime files are adapted from RAEv2 (CC BY-NC 4.0); `THIRD_PARTY_NOTICES.md` gives the attribution and lists the changes. The DINOv3 source and weights, including DINOv3-S/16+ used by RAESR-D, are not redistributed; obtain them from https://github.com/facebookresearch/dinov3 under Meta's DINOv3 License. |
|
|
