--- license: cc-by-nc-4.0 pipeline_tag: image-to-image --- # RAESR Release package for the paper's single-pass super-resolution models: - **RAESR**: frozen DINOv3-L/16 fused latent → released ViT-XL RoPE decoder. - **RAESR-D**: frozen DINOv3-S/16+ fused latent → learned frontend (projector) → the same released decoder. Both models follow the paper protocol: RGB image → bicubic resize to **512×512** → one forward pass → **mean-only color matching** in the float domain, before PNG quantization: per RGB channel, `clamp(sr - mean(sr,H,W) + mean(lr,H,W), 0, 1)`, where `lr` is the same 512×512 bicubic-resized tensor the model consumed. The formula is recorded in every run manifest. There is no hidden color alignment, post-processing, normalization, or implicit fallback model. The released RAESR decoder is the paper's **Stage 3, seed 42, step-600 EMA** checkpoint: - Decoder SHA-256: `2e4de5df246172f067e0437f2346d17ca719a8dd12146624df0042983c580cb3` The encoder layers are named 0-indexed in code and manifests (`layers 1..23` for RAESR, `layers 1..11` for RAESR-D); these are layers 2–24 and 2–12 in the paper's 1-indexed notation. ## Repository contents ```text ema_decoder.pt released decoder checkpoint (ViT-XL, 2D RoPE) weights/raesr_d_frontend.pt released RAESR-D frontend weights raesr/ model runtime (DINOv3 encoders, decoder, RAESR-D frontend) scripts/ infer.py, download_artifacts.py, verify_artifacts.py, smoke_test.py, test_infer.py benchmark/ run_raesr_benchmark.py, score.py, summarize.py, validate_outputs.py, verify_paper_reproduction.py, tests/ manifests/ artifact identities, paper reference metrics, runtime provenance environment/ pinned runtime and metric requirements docs/REPRODUCIBILITY.md exact protocol and validation behavior ``` ## 1. Create the environment The environment file installs Python 3.10, PyTorch 2.7.1 and torchvision 0.22.1 (CUDA 12.8 wheels from the official PyTorch index) and the pinned dependencies: ```bash conda env create -f environment/environment.yml conda activate raesr ``` See `environment/requirements-runtime.txt` and `environment/requirements-metrics.txt` for pinned non-PyTorch packages. ## 2. Download and verify model weights Both released checkpoints are fetched from this repository and verified locally. `download_artifacts.py` writes them to `artifacts/ema_decoder.pt` and `artifacts/weights/raesr_d_frontend.pt`: ```bash python scripts/download_artifacts.py --output-dir artifacts python scripts/verify_artifacts.py --decoder artifacts/ema_decoder.pt --frontend artifacts/weights/raesr_d_frontend.pt ``` `download_artifacts.py` downloads from the `main` revision by default (override with `--revision`) and always checks the recorded SHA-256. Both models need the **native DINOv3** source checkout and the canonical Meta weights, which are not redistributed: ```text /hubconf.py /dinov3_vitl16_pretrain_lvd1689m-8aa4cbdd.pth (RAESR, DINOv3-L/16) /dinov3_vits16plus_pretrain_lvd1689m-4057cbaa.pth (RAESR-D, DINOv3-S/16+) ``` Obtain them from https://github.com/facebookresearch/dinov3 under Meta's DINOv3 License. Verify the weights you intend to use before inference (`--model` selects the DINOv3 checkpoint): ```bash python scripts/verify_artifacts.py \ --decoder artifacts/ema_decoder.pt \ --frontend artifacts/weights/raesr_d_frontend.pt \ --model raesr-d \ --dinov3-root \ --dinov3-weights ``` ## 3. Run folder inference ```bash python scripts/infer.py \ --input-dir \ --output-dir \ --model raesr \ --decoder artifacts/ema_decoder.pt \ --dinov3-root \ --dinov3-weights \ --device cuda ``` Pass `--model raesr-d --frontend artifacts/weights/raesr_d_frontend.pt` to run RAESR-D, with `--dinov3-weights` pointing to the folder that holds the DINOv3-S/16+ weight. Without `--frontend`, the frontend is read from `weights/raesr_d_frontend.pt` in a full clone of this repository. Both models consume the same input protocol and produce 512×512 outputs. Supported inputs are PNG, JPEG, BMP, TIFF, and WebP. Each input stem must be unique. Outputs use the same stem with `.png`; existing valid outputs resume by default. Use `--overwrite` to rerun selected images, or `--num-shards N --shard-index I` to launch disjoint GPU workers. Each worker writes `run_manifest.shard_XX.json` with hashes, runtime versions, inputs, outputs, the model choice, and the fixed protocol. ## 4. Smoke test The default smoke test validates image, output, and package contracts without downloading assets or loading a GPU model: ```bash python scripts/smoke_test.py ``` To verify the released checkpoint hash, add `--decoder artifacts/ema_decoder.pt`. To execute one real forward pass, add `--with-model` plus the DINO paths. ## 5. Evaluate on paired benchmark data Use this layout: ```text / lsdir/LR/ lsdir/GT/ realsr/LR/ realsr/GT/ drealsr/LR/drealsr/GT/ div2k/LR/ div2k/GT/ ``` Run inference and the paper metric set in one command: ```bash python benchmark/run_raesr_benchmark.py \ --data-root \ --output-root \ --model raesr \ --dinov3-root \ --dinov3-weights \ --gpu-ids \ --metric-cache \ --offline-metrics \ --expected-counts ``` Pass `--model raesr-d --frontend artifacts/weights/raesr_d_frontend.pt` for the RAESR-D benchmark. Outputs are written under `/sr//` and metrics under `/metrics//.csv`. `--gpu-ids` is optional. When supplied, the runner launches one deterministic input shard per listed physical GPU with `CUDA_VISIBLE_DEVICES` set for that shard, records the assignment in its manifest, waits for every shard, then scores once on the first listed GPU. Without it, inference remains single-process. Each shard log is stored under `/logs///`. `--metric-cache` is an explicit root holding the pyiqa, Torchvision, CLIP, and Hugging Face metric weights. It sets `TORCH_HOME` and `HUGGINGFACE_HUB_CACHE` for the scorer and checks the selected metric assets before any image is scored. Use `--offline-metrics` with a complete cache to forbid network access. The scorer writes per-image rows and one `MEAN` row with: PSNR-Y, SSIM-Y, LPIPS, DISTS, NIQE, MUSIQ, MANIQA-PIPAL, TOPIQ-NR, and CLIP-IQA. It rejects missing pairs, duplicate stems, mismatched image dimensions, metric failures, and non-finite metric values. ### Verify the paper rows `manifests/paper_reference_metrics.json` (RAESR) and `manifests/paper_reference_metrics_raesr_d.json` (RAESR-D) contain the paper-protocol four-split means, unweighted macro means, split counts, artifact hashes, the accepted protocol, and an absolute tolerance of `1e-4`. Expected paper macro means: RAESR PSNR 24.0825 / SSIM 0.6269; RAESR-D PSNR 23.39 / SSIM 0.6245. After a four-split run, verify every `MEAN` row and the macro result: ```bash python benchmark/verify_paper_reproduction.py \ --metrics-root /metrics/raesr \ --benchmark-provenance /raesr_benchmark.provenance.json \ --out /paper_reproduction.json ``` For RAESR-D, point `--metrics-root` at `/metrics/raesr-d` and add `--reference manifests/paper_reference_metrics_raesr_d.json`. The command checks the benchmark provenance (artifact hashes, per-shard manifests, output inventories, counts, score sidecars) and then compares every per-dataset and macro metric against the reference manifest. It writes every observed/reference/delta triplet and exits nonzero for a provenance, schema, count, finite-value, or tolerance failure. ## Reproducibility files - `manifests/model_artifacts.json`: artifact identities (decoder, RAESR-D frontend, DINOv3 checkpoints) and the architecture contract. - `manifests/paper_reference_metrics.json` / `manifests/paper_reference_metrics_raesr_d.json`: paper-protocol reference means, protocol, and tolerance. - `manifests/runtime_provenance.json`: source hashes of the runtime and the only changes made for this package. - `docs/REPRODUCIBILITY.md`: exact protocol, expected outputs, and validation behavior. ## License and attribution This package is distributed under CC BY-NC 4.0 for non-commercial research use (see `LICENSE`). Several runtime files are adapted from RAEv2 (CC BY-NC 4.0); `THIRD_PARTY_NOTICES.md` gives the attribution and lists the changes. The DINOv3 source and weights, including DINOv3-S/16+ used by RAESR-D, are not redistributed; obtain them from https://github.com/facebookresearch/dinov3 under Meta's DINOv3 License.