Image-to-Image
PyTorch
ONNX
TensorRT
English
super-resolution
image-restoration
sisr
onnxruntime
spandrel
chainner
edge
deployment
mit
Instructions to use inYourOwnBrowser/PULSE with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- TensorRT
How to use inYourOwnBrowser/PULSE with TensorRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Commit ·
fb6c625
0
Parent(s):
Duplicate from Phips/PULSE
Browse filesCo-authored-by: Philip Hofmann <Phips@users.noreply.huggingface.co>
- .gitattributes +45 -0
- ATTRIBUTION.md +15 -0
- LICENSE +21 -0
- PAPER.md +245 -0
- README.md +318 -0
- checksums.txt +24 -0
- configs/2x_pulse_fast_pretrain.yml +82 -0
- configs/2x_pulse_pretrain.yml +82 -0
- configs/4x_pulse_fast_pretrain.yml +82 -0
- configs/4x_pulse_pretrain.yml +82 -0
- configs/pulse_arch.py +165 -0
- examples/4x_pulse_urban100_img_001_input_vs_pretrain.png +3 -0
- examples/4x_pulse_urban100_img_004_input_vs_pretrain.png +3 -0
- examples/4x_pulse_urban100_img_006_input_vs_pretrain.png +3 -0
- models/pulse/2x_pretrain.safetensors +3 -0
- models/pulse/4x_pretrain.safetensors +3 -0
- models/pulse_fast/2x_pretrain.safetensors +3 -0
- models/pulse_fast/4x_pretrain.safetensors +3 -0
- onnx/2x_pulse.onnx +3 -0
- onnx/2x_pulse_fast.onnx +3 -0
- onnx/2x_pulse_fast_fp16.onnx +3 -0
- onnx/2x_pulse_fp16.onnx +3 -0
- onnx/4x_pulse.onnx +3 -0
- onnx/4x_pulse_fast.onnx +3 -0
- onnx/4x_pulse_fast_fp16.onnx +3 -0
- onnx/4x_pulse_fp16.onnx +3 -0
.gitattributes
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
examples/4x_pulse_urban100_img001_ab.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
examples/4x_pulse_urban100_img004_ab.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
examples/4x_pulse_urban100_img006_ab.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
onnx/2x_pulse.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
onnx/2x_pulse_fast.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
onnx/4x_pulse.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
onnx/4x_pulse_fast.onnx.data filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
examples/4x_pulse_urban100_img_001_input_vs_pretrain.png filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
examples/4x_pulse_urban100_img_004_input_vs_pretrain.png filter=lfs diff=lfs merge=lfs -text
|
| 45 |
+
examples/4x_pulse_urban100_img_006_input_vs_pretrain.png filter=lfs diff=lfs merge=lfs -text
|
ATTRIBUTION.md
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Attribution
|
| 2 |
+
|
| 3 |
+
PULSE uses a fixed **Magic Kernel Sharp 2013 (MKS2013)** base upsampler.
|
| 4 |
+
|
| 5 |
+
- Algorithm and original reference: **John Costella**, *Magic Kernel Sharp*,
|
| 6 |
+
https://johncostella.com/magic/
|
| 7 |
+
- Public reference implementation: `felixr/magic-kernel-sharp` (MIT),
|
| 8 |
+
https://github.com/felixr/magic-kernel-sharp
|
| 9 |
+
|
| 10 |
+
PULSE contains an independent PyTorch implementation of the integer-scale
|
| 11 |
+
MKS2013 phase kernels. It is not a copy of the reference C implementation.
|
| 12 |
+
The formula and the MIT-licensed reference are attributed here for clarity.
|
| 13 |
+
|
| 14 |
+
The PULSE release is MIT-licensed. This notice is attribution, not a legal
|
| 15 |
+
opinion; verify the current upstream terms before redistribution.
|
LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Philip Hofmann
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
PAPER.md
ADDED
|
@@ -0,0 +1,245 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# PULSE technical report
|
| 2 |
+
|
| 3 |
+
*Private draft. Companion to the model card.*
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
# PULSE
|
| 7 |
+
|
| 8 |
+
**A tiny, deployment-first super-resolution network.**
|
| 9 |
+
|
| 10 |
+
PULSE is the **tiny/fast sibling of NERVE**: a much smaller convolutional SISR
|
| 11 |
+
model with the same deployment-first philosophy. It trades capacity for size and
|
| 12 |
+
speed, so it is less capable than NERVE and is intended for light cleanup and
|
| 13 |
+
clean upscaling rather than heavy restoration. It trains on a consumer GPU,
|
| 14 |
+
converts without surprises, and embeds almost anywhere.
|
| 15 |
+
|
| 16 |
+
- `pulse`: 48 channels, **353,856 parameters**
|
| 17 |
+
- `pulse_fast`: 32 channels, **162,192 parameters**
|
| 18 |
+
- 2x and 4x models
|
| 19 |
+
- PyTorch `.safetensors` + dynamic ONNX
|
| 20 |
+
- **MIT** code and weights, trained on **CC0 / public-domain images only**
|
| 21 |
+
|
| 22 |
+
**Links:** [HEART](https://huggingface.co/Phips/HEART) · [NERVE](https://huggingface.co/Phips/NERVE) · [PULSE demo](https://huggingface.co/spaces/Phips/PULSE-demo) · [PULSE ONNX converter](https://huggingface.co/spaces/Phips/PULSE-convert)
|
| 23 |
+
|
| 24 |
+
> Short version: load a released `.safetensors` with `pulse_arch.py`, or an
|
| 25 |
+
> `.onnx` with ONNX Runtime. Start with 2x for browser/CPU use.
|
| 26 |
+
|
| 27 |
+
---
|
| 28 |
+
|
| 29 |
+
## Why PULSE
|
| 30 |
+
|
| 31 |
+
PULSE is not different for the sake of being different. Each constraint removes
|
| 32 |
+
a real deployment or maintenance problem.
|
| 33 |
+
|
| 34 |
+
- **No `Resize` op in the exported graph.** The scale is done by a fixed MKS2013
|
| 35 |
+
base plus PixelShuffle. ONNX, TensorRT, NCNN, and other runtimes all see the
|
| 36 |
+
same standard graph; there is no interpolation implementation mismatch.
|
| 37 |
+
- **Only standard ops.** `Conv / Add / ReLU / Pad / DepthToSpace`. No custom
|
| 38 |
+
kernels, no fused/unfused checkpoint pair, no fragile converter step. That
|
| 39 |
+
makes it easy to audit, port, optimize with `onnxsim`, and compile with
|
| 40 |
+
TensorRT.
|
| 41 |
+
- **Fixed MKS2013 base.** The base upsampler has no learned parameters and no
|
| 42 |
+
dynamic branch. It anchors the output, reduces what the network must learn,
|
| 43 |
+
and keeps the graph compact and deterministic.
|
| 44 |
+
- **No attention and no normalization.** That means stable bf16 AMP training,
|
| 45 |
+
fewer memory spikes, and simpler maintenance. The whole model is readable.
|
| 46 |
+
- **Tiny.** 162K–354K parameters. It fits browser/WASM budgets, small GPUs,
|
| 47 |
+
CPU fallbacks, and embedded pipelines where a transformer does not.
|
| 48 |
+
- **Fast training.** The pretrain uses Charbonnier only: no discriminator, no
|
| 49 |
+
perceptual stack, no fragile GAN schedule. It trains on a single RTX 3060.
|
| 50 |
+
- **Dynamic ONNX.** One model works at any H/W; no per-resolution exports.
|
| 51 |
+
|
| 52 |
+
## What it is for
|
| 53 |
+
|
| 54 |
+
PULSE is a **local, low-cost upscaler**. The released pretrains are clean
|
| 55 |
+
2x/4x models, not a general blind-restoration model.
|
| 56 |
+
|
| 57 |
+
- **Browser or desktop upscaling.** ONNX Runtime Web/WASM/WebGPU can run the 2x
|
| 58 |
+
models; 4x is heavier but still small.
|
| 59 |
+
- **Image viewers, galleries, thumbnail pipelines.** A fast local pass before
|
| 60 |
+
display, with no server round-trip.
|
| 61 |
+
- **Anime and illustration stills.** Clean 2x/4x upscaling; a compression-trained
|
| 62 |
+
variant is a separate model.
|
| 63 |
+
- **Batch/offline preprocessing.** Small enough to share a consumer GPU or CPU.
|
| 64 |
+
- **Embedding in apps.** One architecture file, one op set, no fuse step.
|
| 65 |
+
|
| 66 |
+
Compression cleanup is a real use case, but it needs the separate
|
| 67 |
+
**PULSE-Web** finetune. The released clean pretrain was not trained on
|
| 68 |
+
JPEG/WebP/AVIF/H.265/GIF artifacts.
|
| 69 |
+
|
| 70 |
+
A temporal/video cleaner is **not part of this release**. Per-frame PULSE can
|
| 71 |
+
be a component, but flicker-free video needs temporal conditioning.
|
| 72 |
+
|
| 73 |
+
## When to use NERVE or HEART instead
|
| 74 |
+
|
| 75 |
+
| Need | Use |
|
| 76 |
+
|---|---|
|
| 77 |
+
| Browser/CPU, tiny model, fast 2x/4x | **PULSE** |
|
| 78 |
+
| Better quality with a ~1.8M conv-only edge model | **NERVE** |
|
| 79 |
+
| Highest quality / desktop-server GPU budget | **HEART** |
|
| 80 |
+
| Blind OTF restoration, heavy blur/noise | NERVE/HEART OTF or a larger restoration model |
|
| 81 |
+
| Text/UI/screenshot upscaling | a text-aware restoration model, not PULSE |
|
| 82 |
+
| Temporal video restoration | a temporal model; PULSE is per-frame only |
|
| 83 |
+
|
| 84 |
+
PULSE is the **tiny/fast NERVE sibling**: the same practical philosophy, but
|
| 85 |
+
far fewer parameters and a lighter operating envelope. NERVE is the more
|
| 86 |
+
capable restoration model; PULSE is for light degradation, clean upscaling,
|
| 87 |
+
browser/CPU use, and cases where the model itself must be almost free.
|
| 88 |
+
|
| 89 |
+
PULSE is not competing with HEART on PSNR.
|
| 90 |
+
|
| 91 |
+
## Pick a model
|
| 92 |
+
|
| 93 |
+
| Want | Use | Scale | Params |
|
| 94 |
+
|---|---|---:|---:|
|
| 95 |
+
| Best small-model quality | `pulse` | 4x / 2x | 353,856 |
|
| 96 |
+
| Fastest / lowest compute | `pulse_fast` | 4x / 2x | 162,192 |
|
| 97 |
+
| CPU / ORT / engine | `onnx/*_fp16.onnx` | 4x / 2x | same |
|
| 98 |
+
|
| 99 |
+
All released models are pretrained only on **CC0 / public-domain images**. They
|
| 100 |
+
are MIT-licensed and permissive.
|
| 101 |
+
|
| 102 |
+
### Urban100 validation
|
| 103 |
+
|
| 104 |
+
| Model | PSNR | SSIM | TopIQ | LPIPS |
|
| 105 |
+
|---|---:|---:|---:|---:|
|
| 106 |
+
| 4x `pulse` | **25.6523** | **0.7717** | **0.5907** | **0.2452** |
|
| 107 |
+
| 2x `pulse` | **31.4423** | **0.9221** | **0.8614** | **0.0721** |
|
| 108 |
+
| 4x `pulse_fast` | **25.4239** | **0.7638** | **0.5748** | **0.2553** |
|
| 109 |
+
| 2x `pulse_fast` | **31.1371** | **0.9188** | **0.8551** | **0.0755** |
|
| 110 |
+
|
| 111 |
+
EMA weights, standard Urban100 validation. PSNR/SSIM are Y-channel with a 4px border crop; TopIQ/LPIPS are RGB. No TTA.
|
| 112 |
+
|
| 113 |
+
### Inference speed
|
| 114 |
+
|
| 115 |
+
RTX 3060, batch 1, fp16. TensorRT engine built locally.
|
| 116 |
+
|
| 117 |
+
| Model | 256² eager | 720p eager | 1080p eager | 256² TRT | 720p TRT | 1080p TRT |
|
| 118 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 119 |
+
| `pulse_fast` | 3.65 | 47.00 | 103.88 | 1.43 | 17.52 | 39.60 |
|
| 120 |
+
| `pulse` | 6.71 | 90.20 | 199.28 | 2.82 | 35.35 | 79.65 |
|
| 121 |
+
|
| 122 |
+
### Training speed
|
| 123 |
+
|
| 124 |
+
CC0/public-domain 200k fidelity pretrain, RTX 3060, batch 8, bf16:
|
| 125 |
+
|
| 126 |
+
- `pulse` 4x: ~29 it/s
|
| 127 |
+
- `pulse` 2x: ~9.8 it/s
|
| 128 |
+
- `pulse_fast` 4x: ~36 it/s
|
| 129 |
+
- `pulse_fast` 2x: ~20 it/s
|
| 130 |
+
|
| 131 |
+
Training is Charbonnier-only. Perceptual branches are experimental and are not
|
| 132 |
+
part of the pretrain release.
|
| 133 |
+
|
| 134 |
+
## Quickstart
|
| 135 |
+
|
| 136 |
+
### ONNX Runtime
|
| 137 |
+
|
| 138 |
+
```python
|
| 139 |
+
import onnxruntime as ort
|
| 140 |
+
sess = ort.InferenceSession("onnx/4x_pulse_fp16.onnx", providers=["CPUExecutionProvider"])
|
| 141 |
+
# input: float32 [1, 3, H, W] in [0, 1], RGB, dynamic H/W
|
| 142 |
+
```
|
| 143 |
+
|
| 144 |
+
The fp16 file is self-contained. Use `4x_pulse.onnx` if your runtime lacks fp16.
|
| 145 |
+
|
| 146 |
+
### PyTorch / spandrel
|
| 147 |
+
|
| 148 |
+
1. Copy `pulse_arch.py` into `traiNNer/archs/` (or use the file here).
|
| 149 |
+
2. Load the `.safetensors` with spandrel once PULSE is registered.
|
| 150 |
+
3. Input RGB `[0,1]`, output 2x/4x.
|
| 151 |
+
|
| 152 |
+
### Train
|
| 153 |
+
|
| 154 |
+
```bash
|
| 155 |
+
cp pulse_arch.py <traiNNer-redux>/traiNNer/archs/
|
| 156 |
+
cp configs/4x_pulse_pretrain.yml <traiNNer-redux>/options/
|
| 157 |
+
cd <traiNNer-redux>
|
| 158 |
+
python train.py -opt options/4x_pulse_pretrain.yml
|
| 159 |
+
```
|
| 160 |
+
|
| 161 |
+
The sample configs use placeholder data paths and can be trained from scratch.
|
| 162 |
+
The released pretrains are provided as warm starts.
|
| 163 |
+
|
| 164 |
+
## Visual examples
|
| 165 |
+
|
| 166 |
+
Urban100 4x. Left: bicubic input. Right: released PULSE 4x pretrain.
|
| 167 |
+
|
| 168 |
+

|
| 169 |
+

|
| 170 |
+

|
| 171 |
+
|
| 172 |
+
## Files
|
| 173 |
+
|
| 174 |
+
```text
|
| 175 |
+
pulse_arch.py
|
| 176 |
+
models/
|
| 177 |
+
pulse/4x_pretrain.safetensors
|
| 178 |
+
pulse/2x_pretrain.safetensors
|
| 179 |
+
pulse_fast/4x_pretrain.safetensors
|
| 180 |
+
pulse_fast/2x_pretrain.safetensors
|
| 181 |
+
onnx/
|
| 182 |
+
*_fp16.onnx
|
| 183 |
+
*.onnx
|
| 184 |
+
configs/
|
| 185 |
+
4x_pulse_pretrain.yml
|
| 186 |
+
2x_pulse_pretrain.yml
|
| 187 |
+
4x_pulse_fast_pretrain.yml
|
| 188 |
+
2x_pulse_fast_pretrain.yml
|
| 189 |
+
PAPER.md
|
| 190 |
+
examples/
|
| 191 |
+
```
|
| 192 |
+
|
| 193 |
+
## Limitations
|
| 194 |
+
|
| 195 |
+
- Not a blind-OTF model; no degradation embedding.
|
| 196 |
+
- Not a text/UI upscaler.
|
| 197 |
+
- The clean pretrain does not clean compression artifacts; use a web/anime
|
| 198 |
+
finetune for that.
|
| 199 |
+
- No temporal/video model is included.
|
| 200 |
+
- PSNR is below HEART; that is the intended size/speed tradeoff.
|
| 201 |
+
|
| 202 |
+
## Attribution
|
| 203 |
+
|
| 204 |
+
The fixed base uses **Magic Kernel Sharp 2013** by **John Costella**
|
| 205 |
+
(https://johncostella.com/magic/). The public reference implementation
|
| 206 |
+
(`felixr/magic-kernel-sharp`, MIT) is credited in `ATTRIBUTION.md`. PULSE uses
|
| 207 |
+
an independent PyTorch implementation of the integer-scale phase kernels, not
|
| 208 |
+
a copy of the reference C code.
|
| 209 |
+
|
| 210 |
+
## License
|
| 211 |
+
|
| 212 |
+
The code and released weights are **MIT-licensed** and permissive. Keep the
|
| 213 |
+
MIT copyright and permission notice with copies or substantial portions of the
|
| 214 |
+
software. See `LICENSE`.
|
| 215 |
+
|
| 216 |
+
The released weights were trained on a curated CC0 / public-domain image set.
|
| 217 |
+
This is a provenance statement, not a guarantee; source-image rights and
|
| 218 |
+
jurisdiction-specific rules can still matter. No warranty is provided.
|
| 219 |
+
|
| 220 |
+
```bibtex
|
| 221 |
+
@misc{pulse_body_2026,
|
| 222 |
+
title = {PULSE: A Deployment-First Single-Image Super-Resolution Network},
|
| 223 |
+
author = {Philip Hofmann and contributors},
|
| 224 |
+
year = {2026},
|
| 225 |
+
note = {BODY suite}
|
| 226 |
+
}
|
| 227 |
+
```
|
| 228 |
+
|
| 229 |
+
|
| 230 |
+
---
|
| 231 |
+
|
| 232 |
+
## Appendix: ablation context
|
| 233 |
+
|
| 234 |
+
PULSE was selected through equal-budget internal experiments. The published
|
| 235 |
+
weights are the 200k pretrains trained only on CC0/public-domain images.
|
| 236 |
+
|
| 237 |
+
- 48 channels / 8 blocks beat 32 channels / 8 blocks on quality at 2.2x the
|
| 238 |
+
parameters; both keep the same deployment graph.
|
| 239 |
+
- MKS2013 and bicubic bases tied on quality; MKS was kept because it removes
|
| 240 |
+
`Resize` from ONNX.
|
| 241 |
+
- RepConv/PULSE-R gained PSNR but cost 2.1x training time, 3.3x training VRAM,
|
| 242 |
+
10x training parameters, and a fuse step; rejected.
|
| 243 |
+
- Partial depthwise, 5x5 stem, and tiny variants were rejected.
|
| 244 |
+
- Perceptual/compression branches are separate and are not part of this
|
| 245 |
+
pretrain release.
|
README.md
ADDED
|
@@ -0,0 +1,318 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
library_name: pytorch
|
| 6 |
+
pipeline_tag: image-to-image
|
| 7 |
+
tags:
|
| 8 |
+
- super-resolution
|
| 9 |
+
- image-restoration
|
| 10 |
+
- image-to-image
|
| 11 |
+
- sisr
|
| 12 |
+
- pytorch
|
| 13 |
+
- onnx
|
| 14 |
+
- onnxruntime
|
| 15 |
+
- tensorrt
|
| 16 |
+
- spandrel
|
| 17 |
+
- chainner
|
| 18 |
+
- edge
|
| 19 |
+
- deployment
|
| 20 |
+
- mit
|
| 21 |
+
---
|
| 22 |
+
|
| 23 |
+
# PULSE
|
| 24 |
+
|
| 25 |
+
**A tiny, deployment-first super-resolution network.**
|
| 26 |
+
|
| 27 |
+
PULSE is the **tiny/fast sibling of NERVE**: a much smaller convolutional SISR
|
| 28 |
+
model with the same deployment-first philosophy. It trades capacity for size and
|
| 29 |
+
speed, so it is less capable than NERVE and is intended for light cleanup and
|
| 30 |
+
clean upscaling rather than heavy restoration. It trains on a consumer GPU,
|
| 31 |
+
converts without surprises, and embeds almost anywhere.
|
| 32 |
+
|
| 33 |
+
- `pulse`: 48 channels, **353,856 parameters**
|
| 34 |
+
- `pulse_fast`: 32 channels, **162,192 parameters**
|
| 35 |
+
- 2x and 4x models
|
| 36 |
+
- PyTorch `.safetensors` + dynamic ONNX
|
| 37 |
+
- **MIT** code and weights, trained on **CC0 / public-domain images only**
|
| 38 |
+
|
| 39 |
+
**Links:** [HEART](https://huggingface.co/Phips/HEART) · [NERVE](https://huggingface.co/Phips/NERVE) · [PULSE demo](https://huggingface.co/spaces/Phips/PULSE-demo) · [PULSE ONNX converter](https://huggingface.co/spaces/Phips/PULSE-convert)
|
| 40 |
+
|
| 41 |
+
> Short version: load a released `.safetensors` with `pulse_arch.py`, or an
|
| 42 |
+
> `.onnx` with ONNX Runtime. Start with 2x for browser/CPU use.
|
| 43 |
+
|
| 44 |
+
---
|
| 45 |
+
|
| 46 |
+
## Why PULSE
|
| 47 |
+
|
| 48 |
+
PULSE is not different for the sake of being different. Each constraint removes
|
| 49 |
+
a real deployment or maintenance problem.
|
| 50 |
+
|
| 51 |
+
- **No `Resize` op in the exported graph.** The scale is done by a fixed MKS2013
|
| 52 |
+
base plus PixelShuffle. ONNX, TensorRT, NCNN, and other runtimes all see the
|
| 53 |
+
same standard graph; there is no interpolation implementation mismatch.
|
| 54 |
+
- **Only standard ops.** `Conv / Add / ReLU / Pad / DepthToSpace`. No custom
|
| 55 |
+
kernels, no fused/unfused checkpoint pair, no fragile converter step. That
|
| 56 |
+
makes it easy to audit, port, optimize with `onnxsim`, and compile with
|
| 57 |
+
TensorRT.
|
| 58 |
+
- **Fixed MKS2013 base.** The base upsampler has no learned parameters and no
|
| 59 |
+
dynamic branch. It anchors the output, reduces what the network must learn,
|
| 60 |
+
and keeps the graph compact and deterministic.
|
| 61 |
+
- **No attention and no normalization.** That means stable bf16 AMP training,
|
| 62 |
+
fewer memory spikes, and simpler maintenance. The whole model is readable.
|
| 63 |
+
- **Tiny.** 162K–354K parameters. It fits browser/WASM budgets, small GPUs,
|
| 64 |
+
CPU fallbacks, and embedded pipelines where a transformer does not.
|
| 65 |
+
- **Fast training.** The pretrain uses Charbonnier only: no discriminator, no
|
| 66 |
+
perceptual stack, no fragile GAN schedule. It trains on a single RTX 3060.
|
| 67 |
+
- **Dynamic ONNX.** One model works at any H/W; no per-resolution exports.
|
| 68 |
+
|
| 69 |
+
## What it is for
|
| 70 |
+
|
| 71 |
+
PULSE is a **local, low-cost upscaler**. The released pretrains are clean
|
| 72 |
+
2x/4x models, not a general blind-restoration model.
|
| 73 |
+
|
| 74 |
+
- **Browser or desktop upscaling.** ONNX Runtime Web/WASM/WebGPU can run the 2x
|
| 75 |
+
models; 4x is heavier but still small.
|
| 76 |
+
- **Image viewers, galleries, thumbnail pipelines.** A fast local pass before
|
| 77 |
+
display, with no server round-trip.
|
| 78 |
+
- **Anime and illustration stills.** Clean 2x/4x upscaling; a compression-trained
|
| 79 |
+
variant is a separate model.
|
| 80 |
+
- **Batch/offline preprocessing.** Small enough to share a consumer GPU or CPU.
|
| 81 |
+
- **Embedding in apps.** One architecture file, one op set, no fuse step.
|
| 82 |
+
|
| 83 |
+
Compression cleanup is a real use case, but it needs the separate
|
| 84 |
+
**PULSE-Web** finetune. The released clean pretrain was not trained on
|
| 85 |
+
JPEG/WebP/AVIF/H.265/GIF artifacts.
|
| 86 |
+
|
| 87 |
+
A temporal/video cleaner is **not part of this release**. Per-frame PULSE can
|
| 88 |
+
be a component, but flicker-free video needs temporal conditioning.
|
| 89 |
+
|
| 90 |
+
## When to use NERVE or HEART instead
|
| 91 |
+
|
| 92 |
+
| Need | Use |
|
| 93 |
+
|---|---|
|
| 94 |
+
| Browser/CPU, tiny model, fast 2x/4x | **PULSE** |
|
| 95 |
+
| Better quality / heavier restoration with a ~1.8M conv-only edge model | **NERVE** |
|
| 96 |
+
| Highest quality / desktop-server GPU budget | **HEART** |
|
| 97 |
+
| Blind OTF restoration, heavy blur/noise | NERVE/HEART OTF or a larger restoration model |
|
| 98 |
+
| Tiny/fast NERVE-like model for light cleanup | **PULSE** |
|
| 99 |
+
| Text/UI/screenshot upscaling | a text-aware restoration model, not PULSE |
|
| 100 |
+
| Temporal video restoration | a temporal model; PULSE is per-frame only |
|
| 101 |
+
|
| 102 |
+
PULSE is the **tiny/fast NERVE sibling**: the same practical philosophy, but
|
| 103 |
+
far fewer parameters and a lighter operating envelope. NERVE is the more
|
| 104 |
+
capable restoration model; PULSE is for light degradation, clean upscaling,
|
| 105 |
+
browser/CPU use, and cases where the model itself must be almost free.
|
| 106 |
+
|
| 107 |
+
PULSE is not competing with HEART on PSNR.
|
| 108 |
+
|
| 109 |
+
## Pick a model
|
| 110 |
+
|
| 111 |
+
| Want | Use | Scale | Params |
|
| 112 |
+
|---|---|---:|---:|
|
| 113 |
+
| Best small-model quality | `pulse` | 4x / 2x | 353,856 |
|
| 114 |
+
| Fastest / lowest compute | `pulse_fast` | 4x / 2x | 162,192 |
|
| 115 |
+
| CPU / ORT / engine | `onnx/*_fp16.onnx` | 4x / 2x | same |
|
| 116 |
+
|
| 117 |
+
All released models are pretrained only on **CC0 / public-domain images**. They
|
| 118 |
+
are MIT-licensed and permissive.
|
| 119 |
+
|
| 120 |
+
### Urban100 validation
|
| 121 |
+
|
| 122 |
+
| Model | PSNR | SSIM | TopIQ | LPIPS |
|
| 123 |
+
|---|---:|---:|---:|---:|
|
| 124 |
+
| 4x `pulse` | **25.6523** | **0.7717** | **0.5907** | **0.2452** |
|
| 125 |
+
| 2x `pulse` | **31.4423** | **0.9221** | **0.8614** | **0.0721** |
|
| 126 |
+
| 4x `pulse_fast` | **25.4239** | **0.7638** | **0.5748** | **0.2553** |
|
| 127 |
+
| 2x `pulse_fast` | **31.1371** | **0.9188** | **0.8551** | **0.0755** |
|
| 128 |
+
|
| 129 |
+
EMA weights, standard Urban100 validation. PSNR/SSIM are Y-channel with a 4px border crop; TopIQ/LPIPS are RGB. No TTA.
|
| 130 |
+
|
| 131 |
+
### Inference speed
|
| 132 |
+
|
| 133 |
+
RTX 3060, batch 1, fp16. TensorRT engine built locally.
|
| 134 |
+
|
| 135 |
+
| Model | 256² eager | 720p eager | 1080p eager | 256² TRT | 720p TRT | 1080p TRT |
|
| 136 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 137 |
+
| `pulse_fast` | 3.65 | 47.00 | 103.88 | 1.43 | 17.52 | 39.60 |
|
| 138 |
+
| `pulse` | 6.71 | 90.20 | 199.28 | 2.82 | 35.35 | 79.65 |
|
| 139 |
+
|
| 140 |
+
### Training speed
|
| 141 |
+
|
| 142 |
+
CC0/public-domain 200k fidelity pretrain, RTX 3060, batch 8, bf16:
|
| 143 |
+
|
| 144 |
+
- `pulse` 4x: ~29 it/s
|
| 145 |
+
- `pulse` 2x: ~9.8 it/s
|
| 146 |
+
- `pulse_fast` 4x: ~36 it/s
|
| 147 |
+
- `pulse_fast` 2x: ~20 it/s
|
| 148 |
+
|
| 149 |
+
Training is Charbonnier-only. Perceptual branches are experimental and are not
|
| 150 |
+
part of the pretrain release.
|
| 151 |
+
|
| 152 |
+
## Quickstart
|
| 153 |
+
|
| 154 |
+
### ONNX Runtime
|
| 155 |
+
|
| 156 |
+
```python
|
| 157 |
+
import onnxruntime as ort
|
| 158 |
+
sess = ort.InferenceSession("onnx/4x_pulse_fp16.onnx", providers=["CPUExecutionProvider"])
|
| 159 |
+
# input: float32 [1, 3, H, W] in [0, 1], RGB, dynamic H/W
|
| 160 |
+
```
|
| 161 |
+
|
| 162 |
+
The fp16 file is self-contained. Use `4x_pulse.onnx` if your runtime lacks fp16.
|
| 163 |
+
|
| 164 |
+
### PyTorch / spandrel
|
| 165 |
+
|
| 166 |
+
1. Copy `pulse_arch.py` into `traiNNer/archs/` (or use the file here).
|
| 167 |
+
2. Load the `.safetensors` with spandrel once PULSE is registered.
|
| 168 |
+
3. Input RGB `[0,1]`, output 2x/4x.
|
| 169 |
+
|
| 170 |
+
### Train
|
| 171 |
+
|
| 172 |
+
```bash
|
| 173 |
+
cp pulse_arch.py <traiNNer-redux>/traiNNer/archs/
|
| 174 |
+
cp configs/4x_pulse_pretrain.yml <traiNNer-redux>/options/
|
| 175 |
+
cd <traiNNer-redux>
|
| 176 |
+
python train.py -opt options/4x_pulse_pretrain.yml
|
| 177 |
+
```
|
| 178 |
+
|
| 179 |
+
The sample configs use placeholder data paths and can be trained from scratch.
|
| 180 |
+
The released pretrains are provided as warm starts.
|
| 181 |
+
|
| 182 |
+
## Visual examples
|
| 183 |
+
|
| 184 |
+
Urban100 4x. Left: bicubic input. Right: released PULSE 4x pretrain.
|
| 185 |
+
|
| 186 |
+

|
| 187 |
+

|
| 188 |
+

|
| 189 |
+
|
| 190 |
+
## Files
|
| 191 |
+
|
| 192 |
+
```text
|
| 193 |
+
pulse_arch.py
|
| 194 |
+
models/
|
| 195 |
+
pulse/4x_pretrain.safetensors
|
| 196 |
+
pulse/2x_pretrain.safetensors
|
| 197 |
+
pulse_fast/4x_pretrain.safetensors
|
| 198 |
+
pulse_fast/2x_pretrain.safetensors
|
| 199 |
+
onnx/
|
| 200 |
+
*_fp16.onnx
|
| 201 |
+
*.onnx
|
| 202 |
+
configs/
|
| 203 |
+
4x_pulse_pretrain.yml
|
| 204 |
+
2x_pulse_pretrain.yml
|
| 205 |
+
4x_pulse_fast_pretrain.yml
|
| 206 |
+
2x_pulse_fast_pretrain.yml
|
| 207 |
+
PAPER.md
|
| 208 |
+
examples/
|
| 209 |
+
```
|
| 210 |
+
|
| 211 |
+
## Limitations
|
| 212 |
+
|
| 213 |
+
- Not a blind-OTF model; no degradation embedding.
|
| 214 |
+
- Not a text/UI upscaler.
|
| 215 |
+
- The clean pretrain does not clean compression artifacts; use a web/anime
|
| 216 |
+
finetune for that.
|
| 217 |
+
- No temporal/video model is included.
|
| 218 |
+
- PSNR is below HEART; that is the intended size/speed tradeoff.
|
| 219 |
+
|
| 220 |
+
## Attribution
|
| 221 |
+
|
| 222 |
+
The fixed base uses **Magic Kernel Sharp 2013** by **John Costella**
|
| 223 |
+
(https://johncostella.com/magic/). The public reference implementation
|
| 224 |
+
(`felixr/magic-kernel-sharp`, MIT) is credited in `ATTRIBUTION.md`. PULSE uses
|
| 225 |
+
an independent PyTorch implementation of the integer-scale phase kernels, not
|
| 226 |
+
a copy of the reference C code.
|
| 227 |
+
|
| 228 |
+
## License
|
| 229 |
+
|
| 230 |
+
The code and released weights are **MIT-licensed** and permissive. Keep the
|
| 231 |
+
MIT copyright and permission notice with copies or substantial portions of the
|
| 232 |
+
software. See `LICENSE`.
|
| 233 |
+
|
| 234 |
+
The released weights were trained on a curated CC0 / public-domain image set.
|
| 235 |
+
This is a provenance statement, not a guarantee; source-image rights and
|
| 236 |
+
jurisdiction-specific rules can still matter. No warranty is provided.
|
| 237 |
+
|
| 238 |
+
```bibtex
|
| 239 |
+
@misc{pulse_body_2026,
|
| 240 |
+
title = {PULSE: A Deployment-First Single-Image Super-Resolution Network},
|
| 241 |
+
author = {Philip Hofmann and contributors},
|
| 242 |
+
year = {2026},
|
| 243 |
+
note = {BODY suite}
|
| 244 |
+
}
|
| 245 |
+
```
|
| 246 |
+
|
| 247 |
+
---
|
| 248 |
+
|
| 249 |
+
---
|
| 250 |
+
|
| 251 |
+
## BODY suite benchmark — 2026-11-09
|
| 252 |
+
|
| 253 |
+
The **BODY suite** is a set of networks with different deployment goals, not one
|
| 254 |
+
network scaled up and down:
|
| 255 |
+
|
| 256 |
+
- **PULSE_fast** — smallest and fastest, 2x/4x.
|
| 257 |
+
- **PULSE** — ultralight quality step, 2x/4x.
|
| 258 |
+
- **NERVE** — lightweight ~1.8M conv-only edge model.
|
| 259 |
+
- **HEART** — heavier ~16.7M quality model for desktop/workstation GPUs.
|
| 260 |
+
|
| 261 |
+
All numbers use the latest released **CC0 / public-domain pretrains**.
|
| 262 |
+
Validation: [lucid-val-512](https://huggingface.co/datasets/Phips/lucid-val-512),
|
| 263 |
+
full images, Y-channel PSNR/SSIM with a 4px crop, RGB TopIQ/LPIPS/DISTS, EMA,
|
| 264 |
+
no TTA. All models are **4x**.
|
| 265 |
+
|
| 266 |
+
### Quality
|
| 267 |
+
|
| 268 |
+
| Model | Params | Urban100 PSNR / SSIM / TopIQ / LPIPS / DISTS | lucid-val-512 | BHI100 |
|
| 269 |
+
|---|---:|---|---|---|
|
| 270 |
+
| PULSE_fast | 162,192 | 25.4263 / 0.7640 / 0.5749 / 0.2558 / 0.1866 | 28.1909 / 0.7807 / 0.5674 / 0.2825 / 0.1886 | 23.3966 / 0.6801 / 0.4917 / 0.3472 / 0.2275 |
|
| 271 |
+
| PULSE | 353,856 | 25.6559 / 0.7719 / 0.5908 / 0.2456 / 0.1821 | 28.2892 / 0.7842 / 0.5777 / 0.2782 / 0.1859 | 23.5135 / 0.6877 / 0.5041 / 0.3363 / 0.2236 |
|
| 272 |
+
| NERVE | 1,801,920 | 26.2685 / 0.7912 / 0.6230 / 0.2259 / 0.1735 | 28.5381 / 0.7919 / 0.5933 / 0.2710 / 0.1815 | 23.8057 / 0.7052 / 0.5275 / 0.3180 / 0.2156 |
|
| 273 |
+
| HEART | 16,677,399 | **28.0458 / 0.8372 / 0.7055 / 0.1758 / 0.1499** | **29.0150 / 0.8073 / 0.6268 / 0.2484 / 0.1698** | **24.4877 / 0.7419 / 0.5820 / 0.2724 / 0.1945** |
|
| 274 |
+
|
| 275 |
+
### Training speed
|
| 276 |
+
|
| 277 |
+
1k iterations, Charbonnier only, lq96, batch 2, bf16, sequential, RTX 3060.
|
| 278 |
+
|
| 279 |
+
| Model | it/s | Peak VRAM |
|
| 280 |
+
|---|---:|---:|
|
| 281 |
+
| PULSE_fast | 107.59 | 0.05 GB |
|
| 282 |
+
| PULSE | 126.99 | 0.06 GB |
|
| 283 |
+
| NERVE | 54.49 | 0.16 GB |
|
| 284 |
+
| HEART | 1.137 | 1.03 GB |
|
| 285 |
+
|
| 286 |
+
Training is **not strictly monotonic** with size: at this tiny scale, data/launch
|
| 287 |
+
and memory overhead can outweigh the parameter difference, so PULSE is slightly
|
| 288 |
+
faster than PULSE_fast in this 1k microbenchmark even though PULSE_fast is faster
|
| 289 |
+
at inference.
|
| 290 |
+
|
| 291 |
+
### Inference — full lucid-val-512
|
| 292 |
+
|
| 293 |
+
136 images, 128×128 input → 512×512 output, batch 1, no TTA, RTX 3060.
|
| 294 |
+
ONNX is static 128² fp16 + onnxsim, ORT CUDA; TensorRT is native fp16.
|
| 295 |
+
Peak GPU memory includes the CUDA/desktop baseline (~1.3 GB).
|
| 296 |
+
|
| 297 |
+
| Model | Runtime | Total s | images/s | ms/img | Peak GPU MB |
|
| 298 |
+
|---|---|---:|---:|---:|---:|
|
| 299 |
+
| PULSE_fast | eager | 0.151 | 900.9 | 1.11 | 1316 |
|
| 300 |
+
| PULSE_fast | ORT CUDA | 0.190 | 717.6 | 1.39 | 1290 |
|
| 301 |
+
| PULSE_fast | TensorRT | **0.042** | **3249.8** | **0.308** | 1340 |
|
| 302 |
+
| PULSE | eager | 0.402 | 338.1 | 2.96 | 1336 |
|
| 303 |
+
| PULSE | ORT CUDA | 0.329 | 413.7 | 2.42 | 1298 |
|
| 304 |
+
| PULSE | TensorRT | **0.078** | **1740.5** | **0.575** | 1350 |
|
| 305 |
+
| NERVE | eager | 1.064 | 127.8 | 7.83 | 1485 |
|
| 306 |
+
| NERVE | ORT CUDA | 0.946 | 143.8 | 6.95 | 1317 |
|
| 307 |
+
| NERVE | TensorRT | **0.259** | **525.8** | **1.90** | 1374 |
|
| 308 |
+
| HEART | eager | 48.444 | 2.81 | 356.2 | 10454 |
|
| 309 |
+
| HEART | ORT CUDA | 28.706 | 4.74 | 211.1 | 5508 |
|
| 310 |
+
| HEART | TensorRT | **10.149** | **13.40** | **74.63** | 5779 |
|
| 311 |
+
|
| 312 |
+
`torch.compile` was not a win on the full lucid run for these four models and is
|
| 313 |
+
therefore omitted from the summary.
|
| 314 |
+
|
| 315 |
+
**Links:** [PULSE](https://huggingface.co/Phips/PULSE) ·
|
| 316 |
+
[NERVE](https://huggingface.co/Phips/NERVE) ·
|
| 317 |
+
[HEART](https://huggingface.co/Phips/HEART) ·
|
| 318 |
+
[lucid-val-512](https://huggingface.co/datasets/Phips/lucid-val-512)
|
checksums.txt
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
24afac4c03337316f37e1c152369eea8c2960227063970d0fdd89dab8ec3c15c ATTRIBUTION.md
|
| 2 |
+
d16d4d07fd524bf5fb633e6677adee564cfecfcbd49df1870eec946334815f50 LICENSE
|
| 3 |
+
8b20946732f997a51f8758594f85f40215a34abaea75069f156a6139e737f307 PAPER.md
|
| 4 |
+
b3ce65f754c9c60f65d59f2e67d60eac327c911e1873b0e7d8b15c15ba4e2a5a README.md
|
| 5 |
+
4aef8acc39c8a83e2fb32232320f1a82ee0b272d09507fe27b1a78c761e1e100 configs/2x_pulse_fast_pretrain.yml
|
| 6 |
+
dc4e69f32f180229fddaffbaec1110ad7235f743f598b7f0c4416c25a1f4d5a5 configs/2x_pulse_pretrain.yml
|
| 7 |
+
ab6bcfb92d4243e23a6b84d706f61a3cab0deab14a68933a6fc544e167459a79 configs/4x_pulse_fast_pretrain.yml
|
| 8 |
+
a8360fbf7d6777844f33fb49eafabf54f0d08c7e49fa26280661ee98cca2d498 configs/4x_pulse_pretrain.yml
|
| 9 |
+
0026397ecef86da75f26861275edaa73b408bba3611bfc293ba62613be1f8cc2 configs/pulse_arch.py
|
| 10 |
+
7050bea3e3449816d89dd44fb9be367e92db877313bf9952f232317387a6c06d examples/4x_pulse_urban100_img_001_input_vs_pretrain.png
|
| 11 |
+
664410e9af4ab829fbc000a51cb7cf9335fc9e4915e994603c37f9a2fcf3c962 examples/4x_pulse_urban100_img_004_input_vs_pretrain.png
|
| 12 |
+
ca6b1ff8143503b2a87f327beb6fcb60605c719aa8195e346506cc1d5c410b64 examples/4x_pulse_urban100_img_006_input_vs_pretrain.png
|
| 13 |
+
3fd891ff8113ac7caf43de985a2759d915d3a6626d50e1db04bc55865e5b5e13 models/pulse/2x_pretrain.safetensors
|
| 14 |
+
15ac6f2b958b016ad26f1a55700b6530ceeeb08feea920fdf6385029176f2980 models/pulse/4x_pretrain.safetensors
|
| 15 |
+
2957d4af0069001509ad3222a3dfbad97290b120a40916221d3cc6547097c4b4 models/pulse_fast/2x_pretrain.safetensors
|
| 16 |
+
fa9e477807eb65c7340aeadbee374213853eb3c2fe06210898bae910e6f654a8 models/pulse_fast/4x_pretrain.safetensors
|
| 17 |
+
baeeb9f310aa4dfad5f3a5f7961254270ba09256d6fb49fbcb4b2b5e63c34251 onnx/2x_pulse.onnx
|
| 18 |
+
66c8905a7671901623a1f74333a259c39e4b6238069b1fc2c64036672d8421fc onnx/2x_pulse_fast.onnx
|
| 19 |
+
22c277e7cde5d864b8a8ff41d3a74d1afc41ef08ceb4b5349ca272ad525913e8 onnx/2x_pulse_fast_fp16.onnx
|
| 20 |
+
765559129d740c989027ca28d0c7bbd12c1aadd9411aa1e44aa726950cb35352 onnx/2x_pulse_fp16.onnx
|
| 21 |
+
c616de7389281ad0a5cb9fa09c5ca13b4786f72a33ddf1535c0abefb6893a766 onnx/4x_pulse.onnx
|
| 22 |
+
b2c9703ee9b4c82278599c38f6389495944c769bc43a64221c77a3f906ef5cb5 onnx/4x_pulse_fast.onnx
|
| 23 |
+
c62531bd9f49b402234c861c6dbfa81ff29edee4b547d1522f928be0f41bf3bd onnx/4x_pulse_fast_fp16.onnx
|
| 24 |
+
185b51606ba170e9feb207c4778c75e5c21f81b4e5b22c749de9a9a1bd5dbfc3 onnx/4x_pulse_fp16.onnx
|
configs/2x_pulse_fast_pretrain.yml
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: 2x_pulse_fast_PULSE_fast_pretrain
|
| 2 |
+
scale: 2
|
| 3 |
+
use_amp: true
|
| 4 |
+
amp_bf16: true
|
| 5 |
+
use_channels_last: true
|
| 6 |
+
num_gpu: 1
|
| 7 |
+
datasets:
|
| 8 |
+
train:
|
| 9 |
+
name: CC0/public-domain train
|
| 10 |
+
type: pairedimagedataset
|
| 11 |
+
dataroot_gt:
|
| 12 |
+
- datasets/train/cc0/gt
|
| 13 |
+
dataroot_lq:
|
| 14 |
+
- datasets/train/cc0/lr_x2
|
| 15 |
+
lq_size: 256
|
| 16 |
+
use_hflip: true
|
| 17 |
+
use_rot: true
|
| 18 |
+
num_worker_per_gpu: 8
|
| 19 |
+
batch_size_per_gpu: 8
|
| 20 |
+
accum_iter: 1
|
| 21 |
+
val:
|
| 22 |
+
name: Urban100 x4
|
| 23 |
+
type: pairedimagedataset
|
| 24 |
+
dataroot_gt:
|
| 25 |
+
- datasets/val/urban100/hr
|
| 26 |
+
dataroot_lq:
|
| 27 |
+
- datasets/val/urban100/lr_x2
|
| 28 |
+
filename_tmpl: '{}x2'
|
| 29 |
+
network_g:
|
| 30 |
+
type: pulse_fast
|
| 31 |
+
path:
|
| 32 |
+
pretrain_network_g: null
|
| 33 |
+
param_key_g: null
|
| 34 |
+
strict_load_g: false
|
| 35 |
+
resume_state: null
|
| 36 |
+
train:
|
| 37 |
+
ema_decay: 0.999
|
| 38 |
+
ema_power: 0.75
|
| 39 |
+
optim_g:
|
| 40 |
+
type: AdamW
|
| 41 |
+
lr: 0.0001
|
| 42 |
+
weight_decay: 0
|
| 43 |
+
betas:
|
| 44 |
+
- 0.9
|
| 45 |
+
- 0.99
|
| 46 |
+
scheduler:
|
| 47 |
+
type: MultiStepLR
|
| 48 |
+
milestones:
|
| 49 |
+
- 100000
|
| 50 |
+
- 160000
|
| 51 |
+
gamma: 0.5
|
| 52 |
+
total_iter: 200000
|
| 53 |
+
warmup_iter: -1
|
| 54 |
+
losses:
|
| 55 |
+
- type: charbonnierloss
|
| 56 |
+
loss_weight: 1.0
|
| 57 |
+
val:
|
| 58 |
+
val_enabled: true
|
| 59 |
+
val_freq: 5000
|
| 60 |
+
save_img: false
|
| 61 |
+
tile_size: 512
|
| 62 |
+
tile_overlap: 16
|
| 63 |
+
metrics_enabled: true
|
| 64 |
+
metrics:
|
| 65 |
+
psnr:
|
| 66 |
+
type: calculate_psnr
|
| 67 |
+
crop_border: 4
|
| 68 |
+
test_y_channel: true
|
| 69 |
+
ssim:
|
| 70 |
+
type: calculate_ssim
|
| 71 |
+
crop_border: 4
|
| 72 |
+
test_y_channel: true
|
| 73 |
+
topiq:
|
| 74 |
+
type: calculate_topiq
|
| 75 |
+
lpips:
|
| 76 |
+
type: calculate_lpips
|
| 77 |
+
better: lower
|
| 78 |
+
logger:
|
| 79 |
+
print_freq: 250
|
| 80 |
+
save_checkpoint_freq: 5000
|
| 81 |
+
save_checkpoint_format: safetensors
|
| 82 |
+
use_tb_logger: true
|
configs/2x_pulse_pretrain.yml
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: 2x_pulse_PULSE_pretrain
|
| 2 |
+
scale: 2
|
| 3 |
+
use_amp: true
|
| 4 |
+
amp_bf16: true
|
| 5 |
+
use_channels_last: true
|
| 6 |
+
num_gpu: 1
|
| 7 |
+
datasets:
|
| 8 |
+
train:
|
| 9 |
+
name: CC0/public-domain train
|
| 10 |
+
type: pairedimagedataset
|
| 11 |
+
dataroot_gt:
|
| 12 |
+
- datasets/train/cc0/gt
|
| 13 |
+
dataroot_lq:
|
| 14 |
+
- datasets/train/cc0/lr_x2
|
| 15 |
+
lq_size: 256
|
| 16 |
+
use_hflip: true
|
| 17 |
+
use_rot: true
|
| 18 |
+
num_worker_per_gpu: 8
|
| 19 |
+
batch_size_per_gpu: 8
|
| 20 |
+
accum_iter: 1
|
| 21 |
+
val:
|
| 22 |
+
name: Urban100 x4
|
| 23 |
+
type: pairedimagedataset
|
| 24 |
+
dataroot_gt:
|
| 25 |
+
- datasets/val/urban100/hr
|
| 26 |
+
dataroot_lq:
|
| 27 |
+
- datasets/val/urban100/lr_x2
|
| 28 |
+
filename_tmpl: '{}x2'
|
| 29 |
+
network_g:
|
| 30 |
+
type: pulse
|
| 31 |
+
path:
|
| 32 |
+
pretrain_network_g: null
|
| 33 |
+
param_key_g: null
|
| 34 |
+
strict_load_g: false
|
| 35 |
+
resume_state: null
|
| 36 |
+
train:
|
| 37 |
+
ema_decay: 0.999
|
| 38 |
+
ema_power: 0.75
|
| 39 |
+
optim_g:
|
| 40 |
+
type: AdamW
|
| 41 |
+
lr: 0.0001
|
| 42 |
+
weight_decay: 0
|
| 43 |
+
betas:
|
| 44 |
+
- 0.9
|
| 45 |
+
- 0.99
|
| 46 |
+
scheduler:
|
| 47 |
+
type: MultiStepLR
|
| 48 |
+
milestones:
|
| 49 |
+
- 100000
|
| 50 |
+
- 160000
|
| 51 |
+
gamma: 0.5
|
| 52 |
+
total_iter: 200000
|
| 53 |
+
warmup_iter: -1
|
| 54 |
+
losses:
|
| 55 |
+
- type: charbonnierloss
|
| 56 |
+
loss_weight: 1.0
|
| 57 |
+
val:
|
| 58 |
+
val_enabled: true
|
| 59 |
+
val_freq: 5000
|
| 60 |
+
save_img: false
|
| 61 |
+
tile_size: 512
|
| 62 |
+
tile_overlap: 16
|
| 63 |
+
metrics_enabled: true
|
| 64 |
+
metrics:
|
| 65 |
+
psnr:
|
| 66 |
+
type: calculate_psnr
|
| 67 |
+
crop_border: 4
|
| 68 |
+
test_y_channel: true
|
| 69 |
+
ssim:
|
| 70 |
+
type: calculate_ssim
|
| 71 |
+
crop_border: 4
|
| 72 |
+
test_y_channel: true
|
| 73 |
+
topiq:
|
| 74 |
+
type: calculate_topiq
|
| 75 |
+
lpips:
|
| 76 |
+
type: calculate_lpips
|
| 77 |
+
better: lower
|
| 78 |
+
logger:
|
| 79 |
+
print_freq: 250
|
| 80 |
+
save_checkpoint_freq: 5000
|
| 81 |
+
save_checkpoint_format: safetensors
|
| 82 |
+
use_tb_logger: true
|
configs/4x_pulse_fast_pretrain.yml
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: 4x_pulse_fast_PULSE_fast_pretrain
|
| 2 |
+
scale: 4
|
| 3 |
+
use_amp: true
|
| 4 |
+
amp_bf16: true
|
| 5 |
+
use_channels_last: true
|
| 6 |
+
num_gpu: 1
|
| 7 |
+
datasets:
|
| 8 |
+
train:
|
| 9 |
+
name: CC0/public-domain train
|
| 10 |
+
type: pairedimagedataset
|
| 11 |
+
dataroot_gt:
|
| 12 |
+
- datasets/train/cc0/gt
|
| 13 |
+
dataroot_lq:
|
| 14 |
+
- datasets/train/cc0/lr_x4
|
| 15 |
+
lq_size: 128
|
| 16 |
+
use_hflip: true
|
| 17 |
+
use_rot: true
|
| 18 |
+
num_worker_per_gpu: 8
|
| 19 |
+
batch_size_per_gpu: 8
|
| 20 |
+
accum_iter: 1
|
| 21 |
+
val:
|
| 22 |
+
name: Urban100 x4
|
| 23 |
+
type: pairedimagedataset
|
| 24 |
+
dataroot_gt:
|
| 25 |
+
- datasets/val/urban100/hr
|
| 26 |
+
dataroot_lq:
|
| 27 |
+
- datasets/val/urban100/lr_x4
|
| 28 |
+
filename_tmpl: '{}x4'
|
| 29 |
+
network_g:
|
| 30 |
+
type: pulse_fast
|
| 31 |
+
path:
|
| 32 |
+
pretrain_network_g: null
|
| 33 |
+
param_key_g: null
|
| 34 |
+
strict_load_g: false
|
| 35 |
+
resume_state: null
|
| 36 |
+
train:
|
| 37 |
+
ema_decay: 0.999
|
| 38 |
+
ema_power: 0.75
|
| 39 |
+
optim_g:
|
| 40 |
+
type: AdamW
|
| 41 |
+
lr: 0.0001
|
| 42 |
+
weight_decay: 0
|
| 43 |
+
betas:
|
| 44 |
+
- 0.9
|
| 45 |
+
- 0.99
|
| 46 |
+
scheduler:
|
| 47 |
+
type: MultiStepLR
|
| 48 |
+
milestones:
|
| 49 |
+
- 100000
|
| 50 |
+
- 160000
|
| 51 |
+
gamma: 0.5
|
| 52 |
+
total_iter: 200000
|
| 53 |
+
warmup_iter: -1
|
| 54 |
+
losses:
|
| 55 |
+
- type: charbonnierloss
|
| 56 |
+
loss_weight: 1.0
|
| 57 |
+
val:
|
| 58 |
+
val_enabled: true
|
| 59 |
+
val_freq: 5000
|
| 60 |
+
save_img: false
|
| 61 |
+
tile_size: 512
|
| 62 |
+
tile_overlap: 16
|
| 63 |
+
metrics_enabled: true
|
| 64 |
+
metrics:
|
| 65 |
+
psnr:
|
| 66 |
+
type: calculate_psnr
|
| 67 |
+
crop_border: 4
|
| 68 |
+
test_y_channel: true
|
| 69 |
+
ssim:
|
| 70 |
+
type: calculate_ssim
|
| 71 |
+
crop_border: 4
|
| 72 |
+
test_y_channel: true
|
| 73 |
+
topiq:
|
| 74 |
+
type: calculate_topiq
|
| 75 |
+
lpips:
|
| 76 |
+
type: calculate_lpips
|
| 77 |
+
better: lower
|
| 78 |
+
logger:
|
| 79 |
+
print_freq: 250
|
| 80 |
+
save_checkpoint_freq: 5000
|
| 81 |
+
save_checkpoint_format: safetensors
|
| 82 |
+
use_tb_logger: true
|
configs/4x_pulse_pretrain.yml
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
name: 4x_pulse_PULSE_pretrain
|
| 2 |
+
scale: 4
|
| 3 |
+
use_amp: true
|
| 4 |
+
amp_bf16: true
|
| 5 |
+
use_channels_last: true
|
| 6 |
+
num_gpu: 1
|
| 7 |
+
datasets:
|
| 8 |
+
train:
|
| 9 |
+
name: CC0/public-domain train
|
| 10 |
+
type: pairedimagedataset
|
| 11 |
+
dataroot_gt:
|
| 12 |
+
- datasets/train/cc0/gt
|
| 13 |
+
dataroot_lq:
|
| 14 |
+
- datasets/train/cc0/lr_x4
|
| 15 |
+
lq_size: 128
|
| 16 |
+
use_hflip: true
|
| 17 |
+
use_rot: true
|
| 18 |
+
num_worker_per_gpu: 8
|
| 19 |
+
batch_size_per_gpu: 8
|
| 20 |
+
accum_iter: 1
|
| 21 |
+
val:
|
| 22 |
+
name: Urban100 x4
|
| 23 |
+
type: pairedimagedataset
|
| 24 |
+
dataroot_gt:
|
| 25 |
+
- datasets/val/urban100/hr
|
| 26 |
+
dataroot_lq:
|
| 27 |
+
- datasets/val/urban100/lr_x4
|
| 28 |
+
filename_tmpl: '{}x4'
|
| 29 |
+
network_g:
|
| 30 |
+
type: pulse
|
| 31 |
+
path:
|
| 32 |
+
pretrain_network_g: null
|
| 33 |
+
param_key_g: null
|
| 34 |
+
strict_load_g: false
|
| 35 |
+
resume_state: null
|
| 36 |
+
train:
|
| 37 |
+
ema_decay: 0.999
|
| 38 |
+
ema_power: 0.75
|
| 39 |
+
optim_g:
|
| 40 |
+
type: AdamW
|
| 41 |
+
lr: 0.0001
|
| 42 |
+
weight_decay: 0
|
| 43 |
+
betas:
|
| 44 |
+
- 0.9
|
| 45 |
+
- 0.99
|
| 46 |
+
scheduler:
|
| 47 |
+
type: MultiStepLR
|
| 48 |
+
milestones:
|
| 49 |
+
- 100000
|
| 50 |
+
- 160000
|
| 51 |
+
gamma: 0.5
|
| 52 |
+
total_iter: 200000
|
| 53 |
+
warmup_iter: -1
|
| 54 |
+
losses:
|
| 55 |
+
- type: charbonnierloss
|
| 56 |
+
loss_weight: 1.0
|
| 57 |
+
val:
|
| 58 |
+
val_enabled: true
|
| 59 |
+
val_freq: 5000
|
| 60 |
+
save_img: false
|
| 61 |
+
tile_size: 512
|
| 62 |
+
tile_overlap: 16
|
| 63 |
+
metrics_enabled: true
|
| 64 |
+
metrics:
|
| 65 |
+
psnr:
|
| 66 |
+
type: calculate_psnr
|
| 67 |
+
crop_border: 4
|
| 68 |
+
test_y_channel: true
|
| 69 |
+
ssim:
|
| 70 |
+
type: calculate_ssim
|
| 71 |
+
crop_border: 4
|
| 72 |
+
test_y_channel: true
|
| 73 |
+
topiq:
|
| 74 |
+
type: calculate_topiq
|
| 75 |
+
lpips:
|
| 76 |
+
type: calculate_lpips
|
| 77 |
+
better: lower
|
| 78 |
+
logger:
|
| 79 |
+
print_freq: 250
|
| 80 |
+
save_checkpoint_freq: 5000
|
| 81 |
+
save_checkpoint_format: safetensors
|
| 82 |
+
use_tb_logger: true
|
configs/pulse_arch.py
ADDED
|
@@ -0,0 +1,165 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""PULSE — fast, deployable single-image super-resolution.
|
| 2 |
+
|
| 3 |
+
A ~162K parameter SISR network (354K at dim=48). The body is a plain conv
|
| 4 |
+
residual stack; the base image is a fixed Magic Kernel Sharp 2013 resampler
|
| 5 |
+
implemented as a phase-kernel grouped convolution plus PixelShuffle. No
|
| 6 |
+
normalization, no attention, no custom ops: the exported graph is
|
| 7 |
+
Pad / Conv / Add / Relu / DepthToSpace, which is friendly to ONNX Runtime,
|
| 8 |
+
TensorRT, and NCNN.
|
| 9 |
+
"""
|
| 10 |
+
|
| 11 |
+
from __future__ import annotations
|
| 12 |
+
|
| 13 |
+
import torch
|
| 14 |
+
from spandrel.util import store_hyperparameters
|
| 15 |
+
from torch import Tensor, nn
|
| 16 |
+
from torch.nn import functional as F # noqa: N812
|
| 17 |
+
|
| 18 |
+
from traiNNer.utils.icnr import icnr_init
|
| 19 |
+
from traiNNer.utils.registry import ARCH_REGISTRY, SPANDREL_REGISTRY
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _mks_1d(x: Tensor) -> Tensor:
|
| 23 |
+
"""Magic Kernel Sharp 2013, one dimension, support 2.5."""
|
| 24 |
+
x = x.abs()
|
| 25 |
+
return torch.where(
|
| 26 |
+
x <= 0.5,
|
| 27 |
+
17 / 16 - (7 / 4) * x**2,
|
| 28 |
+
torch.where(
|
| 29 |
+
x <= 1.5,
|
| 30 |
+
0.25 * (4 * x**2 - 11 * x + 7),
|
| 31 |
+
torch.where(x <= 2.5, -0.125 * (x - 2.5) ** 2, torch.zeros_like(x)),
|
| 32 |
+
),
|
| 33 |
+
)
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
class MKS2013Base(nn.Module):
|
| 37 |
+
"""Fixed Magic Kernel Sharp 2013 integer upsampler.
|
| 38 |
+
|
| 39 |
+
For an integer scale, each output phase has a fixed 5x5 separable kernel.
|
| 40 |
+
Storing the kernels as a grouped convolution and using PixelShuffle gives
|
| 41 |
+
the exact integer-scale kernel with standard Conv / DepthToSpace ops and no
|
| 42 |
+
Resize. Borders use replicate padding rather than Costella's weight
|
| 43 |
+
renormalization, and the kernel is applied in the input color space (sRGB
|
| 44 |
+
for images), not the linear-light space used by the reference.
|
| 45 |
+
"""
|
| 46 |
+
|
| 47 |
+
def __init__(self, channels: int, upscale: int) -> None:
|
| 48 |
+
super().__init__()
|
| 49 |
+
self.channels = channels
|
| 50 |
+
self.upscale = upscale
|
| 51 |
+
if upscale == 1:
|
| 52 |
+
return
|
| 53 |
+
|
| 54 |
+
# One-time constant kernels; no gradients and no state-dict payload.
|
| 55 |
+
offsets = torch.arange(-2, 3, dtype=torch.float32)
|
| 56 |
+
weight = torch.empty(channels * upscale**2, 1, 5, 5)
|
| 57 |
+
for c in range(channels):
|
| 58 |
+
for py in range(upscale):
|
| 59 |
+
for px in range(upscale):
|
| 60 |
+
fy = (py + 0.5) / upscale - 0.5
|
| 61 |
+
fx = (px + 0.5) / upscale - 0.5
|
| 62 |
+
wy = _mks_1d(fy - offsets)
|
| 63 |
+
wx = _mks_1d(fx - offsets)
|
| 64 |
+
weight[c * upscale**2 + py * upscale + px, 0] = torch.outer(wy, wx)
|
| 65 |
+
self.register_buffer("weight", weight, persistent=False)
|
| 66 |
+
self.shuffle = nn.PixelShuffle(upscale)
|
| 67 |
+
|
| 68 |
+
def forward(self, x: Tensor) -> Tensor:
|
| 69 |
+
if self.upscale == 1:
|
| 70 |
+
return x
|
| 71 |
+
x = F.pad(x, (2, 2, 2, 2), mode="replicate")
|
| 72 |
+
y = F.conv2d(x, self.get_buffer("weight"), groups=self.channels)
|
| 73 |
+
return self.shuffle(y)
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
class PulseBlock(nn.Module):
|
| 77 |
+
"""Conv-ReLU-Conv residual block. No normalization, no gating."""
|
| 78 |
+
|
| 79 |
+
def __init__(self, dim: int) -> None:
|
| 80 |
+
super().__init__()
|
| 81 |
+
self.conv1 = nn.Conv2d(dim, dim, 3, padding=1, bias=False)
|
| 82 |
+
self.conv2 = nn.Conv2d(dim, dim, 3, padding=1, bias=False)
|
| 83 |
+
self.act = nn.ReLU(inplace=True)
|
| 84 |
+
|
| 85 |
+
def forward(self, x: Tensor) -> Tensor:
|
| 86 |
+
return x + self.conv2(self.act(self.conv1(x)))
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
@store_hyperparameters()
|
| 90 |
+
class PULSE(nn.Module):
|
| 91 |
+
"""MKS2013 base + residual convs + PixelShuffle correction."""
|
| 92 |
+
|
| 93 |
+
hyperparameters = {} # noqa: RUF012
|
| 94 |
+
|
| 95 |
+
def __init__(
|
| 96 |
+
self,
|
| 97 |
+
*,
|
| 98 |
+
in_ch: int = 3,
|
| 99 |
+
out_ch: int = 3,
|
| 100 |
+
dim: int = 48,
|
| 101 |
+
n_blocks: int = 8,
|
| 102 |
+
upscale: int = 4,
|
| 103 |
+
) -> None:
|
| 104 |
+
super().__init__()
|
| 105 |
+
# The MKS base is added directly to the output, so channels must match.
|
| 106 |
+
if in_ch != out_ch:
|
| 107 |
+
raise ValueError("PULSE requires in_ch == out_ch for its base residual.")
|
| 108 |
+
|
| 109 |
+
self.upscale = upscale
|
| 110 |
+
self.base = MKS2013Base(in_ch, upscale)
|
| 111 |
+
self.stem = nn.Conv2d(in_ch, dim, 3, padding=1, bias=False)
|
| 112 |
+
self.body = nn.Sequential(*[PulseBlock(dim) for _ in range(n_blocks)])
|
| 113 |
+
# Head bias is intentional: body convs are bias-free and the head
|
| 114 |
+
# learns any residual color shift.
|
| 115 |
+
self.head = nn.Sequential(
|
| 116 |
+
nn.Conv2d(dim, out_ch * upscale**2, 3, padding=1, bias=True),
|
| 117 |
+
nn.PixelShuffle(upscale),
|
| 118 |
+
)
|
| 119 |
+
self._initialize_weights()
|
| 120 |
+
|
| 121 |
+
def _initialize_weights(self) -> None:
|
| 122 |
+
for m in self.modules():
|
| 123 |
+
if isinstance(m, nn.Conv2d):
|
| 124 |
+
nn.init.kaiming_normal_(m.weight, mode="fan_in", nonlinearity="relu")
|
| 125 |
+
if m.bias is not None:
|
| 126 |
+
nn.init.zeros_(m.bias)
|
| 127 |
+
|
| 128 |
+
# ICNR removes phase bias from the PixelShuffle head; the tiny scale
|
| 129 |
+
# keeps the initial output close to the MKS base.
|
| 130 |
+
head = self.head[0]
|
| 131 |
+
assert isinstance(head, nn.Conv2d)
|
| 132 |
+
icnr_init(head, self.upscale)
|
| 133 |
+
with torch.no_grad():
|
| 134 |
+
head.weight.mul_(1e-3)
|
| 135 |
+
if head.bias is not None:
|
| 136 |
+
head.bias.zero_()
|
| 137 |
+
|
| 138 |
+
def forward(self, x: Tensor) -> Tensor:
|
| 139 |
+
base = self.base(x)
|
| 140 |
+
feat = self.stem(x)
|
| 141 |
+
# Direct stem-to-head path; the blocks are already residual.
|
| 142 |
+
feat = feat + self.body(feat)
|
| 143 |
+
return base + self.head(feat)
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
@SPANDREL_REGISTRY.register()
|
| 147 |
+
@ARCH_REGISTRY.register()
|
| 148 |
+
def pulse(
|
| 149 |
+
scale: int = 4,
|
| 150 |
+
*,
|
| 151 |
+
in_ch: int = 3,
|
| 152 |
+
out_ch: int = 3,
|
| 153 |
+
) -> PULSE:
|
| 154 |
+
return PULSE(in_ch=in_ch, out_ch=out_ch, dim=48, n_blocks=8, upscale=scale)
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
@SPANDREL_REGISTRY.register()
|
| 158 |
+
@ARCH_REGISTRY.register()
|
| 159 |
+
def pulse_fast(
|
| 160 |
+
scale: int = 4,
|
| 161 |
+
*,
|
| 162 |
+
in_ch: int = 3,
|
| 163 |
+
out_ch: int = 3,
|
| 164 |
+
) -> PULSE:
|
| 165 |
+
return PULSE(in_ch=in_ch, out_ch=out_ch, dim=32, n_blocks=8, upscale=scale)
|
examples/4x_pulse_urban100_img_001_input_vs_pretrain.png
ADDED
|
Git LFS Details
|
examples/4x_pulse_urban100_img_004_input_vs_pretrain.png
ADDED
|
Git LFS Details
|
examples/4x_pulse_urban100_img_006_input_vs_pretrain.png
ADDED
|
Git LFS Details
|
models/pulse/2x_pretrain.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3fd891ff8113ac7caf43de985a2759d915d3a6626d50e1db04bc55865e5b5e13
|
| 3 |
+
size 1354856
|
models/pulse/4x_pretrain.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:15ac6f2b958b016ad26f1a55700b6530ceeeb08feea920fdf6385029176f2980
|
| 3 |
+
size 1417208
|
models/pulse_fast/2x_pretrain.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2957d4af0069001509ad3222a3dfbad97290b120a40916221d3cc6547097c4b4
|
| 3 |
+
size 608920
|
models/pulse_fast/4x_pretrain.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fa9e477807eb65c7340aeadbee374213853eb3c2fe06210898bae910e6f654a8
|
| 3 |
+
size 650536
|
onnx/2x_pulse.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:baeeb9f310aa4dfad5f3a5f7961254270ba09256d6fb49fbcb4b2b5e63c34251
|
| 3 |
+
size 1361094
|
onnx/2x_pulse_fast.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:66c8905a7671901623a1f74333a259c39e4b6238069b1fc2c64036672d8421fc
|
| 3 |
+
size 615172
|
onnx/2x_pulse_fast_fp16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:22c277e7cde5d864b8a8ff41d3a74d1afc41ef08ceb4b5349ca272ad525913e8
|
| 3 |
+
size 311496
|
onnx/2x_pulse_fp16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:765559129d740c989027ca28d0c7bbd12c1aadd9411aa1e44aa726950cb35352
|
| 3 |
+
size 684456
|
onnx/4x_pulse.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c616de7389281ad0a5cb9fa09c5ca13b4786f72a33ddf1535c0abefb6893a766
|
| 3 |
+
size 1427048
|
onnx/4x_pulse_fast.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b2c9703ee9b4c82278599c38f6389495944c769bc43a64221c77a3f906ef5cb5
|
| 3 |
+
size 660392
|
onnx/4x_pulse_fast_fp16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c62531bd9f49b402234c861c6dbfa81ff29edee4b547d1522f928be0f41bf3bd
|
| 3 |
+
size 334106
|
onnx/4x_pulse_fp16.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:185b51606ba170e9feb207c4778c75e5c21f81b4e5b22c749de9a9a1bd5dbfc3
|
| 3 |
+
size 717434
|