Spaces:
Sleeping
Sleeping
Commit ·
819311d
1
Parent(s): f5e3ffa
Add 4x super-resolution upscaling stage (FR-4, FR-5, ID-2, CR-2)
Browse filespostprocess/upscale/: 4xLSDIRCompact (SRVGGNetCompact, num_conv=16) tiled CPU upscaling,
vendored from the reference project (inference structure adapted from xinntao/Real-ESRGAN,
BSD-3-Clause). Runs after interpolation and outside the @spaces.GPU call, so it never bids
for ZeroGPU time. Wired into app.py's "Upscale 4x" checkbox.
Also adds .gitignore for __pycache__/ and the RIFE weights downloaded at runtime.
Closes #6
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- .gitignore +5 -0
- LICENSES/4xLSDIRCompact-LICENSE +12 -0
- LICENSES/REAL-ESRGAN-LICENSE +35 -0
- README.md +26 -0
- app.py +22 -0
- postprocess/upscale/__init__.py +3 -0
- postprocess/upscale/srvgg_arch.py +68 -0
- postprocess/upscale/upscale.py +118 -0
.gitignore
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
__pycache__/
|
| 2 |
+
*.pyc
|
| 3 |
+
train_log/
|
| 4 |
+
RIFEv4.26_0921.zip
|
| 5 |
+
flagged/
|
LICENSES/4xLSDIRCompact-LICENSE
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Creative Commons Attribution 4.0 International (CC BY 4.0)
|
| 2 |
+
|
| 3 |
+
Full license text: https://creativecommons.org/licenses/by/4.0/legalcode
|
| 4 |
+
Human-readable summary: https://creativecommons.org/licenses/by/4.0/
|
| 5 |
+
|
| 6 |
+
---
|
| 7 |
+
Applies to: the model weights downloaded at runtime from
|
| 8 |
+
https://huggingface.co/Phips/4xLSDIRCompact (postprocess/upscale/upscale.py, WEIGHTS_URL).
|
| 9 |
+
|
| 10 |
+
Attribution: "4xLSDIRCompact" by Philip Hofmann (https://github.com/Phhofm/models), licensed under
|
| 11 |
+
CC BY 4.0. Architecture: SRVGGNetCompact, trained on the LSDIR dataset. No modifications were made to
|
| 12 |
+
the weights; they are loaded as published.
|
LICENSES/REAL-ESRGAN-LICENSE
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
BSD 3-Clause License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2021, Xintao Wang
|
| 4 |
+
All rights reserved.
|
| 5 |
+
|
| 6 |
+
Redistribution and use in source and binary forms, with or without
|
| 7 |
+
modification, are permitted provided that the following conditions are met:
|
| 8 |
+
|
| 9 |
+
1. Redistributions of source code must retain the above copyright notice, this
|
| 10 |
+
list of conditions and the following disclaimer.
|
| 11 |
+
|
| 12 |
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
| 13 |
+
this list of conditions and the following disclaimer in the documentation
|
| 14 |
+
and/or other materials provided with the distribution.
|
| 15 |
+
|
| 16 |
+
3. Neither the name of the copyright holder nor the names of its
|
| 17 |
+
contributors may be used to endorse or promote products derived from
|
| 18 |
+
this software without specific prior written permission.
|
| 19 |
+
|
| 20 |
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
| 21 |
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
| 22 |
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
| 23 |
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
| 24 |
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
| 25 |
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
| 26 |
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
| 27 |
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
| 28 |
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
| 29 |
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
| 30 |
+
|
| 31 |
+
---
|
| 32 |
+
Applies to: postprocess/upscale/srvgg_arch.py (vendored from xinntao/Real-ESRGAN,
|
| 33 |
+
https://github.com/xinntao/Real-ESRGAN, realesrgan/archs/srvgg_arch.py) and postprocess/upscale/upscale.py
|
| 34 |
+
(inference structure adapted from realesrgan/utils.py's RealESRGANer). The model weights themselves are no
|
| 35 |
+
longer from this project — see LICENSES/4xLSDIRCompact-LICENSE.
|
README.md
CHANGED
|
@@ -17,7 +17,33 @@ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-
|
|
| 17 |
Image-to-video generation from [`thornmaze/WAMU_v3_WAN2.2_I2V_LIGHTNING`](https://huggingface.co/thornmaze/WAMU_v3_WAN2.2_I2V_LIGHTNING)
|
| 18 |
via `WanImageToVideoPipeline`, running on Hugging Face ZeroGPU.
|
| 19 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
## Status
|
| 21 |
|
| 22 |
Baseline (non-AOT) pipeline. AOT-compiled inference (faster, but requires a compatible
|
| 23 |
precompiled package for WAMU_v3) is tracked separately — see the repo's open issues.
|
|
|
|
|
|
|
|
|
| 17 |
Image-to-video generation from [`thornmaze/WAMU_v3_WAN2.2_I2V_LIGHTNING`](https://huggingface.co/thornmaze/WAMU_v3_WAN2.2_I2V_LIGHTNING)
|
| 18 |
via `WanImageToVideoPipeline`, running on Hugging Face ZeroGPU.
|
| 19 |
|
| 20 |
+
## Post-processing
|
| 21 |
+
|
| 22 |
+
Two optional steps can run after generation:
|
| 23 |
+
|
| 24 |
+
| step | what | runs on |
|
| 25 |
+
|---|---|---|
|
| 26 |
+
| Frame interpolation ("Video Fluidity") | RIFE v4.26 (`thornmaze/RIFE`), half precision. 2x/4x/8x the native 16 fps. | GPU, inside the generation call |
|
| 27 |
+
| Upscaling ("Upscale 4×") | `4xLSDIRCompact` (SRVGGNetCompact, num_conv=16), forced fp32, tiled to bound peak CPU memory. Fixed 4x. | CPU, after the GPU call returns |
|
| 28 |
+
|
| 29 |
+
Upscaling runs after interpolation and outside the `@spaces.GPU` call, so it never bids for or consumes
|
| 30 |
+
ZeroGPU worker time — only wall-clock time on the Space's regular CPU.
|
| 31 |
+
|
| 32 |
+
RIFE support code (`model/warplayer.py`, `model/loss.py`, `model/pytorch_msssim/`) is vendored from
|
| 33 |
+
[`hzwer/Practical-RIFE`](https://github.com/hzwer/Practical-RIFE) (MIT, see
|
| 34 |
+
[`LICENSES/RIFE-LICENSE`](LICENSES/RIFE-LICENSE)) — required as sibling-import targets for the
|
| 35 |
+
`train_log/RIFE_HDv3.py` module downloaded at runtime from `thornmaze/RIFE`.
|
| 36 |
+
|
| 37 |
+
The upscaling *code* (`postprocess/upscale/`) is vendored/adapted from
|
| 38 |
+
[`xinntao/Real-ESRGAN`](https://github.com/xinntao/Real-ESRGAN) (BSD-3-Clause License, see
|
| 39 |
+
[`LICENSES/REAL-ESRGAN-LICENSE`](LICENSES/REAL-ESRGAN-LICENSE)). The *weights* are
|
| 40 |
+
[`Phips/4xLSDIRCompact`](https://huggingface.co/Phips/4xLSDIRCompact) (CC BY 4.0, see
|
| 41 |
+
[`LICENSES/4xLSDIRCompact-LICENSE`](LICENSES/4xLSDIRCompact-LICENSE)), not Real-ESRGAN's own — see the
|
| 42 |
+
comment at the top of `postprocess/upscale/upscale.py` for why.
|
| 43 |
+
|
| 44 |
## Status
|
| 45 |
|
| 46 |
Baseline (non-AOT) pipeline. AOT-compiled inference (faster, but requires a compatible
|
| 47 |
precompiled package for WAMU_v3) is tracked separately — see the repo's open issues.
|
| 48 |
+
|
| 49 |
+
LoRA loading is out of scope for this version (see SRS FR-8/C-4).
|
app.py
CHANGED
|
@@ -6,6 +6,7 @@ import numpy as np
|
|
| 6 |
import spaces
|
| 7 |
import torch
|
| 8 |
from diffusers.utils.export_utils import export_to_video
|
|
|
|
| 9 |
|
| 10 |
from model.pipeline import (
|
| 11 |
DEFAULT_NEGATIVE_PROMPT,
|
|
@@ -92,6 +93,7 @@ def generate_video(
|
|
| 92 |
seed,
|
| 93 |
randomize_seed,
|
| 94 |
frame_multiplier,
|
|
|
|
| 95 |
progress=gr.Progress(track_tqdm=True),
|
| 96 |
):
|
| 97 |
if input_image is None:
|
|
@@ -113,6 +115,19 @@ def generate_video(
|
|
| 113 |
progress,
|
| 114 |
)
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tmpfile:
|
| 117 |
video_path = tmpfile.name
|
| 118 |
export_to_video(final_frames, video_path, fps=final_fps, quality=6)
|
|
@@ -137,6 +152,12 @@ with gr.Blocks() as demo:
|
|
| 137 |
label="Video Fluidity (Frames per Second)",
|
| 138 |
info="Extra frames are generated with RIFE frame interpolation to smooth motion.",
|
| 139 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 140 |
with gr.Accordion("Advanced Settings", open=False):
|
| 141 |
negative_prompt_input = gr.Textbox(label="Negative Prompt", value=DEFAULT_NEGATIVE_PROMPT, lines=3)
|
| 142 |
steps_slider = gr.Slider(minimum=1, maximum=30, step=1, value=6, label="Inference Steps")
|
|
@@ -160,6 +181,7 @@ with gr.Blocks() as demo:
|
|
| 160 |
seed_input,
|
| 161 |
randomize_seed_checkbox,
|
| 162 |
frame_multiplier_input,
|
|
|
|
| 163 |
],
|
| 164 |
outputs=[video_output, file_output, seed_input],
|
| 165 |
)
|
|
|
|
| 6 |
import spaces
|
| 7 |
import torch
|
| 8 |
from diffusers.utils.export_utils import export_to_video
|
| 9 |
+
from PIL import Image
|
| 10 |
|
| 11 |
from model.pipeline import (
|
| 12 |
DEFAULT_NEGATIVE_PROMPT,
|
|
|
|
| 93 |
seed,
|
| 94 |
randomize_seed,
|
| 95 |
frame_multiplier,
|
| 96 |
+
upscale_output,
|
| 97 |
progress=gr.Progress(track_tqdm=True),
|
| 98 |
):
|
| 99 |
if input_image is None:
|
|
|
|
| 115 |
progress,
|
| 116 |
)
|
| 117 |
|
| 118 |
+
# CPU-only post-processing: outside the @spaces.GPU call, so it never bids for ZeroGPU time.
|
| 119 |
+
if upscale_output:
|
| 120 |
+
from postprocess.upscale import upscale_frames
|
| 121 |
+
|
| 122 |
+
def _report_upscale_progress(done: int, total: int) -> None:
|
| 123 |
+
progress(0.95 + 0.04 * (done / total), desc=f"Upscaling 4x on CPU ({done}/{total} frames)...")
|
| 124 |
+
|
| 125 |
+
progress(0.95, desc=f"Upscaling 4x on CPU (0/{len(final_frames)} frames)...")
|
| 126 |
+
pil_frames = [
|
| 127 |
+
Image.fromarray((np.asarray(frame) * 255.0).round().astype(np.uint8)) for frame in final_frames
|
| 128 |
+
]
|
| 129 |
+
final_frames = upscale_frames(pil_frames, progress_callback=_report_upscale_progress)
|
| 130 |
+
|
| 131 |
with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tmpfile:
|
| 132 |
video_path = tmpfile.name
|
| 133 |
export_to_video(final_frames, video_path, fps=final_fps, quality=6)
|
|
|
|
| 152 |
label="Video Fluidity (Frames per Second)",
|
| 153 |
info="Extra frames are generated with RIFE frame interpolation to smooth motion.",
|
| 154 |
)
|
| 155 |
+
upscale_checkbox = gr.Checkbox(
|
| 156 |
+
label="Upscale 4× (4xLSDIRCompact)",
|
| 157 |
+
value=False,
|
| 158 |
+
info="Upscales the finished video 4x on CPU after generation. Runs after frame "
|
| 159 |
+
"interpolation and doesn't use GPU quota, but adds wall-clock time.",
|
| 160 |
+
)
|
| 161 |
with gr.Accordion("Advanced Settings", open=False):
|
| 162 |
negative_prompt_input = gr.Textbox(label="Negative Prompt", value=DEFAULT_NEGATIVE_PROMPT, lines=3)
|
| 163 |
steps_slider = gr.Slider(minimum=1, maximum=30, step=1, value=6, label="Inference Steps")
|
|
|
|
| 181 |
seed_input,
|
| 182 |
randomize_seed_checkbox,
|
| 183 |
frame_multiplier_input,
|
| 184 |
+
upscale_checkbox,
|
| 185 |
],
|
| 186 |
outputs=[video_output, file_output, seed_input],
|
| 187 |
)
|
postprocess/upscale/__init__.py
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from .upscale import upscale_frames
|
| 2 |
+
|
| 3 |
+
__all__ = ["upscale_frames"]
|
postprocess/upscale/srvgg_arch.py
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""SRVGGNetCompact, the architecture behind `realesr-general-x4v3`. Vendored from xinntao/Real-ESRGAN
|
| 2 |
+
(BSD-3-Clause License, see `LICENSES/REAL-ESRGAN-LICENSE`), `realesrgan/archs/srvgg_arch.py`, with the
|
| 3 |
+
`basicsr.utils.registry.ARCH_REGISTRY` import and decorator dropped — that registry only matters for basicsr's
|
| 4 |
+
config-driven model zoo, and this Space instantiates the class directly. Dropping it also means this module has no
|
| 5 |
+
`basicsr` dependency, which matters: `basicsr`'s own imports break on current torchvision (its
|
| 6 |
+
`degradations.py` imports `torchvision.transforms.functional_tensor`, removed upstream) — the reason issue #1
|
| 7 |
+
chose the compact architecture over pulling in the full `realesrgan` package.
|
| 8 |
+
"""
|
| 9 |
+
|
| 10 |
+
from torch import nn as nn
|
| 11 |
+
from torch.nn import functional as F
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
class SRVGGNetCompact(nn.Module):
|
| 15 |
+
"""A compact VGG-style network structure for super-resolution.
|
| 16 |
+
|
| 17 |
+
It is a compact network structure, which performs upsampling in the last layer and no convolution is
|
| 18 |
+
conducted on the HR feature space.
|
| 19 |
+
|
| 20 |
+
Args:
|
| 21 |
+
num_in_ch (int): Channel number of inputs. Default: 3.
|
| 22 |
+
num_out_ch (int): Channel number of outputs. Default: 3.
|
| 23 |
+
num_feat (int): Channel number of intermediate features. Default: 64.
|
| 24 |
+
num_conv (int): Number of convolution layers in the body network. Default: 16.
|
| 25 |
+
upscale (int): Upsampling factor. Default: 4.
|
| 26 |
+
act_type (str): Activation type, options: 'relu', 'prelu', 'leakyrelu'. Default: prelu.
|
| 27 |
+
"""
|
| 28 |
+
|
| 29 |
+
def __init__(self, num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=16, upscale=4, act_type='prelu'):
|
| 30 |
+
super(SRVGGNetCompact, self).__init__()
|
| 31 |
+
self.num_in_ch = num_in_ch
|
| 32 |
+
self.num_out_ch = num_out_ch
|
| 33 |
+
self.num_feat = num_feat
|
| 34 |
+
self.num_conv = num_conv
|
| 35 |
+
self.upscale = upscale
|
| 36 |
+
self.act_type = act_type
|
| 37 |
+
|
| 38 |
+
self.body = nn.ModuleList()
|
| 39 |
+
self.body.append(nn.Conv2d(num_in_ch, num_feat, 3, 1, 1))
|
| 40 |
+
self.body.append(self._activation(act_type, num_feat))
|
| 41 |
+
|
| 42 |
+
for _ in range(num_conv):
|
| 43 |
+
self.body.append(nn.Conv2d(num_feat, num_feat, 3, 1, 1))
|
| 44 |
+
self.body.append(self._activation(act_type, num_feat))
|
| 45 |
+
|
| 46 |
+
self.body.append(nn.Conv2d(num_feat, num_out_ch * upscale * upscale, 3, 1, 1))
|
| 47 |
+
self.upsampler = nn.PixelShuffle(upscale)
|
| 48 |
+
|
| 49 |
+
@staticmethod
|
| 50 |
+
def _activation(act_type, num_feat):
|
| 51 |
+
if act_type == 'relu':
|
| 52 |
+
return nn.ReLU(inplace=True)
|
| 53 |
+
if act_type == 'prelu':
|
| 54 |
+
return nn.PReLU(num_parameters=num_feat)
|
| 55 |
+
if act_type == 'leakyrelu':
|
| 56 |
+
return nn.LeakyReLU(negative_slope=0.1, inplace=True)
|
| 57 |
+
raise ValueError(f"Unsupported act_type {act_type!r}")
|
| 58 |
+
|
| 59 |
+
def forward(self, x):
|
| 60 |
+
out = x
|
| 61 |
+
for i in range(len(self.body)):
|
| 62 |
+
out = self.body[i](out)
|
| 63 |
+
|
| 64 |
+
out = self.upsampler(out)
|
| 65 |
+
# add the nearest upsampled image, so that the network learns the residual
|
| 66 |
+
base = F.interpolate(x, scale_factor=self.upscale, mode='nearest')
|
| 67 |
+
out += base
|
| 68 |
+
return out
|
postprocess/upscale/upscale.py
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""`4xLSDIRCompact` tiled super-resolution, CPU-only, forced fp32.
|
| 2 |
+
|
| 3 |
+
Inference structure (pre-pad, tile, merge, crop) is adapted from xinntao/Real-ESRGAN's `RealESRGANer`
|
| 4 |
+
(BSD-3-Clause License, see `LICENSES/REAL-ESRGAN-LICENSE`, `realesrgan/utils.py`), trimmed to the single code path
|
| 5 |
+
this Space needs: one fixed compact model, RGB `PIL.Image` in and out (no alpha channel, no 16-bit, no cv2/BGR
|
| 6 |
+
round-trip, no `dni` model blending, no CLI prefetch/IO-queue threads — none of those apply to frames coming
|
| 7 |
+
straight out of the Wan 2.2 decode step).
|
| 8 |
+
|
| 9 |
+
The weights are `4xLSDIRCompact` (Philip Hofmann, CC BY 4.0, see `LICENSES/4xLSDIRCompact-LICENSE`) rather than
|
| 10 |
+
Real-ESRGAN's own `realesr-general-x4v3`: same `SRVGGNetCompact` architecture (just `num_conv=16` instead of 32,
|
| 11 |
+
so if anything it's cheaper per tile), but trained on clean LSDIR photos with no synthetic noise/blur/JPEG
|
| 12 |
+
augmentation. `realesr-general-x4v3` bakes in fairly aggressive denoising to cope with that augmentation, which
|
| 13 |
+
flattens real skin/fabric/hair texture into a smooth, CG-like look on frames that were never actually degraded
|
| 14 |
+
(e.g. straight out of a video decode step, as here) — `4xLSDIRCompact` doesn't have that bias.
|
| 15 |
+
|
| 16 |
+
No `@spaces.GPU` here: this runs on the Space's regular CPU after generation returns, so it never bids for or
|
| 17 |
+
consumes paid ZeroGPU worker time.
|
| 18 |
+
"""
|
| 19 |
+
|
| 20 |
+
from __future__ import annotations
|
| 21 |
+
|
| 22 |
+
import math
|
| 23 |
+
from functools import lru_cache
|
| 24 |
+
from pathlib import Path
|
| 25 |
+
from typing import Callable
|
| 26 |
+
|
| 27 |
+
import numpy as np
|
| 28 |
+
import torch
|
| 29 |
+
import torch.nn.functional as F
|
| 30 |
+
from PIL import Image
|
| 31 |
+
from safetensors.torch import load_file
|
| 32 |
+
from torch.hub import download_url_to_file, get_dir
|
| 33 |
+
|
| 34 |
+
from .srvgg_arch import SRVGGNetCompact
|
| 35 |
+
|
| 36 |
+
WEIGHTS_URL = "https://huggingface.co/Phips/4xLSDIRCompact/resolve/main/4xLSDIRCompact.safetensors"
|
| 37 |
+
SCALE = 4
|
| 38 |
+
NUM_CONV = 16
|
| 39 |
+
TILE_SIZE = 256
|
| 40 |
+
TILE_PAD = 10
|
| 41 |
+
device = torch.device("cpu")
|
| 42 |
+
|
| 43 |
+
ProgressCallback = Callable[[int, int], None]
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def _download_weights() -> Path:
|
| 47 |
+
cache_dir = Path(get_dir()) / "checkpoints"
|
| 48 |
+
cache_dir.mkdir(parents=True, exist_ok=True)
|
| 49 |
+
weights_path = cache_dir / Path(WEIGHTS_URL).name
|
| 50 |
+
if not weights_path.exists():
|
| 51 |
+
download_url_to_file(WEIGHTS_URL, str(weights_path))
|
| 52 |
+
return weights_path
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
@lru_cache(maxsize=1)
|
| 56 |
+
def _load_model() -> SRVGGNetCompact:
|
| 57 |
+
model = SRVGGNetCompact(
|
| 58 |
+
num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=NUM_CONV, upscale=SCALE, act_type="prelu"
|
| 59 |
+
)
|
| 60 |
+
state_dict = load_file(_download_weights())
|
| 61 |
+
model.load_state_dict(state_dict, strict=True)
|
| 62 |
+
model.to(device).float().eval()
|
| 63 |
+
return model
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def _pre_pad(tensor: torch.Tensor, pad: int) -> torch.Tensor:
|
| 67 |
+
return F.pad(tensor, (0, pad, 0, pad), mode="reflect")
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
@torch.no_grad()
|
| 71 |
+
def _tile_process(model: SRVGGNetCompact, img: torch.Tensor) -> torch.Tensor:
|
| 72 |
+
batch, channel, height, width = img.shape
|
| 73 |
+
output = img.new_zeros((batch, channel, height * SCALE, width * SCALE))
|
| 74 |
+
tiles_x = math.ceil(width / TILE_SIZE)
|
| 75 |
+
tiles_y = math.ceil(height / TILE_SIZE)
|
| 76 |
+
|
| 77 |
+
for y in range(tiles_y):
|
| 78 |
+
for x in range(tiles_x):
|
| 79 |
+
in_x0, in_x1 = x * TILE_SIZE, min((x + 1) * TILE_SIZE, width)
|
| 80 |
+
in_y0, in_y1 = y * TILE_SIZE, min((y + 1) * TILE_SIZE, height)
|
| 81 |
+
pad_x0, pad_x1 = max(in_x0 - TILE_PAD, 0), min(in_x1 + TILE_PAD, width)
|
| 82 |
+
pad_y0, pad_y1 = max(in_y0 - TILE_PAD, 0), min(in_y1 + TILE_PAD, height)
|
| 83 |
+
|
| 84 |
+
tile_out = model(img[:, :, pad_y0:pad_y1, pad_x0:pad_x1])
|
| 85 |
+
|
| 86 |
+
trim_x0, trim_y0 = (in_x0 - pad_x0) * SCALE, (in_y0 - pad_y0) * SCALE
|
| 87 |
+
trim_x1 = trim_x0 + (in_x1 - in_x0) * SCALE
|
| 88 |
+
trim_y1 = trim_y0 + (in_y1 - in_y0) * SCALE
|
| 89 |
+
output[:, :, in_y0 * SCALE:in_y1 * SCALE, in_x0 * SCALE:in_x1 * SCALE] = (
|
| 90 |
+
tile_out[:, :, trim_y0:trim_y1, trim_x0:trim_x1]
|
| 91 |
+
)
|
| 92 |
+
return output
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def upscale_frames(
|
| 96 |
+
frames: list[Image.Image], progress_callback: ProgressCallback | None = None
|
| 97 |
+
) -> list[Image.Image]:
|
| 98 |
+
"""Upscale every frame 4x with `4xLSDIRCompact`, tiled to bound peak CPU memory.
|
| 99 |
+
|
| 100 |
+
progress_callback(done, total), if given, fires after each frame finishes.
|
| 101 |
+
"""
|
| 102 |
+
if not frames:
|
| 103 |
+
return []
|
| 104 |
+
model = _load_model()
|
| 105 |
+
out_frames = []
|
| 106 |
+
for frame in frames:
|
| 107 |
+
w, h = frame.size
|
| 108 |
+
array = np.asarray(frame.convert("RGB"), dtype=np.float32) / 255.0
|
| 109 |
+
tensor = torch.from_numpy(array).permute(2, 0, 1).unsqueeze(0).to(device)
|
| 110 |
+
padded = _pre_pad(tensor, TILE_PAD)
|
| 111 |
+
upscaled = _tile_process(model, padded)
|
| 112 |
+
# crop the pre-pad border (scaled) back off
|
| 113 |
+
upscaled = upscaled[:, :, : h * SCALE, : w * SCALE]
|
| 114 |
+
result = upscaled.squeeze(0).clamp(0, 1).permute(1, 2, 0).numpy()
|
| 115 |
+
out_frames.append(Image.fromarray((result * 255.0).round().astype(np.uint8)))
|
| 116 |
+
if progress_callback is not None:
|
| 117 |
+
progress_callback(len(out_frames), len(frames))
|
| 118 |
+
return out_frames
|