someone-in-the-world Claude Sonnet 5 commited on
Commit
819311d
·
1 Parent(s): f5e3ffa

Add 4x super-resolution upscaling stage (FR-4, FR-5, ID-2, CR-2)

Browse files

postprocess/upscale/: 4xLSDIRCompact (SRVGGNetCompact, num_conv=16) tiled CPU upscaling,
vendored from the reference project (inference structure adapted from xinntao/Real-ESRGAN,
BSD-3-Clause). Runs after interpolation and outside the @spaces.GPU call, so it never bids
for ZeroGPU time. Wired into app.py's "Upscale 4x" checkbox.

Also adds .gitignore for __pycache__/ and the RIFE weights downloaded at runtime.

Closes #6

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.gitignore ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ __pycache__/
2
+ *.pyc
3
+ train_log/
4
+ RIFEv4.26_0921.zip
5
+ flagged/
LICENSES/4xLSDIRCompact-LICENSE ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Creative Commons Attribution 4.0 International (CC BY 4.0)
2
+
3
+ Full license text: https://creativecommons.org/licenses/by/4.0/legalcode
4
+ Human-readable summary: https://creativecommons.org/licenses/by/4.0/
5
+
6
+ ---
7
+ Applies to: the model weights downloaded at runtime from
8
+ https://huggingface.co/Phips/4xLSDIRCompact (postprocess/upscale/upscale.py, WEIGHTS_URL).
9
+
10
+ Attribution: "4xLSDIRCompact" by Philip Hofmann (https://github.com/Phhofm/models), licensed under
11
+ CC BY 4.0. Architecture: SRVGGNetCompact, trained on the LSDIR dataset. No modifications were made to
12
+ the weights; they are loaded as published.
LICENSES/REAL-ESRGAN-LICENSE ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ BSD 3-Clause License
2
+
3
+ Copyright (c) 2021, Xintao Wang
4
+ All rights reserved.
5
+
6
+ Redistribution and use in source and binary forms, with or without
7
+ modification, are permitted provided that the following conditions are met:
8
+
9
+ 1. Redistributions of source code must retain the above copyright notice, this
10
+ list of conditions and the following disclaimer.
11
+
12
+ 2. Redistributions in binary form must reproduce the above copyright notice,
13
+ this list of conditions and the following disclaimer in the documentation
14
+ and/or other materials provided with the distribution.
15
+
16
+ 3. Neither the name of the copyright holder nor the names of its
17
+ contributors may be used to endorse or promote products derived from
18
+ this software without specific prior written permission.
19
+
20
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
22
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
23
+ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
24
+ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
25
+ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
26
+ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
27
+ CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
28
+ OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
29
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
30
+
31
+ ---
32
+ Applies to: postprocess/upscale/srvgg_arch.py (vendored from xinntao/Real-ESRGAN,
33
+ https://github.com/xinntao/Real-ESRGAN, realesrgan/archs/srvgg_arch.py) and postprocess/upscale/upscale.py
34
+ (inference structure adapted from realesrgan/utils.py's RealESRGANer). The model weights themselves are no
35
+ longer from this project — see LICENSES/4xLSDIRCompact-LICENSE.
README.md CHANGED
@@ -17,7 +17,33 @@ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-
17
  Image-to-video generation from [`thornmaze/WAMU_v3_WAN2.2_I2V_LIGHTNING`](https://huggingface.co/thornmaze/WAMU_v3_WAN2.2_I2V_LIGHTNING)
18
  via `WanImageToVideoPipeline`, running on Hugging Face ZeroGPU.
19
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
  ## Status
21
 
22
  Baseline (non-AOT) pipeline. AOT-compiled inference (faster, but requires a compatible
23
  precompiled package for WAMU_v3) is tracked separately — see the repo's open issues.
 
 
 
17
  Image-to-video generation from [`thornmaze/WAMU_v3_WAN2.2_I2V_LIGHTNING`](https://huggingface.co/thornmaze/WAMU_v3_WAN2.2_I2V_LIGHTNING)
18
  via `WanImageToVideoPipeline`, running on Hugging Face ZeroGPU.
19
 
20
+ ## Post-processing
21
+
22
+ Two optional steps can run after generation:
23
+
24
+ | step | what | runs on |
25
+ |---|---|---|
26
+ | Frame interpolation ("Video Fluidity") | RIFE v4.26 (`thornmaze/RIFE`), half precision. 2x/4x/8x the native 16 fps. | GPU, inside the generation call |
27
+ | Upscaling ("Upscale 4×") | `4xLSDIRCompact` (SRVGGNetCompact, num_conv=16), forced fp32, tiled to bound peak CPU memory. Fixed 4x. | CPU, after the GPU call returns |
28
+
29
+ Upscaling runs after interpolation and outside the `@spaces.GPU` call, so it never bids for or consumes
30
+ ZeroGPU worker time — only wall-clock time on the Space's regular CPU.
31
+
32
+ RIFE support code (`model/warplayer.py`, `model/loss.py`, `model/pytorch_msssim/`) is vendored from
33
+ [`hzwer/Practical-RIFE`](https://github.com/hzwer/Practical-RIFE) (MIT, see
34
+ [`LICENSES/RIFE-LICENSE`](LICENSES/RIFE-LICENSE)) — required as sibling-import targets for the
35
+ `train_log/RIFE_HDv3.py` module downloaded at runtime from `thornmaze/RIFE`.
36
+
37
+ The upscaling *code* (`postprocess/upscale/`) is vendored/adapted from
38
+ [`xinntao/Real-ESRGAN`](https://github.com/xinntao/Real-ESRGAN) (BSD-3-Clause License, see
39
+ [`LICENSES/REAL-ESRGAN-LICENSE`](LICENSES/REAL-ESRGAN-LICENSE)). The *weights* are
40
+ [`Phips/4xLSDIRCompact`](https://huggingface.co/Phips/4xLSDIRCompact) (CC BY 4.0, see
41
+ [`LICENSES/4xLSDIRCompact-LICENSE`](LICENSES/4xLSDIRCompact-LICENSE)), not Real-ESRGAN's own — see the
42
+ comment at the top of `postprocess/upscale/upscale.py` for why.
43
+
44
  ## Status
45
 
46
  Baseline (non-AOT) pipeline. AOT-compiled inference (faster, but requires a compatible
47
  precompiled package for WAMU_v3) is tracked separately — see the repo's open issues.
48
+
49
+ LoRA loading is out of scope for this version (see SRS FR-8/C-4).
app.py CHANGED
@@ -6,6 +6,7 @@ import numpy as np
6
  import spaces
7
  import torch
8
  from diffusers.utils.export_utils import export_to_video
 
9
 
10
  from model.pipeline import (
11
  DEFAULT_NEGATIVE_PROMPT,
@@ -92,6 +93,7 @@ def generate_video(
92
  seed,
93
  randomize_seed,
94
  frame_multiplier,
 
95
  progress=gr.Progress(track_tqdm=True),
96
  ):
97
  if input_image is None:
@@ -113,6 +115,19 @@ def generate_video(
113
  progress,
114
  )
115
 
 
 
 
 
 
 
 
 
 
 
 
 
 
116
  with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tmpfile:
117
  video_path = tmpfile.name
118
  export_to_video(final_frames, video_path, fps=final_fps, quality=6)
@@ -137,6 +152,12 @@ with gr.Blocks() as demo:
137
  label="Video Fluidity (Frames per Second)",
138
  info="Extra frames are generated with RIFE frame interpolation to smooth motion.",
139
  )
 
 
 
 
 
 
140
  with gr.Accordion("Advanced Settings", open=False):
141
  negative_prompt_input = gr.Textbox(label="Negative Prompt", value=DEFAULT_NEGATIVE_PROMPT, lines=3)
142
  steps_slider = gr.Slider(minimum=1, maximum=30, step=1, value=6, label="Inference Steps")
@@ -160,6 +181,7 @@ with gr.Blocks() as demo:
160
  seed_input,
161
  randomize_seed_checkbox,
162
  frame_multiplier_input,
 
163
  ],
164
  outputs=[video_output, file_output, seed_input],
165
  )
 
6
  import spaces
7
  import torch
8
  from diffusers.utils.export_utils import export_to_video
9
+ from PIL import Image
10
 
11
  from model.pipeline import (
12
  DEFAULT_NEGATIVE_PROMPT,
 
93
  seed,
94
  randomize_seed,
95
  frame_multiplier,
96
+ upscale_output,
97
  progress=gr.Progress(track_tqdm=True),
98
  ):
99
  if input_image is None:
 
115
  progress,
116
  )
117
 
118
+ # CPU-only post-processing: outside the @spaces.GPU call, so it never bids for ZeroGPU time.
119
+ if upscale_output:
120
+ from postprocess.upscale import upscale_frames
121
+
122
+ def _report_upscale_progress(done: int, total: int) -> None:
123
+ progress(0.95 + 0.04 * (done / total), desc=f"Upscaling 4x on CPU ({done}/{total} frames)...")
124
+
125
+ progress(0.95, desc=f"Upscaling 4x on CPU (0/{len(final_frames)} frames)...")
126
+ pil_frames = [
127
+ Image.fromarray((np.asarray(frame) * 255.0).round().astype(np.uint8)) for frame in final_frames
128
+ ]
129
+ final_frames = upscale_frames(pil_frames, progress_callback=_report_upscale_progress)
130
+
131
  with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tmpfile:
132
  video_path = tmpfile.name
133
  export_to_video(final_frames, video_path, fps=final_fps, quality=6)
 
152
  label="Video Fluidity (Frames per Second)",
153
  info="Extra frames are generated with RIFE frame interpolation to smooth motion.",
154
  )
155
+ upscale_checkbox = gr.Checkbox(
156
+ label="Upscale 4× (4xLSDIRCompact)",
157
+ value=False,
158
+ info="Upscales the finished video 4x on CPU after generation. Runs after frame "
159
+ "interpolation and doesn't use GPU quota, but adds wall-clock time.",
160
+ )
161
  with gr.Accordion("Advanced Settings", open=False):
162
  negative_prompt_input = gr.Textbox(label="Negative Prompt", value=DEFAULT_NEGATIVE_PROMPT, lines=3)
163
  steps_slider = gr.Slider(minimum=1, maximum=30, step=1, value=6, label="Inference Steps")
 
181
  seed_input,
182
  randomize_seed_checkbox,
183
  frame_multiplier_input,
184
+ upscale_checkbox,
185
  ],
186
  outputs=[video_output, file_output, seed_input],
187
  )
postprocess/upscale/__init__.py ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ from .upscale import upscale_frames
2
+
3
+ __all__ = ["upscale_frames"]
postprocess/upscale/srvgg_arch.py ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """SRVGGNetCompact, the architecture behind `realesr-general-x4v3`. Vendored from xinntao/Real-ESRGAN
2
+ (BSD-3-Clause License, see `LICENSES/REAL-ESRGAN-LICENSE`), `realesrgan/archs/srvgg_arch.py`, with the
3
+ `basicsr.utils.registry.ARCH_REGISTRY` import and decorator dropped — that registry only matters for basicsr's
4
+ config-driven model zoo, and this Space instantiates the class directly. Dropping it also means this module has no
5
+ `basicsr` dependency, which matters: `basicsr`'s own imports break on current torchvision (its
6
+ `degradations.py` imports `torchvision.transforms.functional_tensor`, removed upstream) — the reason issue #1
7
+ chose the compact architecture over pulling in the full `realesrgan` package.
8
+ """
9
+
10
+ from torch import nn as nn
11
+ from torch.nn import functional as F
12
+
13
+
14
+ class SRVGGNetCompact(nn.Module):
15
+ """A compact VGG-style network structure for super-resolution.
16
+
17
+ It is a compact network structure, which performs upsampling in the last layer and no convolution is
18
+ conducted on the HR feature space.
19
+
20
+ Args:
21
+ num_in_ch (int): Channel number of inputs. Default: 3.
22
+ num_out_ch (int): Channel number of outputs. Default: 3.
23
+ num_feat (int): Channel number of intermediate features. Default: 64.
24
+ num_conv (int): Number of convolution layers in the body network. Default: 16.
25
+ upscale (int): Upsampling factor. Default: 4.
26
+ act_type (str): Activation type, options: 'relu', 'prelu', 'leakyrelu'. Default: prelu.
27
+ """
28
+
29
+ def __init__(self, num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=16, upscale=4, act_type='prelu'):
30
+ super(SRVGGNetCompact, self).__init__()
31
+ self.num_in_ch = num_in_ch
32
+ self.num_out_ch = num_out_ch
33
+ self.num_feat = num_feat
34
+ self.num_conv = num_conv
35
+ self.upscale = upscale
36
+ self.act_type = act_type
37
+
38
+ self.body = nn.ModuleList()
39
+ self.body.append(nn.Conv2d(num_in_ch, num_feat, 3, 1, 1))
40
+ self.body.append(self._activation(act_type, num_feat))
41
+
42
+ for _ in range(num_conv):
43
+ self.body.append(nn.Conv2d(num_feat, num_feat, 3, 1, 1))
44
+ self.body.append(self._activation(act_type, num_feat))
45
+
46
+ self.body.append(nn.Conv2d(num_feat, num_out_ch * upscale * upscale, 3, 1, 1))
47
+ self.upsampler = nn.PixelShuffle(upscale)
48
+
49
+ @staticmethod
50
+ def _activation(act_type, num_feat):
51
+ if act_type == 'relu':
52
+ return nn.ReLU(inplace=True)
53
+ if act_type == 'prelu':
54
+ return nn.PReLU(num_parameters=num_feat)
55
+ if act_type == 'leakyrelu':
56
+ return nn.LeakyReLU(negative_slope=0.1, inplace=True)
57
+ raise ValueError(f"Unsupported act_type {act_type!r}")
58
+
59
+ def forward(self, x):
60
+ out = x
61
+ for i in range(len(self.body)):
62
+ out = self.body[i](out)
63
+
64
+ out = self.upsampler(out)
65
+ # add the nearest upsampled image, so that the network learns the residual
66
+ base = F.interpolate(x, scale_factor=self.upscale, mode='nearest')
67
+ out += base
68
+ return out
postprocess/upscale/upscale.py ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """`4xLSDIRCompact` tiled super-resolution, CPU-only, forced fp32.
2
+
3
+ Inference structure (pre-pad, tile, merge, crop) is adapted from xinntao/Real-ESRGAN's `RealESRGANer`
4
+ (BSD-3-Clause License, see `LICENSES/REAL-ESRGAN-LICENSE`, `realesrgan/utils.py`), trimmed to the single code path
5
+ this Space needs: one fixed compact model, RGB `PIL.Image` in and out (no alpha channel, no 16-bit, no cv2/BGR
6
+ round-trip, no `dni` model blending, no CLI prefetch/IO-queue threads — none of those apply to frames coming
7
+ straight out of the Wan 2.2 decode step).
8
+
9
+ The weights are `4xLSDIRCompact` (Philip Hofmann, CC BY 4.0, see `LICENSES/4xLSDIRCompact-LICENSE`) rather than
10
+ Real-ESRGAN's own `realesr-general-x4v3`: same `SRVGGNetCompact` architecture (just `num_conv=16` instead of 32,
11
+ so if anything it's cheaper per tile), but trained on clean LSDIR photos with no synthetic noise/blur/JPEG
12
+ augmentation. `realesr-general-x4v3` bakes in fairly aggressive denoising to cope with that augmentation, which
13
+ flattens real skin/fabric/hair texture into a smooth, CG-like look on frames that were never actually degraded
14
+ (e.g. straight out of a video decode step, as here) — `4xLSDIRCompact` doesn't have that bias.
15
+
16
+ No `@spaces.GPU` here: this runs on the Space's regular CPU after generation returns, so it never bids for or
17
+ consumes paid ZeroGPU worker time.
18
+ """
19
+
20
+ from __future__ import annotations
21
+
22
+ import math
23
+ from functools import lru_cache
24
+ from pathlib import Path
25
+ from typing import Callable
26
+
27
+ import numpy as np
28
+ import torch
29
+ import torch.nn.functional as F
30
+ from PIL import Image
31
+ from safetensors.torch import load_file
32
+ from torch.hub import download_url_to_file, get_dir
33
+
34
+ from .srvgg_arch import SRVGGNetCompact
35
+
36
+ WEIGHTS_URL = "https://huggingface.co/Phips/4xLSDIRCompact/resolve/main/4xLSDIRCompact.safetensors"
37
+ SCALE = 4
38
+ NUM_CONV = 16
39
+ TILE_SIZE = 256
40
+ TILE_PAD = 10
41
+ device = torch.device("cpu")
42
+
43
+ ProgressCallback = Callable[[int, int], None]
44
+
45
+
46
+ def _download_weights() -> Path:
47
+ cache_dir = Path(get_dir()) / "checkpoints"
48
+ cache_dir.mkdir(parents=True, exist_ok=True)
49
+ weights_path = cache_dir / Path(WEIGHTS_URL).name
50
+ if not weights_path.exists():
51
+ download_url_to_file(WEIGHTS_URL, str(weights_path))
52
+ return weights_path
53
+
54
+
55
+ @lru_cache(maxsize=1)
56
+ def _load_model() -> SRVGGNetCompact:
57
+ model = SRVGGNetCompact(
58
+ num_in_ch=3, num_out_ch=3, num_feat=64, num_conv=NUM_CONV, upscale=SCALE, act_type="prelu"
59
+ )
60
+ state_dict = load_file(_download_weights())
61
+ model.load_state_dict(state_dict, strict=True)
62
+ model.to(device).float().eval()
63
+ return model
64
+
65
+
66
+ def _pre_pad(tensor: torch.Tensor, pad: int) -> torch.Tensor:
67
+ return F.pad(tensor, (0, pad, 0, pad), mode="reflect")
68
+
69
+
70
+ @torch.no_grad()
71
+ def _tile_process(model: SRVGGNetCompact, img: torch.Tensor) -> torch.Tensor:
72
+ batch, channel, height, width = img.shape
73
+ output = img.new_zeros((batch, channel, height * SCALE, width * SCALE))
74
+ tiles_x = math.ceil(width / TILE_SIZE)
75
+ tiles_y = math.ceil(height / TILE_SIZE)
76
+
77
+ for y in range(tiles_y):
78
+ for x in range(tiles_x):
79
+ in_x0, in_x1 = x * TILE_SIZE, min((x + 1) * TILE_SIZE, width)
80
+ in_y0, in_y1 = y * TILE_SIZE, min((y + 1) * TILE_SIZE, height)
81
+ pad_x0, pad_x1 = max(in_x0 - TILE_PAD, 0), min(in_x1 + TILE_PAD, width)
82
+ pad_y0, pad_y1 = max(in_y0 - TILE_PAD, 0), min(in_y1 + TILE_PAD, height)
83
+
84
+ tile_out = model(img[:, :, pad_y0:pad_y1, pad_x0:pad_x1])
85
+
86
+ trim_x0, trim_y0 = (in_x0 - pad_x0) * SCALE, (in_y0 - pad_y0) * SCALE
87
+ trim_x1 = trim_x0 + (in_x1 - in_x0) * SCALE
88
+ trim_y1 = trim_y0 + (in_y1 - in_y0) * SCALE
89
+ output[:, :, in_y0 * SCALE:in_y1 * SCALE, in_x0 * SCALE:in_x1 * SCALE] = (
90
+ tile_out[:, :, trim_y0:trim_y1, trim_x0:trim_x1]
91
+ )
92
+ return output
93
+
94
+
95
+ def upscale_frames(
96
+ frames: list[Image.Image], progress_callback: ProgressCallback | None = None
97
+ ) -> list[Image.Image]:
98
+ """Upscale every frame 4x with `4xLSDIRCompact`, tiled to bound peak CPU memory.
99
+
100
+ progress_callback(done, total), if given, fires after each frame finishes.
101
+ """
102
+ if not frames:
103
+ return []
104
+ model = _load_model()
105
+ out_frames = []
106
+ for frame in frames:
107
+ w, h = frame.size
108
+ array = np.asarray(frame.convert("RGB"), dtype=np.float32) / 255.0
109
+ tensor = torch.from_numpy(array).permute(2, 0, 1).unsqueeze(0).to(device)
110
+ padded = _pre_pad(tensor, TILE_PAD)
111
+ upscaled = _tile_process(model, padded)
112
+ # crop the pre-pad border (scaled) back off
113
+ upscaled = upscaled[:, :, : h * SCALE, : w * SCALE]
114
+ result = upscaled.squeeze(0).clamp(0, 1).permute(1, 2, 0).numpy()
115
+ out_frames.append(Image.fromarray((result * 255.0).round().astype(np.uint8)))
116
+ if progress_callback is not None:
117
+ progress_callback(len(out_frames), len(frames))
118
+ return out_frames