mlx-community/RelayVSR-bf16

RelayVSR (kopper/RelayVSR, kopperx/RelayVSR, arXiv 2609.37850) — streaming ×4 video super-resolution by large–small model collaboration — converted to MLX for the Swift/MLX port xocialize/mlx-relayvsr-swift.

A large generative model (the Wan2.2-TI2V-5B DiT, distilled to one step) runs on one frame in fifteen and proposes a reference latent. A 32 M-parameter transformer then super-resolves every frame from the low-resolution frame and the two reference latents around it. There is no VAE and no text encoder at runtime.

What to expect

  • Faithful. It stays close to the source: defocus stays soft and a drawing stays a drawing, so it suits anime and graphics as well as live action.
  • Soft. It removes film grain and adds little high-frequency detail, and it does not repair codec damage.
  • ×4 only; output lags input by up to 15 frames; every frame is kept.

Files

file component params notes
generator_bf16.safetensors the reference generator: Wan2.2-TI2V-5B DiT (LoRA merged, one step at t = 1000) + the LQ projector 5,031,059,136 released dtype (bf16), bit for bit; patch_embedding Conv3d (O,D,H,W,I), lq_proj Conv2d (O,H,W,I)
decoder_fp32.safetensors the lightweight VSR network (upstream flash.safetensors, variant S, step 60000 EMA) 31,934,208 released dtype (fp32), bit for bit; Conv2d (O,H,W,I)
prompt_bf16.safetensors the fixed prompt embedding (1, 512, 4096) — split out of upstream's wan.safetensors
config.json both upstream configs, sampling constants, provenance (source sha256s)

Keys are upstream's verbatim; only conv layouts change (channels-last), and the two lq_proj norm weights are flattened from (C, 1, 1) to (C). Produced by the port's oracle/convert_weights.py from kopper/RelayVSR @ 751954dc.

One lane serves both precisions: the port runs the generator in bf16 and the small network in fp16 by default, and upcasts the same files exactly for its fp32 parity lane.

Parity with upstream

Gated against upstream's own code executed verbatim on the CPU in fp32 (a 9-frame clip chosen so that both window masks bite and the generator's 4-keyframe cache wraps):

  • key contract: 833 + 200 tensors, no missing and no unused key;
  • every component within 3e-5 (max abs / reference max) of upstream;
  • the whole stream at least 120 dB per frame, on the CPU and on the GPU.

The default half-precision setting stays 49–53 dB from the fp32 lane on real footage, level with upstream's own bf16. Not yet gated against a CUDA run of upstream's Triton attention kernels; the port uses dense attention under the same masks.

Use

swift build -c release --build-system swiftbuild --product relayvsr-smoke
.build/out/Products/Release/relayvsr-smoke run <lrFramesDir> <outDir> <this directory>
let pipe = try RelayVSRPipeline.load(directory: laneDirectory)
try pipe.run(frames: lrFrames) { index, frame in … }        // (4H, 4W, 3) in [−1, 1], in order

Through MLXEngine the MLXRelayVSR package downloads this repository on first use.

Measured on an M5 Max (GPU shared with other apps): 0.075 s per 1280×768 output frame and 0.245 s at 1920×1080; process peak 14.3 GB and 16.8 GB through the engine.

Licence

Apache-2.0, as released by the RelayVSR authors; the base model Wan2.2-TI2V-5B is Apache-2.0. The training data is not disclosed by the authors.

Downloads last month
24
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for mlx-community/RelayVSR-bf16

Finetuned
kopper/RelayVSR
Finetuned
(3)
this model

Paper for mlx-community/RelayVSR-bf16