Instructions to use mlx-community/RelayVSR-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use mlx-community/RelayVSR-bf16 with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download mlx-community/RelayVSR-bf16 --local-dir RelayVSR-bf16
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
mlx-community/RelayVSR-bf16
RelayVSR (kopper/RelayVSR,
kopperx/RelayVSR, arXiv 2609.37850) —
streaming ×4 video super-resolution by large–small model collaboration — converted to MLX for the Swift/MLX port
xocialize/mlx-relayvsr-swift.
A large generative model (the Wan2.2-TI2V-5B DiT, distilled to one step) runs on one frame in fifteen and proposes a reference latent. A 32 M-parameter transformer then super-resolves every frame from the low-resolution frame and the two reference latents around it. There is no VAE and no text encoder at runtime.
What to expect
- Faithful. It stays close to the source: defocus stays soft and a drawing stays a drawing, so it suits anime and graphics as well as live action.
- Soft. It removes film grain and adds little high-frequency detail, and it does not repair codec damage.
- ×4 only; output lags input by up to 15 frames; every frame is kept.
Files
| file | component | params | notes |
|---|---|---|---|
generator_bf16.safetensors |
the reference generator: Wan2.2-TI2V-5B DiT (LoRA merged, one step at t = 1000) + the LQ projector | 5,031,059,136 | released dtype (bf16), bit for bit; patch_embedding Conv3d (O,D,H,W,I), lq_proj Conv2d (O,H,W,I) |
decoder_fp32.safetensors |
the lightweight VSR network (upstream flash.safetensors, variant S, step 60000 EMA) |
31,934,208 | released dtype (fp32), bit for bit; Conv2d (O,H,W,I) |
prompt_bf16.safetensors |
the fixed prompt embedding (1, 512, 4096) | — | split out of upstream's wan.safetensors |
config.json |
both upstream configs, sampling constants, provenance (source sha256s) |
Keys are upstream's verbatim; only conv layouts change (channels-last), and the two lq_proj norm weights are
flattened from (C, 1, 1) to (C). Produced by the port's oracle/convert_weights.py from kopper/RelayVSR @ 751954dc.
One lane serves both precisions: the port runs the generator in bf16 and the small network in fp16 by default, and upcasts the same files exactly for its fp32 parity lane.
Parity with upstream
Gated against upstream's own code executed verbatim on the CPU in fp32 (a 9-frame clip chosen so that both window masks bite and the generator's 4-keyframe cache wraps):
- key contract: 833 + 200 tensors, no missing and no unused key;
- every component within 3e-5 (max abs / reference max) of upstream;
- the whole stream at least 120 dB per frame, on the CPU and on the GPU.
The default half-precision setting stays 49–53 dB from the fp32 lane on real footage, level with upstream's own bf16. Not yet gated against a CUDA run of upstream's Triton attention kernels; the port uses dense attention under the same masks.
Use
swift build -c release --build-system swiftbuild --product relayvsr-smoke
.build/out/Products/Release/relayvsr-smoke run <lrFramesDir> <outDir> <this directory>
let pipe = try RelayVSRPipeline.load(directory: laneDirectory)
try pipe.run(frames: lrFrames) { index, frame in … } // (4H, 4W, 3) in [−1, 1], in order
Through MLXEngine the MLXRelayVSR package downloads this repository
on first use.
Measured on an M5 Max (GPU shared with other apps): 0.075 s per 1280×768 output frame and 0.245 s at 1920×1080; process peak 14.3 GB and 16.8 GB through the engine.
Licence
Apache-2.0, as released by the RelayVSR authors; the base model Wan2.2-TI2V-5B is Apache-2.0. The training data is not disclosed by the authors.
- Downloads last month
- 24
Quantized