🎞️ OmniVR

Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films

Xin Lu^†   Zihao Fan^†   Jie Huang^‡   Mingchen Zhong   Xueyang Fu^✉   Zheng-Jun Zha

University of Science and Technology of China (USTC)

^† Equal contribution    ^‡ Project leader    ^✉ Corresponding author

     

---

🌟 Overview

Historical films suffer from co-occurring visual and audio degradations — blur, noise, flicker, hiss, clipping, and dropout — yet existing methods restore each modality independently, leaving quality gaps and cross-modal inconsistency.

OmniVR is the first joint audio-video generative restoration model. Built on a 22B-parameter audio-video generation backbone (LTX-2.3), it formulates restoration as conditional generation inside a unified multimodal DiT: the low-quality video and audio are encoded as latent conditions, combined with a fixed restoration prompt, and jointly denoised to recover visual structure, temporal motion, and acoustic detail under one coordinated objective.

Three key designs enable this adaptation:

	Design	What it does
1	Joint AV Degradation Pipeline	Simulates real old-film characteristics — blur, noise, flicker, hiss, clipping, dropout — from Internet-collected data, producing paired LQ audio-video for training.
2	Prior-Preserving T2AV → AV2AV	An architecture-preserving transition with prompt annealing recasts text-to-audio-video generation as audio-video-to-audio-video restoration, maximally retaining the generative prior.
3	I2V Anchoring & Waveform Supervision	First-frame image-to-video anchoring with loss reweighting and waveform supervision enables long-video extrapolation and high audio fidelity.

We also propose OmniVRBench, the first benchmark evaluating audio-video restoration across visual quality, audio quality, temporal consistency, and audio-visual synchrony on 200 real historical clips.

---

📊 Results

OmniVR surpasses all baselines on both OmniVRBench tracks across visual, audio, and sync metrics:

Track	MUSIQ ↑	CLIP-IQA ↑	DNSMOS ↑	LSE-C ↑
Controlled (with_gt) — best baseline	45.54	0.268	2.12	2.32
Controlled (with_gt) — OmniVR	71.17	0.543	2.70	3.52
Real (no_gt) — best baseline	52.38	0.421	2.21	1.05
Real (no_gt) — OmniVR	61.87	0.444	2.43	1.12

A pairwise human preference study (12 annotators, 129 real clips) prefers OmniVR 80.0% overall — a +56.7 percentage point gain over the strongest baseline. OmniVR is also the first method to produce natural colorization while jointly addressing all three aspects.

---

📦 What's in This Repository

Everything OmniVR needs — weights, benchmark, and reference outputs — lives in this single repo:

xin1u/OmniVR/
├── weights/
│   ├── ominivr_lora_step_01800.safetensors   # LoRA rank 384 — no_gt / real track
│   └── ominivr_lora_step_02400.safetensors   # LoRA rank 384 — with_gt / controlled track
├── tinydecoder/
│   └── taeltx2_3_wide.pth                    # fast HR decoder (optional, recommended)
├── omnibench/                                # OmniVRBench: 200 evaluation clips
│   ├── with_gt/  (71 clips: lq/ + gt/)       # controlled, full-reference track
│   ├── no_gt/    (129 clips)                 # real archival, no-reference track
│   └── manifest.json                         # unified metadata for all 200 clips
└── predictions/                              # precomputed neg_cfg restorations (both tracks)

We release two LoRA checkpoints from the same training run (v5_tav2av_hq), matching the two evaluation tracks in the paper:

• step_01800 — used for the no_gt / real-footage (no-reference) track
• step_02400 — used for the with_gt / talking-head (full-reference) track

⚠️ These weights are a Derivative of LTX-2 and are governed by the LTX-2 Community License Agreement — not Apache-2.0. The Apache-2.0 license on this repo covers OmniVR's own code only. See NOTICE.

---

🚀 Quick Start

1️⃣ Install

OmniVR is a LoRA adapter for Lightricks' LTX-2 and depends on its ltx-core / ltx-trainer / ltx-pipelines packages:

git clone https://github.com/xin1u/OminiVR.git && cd OminiVR
pip install -e .

git clone https://github.com/Lightricks/LTX-2.git /path/to/ltx-2
export OMINIVR_DEPS_ROOT=/path/to/ltx-2/packages

2️⃣ Download Weights & Data

# OmniVR LoRA weights + fast decoder
huggingface-cli download xin1u/OmniVR weights/ominivr_lora_step_01800.safetensors --local-dir .
huggingface-cli download xin1u/OmniVR weights/ominivr_lora_step_02400.safetensors --local-dir .
huggingface-cli download xin1u/OmniVR tinydecoder/taeltx2_3_wide.pth --local-dir ckpt

# OmniVRBench evaluation set (optional)
huggingface-cli download xin1u/OmniVR --include "omnibench/*" --repo-type model --local-dir data

The base LTX-2.3 model and Gemma text encoder must be obtained from their official sources:

export OMINIVR_MODEL_PATH=/path/to/ltx-2.3-22b-dev.safetensors
export OMINIVR_TEXT_ENCODER_PATH=/path/to/gemma-3-12b-it-qat-q4_0-unquantized
export OMINIVR_LORA_STRUCTURE=/path/to/ltx-2.3-22b-distilled-lora-384.safetensors

3️⃣ Restore a Video

python scripts/infer.py \
    --input_dir data/omnibench/no_gt \
    --output_dir /path/to/output \
    --checkpoint /path/to/ominivr_lora_step_01800.safetensors \
    --strategy neg_cfg \
    --guidance_scale 3.0 \
    --num_steps 30 \
    --condition_noise 0.3 \
    --width 1920 --height 1088 --max_frames 121 --frame_rate 24 \
    --stg_scale 1.0 --stg_blocks 29 --stg_mode stg_av \
    --audio_denoise --audio_denoise_highpass 200 --audio_denoise_over 2.5 --audio_denoise_floor 0.03

---

⚙️ Key Options

Option	Description
--strategy no_cfg	guidance_scale=1.0, single forward per step (fastest)
--strategy empty_cfg	positive=SR prompt, negative=empty string (matches training distribution)
--strategy neg_cfg	positive=SR prompt, negative=negative prompt (strongest enhancement, used for paper numbers)
--strategy all	runs all three on the same inputs, for comparison
--lq_grayscale	Pass when restoring genuinely black-and-white footage — training degraded ~50% of clips to grayscale, so the model expects a grayscale LQ signal for that case.
--num_gpus N	Data-parallel across N GPUs (each GPU takes a disjoint subset of input videos, round-robin).

TinyDecoder: if ckpt/tinydecoder/taeltx2_3_wide.pth is present, inference uses the fast TinyDecoder path (keeps the transformer resident on GPU). If absent, it falls back automatically to the full VAE decoder (slower, tiled to control memory).

---

📈 OmniVRBench

200 clips at 5.04s / 24fps / 121 frames, split into two tracks by whether ground truth exists:

	with_gt/ (71 clips)	no_gt/ (129 clips)
Content	Talking-head / speaking subjects	Real historical film footage
Source	LQ synthesized from HQ via degradation pipeline; GT = clean HQ	Collected real old films (inherently LQ)
Ground truth	✅ frame-aligned, same filenames	❌ does not exist
Evaluation track	Controlled (full-reference)	Real (no-reference)
Applicable metrics	PSNR/SSIM/LPIPS, PESQ/STOI/SI-SDR, LSE-C/D, AV-Align, plus NR-IQA	NR-IQA only: MUSIQ/CLIP-IQA/MANIQA + no-reference audio

The with_gt track is purpose-built for lip-sync (LSE-C/D) and audio-visual synchrony ablations: every clip is confirmed by face detection to have a visible speaker and by silero-VAD to have speech ratio ≥ 0.40.

Predictions and ground truth match by filename, so any standard metric implementation plugs in directly. Precomputed neg_cfg reference outputs for both tracks are under predictions/ if you want to compare without rerunning inference.

---

📄 License & Attribution

• OmniVR's own code (src/, scripts/) — Apache-2.0
• Released LoRA weights — a Derivative of LTX-2, governed by the LTX-2 Community License Agreement
• tiny_decoder.py — adapted from Ollin Boer Bohan's Seraena/TAESD (MIT)

We gratefully acknowledge LTX-2 and Seraena / TAESD.

---

📞 Contact

Please feel free to contact us if there is any question (luxion@mail.ustc.edu.cn).

---

📜 Citation

Paper: arXiv:2608.04224

@article{lu2026omnivr,
  title={OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films},
  author={Lu, Xin and Fan, Zihao and Zhong, Mingchen and Huang, Jie and Fu, Xueyang and Zha, Zheng-Jun},
  journal={arXiv preprint arXiv:2608.04224},
  year={2026}
}