TurboT2VA-Ovi

Experimental four-step, text-conditioned audio/video distillation snapshots. This includes DCM and independent corrected sCM research snapshots, plus explicitly archival DMD-only and legacy rCM ablations. It is NOT a validated final rCM model, an upstream official release or a drop-in Diffusers pipeline.

Code: liuyuxiang1021/ovi-t2av-distillation. HF target: luyu1021/turbot2va-ovi, public with explicit user approval on 2026-11-19. The initial DCM500/sCM300 release passed remote verification; corrected sCM343 was subsequently uploaded and verified as a later safe-stop snapshot without a same-step benchmark. That initial three-student package was about140.17GB. The current five-student package including experimental DMD/rCM is about233.67GB; all optimizers remain local. GitHub remains private, so accessing the matching training/inference code requires repository access; this is not a self-contained public inference bundle. The expanded weight-upload commit is 44d049be9ce3686500cedff1a025fbc2940f4c62. The DMD2000/rCM1663 extension was uploaded under the user's explicit request. These are experimental archives, not recommended quality weights. The expanded weight-upload commit 2a666163401f3637a84d354fc078fb95a60274b2 passed remote size/SHA-256 verification of all84 new model/evaluation artifacts. Use the current release manifest for exact available files, byte counts, evaluation steps and per-snapshot provenance. Check release_manifest.json for the exact source code commit, included files, SHA-256 hashes, byte counts and optimizer availability. This card is also kept in the code repository. code_commit identifies the unchanged training/release implementation; documentation_commit, when present, identifies the later documentation-only update. The manifest and checksums include those updated docs.

Snapshots

HF Folder Purpose Full Same-Step Benchmark Optimizer
checkpoints/dcm-step500/ Anchored DCM result; initializes a new sCM stage Eight Prompts, DCM500 Not Included In Default Package
checkpoints/scm-v2-step300/ Latest corrected sCM snapshot with evaluated outputs Eight Prompts, sCM300 Not Included In Default Package
checkpoints/scm-v2-step343/ Later safe-stop snapshot; no same-step quality claim None; nearest benchmark is300 Not Included
checkpoints/dmd-reference-step2000/ Archival reference-objective text-only DMD Eight Prompts, DMD2000 Not Included
checkpoints/rcm-av3to2-step1663/ Archival legacy combined3:2 safe stop, NOT corrected-sCM continuation None; included evaluation is1650 Not Included

Each student comprises eight PyTorch DCP .distcp shards and .metadata, about 46.67 GB decimal in total, plus original meta.json. Do not load a single shard or rename DCP as safetensors. Five students total about233.36GB. An optional sCM343 resume layer adds eight AdamW shards, about93.29 GB. Exact optimizer continuation requires that layer AND matching inputs/runtime/settings; model-only snapshots are not full resume packages. The historical two-model private package was about 93.5 GB. The user approved public HF visibility to expand to three model snapshots, about 140.17 GB, then separately requested archival DMD/rCM. No critic states or optimizer files are included for either experiment; exact continuation would additionally require those full states and validated runtime compatibility. Complete source states remain local.

evaluations/dcm-step500/ and evaluations/scm-v2-step300/ contain eight MP4s, generated student latents, and per-source latent/media metrics. No raw training media, training latents/text embeddings, train CSV, held-out input embeddings, teacher model, teacher videos, decoders or tokens are bundled. The extension includes evaluations/dmd-reference-step2000/ and evaluations/rcm-av3to2-step1650/, each eight videos, generated latents and metrics. The latter is NOT an evaluation of1663. Parameter/reproduction/loading details are in docs/DMD_RCM_EXPERIMENTS.md; historical rCM critical source is retained under docs/provenance/rcm-av3to2/.

Recipe

The following recipe describes DCM/corrected sCM only. DMD uses text-only audited inputs, AdamW beta1=0, student/critic LR2.5e-6/5e-7 and critic-every-step reference cadence. Legacy rCM uses paired inputs, beta1=0.9, LR2e-6/2e-6, exclusive1:4, AV weights0.60/0.40, old sCM objective and DMD coefficient0.002. These objectives and schedules are not interchangeable. See the experiment parameter table.

Eight H20 GPUs; 20,000 audited disjoint paired training samples; seed1234; AdamW betas0.9/0.999, default weight decay0.01, grad clip1; BF16 ordinary forwards with FP32 student weights and gradient reductions; optimizer CPU offload. Video/audio loss weights0.85/0.15.

DCM forks the original unanchored step50 checkpoint, adding conditional teacher anchor0.25; base LR2e-6, LR warmup100, cosine horizon2000/floor0.1; trig grid48, skip1, teacher CFG4/3, loss scale100. Corrected sCM starts a NEW stage from DCM500: fresh AdamW, LR2e-6, LR warmup500 AND both tangent terms warmup500, exact FSDP-compatible unsharded JVP, p_g mean1.5, tangent p99 clip20, loss scale100. Objective ID: rcm_tangent_warmup_v2_unsharded.

The old target incorrectly omitted warmup from the JVP term. Old high-LR sCM branches showed cross-prompt audio-latent collapse and are NOT included here. The code fix is established by regression tests; there is no controlled proof that the bug alone caused collapse because LR also varied.

Evaluation And Limitations

Eight fixed held-out prompts, seed103, matching initial noise, four-step student versus50-step recipe teacher. These are paired diagnostic means, not perceptual or large-scale benchmark scores:

Snapshot Video Latent Cosine Audio Latent Cosine RGB SSIM
DCM500 / sCM0 0.60283 0.45035 0.41159
Corrected sCM300 0.62233 0.44530 0.36119
Archival DMD2000 0.21951 0.29864 0.11028
Archival rCM1650 (NOT1663) 0.47028 0.31574 0.22518

At sCM300 cross-prompt audio cosine is0.671 and mean-centered residual/audio norm ratio0.521. The previously observed near-identical audio-latent collapse has not recurred up to this evaluation, but visual SSIM fell and subjective sound/video quality still needs inspection. Step343 has no same-step benchmark; do NOT label step300 metrics as step343 measurements. Neither sound fidelity nor AV sync is established by latent cosine. Later training stability is unknown.

The legacy combined rCM3:2 ablation was stopped at1663. Its latest1650 audio alignment deteriorated despite increased RMS; it is not a recommended weight release and is documented separately in docs/RCM_EXPERIMENTS_2026-11-19.md. It is now included only as an explicitly requested archival ablation. DMD2000 also deteriorated relative to its pre-training baseline; neither extension establishes improved subjective sound fidelity, visual quality or AV sync.

Loading And Provenance

Use the matching code repository, upstream Ovi model configuration and teacher initialization. compare_ovi_inference.py loads DCP into an ordinary FP32 model with CPU resharding; use explicit --preview-stage dcm or --preview-stage scm. The loader then requires original Ovi decoders and your own input text/latents to render outputs. See docs/TRAINING_GUIDE.md for commands and prerequisites. For DMD student loading explicitly add --diagnostic-dmd; for combined rCM omit that flag and --preview-stage. DMD's original metadata predates newer strict trainer fields: exact optimizer continuation is unverified and cannot be enabled by silently rewriting the published provenance.

Original meta.json includes the source machine's absolute paths and is preserved unchanged. Strict loader/trainer checks require matching teacher/index/benchmark identity. Controlled cross-machine path remapping, full-weight GPU recovery and exact resume on a different host have NOT been validated. File completeness and SHA-256 verification are not a full-weight resume test.

Ovi source and upstream model card identify Apache-2.0; attribution and license are retained. Decoder/encoder components keep their own upstream licenses. This release does not relicense input datasets or redistribute upstream decoder weights. Avoid presenting generated videos/audio as real recordings; subject to applicable rights and policies, use these experimental artifacts responsibly.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for luyu1021/turbot2va-ovi

Finetuned
chetwinlow1/Ovi
Finetuned
(3)
this model