Hugging Face's logo

Massyzs
/
smart-insertion-v-stage1

Diffusers
Safetensors
t2v
video-editing
video-insertion
diffusion
wan2.1
qwen3-vl
Model card Files
xet
 Community
How to use from the
Use from the
Diffusers library
# Gated model: Login with a HF token with gated access permission
hf auth login
pip install -U diffusers transformers accelerate
import torch
from diffusers import DiffusionPipeline

# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("Massyzs/smart-insertion-v-stage1", dtype=torch.bfloat16, device_map="cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Smart-Insertion-V: Stage-1 Weights

Stage-1 (pre-training) transformer weights for Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework (SIGGRAPH Asia 2026).

Release policy. Due to policy restrictions, the fine-tuned (stage-2) model is not released. This stage-1 checkpoint covers the most compute-intensive part of training. The fine-tuned model can be reproduced by running stage 2 (scripts/wan2.1/finetune.sh in the code repository) on the released dataset, starting from this checkpoint.

Files

File Description
config.json Transformer architecture config (same as Wan2.1-T2V-14B)
diffusion_pytorch_model.safetensors Stage-1 transformer weights, bf16, 14.44B parameters (28.9 GB)

What stage 1 trains

Stage 1 starts from Wan2.1-T2V-14B and connects the Qwen3-VL-8B-Instruct encoder to the diffusion transformer. A new two-layer MLP, qwen_project_in, maps Qwen3-VL hidden states (4096) to the transformer width (5120). The projected Qwen tokens are concatenated with the T5 text tokens and used as cross-attention context. During stage 1, Qwen3-VL encodes the first frame of each training clip (--qwen_encoder_mode i2v). Only qwen_project_in and the cross_attn / norm3 layers of every block are trained. The recipe is scripts/wan2.1/pretrain.sh in the code repository.

Usage

These weights initialize stage 2 (scripts/wan2.1/finetune.sh). The fine-tuning script expects MODEL_NAME to be a directory laid out like Wan2.1-T2V-14B. Combine the official VAE, T5 encoder and tokenizer with the stage-1 transformer:

hf download Wan-AI/Wan2.1-T2V-14B --local-dir Wan2.1-T2V-14B
hf download Massyzs/smart-insertion-v-stage1 --local-dir smart-insertion-v-stage1

mkdir -p Wan2.1-T2V-14B-stage1
ln -s "$PWD"/Wan2.1-T2V-14B/{Wan2.1_VAE.pth,models_t5_umt5-xxl-enc-bf16.pth,google} Wan2.1-T2V-14B-stage1/
ln -s "$PWD"/smart-insertion-v-stage1/{config.json,diffusion_pytorch_model.safetensors} Wan2.1-T2V-14B-stage1/

Then set MODEL_NAME in finetune.sh to the absolute path of the Wan2.1-T2V-14B-stage1 directory.

License

The weights are derived from Wan2.1-T2V-14B and are released under the Apache 2.0 license.

Downloads last month
-
Safetensors
Model size
14B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Massyzs/smart-insertion-v-stage1

Base model

Wan-AI/Wan2.1-T2V-14B
Finetuned
(88)
this model