--- license: apache-2.0 pipeline_tag: video-classification base_model: openai/clip-vit-large-patch14 tags: - deepfake-detection - video-forgery-detection - video-classification - onnx - safetensors base_model_relation: adapter co2_eq_emissions: emissions: 1605.4 source: CodeCarbon, recorded with ai-impact-accounting training_type: fine-tuning geographical_location: Ontario, Canada hardware_used: 4 x NVIDIA A40 dia_version: '0.1' dia_report: scope: incremental lineage: - model: openai/clip-vit-large-patch14 relation: adapter compute: hardware: gpu: NVIDIA A40 count: 4 duration_gpu_hours: 86.1089 footprint: energy_kwh: value: 24.8049 quality: measured carbon_kgco2eq: value: 1.6054 quality: measured water_liters: value: - 44.649 - 99.22 quality: estimated-from-default-wue context: region: ca-on carbon_intensity: 0.03 wue_l_per_kwh: - 1.8 - 4.0 tool: codecarbon --- # DFD-HR, re-implemented and trained under the MARS protocol [![arXiv](https://img.shields.io/badge/arXiv-2611.12508-b31b1b.svg)](https://arxiv.org/abs/2611.12508) [![Code](https://img.shields.io/badge/GitHub-VectorInstitute%2Fmars-181717.svg?logo=github)](https://github.com/VectorInstitute/mars) [![MARS](https://img.shields.io/badge/%F0%9F%A4%97%20Model-MARS-ffd21e.svg)](https://huggingface.co/vector-institute/mars) [![Original DFD-HR code](https://img.shields.io/badge/Original%20code-Disguiser15%2FDFD--HR-555555.svg?logo=github)](https://github.com/Disguiser15/DFD-HR) This is the DFD-HR baseline from the MARS paper: our re-implementation of DFD-HR (Sun et al., CVPR 2026), trained and evaluated under the same protocol as MARS. It is not the original authors' model or weights; training data, inputs, optimization and checkpoint selection follow the MARS protocol, so its numbers differ from those in the DFD-HR paper. DFD-HR scores each frame with a frozen CLIP ViT-L/14 carrying trainable mixture-of-experts adapters. Every frame is seen twice, as one 224 x 224 global view and as 2 x 2 tiles of 224 x 224 cut from the 448 x 448 frame. In the last blocks, a token router passes only the top-scoring tokens and a layer router lets a view stop early. The clip score is the mean of the frame scores. ## Files | file | contents | size | |---|---|---| | `model.safetensors` | trained adapters, routers and head; the frozen CLIP ViT-L/14 is not included and is downloaded from OpenAI on first use | 415 MB | | `config.yaml` | model and data settings, read by the `mars` package | 1 KB | | `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone | 1.63 GB | This is the checkpoint behind the DFD-HR results in the MARS paper. ## Input 1. Decode the video as RGB and keep every second frame (0, 2, 4, ...). 2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is padded by repeating its last frame; `n_valid` is the number of real frames. 3. Resize each frame to **448 x 448** (square, no crop, bicubic with antialiasing), scale to [0, 1], then to [-1, 1]. The model applies CLIP's normalization itself. 4. The video score is the mean P(manipulated) over its clips. Scores are not calibrated. If you need a decision, choose a threshold on validation data from your own setting. ## Use with safetensors (PyTorch) Requires the code repository: ```bash git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync ``` ```python import torch from huggingface_hub import snapshot_download from safetensors.torch import load_file from mars.config import load_yaml from mars.data.sampling import clips_for_video from mars.data.transforms import preprocess from mars.data.video import count_frames, read_frames from mars.models.factory import build_detector folder = snapshot_download("vector-institute/mars-dfd-hr", allow_patterns=["config.yaml", "model.safetensors"]) raw = load_yaml(f"{folder}/config.yaml") detector = build_detector(raw["model"]["name"], raw) model = detector.build_network() # downloads CLIP ViT-L/14 on first use missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False) assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder is absent model = model.cuda().eval() data = detector.cfg.data def score(path: str) -> float: """Mean P(manipulated) over the clips of one video.""" probs = [] for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride): frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda() valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda() with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): logits = model(frames, valid).logits probs.append(logits.float().softmax(-1)[0, 1].item()) return sum(probs) / len(probs) print(score("video.mp4")) ``` ## Use with ONNX (no PyTorch, no code repository) ```bash pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU ``` | name | shape, type | meaning | |---|---|---| | input `frames` | (1, 64, 3, 448, 448), float32 | one clip in [-1, 1] | | input `n_valid` | (1,), int64 | number of real frames in the clip, 1 to 64 | | output `fake_prob` | (1,), float32 | P(manipulated) for the clip | | output `logits` | (1, 2), float32 | authentic and manipulated logits | Clip length, frame stride and image size are also stored in the file's metadata. ```python import av import numpy as np import onnxruntime as ort from huggingface_hub import hf_hub_download from PIL import Image session = ort.InferenceSession( hf_hub_download("vector-institute/mars-dfd-hr", "model.onnx"), providers=["CUDAExecutionProvider", "CPUExecutionProvider"], ) meta = session.get_modelmeta().custom_metadata_map T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size")) def to_input(rgb: np.ndarray) -> np.ndarray: """(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1].""" x = rgb.astype(np.float32) / 255.0 if x.shape[:2] != (S, S): channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)] x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0) return ((x - 0.5) / 0.5).transpose(2, 0, 1) def score(path: str) -> float: """Mean P(manipulated) over the clips of one video.""" with av.open(path) as container: kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0] probs = [] for start in range(0, len(kept), T): clip = kept[start : start + T] n_valid = len(clip) clip = clip + [clip[-1]] * (T - n_valid) inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)} probs.append(float(session.run(["fake_prob"], inputs)[0][0])) return float(np.mean(probs)) print(score("video.mp4")) ``` The ONNX graph runs in float32, while the paper's evaluation ran PyTorch in bfloat16, so scores differ slightly. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with CUDA is installed, `import torch` before creating the session also works. ## Training | | | |---|---| | training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) | | validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3) | | clips | 64 frames, every second frame, full frames at 448 x 448 | | encoder | OpenAI CLIP ViT-L/14 at 224, frozen; MoE adapters (4 experts) after every attention and MLP, token and layer routing in blocks 21 to 23 | | loss | cross-entropy plus the ranking term (weight 0.1) | | optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0 | | hardware | 4 x NVIDIA A40 | ## Results Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's score is the mean over its clips. | test set | videos | AUROC | AP | |---|---|---|---| | FaceForensics++ (test split) | 700 | 99.80 | 99.95 | | Celeb-DF v2 | 518 | 93.89 | 96.70 | | DeepfakeTIMIT | 960 | 99.59 | 99.82 | | UADFV | 98 | 96.25 | 96.96 | | HiFiFace (paired with FF++ targets) | 280 | 86.26 | 88.91 | | GenVideo (validation split, 10 generators) | 18,286 | 79.48 | 79.10 | | DF26 (10 generators) | 2,691 | 74.95 | 95.83 | | **average** | | **90.03** | | ## License The weights in this repository are released under Apache 2.0. The frozen encoder is OpenAI's CLIP ViT-L/14, downloaded from OpenAI under its own license. ## Citation Please cite the MARS paper for these weights and the protocol, and the DFD-HR paper for the method. ```bibtex @misc{radwan2026marsmultiscaleappearanceresidualstreams, title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection}, author={Ahmed Y. Radwan and Shaina Raza}, year={2026}, eprint={2611.12508}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2611.12508}, } @InProceedings{Sun_2026_CVPR, author = {Sun, Jiamu and Yan, Zhiyuan and Zhang, Ke-Yue and Yao, Taiping and Ding, Shouhong}, title = {DFD-HR: Generalizable Deepfake Detection via Hierarchical Routing Learning}, booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, month = {June}, year = {2026}, pages = {13984-13995} } ```