mars-dfd-hr / README.md
ahmedyradwan02's picture
Model card: badge links
640229d verified
|
Raw History Blame Contribute Delete
9.83 kB
---
license: apache-2.0
pipeline_tag: video-classification
base_model: openai/clip-vit-large-patch14
tags:
- deepfake-detection
- video-forgery-detection
- video-classification
- onnx
- safetensors
base_model_relation: adapter
co2_eq_emissions:
emissions: 1605.4
source: CodeCarbon, recorded with ai-impact-accounting
training_type: fine-tuning
geographical_location: Ontario, Canada
hardware_used: 4 x NVIDIA A40
dia_version: '0.1'
dia_report:
scope: incremental
lineage:
- model: openai/clip-vit-large-patch14
relation: adapter
compute:
hardware:
gpu: NVIDIA A40
count: 4
duration_gpu_hours: 86.1089
footprint:
energy_kwh:
value: 24.8049
quality: measured
carbon_kgco2eq:
value: 1.6054
quality: measured
water_liters:
value:
- 44.649
- 99.22
quality: estimated-from-default-wue
context:
region: ca-on
carbon_intensity: 0.03
wue_l_per_kwh:
- 1.8
- 4.0
tool: codecarbon
---
# DFD-HR, re-implemented and trained under the MARS protocol
[![arXiv](https://img.shields.io/badge/arXiv-2611.12508-b31b1b.svg)](https://arxiv.org/abs/2611.12508)
[![Code](https://img.shields.io/badge/GitHub-VectorInstitute%2Fmars-181717.svg?logo=github)](https://github.com/VectorInstitute/mars)
[![MARS](https://img.shields.io/badge/%F0%9F%A4%97%20Model-MARS-ffd21e.svg)](https://huggingface.co/vector-institute/mars)
[![Original DFD-HR code](https://img.shields.io/badge/Original%20code-Disguiser15%2FDFD--HR-555555.svg?logo=github)](https://github.com/Disguiser15/DFD-HR)
This is the DFD-HR baseline from the MARS paper: our re-implementation of DFD-HR
(Sun et al., CVPR 2026), trained and evaluated under the same protocol as MARS. It is not
the original authors' model or weights; training data, inputs, optimization and checkpoint
selection follow the MARS protocol, so its numbers differ from those in the DFD-HR paper.
DFD-HR scores each frame with a frozen CLIP ViT-L/14 carrying trainable mixture-of-experts
adapters. Every frame is seen twice, as one 224 x 224 global view and as 2 x 2 tiles of
224 x 224 cut from the 448 x 448 frame. In the last blocks, a token router passes only the
top-scoring tokens and a layer router lets a view stop early. The clip score is the mean of
the frame scores.
## Files
| file | contents | size |
|---|---|---|
| `model.safetensors` | trained adapters, routers and head; the frozen CLIP ViT-L/14 is not included and is downloaded from OpenAI on first use | 415 MB |
| `config.yaml` | model and data settings, read by the `mars` package | 1 KB |
| `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone | 1.63 GB |
This is the checkpoint behind the DFD-HR results in the MARS paper.
## Input
1. Decode the video as RGB and keep every second frame (0, 2, 4, ...).
2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is
padded by repeating its last frame; `n_valid` is the number of real frames.
3. Resize each frame to **448 x 448** (square, no crop, bicubic with antialiasing), scale to
[0, 1], then to [-1, 1]. The model applies CLIP's normalization itself.
4. The video score is the mean P(manipulated) over its clips.
Scores are not calibrated. If you need a decision, choose a threshold on validation data from
your own setting.
## Use with safetensors (PyTorch)
Requires the code repository:
```bash
git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync
```
```python
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from mars.config import load_yaml
from mars.data.sampling import clips_for_video
from mars.data.transforms import preprocess
from mars.data.video import count_frames, read_frames
from mars.models.factory import build_detector
folder = snapshot_download("vector-institute/mars-dfd-hr", allow_patterns=["config.yaml", "model.safetensors"])
raw = load_yaml(f"{folder}/config.yaml")
detector = build_detector(raw["model"]["name"], raw)
model = detector.build_network() # downloads CLIP ViT-L/14 on first use
missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False)
assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder is absent
model = model.cuda().eval()
data = detector.cfg.data
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
probs = []
for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride):
frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda()
valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
logits = model(frames, valid).logits
probs.append(logits.float().softmax(-1)[0, 1].item())
return sum(probs) / len(probs)
print(score("video.mp4"))
```
## Use with ONNX (no PyTorch, no code repository)
```bash
pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU
```
| name | shape, type | meaning |
|---|---|---|
| input `frames` | (1, 64, 3, 448, 448), float32 | one clip in [-1, 1] |
| input `n_valid` | (1,), int64 | number of real frames in the clip, 1 to 64 |
| output `fake_prob` | (1,), float32 | P(manipulated) for the clip |
| output `logits` | (1, 2), float32 | authentic and manipulated logits |
Clip length, frame stride and image size are also stored in the file's metadata.
```python
import av
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image
session = ort.InferenceSession(
hf_hub_download("vector-institute/mars-dfd-hr", "model.onnx"),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
meta = session.get_modelmeta().custom_metadata_map
T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size"))
def to_input(rgb: np.ndarray) -> np.ndarray:
"""(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1]."""
x = rgb.astype(np.float32) / 255.0
if x.shape[:2] != (S, S):
channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)]
x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0)
return ((x - 0.5) / 0.5).transpose(2, 0, 1)
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
with av.open(path) as container:
kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0]
probs = []
for start in range(0, len(kept), T):
clip = kept[start : start + T]
n_valid = len(clip)
clip = clip + [clip[-1]] * (T - n_valid)
inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)}
probs.append(float(session.run(["fake_prob"], inputs)[0][0]))
return float(np.mean(probs))
print(score("video.mp4"))
```
The ONNX graph runs in float32, while the paper's evaluation ran PyTorch in bfloat16, so
scores differ slightly. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with
CUDA is installed, `import torch` before creating the session also works.
## Training
| | |
|---|---|
| training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) |
| validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3) |
| clips | 64 frames, every second frame, full frames at 448 x 448 |
| encoder | OpenAI CLIP ViT-L/14 at 224, frozen; MoE adapters (4 experts) after every attention and MLP, token and layer routing in blocks 21 to 23 |
| loss | cross-entropy plus the ranking term (weight 0.1) |
| optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0 |
| hardware | 4 x NVIDIA A40 |
## Results
Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's
score is the mean over its clips.
| test set | videos | AUROC | AP |
|---|---|---|---|
| FaceForensics++ (test split) | 700 | 99.80 | 99.95 |
| Celeb-DF v2 | 518 | 93.89 | 96.70 |
| DeepfakeTIMIT | 960 | 99.59 | 99.82 |
| UADFV | 98 | 96.25 | 96.96 |
| HiFiFace (paired with FF++ targets) | 280 | 86.26 | 88.91 |
| GenVideo (validation split, 10 generators) | 18,286 | 79.48 | 79.10 |
| DF26 (10 generators) | 2,691 | 74.95 | 95.83 |
| **average** | | **90.03** | |
## License
The weights in this repository are released under Apache 2.0. The frozen encoder is OpenAI's
CLIP ViT-L/14, downloaded from OpenAI under its own license.
## Citation
Please cite the MARS paper for these weights and the protocol, and the DFD-HR paper for the
method.
```bibtex
@misc{radwan2026marsmultiscaleappearanceresidualstreams,
title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection},
author={Ahmed Y. Radwan and Shaina Raza},
year={2026},
eprint={2611.12508},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2611.12508},
}
@InProceedings{Sun_2026_CVPR,
author = {Sun, Jiamu and Yan, Zhiyuan and Zhang, Ke-Yue and Yao, Taiping and Ding, Shouhong},
title = {DFD-HR: Generalizable Deepfake Detection via Hierarchical Routing Learning},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2026},
pages = {13984-13995}
}
```