File size: 9,827 Bytes
878ea79 640229d 878ea79 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 | ---
license: apache-2.0
pipeline_tag: video-classification
base_model: openai/clip-vit-large-patch14
tags:
- deepfake-detection
- video-forgery-detection
- video-classification
- onnx
- safetensors
base_model_relation: adapter
co2_eq_emissions:
emissions: 1605.4
source: CodeCarbon, recorded with ai-impact-accounting
training_type: fine-tuning
geographical_location: Ontario, Canada
hardware_used: 4 x NVIDIA A40
dia_version: '0.1'
dia_report:
scope: incremental
lineage:
- model: openai/clip-vit-large-patch14
relation: adapter
compute:
hardware:
gpu: NVIDIA A40
count: 4
duration_gpu_hours: 86.1089
footprint:
energy_kwh:
value: 24.8049
quality: measured
carbon_kgco2eq:
value: 1.6054
quality: measured
water_liters:
value:
- 44.649
- 99.22
quality: estimated-from-default-wue
context:
region: ca-on
carbon_intensity: 0.03
wue_l_per_kwh:
- 1.8
- 4.0
tool: codecarbon
---
# DFD-HR, re-implemented and trained under the MARS protocol
[](https://arxiv.org/abs/2611.12508)
[](https://github.com/VectorInstitute/mars)
[](https://huggingface.co/vector-institute/mars)
[](https://github.com/Disguiser15/DFD-HR)
This is the DFD-HR baseline from the MARS paper: our re-implementation of DFD-HR
(Sun et al., CVPR 2026), trained and evaluated under the same protocol as MARS. It is not
the original authors' model or weights; training data, inputs, optimization and checkpoint
selection follow the MARS protocol, so its numbers differ from those in the DFD-HR paper.
DFD-HR scores each frame with a frozen CLIP ViT-L/14 carrying trainable mixture-of-experts
adapters. Every frame is seen twice, as one 224 x 224 global view and as 2 x 2 tiles of
224 x 224 cut from the 448 x 448 frame. In the last blocks, a token router passes only the
top-scoring tokens and a layer router lets a view stop early. The clip score is the mean of
the frame scores.
## Files
| file | contents | size |
|---|---|---|
| `model.safetensors` | trained adapters, routers and head; the frozen CLIP ViT-L/14 is not included and is downloaded from OpenAI on first use | 415 MB |
| `config.yaml` | model and data settings, read by the `mars` package | 1 KB |
| `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone | 1.63 GB |
This is the checkpoint behind the DFD-HR results in the MARS paper.
## Input
1. Decode the video as RGB and keep every second frame (0, 2, 4, ...).
2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is
padded by repeating its last frame; `n_valid` is the number of real frames.
3. Resize each frame to **448 x 448** (square, no crop, bicubic with antialiasing), scale to
[0, 1], then to [-1, 1]. The model applies CLIP's normalization itself.
4. The video score is the mean P(manipulated) over its clips.
Scores are not calibrated. If you need a decision, choose a threshold on validation data from
your own setting.
## Use with safetensors (PyTorch)
Requires the code repository:
```bash
git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync
```
```python
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from mars.config import load_yaml
from mars.data.sampling import clips_for_video
from mars.data.transforms import preprocess
from mars.data.video import count_frames, read_frames
from mars.models.factory import build_detector
folder = snapshot_download("vector-institute/mars-dfd-hr", allow_patterns=["config.yaml", "model.safetensors"])
raw = load_yaml(f"{folder}/config.yaml")
detector = build_detector(raw["model"]["name"], raw)
model = detector.build_network() # downloads CLIP ViT-L/14 on first use
missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False)
assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder is absent
model = model.cuda().eval()
data = detector.cfg.data
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
probs = []
for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride):
frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda()
valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
logits = model(frames, valid).logits
probs.append(logits.float().softmax(-1)[0, 1].item())
return sum(probs) / len(probs)
print(score("video.mp4"))
```
## Use with ONNX (no PyTorch, no code repository)
```bash
pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU
```
| name | shape, type | meaning |
|---|---|---|
| input `frames` | (1, 64, 3, 448, 448), float32 | one clip in [-1, 1] |
| input `n_valid` | (1,), int64 | number of real frames in the clip, 1 to 64 |
| output `fake_prob` | (1,), float32 | P(manipulated) for the clip |
| output `logits` | (1, 2), float32 | authentic and manipulated logits |
Clip length, frame stride and image size are also stored in the file's metadata.
```python
import av
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image
session = ort.InferenceSession(
hf_hub_download("vector-institute/mars-dfd-hr", "model.onnx"),
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
meta = session.get_modelmeta().custom_metadata_map
T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size"))
def to_input(rgb: np.ndarray) -> np.ndarray:
"""(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1]."""
x = rgb.astype(np.float32) / 255.0
if x.shape[:2] != (S, S):
channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)]
x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0)
return ((x - 0.5) / 0.5).transpose(2, 0, 1)
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
with av.open(path) as container:
kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0]
probs = []
for start in range(0, len(kept), T):
clip = kept[start : start + T]
n_valid = len(clip)
clip = clip + [clip[-1]] * (T - n_valid)
inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)}
probs.append(float(session.run(["fake_prob"], inputs)[0][0]))
return float(np.mean(probs))
print(score("video.mp4"))
```
The ONNX graph runs in float32, while the paper's evaluation ran PyTorch in bfloat16, so
scores differ slightly. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with
CUDA is installed, `import torch` before creating the session also works.
## Training
| | |
|---|---|
| training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) |
| validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3) |
| clips | 64 frames, every second frame, full frames at 448 x 448 |
| encoder | OpenAI CLIP ViT-L/14 at 224, frozen; MoE adapters (4 experts) after every attention and MLP, token and layer routing in blocks 21 to 23 |
| loss | cross-entropy plus the ranking term (weight 0.1) |
| optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0 |
| hardware | 4 x NVIDIA A40 |
## Results
Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's
score is the mean over its clips.
| test set | videos | AUROC | AP |
|---|---|---|---|
| FaceForensics++ (test split) | 700 | 99.80 | 99.95 |
| Celeb-DF v2 | 518 | 93.89 | 96.70 |
| DeepfakeTIMIT | 960 | 99.59 | 99.82 |
| UADFV | 98 | 96.25 | 96.96 |
| HiFiFace (paired with FF++ targets) | 280 | 86.26 | 88.91 |
| GenVideo (validation split, 10 generators) | 18,286 | 79.48 | 79.10 |
| DF26 (10 generators) | 2,691 | 74.95 | 95.83 |
| **average** | | **90.03** | |
## License
The weights in this repository are released under Apache 2.0. The frozen encoder is OpenAI's
CLIP ViT-L/14, downloaded from OpenAI under its own license.
## Citation
Please cite the MARS paper for these weights and the protocol, and the DFD-HR paper for the
method.
```bibtex
@misc{radwan2026marsmultiscaleappearanceresidualstreams,
title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection},
author={Ahmed Y. Radwan and Shaina Raza},
year={2026},
eprint={2611.12508},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2611.12508},
}
@InProceedings{Sun_2026_CVPR,
author = {Sun, Jiamu and Yan, Zhiyuan and Zhang, Ke-Yue and Yao, Taiping and Ding, Shouhong},
title = {DFD-HR: Generalizable Deepfake Detection via Hierarchical Routing Learning},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2026},
pages = {13984-13995}
}
```
|