|
Download README.md from vector-institute/mars-dfd-hr: direct link, hf CLI and curl.
- Browser
- Download file 9.83 kB
-
https://huggingface.co/vector-institute/mars-dfd-hr/resolve/main/README.md
- Command line
-
hf download hf://vector-institute/mars-dfd-hr/README.md
-
curl -L -o README.md https://huggingface.co/vector-institute/mars-dfd-hr/resolve/main/README.md
9.83 kB
| license: apache-2.0 | |
| pipeline_tag: video-classification | |
| base_model: openai/clip-vit-large-patch14 | |
| tags: | |
| - deepfake-detection | |
| - video-forgery-detection | |
| - video-classification | |
| - onnx | |
| - safetensors | |
| base_model_relation: adapter | |
| co2_eq_emissions: | |
| emissions: 1605.4 | |
| source: CodeCarbon, recorded with ai-impact-accounting | |
| training_type: fine-tuning | |
| geographical_location: Ontario, Canada | |
| hardware_used: 4 x NVIDIA A40 | |
| dia_version: '0.1' | |
| dia_report: | |
| scope: incremental | |
| lineage: | |
| - model: openai/clip-vit-large-patch14 | |
| relation: adapter | |
| compute: | |
| hardware: | |
| gpu: NVIDIA A40 | |
| count: 4 | |
| duration_gpu_hours: 86.1089 | |
| footprint: | |
| energy_kwh: | |
| value: 24.8049 | |
| quality: measured | |
| carbon_kgco2eq: | |
| value: 1.6054 | |
| quality: measured | |
| water_liters: | |
| value: | |
| - 44.649 | |
| - 99.22 | |
| quality: estimated-from-default-wue | |
| context: | |
| region: ca-on | |
| carbon_intensity: 0.03 | |
| wue_l_per_kwh: | |
| - 1.8 | |
| - 4.0 | |
| tool: codecarbon | |
| # DFD-HR, re-implemented and trained under the MARS protocol | |
| [](https://arxiv.org/abs/2611.12508) | |
| [](https://github.com/VectorInstitute/mars) | |
| [](https://huggingface.co/vector-institute/mars) | |
| [](https://github.com/Disguiser15/DFD-HR) | |
| This is the DFD-HR baseline from the MARS paper: our re-implementation of DFD-HR | |
| (Sun et al., CVPR 2026), trained and evaluated under the same protocol as MARS. It is not | |
| the original authors' model or weights; training data, inputs, optimization and checkpoint | |
| selection follow the MARS protocol, so its numbers differ from those in the DFD-HR paper. | |
| DFD-HR scores each frame with a frozen CLIP ViT-L/14 carrying trainable mixture-of-experts | |
| adapters. Every frame is seen twice, as one 224 x 224 global view and as 2 x 2 tiles of | |
| 224 x 224 cut from the 448 x 448 frame. In the last blocks, a token router passes only the | |
| top-scoring tokens and a layer router lets a view stop early. The clip score is the mean of | |
| the frame scores. | |
| ## Files | |
| | file | contents | size | | |
| |---|---|---| | |
| | `model.safetensors` | trained adapters, routers and head; the frozen CLIP ViT-L/14 is not included and is downloaded from OpenAI on first use | 415 MB | | |
| | `config.yaml` | model and data settings, read by the `mars` package | 1 KB | | |
| | `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone | 1.63 GB | | |
| This is the checkpoint behind the DFD-HR results in the MARS paper. | |
| ## Input | |
| 1. Decode the video as RGB and keep every second frame (0, 2, 4, ...). | |
| 2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is | |
| padded by repeating its last frame; `n_valid` is the number of real frames. | |
| 3. Resize each frame to **448 x 448** (square, no crop, bicubic with antialiasing), scale to | |
| [0, 1], then to [-1, 1]. The model applies CLIP's normalization itself. | |
| 4. The video score is the mean P(manipulated) over its clips. | |
| Scores are not calibrated. If you need a decision, choose a threshold on validation data from | |
| your own setting. | |
| ## Use with safetensors (PyTorch) | |
| Requires the code repository: | |
| ```bash | |
| git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync | |
| ``` | |
| ```python | |
| import torch | |
| from huggingface_hub import snapshot_download | |
| from safetensors.torch import load_file | |
| from mars.config import load_yaml | |
| from mars.data.sampling import clips_for_video | |
| from mars.data.transforms import preprocess | |
| from mars.data.video import count_frames, read_frames | |
| from mars.models.factory import build_detector | |
| folder = snapshot_download("vector-institute/mars-dfd-hr", allow_patterns=["config.yaml", "model.safetensors"]) | |
| raw = load_yaml(f"{folder}/config.yaml") | |
| detector = build_detector(raw["model"]["name"], raw) | |
| model = detector.build_network() # downloads CLIP ViT-L/14 on first use | |
| missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False) | |
| assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder is absent | |
| model = model.cuda().eval() | |
| data = detector.cfg.data | |
| def score(path: str) -> float: | |
| """Mean P(manipulated) over the clips of one video.""" | |
| probs = [] | |
| for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride): | |
| frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda() | |
| valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda() | |
| with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): | |
| logits = model(frames, valid).logits | |
| probs.append(logits.float().softmax(-1)[0, 1].item()) | |
| return sum(probs) / len(probs) | |
| print(score("video.mp4")) | |
| ``` | |
| ## Use with ONNX (no PyTorch, no code repository) | |
| ```bash | |
| pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU | |
| ``` | |
| | name | shape, type | meaning | | |
| |---|---|---| | |
| | input `frames` | (1, 64, 3, 448, 448), float32 | one clip in [-1, 1] | | |
| | input `n_valid` | (1,), int64 | number of real frames in the clip, 1 to 64 | | |
| | output `fake_prob` | (1,), float32 | P(manipulated) for the clip | | |
| | output `logits` | (1, 2), float32 | authentic and manipulated logits | | |
| Clip length, frame stride and image size are also stored in the file's metadata. | |
| ```python | |
| import av | |
| import numpy as np | |
| import onnxruntime as ort | |
| from huggingface_hub import hf_hub_download | |
| from PIL import Image | |
| session = ort.InferenceSession( | |
| hf_hub_download("vector-institute/mars-dfd-hr", "model.onnx"), | |
| providers=["CUDAExecutionProvider", "CPUExecutionProvider"], | |
| ) | |
| meta = session.get_modelmeta().custom_metadata_map | |
| T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size")) | |
| def to_input(rgb: np.ndarray) -> np.ndarray: | |
| """(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1].""" | |
| x = rgb.astype(np.float32) / 255.0 | |
| if x.shape[:2] != (S, S): | |
| channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)] | |
| x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0) | |
| return ((x - 0.5) / 0.5).transpose(2, 0, 1) | |
| def score(path: str) -> float: | |
| """Mean P(manipulated) over the clips of one video.""" | |
| with av.open(path) as container: | |
| kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0] | |
| probs = [] | |
| for start in range(0, len(kept), T): | |
| clip = kept[start : start + T] | |
| n_valid = len(clip) | |
| clip = clip + [clip[-1]] * (T - n_valid) | |
| inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)} | |
| probs.append(float(session.run(["fake_prob"], inputs)[0][0])) | |
| return float(np.mean(probs)) | |
| print(score("video.mp4")) | |
| ``` | |
| The ONNX graph runs in float32, while the paper's evaluation ran PyTorch in bfloat16, so | |
| scores differ slightly. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with | |
| CUDA is installed, `import torch` before creating the session also works. | |
| ## Training | |
| | | | | |
| |---|---| | |
| | training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) | | |
| | validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3) | | |
| | clips | 64 frames, every second frame, full frames at 448 x 448 | | |
| | encoder | OpenAI CLIP ViT-L/14 at 224, frozen; MoE adapters (4 experts) after every attention and MLP, token and layer routing in blocks 21 to 23 | | |
| | loss | cross-entropy plus the ranking term (weight 0.1) | | |
| | optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0 | | |
| | hardware | 4 x NVIDIA A40 | | |
| ## Results | |
| Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's | |
| score is the mean over its clips. | |
| | test set | videos | AUROC | AP | | |
| |---|---|---|---| | |
| | FaceForensics++ (test split) | 700 | 99.80 | 99.95 | | |
| | Celeb-DF v2 | 518 | 93.89 | 96.70 | | |
| | DeepfakeTIMIT | 960 | 99.59 | 99.82 | | |
| | UADFV | 98 | 96.25 | 96.96 | | |
| | HiFiFace (paired with FF++ targets) | 280 | 86.26 | 88.91 | | |
| | GenVideo (validation split, 10 generators) | 18,286 | 79.48 | 79.10 | | |
| | DF26 (10 generators) | 2,691 | 74.95 | 95.83 | | |
| | **average** | | **90.03** | | | |
| ## License | |
| The weights in this repository are released under Apache 2.0. The frozen encoder is OpenAI's | |
| CLIP ViT-L/14, downloaded from OpenAI under its own license. | |
| ## Citation | |
| Please cite the MARS paper for these weights and the protocol, and the DFD-HR paper for the | |
| method. | |
| ```bibtex | |
| @misc{radwan2026marsmultiscaleappearanceresidualstreams, | |
| title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection}, | |
| author={Ahmed Y. Radwan and Shaina Raza}, | |
| year={2026}, | |
| eprint={2611.12508}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CV}, | |
| url={https://arxiv.org/abs/2611.12508}, | |
| } | |
| @InProceedings{Sun_2026_CVPR, | |
| author = {Sun, Jiamu and Yan, Zhiyuan and Zhang, Ke-Yue and Yao, Taiping and Ding, Shouhong}, | |
| title = {DFD-HR: Generalizable Deepfake Detection via Hierarchical Routing Learning}, | |
| booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, | |
| month = {June}, | |
| year = {2026}, | |
| pages = {13984-13995} | |
| } | |
| ``` | |