mars / README.md
ahmedyradwan02's picture
ONNX: batch axis, attribution outputs, float16 file, usual ranges, serving facts
9baf809 verified
|
Raw History Blame Contribute Delete
11.2 kB
---
license: apache-2.0
pipeline_tag: video-classification
base_model: timm/vit_pe_core_large_patch14_336.fb
tags:
- deepfake-detection
- video-forgery-detection
- video-classification
- onnx
- safetensors
base_model_relation: adapter
co2_eq_emissions:
emissions: 353.1
source: CodeCarbon, recorded with ai-impact-accounting
training_type: fine-tuning
geographical_location: Ontario, Canada
hardware_used: 4 x NVIDIA A40
dia_version: '0.1'
dia_report:
scope: incremental
lineage:
- model: timm/vit_pe_core_large_patch14_336.fb
relation: adapter
compute:
hardware:
gpu: NVIDIA A40
count: 4
duration_gpu_hours: 17.3576
footprint:
energy_kwh:
value: 5.4554
quality: measured
carbon_kgco2eq:
value: 0.3531
quality: measured
water_liters:
value:
- 9.82
- 21.821
quality: estimated-from-default-wue
context:
region: ca-on
carbon_intensity: 0.03
wue_l_per_kwh:
- 1.8
- 4.0
tool: codecarbon
---
# MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection
[![arXiv](https://img.shields.io/badge/arXiv-2611.12508-b31b1b.svg)](https://arxiv.org/abs/2611.12508)
[![Code](https://img.shields.io/badge/GitHub-VectorInstitute%2Fmars-181717.svg?logo=github)](https://github.com/VectorInstitute/mars)
[![Project page](https://img.shields.io/badge/Project-page-0b2e6f.svg)](https://vectorinstitute.github.io/mars)
MARS is a full-frame video forgery detector that covers both facial manipulation (face
swapping, reenactment) and fully generated video. It reads one frozen Perception Encoder
(PE-Core-L) in two ways:
1. A **residual path** compares the encoder's intermediate patch features with smoothed
copies of themselves, across space and time at several scales, and a temporal transformer
models how these differences develop over the clip.
2. A **semantic path** classifies sampled frames with the complete encoder, tuning only its
own copies of the LayerNorm parameters.
A gate mixes the two paths per clip and returns P(manipulated) for the clip.
## Files
| file | contents | size |
|---|---|---|
| `model.safetensors` | trained weights; the frozen PE-Core-L encoder is not included and is downloaded from timm on first use | 270 MB |
| `config.yaml` | model and data settings, read by the `mars` package | 2 KB |
| `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone, float32 | 1.59 GB |
| `model.fp16.onnx` | the same graph in float16, inputs and outputs still float32; less GPU memory | 0.80 GB |
| `usual_ranges.json` | percentiles of the ONNX video score on the authentic and manipulated test videos, with `model.onnx` | 3 KB |
| `usual_ranges.fp16.json` | the same with `model.fp16.onnx` | 3 KB |
| `serving.json` | measured GPU memory and time per clip with ONNX Runtime (both files, batch 1 and 2), opset, outputs, encoder chunk, ONNX Runtime version | 1 KB |
This is the checkpoint behind the MARS results in the paper.
## Input
Every clip is built the same way, whichever file you use:
1. Decode the video as RGB and keep every second frame (0, 2, 4, ...).
2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is
padded by repeating its last frame; `n_valid` is the number of real frames.
3. Resize each frame to 336 x 336 (square, no crop, bicubic with antialiasing), scale to
[0, 1], then to [-1, 1].
4. The video score is the mean P(manipulated) over its clips.
Scores are not calibrated. If you need a decision, choose a threshold on validation data from
your own setting.
## Use with safetensors (PyTorch)
Requires the code repository:
```bash
git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync
```
```python
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from mars.config import load_yaml
from mars.data.sampling import clips_for_video
from mars.data.transforms import preprocess
from mars.data.video import count_frames, read_frames
from mars.models.factory import build_detector
folder = snapshot_download("vector-institute/mars", allow_patterns=["config.yaml", "model.safetensors"])
raw = load_yaml(f"{folder}/config.yaml")
detector = build_detector(raw["model"]["name"], raw)
model = detector.build_network() # downloads the frozen encoder on first use
missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False)
assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder is absent
model = model.cuda().eval()
data = detector.cfg.data
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
probs = []
for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride):
frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda()
valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
logits = model(frames, valid).logits
probs.append(logits.float().softmax(-1)[0, 1].item())
return sum(probs) / len(probs)
print(score("video.mp4"))
```
## Use with ONNX (no PyTorch, no code repository)
```bash
pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU
```
| name | shape, type | meaning |
|---|---|---|
| input `frames` | (B, 64, 3, 336, 336), float32 | B clips in [-1, 1] |
| input `n_valid` | (B,), int64 | number of real frames in each clip, 1 to 64 |
| output `fake_prob` | (B,), float32 | P(manipulated) for each clip |
| output `logits` | (B, 2), float32 | authentic and manipulated logits |
| output `frame_contrib` | (B, 64), float32 | signed contribution of each frame to the clip's fake-minus-real logit margin; positive pushes toward manipulated, 0 on padded frames |
| output `patch_contrib` | (B, 64, 24, 24), float32 | signed contribution of each frame's 14 x 14-pixel patches through the residual path, same units, 0 on padded frames |
| output `gate` | (B, 2), float32 | weight given to the residual and the semantic path; the two sum to 1 |
The batch size B is free. Clip length, frame stride and image size are also stored in the
file's metadata.
Both files take the full 64-frame clip. Inside the graph the encoder processes 16 frames of
each clip at a time; frames are encoded independently, so this lowers memory without
changing the outputs. On an A100, one clip peaks at about 6.8 GiB of GPU memory with
`model.onnx` and 2.9 GiB with `model.fp16.onnx` (details in [`serving.json`](serving.json)).
The float16 video scores are within 0.003 of float32 on UADFV and Celeb-DF. Use `model.onnx`
on CPU, or as the source for a TensorRT engine.
The contributions are those behind the paper's figures and faithfulness tests: the clip's
margin (logit of `fake_prob`) equals the sum of `frame_contrib` plus a per-clip constant.
A frame's `frame_contrib` is the sum of its `patch_contrib` cells, plus, for the 16 frames
the semantic path samples, that path's share.
[`usual_ranges.json`](usual_ranges.json) gives the percentiles (0 to 100) of the video score
on the authentic and on the manipulated test videos listed under Results, computed with
`model.onnx`; [`usual_ranges.fp16.json`](usual_ranges.fp16.json) is the same for `model.fp16.onnx`.
Use them to place a score in context, not as a threshold.
```python
import av
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image
session = ort.InferenceSession(
hf_hub_download("vector-institute/mars", "model.onnx"), # or "model.fp16.onnx" for less GPU memory
providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
meta = session.get_modelmeta().custom_metadata_map
T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size"))
def to_input(rgb: np.ndarray) -> np.ndarray:
"""(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1]."""
x = rgb.astype(np.float32) / 255.0
if x.shape[:2] != (S, S):
channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)]
x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0)
return ((x - 0.5) / 0.5).transpose(2, 0, 1)
def score(path: str) -> float:
"""Mean P(manipulated) over the clips of one video."""
with av.open(path) as container:
kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0]
probs = []
for start in range(0, len(kept), T):
clip = kept[start : start + T]
n_valid = len(clip)
clip = clip + [clip[-1]] * (T - n_valid)
inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)}
probs.append(float(session.run(["fake_prob"], inputs)[0][0]))
return float(np.mean(probs))
print(score("video.mp4"))
```
Neither ONNX file runs in bfloat16 like the evaluation behind Results, so scores differ
slightly from that table. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with
CUDA is installed, `import torch` before creating the session also works.
## Training
| | |
|---|---|
| training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) |
| validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3) |
| clips | 64 frames, every second frame, full frames at 336 x 336 |
| encoder | `vit_pe_core_large_patch14_336.fb`, frozen; the semantic path tunes its own LayerNorm copies |
| optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0 |
| hardware | 4 x NVIDIA A40 |
## Results
Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's
score is the mean over its clips.
| test set | videos | AUROC | AP |
|---|---|---|---|
| FaceForensics++ (test split) | 700 | 98.98 | 99.74 |
| Celeb-DF v2 | 518 | 91.22 | 94.73 |
| DeepfakeTIMIT | 960 | 99.95 | 99.97 |
| UADFV | 98 | 96.58 | 97.29 |
| HiFiFace (paired with FF++ targets) | 280 | 84.05 | 85.95 |
| GenVideo (validation split, 10 generators) | 18,286 | 93.44 | 92.60 |
| DF26 (10 generators) | 2,691 | 75.81 | 95.76 |
| **average** | | **91.43** | |
## License
The weights in this repository are released under Apache 2.0. The frozen encoder is Meta's
Perception Encoder (PE-Core-L), downloaded from timm under its own license (Apache 2.0).
## Citation
```bibtex
@misc{radwan2026marsmultiscaleappearanceresidualstreams,
title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection},
author={Ahmed Y. Radwan and Shaina Raza},
year={2026},
eprint={2611.12508},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2611.12508},
}
```