--- license: apache-2.0 pipeline_tag: video-classification base_model: timm/vit_pe_core_large_patch14_336.fb tags: - deepfake-detection - video-forgery-detection - video-classification - onnx - safetensors base_model_relation: adapter co2_eq_emissions: emissions: 353.1 source: CodeCarbon, recorded with ai-impact-accounting training_type: fine-tuning geographical_location: Ontario, Canada hardware_used: 4 x NVIDIA A40 dia_version: '0.1' dia_report: scope: incremental lineage: - model: timm/vit_pe_core_large_patch14_336.fb relation: adapter compute: hardware: gpu: NVIDIA A40 count: 4 duration_gpu_hours: 17.3576 footprint: energy_kwh: value: 5.4554 quality: measured carbon_kgco2eq: value: 0.3531 quality: measured water_liters: value: - 9.82 - 21.821 quality: estimated-from-default-wue context: region: ca-on carbon_intensity: 0.03 wue_l_per_kwh: - 1.8 - 4.0 tool: codecarbon --- # MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection [![arXiv](https://img.shields.io/badge/arXiv-2611.12508-b31b1b.svg)](https://arxiv.org/abs/2611.12508) [![Code](https://img.shields.io/badge/GitHub-VectorInstitute%2Fmars-181717.svg?logo=github)](https://github.com/VectorInstitute/mars) [![Project page](https://img.shields.io/badge/Project-page-0b2e6f.svg)](https://vectorinstitute.github.io/mars) MARS is a full-frame video forgery detector that covers both facial manipulation (face swapping, reenactment) and fully generated video. It reads one frozen Perception Encoder (PE-Core-L) in two ways: 1. A **residual path** compares the encoder's intermediate patch features with smoothed copies of themselves, across space and time at several scales, and a temporal transformer models how these differences develop over the clip. 2. A **semantic path** classifies sampled frames with the complete encoder, tuning only its own copies of the LayerNorm parameters. A gate mixes the two paths per clip and returns P(manipulated) for the clip. ## Files | file | contents | size | |---|---|---| | `model.safetensors` | trained weights; the frozen PE-Core-L encoder is not included and is downloaded from timm on first use | 270 MB | | `config.yaml` | model and data settings, read by the `mars` package | 2 KB | | `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone | 1.56 GB | This is the checkpoint behind the MARS results in the paper. ## Input Every clip is built the same way, whichever file you use: 1. Decode the video as RGB and keep every second frame (0, 2, 4, ...). 2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is padded by repeating its last frame; `n_valid` is the number of real frames. 3. Resize each frame to 336 x 336 (square, no crop, bicubic with antialiasing), scale to [0, 1], then to [-1, 1]. 4. The video score is the mean P(manipulated) over its clips. Scores are not calibrated. If you need a decision, choose a threshold on validation data from your own setting. ## Use with safetensors (PyTorch) Requires the code repository: ```bash git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync ``` ```python import torch from huggingface_hub import snapshot_download from safetensors.torch import load_file from mars.config import load_yaml from mars.data.sampling import clips_for_video from mars.data.transforms import preprocess from mars.data.video import count_frames, read_frames from mars.models.factory import build_detector folder = snapshot_download("vector-institute/mars", allow_patterns=["config.yaml", "model.safetensors"]) raw = load_yaml(f"{folder}/config.yaml") detector = build_detector(raw["model"]["name"], raw) model = detector.build_network() # downloads the frozen encoder on first use missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False) assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder is absent model = model.cuda().eval() data = detector.cfg.data def score(path: str) -> float: """Mean P(manipulated) over the clips of one video.""" probs = [] for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride): frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda() valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda() with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): logits = model(frames, valid).logits probs.append(logits.float().softmax(-1)[0, 1].item()) return sum(probs) / len(probs) print(score("video.mp4")) ``` ## Use with ONNX (no PyTorch, no code repository) ```bash pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU ``` | name | shape, type | meaning | |---|---|---| | input `frames` | (1, 64, 3, 336, 336), float32 | one clip in [-1, 1] | | input `n_valid` | (1,), int64 | number of real frames in the clip, 1 to 64 | | output `fake_prob` | (1,), float32 | P(manipulated) for the clip | | output `logits` | (1, 2), float32 | authentic and manipulated logits | Clip length, frame stride and image size are also stored in the file's metadata. ```python import av import numpy as np import onnxruntime as ort from huggingface_hub import hf_hub_download from PIL import Image session = ort.InferenceSession( hf_hub_download("vector-institute/mars", "model.onnx"), providers=["CUDAExecutionProvider", "CPUExecutionProvider"], ) meta = session.get_modelmeta().custom_metadata_map T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size")) def to_input(rgb: np.ndarray) -> np.ndarray: """(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1].""" x = rgb.astype(np.float32) / 255.0 if x.shape[:2] != (S, S): channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)] x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0) return ((x - 0.5) / 0.5).transpose(2, 0, 1) def score(path: str) -> float: """Mean P(manipulated) over the clips of one video.""" with av.open(path) as container: kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0] probs = [] for start in range(0, len(kept), T): clip = kept[start : start + T] n_valid = len(clip) clip = clip + [clip[-1]] * (T - n_valid) inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)} probs.append(float(session.run(["fake_prob"], inputs)[0][0])) return float(np.mean(probs)) print(score("video.mp4")) ``` The ONNX graph runs in float32, while the paper's evaluation ran PyTorch in bfloat16, so scores differ slightly. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with CUDA is installed, `import torch` before creating the session also works. ## Training | | | |---|---| | training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) | | validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3) | | clips | 64 frames, every second frame, full frames at 336 x 336 | | encoder | `vit_pe_core_large_patch14_336.fb`, frozen; the semantic path tunes its own LayerNorm copies | | optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0 | | hardware | 4 x NVIDIA A40 | ## Results Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's score is the mean over its clips. | test set | videos | AUROC | AP | |---|---|---|---| | FaceForensics++ (test split) | 700 | 98.98 | 99.74 | | Celeb-DF v2 | 518 | 91.22 | 94.73 | | DeepfakeTIMIT | 960 | 99.95 | 99.97 | | UADFV | 98 | 96.58 | 97.29 | | HiFiFace (paired with FF++ targets) | 280 | 84.05 | 85.95 | | GenVideo (validation split, 10 generators) | 18,286 | 93.44 | 92.60 | | DF26 (10 generators) | 2,691 | 75.81 | 95.76 | | **average** | | **91.43** | | ## License The weights in this repository are released under Apache 2.0. The frozen encoder is Meta's Perception Encoder (PE-Core-L), downloaded from timm under its own license (Apache 2.0). ## Citation ```bibtex @misc{radwan2026marsmultiscaleappearanceresidualstreams, title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection}, author={Ahmed Y. Radwan and Shaina Raza}, year={2026}, eprint={2611.12508}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2611.12508}, } ```