MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection

arXiv Code Project page

MARS is a full-frame video forgery detector that covers both facial manipulation (face swapping, reenactment) and fully generated video. It reads one frozen Perception Encoder (PE-Core-L) in two ways:

  1. A residual path compares the encoder's intermediate patch features with smoothed copies of themselves, across space and time at several scales, and a temporal transformer models how these differences develop over the clip.
  2. A semantic path classifies sampled frames with the complete encoder, tuning only its own copies of the LayerNorm parameters.

A gate mixes the two paths per clip and returns P(manipulated) for the clip.

Files

file contents size
model.safetensors trained weights; the frozen PE-Core-L encoder is not included and is downloaded from timm on first use 270 MB
config.yaml model and data settings, read by the mars package 2 KB
model.onnx complete graph, frozen encoder included; runs with ONNX Runtime alone 1.56 GB

This is the checkpoint behind the MARS results in the paper.

Input

Every clip is built the same way, whichever file you use:

  1. Decode the video as RGB and keep every second frame (0, 2, 4, ...).
  2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is padded by repeating its last frame; n_valid is the number of real frames.
  3. Resize each frame to 336 x 336 (square, no crop, bicubic with antialiasing), scale to [0, 1], then to [-1, 1].
  4. The video score is the mean P(manipulated) over its clips.

Scores are not calibrated. If you need a decision, choose a threshold on validation data from your own setting.

Use with safetensors (PyTorch)

Requires the code repository:

git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file

from mars.config import load_yaml
from mars.data.sampling import clips_for_video
from mars.data.transforms import preprocess
from mars.data.video import count_frames, read_frames
from mars.models.factory import build_detector

folder = snapshot_download("vector-institute/mars", allow_patterns=["config.yaml", "model.safetensors"])
raw = load_yaml(f"{folder}/config.yaml")
detector = build_detector(raw["model"]["name"], raw)
model = detector.build_network()  # downloads the frozen encoder on first use
missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False)
assert not unexpected and all(key.startswith("encoder.") for key in missing)  # only the frozen encoder is absent
model = model.cuda().eval()
data = detector.cfg.data


def score(path: str) -> float:
    """Mean P(manipulated) over the clips of one video."""
    probs = []
    for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride):
        frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda()
        valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda()
        with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
            logits = model(frames, valid).logits
        probs.append(logits.float().softmax(-1)[0, 1].item())
    return sum(probs) / len(probs)


print(score("video.mp4"))

Use with ONNX (no PyTorch, no code repository)

pip install onnxruntime-gpu av numpy pillow huggingface_hub   # or onnxruntime for CPU
name shape, type meaning
input frames (1, 64, 3, 336, 336), float32 one clip in [-1, 1]
input n_valid (1,), int64 number of real frames in the clip, 1 to 64
output fake_prob (1,), float32 P(manipulated) for the clip
output logits (1, 2), float32 authentic and manipulated logits

Clip length, frame stride and image size are also stored in the file's metadata.

import av
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from PIL import Image

session = ort.InferenceSession(
    hf_hub_download("vector-institute/mars", "model.onnx"),
    providers=["CUDAExecutionProvider", "CPUExecutionProvider"],
)
meta = session.get_modelmeta().custom_metadata_map
T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size"))


def to_input(rgb: np.ndarray) -> np.ndarray:
    """(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1]."""
    x = rgb.astype(np.float32) / 255.0
    if x.shape[:2] != (S, S):
        channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)]
        x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0)
    return ((x - 0.5) / 0.5).transpose(2, 0, 1)


def score(path: str) -> float:
    """Mean P(manipulated) over the clips of one video."""
    with av.open(path) as container:
        kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0]
    probs = []
    for start in range(0, len(kept), T):
        clip = kept[start : start + T]
        n_valid = len(clip)
        clip = clip + [clip[-1]] * (T - n_valid)
        inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)}
        probs.append(float(session.run(["fake_prob"], inputs)[0][0]))
    return float(np.mean(probs))


print(score("video.mp4"))

The ONNX graph runs in float32, while the paper's evaluation ran PyTorch in bfloat16, so scores differ slightly. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with CUDA is installed, import torch before creating the session also works.

Training

training data FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic)
validation FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3)
clips 64 frames, every second frame, full frames at 336 x 336
encoder vit_pe_core_large_patch14_336.fb, frozen; the semantic path tunes its own LayerNorm copies
optimization AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0
hardware 4 x NVIDIA A40

Results

Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's score is the mean over its clips.

test set videos AUROC AP
FaceForensics++ (test split) 700 98.98 99.74
Celeb-DF v2 518 91.22 94.73
DeepfakeTIMIT 960 99.95 99.97
UADFV 98 96.58 97.29
HiFiFace (paired with FF++ targets) 280 84.05 85.95
GenVideo (validation split, 10 generators) 18,286 93.44 92.60
DF26 (10 generators) 2,691 75.81 95.76
average 91.43

License

The weights in this repository are released under Apache 2.0. The frozen encoder is Meta's Perception Encoder (PE-Core-L), downloaded from timm under its own license (Apache 2.0).

Citation

@misc{radwan2026marsmultiscaleappearanceresidualstreams,
      title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection},
      author={Ahmed Y. Radwan and Shaina Raza},
      year={2026},
      eprint={2611.12508},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2611.12508},
}
Downloads last month
6
Safetensors
Model size
67.5M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for vector-institute/mars

Adapter
(1)
this model

Paper for vector-institute/mars