|
Download README.md from vector-institute/mars: direct link, hf CLI and curl.
- Browser
- Download file 11.2 kB
-
https://huggingface.co/vector-institute/mars/resolve/main/README.md
- Command line
-
hf download hf://vector-institute/mars/README.md
-
curl -L -o README.md https://huggingface.co/vector-institute/mars/resolve/main/README.md
11.2 kB
| license: apache-2.0 | |
| pipeline_tag: video-classification | |
| base_model: timm/vit_pe_core_large_patch14_336.fb | |
| tags: | |
| - deepfake-detection | |
| - video-forgery-detection | |
| - video-classification | |
| - onnx | |
| - safetensors | |
| base_model_relation: adapter | |
| co2_eq_emissions: | |
| emissions: 353.1 | |
| source: CodeCarbon, recorded with ai-impact-accounting | |
| training_type: fine-tuning | |
| geographical_location: Ontario, Canada | |
| hardware_used: 4 x NVIDIA A40 | |
| dia_version: '0.1' | |
| dia_report: | |
| scope: incremental | |
| lineage: | |
| - model: timm/vit_pe_core_large_patch14_336.fb | |
| relation: adapter | |
| compute: | |
| hardware: | |
| gpu: NVIDIA A40 | |
| count: 4 | |
| duration_gpu_hours: 17.3576 | |
| footprint: | |
| energy_kwh: | |
| value: 5.4554 | |
| quality: measured | |
| carbon_kgco2eq: | |
| value: 0.3531 | |
| quality: measured | |
| water_liters: | |
| value: | |
| - 9.82 | |
| - 21.821 | |
| quality: estimated-from-default-wue | |
| context: | |
| region: ca-on | |
| carbon_intensity: 0.03 | |
| wue_l_per_kwh: | |
| - 1.8 | |
| - 4.0 | |
| tool: codecarbon | |
| # MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection | |
| [](https://arxiv.org/abs/2611.12508) | |
| [](https://github.com/VectorInstitute/mars) | |
| [](https://vectorinstitute.github.io/mars) | |
| MARS is a full-frame video forgery detector that covers both facial manipulation (face | |
| swapping, reenactment) and fully generated video. It reads one frozen Perception Encoder | |
| (PE-Core-L) in two ways: | |
| 1. A **residual path** compares the encoder's intermediate patch features with smoothed | |
| copies of themselves, across space and time at several scales, and a temporal transformer | |
| models how these differences develop over the clip. | |
| 2. A **semantic path** classifies sampled frames with the complete encoder, tuning only its | |
| own copies of the LayerNorm parameters. | |
| A gate mixes the two paths per clip and returns P(manipulated) for the clip. | |
| ## Files | |
| | file | contents | size | | |
| |---|---|---| | |
| | `model.safetensors` | trained weights; the frozen PE-Core-L encoder is not included and is downloaded from timm on first use | 270 MB | | |
| | `config.yaml` | model and data settings, read by the `mars` package | 2 KB | | |
| | `model.onnx` | complete graph, frozen encoder included; runs with ONNX Runtime alone, float32 | 1.59 GB | | |
| | `model.fp16.onnx` | the same graph in float16, inputs and outputs still float32; less GPU memory | 0.80 GB | | |
| | `usual_ranges.json` | percentiles of the ONNX video score on the authentic and manipulated test videos, with `model.onnx` | 3 KB | | |
| | `usual_ranges.fp16.json` | the same with `model.fp16.onnx` | 3 KB | | |
| | `serving.json` | measured GPU memory and time per clip with ONNX Runtime (both files, batch 1 and 2), opset, outputs, encoder chunk, ONNX Runtime version | 1 KB | | |
| This is the checkpoint behind the MARS results in the paper. | |
| ## Input | |
| Every clip is built the same way, whichever file you use: | |
| 1. Decode the video as RGB and keep every second frame (0, 2, 4, ...). | |
| 2. Cut the kept frames into consecutive clips of 64. A short last clip (or a short video) is | |
| padded by repeating its last frame; `n_valid` is the number of real frames. | |
| 3. Resize each frame to 336 x 336 (square, no crop, bicubic with antialiasing), scale to | |
| [0, 1], then to [-1, 1]. | |
| 4. The video score is the mean P(manipulated) over its clips. | |
| Scores are not calibrated. If you need a decision, choose a threshold on validation data from | |
| your own setting. | |
| ## Use with safetensors (PyTorch) | |
| Requires the code repository: | |
| ```bash | |
| git clone https://github.com/VectorInstitute/mars.git && cd mars && uv sync | |
| ``` | |
| ```python | |
| import torch | |
| from huggingface_hub import snapshot_download | |
| from safetensors.torch import load_file | |
| from mars.config import load_yaml | |
| from mars.data.sampling import clips_for_video | |
| from mars.data.transforms import preprocess | |
| from mars.data.video import count_frames, read_frames | |
| from mars.models.factory import build_detector | |
| folder = snapshot_download("vector-institute/mars", allow_patterns=["config.yaml", "model.safetensors"]) | |
| raw = load_yaml(f"{folder}/config.yaml") | |
| detector = build_detector(raw["model"]["name"], raw) | |
| model = detector.build_network() # downloads the frozen encoder on first use | |
| missing, unexpected = model.load_state_dict(load_file(f"{folder}/model.safetensors"), strict=False) | |
| assert not unexpected and all(key.startswith("encoder.") for key in missing) # only the frozen encoder is absent | |
| model = model.cuda().eval() | |
| data = detector.cfg.data | |
| def score(path: str) -> float: | |
| """Mean P(manipulated) over the clips of one video.""" | |
| probs = [] | |
| for clip in clips_for_video(path, count_frames(path), data.n_frames, data.frame_stride): | |
| frames = preprocess(read_frames(path, list(clip.frame_indices)), data.image_size)[None].cuda() | |
| valid = (torch.arange(data.n_frames) < clip.n_valid)[None].cuda() | |
| with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16): | |
| logits = model(frames, valid).logits | |
| probs.append(logits.float().softmax(-1)[0, 1].item()) | |
| return sum(probs) / len(probs) | |
| print(score("video.mp4")) | |
| ``` | |
| ## Use with ONNX (no PyTorch, no code repository) | |
| ```bash | |
| pip install onnxruntime-gpu av numpy pillow huggingface_hub # or onnxruntime for CPU | |
| ``` | |
| | name | shape, type | meaning | | |
| |---|---|---| | |
| | input `frames` | (B, 64, 3, 336, 336), float32 | B clips in [-1, 1] | | |
| | input `n_valid` | (B,), int64 | number of real frames in each clip, 1 to 64 | | |
| | output `fake_prob` | (B,), float32 | P(manipulated) for each clip | | |
| | output `logits` | (B, 2), float32 | authentic and manipulated logits | | |
| | output `frame_contrib` | (B, 64), float32 | signed contribution of each frame to the clip's fake-minus-real logit margin; positive pushes toward manipulated, 0 on padded frames | | |
| | output `patch_contrib` | (B, 64, 24, 24), float32 | signed contribution of each frame's 14 x 14-pixel patches through the residual path, same units, 0 on padded frames | | |
| | output `gate` | (B, 2), float32 | weight given to the residual and the semantic path; the two sum to 1 | | |
| The batch size B is free. Clip length, frame stride and image size are also stored in the | |
| file's metadata. | |
| Both files take the full 64-frame clip. Inside the graph the encoder processes 16 frames of | |
| each clip at a time; frames are encoded independently, so this lowers memory without | |
| changing the outputs. On an A100, one clip peaks at about 6.8 GiB of GPU memory with | |
| `model.onnx` and 2.9 GiB with `model.fp16.onnx` (details in [`serving.json`](serving.json)). | |
| The float16 video scores are within 0.003 of float32 on UADFV and Celeb-DF. Use `model.onnx` | |
| on CPU, or as the source for a TensorRT engine. | |
| The contributions are those behind the paper's figures and faithfulness tests: the clip's | |
| margin (logit of `fake_prob`) equals the sum of `frame_contrib` plus a per-clip constant. | |
| A frame's `frame_contrib` is the sum of its `patch_contrib` cells, plus, for the 16 frames | |
| the semantic path samples, that path's share. | |
| [`usual_ranges.json`](usual_ranges.json) gives the percentiles (0 to 100) of the video score | |
| on the authentic and on the manipulated test videos listed under Results, computed with | |
| `model.onnx`; [`usual_ranges.fp16.json`](usual_ranges.fp16.json) is the same for `model.fp16.onnx`. | |
| Use them to place a score in context, not as a threshold. | |
| ```python | |
| import av | |
| import numpy as np | |
| import onnxruntime as ort | |
| from huggingface_hub import hf_hub_download | |
| from PIL import Image | |
| session = ort.InferenceSession( | |
| hf_hub_download("vector-institute/mars", "model.onnx"), # or "model.fp16.onnx" for less GPU memory | |
| providers=["CUDAExecutionProvider", "CPUExecutionProvider"], | |
| ) | |
| meta = session.get_modelmeta().custom_metadata_map | |
| T, STRIDE, S = (int(meta[key]) for key in ("n_frames", "frame_stride", "image_size")) | |
| def to_input(rgb: np.ndarray) -> np.ndarray: | |
| """(H, W, 3) uint8 RGB to (3, S, S) float32 in [-1, 1].""" | |
| x = rgb.astype(np.float32) / 255.0 | |
| if x.shape[:2] != (S, S): | |
| channels = [Image.fromarray(np.ascontiguousarray(x[..., c])).resize((S, S), Image.BICUBIC) for c in range(3)] | |
| x = np.clip(np.stack([np.asarray(c) for c in channels], axis=-1), 0.0, 1.0) | |
| return ((x - 0.5) / 0.5).transpose(2, 0, 1) | |
| def score(path: str) -> float: | |
| """Mean P(manipulated) over the clips of one video.""" | |
| with av.open(path) as container: | |
| kept = [to_input(f.to_ndarray(format="rgb24")) for i, f in enumerate(container.decode(video=0)) if i % STRIDE == 0] | |
| probs = [] | |
| for start in range(0, len(kept), T): | |
| clip = kept[start : start + T] | |
| n_valid = len(clip) | |
| clip = clip + [clip[-1]] * (T - n_valid) | |
| inputs = {"frames": np.stack(clip)[None].astype(np.float32), "n_valid": np.array([n_valid], dtype=np.int64)} | |
| probs.append(float(session.run(["fake_prob"], inputs)[0][0])) | |
| return float(np.mean(probs)) | |
| print(score("video.mp4")) | |
| ``` | |
| Neither ONNX file runs in bfloat16 like the evaluation behind Results, so scores differ | |
| slightly from that table. To use the GPU, ONNX Runtime needs CUDA and cuDNN; if PyTorch with | |
| CUDA is installed, `import torch` before creating the session also works. | |
| ## Training | |
| | | | | |
| |---|---| | |
| | training data | FaceForensics++ (c23; original videos and Deepfakes, Face2Face, FaceSwap, NeuralTextures; official splits) and GTA-V (rendered sequences from SAIL-VOS 3D, all synthetic) | | |
| | validation | FaceForensics++ validation split; the checkpoint is the epoch with the best video AUROC (the last of 3) | | |
| | clips | 64 frames, every second frame, full frames at 336 x 336 | | |
| | encoder | `vit_pe_core_large_patch14_336.fb`, frozen; the semantic path tunes its own LayerNorm copies | | |
| | optimization | AdamW, learning rate 1e-4, weight decay 1e-4, batch 32 clips, step decay x0.5 every 1,000 steps, 3 epochs, bfloat16, seed 0 | | |
| | hardware | 4 x NVIDIA A40 | | |
| ## Results | |
| Video-level AUROC and average precision (%), trained on FaceForensics++ and GTA-V. A video's | |
| score is the mean over its clips. | |
| | test set | videos | AUROC | AP | | |
| |---|---|---|---| | |
| | FaceForensics++ (test split) | 700 | 98.98 | 99.74 | | |
| | Celeb-DF v2 | 518 | 91.22 | 94.73 | | |
| | DeepfakeTIMIT | 960 | 99.95 | 99.97 | | |
| | UADFV | 98 | 96.58 | 97.29 | | |
| | HiFiFace (paired with FF++ targets) | 280 | 84.05 | 85.95 | | |
| | GenVideo (validation split, 10 generators) | 18,286 | 93.44 | 92.60 | | |
| | DF26 (10 generators) | 2,691 | 75.81 | 95.76 | | |
| | **average** | | **91.43** | | | |
| ## License | |
| The weights in this repository are released under Apache 2.0. The frozen encoder is Meta's | |
| Perception Encoder (PE-Core-L), downloaded from timm under its own license (Apache 2.0). | |
| ## Citation | |
| ```bibtex | |
| @misc{radwan2026marsmultiscaleappearanceresidualstreams, | |
| title={MARS: Multiscale Appearance-Residual Streams for Video Forgery Detection}, | |
| author={Ahmed Y. Radwan and Shaina Raza}, | |
| year={2026}, | |
| eprint={2611.12508}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CV}, | |
| url={https://arxiv.org/abs/2611.12508}, | |
| } | |
| ``` | |