Buckets:

332F
/

DF26-bucket

Files

xet

332F/DF26-bucket / README.md

332F

16 days ago

preview code

download

raw

8 kB

---
license: other
license_name: df26-evaluation-only-license
license_link: LICENSE.md
pretty_name: DF26
viewer: false
size_categories:
  - 1K<n<10K
task_categories:
  - video-classification
tags:
  - deepfake-detection
  - synthetic-media
  - talking-head
  - benchmark
  - video
extra_gated_heading: "Request access to DF26"
extra_gated_prompt: >-
  DF26 is a controlled-access benchmark of real and AI-generated
  talking-head videos. By clicking Agree you accept the
  [DF26 Dataset License](LICENSE.md). Access is limited to researchers
  who identify themselves on this form. You may not use this dataset
  (or any content derived from it) to train, fine-tune, or otherwise
  develop a generative model. The closed-source generator subset is
  evaluation-only. Redistribution outside this repository is not
  permitted. Requests are approved automatically once submitted —
  fill out every field accurately, since incomplete or clearly false
  entries may be revoked on review.
extra_gated_button_content: "Agree and request access"
extra_gated_fields:
  Full name (first and last): text
  Institutional / lab affiliation: text
  Country: country
  Institutional email: text
  Intended use of the dataset: text
  I agree not to use DF26, or any content derived from it, to train, fine-tune, or otherwise develop a generative model: checkbox
  I agree not to redistribute DF26, or any portion of it, to any third party: checkbox
  I agree to the DF26 Dataset License, including the closed-source evaluation-only restriction, the leave-one-open-generator-out training rule, and all source-dataset terms: checkbox
  I confirm the information above is accurate and that I am requesting access as an individually accountable researcher: checkbox
configs:
  - config_name: default
    default: true
    data_files:
      - split: test
        path: metadata/videos.csv
  - config_name: frames
    data_files:
      - split: test
        path: metadata/frames.csv
---

# DF26

DF26 is an evaluation set for video-level (temporal) and frame-based deepfake detectors, covering AI-generated talking-head video: one speaker, in casual address, studio interview, or official statement. Each real clip has a single scene prompt used by every generator, so identity and scene stay fixed and the generator is what changes. Paper: [arXiv:2609.07369](https://arxiv.org/abs/2609.07369).

There are 271 reals and 2,420 fakes, stored as 1280×720 H.264. Duration is targeted at 5 s (4.01–6.04s in reality: Grok Imagine 6.04 s, most Veo 3.1 clips 4.01 s, the rest ≈5 s). [`metadata/videos.csv`](metadata/videos.csv) has one row per video.

Real clips are under `real/{scenario}/` and come from OpenVid-1M, MAVOS-DD, and TalkingCelebs:

| Scenario                |  Videos | MAVOS-DD | OpenVid | TalkingCelebs |
| :---------------------- | :-----: | :------: | :-----: | :-----------: |
| Direct_to_Camera_Casual |   100   |    3     |    97   |       0       |
| Official_Statement      |    71   |    4     |    5    |       62      |
| Studio_Interview        |   100   |    5     |    95   |       0       |
| **Total**               | **271** |  **12**  | **197** |     **62**    |

Fakes are `fake/{scenario}/{generator}/{clip_id}.mp4` with the same `clip_id` as the real. Open-weight models provide I2V (real frame + prompt) and T2V (prompt only). Commercial APIs are I2V only and were not run on `Official_Statement`.

| Generator                | Availability   | Mode  | Casual | Studio | Official | Total |
| :----------------------- | :-------------: | :----: | :----: | :----: | :------: | :---: |
| HunyuanVideo_1.5_14b_i2v | Open-weight    | I2V   |  100   |  100   |    71    |  271  |
| HunyuanVideo_1.5_14b_t2v | Open-weight    | T2V   |  100   |  100   |    71    |  271  |
| LTX_2.3_distilled_i2v    | Open-weight    | I2V   |  100   |  100   |    71    |  271  |
| LTX_2.3_distilled_t2v    | Open-weight    | T2V   |  100   |  100   |    71    |  271  |
| Wan_2.2_14b_i2v          | Open-weight    | I2V   |  100   |  100   |    71    |  271  |
| Wan_2.2_14b_t2v          | Open-weight    | T2V   |  100   |  100   |    71    |  271  |
| Grok_Imagine             | Commercial API | I2V   |   99   |  100   |    —     |  199  |
| Kling_3.0                | Commercial API | I2V   |   98   |  100   |    —     |  198  |
| Veo_3.1                  | Commercial API | I2V   |  100   |  100   |    —     |  200  |
| Wan_2.6                  | Commercial API | I2V   |   97   |  100   |    —     |  197  |

```text
real/{scenario}/{clip_id}.mp4
fake/{scenario}/{generator}/{clip_id}.mp4
frames/{scenario}/{clip_id}/{clip_id}_frame{0-3}.jpg
prompts.csv
metadata/videos.csv
metadata/frames.csv
metadata/generators.csv
dataset-creation-and-evaluation/
```

`clip_id` joins the real, its fakes, four keyframes, and the `prompts.csv` row (`{scenario}/{clip_id}.mp4`). Paths in the metadata CSVs are relative to the repo root.

## Baselines

Cross-generator video-level AUROC (%) of state-of-the-art deepfake detectors trained on FF++ and evaluated on DF26 without fine-tuning (Table 6 of the [paper](https://arxiv.org/abs/2609.07369)). Mean is macro-averaged over the ten generator subsets. Higher is better; 50 is chance.

| Detector    | Venue   | Type     | Grok 1.0 | Kling 3.0 | Veo 3.1  | Wan 2.6  | HV 1.5 I2V | HV 1.5 T2V | Wan 2.2 I2V | Wan 2.2 T2V | LTX 2.3 I2V | LTX 2.3 T2V |   Mean   |
| :---------- | ------: | :------- | :------: | :-------: | :------: | :------: | :--------: | :--------: | :---------: | :---------: | :---------: | :---------: | :------: |
| DFD-FCG     | CVPR'25 | Temporal |   20.5   |    27.4   |   29.6   |   60.3   |    41.1    |    76.5    |     61.3    |     58.4    |     61.1    |     45.9    |   48.2   |
| PwTF-DVD    | ICCV'25 | Temporal |   34.5   |    60.4   |   26.7   |   71.6   |    41.9    |    56.7    |     **87.7**    |     **92.9**    |     79.5    |     64.0    |   61.6   |
| ForAda      | CVPR'25 | Frame    |   38.9   |    53.4   |   39.9   |   63.8   |    50.8    |    78.0    |     62.6    |     48.4    |     70.8    |     57.7    |   56.4   |
| Effort      | ICML'25 | Frame    |   13.6   |    32.3   |   21.3   |   29.0   |    44.3    |    68.8    |     61.7    |     30.4    |     61.6    |     45.5    |   40.9   |
| FSFM        | CVPR'25 | Frame    |   36.3   |    49.7   |   38.2   |   69.2   |    46.3    |    75.4    |     62.7    |     55.0    |     65.2    |     41.3    |   53.9   |
| GenD-CLIP   | WACV'26 | Frame    |   47.7   |    62.5   |   39.6   |   62.6   |    52.2    |    73.9    |     66.5    |     35.5    |     72.2    |     43.5    |   55.6   |
| **GenD-PE** | WACV'26 | Frame    | **66.5** |  **71.0** | **76.6** | **88.3** |  **64.8**  |    82.7    |     66.1    |     58.6    |     81.5    |   **71.4**  | **72.7** |
| GenD-DINO   | WACV'26 | Frame    |   35.4   |    70.5   |   42.4   |   76.6   |    55.6    |    73.0    |     63.0    |     34.7    |     72.0    |     46.5    |   57.0   |
| DFD-HR      | CVPR'26 | Frame    |   43.9   |    72.1   |   49.9   |   64.4   |    61.3    |  **84.5**  |     73.1    |     48.3    |   **83.8**  |     56.5    |   63.8   |

The first four columns are commercial generators, the rest open-weight. No detector is uniformly strong: PwTF-DVD reaches 92.9 AUROC on Wan 2.2 T2V yet is at chance on HunyuanVideo 1.5 I2V.

## Evaluation

Report AUC and EER per generator. Training rules are in [LICENSE.md](LICENSE.md).

## Citation

```bibtex
@misc{shykula2026df26,
  title         = {DF26: We Cannot Tell Fake From Real Anymore},
  author        = {Shykula, Severyn and Yermakov, Andrii and Samarskyi, Ivan and Mishkin, Dmytro and Cech, Jan and Mishchuk, Anastasiia},
  year          = {2026},
  eprint        = {2609.07369},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2609.07369}
}
```

Xet Storage Details

Size:: 8 kB
Xet hash:: 8898dcba8d402b170ef0d88006eb85e3a66049edf8d22ade94b8d3a211cb693e

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	---
	license: other
	license_name: df26-evaluation-only-license
	license_link: LICENSE.md
	pretty_name: DF26
	viewer: false
	size_categories:
	- 1K<n<10K
	task_categories:
	- video-classification
	tags:
	- deepfake-detection
	- synthetic-media
	- talking-head
	- benchmark
	- video
	extra_gated_heading: "Request access to DF26"
	extra_gated_prompt: >-
	DF26 is a controlled-access benchmark of real and AI-generated
	talking-head videos. By clicking Agree you accept the
	[DF26 Dataset License](LICENSE.md). Access is limited to researchers
	who identify themselves on this form. You may not use this dataset
	(or any content derived from it) to train, fine-tune, or otherwise
	develop a generative model. The closed-source generator subset is
	evaluation-only. Redistribution outside this repository is not
	permitted. Requests are approved automatically once submitted —
	fill out every field accurately, since incomplete or clearly false
	entries may be revoked on review.
	extra_gated_button_content: "Agree and request access"
	extra_gated_fields:
	Full name (first and last): text
	Institutional / lab affiliation: text
	Country: country
	Institutional email: text
	Intended use of the dataset: text
	I agree not to use DF26, or any content derived from it, to train, fine-tune, or otherwise develop a generative model: checkbox
	I agree not to redistribute DF26, or any portion of it, to any third party: checkbox
	I agree to the DF26 Dataset License, including the closed-source evaluation-only restriction, the leave-one-open-generator-out training rule, and all source-dataset terms: checkbox
	I confirm the information above is accurate and that I am requesting access as an individually accountable researcher: checkbox
	configs:
	- config_name: default
	default: true
	data_files:
	- split: test
	path: metadata/videos.csv
	- config_name: frames
	data_files:
	- split: test
	path: metadata/frames.csv
	---

	# DF26

	DF26 is an evaluation set for video-level (temporal) and frame-based deepfake detectors, covering AI-generated talking-head video: one speaker, in casual address, studio interview, or official statement. Each real clip has a single scene prompt used by every generator, so identity and scene stay fixed and the generator is what changes. Paper: [arXiv:2609.07369](https://arxiv.org/abs/2609.07369).

	There are 271 reals and 2,420 fakes, stored as 1280×720 H.264. Duration is targeted at 5 s (4.01–6.04s in reality: Grok Imagine 6.04 s, most Veo 3.1 clips 4.01 s, the rest ≈5 s). [`metadata/videos.csv`](metadata/videos.csv) has one row per video.

	Real clips are under `real/{scenario}/` and come from OpenVid-1M, MAVOS-DD, and TalkingCelebs:

	\| Scenario \| Videos \| MAVOS-DD \| OpenVid \| TalkingCelebs \|
	\| :---------------------- \| :-----: \| :------: \| :-----: \| :-----------: \|
	\| Direct_to_Camera_Casual \| 100 \| 3 \| 97 \| 0 \|
	\| Official_Statement \| 71 \| 4 \| 5 \| 62 \|
	\| Studio_Interview \| 100 \| 5 \| 95 \| 0 \|
	\| Total \| 271 \| 12 \| 197 \| 62 \|

	Fakes are `fake/{scenario}/{generator}/{clip_id}.mp4` with the same `clip_id` as the real. Open-weight models provide I2V (real frame + prompt) and T2V (prompt only). Commercial APIs are I2V only and were not run on `Official_Statement`.

	\| Generator \| Availability \| Mode \| Casual \| Studio \| Official \| Total \|
	\| :----------------------- \| :-------------: \| :----: \| :----: \| :----: \| :------: \| :---: \|
	\| HunyuanVideo_1.5_14b_i2v \| Open-weight \| I2V \| 100 \| 100 \| 71 \| 271 \|
	\| HunyuanVideo_1.5_14b_t2v \| Open-weight \| T2V \| 100 \| 100 \| 71 \| 271 \|
	\| LTX_2.3_distilled_i2v \| Open-weight \| I2V \| 100 \| 100 \| 71 \| 271 \|
	\| LTX_2.3_distilled_t2v \| Open-weight \| T2V \| 100 \| 100 \| 71 \| 271 \|
	\| Wan_2.2_14b_i2v \| Open-weight \| I2V \| 100 \| 100 \| 71 \| 271 \|
	\| Wan_2.2_14b_t2v \| Open-weight \| T2V \| 100 \| 100 \| 71 \| 271 \|
	\| Grok_Imagine \| Commercial API \| I2V \| 99 \| 100 \| — \| 199 \|
	\| Kling_3.0 \| Commercial API \| I2V \| 98 \| 100 \| — \| 198 \|
	\| Veo_3.1 \| Commercial API \| I2V \| 100 \| 100 \| — \| 200 \|
	\| Wan_2.6 \| Commercial API \| I2V \| 97 \| 100 \| — \| 197 \|

	```text
	real/{scenario}/{clip_id}.mp4
	fake/{scenario}/{generator}/{clip_id}.mp4
	frames/{scenario}/{clip_id}/{clip_id}_frame{0-3}.jpg
	prompts.csv
	metadata/videos.csv
	metadata/frames.csv
	metadata/generators.csv
	dataset-creation-and-evaluation/
	```

	`clip_id` joins the real, its fakes, four keyframes, and the `prompts.csv` row (`{scenario}/{clip_id}.mp4`). Paths in the metadata CSVs are relative to the repo root.

	## Baselines

	Cross-generator video-level AUROC (%) of state-of-the-art deepfake detectors trained on FF++ and evaluated on DF26 without fine-tuning (Table 6 of the [paper](https://arxiv.org/abs/2609.07369)). Mean is macro-averaged over the ten generator subsets. Higher is better; 50 is chance.

	\| Detector \| Venue \| Type \| Grok 1.0 \| Kling 3.0 \| Veo 3.1 \| Wan 2.6 \| HV 1.5 I2V \| HV 1.5 T2V \| Wan 2.2 I2V \| Wan 2.2 T2V \| LTX 2.3 I2V \| LTX 2.3 T2V \| Mean \|
	\| :---------- \| ------: \| :------- \| :------: \| :-------: \| :------: \| :------: \| :--------: \| :--------: \| :---------: \| :---------: \| :---------: \| :---------: \| :------: \|
	\| DFD-FCG \| CVPR'25 \| Temporal \| 20.5 \| 27.4 \| 29.6 \| 60.3 \| 41.1 \| 76.5 \| 61.3 \| 58.4 \| 61.1 \| 45.9 \| 48.2 \|
	\| PwTF-DVD \| ICCV'25 \| Temporal \| 34.5 \| 60.4 \| 26.7 \| 71.6 \| 41.9 \| 56.7 \| 87.7 \| 92.9 \| 79.5 \| 64.0 \| 61.6 \|
	\| ForAda \| CVPR'25 \| Frame \| 38.9 \| 53.4 \| 39.9 \| 63.8 \| 50.8 \| 78.0 \| 62.6 \| 48.4 \| 70.8 \| 57.7 \| 56.4 \|
	\| Effort \| ICML'25 \| Frame \| 13.6 \| 32.3 \| 21.3 \| 29.0 \| 44.3 \| 68.8 \| 61.7 \| 30.4 \| 61.6 \| 45.5 \| 40.9 \|
	\| FSFM \| CVPR'25 \| Frame \| 36.3 \| 49.7 \| 38.2 \| 69.2 \| 46.3 \| 75.4 \| 62.7 \| 55.0 \| 65.2 \| 41.3 \| 53.9 \|
	\| GenD-CLIP \| WACV'26 \| Frame \| 47.7 \| 62.5 \| 39.6 \| 62.6 \| 52.2 \| 73.9 \| 66.5 \| 35.5 \| 72.2 \| 43.5 \| 55.6 \|
	\| GenD-PE \| WACV'26 \| Frame \| 66.5 \| 71.0 \| 76.6 \| 88.3 \| 64.8 \| 82.7 \| 66.1 \| 58.6 \| 81.5 \| 71.4 \| 72.7 \|
	\| GenD-DINO \| WACV'26 \| Frame \| 35.4 \| 70.5 \| 42.4 \| 76.6 \| 55.6 \| 73.0 \| 63.0 \| 34.7 \| 72.0 \| 46.5 \| 57.0 \|
	\| DFD-HR \| CVPR'26 \| Frame \| 43.9 \| 72.1 \| 49.9 \| 64.4 \| 61.3 \| 84.5 \| 73.1 \| 48.3 \| 83.8 \| 56.5 \| 63.8 \|

	The first four columns are commercial generators, the rest open-weight. No detector is uniformly strong: PwTF-DVD reaches 92.9 AUROC on Wan 2.2 T2V yet is at chance on HunyuanVideo 1.5 I2V.

	## Evaluation

	Report AUC and EER per generator. Training rules are in [LICENSE.md](LICENSE.md).

	## Citation

	```bibtex
	@misc{shykula2026df26,
	title = {DF26: We Cannot Tell Fake From Real Anymore},
	author = {Shykula, Severyn and Yermakov, Andrii and Samarskyi, Ivan and Mishkin, Dmytro and Cech, Jan and Mishchuk, Anastasiia},
	year = {2026},
	eprint = {2609.07369},
	archivePrefix = {arXiv},
	primaryClass = {cs.CV},
	url = {https://arxiv.org/abs/2609.07369}
	}
	```