diff --git a/.gitattributes b/.gitattributes index d2db7edd54b381c486377e1896d6927a4886f954..252d4c4005ce83213ee5ea05c4fe48bcd8658432 100644 --- a/.gitattributes +++ b/.gitattributes @@ -89,3 +89,18 @@ clips/RVRA__real__77f__main__id08710_EMa5tWHB6Mo_00048__id01858.mp4 filter=lfs d clips/RVRA__real__77f__main__id08808_vQV6tYM_-3M_00053__id07697.mp4 filter=lfs diff=lfs merge=lfs -text clips/RVRA__real__77f__vocoded__id01104_wmkjwD45aYo_00109__id08710.mp4 filter=lfs diff=lfs merge=lfs -text clips/RVRA__real__77f__vocoded__id05623_HbnUHzNq8XI_00003__id00739.mp4 filter=lfs diff=lfs merge=lfs -text +clips/FVFA__facefusion__77f__main__id05670_8I_YYqRsihM_00041__id04900.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVFA__seed_vc__153f__main__id02359_QiCI0mefCx4_00030__id03078.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVFA__seed_vc__153f__main__id04049_JCDeI7UwWnE_00080__id08096.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVFA__seed_vc__77f__main__id01858_HBSDURviflg_00015__id01156.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVFA__seed_vc__77f__main__id07574_DbcyMh1l_Ng_00043__id06429.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__153f__main__id01124_2hg7q9O8xLo_00015__id08748.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__153f__main__id03078_4XN_hNJCe9E_00031__id06987.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__153f__main__id04049_Flu_4soH0mo_00003__id04528.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__153f__main__id05042_K1HMS1K0kWI_00010__id00144.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__153f__main__id05623_UXUHOpRirtg_00079__id03123.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__153f__main__id06713_5qTLKWiq73k_00074__id05042.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__153f__vocoded__id00885_xiPMwAYISG0_00135__id05623.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__153f__vocoded__id01124_7Y-51ooBIWQ_00040__id09218.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__77f__main__id06429_0aTJ9yNMAx0_00002__id06427.mp4 filter=lfs diff=lfs merge=lfs -text +clips/RVRA__real__77f__vocoded__id00948_PCy1KCIsNxo_00024__id04278.mp4 filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md index 61e4ae91361d61cc0e82902733fd1112f9c62a88..e407116434446a9e60ddbf2ab4944460b869aa7c 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,7 @@ short_description: Gemini zero-shot answers on MOZAIC deepfake samples A small pilot from MOZAIC, an audio-visual deepfake benchmark built on VoxCeleb2 (code: https://github.com/VectorInstitute/mozaic). It shows Gemini's zero-shot answers on -example questions from each task (T1 spot the fake, T2 manipulation type, T3 same person, -T3.5 real same person), with the clips, the correct answers and the pilot-run scores. +example questions from each task (T1 spot the fake, T2 manipulation type, T3 identity check), with the clips, the correct answers and the pilot-run scores. Clips are copied from the MOZAIC release unchanged (video stream untouched; audio re-encoded to AAC so browsers can play it). Fake clips are synthetic media made for diff --git a/clips/FVFA__facefusion__77f__main__id05670_8I_YYqRsihM_00041__id04900.mp4 b/clips/FVFA__facefusion__77f__main__id05670_8I_YYqRsihM_00041__id04900.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..decbc6ad80f70ae746a132cfe19c045b20ae3cd3 --- /dev/null +++ b/clips/FVFA__facefusion__77f__main__id05670_8I_YYqRsihM_00041__id04900.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4a39bdcc7fb92c57cf5b1510c8429fec97ec5161d81c712c16685890102c3e8b +size 146226 diff --git a/clips/FVRA__animate__77f__main__id07054_zTKKYEviZvQ_00073__id01052.mp4 b/clips/FVRA__animate__77f__main__id07054_zTKKYEviZvQ_00073__id01052.mp4 deleted file mode 100644 index f807ae03b0cbc01656e315def922992bcb2c5a36..0000000000000000000000000000000000000000 --- a/clips/FVRA__animate__77f__main__id07054_zTKKYEviZvQ_00073__id01052.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:589f0118c62983820afb775873deb66f59e0ac258f04079a6a787082836eab0f -size 120821 diff --git a/clips/FVRA__animate__77f__main__id07839__QTWWQBdhWY_00029__id07683.mp4 b/clips/FVRA__animate__77f__main__id07839__QTWWQBdhWY_00029__id07683.mp4 deleted file mode 100644 index 3e268c3b90f0b27dfddeb427e723385713c74262..0000000000000000000000000000000000000000 --- a/clips/FVRA__animate__77f__main__id07839__QTWWQBdhWY_00029__id07683.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:fb541e2410de06410bcd71789b65c745ba1b36abc98305252bb25630c731f405 -size 107958 diff --git a/clips/FVRA__facefusion__153f__main__id04171_S7q4HA1KM-8_00038__id04126.mp4 b/clips/FVRA__facefusion__153f__main__id04171_S7q4HA1KM-8_00038__id04126.mp4 deleted file mode 100644 index f740e7dc67640d5ab417c13f3553c96f89bfae4b..0000000000000000000000000000000000000000 --- a/clips/FVRA__facefusion__153f__main__id04171_S7q4HA1KM-8_00038__id04126.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:213a0ace479e8948e91db61a80fb1b685aec97d1cabb2c1f216325272c619cfa -size 202819 diff --git a/clips/FVRA__s2v__77f__main__id06427_FE4m2_FwOTA_00050__id08209.mp4 b/clips/FVRA__s2v__77f__main__id06427_FE4m2_FwOTA_00050__id08209.mp4 deleted file mode 100644 index 8c429bbe44683c63002386ab2efdd39ee8bfb26d..0000000000000000000000000000000000000000 --- a/clips/FVRA__s2v__77f__main__id06427_FE4m2_FwOTA_00050__id08209.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:441c2e97f0a174a089ca6b1a39b0a187230d8171a0f3d70df3bb19bdd7c424f3 -size 181776 diff --git a/clips/RVFA__seed_vc__153f__main__id02359_QiCI0mefCx4_00030__id03078.mp4 b/clips/RVFA__seed_vc__153f__main__id02359_QiCI0mefCx4_00030__id03078.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..61f0dcff0bca3e6cbb925c2251a7907099dedf05 --- /dev/null +++ b/clips/RVFA__seed_vc__153f__main__id02359_QiCI0mefCx4_00030__id03078.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e3af378281f366520d7af0de31cedeb0f071cc09643cd4b8524825223b10ec36 +size 227550 diff --git a/clips/RVFA__seed_vc__153f__main__id04049_JCDeI7UwWnE_00080__id08096.mp4 b/clips/RVFA__seed_vc__153f__main__id04049_JCDeI7UwWnE_00080__id08096.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..c18fcbc10ea93af66072746ea91a08be27e910a6 --- /dev/null +++ b/clips/RVFA__seed_vc__153f__main__id04049_JCDeI7UwWnE_00080__id08096.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4183d873625530007d910fc817de2a122a38af9d9f064de499f3c933a0d31839 +size 197732 diff --git a/clips/RVFA__seed_vc__153f__main__id06210_ZG5rWainMdg_00075__id01643.mp4 b/clips/RVFA__seed_vc__153f__main__id06210_ZG5rWainMdg_00075__id01643.mp4 deleted file mode 100644 index c07f1eea5474e686cc24e83a02d566ff0dfd6426..0000000000000000000000000000000000000000 --- a/clips/RVFA__seed_vc__153f__main__id06210_ZG5rWainMdg_00075__id01643.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:5b943642a1836cf4cf1402ff7f721c552b9e4410c37e331060c12cc00fee4400 -size 274461 diff --git a/clips/RVFA__seed_vc__153f__main__id07780_9oRm6SDT2Ak_00002__id05042.mp4 b/clips/RVFA__seed_vc__153f__main__id07780_9oRm6SDT2Ak_00002__id05042.mp4 deleted file mode 100644 index e788611cf6ddad6f5a348bc30417916984aafe01..0000000000000000000000000000000000000000 --- a/clips/RVFA__seed_vc__153f__main__id07780_9oRm6SDT2Ak_00002__id05042.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:28346f508d4c4f1ea4679cd4d08b6c3ccfc2c5e86dc7bc2b656f43071d261edf -size 165684 diff --git a/clips/RVFA__seed_vc__77f__main__id01505_Lf7-yDcu0NA_00049__id00761.mp4 b/clips/RVFA__seed_vc__77f__main__id01505_Lf7-yDcu0NA_00049__id00761.mp4 deleted file mode 100644 index 620c73813f48b637c6a0705ff2e86d89bad6c966..0000000000000000000000000000000000000000 --- a/clips/RVFA__seed_vc__77f__main__id01505_Lf7-yDcu0NA_00049__id00761.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:71cdac9a9a8d873bce4fe52b45238ae6fe6b70ba8d9da2a09ea2bff06d04ae68 -size 156378 diff --git a/clips/RVFA__seed_vc__77f__main__id01858_HBSDURviflg_00015__id01156.mp4 b/clips/RVFA__seed_vc__77f__main__id01858_HBSDURviflg_00015__id01156.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..37216bb8280c81d7466acf478afaa41a8abf60dd --- /dev/null +++ b/clips/RVFA__seed_vc__77f__main__id01858_HBSDURviflg_00015__id01156.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7747e6c1410f9a62fccef95fb0a53058b7918d506c52f6e292929837895c9980 +size 169052 diff --git a/clips/RVFA__seed_vc__77f__main__id04528_UF0U3evnpC4_00063__id03281.mp4 b/clips/RVFA__seed_vc__77f__main__id04528_UF0U3evnpC4_00063__id03281.mp4 deleted file mode 100644 index 4332a06f3d11f360d7d0b4c31092c9615c459c00..0000000000000000000000000000000000000000 --- a/clips/RVFA__seed_vc__77f__main__id04528_UF0U3evnpC4_00063__id03281.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:829e0f1b9b043d2e500bdc2633f606a1e0b4e97810320dd554b9d2644330bea7 -size 106771 diff --git a/clips/RVFA__seed_vc__77f__main__id05564_5eBl0n59574_00005__id05877.mp4 b/clips/RVFA__seed_vc__77f__main__id05564_5eBl0n59574_00005__id05877.mp4 deleted file mode 100644 index 86b96eaaf7c8c1cbb8645c1faf76f2a827c29429..0000000000000000000000000000000000000000 Binary files a/clips/RVFA__seed_vc__77f__main__id05564_5eBl0n59574_00005__id05877.mp4 and /dev/null differ diff --git a/clips/RVFA__seed_vc__77f__main__id07574_DbcyMh1l_Ng_00043__id06429.mp4 b/clips/RVFA__seed_vc__77f__main__id07574_DbcyMh1l_Ng_00043__id06429.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..0ed6791bebb01e4b1546e35849c8e8b460154359 --- /dev/null +++ b/clips/RVFA__seed_vc__77f__main__id07574_DbcyMh1l_Ng_00043__id06429.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4e131883cdc4bc77bfa711823b4b0e2bd2869c99a747808530f285b9bc8ff37c +size 101171 diff --git a/clips/RVRA__real__153f__main__id00739_KZti9h4Uyrc_00016__id00885.mp4 b/clips/RVRA__real__153f__main__id00739_KZti9h4Uyrc_00016__id00885.mp4 deleted file mode 100644 index 25d690659ee1cfb8a5fb9f77186f99e89c666375..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id00739_KZti9h4Uyrc_00016__id00885.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:03c07dce597f3950ba870f2b75894d627ccd3221753234df848141762394110c -size 359098 diff --git a/clips/RVRA__real__153f__main__id00739_OKXRt9jmIZQ_00050__id04597.mp4 b/clips/RVRA__real__153f__main__id00739_OKXRt9jmIZQ_00050__id04597.mp4 deleted file mode 100644 index 7bbe5d67ec5b3b1a71b0edff9133ba04627867f0..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id00739_OKXRt9jmIZQ_00050__id04597.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:f108b1ca21ba9fd6e97a9699aafdcd9535e6cac5116df8101cad9be0d0c6a55b -size 289319 diff --git a/clips/RVRA__real__153f__main__id01124_2hg7q9O8xLo_00015__id08748.mp4 b/clips/RVRA__real__153f__main__id01124_2hg7q9O8xLo_00015__id08748.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..8891eefe232713c7a7c397777b8a3db4687663ab --- /dev/null +++ b/clips/RVRA__real__153f__main__id01124_2hg7q9O8xLo_00015__id08748.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4a4b9bb299239c39a23b965e276acc4dea43f99fcb6a5b890639878d4c9f9149 +size 220006 diff --git a/clips/RVRA__real__153f__main__id01643_JsdoU91plJ0_00073__id08338.mp4 b/clips/RVRA__real__153f__main__id01643_JsdoU91plJ0_00073__id08338.mp4 deleted file mode 100644 index 9ec19568283c402628dd2391e358faccf0c4588e..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id01643_JsdoU91plJ0_00073__id08338.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:898959c8dd579d5580b3b608055e7d5e31e77babf7c75e88bd5bbf5491ab8f16 -size 283747 diff --git a/clips/RVRA__real__153f__main__id03078_4XN_hNJCe9E_00031__id06987.mp4 b/clips/RVRA__real__153f__main__id03078_4XN_hNJCe9E_00031__id06987.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..959299864f83428edf8a996b9e9b3ad5b4452381 --- /dev/null +++ b/clips/RVRA__real__153f__main__id03078_4XN_hNJCe9E_00031__id06987.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:31102a747052d2244b99bc5ff57e537ba7faa1120f0a96d85412e8610e0a9794 +size 259710 diff --git a/clips/RVRA__real__153f__main__id03710_gvuFmC3nXFw_00094__id06015.mp4 b/clips/RVRA__real__153f__main__id03710_gvuFmC3nXFw_00094__id06015.mp4 deleted file mode 100644 index ca261fac90bf3c9880fbd6d1bd7ca1276b761850..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id03710_gvuFmC3nXFw_00094__id06015.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:978883bce7b85e255e70d3f90c3de2fadede5af4555c48534563a5bc03c26d3d -size 201423 diff --git a/clips/RVRA__real__153f__main__id04049_Flu_4soH0mo_00003__id04528.mp4 b/clips/RVRA__real__153f__main__id04049_Flu_4soH0mo_00003__id04528.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..88e2ee3d30f83e7fa8f5876d1151440c3913b68a --- /dev/null +++ b/clips/RVRA__real__153f__main__id04049_Flu_4soH0mo_00003__id04528.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cc4f6c9de3cdbbe7fb6a1c71262390705d8fa479a073459bdaad7f0733eb0c99 +size 238820 diff --git a/clips/RVRA__real__153f__main__id04126_XcG4iWxXxUM_00120__id04171.mp4 b/clips/RVRA__real__153f__main__id04126_XcG4iWxXxUM_00120__id04171.mp4 deleted file mode 100644 index ba7a77851dae64e37d18f923a5effb2e0f5d7033..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id04126_XcG4iWxXxUM_00120__id04171.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ecab698cd73f4bb7e6a6148ce07a0edcf03937a40c75bf2dfc0cfe93200067f1 -size 211480 diff --git a/clips/RVRA__real__153f__main__id04495__k8mU1T87Pw_00134__id06471.mp4 b/clips/RVRA__real__153f__main__id04495__k8mU1T87Pw_00134__id06471.mp4 deleted file mode 100644 index 660392a06498b2d293e9f3012a4ef68f25510fab..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id04495__k8mU1T87Pw_00134__id06471.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:ce312a1cfb49abd2952c9f2ff04f2633ae42740442e138e79b68d35d83e194d6 -size 252712 diff --git a/clips/RVRA__real__153f__main__id04495_vDkHbTjw_pw_00255__id01664.mp4 b/clips/RVRA__real__153f__main__id04495_vDkHbTjw_pw_00255__id01664.mp4 deleted file mode 100644 index caf1f2ceeeeb02aecf145d5aaf2ffe1b135e2155..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id04495_vDkHbTjw_pw_00255__id01664.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:8c54f333f0100f7928435cc1fb8ea59d95cc3645a5175dc0ee7a94dfe67f8e06 -size 190995 diff --git a/clips/RVRA__real__153f__main__id04900_mwJnKgcj82w_00026__id05670.mp4 b/clips/RVRA__real__153f__main__id04900_mwJnKgcj82w_00026__id05670.mp4 deleted file mode 100644 index c4a27fbfdfabb59d56de8e2a9c0353e97785ca95..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id04900_mwJnKgcj82w_00026__id05670.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:bf0de48120810f2bf2607db9cc2d332c661780030895773f7d7677ca5cea82d4 -size 234444 diff --git a/clips/RVRA__real__153f__main__id05042_K1HMS1K0kWI_00010__id00144.mp4 b/clips/RVRA__real__153f__main__id05042_K1HMS1K0kWI_00010__id00144.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..0d72895eca3b4b749fff4077851aa5c9e9ef4d5d --- /dev/null +++ b/clips/RVRA__real__153f__main__id05042_K1HMS1K0kWI_00010__id00144.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c3abc2efe1a89afc52471243a34ac9570e6f4e3f988be6279b442ecc397a5110 +size 182947 diff --git a/clips/RVRA__real__153f__main__id05042_SIx5G21ZMN8_00067__id06343.mp4 b/clips/RVRA__real__153f__main__id05042_SIx5G21ZMN8_00067__id06343.mp4 deleted file mode 100644 index edf953725f028bb390f29d933086a17967eb49c7..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id05042_SIx5G21ZMN8_00067__id06343.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:eed2e245ca20c49e07d1ebeb8b572375624facaac9c22038159448d9a89ea105 -size 219735 diff --git a/clips/RVRA__real__153f__main__id05623_UXUHOpRirtg_00079__id03123.mp4 b/clips/RVRA__real__153f__main__id05623_UXUHOpRirtg_00079__id03123.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..82ff41a21a7f0e902d41802ab0b9277ee00dd445 --- /dev/null +++ b/clips/RVRA__real__153f__main__id05623_UXUHOpRirtg_00079__id03123.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:79bf7bc597827b48744b12a8cc1875aaa5aae96473684dfece32bd060f056556 +size 278846 diff --git a/clips/RVRA__real__153f__main__id06015_huOOE9WL2pk_00212__id01376.mp4 b/clips/RVRA__real__153f__main__id06015_huOOE9WL2pk_00212__id01376.mp4 deleted file mode 100644 index c9199a6829ee9c11e16cb2a5bf786443630f1bca..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id06015_huOOE9WL2pk_00212__id01376.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4ac6f045baa30e8c5d57a59c42667d7ce02aec681a365dd1d51dad67fb2f2186 -size 292086 diff --git a/clips/RVRA__real__153f__main__id06713_5qTLKWiq73k_00074__id05042.mp4 b/clips/RVRA__real__153f__main__id06713_5qTLKWiq73k_00074__id05042.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..b83b5104673fba2a4222b607159c34b947839ddb --- /dev/null +++ b/clips/RVRA__real__153f__main__id06713_5qTLKWiq73k_00074__id05042.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:523d8338f02694b7860f79a5bfb6a82046fb9ab03ac8bdcd4bee9f693c227b9b +size 303718 diff --git a/clips/RVRA__real__153f__main__id07398_uXZycbvHaAw_00063__id08338.mp4 b/clips/RVRA__real__153f__main__id07398_uXZycbvHaAw_00063__id08338.mp4 deleted file mode 100644 index 8f4755eda8423806067b32ae1b08902702acad4e..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id07398_uXZycbvHaAw_00063__id08338.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:efe569f216a7131c7d5683e55c61505096eebd9bfa9bf86f055683d42871cbd2 -size 240656 diff --git a/clips/RVRA__real__153f__main__id07839_41lss_yYWIM_00008__id05768.mp4 b/clips/RVRA__real__153f__main__id07839_41lss_yYWIM_00008__id05768.mp4 deleted file mode 100644 index db512c1d8ca7108a119214738b04e54b0a566b40..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__main__id07839_41lss_yYWIM_00008__id05768.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4d904b59d9518296b9c9447f14f2d8cfb50cfea02c02ac34c09a4bdcd5ffe7bf -size 238593 diff --git a/clips/RVRA__real__153f__vocoded__id00885_xiPMwAYISG0_00135__id05623.mp4 b/clips/RVRA__real__153f__vocoded__id00885_xiPMwAYISG0_00135__id05623.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..5d75fc48e96b117b7c614759e7e37d9b2a99d64f --- /dev/null +++ b/clips/RVRA__real__153f__vocoded__id00885_xiPMwAYISG0_00135__id05623.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:62c2e2183f7cec645e78aa39fa0c6ba9d987e68988ddedbb83a88d05e138549f +size 275143 diff --git a/clips/RVRA__real__153f__vocoded__id01124_7Y-51ooBIWQ_00040__id09218.mp4 b/clips/RVRA__real__153f__vocoded__id01124_7Y-51ooBIWQ_00040__id09218.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..db82d11ec734161aa0a3a182719cdc696537ae78 --- /dev/null +++ b/clips/RVRA__real__153f__vocoded__id01124_7Y-51ooBIWQ_00040__id09218.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6ac24434c181be9c3c93be9ade44304eb7c7ae6133415a9a8d1772a01d9cab0c +size 236927 diff --git a/clips/RVRA__real__153f__vocoded__id07398_tzZJzDzgF-g_00049__id06071.mp4 b/clips/RVRA__real__153f__vocoded__id07398_tzZJzDzgF-g_00049__id06071.mp4 deleted file mode 100644 index 9960aada4e6c8cacb46a1ba755d7df1ce27fb5ff..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__vocoded__id07398_tzZJzDzgF-g_00049__id06071.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:77ca5ad14d89d0b53fa822d4833117ade9dab77298659860b7b79c3340f47978 -size 238683 diff --git a/clips/RVRA__real__153f__vocoded__id07683_YCmGkErqGAc_00036__id07839.mp4 b/clips/RVRA__real__153f__vocoded__id07683_YCmGkErqGAc_00036__id07839.mp4 deleted file mode 100644 index 2fa2d90c90aef1ca7c0b9988ce775b9a76ca3e7b..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__153f__vocoded__id07683_YCmGkErqGAc_00036__id07839.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:30f8f36c3bec877c9982c6d58b425bd84015e3659eb6a9f70306b4bd90a6d2ac -size 228641 diff --git a/clips/RVRA__real__77f__main__id00948_z-P1KSBdNU8_00128__id08581.mp4 b/clips/RVRA__real__77f__main__id00948_z-P1KSBdNU8_00128__id08581.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..5e360432c72ff65fdf07899dae68bd9288d3255e Binary files /dev/null and b/clips/RVRA__real__77f__main__id00948_z-P1KSBdNU8_00128__id08581.mp4 differ diff --git a/clips/RVRA__real__77f__main__id01052_BPYYdOKuXSs_00029__id08590.mp4 b/clips/RVRA__real__77f__main__id01052_BPYYdOKuXSs_00029__id08590.mp4 deleted file mode 100644 index 790da9dc22bac59dc5450065f966242efd40b5da..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id01052_BPYYdOKuXSs_00029__id08590.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:07dcf974db3e6250cb2a002f8f3cd8e0710a1d44991210dcf0ac81c69266395e -size 102641 diff --git a/clips/RVRA__real__77f__main__id01505_dVKZSQDG9X4_00079__id00762.mp4 b/clips/RVRA__real__77f__main__id01505_dVKZSQDG9X4_00079__id00762.mp4 deleted file mode 100644 index a7e04d84d0efe476a59e85c30191f8e5c0e81400..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id01505_dVKZSQDG9X4_00079__id00762.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:0266b49af698f68b2920f01958bc61ab9a69ba1701f674815939efcd3e241d44 -size 104450 diff --git a/clips/RVRA__real__77f__main__id01534_htQIn8Um9Tg_00081__id04234.mp4 b/clips/RVRA__real__77f__main__id01534_htQIn8Um9Tg_00081__id04234.mp4 deleted file mode 100644 index 2f26dedccc709abe1bd3c849d34d94aed0434f60..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id01534_htQIn8Um9Tg_00081__id04234.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:46642cfa929dd01cc37d2306450ef8d375aae94c8bc49c8665fe9840f7037f29 -size 136303 diff --git a/clips/RVRA__real__77f__main__id01534_zaac3OQqEQE_00097__id06480.mp4 b/clips/RVRA__real__77f__main__id01534_zaac3OQqEQE_00097__id06480.mp4 deleted file mode 100644 index a371a69c8ecbf73f2b3066780bf859278ad619a0..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id01534_zaac3OQqEQE_00097__id06480.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:55900c049224d35fb0d4fcfb2267ded3531a9f1c5da7b487dce98c6da61e7707 -size 140422 diff --git a/clips/RVRA__real__77f__main__id03281_B8yQH7WxPc0_00033__id04049.mp4 b/clips/RVRA__real__77f__main__id03281_B8yQH7WxPc0_00033__id04049.mp4 deleted file mode 100644 index b9583c6259066d52c20dc23ef25f95d9891ab7c6..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id03281_B8yQH7WxPc0_00033__id04049.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:093ee35fae90ed0869c8963f29db42545a7c9f41dbb5503626fa3e585435896b -size 151556 diff --git a/clips/RVRA__real__77f__main__id05877_hy-uGrBUfPM_00046__id00172.mp4 b/clips/RVRA__real__77f__main__id05877_hy-uGrBUfPM_00046__id00172.mp4 deleted file mode 100644 index 86f418929278a2a615377041f28d495c9133fed2..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id05877_hy-uGrBUfPM_00046__id00172.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:26adcdc89f0bfefc29f7ec0c3ca13d7c1f547c968c0f3c82bb3ea7ed3c507b0f -size 129286 diff --git a/clips/RVRA__real__77f__main__id06429_0aTJ9yNMAx0_00002__id06427.mp4 b/clips/RVRA__real__77f__main__id06429_0aTJ9yNMAx0_00002__id06427.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..953725cdb4957d07cc9525fe1298edeab55acae2 --- /dev/null +++ b/clips/RVRA__real__77f__main__id06429_0aTJ9yNMAx0_00002__id06427.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ed6ba8f8b77a6698696d77b0c1b40dbed5665136042d001717bbf41363ea277c +size 161246 diff --git a/clips/RVRA__real__77f__main__id07697_3ysHlmEOElg_00022__id05444.mp4 b/clips/RVRA__real__77f__main__id07697_3ysHlmEOElg_00022__id05444.mp4 deleted file mode 100644 index 233becfbee66c2610aac81b4995dade69c493c64..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id07697_3ysHlmEOElg_00022__id05444.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:26a7532d09b32c51ade686b89850b3f133768949a8e5e6bf441ef605db49cafa -size 135232 diff --git a/clips/RVRA__real__77f__main__id07839_gJgj1DxOfKA_00045__id07786.mp4 b/clips/RVRA__real__77f__main__id07839_gJgj1DxOfKA_00045__id07786.mp4 deleted file mode 100644 index 25fd5a93686861d3653fa808e9b85dd275faee5c..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id07839_gJgj1DxOfKA_00045__id07786.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:51819f6865267482967c0b79c5e42203166c3b24c138f881961093c9674fb25f -size 135567 diff --git a/clips/RVRA__real__77f__main__id08209_O8r8MtzKdfI_00034__id06429.mp4 b/clips/RVRA__real__77f__main__id08209_O8r8MtzKdfI_00034__id06429.mp4 deleted file mode 100644 index 43c04aba8d23d74836ff034df5ce76845b3861c2..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id08209_O8r8MtzKdfI_00034__id06429.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4d964032cd4bc012eae8668a96d4b7423b24838fce3c6279eb72cb5ed38da0bc -size 123948 diff --git a/clips/RVRA__real__77f__main__id08590_3BGHjstAIzg_00003__id08610.mp4 b/clips/RVRA__real__77f__main__id08590_3BGHjstAIzg_00003__id08610.mp4 deleted file mode 100644 index ff2c2ea6641302e8ea9f6e652b94471800916c93..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id08590_3BGHjstAIzg_00003__id08610.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:4a0ddef36b147c327f87fb195ad4d1833aed0e172f306c5115b937f067dc3719 -size 122838 diff --git a/clips/RVRA__real__77f__main__id08590_UhRFZwnafIk_00068__id04236.mp4 b/clips/RVRA__real__77f__main__id08590_UhRFZwnafIk_00068__id04236.mp4 deleted file mode 100644 index f3bd71e2a2e501c93c6e9f3e85c08e4ca89bf203..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id08590_UhRFZwnafIk_00068__id04236.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:2e986588ba2186ee74cb67e10260c30368b62db9bb9ad2d1a7f19026959896a9 -size 125090 diff --git a/clips/RVRA__real__77f__main__id08710_EMa5tWHB6Mo_00048__id01858.mp4 b/clips/RVRA__real__77f__main__id08710_EMa5tWHB6Mo_00048__id01858.mp4 deleted file mode 100644 index 7f980d8a38c0b78f8c1b897d371471143a263590..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__main__id08710_EMa5tWHB6Mo_00048__id01858.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:b5f2cd4508a200f60c2dc04037f5ef75f8c695c034813ed0c74b21692f19ca5f -size 113452 diff --git a/clips/RVRA__real__77f__vocoded__id00948_PCy1KCIsNxo_00024__id04278.mp4 b/clips/RVRA__real__77f__vocoded__id00948_PCy1KCIsNxo_00024__id04278.mp4 new file mode 100644 index 0000000000000000000000000000000000000000..b1715650b58913c5a5e5f9a17f94cd43bce7b794 --- /dev/null +++ b/clips/RVRA__real__77f__vocoded__id00948_PCy1KCIsNxo_00024__id04278.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8ac75d2201b9d40e4b0ed9ca2fa7e5bac9a16d33830e8a5fc9500881a54da9df +size 168657 diff --git a/clips/RVRA__real__77f__vocoded__id01104_wmkjwD45aYo_00109__id08710.mp4 b/clips/RVRA__real__77f__vocoded__id01104_wmkjwD45aYo_00109__id08710.mp4 deleted file mode 100644 index 7c373d457d81aa6cfeb987fb28c2d31d743127ef..0000000000000000000000000000000000000000 --- a/clips/RVRA__real__77f__vocoded__id01104_wmkjwD45aYo_00109__id08710.mp4 +++ /dev/null @@ -1,3 +0,0 @@ -version https://git-lfs.github.com/spec/v1 -oid sha256:e3ec454c3637ede119535f925062d2c0f5f4ade54b5c0b4383fff722dd576a58 -size 102233 diff --git a/index.html b/index.html index 11a4a200c3c75dc5d2d3702c0ee62da8d8cf538d..c26cb385c94e1c7de5866fb934fc2bff515e36d2 100644 --- a/index.html +++ b/index.html @@ -79,16 +79,14 @@ const TASKS = { t1: ["T1 · Spot the fake", "Which part is fake: the video, the audio, both, or neither?", "The model saw 32 frames and heard the audio."], - t2: ["T2 · Manipulation type", "How was the video made?", "The model saw 32 frames only, no sound."], - t3: ["T3 · Same person", "Is the face (or voice) in clip 1 the same person as in clip 2?", - "Face questions: frames of both clips, no sound. Voice questions: sound of both clips, no frames."], - t3_5: ["T3.5 · Real same person", "Is clip 1 a real recording of clip 2's person?", + t2: ["T2 · Manipulation type", "Which part of this video, if any, was generated or altered by AI?", "The model saw 32 frames only, no sound."], + t3: ["T3 · Identity check", "Is clip 1 real or fake, and is it the same person as clip 2?", "Face questions: frames of both clips, no sound. Voice questions: sound of both clips, no frames."], }; const LABELS = { RVRA: "real video · real audio", RVFA: "real video · fake audio", FVRA: "fake video · real audio", FVFA: "fake video · fake audio" }; -const METHODS = { real: "camera (real)", facefusion: "FaceFusion face swap", s2v: "Wan2.2-S2V (audio-driven)", - animate: "Wan2.2-Animate (video-driven)", seed_vc: "Seed-VC voice conversion" }; +const METHODS = { real: "camera (real)", facefusion: "FaceFusion face swap", s2v: "Wan2.2-S2V (fully generated)", + animate: "Wan2.2-Animate (full-body swap)", seed_vc: "Seed-VC voice conversion" }; const esc = s => String(s ?? "").replace(/[&<>"']/g, c => ({ "&": "&", "<": "<", ">": ">", '"': """, "'": "'" }[c])); function clipDesc(clip) { diff --git a/samples.json b/samples.json index 786cb8c7e81dc2e0d92b1705775739af797afb8a..60a75bd9b47900ccb7a000cf94e1253668273db7 100644 --- a/samples.json +++ b/samples.json @@ -5,13 +5,13 @@ "t1": { "scores": { "n": 10, - "accuracy": 0.2, + "accuracy": 0.4, "ci95": [ - 0.0567, - 0.5098 + 0.1682, + 0.6873 ], - "balanced_accuracy": 0.1875, - "cohen_kappa": -0.0256 + "balanced_accuracy": 0.4167, + "cohen_kappa": 0.2405 }, "items": [ { @@ -66,25 +66,7 @@ "gt_answer": "neither", "pred_answer": "neither", "correct": true, - "raw": "{\"answer\": \"neither\"}" - }, - { - "id": "t1_RVRA__real__77f__main__id07697_3ysHlmEOElg_00022__id05444", - "clips": [ - "RVRA__real__77f__main__id07697_3ysHlmEOElg_00022__id05444.mp4" - ], - "part": null, - "question": "Which part of this audio-visual clip, if any, is AI-generated or manipulated? A real recording that was only processed, without changing who is seen or heard, still counts as real.", - "options": [ - "the audio", - "both", - "the video", - "neither" - ], - "gt_answer": "neither", - "pred_answer": "the video", - "correct": false, - "raw": "```json\n{\"answer\": \"the video\"}\n```" + "raw": "```json\n{\"answer\": \"neither\"}\n```" }, { "id": "t1_RVRA__real__153f__vocoded__id02959_hp-EccEnxCQ_00026__id08803", @@ -100,9 +82,9 @@ "the audio" ], "gt_answer": "neither", - "pred_answer": "the video", - "correct": false, - "raw": "{\"answer\": \"the video\"}" + "pred_answer": "neither", + "correct": true, + "raw": "```json\n{\"answer\": \"neither\"}\n```" }, { "id": "t1_FVFA__s2v__153f__main__id03084_BQIuTs2QZUU_00021__id08748", @@ -141,27 +123,27 @@ "raw": "```json\n{\"answer\": \"neither\"}\n```" }, { - "id": "t1_RVRA__real__153f__main__id04900_mwJnKgcj82w_00026__id05670", + "id": "t1_FVFA__s2v__77f__main__id06390_17l4_o3k29c_00007__id07683", "clips": [ - "RVRA__real__153f__main__id04900_mwJnKgcj82w_00026__id05670.mp4" + "FVFA__s2v__77f__main__id06390_17l4_o3k29c_00007__id07683.mp4" ], "part": null, "question": "Which part of this audio-visual clip, if any, is AI-generated or manipulated? A real recording that was only processed, without changing who is seen or heard, still counts as real.", "options": [ + "neither", "both", - "the video", "the audio", - "neither" + "the video" ], - "gt_answer": "neither", - "pred_answer": "the audio", + "gt_answer": "both", + "pred_answer": "neither", "correct": false, - "raw": "```json\n{\"answer\": \"the audio\"}\n```" + "raw": "```json\n{\"answer\": \"neither\"}\n```" }, { - "id": "t1_FVFA__s2v__77f__main__id06390_17l4_o3k29c_00007__id07683", + "id": "t1_FVFA__facefusion__153f__main__id07786_bFZAH657cFs_00051__id07683", "clips": [ - "FVFA__s2v__77f__main__id06390_17l4_o3k29c_00007__id07683.mp4" + "FVFA__facefusion__153f__main__id07786_bFZAH657cFs_00051__id07683.mp4" ], "part": null, "question": "Which part of this audio-visual clip, if any, is AI-generated or manipulated? A real recording that was only processed, without changing who is seen or heard, still counts as real.", @@ -172,26 +154,44 @@ "the video" ], "gt_answer": "both", + "pred_answer": "the audio", + "correct": false, + "raw": "```json\n{\"answer\": \"the audio\"}\n```" + }, + { + "id": "t1_FVFA__facefusion__77f__main__id05670_8I_YYqRsihM_00041__id04900", + "clips": [ + "FVFA__facefusion__77f__main__id05670_8I_YYqRsihM_00041__id04900.mp4" + ], + "part": null, + "question": "Which part of this audio-visual clip, if any, is AI-generated or manipulated? A real recording that was only processed, without changing who is seen or heard, still counts as real.", + "options": [ + "the video", + "both", + "neither", + "the audio" + ], + "gt_answer": "both", "pred_answer": "neither", "correct": false, "raw": "```json\n{\"answer\": \"neither\"}\n```" }, { - "id": "t1_FVFA__facefusion__153f__main__id07786_bFZAH657cFs_00051__id07683", + "id": "t1_RVFA__seed_vc__77f__main__id01858_HBSDURviflg_00015__id01156", "clips": [ - "FVFA__facefusion__153f__main__id07786_bFZAH657cFs_00051__id07683.mp4" + "RVFA__seed_vc__77f__main__id01858_HBSDURviflg_00015__id01156.mp4" ], "part": null, "question": "Which part of this audio-visual clip, if any, is AI-generated or manipulated? A real recording that was only processed, without changing who is seen or heard, still counts as real.", "options": [ "neither", - "both", "the audio", + "both", "the video" ], - "gt_answer": "both", + "gt_answer": "the audio", "pred_answer": "the audio", - "correct": false, + "correct": true, "raw": "```json\n{\"answer\": \"the audio\"}\n```" } ] @@ -199,13 +199,13 @@ "t2": { "scores": { "n": 10, - "accuracy": 0.5, + "accuracy": 0.2, "ci95": [ - 0.2366, - 0.7634 + 0.0567, + 0.5098 ], - "balanced_accuracy": 0.5, - "cohen_kappa": 0.3333 + "balanced_accuracy": 0.25, + "cohen_kappa": 0.0 }, "items": [ { @@ -214,17 +214,17 @@ "FVRA__s2v__77f__main__id08406_qr5lYg7ej40_00213__id05444.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ - "video-driven animation", - "audio-driven animation", + "full-body swap", + "fully generated", "real", "face swap" ], - "gt_answer": "audio-driven animation", - "pred_answer": "video-driven animation", + "gt_answer": "fully generated", + "pred_answer": "real", "correct": false, - "raw": "{\"answer\": \"video-driven animation\"}" + "raw": "```json\n{\"answer\": \"real\"}\n```" }, { "id": "t2_FVRA__facefusion__77f__main__id08940_B5cvspWteMI_00009__id03946", @@ -232,11 +232,11 @@ "FVRA__facefusion__77f__main__id08940_B5cvspWteMI_00009__id03946.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ "face swap", - "video-driven animation", - "audio-driven animation", + "full-body swap", + "fully generated", "real" ], "gt_answer": "face swap", @@ -250,17 +250,17 @@ "RVRA__real__77f__main__id05444_QIxNYNGaVg0_00040__id08406.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ - "video-driven animation", + "full-body swap", "face swap", "real", - "audio-driven animation" + "fully generated" ], "gt_answer": "real", - "pred_answer": "video-driven animation", + "pred_answer": "face swap", "correct": false, - "raw": "```json\n{\"answer\": \"video-driven animation\"}\n```" + "raw": "{\"answer\": \"face swap\"}" }, { "id": "t2_FVRA__animate__77f__main__id07054_gLIAqAQ40LQ_00053__id08610", @@ -268,17 +268,17 @@ "FVRA__animate__77f__main__id07054_gLIAqAQ40LQ_00053__id08610.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ - "audio-driven animation", + "fully generated", "real", - "video-driven animation", + "full-body swap", "face swap" ], - "gt_answer": "video-driven animation", - "pred_answer": "video-driven animation", - "correct": true, - "raw": "```json\n{\"answer\": \"video-driven animation\"}\n```" + "gt_answer": "full-body swap", + "pred_answer": "face swap", + "correct": false, + "raw": "{\"answer\": \"face swap\"}" }, { "id": "t2_RVRA__real__77f__main__id00761_C_ZBK3E8bJk_00027__id07616", @@ -286,12 +286,12 @@ "RVRA__real__77f__main__id00761_C_ZBK3E8bJk_00027__id07616.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ "real", - "audio-driven animation", + "fully generated", "face swap", - "video-driven animation" + "full-body swap" ], "gt_answer": "real", "pred_answer": "real", @@ -304,12 +304,12 @@ "FVRA__facefusion__77f__main__id06480_YQD4AsloKcY_00076__id01534.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ "real", "face swap", - "audio-driven animation", - "video-driven animation" + "fully generated", + "full-body swap" ], "gt_answer": "face swap", "pred_answer": "real", @@ -322,17 +322,17 @@ "FVRA__s2v__77f__main__id03854_yfQGoMSBDx0_00151__id00509.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ "face swap", - "video-driven animation", + "full-body swap", "real", - "audio-driven animation" + "fully generated" ], - "gt_answer": "audio-driven animation", - "pred_answer": "audio-driven animation", - "correct": true, - "raw": "```json\n{\"answer\": \"audio-driven animation\"}\n```" + "gt_answer": "fully generated", + "pred_answer": "face swap", + "correct": false, + "raw": "{\"answer\": \"face swap\"}" }, { "id": "t2_FVRA__animate__77f__main__id02959_h9Qwwv-DApg_00016__id03078", @@ -340,14 +340,14 @@ "FVRA__animate__77f__main__id02959_h9Qwwv-DApg_00016__id03078.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ "face swap", - "audio-driven animation", - "video-driven animation", + "fully generated", + "full-body swap", "real" ], - "gt_answer": "video-driven animation", + "gt_answer": "full-body swap", "pred_answer": "face swap", "correct": false, "raw": "```json\n{\"answer\": \"face swap\"}\n```" @@ -358,17 +358,17 @@ "FVRA__s2v__77f__main__id02359_hoPN5dCzSC8_00093__id06987.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ - "audio-driven animation", - "video-driven animation", + "fully generated", + "full-body swap", "real", "face swap" ], - "gt_answer": "audio-driven animation", - "pred_answer": "audio-driven animation", - "correct": true, - "raw": "```json\n{\"answer\": \"audio-driven animation\"}\n```" + "gt_answer": "fully generated", + "pred_answer": "real", + "correct": false, + "raw": "{\"answer\": \"real\"}" }, { "id": "t2_FVRA__animate__77f__main__id03681_Szvjxba6BTg_00066__id01664", @@ -376,385 +376,221 @@ "FVRA__animate__77f__main__id03681_Szvjxba6BTg_00066__id01664.mp4" ], "part": null, - "question": "How was the video in this clip made? This question is only about the video (what you see), not the sound. Normal processing such as brightness, colour or compression does not count.", + "question": "Which part of this video, if any, was generated or altered by AI? Ignore normal processing such as brightness, colour or compression.", "options": [ "real", - "video-driven animation", + "full-body swap", "face swap", - "audio-driven animation" + "fully generated" ], - "gt_answer": "video-driven animation", - "pred_answer": "face swap", + "gt_answer": "full-body swap", + "pred_answer": "real", "correct": false, - "raw": "```json\n{\"answer\": \"face swap\"}\n```" + "raw": "```json\n{\"answer\": \"real\"}\n```" } ] }, "t3": { "scores": { "n": 10, - "accuracy": 0.7, + "accuracy": 0.5, "ci95": [ - 0.3968, - 0.8922 + 0.2366, + 0.7634 ], - "balanced_accuracy": 0.7084, - "cohen_kappa": 0.4 + "balanced_accuracy": 0.3542, + "cohen_kappa": 0.2647 }, "items": [ { - "id": "t3_face_A_FVRA__animate__77f__main__id07839__QTWWQBdhWY_00029__id07683", + "id": "t3_voice_A_RVRA__real__77f__vocoded__id00948_PCy1KCIsNxo_00024__id04278", "clips": [ - "FVRA__animate__77f__main__id07839__QTWWQBdhWY_00029__id07683.mp4", - "RVRA__real__77f__main__id07839_gJgj1DxOfKA_00045__id07786.mp4" + "RVRA__real__77f__vocoded__id00948_PCy1KCIsNxo_00024__id04278.mp4", + "RVRA__real__77f__main__id00948_z-P1KSBdNU8_00128__id08581.mp4" ], - "part": "face", - "question": "Clip 2 is a real recording of one person. Look only at the faces: is the face in clip 1 the same person as in clip 2? Face means the inner face: eyes, nose and mouth.", + "part": "voice", + "question": "Clip 2 is a real recording of one person. Listen only to the voices. Is the voice in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose voice it is, still counts as real.", "options": [ - "yes", - "no" + "fake of a different person", + "fake of this person", + "real, same person", + "real, different person" ], - "gt_answer": "no", - "pred_answer": "no", + "gt_answer": "real, same person", + "pred_answer": "real, same person", "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" + "raw": "```json\n{\"answer\": \"real, same person\"}\n```" }, { - "id": "t3_face_B_FVRA__animate__77f__main__id07054_zTKKYEviZvQ_00073__id01052", + "id": "t3_face_A_RVRA__real__77f__main__id05877_q_ZAQf6KfTQ_00093__id05564", "clips": [ - "FVRA__animate__77f__main__id07054_zTKKYEviZvQ_00073__id01052.mp4", - "RVRA__real__77f__main__id01052_BPYYdOKuXSs_00029__id08590.mp4" + "RVRA__real__77f__main__id05877_q_ZAQf6KfTQ_00093__id05564.mp4", + "RVRA__real__77f__main__id05877_6QUcq_6B-LU_00011__id07768.mp4" ], "part": "face", - "question": "Clip 2 is a real recording of one person. Look only at the faces: is the face in clip 1 the same person as in clip 2? Face means the inner face: eyes, nose and mouth.", + "question": "Clip 2 is a real recording of one person. Look only at the faces (the inner face: eyes, nose and mouth). Is the face in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose face it is, still counts as real.", "options": [ - "yes", - "no" + "real, different person", + "real, same person", + "fake of this person", + "fake of a different person" ], - "gt_answer": "yes", - "pred_answer": "yes", + "gt_answer": "real, same person", + "pred_answer": "real, same person", "correct": true, - "raw": "```json\n{\"answer\": \"yes\"}\n```" + "raw": "```json\n{\"answer\": \"real, same person\"}\n```" }, { - "id": "t3_voice_B_RVRA__real__153f__vocoded__id07683_YCmGkErqGAc_00036__id07839", + "id": "t3_voice_B_RVRA__real__153f__vocoded__id00885_xiPMwAYISG0_00135__id05623", "clips": [ - "RVRA__real__153f__vocoded__id07683_YCmGkErqGAc_00036__id07839.mp4", - "RVRA__real__153f__main__id07839_41lss_yYWIM_00008__id05768.mp4" + "RVRA__real__153f__vocoded__id00885_xiPMwAYISG0_00135__id05623.mp4", + "RVRA__real__153f__main__id05623_UXUHOpRirtg_00079__id03123.mp4" ], "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is the voice in clip 1 the same person as in clip 2? Judge whose voice it is, not the sound quality.", + "question": "Clip 2 is a real recording of one person. Listen only to the voices. Is the voice in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose voice it is, still counts as real.", "options": [ - "yes", - "no" + "real, same person", + "fake of this person", + "real, different person", + "fake of a different person" ], - "gt_answer": "no", - "pred_answer": "no", - "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" - }, - { - "id": "t3_voice_A_RVFA__seed_vc__77f__main__id01505_Lf7-yDcu0NA_00049__id00761", - "clips": [ - "RVFA__seed_vc__77f__main__id01505_Lf7-yDcu0NA_00049__id00761.mp4", - "RVRA__real__77f__main__id01505_dVKZSQDG9X4_00079__id00762.mp4" - ], - "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is the voice in clip 1 the same person as in clip 2? Judge whose voice it is, not the sound quality.", - "options": [ - "yes", - "no" - ], - "gt_answer": "no", - "pred_answer": "yes", - "correct": false, - "raw": "```json\n{\"answer\": \"yes\"}\n```" - }, - { - "id": "t3_voice_B_RVFA__seed_vc__153f__main__id06210_ZG5rWainMdg_00075__id01643", - "clips": [ - "RVFA__seed_vc__153f__main__id06210_ZG5rWainMdg_00075__id01643.mp4", - "RVRA__real__153f__main__id01643_JsdoU91plJ0_00073__id08338.mp4" - ], - "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is the voice in clip 1 the same person as in clip 2? Judge whose voice it is, not the sound quality.", - "options": [ - "yes", - "no" - ], - "gt_answer": "yes", - "pred_answer": "no", + "gt_answer": "real, different person", + "pred_answer": "real, same person", "correct": false, - "raw": "```json\n{\"answer\": \"no\"}\n```" - }, - { - "id": "t3_voice_A_RVRA__real__77f__main__id01534_zaac3OQqEQE_00097__id06480", - "clips": [ - "RVRA__real__77f__main__id01534_zaac3OQqEQE_00097__id06480.mp4", - "RVRA__real__77f__main__id01534_htQIn8Um9Tg_00081__id04234.mp4" - ], - "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is the voice in clip 1 the same person as in clip 2? Judge whose voice it is, not the sound quality.", - "options": [ - "yes", - "no" - ], - "gt_answer": "yes", - "pred_answer": "yes", - "correct": true, - "raw": "```json\n{\"answer\": \"yes\"}\n```" - }, - { - "id": "t3_voice_B_RVRA__real__153f__main__id03710_gvuFmC3nXFw_00094__id06015", - "clips": [ - "RVRA__real__153f__main__id03710_gvuFmC3nXFw_00094__id06015.mp4", - "RVRA__real__153f__main__id06015_huOOE9WL2pk_00212__id01376.mp4" - ], - "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is the voice in clip 1 the same person as in clip 2? Judge whose voice it is, not the sound quality.", - "options": [ - "yes", - "no" - ], - "gt_answer": "no", - "pred_answer": "no", - "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" - }, - { - "id": "t3_voice_A_RVRA__real__153f__main__id04495_vDkHbTjw_pw_00255__id01664", - "clips": [ - "RVRA__real__153f__main__id04495_vDkHbTjw_pw_00255__id01664.mp4", - "RVRA__real__153f__main__id04495__k8mU1T87Pw_00134__id06471.mp4" - ], - "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is the voice in clip 1 the same person as in clip 2? Judge whose voice it is, not the sound quality.", - "options": [ - "yes", - "no" - ], - "gt_answer": "yes", - "pred_answer": "yes", - "correct": true, - "raw": "```json\n{\"answer\": \"yes\"}\n```" - }, - { - "id": "t3_voice_A_RVRA__real__153f__vocoded__id07398_tzZJzDzgF-g_00049__id06071", - "clips": [ - "RVRA__real__153f__vocoded__id07398_tzZJzDzgF-g_00049__id06071.mp4", - "RVRA__real__153f__main__id07398_uXZycbvHaAw_00063__id08338.mp4" - ], - "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is the voice in clip 1 the same person as in clip 2? Judge whose voice it is, not the sound quality.", - "options": [ - "yes", - "no" - ], - "gt_answer": "yes", - "pred_answer": "yes", - "correct": true, - "raw": "```json\n{\"answer\": \"yes\"}\n```" + "raw": "```json\n{\"answer\": \"real, same person\"}\n```" }, { - "id": "t3_voice_B_RVFA__seed_vc__77f__main__id05564_5eBl0n59574_00005__id05877", - "clips": [ - "RVFA__seed_vc__77f__main__id05564_5eBl0n59574_00005__id05877.mp4", - "RVRA__real__77f__main__id05877_hy-uGrBUfPM_00046__id00172.mp4" - ], - "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is the voice in clip 1 the same person as in clip 2? Judge whose voice it is, not the sound quality.", - "options": [ - "yes", - "no" - ], - "gt_answer": "yes", - "pred_answer": "no", - "correct": false, - "raw": "```json\n{\"answer\": \"no\"}\n```" - } - ] - }, - "t3_5": { - "scores": { - "n": 10, - "accuracy": 0.8, - "ci95": [ - 0.4902, - 0.9433 - ], - "balanced_accuracy": 0.75, - "cohen_kappa": 0.5455 - }, - "items": [ - { - "id": "t35_face_A_RVRA__real__77f__main__id05877_q_ZAQf6KfTQ_00093__id05564", + "id": "t3_face_B_RVRA__real__153f__main__id06713_5qTLKWiq73k_00074__id05042", "clips": [ - "RVRA__real__77f__main__id05877_q_ZAQf6KfTQ_00093__id05564.mp4", - "RVRA__real__77f__main__id05877_6QUcq_6B-LU_00011__id07768.mp4" + "RVRA__real__153f__main__id06713_5qTLKWiq73k_00074__id05042.mp4", + "RVRA__real__153f__main__id05042_K1HMS1K0kWI_00010__id00144.mp4" ], "part": "face", - "question": "Clip 2 is a real recording of one person. Look only at the faces: is clip 1 a real recording of that same person's face? Answer no if the face in clip 1 belongs to someone else, or was swapped, changed or made by a computer, even if it looks like that person. A real recording that was only processed, without changing whose face it is, still counts as real. Face means the inner face: eyes, nose and mouth.", + "question": "Clip 2 is a real recording of one person. Look only at the faces (the inner face: eyes, nose and mouth). Is the face in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose face it is, still counts as real.", "options": [ - "yes", - "no" + "real, same person", + "fake of a different person", + "fake of this person", + "real, different person" ], - "gt_answer": "yes", - "pred_answer": "yes", + "gt_answer": "real, different person", + "pred_answer": "real, different person", "correct": true, - "raw": "```json\n{\"answer\": \"yes\"}\n```" + "raw": "```json\n{\"answer\": \"real, different person\"}\n```" }, { - "id": "t35_voice_A_RVRA__real__77f__main__id08590_UhRFZwnafIk_00068__id04236", + "id": "t3_voice_A_RVFA__seed_vc__153f__main__id04049_JCDeI7UwWnE_00080__id08096", "clips": [ - "RVRA__real__77f__main__id08590_UhRFZwnafIk_00068__id04236.mp4", - "RVRA__real__77f__main__id08590_3BGHjstAIzg_00003__id08610.mp4" + "RVFA__seed_vc__153f__main__id04049_JCDeI7UwWnE_00080__id08096.mp4", + "RVRA__real__153f__main__id04049_Flu_4soH0mo_00003__id04528.mp4" ], "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is clip 1 a real recording of that same person's voice? Answer no if the voice in clip 1 belongs to someone else, or was converted, changed or made by a computer, even if it sounds like that person. A real recording that was only processed, without changing whose voice it is, still counts as real.", + "question": "Clip 2 is a real recording of one person. Listen only to the voices. Is the voice in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose voice it is, still counts as real.", "options": [ - "yes", - "no" + "fake of a different person", + "real, same person", + "real, different person", + "fake of this person" ], - "gt_answer": "yes", - "pred_answer": "no", + "gt_answer": "fake of a different person", + "pred_answer": "real, different person", "correct": false, - "raw": "```json\n{\"answer\": \"no\"}\n```" + "raw": "```json\n{\"answer\": \"real, different person\"}\n```" }, { - "id": "t35_voice_B_RVFA__seed_vc__77f__main__id04528_UF0U3evnpC4_00063__id03281", + "id": "t3_voice_B_RVFA__seed_vc__77f__main__id07574_DbcyMh1l_Ng_00043__id06429", "clips": [ - "RVFA__seed_vc__77f__main__id04528_UF0U3evnpC4_00063__id03281.mp4", - "RVRA__real__77f__main__id03281_B8yQH7WxPc0_00033__id04049.mp4" + "RVFA__seed_vc__77f__main__id07574_DbcyMh1l_Ng_00043__id06429.mp4", + "RVRA__real__77f__main__id06429_0aTJ9yNMAx0_00002__id06427.mp4" ], "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is clip 1 a real recording of that same person's voice? Answer no if the voice in clip 1 belongs to someone else, or was converted, changed or made by a computer, even if it sounds like that person. A real recording that was only processed, without changing whose voice it is, still counts as real.", + "question": "Clip 2 is a real recording of one person. Listen only to the voices. Is the voice in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose voice it is, still counts as real.", "options": [ - "yes", - "no" + "fake of this person", + "real, different person", + "real, same person", + "fake of a different person" ], - "gt_answer": "no", - "pred_answer": "no", - "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" + "gt_answer": "fake of this person", + "pred_answer": "fake of a different person", + "correct": false, + "raw": "```json\n{\"answer\": \"fake of a different person\"}\n```" }, { - "id": "t35_face_A_RVRA__real__153f__main__id00885_yMWJySOEuTI_00138__id00739", + "id": "t3_face_A_RVRA__real__153f__main__id00885_yMWJySOEuTI_00138__id00739", "clips": [ "RVRA__real__153f__main__id00885_yMWJySOEuTI_00138__id00739.mp4", "RVRA__real__153f__main__id00885_xiPMwAYISG0_00135__id05623.mp4" ], "part": "face", - "question": "Clip 2 is a real recording of one person. Look only at the faces: is clip 1 a real recording of that same person's face? Answer no if the face in clip 1 belongs to someone else, or was swapped, changed or made by a computer, even if it looks like that person. A real recording that was only processed, without changing whose face it is, still counts as real. Face means the inner face: eyes, nose and mouth.", + "question": "Clip 2 is a real recording of one person. Look only at the faces (the inner face: eyes, nose and mouth). Is the face in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose face it is, still counts as real.", "options": [ - "yes", - "no" + "real, different person", + "fake of this person", + "real, same person", + "fake of a different person" ], - "gt_answer": "yes", - "pred_answer": "no", + "gt_answer": "real, same person", + "pred_answer": "real, different person", "correct": false, - "raw": "```json\n{\"answer\": \"no\"}\n```" + "raw": "```json\n{\"answer\": \"real, different person\"}\n```" }, { - "id": "t35_face_B_FVRA__facefusion__153f__main__id04171_S7q4HA1KM-8_00038__id04126", - "clips": [ - "FVRA__facefusion__153f__main__id04171_S7q4HA1KM-8_00038__id04126.mp4", - "RVRA__real__153f__main__id04126_XcG4iWxXxUM_00120__id04171.mp4" - ], - "part": "face", - "question": "Clip 2 is a real recording of one person. Look only at the faces: is clip 1 a real recording of that same person's face? Answer no if the face in clip 1 belongs to someone else, or was swapped, changed or made by a computer, even if it looks like that person. A real recording that was only processed, without changing whose face it is, still counts as real. Face means the inner face: eyes, nose and mouth.", - "options": [ - "yes", - "no" - ], - "gt_answer": "no", - "pred_answer": "no", - "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" - }, - { - "id": "t35_face_B_RVRA__real__77f__main__id08808_vQV6tYM_-3M_00053__id07697", + "id": "t3_face_B_RVRA__real__77f__main__id08808_vQV6tYM_-3M_00053__id07697", "clips": [ "RVRA__real__77f__main__id08808_vQV6tYM_-3M_00053__id07697.mp4", "RVRA__real__77f__main__id07697_diNcpyVMCk8_00047__id07630.mp4" ], "part": "face", - "question": "Clip 2 is a real recording of one person. Look only at the faces: is clip 1 a real recording of that same person's face? Answer no if the face in clip 1 belongs to someone else, or was swapped, changed or made by a computer, even if it looks like that person. A real recording that was only processed, without changing whose face it is, still counts as real. Face means the inner face: eyes, nose and mouth.", + "question": "Clip 2 is a real recording of one person. Look only at the faces (the inner face: eyes, nose and mouth). Is the face in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose face it is, still counts as real.", "options": [ - "yes", - "no" + "real, same person", + "fake of this person", + "fake of a different person", + "real, different person" ], - "gt_answer": "no", - "pred_answer": "no", + "gt_answer": "real, different person", + "pred_answer": "real, different person", "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" + "raw": "```json\n{\"answer\": \"real, different person\"}\n```" }, { - "id": "t35_voice_B_RVFA__seed_vc__153f__main__id07780_9oRm6SDT2Ak_00002__id05042", + "id": "t3_voice_B_RVFA__seed_vc__153f__main__id02359_QiCI0mefCx4_00030__id03078", "clips": [ - "RVFA__seed_vc__153f__main__id07780_9oRm6SDT2Ak_00002__id05042.mp4", - "RVRA__real__153f__main__id05042_SIx5G21ZMN8_00067__id06343.mp4" + "RVFA__seed_vc__153f__main__id02359_QiCI0mefCx4_00030__id03078.mp4", + "RVRA__real__153f__main__id03078_4XN_hNJCe9E_00031__id06987.mp4" ], "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is clip 1 a real recording of that same person's voice? Answer no if the voice in clip 1 belongs to someone else, or was converted, changed or made by a computer, even if it sounds like that person. A real recording that was only processed, without changing whose voice it is, still counts as real.", + "question": "Clip 2 is a real recording of one person. Listen only to the voices. Is the voice in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose voice it is, still counts as real.", "options": [ - "yes", - "no" - ], - "gt_answer": "no", - "pred_answer": "no", - "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" - }, - { - "id": "t35_voice_B_RVRA__real__77f__vocoded__id01104_wmkjwD45aYo_00109__id08710", - "clips": [ - "RVRA__real__77f__vocoded__id01104_wmkjwD45aYo_00109__id08710.mp4", - "RVRA__real__77f__main__id08710_EMa5tWHB6Mo_00048__id01858.mp4" + "real, same person", + "real, different person", + "fake of a different person", + "fake of this person" ], - "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is clip 1 a real recording of that same person's voice? Answer no if the voice in clip 1 belongs to someone else, or was converted, changed or made by a computer, even if it sounds like that person. A real recording that was only processed, without changing whose voice it is, still counts as real.", - "options": [ - "yes", - "no" - ], - "gt_answer": "no", - "pred_answer": "no", - "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" - }, - { - "id": "t35_face_B_FVRA__s2v__77f__main__id06427_FE4m2_FwOTA_00050__id08209", - "clips": [ - "FVRA__s2v__77f__main__id06427_FE4m2_FwOTA_00050__id08209.mp4", - "RVRA__real__77f__main__id08209_O8r8MtzKdfI_00034__id06429.mp4" - ], - "part": "face", - "question": "Clip 2 is a real recording of one person. Look only at the faces: is clip 1 a real recording of that same person's face? Answer no if the face in clip 1 belongs to someone else, or was swapped, changed or made by a computer, even if it looks like that person. A real recording that was only processed, without changing whose face it is, still counts as real. Face means the inner face: eyes, nose and mouth.", - "options": [ - "yes", - "no" - ], - "gt_answer": "no", - "pred_answer": "no", - "correct": true, - "raw": "```json\n{\"answer\": \"no\"}\n```" + "gt_answer": "fake of this person", + "pred_answer": "real, different person", + "correct": false, + "raw": "```json\n{\"answer\": \"real, different person\"}\n```" }, { - "id": "t35_voice_A_RVRA__real__153f__main__id00739_OKXRt9jmIZQ_00050__id04597", + "id": "t3_voice_A_RVRA__real__153f__vocoded__id01124_7Y-51ooBIWQ_00040__id09218", "clips": [ - "RVRA__real__153f__main__id00739_OKXRt9jmIZQ_00050__id04597.mp4", - "RVRA__real__153f__main__id00739_KZti9h4Uyrc_00016__id00885.mp4" + "RVRA__real__153f__vocoded__id01124_7Y-51ooBIWQ_00040__id09218.mp4", + "RVRA__real__153f__main__id01124_2hg7q9O8xLo_00015__id08748.mp4" ], "part": "voice", - "question": "Clip 2 is a real recording of one person. Listen only to the voices: is clip 1 a real recording of that same person's voice? Answer no if the voice in clip 1 belongs to someone else, or was converted, changed or made by a computer, even if it sounds like that person. A real recording that was only processed, without changing whose voice it is, still counts as real.", + "question": "Clip 2 is a real recording of one person. Listen only to the voices. Is the voice in clip 1 real or fake, and is it the same person as in clip 2? A real recording that was only processed, without changing whose voice it is, still counts as real.", "options": [ - "yes", - "no" + "real, same person", + "fake of this person", + "real, different person", + "fake of a different person" ], - "gt_answer": "yes", - "pred_answer": "yes", + "gt_answer": "real, same person", + "pred_answer": "real, same person", "correct": true, - "raw": "```json\n{\"answer\": \"yes\"}\n```" + "raw": "{\"answer\": \"real, same person\"}" } ] }