Download RESEARCH_LOG.md from assa8945/MoE_detector: direct link, hf CLI and curl.
- Browser
- Download file 49.5 kB
-
https://huggingface.co/assa8945/MoE_detector/resolve/main/RESEARCH_LOG.md
- Command line
-
hf download hf://assa8945/MoE_detector/RESEARCH_LOG.md
-
curl -L -o RESEARCH_LOG.md https://huggingface.co/assa8945/MoE_detector/resolve/main/RESEARCH_LOG.md
MoE detector: research log (Setting G)
Last update: 2026-11-15 (section 16). This is a private working log. All numbers come from local runs on one test set (Setting G). There is no peer review yet. The text follows ASD-STE100 Simplified Technical English.
Working title: Route by Reliability: a degradation-aware mixture of forensic experts for the detection of AI-generated video.
0. Target test set: mix_setting_v1 (added 2026-10-25)
Since 2026-10-24 the only target test set is mix_setting_v1. Sections 1β7 report Setting G and stay as development history. The term "re-encodes" replaces "hops" for the length of a repost chain.
- Set:
batch_infer/combination_test/mix_setting_v1/manifest.csv. 10,000 slots; 9,939 videos are ready (2,949 real, 6,990 fake). 61 SkyReels-V3 extension slots are not encoded yet. - Re-encodes: every video has 1, 5, 10 or 20 re-encodes (4:3:2:1). There is no set with 0 re-encodes.
- Platforms: 12 profiles (6 old + 6 measured
*_real). Chains with 5 or more re-encodes mix the platforms. No mix platform is a router platform. - Protocol: as in section 2. Platt on the router calib_fit, Youden threshold on the router calib_select. No mix labels are fitted.
0.1 Main result (n = 9,939; 95% CI from 1,000 paired bootstrap resamples grouped by source video, 4,689 groups)
| Model | AUC [95% CI] | Macro F1 [95% CI] | TPR | TNR |
|---|---|---|---|---|
| E2 D3 | 0.693 [0.682, 0.705] | 0.554 [0.543, 0.564] | 0.476 | 0.757 |
| E3β² WaveRep-DINOv3 | 0.803 [0.794, 0.811] | 0.695 [0.685, 0.705] | 0.841 | 0.540 |
| Mean of z (E2, E3β²) | 0.842 [0.834, 0.849] | 0.735 [0.726, 0.745] | 0.844 | 0.627 |
- The mean of z is better than E3β² alone: ΞAUC +0.039 [0.034, 0.043], ΞF1 +0.041 [0.034, 0.049].
- E1 (released WaveRep) is not scored on mix_setting_v1 yet.
| Re-encodes (mean of z) | 1 | 5 | 10 | 20 |
|---|---|---|---|---|
| AUC / Macro F1 | 0.904 / 0.803 | 0.819 / 0.711 | 0.792 / 0.694 | 0.736 / 0.588 |
| Real TNR | 0.71 | 0.68 | 0.57 | 0.25 |
| Fake family vs all reals (mean of z) | AUC / Macro F1 |
|---|---|
| HunyuanVideo-1.5 | 0.963 / 0.746 |
| Wan2.2 | 0.874 / 0.716 |
| LTX-2 | 0.832 / 0.720 |
| Wan-animate / replace | 0.826 / 0.476 |
| SkyReels-V3 | 0.700 / 0.613 |
| Relight-Video | 0.641 / 0.455 |
| Platforms (mean of z) | AUC / Macro F1 |
|---|---|
| Old profiles only | 0.896 / 0.792 |
| Measured profiles only | 0.914 / 0.813 |
| Mixed chains | 0.790 / 0.690 |
- The weak points are the same as on G: false alarms on real videos after many re-encodes, SkyReels-V3 and Relight-Video.
- Mixed-platform chains are harder than chains on one profile group.
0.2 Studies on mix_setting_v1 (training-free, no mix labels fitted)
| Study | Model | AUC / Macro F1 | Key detail |
|---|---|---|---|
| Problem 1: real-reference score (Β§7.5, one bin) | Real reference alone | 0.798 / 0.685 | 20 re-encodes: 0.718 / 0.658, real TNR 0.63 (E3β²: 0.676 / 0.540, TNR 0.18) |
| Mean of z (E2, E3β², real reference) | 0.837 / 0.723 | no overall gain | |
| Problem 2: change-point scores on the E3β² frame logits | Mean of z (E2, E3β², CUSUM) | 0.839 / 0.730 | SkyReels-V3 0.700 β 0.761; LTX-2 0.832 β 0.798 |
| Mean of z (E2, E3β², late β early) | 0.839 / 0.731 | SkyReels-V3 0.739 | |
| Feature change, distance to start | 0.499, 0.480 | no signal | |
| Design 2: periodic VAE pattern (F4, frames de-duplicated) | F4 alone | 0.611 / 0.515 | no help on SkyReels-V3 (0.508) |
| Mean of z (E2, E3β², F4) | 0.844 / 0.733 | +0.002 AUC, no clear gain | |
| Design 3 pilot: change-point transformer on feature splices | Transformer alone | 0.749 / 0.668 | trained on router_train feature splices |
| Mean of z (E2, E3β², transformer) | 0.829 / 0.728 | worse; SkyReels-V3 0.721 |
- Problem 2: on mix, the SkyReels-V3 frame logit also goes up over the clip (β4.0 at frame 3, β0.5 at frame 15). A fixed temporal statistic helps SkyReels-V3 but makes the other generators worse.
- Real videos with shot cuts do not give more false alarms with these scores (TNR 0.67 vs 0.56 for the other real videos).
- Design 2 (analysis on our own videos, no mix):
- Native gen4 videos have a clear 4-frame pattern in the noise residual: median F4 3β12; period 4 is the strongest period in 57β78% of the videos.
- Platform frame-rate conversion adds a period-5 pattern to all videos.
- With duplicate frames removed, the F4 AUC on router videos is 0.72 at 0 re-encodes and 0.59β0.66 after 1β20 re-encodes. The cue is real but weak.
- Design 3 pilot:
- The model finds the seam in router_val feature splices to within about 2 frames (router_val AUC 0.90).
- It does not transfer to the new generators in mix. The router fakes are old generators, and feature splices have no real VAE seam.
- Design 3 needs real extension clips from current generators.
- Data:
claude_workspace/csv/{mix_moe_*,mix_bootstrap,problem1_mix,problem2_changepoint_mix,mix_extra_expert_design2,design2_periodic*,design3_pilot_mix}.csv.
0.3 Expert studies on mix_setting_v1 (added 2026-10-26; GPU runs, no mix labels fitted)
Each new model replaces E3β² (or is added to it) in the mean of z with E2. Same protocol as Β§0.1. Reference: E3β² 0.803 / 0.695; mean of z (E2, E3β²) 0.842 / 0.735.
| Study | Model alone AUC / Macro F1 | Mean of z with E2 AUC / Macro F1 | 20 re-encodes (with E2) | Key detail |
|---|---|---|---|---|
| E1 released WaveRep, added to (E2, E3β²) | 0.637 / 0.495 | 0.837 / 0.709 (E2, E3β², E1) | 0.731 / 0.654 | E1 is near chance after 5+ re-encodes; no gain |
| E3β² without HunyuanVideo-1.5 | 0.787 / 0.687 | 0.834 / 0.719 | 0.715 / 0.633 | HunyuanVideo-1.5 (E3β² alone) 0.940 β 0.891 |
| E3β² without LTX-2 | 0.763 / 0.667 | 0.810 / 0.708 | 0.705 / 0.546 | LTX-2 (E3β² alone) 0.791 β 0.723 |
| E3β² without SkyReels-V3 | 0.796 / 0.700 | 0.840 / 0.715 | 0.737 / 0.667 | SkyReels-V3 (E3β² alone) 0.681 β 0.651 |
| E3β² without Wan2.2 | 0.798 / 0.696 | 0.839 / 0.737 | 0.720 / 0.611 | Wan2.2 (E3β² alone) 0.810 β 0.809 |
| Paired views, Ξ» = 0 (BCE on clean + re-encoded view) | 0.790 / 0.690 | 0.833 / 0.718 | 0.696 / 0.606 | control for Ξ» = 1 |
| Paired views, Ξ» = 1 (+ consistency loss) | 0.776 / 0.679 | 0.824 / 0.721 | 0.707 / 0.624 | worse than E3β² and than Ξ» = 0 |
E1: the released WaveRep adds no information to (E2, E3β²) on mix.
Leave one generator out: overall AUC with E2 drops by 0.002β0.032. The left-out family loses the most (LTX-2 β0.068 for E3β² alone). No single training generator is necessary; together they give the full result.
Paired consistency (problem 1): an explicit compression-invariance loss does not help. Both paired runs are below E3β² at all re-encode levels, also at 20 re-encodes. The stochastic re-encode augmentation of E3β² (p 0.3) is sufficient.
Re-encode-aware gate (crf_hat): the router CRF head (router_train only) predicts the re-encode level of each mix video (Spearman 0.65 with the true count; it underestimates mix chains: median crf_hat 1.46 at 5 re-encodes vs 2.80 on router). Six designs on (z_d, z_v), all fitted on router calib only, 3 or 5 crf_hat bins:
Design (3 bins) AUC / Macro F1 ΞAUC [95% CI] vs mean of z 20 re-encodes AUC / F1 / TNR Mean of z (E2, E3β²), one threshold (current) 0.842 / 0.735 β 0.736 / 0.588 / 0.25 Threshold per bin 0.842 / 0.737 +0.000 [+0.000, +0.000] 0.736 / 0.618 / 0.33 Expert weights per bin (E2 weight 0.42 β 0.59 with depth) 0.842 / 0.736 +0.001 [β0.001, +0.002] 0.736 / 0.631 / 0.38 Weights + threshold per bin 0.842 / 0.735 +0.001 [β0.001, +0.002] 0.736 / 0.633 / 0.38 Stacker with crf_hat interactions 0.841 / 0.734 β0.001 [β0.003, +0.001] 0.738 / 0.654 / 0.47 Soft gate w(crf_hat) 0.842 / 0.730 +0.000 [β0.001, +0.001] 0.743 / 0.656 / 0.45 No design changes the overall AUC or Macro F1 (all CIs include 0). They only move the false-alarm / miss balance at 20 re-encodes (TNR 0.25 β 0.47). This agrees with Β§7.3 on G: the plain mean of z stays the MoE. Data:
csv/mix_crf_gate.csv,moe_mix/crf_hat_mix.csv.UCF-Net-style feature fusion on mix (Β§7.1 method, frozen features, trained on router_train, 5 seeds;
ucf_fusion_mix.py):Method (E2 + E3β²) AUC / Macro F1 (5-seed ensemble) Mean Β± sd over seeds 20 re-encodes AUC / F1 Mean of z (score level, current) 0.842 / 0.735 β 0.736 / 0.588 Logistic stacker on z (score level) 0.843 / 0.737 β 0.739 / 0.605 UCF: LEA + UAF 0.842 / 0.740 0.838 Β± 0.006 / 0.736 Β± 0.008 0.720 / 0.620 LEA + sum 0.841 / 0.743 0.839 Β± 0.004 / 0.736 Β± 0.005 0.722 / 0.638 LEA + concatenation 0.841 / 0.739 0.838 Β± 0.005 / 0.735 Β± 0.005 0.721 / 0.631 LEA + cross-attention 0.840 / 0.737 0.835 Β± 0.006 / 0.732 Β± 0.006 0.716 / 0.626 UAF without LEA 0.815 / 0.713 0.812 Β± 0.004 / 0.710 Β± 0.002 β Same result as on G: feature fusion equals score fusion (AUC 0.838β0.842 vs 0.842); UAF equals a plain sum; the UAF weights stay near 0.43 / 0.57 at all re-encode levels. Feature fusion is worse at 20 re-encodes in AUC (0.720 vs 0.736). Data:
csv/ucf_fusion_mix*.csv.Data:
claude_workspace/csv/mix_extra_expert_{e1,logo,pair}.csv; logsclaude_workspace/log/mix_extra_expert_{e1,logo,pair}.log.
1. Test set
- The test set is Setting G. It has 9,468 videos that the experts can score: 3,010 real and 6,458 fake.
- The fakes come from 7 generators.
- Each video has a re-encode depth of 0, 1, 5, 10 or 20 hops. One hop is one re-encode by a simulated platform. The chains use 6 platforms.
- The benchmark log gives more data about Setting G.
2. Protocol (all MoE results)
- We do not use G labels in a part that we fit.
- We fit Platt scaling for each expert on
calib_fit. - We train the router heads and the stackers on
router_train. - We select the temperature Ο on
router_val. - The decision threshold is the Youden J point on
calib_select. We test on G. - The router and calibration set has 9,577 videos:
- 5,246 real videos from ActivityNet and InternVid.
- Fakes from CogVideoX (3,064), HunyuanVideo (767), Kling (250), OpenSora (199) and Sora (51). 500 of these videos come from GenVidBench.
- Each video is re-encoded 0, 1, 5, 10 or 20 hops. The platforms are not the G platforms:
whatsapp_hd,wechat_chat,genericand their bad-connection variants.
- Metrics: ACC, Macro F1, AUC, TPR (fake), TNR (real) and TPR at 1% FPR. We always show AUC and Macro F1 together.
- Bootstrap: 1,000 paired resamples, grouped by source video name. We give the 95% percentile interval.
3. Experts
| ID | Expert | Training data | G AUC / Macro F1 (alone) |
|---|---|---|---|
| E1 | WaveRep: the released checkpoint (DINOv2 ViT-B/14 reg4, weights_dinov2_G4.ckpt). Score = mean frame logit. |
Data of the authors; frozen | 0.749 / 0.563 (threshold 0.5) |
| E2 | D3: X-CLIP-B/16 features; std of the second-order frame-feature distance. Real if dis_std > 5. | No training | 0.695 / 0.646 |
| E3 (old) | Frozen DINOv3 ViT-L/16 temporal-difference features and a linear probe | expert_train (CogVideoX fakes only) | 0.606 / 0.560 |
| E3β² (new) | WaveRep-DINOv3: our re-implementation of the WaveRep recipe on DINOv3 ViT-L/16. Full fine-tune, 5 s clips, re-encode augmentation. | gen4 (4 modern generators, see section 8) | 0.871 / 0.756 |
4. Main result: Setting G (n = 9,468)
| Method | ACC | Macro F1 | AUC | TPR fake | TNR real | TPR@1%FPR |
|---|---|---|---|---|---|---|
| Old router E1βE3 (Ο = 2) | 0.746 | 0.718 | 0.826 | 0.774 | 0.684 | 0.349 |
| Old mean of z (E1, E2, E3) | 0.740 | 0.713 | 0.819 | 0.768 | 0.679 | 0.340 |
| E3β² WaveRep-DINOv3 alone | 0.794 | 0.751 | 0.871 | 0.887 | 0.594 | 0.432 |
| Router E1, E2, E3β² (Ο = 2) | 0.805 | 0.785 | 0.887 | 0.816 | 0.783 | 0.432 |
| Mean of z (E2, E3β²) | 0.816 | 0.783 | 0.892 | 0.885 | 0.668 | 0.449 |
| Mean of z (E1, E2, E3β²) | 0.799 | 0.784 | 0.894 | 0.778 | 0.846 | 0.513 |
| Logistic stacker on z (E1, E2, E3β²): best model | 0.813 | 0.795 | 0.896 | 0.818 | 0.804 | 0.494 |
| [reference, uses G labels] in-G logistic on z | 0.820 | 0.792 | 0.896 | 0.866 | 0.720 | 0.506 |
The best model is the logistic stacker on the Platt scores z of E1, E2 and E3β². It is equal to the in-G reference that uses G labels.
4.1 Results for each hop (AUC / Macro F1)
| Model | 0 | 1 | 5 | 10 | 20 |
|---|---|---|---|---|---|
| E1 WaveRep (released) | 0.926 / 0.826 | 0.772 / 0.627 | 0.556 / 0.326 | 0.477 / 0.307 | 0.490 / 0.334 |
| E2 D3 | 0.700 / 0.580 | 0.681 / 0.582 | 0.712 / 0.560 | 0.678 / 0.535 | 0.702 / 0.510 |
| E3β² WaveRep-DINOv3 | 0.936 / 0.824 | 0.879 / 0.770 | 0.766 / 0.676 | 0.706 / 0.566 | 0.683 / 0.483 |
| Mean of z (E2, E3β²) | 0.941 / 0.839 | 0.894 / 0.794 | 0.821 / 0.727 | 0.764 / 0.652 | 0.752 / 0.584 |
| Mean of z (E1, E2, E3β²) | 0.952 / 0.872 | 0.904 / 0.795 | 0.815 / 0.646 | 0.747 / 0.644 | 0.753 / 0.640 |
| Old router E1βE3 | 0.927 / 0.815 | 0.820 / 0.726 | 0.682 / 0.582 | 0.665 / 0.580 | 0.632 / 0.549 |
- Each cell is the real-vs-fake result on the G videos with that hop count. Each cell contains real and fake videos.
- The threshold is the same at all hops.
- The fusion degrades less than each single expert. But it still degrades: AUC goes from 0.952 to 0.753 between 0 and 20 hops.
4.2 Findings
- E3β² in place of the old E3: AUC increases by 0.06β0.07 and Macro F1 by 0.07β0.08 at all hop depths.
- E3β² in place of E1 is worse: router 0.884 / 0.761, mean of z 0.886 / 0.755. Keep WaveRep.
- Routing gives no gain over simple fusion: router 0.887, mean of z 0.894. The gates stay almost uniform (0.28β0.40).
- Contribution of each expert. We use a paired bootstrap on G at threshold 0, with E3β² seed 0 / seed 1.
- Add WaveRep to (D3, E3β²): ΞAUC +0.002 / +0.006; ΞTPR@1%FPR +0.057 [+0.026, +0.079] / +0.089 [+0.047, +0.113].
- Add E3β² to (WaveRep, D3): ΞAUC +0.073 [+0.065, +0.080] / +0.071 [+0.063, +0.078]; ΞMacro F1 +0.050 / +0.059.
- Add D3 to (WaveRep, E3β²): ΞAUC +0.019 [+0.017, +0.021] / +0.018 [+0.016, +0.021].
- E1 is not necessary for AUC or F1. Mean of z (E2, E3β²) gives 0.892 / 0.783. E1 adds TPR@1%FPR (+0.064) and results at 0β1 hops.
- WaveRep and E3β² are complementary.
- On GenVideo (clean video, older generators, n = 977): WaveRep 0.998 / 0.955; E3β² 0.948 / 0.818. WaveRep wins on all 10 generators.
- On G (modern generators, repost chains): E3β² wins at all hops, on all platforms and on 6 of 7 generators.
- WaveRep wins only on Relight-Video (0.664 vs 0.578). E3β² did not see this generator in training.
5. Failure analysis: mean of z (E2, E3β²)
The model makes 1,740 errors on G: 999 false alarms on real videos and 741 missed fakes.
5.1 False alarms on real videos (57% of errors)
| Hops | 0 | 1 | 5 | 10 | 20 |
|---|---|---|---|---|---|
| Error rate on real videos | 0.26 | 0.27 | 0.34 | 0.56 | 0.79 |
- Deep re-encoding makes real videos look fake.
- All platforms have this problem (0.36β0.44). Both real sources have it: "cuts" 30%, "crawl" 38%.
5.2 Missed fakes (43% of errors)
| Generator | Miss rate | Part of missed fakes |
|---|---|---|
| SkyReels-V3 | 0.32 | 51% |
| Relight-Video | 0.45 | 12% |
| LTX-Video | 0.10 (0.19β0.25 at 5β10 hops) | 24% |
| HunyuanVideo, Wan-2.2, Wan-Lightning | β€ 0.06 | small |
5.3 Which expert is wrong
| Case | Part of errors |
|---|---|
| Both experts are wrong (a better fusion cannot fix these) | 41% |
| Only D3 is correct | 39% |
| Only E3β² is correct | 20% |
6. SkyReels-V3: real start, generated end
SkyReels-V3 videos in G start with real frames, and the generated part comes later. The per-frame E3β² logits show this pattern (16 frames, uniform over the clip, 0 hops):
| Frame position | 0 | 4 | 8 | 12 | 15 |
|---|---|---|---|---|---|
| SkyReels-V3 | β3.3 | β5.2 | β2.7 | β0.3 | +1.3 |
| Real | β4.8 | β5.5 | β5.5 | β5.5 | β5.6 |
| Wan-2.2 | β3.0 | +5.1 | +5.4 | +5.6 | +5.4 |
The mean over all frames adds the real start to the score. Thus the score goes toward "real". We tested other temporal pooling methods on E3β² alone, with no retraining and the same protocol:
| E3β² pooling | G AUC / Macro F1 | SkyReels AUC | Relight AUC | LTX AUC |
|---|---|---|---|---|
| Mean (current) | 0.871 / 0.751 | 0.724 | 0.578 | 0.885 |
| Maximum of sliding 4-frame windows | 0.877 / 0.764 | 0.764 | 0.580 | 0.883 |
| Mean of the 8 highest frames | 0.876 / 0.759 | 0.749 | 0.574 | 0.889 |
| Mean of the last quarter | 0.880 / 0.765 | 0.800 | 0.575 | 0.862 |
- The last-quarter mean is the best, but it is correct only for one format: a generated end. The sliding-window maximum does not use a position, so we recommend it.
- Possible next steps:
- Multiple-instance scoring at inference: more frames and the maximum window score, for E3β² and D3.
- Training on partly fake clips: a real start and a generated end, with a multiple-instance loss.
- A seam feature: a change point in the per-frame score. The late-minus-early score alone gives SkyReels AUC 0.740.
7. Studies with negative or small results
7.1 UCF-Net-style feature fusion (E2, E3β²)
- We adapted the fusion of UCF-Net (Zou et al., arXiv 2609.07670) to E2 and E3β².
- Branches: both backbones are frozen, with no LoRA.
- E3β² branch: the CLS token of the 25 layers, mean over 16 frames.
- E2 branch: one temporal token per X-CLIP layer (13 layers). The token is the per-channel std over time of the frame differences.
- LEA: a layer gate and a shared bottleneck expert (d = 256), into a common dimension of 1024.
- UAF: branch weight = 1 β normalized channel entropy.
- Training: focal loss. We train on router_train (5,548 videos), with 5 seeds.
| Method (E2 + E3β²) | ACC | Macro F1 | AUC | TPR | TNR | TPR@1% |
|---|---|---|---|---|---|---|
| Mean of z (score level) | 0.816 | 0.783 | 0.892 | 0.885 | 0.668 | 0.449 |
| Logistic stacker (score level) | 0.817 | 0.786 | 0.892 | 0.878 | 0.687 | 0.446 |
| UCF: LEA + UAF | 0.816 | 0.790 | 0.888 | 0.854 | 0.734 | 0.390 |
| LEA + sum (average) | 0.816 | 0.791 | 0.889 | 0.855 | 0.734 | 0.389 |
| LEA + concatenation | 0.814 | 0.789 | 0.886 | 0.850 | 0.737 | 0.380 |
| LEA + cross-attention | 0.816 | 0.793 | 0.891 | 0.844 | 0.756 | 0.408 |
| UAF without LEA | 0.798 | 0.769 | 0.873 | 0.848 | 0.693 | 0.386 |
- UAF is equal to a plain sum (0.888 vs 0.889). This does not agree with the +2.03 mAUC gain in the paper.
- The UAF weights stay near 0.42 / 0.58 at all hops.
- Feature fusion does not do better than score fusion. At 20 hops it is worse (AUC 0.711 vs 0.752).
7.2 Light stability as a fusion input (for Relight-Video)
- We used the smoothness score of
relit_video_evaluation.py:- Signals: the mean of the bright pixels (gray > 125), the number of bright pixels, and the frame-to-frame change.
- S = exp(βa Β· mean|Ξx| / range(x)). The final score is the mean of the 3 values of S.
- Clip length is a confound:
- On G, S gives relit-vs-real AUC 0.801. But relit videos are short: median 85 frames, real videos 301 frames.
- Clip length alone gives AUC 0.973.
- Against real videos of the same length, S gives only 0.598.
- The direction is the opposite of the hypothesis:
- We controlled the length: 24 fps, 64-frame windows, mean over the windows.
- With this control, relit videos have more stable light than real videos (AUC 0.42 with "low stability = fake").
- In-G 7:3 split (uses G labels):
- We split the 173 relit source videos by source video and repeated the split 20 times.
- We added the windowed stability to the stacker.
- Relit AUC goes from 0.717 to 0.744, and it increases in 18 of 20 splits. All-G AUC goes from 0.895 to 0.902.
- The original metric does not help (relit AUC 0.701).
- Clip length alone gives relit AUC 0.855. This is a shortcut in the benchmark.
- Data limit: all relit videos on the disk are in G. A correct test needs new relit videos from expert_train sources, with the same length and fps as the real videos.
7.3 Degradation descriptor (contribution 2 of the plan)
The descriptor is d = [crf_hat, log motion, log Laplacian variance, log height].
crf_hat follows the hop depth on G (Spearman 0.66) and does not show the label:
Hops 0 1 5 10 20 Median crf_hat, real 0.93 0.98 2.23 3.34 4.66 Median crf_hat, fake 0.62 0.80 2.15 3.25 4.73 We compared the stacker with and without d. We used the same protocol and a paired bootstrap.
| Experts | Stacker input | ΞAUC [95% CI] | ΞMacro F1 [95% CI] |
|---|---|---|---|
| E1, E2, E3β² | + crf_hat | +0.001 [+0.001, +0.002] | β0.005 [β0.009, β0.000] |
| E1, E2, E3β² | + crf_hat + zΒ·crf_hat | +0.001 [β0.001, +0.002] | β0.003 [β0.007, +0.002] |
| E1, E2, E3β² | + all 4 descriptors | β0.009 [β0.011, β0.007] | β0.039 [β0.046, β0.033] |
| E1, E2, E3β² | + all 4 descriptors + interactions | β0.010 [β0.012, β0.006] | β0.021 [β0.027, β0.015] |
| E2, E3β² | + crf_hat | β0.001 [β0.001, β0.000] | β0.003 [β0.005, +0.000] |
| E2, E3β² | + all 4 descriptors + interactions | β0.011 [β0.014, β0.008] | β0.012 [β0.017, β0.005] |
- The descriptor does not improve the ranking. It only moves the balance between false alarms and misses.
- Example (all 4 descriptors, E1, E2, E3β²):
- TNR at 20 hops goes from 0.335 to 0.742.
- But all-G TPR goes from 0.818 to 0.688.
- Conclusion: the current results do not support degradation-aware routing. The router and the condition-aware logistic (0.884) agree with this.
7.5 Repost-chain compression: layer probes and a real-reference score (added 2026-10-21)
Question: after deep compression, do real and fake videos still have different distributions in the feature space?
Data: the stored frozen features in wr3/ucf_feat (16 frames for each video, router and G). No G labels are fitted.
(a) Linear probe on each layer (router_train; C selected on router_val; Youden threshold on calib_select).
| Token | G AUC / Macro F1 | Hop 0 AUC | Hop 20 AUC | Hop 20 TNR |
|---|---|---|---|---|
| E3β² last layer (pooler, layer 24) | 0.868 / 0.763 | 0.932 | 0.681 | 0.245 |
| E3β² middle layers (15β17) | 0.84 / 0.73 | 0.93 | 0.61β0.63 | 0.64β0.72 |
| E2 X-CLIP layer 0, temporal token (std of frame differences) | 0.673 / 0.504 | 0.684 | 0.673 | 0.761 |
- Our hypothesis was that the middle layers keep more separation. This is not correct for E3β². The last layer is the best at all hops.
- The low-level temporal token of E2 does not change with compression (0.684 β 0.673), but it is weak.
- Main finding: at 20 hops, E3β² still ranks the videos (AUC 0.68), but its TNR is only 0.08. Most of the failure is a shift of all scores toward "fake". The separation is not fully lost.
(b) Distance to real videos at the same degradation level.
- The score is the Mahalanobis distance (Ledoit-Wolf, PCA to 64 dimensions) from the E3β² pooler token to the router real videos (router_train + calib_fit) in the same crf_hat quantile bin. It does not use fake videos.
- Platt on calib_fit. Paired bootstrap (1,000 resamples, grouped by source video) against the mean of z (E2, E3β²).
| Model (G) | AUC / Macro F1 | Hop 0 AUC / F1 | Hop 20 AUC / F1 | Hop 20 TNR | ΞAUC vs mean (E2, E3β²) [95% CI] |
|---|---|---|---|---|---|
| E1 WaveRep (released, DINOv2-B) | 0.749 / 0.563 | 0.926 / 0.826 | 0.490 / 0.334 | not calculated | not calculated |
| E3β² alone | 0.871 / 0.751 | 0.936 / 0.824 | 0.683 / 0.483 | 0.084 | β0.020 [β0.023, β0.017] |
| Real reference, 1 bin | 0.858 / 0.751 | 0.920 / 0.831 | 0.733 / 0.651 | 0.452 | β0.034 [β0.038, β0.029] |
| Real reference, 5 crf_hat bins | 0.860 / 0.755 | 0.925 / 0.834 | 0.743 / 0.653 | 0.645 | β0.032 [β0.037, β0.027] |
| Mean of z (E2, E3β²) | 0.892 / 0.783 | 0.941 / 0.839 | 0.752 / 0.584 | 0.213 | 0 |
| Mean of z (E2, real reference 5 bins) | 0.866 / 0.756 | 0.923 / 0.816 | 0.779 / 0.685 | 0.626 | β0.026 [β0.030, β0.021] |
| Mean of z (E2, E3β², real reference 5 bins) | 0.888 / 0.788 | 0.938 / 0.856 | 0.760 / 0.668 | 0.400 | β0.004 [β0.006, β0.001] |
- The real-reference score is better than E3β² at deep hops (hop 20: AUC +0.06, F1 +0.17). It is a little worse at hop 0.
- The crf_hat bins raise the TNR at 20 hops from 0.45 to 0.65, but they do not change the AUC. Thus the degradation descriptor helps the calibration, not the ranking (this agrees with Β§7.3).
- In the 3-expert mean, Macro F1 increases (0.783 β 0.788; hop 20: 0.584 β 0.668), but AUC decreases by 0.004. There is no overall AUC gain.
- Conclusion: after compression, real and fake videos are still separable. The detector features are not invariant to compression, so all scores move toward "fake". The next step is to train for this invariance (Β§10, item 1).
7.4 Older negative results
- The old E3 does not improve the ranking of (E1, E2): ΞAUC 0.000. It moves the threshold: F1 +0.021 [0.013, 0.030], TNR β0.16, TPR@1% β0.053. It calls G real videos fake at all hops (TNR 0.17β0.31).
- The router E1βE3 with seed 0 gives 0.816 AUC. The mean of 5 seeds is 0.798 Β± 0.015. A router that knows the true hop count gives 0.811.
- Label-free accuracy estimation (SML, GNB-EM) gives no gain. A self-probe by re-encoding fails, because the compression does not change after one re-encode.
- E3 v2 (fixed 125 ms step): 0.594 vs 0.606. Frame rate is not a shortcut in G (AUC 0.50).
- Training-free trajectory statistics (dense DINOv3, Wan2.1 and CogVideoX VAE latents): best G AUC 0.703. They give no fusion gain.
- Other frozen E3 candidates (linear probe, CogVideoX-only training data): SigLIP 2 0.733 / 0.612, DINOv3 RGB 0.677 / 0.591, V-JEPA 2 0.598 / 0.516. None of them improves the (WaveRep, D3) fusion.
8. WaveRep re-implementation (wr3): development of E3β²
- The official training code of WaveRep is not available. We re-implemented the recipe of the paper (Corvi et al., NeurIPS 2025, arXiv 2506.16802).
- Model and loss: ViT and a linear head, trained end to end, with BCE on random frame crops and balanced batches.
- WaveRep augmentation:
- p = 0.1 on fake crops; 3-level Haar fully separable wavelet transform.
- The baseband and the horizontal and vertical bands come from a real crop. The 9 diagonal bands stay from the fake.
- Other augmentation:
- JPEG and blur, p = 0.1 each.
- Resize of both classes to the same short-side range.
- Training settings: Adam, lr 1e-5 (cosine), weight decay 1e-6, batch 32, 20k iterations, crop 448. We select the checkpoint by the validation AUC.
- Re-encode augmentation (
--reenc): p = 0.3, 1β3 H.264 passes (CRF 23β35, scale 0.6β1.0) on a 16-frame window.
gen4 training fakes:
- Input: the first frame of a real expert_train source video and a Qwen2.5-VL-7B caption. Each fake uses a different source video.
- Generators:
- Wan2.2-TI2V-5B (704Γ1056, 40 steps).
- HunyuanVideo-1.5 480p I2V step-distilled (12 steps).
- LTX-2 19B I2V with the distilled LoRA (8 steps).
- SkyReels-V3 R2V-14B (540P, 8 steps).
- Encoding: H.264, CRF 18β28, 24 fps.
- 2 s set: 49 frames, 300 for each generator. 5 s set: 121 frames, 300 for each generator.
- Real videos: expert_train ActivityNet and InternVid (49 frames at 8 fps).
- Data of E3β² (
v3_5s300): train 4,498 real and 1,086 fake; validation 456 real and 114 fake.
| Run (G, Youden threshold on calib_select) | ACC | Macro F1 | AUC | TPR | TNR | TPR@1% | GenVideo AUC |
|---|---|---|---|---|---|---|---|
| WaveRep released (DINOv2-B) | 0.565 | 0.563 | 0.749 | 0.364 | 0.995 | 0.398 | 0.998 |
| DINOv2-B, 2 s | 0.747 | 0.687 | 0.811 | 0.868 | 0.486 | 0.284 | 0.715 |
| DINOv3-L, 2 s | 0.753 | 0.711 | 0.822 | 0.829 | 0.588 | 0.369 | 0.902 |
| DINOv3-L, 2 s + re-encode | 0.770 | 0.723 | 0.846 | 0.867 | 0.562 | 0.358 | 0.912 |
| DINOv2-B, 5 s + re-encode (seed 0 / 1) | 0.749 / 0.762 | 0.663 / 0.704 | 0.830 / 0.837 | 0.919 / 0.884 | 0.384 / 0.501 | 0.303 / 0.322 | 0.832 / 0.869 |
| DINOv2-L, 5 s + re-encode | 0.753 | 0.674 | 0.832 | 0.912 | 0.412 | 0.316 | 0.795 |
| DINOv3-L, 5 s + re-encode (seed 0 / 1) = E3β² | 0.796 / 0.793 | 0.756 / 0.748 | 0.871 / 0.865 | 0.881 / 0.892 | 0.613 / 0.582 | 0.432 / 0.402 | 0.948 / 0.944 |
- DINOv3 is better than DINOv2 also at the same model size (DINOv2-L 0.832, DINOv2-B 0.834). The gain comes from the pre-training, not from the size.
- 2 s vs 5 s (100 fakes per generator, 2 seeds each): G AUC 0.802 vs 0.825; Macro F1 0.706 vs 0.712. The source videos are different in the two sets.
- More fakes help more than longer clips (2 s: 100 per generator 0.802; 300 per generator 0.846).
8.1 Ablation of the WaveRep wavelet mix in E3β² (added 2026-10-25; G, seed 0 for each variant)
| E3β² variant (DINOv3-L, 5 s, re-encode) | Share of a mixed crop from the real crop | G AUC / Macro F1 | G 0 re-encodes | G 20 re-encodes (AUC) | GenVideo AUC / F1 |
|---|---|---|---|---|---|
| 3-level Haar, p = 0.1 (E3β²) | 23% | 0.871 / 0.756 | 0.936 / 0.831 | 0.683 | 0.948 / 0.818 |
| No wavelet mix (p = 0) | 0 | 0.829 / 0.677 | 0.912 / 0.735 | 0.606 | 0.920 / 0.741 |
| 1-level Haar | 75% | 0.865 / 0.708 | 0.933 / 0.767 | 0.645 | 0.945 / 0.770 |
| 2-level Haar | 44% | 0.860 / 0.711 | 0.928 / 0.762 | 0.673 | 0.958 / 0.806 |
| 3-level Haar, p = 0.3 | 23% | 0.845 / 0.666 | 0.926 / 0.724 | 0.658 | 0.953 / 0.735 |
| DINOv2-B retrain (3-level, p = 0.1) | 23% | 0.830 / 0.663 | 0.904 / 0.719 | 0.600 | 0.832 / 0.611 |
| WaveRep released (DINOv2-B) | β | 0.749 / 0.563 | 0.926 / 0.772 | 0.490 | 0.998 |
- The wavelet mix is necessary: β0.042 AUC and β0.079 Macro F1 without it. This is about 7 times the seed difference of E3β² (0.006).
- The setting of the paper (3 levels, p = 0.1) is the best on G. More mixing (p = 0.3) is worse.
- Without the mix, DINOv3-L (0.829) is at the level of the DINOv2-B retrain with the mix (0.830). The backbone and the mix both give a gain.
- Limit: one seed for each variant. The 1- and 2-level variants are within 2 seed differences of E3β².
- Script:
wr3_train.py --wr_p --wr_levels; datacsv/wr3_eval_G_wrablation.csv.
9. Known problems and limits
- D3 prediction bug in the benchmark CSVs:
tools/build_setting_{F,G}_per_video.pyswaps the predictions when they flip the score sign. Thuspred_D3is inverted on the rows with 5, 10 and 20 hops in F and G.- The correct rule: fake if
score_D3 β₯ β5. Corrected D3 on G: 0.686 / 0.646 / 0.695 / 0.750 / 0.550 / 0.013. - All MoE scripts use the correct rule. The fix of the project CSVs needs approval.
- Generator overlap:
- gen4 uses the Wan2.2, HunyuanVideo, LTX and SkyReels families. G also has these families. Thus G is not an unseen-generator test for E3β².
- Evidence on unseen generators: Relight-Video (E3β² weak, 0.54β0.58) and GenVideo.
- Shortcut in G: Relight-Video clips are much shorter than real clips (clip length alone gives AUC 0.97). G should use clips of the same length, or the paper must report this limit.
- WaveRep training generators (checked 2026-10-25): the released
G4checkpoint is trained on Pyramid Flow, CogVideoX-1.5, Allegro and OpenSora-Plan, with Panda70M real videos (Corvi et al., arXiv 2506.16802). Our GenVideo test set has "COG" (same family as CogVideoX) and "OpenSora" (HPC-AI, a different project from PKU OpenSora-Plan). The CogVideo family overlap partly explains the released WaveRep GenVideo AUC of 0.998. - Evidence limits: one test set; E3β² results from 2 seeds; no confidence intervals yet for router vs mean of z vs stacker with E3β².
10. Next steps (status 2026-10-26; target = mix_setting_v1)
| # | Experiment | Status |
|---|---|---|
| 1 | Compression-invariant E3β²: paired clean / re-encoded views, BCE on both views + consistency loss (wr3_train_pair.py, Ξ» = 1 and the Ξ» = 0 control) |
Done: negative (Β§0.3) |
| 2 | Leave-one-generator-out E3β² (without SkyReels-V3, LTX-2, HunyuanVideo-1.5 or Wan2.2) | Done: each generator gives a small part; no single one is necessary (Β§0.3) |
| 3 | Video extension: real extension clips from current generators (not SkyReels-V3), then the change-point transformer (Design 3) | Deferred (user, 2026-10-26); the feature-splice pilot is negative (Β§0.2). The 61 pending SkyReels-V3 extension slots of mix are also deferred |
| 4 | Relit training videos with the same length and fps as the real videos | Not run (user, 2026-10-26: stop here; MoE study ends with mean of z (E2, E3β²)) |
| 5 | Score E1 (released WaveRep) on mix_setting_v1 | Done: no gain (Β§0.3) |
| 6 | Augmentation study (VAE-reconstruction fakes, 12-profile chains) | Running in a separate session (claude_workspace/aug_study/STATUS.md) |
| 7 | Re-encode-aware gate / stacker on (E2, E3β²) with crf_hat on mix | Done: no gain (Β§0.3) |
| 8 | UCF-Net-style feature fusion (E2, E3β²) on mix | Done: equal to score fusion (Β§0.3) |
11. Update 2026-11-02: LevelNet and the data flywheel for the pixel expert
The full record is in AUG_STUDY_LOG.md Sections 19β23. Short summary (mix_setting_v1, n = 9,939, fused with E2; AUC / Macro F1):
| Study | Result | (E2, model) on mix |
|---|---|---|
| LevelNet (detail bands per SWT level; PREREG_LV) | Proxy rule not met; full run LVBN is descriptive | LVBN 0.829 / 0.727 |
| FW4: one flywheel round, 70% base | Round 1 passes (mixval deep +0.015 vs random selection) | M_1β² 0.833 / 0.729; C-randβ² 0.828 / 0.728 |
| FW5: multi-round, 30% base, new Kinetics + gen4 data on demand | Round 1 fails (error-directed = random); loop stops | M_0 0.808 / 0.709; C-all 0.829 / 0.729 |
| FW6: depth-directed loop (probe at 1β20 re-encodes, deep training units) | Preregistered round 1 fails narrowly; exploratory rounds 2β3: error-directed > uniform on mixval in every round | M_3 0.819 / 0.719 = C-rand_3 0.820 / 0.721 |
- The loop improves mixval strongly (deep AUC 0.750 β 0.815), but the gain does not transfer to mix_setting_v1.
- The best system on mix stays (E2, F3_s0) 0.849 / 0.739 (augmentation study,
AUG_STUDY_LOG.mdSection 13).
12. Update 2026-11-02 evening: FW7 and the model-size control
The full record is in AUG_STUDY_LOG.md Section 25. Short summary (fused with E2; AUC / Macro F1):
| Study | Result |
|---|---|
| FW6-X X4 (round 2, 2 seeds) | mixval: pass (+0.019 vs C-rand); mix: M_2 0.813 / 0.711 = C-rand_2 0.812 / 0.710 |
| FW7 pilots | The mix normalize step is not the cause (ΞAUC β€ 0.02); new high-quality Kinetics reals give many false positives (48% vs 27% for the probe reals) |
| FW7 round 4 (real-source-directed loop) | PASS on srcval: S_src M_4 0.890 vs C-rand_4 0.884 (+0.0065, CI low +0.0031) vs M_3 0.846. Round 5 is running; mix scores follow |
| F3B (F3 recipe on ViT-B/16, 2 seeds) | mix 0.833 / 0.728 and 0.828 / 0.718 vs F3_s0 (ViT-L/16) 0.849 / 0.739 and LVBN 0.829 / 0.727: the backbone size is the main factor; LevelConcat = full image |
- The best system on mix stays (E2, F3_s0) 0.849 / 0.739. A next loop must start from a ViT-L model.
13. Update 2026-11-03: FW7 and FW8 results on mix
Full record: AUG_STUDY_LOG.md Section 26 (fused with E2; AUC / Macro F1).
| Study | Validation decision | mix_setting_v1 |
|---|---|---|
| FW7 (ViT-B, real-source-directed, rounds 4β5) | Both rounds pass on srcval (S_src 0.846 β 0.908) | M_5 0.823 / 0.723 vs start 0.819 / 0.719 and C-rand_5 0.826 / 0.721 |
| FW8 (same loop on F3_s0, ViT-L) | Round 1 fails narrowly (+0.0047 vs C-rand, need +0.005) | M_1 0.850 / 0.735 vs F3_s0 0.849 / 0.739 |
- No flywheel variant (FW4βFW8) transfers its validation gain to mix_setting_v1. The best system stays (E2, F3_s0) 0.849 / 0.739.
14. Update 2026-11-04: error-directed fakes, FW7 round 6
Full record: AUG_STUDY_LOG.md Section 27 (fused with E2; AUC / Macro F1). Since 2026-11-03, new training uses only LevelConcat and level-attention. F3_s0 and LVBN are reference rows.
| Study | Validation decision | mix_setting_v1 |
|---|---|---|
| FW7 round 6 (LevelConcat ViT-B; new SkyReels-V3 extensions and Relight fakes from Kinetics) | Passes: S_hold 0.752 vs C-real 0.568 (CI low +0.158), no harm | M_6 0.837 / 0.737 vs M_5 0.823 / 0.723 and C-real_6 0.825 / 0.721 |
- This is the first flywheel round with a mix gain over its controls: +0.014 AUC [+0.011, +0.017] against C-real_6, for both seeds.
- The gain comes from Relight: the mix miss rate goes from 0.56 to 0.18. The SkyReels-V3 miss rate goes from 0.54 to 0.49.
- (E2, M6_s0) is still 0.012 AUC below (E2, F3_s0) 0.849 / 0.739, and the Macro F1 is equal. The DINOv2 rows are (E2, D2wr) 0.795 / 0.696 and (E2, E1) 0.759 / 0.583.
- Round 7 (200 more SkyReels-V3 extensions) is pre-registered and paused at 114 / 200 clips, because of a GPU driver mismatch on the server.
15. Update 2026-11-13: QL data flywheel (rounds 1β7, controls, external sets)
QL = LevelNet --head xattn --rgb 4 on one DINOv3 ViT-B/16 (query = LL3, keys = Haar SWT bands). Test set = mix_setting_v1 (RPG, n = 9,939). Protocol: logistic calibration on router calib_fit, Youden J on router calib_select, 1,000 resamples grouped by source video, labels Β±0.005 with the 95% CI. Main value = QL alone; (E2, QL) next to it. AUC / Macro F1. Records: aug_study/levelnet/PREREG_ARCH_FW.md (rounds 1β5), PREREG_AFW2.md (rounds 6+), PREREG_AFW2X.md (external sets), PREREG_AFW2S.md (same-start selection test).
Chains. Mk = data selected by model errors (rounds 1β5: LevelConcat errors; rounds 6+: QL's own errors on the FW6 depth probe, 600 re-encoded deep units of 4 s per round). C-rand (random fakes) = same budget, random / uniform selection (from round 3). C-real (rounds 3β5) = new reals only. C-steps = same training steps, no new data.
| mix_setting_v1, QL alone | AUC / Macro F1 | (E2, QL) AUC |
|---|---|---|
| Round 0 (AR2_QL_s0) | 0.757 / 0.650 | β |
| Mk round 5 | 0.815 / 0.713 | 0.849 |
| Mk round 6 (stop round K = 6) | 0.819 / 0.717 | 0.850 |
| Mk round 7 (confirmation, report only) | 0.821 / 0.717 | β |
| C-rand (random fakes) round 6 | 0.773 / 0.672 | 0.822 |
| C-real round 5 | 0.773 / 0.663 | β |
| C-steps round 6 | 0.744 / 0.640 | 0.794 |
| F3_s0 (ViT-L, reference) | 0.821 / 0.709 | β |
| dinov2_wr / released WaveRep (reference) | 0.737 / 0.638, 0.637 / 0.495 | β |
Pre-registered reading (PREREG_AFW2): NOT convincing.
- (a) Plateau: pass. Mk 5β6 +0.004 (one-third RPG +0.005 [0.001, 0.009]), 6β7 +0.003. Both Plateau.
- (b) Mk better than both controls at K = 6: pass. +0.045 vs C-rand (random fakes), +0.075 vs C-steps.
- (c) Mk 5β6 gain larger than C-rand 5β6 gain: fail. β0.002 [β0.005, 0.001], Equal.
- Post hoc (not pre-registered): (Mk 3β6) β (C-rand 3β6) = +0.034 [0.030, 0.039]. The gain over the control builds in rounds 3β5.
External sets (PREREG_AFW2X): the Mk advantage holds. QL alone, round 6.
| Set | Mk | C-rand (random fakes) | C-steps | Mk β C-rand | Mk β C-steps |
|---|---|---|---|---|---|
| AIGVDBench (2,000) | 0.894 / 0.817 | 0.802 / 0.752 | 0.746 / 0.701 | +0.091 Better | +0.147 Better |
| r5βr30 pooled (5,000) | 0.701 / 0.635 | 0.579 / 0.559 | 0.579 / 0.560 | +0.122 Better | +0.122 Better |
| GenVideo (report only) | 0.953 / 0.891 | 0.904 / 0.829 | 0.834 / 0.810 | +0.049 | +0.120 |
- r5βr30 use RPG source videos with new re-encodes; only AIGVDBench is fully outside RPG.
Same-start selection test (PREREG_AFW2S, 2 seeds). From Mk round 5: error-selected vs uniform round-6 units.
- Sel β Uni on mix: seed 0 +0.004, seed 1 β0.002. Both Equal; stable across seeds.
- All four runs (0.815β0.819) equal the start model Mk round 5 (0.815).
- External Sel β Uni changes sign between seeds (about Β±0.015): training-seed noise.
Conclusion. Error-selected data gives a large gain over the control chains, on RPG and outside RPG. The gain builds in rounds 1β5 and then reaches a plateau at round 6. At the plateau, error selection equals uniform selection. One seed for the chains (two for the same-start test).
Running (report only, user request 2026-11-13): C-rand (random fakes) and C-steps rounds 7β10, Mk rounds 8β10, on GPUs 0β2. Figure: aug_study/levelnet/arch_fw/ql_flywheel.png (one-third RPG, all rounds).
16. Update 2026-11-15: MoE table, fusion and frame-selection tests, temporal keys, second-detector flywheels, last-checkpoint rule
Test set = mix_setting_v1 (RPG, n = 9,939) unless the text says "prop33" (the proportional one-third subset, n = 3,277, csv/rpg_prop33.csv; progress and stop decisions only). Protocol as in Section 15. AUC / Macro F1; Β± and [ ] = 95% CI from 1,000 resamples grouped by source video. Each detector is reported alone unless the row says "with D3".
16.1 Main MoE result with QL round 7 (misc/moe3/moe3_full.md)
QL = AFW_QL_Mk_r7 (Section 15). Pairs tested on prop33: (QL, D3) 0.851 / 0.749 is better than (QL, STALL) and (QL, SPLIT) (both 0.821, +0.030). STALL and SPLIT add nothing to QL.
| Full RPG | AUC | Macro F1 | TPR@1%FPR |
|---|---|---|---|
| Spatial-frequency expert (QL r7) | 0.821 Β± 0.008 | 0.717 Β± 0.009 | 0.293 Β± 0.035 |
| Temporal expert (D3) | 0.693 Β± 0.011 | 0.554 Β± 0.010 | 0.013 Β± 0.005 |
| MoE (QL, D3) | 0.852 Β± 0.007 | 0.742 Β± 0.009 | 0.290 Β± 0.034 |
- GenVideo test split: (QL, D3) 0.986 / 0.918, TPR@1%FPR 0.880. AIGVDBench closed-source sample (2,000 reals + 2,000 fakes of 11 generators,
csv/aigvdb_cs_manifest.csv): 0.965 / 0.818, TPR@1%FPR 0.766. - Baselines run a second time on full RPG (
misc/rpg2/rpg2_table.tex): 9 of 11 give the same scores (r β₯ 0.9999). Only SPLIT (0.568 β 0.577) and STALL (0.517 β 0.516) change (seeded sampling).
16.2 Round-0 ablation of QL (full RPG, seed 0; csv/arch_r0_tpr1.csv, PREREG_QLD0.md)
| Variant | Alone: AUC / Macro F1 / TPR@1%FPR | With D3: AUC / Macro F1 / TPR@1%FPR |
|---|---|---|
| QL_s0 (full model) | 0.757 / 0.650 / 0.192 | 0.803 / 0.710 / 0.191 |
| w/o wavelet bands (R1_s0) | 0.770 / 0.661 / 0.234 | 0.812 / 0.710 / 0.238 |
| w/o content queries (X4_s0) | 0.769 / 0.665 / 0.223 | 0.807 / 0.705 / 0.237 |
| w/o fine-level dropout (AR3_QLD0_s0) | running | running |
- At round 0, both finished ablations are higher than the full model. The round-0 table thus does not support these two components. The main-table model is QL after the flywheel rounds, not round 0.
- AR3_QLD0_s0 = AR2_QL_s0 with
--p_drop 0(pre-registered reading: "fine-level dropout helps" only if AUC with D3 is Worse). On the in-distribution validation set, AUC at 5 re-encodes is 0.005β0.028 lower than QL at all 16 points to iteration 16,000; AUC at 0 re-encodes is equal. The RPG result is not available yet.
16.3 Fusion tests: the mean of calibrated log-odds stays (PREREG_FUS.md, PREREG_RES.md; prop33, QL = AFW_QL_Mk_r6)
Reference: QL 0.819 / 0.721, D3 0.701 / 0.557, mean 0.849 / 0.754.
| Idea | Result | Decision |
|---|---|---|
| Detail-survival gate (weight by an estimate of re-encodes) | β0.004 Equal; even the best weight per re-encode count, fitted on the subset, gives β0.0001 | STOP |
| Second-order dynamics of QL frame logits | +0.000 over QL; β0.030 Worse as a D3 replacement | STOP |
| Residual expert (train QL on BCE(logit + calibrated D3 log-odds)) | 0.837 / 0.740 = β0.012 Worse | STOP |
| UCF-Net uncertainty weight at the score level | 0.842 / 0.757 = β0.007 Worse; the learned version returns to the plain mean | STOP |
- Useful for the paper: mean β QL alone = +0.008 / +0.051 / +0.056 / +0.046 at 1 / 5 / 10 / 20 re-encodes. D3 helps most after re-encodes.
16.4 D3 as a frame sampler (PREREG_D3SEL.md, PREREG_D3Q.md): STOP
- LevelConcat M8, exact decode (
src/exact_decode.py; decord gives wrong frames on 4β7% of RPG videos): Uniform-32 = Uniform-16 (0.815 vs 0.813). Uniform-32 β random window-32 = +0.015 Better (coverage matters, not frame count). D3-frame-32 β random-frame-32 = +0.001 Equal. - QL r6 with equal coverage (16 time strata; D3 frame vs random frame per stratum): +0.001 Equal; D3-weighted mean β0.003 Equal (SkyReels-V3 Worse).
16.5 Temporal keys (QLT; PREREG_QLT.md, PREREG_QLT_FB.md): negative
- QLT adds temporal-difference keys to QL. Stage 1, prop33: QLT 0.640 / 0.572 vs QL 0.751 / 0.647 alone; 0.737 vs 0.800 with D3. All hypotheses Worse.
- Stage 2, own-error feedback from a held-out pool of old flywheel videos (26,306 units; top 1,500 loss of a 6,000 sample per round), 5 rounds: Mk 0.640 β 0.613 (Decrease); C-rand (random fakes) 0.640 β 0.636; Mk_5 β C-rand_5 = β0.023 Worse.
16.6 Flywheel on second detectors (5-round layout; full RPG, alone)
| Detector (record) | Mk | C-rand (random fakes) | C-steps | Reading |
|---|---|---|---|---|
FreqNet own loop, r7 (PREREG_FL.md) |
0.596 / 0.529 | 0.603 / 0.515 | 0.575 / 0.549 | Fails (Worse than C-rand β0.007) |
VideoFACT own loop, r4 (PREREG_VFL.md) |
0.622 / 0.569 | 0.628 / 0.578 | β | Worse than C-rand (β0.006) |
VideoFACT round 9 (PREREG_VF9.md) |
0.690 / 0.596 | 0.672 / 0.582 | 0.629 / 0.573 | Gaps Better, but Mk 8β9 Plateau: fails |
GenD r8, lr 1e-4 (PREREG_GD.md) |
0.591 / 0.510 | 0.594 / 0.499 | 0.585 / 0.508 | Mk = C-rand; all chains still rise |
UnivFD fc-only r5 (PREREG_UFL.md) |
0.679 / 0.602 | 0.690 / 0.620 | 0.666 / 0.599 | Worse than C-rand (β0.011) |
VideoFACT feedback from the QLT pool, K = 3 (PREREG_VFFB.md) |
0.662 / 0.588 | 0.650 / 0.587 | 0.604 / 0.563 | Helps: +0.012 vs C-rand, +0.058 vs C-steps (both Better) |
- VFFB is the first own-error loop on a second detector that passes. But selection adds only +0.012; coverage of the pool adds +0.055 (C-rand β C-steps). Round 0 = FW7_VFL1_s0 0.595 / 0.530.
16.7 VideoFACT feedback extension (PREREG_VFFB2.md; running)
Rounds 6β12 from the round-5 models; stop after 2 extension rounds in a row without Improve on prop33. On 2026-11-15 the user replaced C-rand with C-match (Mk's type Γ label counts per round, random units inside each cell). C-rand stopped at round 7 (report only).
| prop33, alone | Round 5 | Round 7 | Round 9 | Round 10 | Round 11 |
|---|---|---|---|---|---|
| Mk (own errors) | 0.672 / 0.585 | 0.690 / 0.609 Improve | 0.702 / 0.586 Improve (K2 = 9) | 0.698 / 0.611 Plateau | 0.692 / 0.573 Inconclusive (β0.006) |
| C-match (Mk's mix) | 0.679 / 0.597 | running | |||
| C-steps (base set only) | 0.589 / 0.544 | 0.594 / 0.548 | 0.617 / 0.569 | 0.603 / 0.539 | running |
- Mk stopped at round 11 (N2 = 11, K2 = 9). The controls run to round 11; then full-RPG scoring and the pre-registered reading.
- C-match equals or beats Mk at rounds 1β5 (round 4: 0.683 vs 0.675). This suggests that most of the gain comes from the type mix of the pool, not from the selection of single units. The reading at K2 will test this.
- Figure:
image/vffb_chains_prop33_live.png.
16.8 Last-checkpoint rule for chain rounds (user decision 2026-11-15)
- Problem: VideoFACT chains move up and down from round to round (C-steps spread Β±0.012 prop33 AUC; QL Β±0.003). The best-validation checkpoint was picked on an in-distribution validation set (AUC about 0.90) that does not follow RPG. Mk kept non-final weights in 5 of 9 rounds (iterations 500, 1,000, 2,000, 2,000, 3,000), near the peak of the restarted learning-rate schedule. QL validated only at 2,500 and 5,000 and kept the final weights in 28 of 33 runs.
- Rule: from round 10, all chains keep the final weights (
train_vfact_fw6.py --keep last; the validation-best weights are kept asbest_val.pt). Rounds 1β9 keep the old rule in all chains. - First data: Mk had no Decrease label in rounds 10β11 (Decrease at rounds 2 and 6 before). Two rounds are not sufficient to judge the change.
- Mk_9 has a higher AUC but a lower Macro F1 than Mk_8. The Youden threshold from router calib_select is low on RPG: Mk_9 calls 79% of videos fake (TNR 0.322). At the best test threshold (report only), Macro F1 is 0.627 vs 0.613.
16.9 Other results
- QL chains, rounds 7β9 (report only): Mk 0.822 (r7), 0.821 (r8) prop33, Plateau. C-steps 0.759 (r7), 0.749 (r8). The chains are paused before round 10 (user decision).
- GenVideo re-encode test (200 GenVideo reals vs 800 fakes, 0/1/5/10/20 re-encodes): ours 0.962 β 0.799, SPLIT 0.988 β 0.979, D3 0.812 β 0.778, WaveRep 0.824 β 0.369. High AUCs at 0 re-encodes come partly from a format shortcut (reals 3 fps, 298Γ224).
- Trace by level (
aug_study/fig2_trace_by_level/report.md): the absolute trace in b1βb3 rises with re-encodes after one re-encode (unsharp masks in the TikTok / RedNote profiles amplify high frequencies). The claim "the finest level loses its trace fastest" is not supported.
16.10 Open items
- AR3_QLD0_s0 RPG row (w/o fine-level dropout).
- PREREG_VFFB2: C-match rounds 6β11 and C-steps round 11, then full RPG and the reading at K2 = 9 and N2 = 11.
- QL chains round 10: waits for user approval.