Spaces:
Running on Zero
Running on Zero
|
Download docs/PHASE5_FUSION_FEASIBILITY_REPORT.md from Thiru1122/Multimodel_setup: direct link, hf CLI and curl.
- Browser
- Download file 24 kB
-
https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE5_FUSION_FEASIBILITY_REPORT.md
- Command line
-
hf download hf://spaces/Thiru1122/Multimodel_setup/docs/PHASE5_FUSION_FEASIBILITY_REPORT.md
-
curl -L -o PHASE5_FUSION_FEASIBILITY_REPORT.md https://huggingface.co/spaces/Thiru1122/Multimodel_setup/resolve/main/docs/PHASE5_FUSION_FEASIBILITY_REPORT.md
24 kB
| # Phase 5: Decision-Fusion Feasibility and Controlled Evaluation Report | |
| ## Empirical Investigation of Multi-Model Ensemble Strategies on Independent Development Validation Split | |
| **Document Date:** 2026-10-30 | |
| **Evaluation Partition:** `development_val` (Master Manifest: `benchmark/evaluation_manifest.csv`) | |
| **Sample Composition:** 50 Facial Samples (25 Authentic FFHQ Portraits vs. 25 StyleGAN2 Synthetic Faces) | |
| **Execution Environment:** Windows (CPU Inference), Python 3.13.13, PyTorch 2.13.0+cpu | |
| **Evaluation Script:** `scripts/evaluate_phase5_fusion.py` | |
| **Fusion Implementation:** `src/decision_fusion.py` | |
| **Test Suite:** `tests/test_decision_fusion.py` (21/21 passed) | |
| **Metrics Deliverable:** `benchmark/results/phase5_validation_metrics.json` | |
| **Raw Predictions:** `benchmark/results/phase5_val_raw_predictions.csv` | |
| --- | |
| ## 1. Executive Summary | |
| The primary objective of Phase 5 is to empirically determine whether combining three independently trained and integrated deepfake detectors—**ConvNeXt-Base/DRCT** (`chen2024_convnext`), **UniversalFakeDetect** (`univfd`), and **Vision Transformer** (`vit`)—improves detection reliability over the strongest individual model (`vit`). In accordance with rigorous scientific methodology, decision fusion was treated as an empirical hypothesis to be tested rather than a predetermined improvement. | |
| ### Core Empirical Findings: | |
| 1. **Unweighted Fusion Degrades Facial Detection Reliability**: | |
| - **ViT Alone** achieved **100.00% accuracy** (50/50, F1: 1.0000) on `development_val`. | |
| - **Majority Voting** degraded accuracy to **74.00%** (37/50, F1: 0.6486), introducing **13 false negatives** (fake recall plummeted from 100.0% to 48.00%). | |
| - **Mean Fake Probability** degraded accuracy to **76.00%** (38/50, F1: 0.6842), introducing **12 false negatives** (fake recall dropped to 52.00%). | |
| 2. **Root Cause of Fusion Degradation**: | |
| - On facial portraits, ConvNeXt and UnivFD exhibit severe false negative rates (76.00% and 68.00% false negative rates, respectively). | |
| - In hard majority voting, ConvNeXt and UnivFD regularly agree on `REAL` for synthetic faces, creating a 2-to-1 majority that overrules ViT's correct `FAKE` prediction. | |
| - In linear probability averaging, ConvNeXt's mean fake probability on synthetic faces is only $29.29\%$ and UnivFD's is $34.84\%$. These low probabilities drag ViT's high confidence ($\approx 99.87\%$) below the standard $0.50$ decision threshold. | |
| 3. **ViT-Priority Strategy Selected and Frozen**: | |
| - The **ViT-Priority** strategy achieves **100.00% accuracy** on `development_val` (50/50, balanced accuracy: 100.00%, F1: 1.0000). | |
| - It maintains ViT's ceiling performance on facial media while providing explicit, deterministic fallback to secondary detectors (UnivFD, ConvNeXt) in the event of ViT runtime failures or corrupted inferences. | |
| 4. **Boundary Integrity & Protocol Freeze**: | |
| - All strategy comparisons and design decisions were conducted strictly on `development_val`. | |
| - The `held_out_test` split (105 samples) was strictly quarantined during this phase. | |
| - Per protocol instructions, the reproducible command for evaluating the frozen strategy against `held_out_test` has been prepared, and execution is paused awaiting user review and approval. | |
| --- | |
| ## 2. Experimental Rules & Dataset Boundary Protections | |
| ### 2.1 Partition Integrity | |
| All experimental decisions were governed by the data integrity boundaries established in Phase 3 (`docs/DATASET_AUDIT_AND_EVALUATION_PLAN.md`): | |
| | Partition Name | Samples | Role in Phase 5 | Integrity Status | | |
| | :--- | :---: | :--- | :--- | | |
| | **`development_val`** | **50** | **Sole active dataset** for baseline comparison, probability analysis, and fusion selection. | Completely isolated from Phase 4; evaluated for the first time in Phase 5. | | |
| | **`held_out_test`** | 105 | **Strictly quarantined**. No tuning, threshold optimization, or iterative decisions. | Unmodified. Frozen command prepared for single final evaluation. | | |
| | **`operational_smoke`** | 4 | Excluded from evaluation metrics; reserved for software contract checks. | Preserved. | | |
| ### 2.2 Dataset Domain Composition Notice | |
| The `development_val` partition comprises strictly: | |
| - **25 Authentic Human Portraits**: Photographic captures from Flickr-Faces-HQ (FFHQ), tightly cropped and aligned to $256 \times 256$ pixels (`real_1550.jpg` through `real_1574.jpg`). | |
| - **25 Synthetic Faces**: StyleGAN2 generative adversarial network synthesis, tightly cropped and aligned to $256 \times 256$ pixels (`fake_1550.jpg` through `fake_1574.jpg`). | |
| > [!CRITICAL] | |
| > **Domain Limitation**: Because `development_val` contains **only facial crops**, performance on this split **does NOT validate general-scene fusion**, latent diffusion detection (e.g. Imagen 3, Stable Diffusion), or localized digital splicing. Claims of general-domain effectiveness cannot be made from this partition. | |
| --- | |
| ## 3. Probability Semantics, Comparability, and Calibration Suitability | |
| ### 3.1 Detector Output Schema & Probability Semantics | |
| All detectors conform to `DetectorResult` (`src/detector_schema.py`): | |
| - `fake_probability` represents the model's estimated posterior $P(Y = \text{FAKE} \mid X) \in [0.0, 1.0]$. | |
| - Standard binary decision threshold is fixed at $\tau = 0.50$: | |
| $$\hat{Y} = \begin{cases} \text{FAKE} & \text{if } P(\text{fake}) \ge 0.50 \\ \text{REAL} & \text{if } P(\text{fake}) < 0.50 \end{cases}$$ | |
| ### 3.2 Measured Probability Distributions on `development_val` ($N=50$) | |
| | Detector Key | Detector Name | Metric | Real Faces ($N=25$) | Fake Faces ($N=25$) | Separation Margin $\Delta$ | | |
| | :--- | :--- | :--- | :---: | :---: | :---: | | |
| | `chen2024_convnext` | Chen2024ConvNeXtDetector | Mean $P(\text{fake})$<br>Std Dev<br>Median<br>[Min, Max] | 9.83%<br>$\pm 25.59\%$<br>0.33%<br>[0.00%, 99.78%] | 29.29%<br>$\pm 38.20\%$<br>7.57%<br>[0.00%, 99.84%] | **+19.46%** | | |
| | `univfd` | UniversalFakeDetect | Mean $P(\text{fake})$<br>Std Dev<br>Median<br>[Min, Max] | 11.73%<br>$\pm 18.44\%$<br>0.92%<br>[0.00%, 70.01%] | 34.84%<br>$\pm 33.94\%$<br>20.64%<br>[0.00%, 99.56%] | **+23.11%** | | |
| | `vit` | ViTDeepfakeDetector | Mean $P(\text{fake})$<br>Std Dev<br>Median<br>[Min, Max] | 0.26%<br>$\pm 0.53\%$<br>0.10%<br>[0.05%, 2.72%] | 99.87%<br>$\pm 0.12\%$<br>99.90%<br>[99.43%, 99.96%] | **+99.61%** | | |
| ### 3.3 Probability Incomparability Analysis | |
| The empirical distributions reveal that raw detector outputs are **not on a comparable scale**: | |
| 1. **ViT (dima806)** exhibits almost complete bipolar separation ($\Delta = +99.61\%$). Its probabilities are sharp: real portraits never exceed $2.72\%$ fake probability, and fake portraits never drop below $99.43\%$. | |
| 2. **UnivFD (CVPR 2023)** produces heavily compressed probabilities on facial crops. Its median fake probability on fake faces is only $20.64\%$, and its mean is $34.84\%$. Because it was trained on ProGAN objects/scenes, StyleGAN facial artifacts do not strongly activate its linear probe. | |
| 3. **ConvNeXt (Chen et al. 2024 / RAID)** has a median fake probability on fake faces of only $7.57\%$, with an overall mean of $29.29\%$. Trained on general diffusion scenes (RAID benchmark), its frequency-domain features fail to register GAN blending artifacts in aligned face crops. | |
| ### 3.4 Evaluation of Probability Calibration Suitability | |
| Consideration was given to post-hoc probability calibration (e.g., Platt scaling / logistic regression or isotonic regression). However: | |
| - **Sample Size Constraint**: The validation partition contains only 50 samples (25 real, 25 fake). Fitting parametric scaling parameters (temperature $T$, bias $b$) or non-parametric monotonic steps on 50 samples is severely under-powered, prone to extreme variance, and risks severe overfitting. | |
| - **Fundamental Domain Shift**: Calibration cannot restore information that the feature extractor failed to capture. When ConvNeXt assigns $P(\text{fake}) = 0.004\%$ to a synthetic face, monotonic calibration cannot elevate that specific sample without simultaneously inflating false positives on authentic faces. | |
| - **Methodological Recommendation**: As instructed, raw model probabilities were not assumed calibrated, and no complex meta-classifier was fitted on this small split. | |
| --- | |
| ## 4. Individual Detector Baselines on `development_val` ($N=50$) | |
| Before evaluating multi-model fusion, individual baseline metrics were established across all 50 validation samples: | |
| ### 4.1 Baseline Performance Table | |
| | Detector Key | Detector Name | TP | FP | TN | FN | Accuracy (95% CI) | Balanced Acc | Precision (95% CI) | Recall (95% CI) | Specificity (95% CI) | F1 Score | | |
| | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | `chen2024_convnext` | Chen2024ConvNeXtDetector | 6 | 2 | 23 | 19 | 58.00%<br>[44.2%, 70.6%] | 58.00% | 75.00%<br>[40.9%, 92.9%] | 24.00%<br>[11.5%, 43.4%] | 92.00%<br>[75.0%, 97.8%] | 0.3636 | | |
| | `univfd` | UniversalFakeDetect | 8 | 1 | 24 | 17 | 64.00%<br>[50.1%, 75.9%] | 64.00% | 88.89%<br>[56.5%, 98.0%] | 32.00%<br>[17.2%, 51.6%] | 96.00%<br>[80.5%, 99.3%] | 0.4706 | | |
| | `vit` | ViTDeepfakeDetector | 25 | 0 | 25 | 0 | **100.00%**<br>[92.9%, 100.0%] | **100.00%** | **100.00%**<br>[86.7%, 100.0%] | **100.00%**<br>[86.7%, 100.0%] | **100.00%**<br>[86.7%, 100.0%] | **1.0000** | | |
| ### 4.2 Key Observations: | |
| - **ViT** achieves perfect discrimination on this split (25/25 real portraits correctly rejected, 25/25 synthetic faces correctly detected). | |
| - **ConvNeXt** misses 19 out of 25 synthetic faces (76.0% false negative rate). | |
| - **UnivFD** misses 17 out of 25 synthetic faces (68.0% false negative rate). | |
| --- | |
| ## 5. Decision Fusion Strategy Evaluation & Comparison | |
| Four canonical strategies implemented in `src/decision_fusion.py` were evaluated on the 50 validation samples: | |
| 1. **`vit_only`**: ViT baseline. | |
| 2. **`majority_vote`**: Unweighted 2-out-of-3 majority vote across all three detectors. | |
| 3. **`mean_fake_probability`**: Arithmetic mean of valid fake probabilities compared to threshold $\tau = 0.50$. | |
| 4. **`vit_priority`**: Prioritizes ViT, falling back to secondary detectors only if ViT encounters a runtime or prediction failure. | |
| ### 5.1 Fusion Strategy Performance Table ($N=50$) | |
| | Strategy Code | Strategy Name | TP | FP | TN | FN | Accuracy (95% CI) | Balanced Acc | Precision | Recall | Specificity | F1 Score | Status | | |
| | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | `vit_only` | ViT Alone (Baseline) | 25 | 0 | 25 | 0 | **100.00%**<br>[92.9%, 100.0%] | **100.00%** | 100.00% | 100.00% | 100.00% | **1.0000** | Baseline | | |
| | `majority_vote` | Hard Majority Vote | 12 | 0 | 25 | 13 | **74.00%**<br>[60.5%, 84.1%] | **74.00%** | 100.00% | 48.00% | 100.00% | **0.6486** | **DEGRADED** | | |
| | `mean_fake_probability` | Mean Probability ($\tau=0.5$) | 13 | 0 | 25 | 12 | **76.00%**<br>[62.6%, 85.7%] | **76.00%** | 100.00% | 52.00% | 100.00% | **0.6842** | **DEGRADED** | | |
| | `vit_priority` | ViT-Priority with Fallback | 25 | 0 | 25 | 0 | **100.00%**<br>[92.9%, 100.0%] | **100.00%** | 100.00% | 100.00% | 100.00% | **1.0000** | **PARITY / ROBUST** | | |
| --- | |
| ## 6. Comparative Transition Analysis Relative to ViT Alone | |
| To rigorously audit the effect of fusion, every sample transition from ViT alone to each fusion strategy was tracked: | |
| ### 6.1 Relative Metric Deltas | |
| | Strategy | $\Delta$ Accuracy | $\Delta$ Balanced Acc | $\Delta$ F1 Score | Improved Samples | Regressed Samples | Both Correct | Both Incorrect | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | `majority_vote` | **-26.00%** | **-26.00%** | **-0.3514** | 0 | **13** | 37 | 0 | | |
| | `mean_fake_probability` | **-24.00%** | **-24.00%** | **-0.3158** | 0 | **12** | 38 | 0 | | |
| | `vit_priority` | **+0.00%** | **+0.00%** | **+0.0000** | 0 | **0** | 50 | 0 | | |
| ### 6.2 Detailed Sample Regression Catalog | |
| #### 1. Regressed Under `majority_vote` (13 Samples): | |
| The following 13 synthetic faces were correctly classified as `FAKE` by ViT, but were **misclassified as `REAL`** by majority voting because ConvNeXt and UnivFD both falsely predicted `REAL` (creating a 2 REAL vs 1 FAKE vote): | |
| - `val_face_fake_001` | |
| - `val_face_fake_002` | |
| - `val_face_fake_005` | |
| - `val_face_fake_008` | |
| - `val_face_fake_009` | |
| - `val_face_fake_012` | |
| - `val_face_fake_013` | |
| - `val_face_fake_015` | |
| - `val_face_fake_016` | |
| - `val_face_fake_017` | |
| - `val_face_fake_023` | |
| - `val_face_fake_024` | |
| - `val_face_fake_025` | |
| #### 2. Regressed Under `mean_fake_probability` (12 Samples): | |
| The following 12 synthetic faces were correctly classified as `FAKE` by ViT ($P_\text{ViT} \approx 99.8\%$), but had their mean fake probability dragged below $0.50$ by ConvNeXt and UnivFD: | |
| - `val_face_fake_001` (Mean $P(\text{fake}) = 35.40\%$) | |
| - `val_face_fake_002` (Mean $P(\text{fake}) = 35.99\%$) | |
| - `val_face_fake_008` (Mean $P(\text{fake}) = 42.72\%$) | |
| - `val_face_fake_009` (Mean $P(\text{fake}) = 40.77\%$) | |
| - `val_face_fake_012` (Mean $P(\text{fake}) = 43.96\%$) | |
| - `val_face_fake_013` (Mean $P(\text{fake}) = 34.48\%$) | |
| - `val_face_fake_015` (Mean $P(\text{fake}) = 49.10\%$) | |
| - `val_face_fake_016` (Mean $P(\text{fake}) = 47.80\%$) | |
| - `val_face_fake_017` (Mean $P(\text{fake}) = 47.32\%$) | |
| - `val_face_fake_023` (Mean $P(\text{fake}) = 34.79\%$) | |
| - `val_face_fake_024` (Mean $P(\text{fake}) = 34.23\%$) | |
| - `val_face_fake_025` (Mean $P(\text{fake}) = 33.42\%$) | |
| *(Note: `val_face_fake_005` barely survived mean probability with $P=50.2\%$, but regressed under majority voting).* | |
| ### 6.3 Scientific Diagnosis: Why Unweighted Fusion Fails | |
| An ensemble improves performance only when constituent models make **independent, uncorrelated errors** and have individual accuracies well above chance. On facial deepfakes: | |
| 1. **Correlated Errors**: ConvNeXt and UnivFD share the same blindspot—they miss StyleGAN2 synthetic faces at rates exceeding $68\%$. Their errors are positively correlated. | |
| 2. **Asymmetric Capability**: ViT is virtually saturated on this domain ($100.0\%$), while the other two models perform near chance ($58\%$ and $64\%$). | |
| 3. **Voting Paradox**: Diluting a $100\%$ accurate specialist with two $60\%$ generalists guarantees that the majority vote will frequently overrule the correct specialist. | |
| --- | |
| ## 7. Software Contract & Unit Test Verification | |
| To ensure that `src/decision_fusion.py` behaves deterministically under all operational conditions, a comprehensive test suite was implemented in `tests/test_decision_fusion.py`. All 21 tests execute and pass: | |
| ### 7.1 Test Matrix & Verification Coverage | |
| | Test Case Name | Behavioral Condition Tested | Expected Outcome | Verification Status | | |
| | :--- | :--- | :--- | :---: | | |
| | `test_all_models_agree_real` | All 3 models predict REAL ($P < 0.10$) | All strategies predict REAL; `status="SUCCESS"` | **PASS** | | |
| | `test_all_models_agree_fake` | All 3 models predict FAKE ($P > 0.85$) | All strategies predict FAKE; `status="SUCCESS"` | **PASS** | | |
| | `test_models_disagree_majority_vs_vit` | ViT predicts FAKE; ConvNeXt and UnivFD predict REAL | Majority: REAL; ViT-Only: FAKE; Priority: FAKE | **PASS** | | |
| | `test_one_model_fails_convnext` | ConvNeXt raises CUDA OOM / ERROR; ViT & UnivFD valid | Surviving models used; ViT Priority unaffected | **PASS** | | |
| | `test_one_model_fails_vit_triggers_fallback` | ViT fails with memory error; ConvNeXt & UnivFD valid | ViT Priority triggers fallback (`status="FALLBACK"`) | **PASS** | | |
| | `test_multiple_models_fail_single_survivor` | ConvNeXt & UnivFD fail; ViT survives | Single surviving model used deterministically | **PASS** | | |
| | `test_multiple_models_fail_including_vit` | ViT & UnivFD fail; ConvNeXt survives | Fallback selects ConvNeXt; `fallback_applied=True` | **PASS** | | |
| | `test_invalid_or_missing_probabilities` | `fake_probability` is `None` or `NaN` | Majority vote unaffected; Mean prob filters invalid | **PASS** | | |
| | `test_all_missing_probabilities` | All detectors lack numeric probabilities | Mean prob returns clean `PREDICTION_ERROR` | **PASS** | | |
| | `test_deterministic_tie_handling_majority` | Even 1-vs-1 tie with 1 failure | Deterministic resolution via specified policy | **PASS** | | |
| | `test_tie_handling_with_vit_preference` | 1-vs-1 tie including ViT | Deterministically resolves in favor of ViT vote | **PASS** | | |
| | `test_mean_probability_exact_threshold_tie` | Mean probability equals exactly $0.500000$ | Deterministic tie breaking (`real` vs `fake`) | **PASS** | | |
| | `test_no_valid_predictions_remain` | All 3 models report `PREDICTION_ERROR` | All strategies return `PREDICTION_ERROR` without crash | **PASS** | | |
| | `test_rejection_controls` | Malformed image triggers adapter rejection | Returns `PREDICTION_ERROR`; excluded from TP/TN | **PASS** | | |
| | `test_to_detector_result` | Conversion of `FusionResult` to `DetectorResult` | Full schema compliance and field preservation | **PASS** | | |
| | `test_boolean_probabilities_rejected` | `fake_probability` is boolean `True` or `False` | Sanitized to `None`; booleans rejected as probabilities | **PASS** | | |
| | `test_all_models_uncertain` | All detectors report `PREDICTION_UNCERTAIN` | Fused prediction is `UNCERTAIN` (not `ERROR`) | **PASS** | | |
| | `test_vit_uncertain_fallback_to_decisive` | ViT is `UNCERTAIN`; secondary models are decisive | Triggers fallback to secondary; `status="FALLBACK"` | **PASS** | | |
| | `test_mean_probability_tie_breaker_uncertain` | Mean probability equals threshold; tie policy uncertain | Returns `PREDICTION_UNCERTAIN` with `status="UNCERTAIN"` | **PASS** | | |
| | `test_majority_vote_with_abstained_and_error` | 1 model ERROR, 2 models UNCERTAIN | Fused verdict is `UNCERTAIN`; failed count is strictly 1 | **PASS** | | |
| | `test_invalid_strategy_raises_value_error` | Pipeline requested with unknown strategy name | Raises `ValueError` with valid strategy names | **PASS** | | |
| --- | |
| ## 8. Frozen Strategy Selection & Held-Out Evaluation Protocol | |
| ### 8.1 Strategy Selection | |
| Based on the empirical evidence on `development_val`: | |
| - **Selected Strategy**: **`vit_priority`** (ViT-Priority with Secondary Fallback). | |
| - **Selection Rationale**: | |
| 1. It preserves ViT's **100.00% accuracy** on facial deepfakes, preventing the catastrophic 24–26% regression introduced by unweighted averaging or voting. | |
| 2. It introduces critical **fault tolerance**: if ViT fails due to memory exhaustion, paging file constraints (Windows Error 1455), or model corruption, the pipeline seamlessly falls back to the secondary models (`univfd` and `chen2024_convnext`) rather than failing completely. | |
| ### 8.2 Reproducible Command for Held-Out Evaluation | |
| The strategy selection has been frozen. The evaluation harness supports running this frozen strategy against the pristine `held_out_test` partition via: | |
| ```powershell | |
| # Step 1: Protocol Inspection / Safety Gate check (pauses per Phase 5 stopping rule): | |
| python scripts/evaluate_phase5_fusion.py --eval-test --strategy vit_priority | |
| # Step 2: Protocol Execution (Runs only upon explicit confirmation / approval): | |
| python scripts/evaluate_phase5_fusion.py --eval-test --strategy vit_priority --confirm | |
| ``` | |
| ### 8.3 Scientific Disclosures & Limitations: | |
| 1. **Prior Observation of Test Set**: The `held_out_test` split (105 samples) was evaluated during Phase 4 (`docs/PHASE4_BASELINE_EVALUATION_REPORT.md`). Consequently, results on that partition are no longer a strictly blind test for this repository. | |
| 2. **Domain Specificity**: The validation split tested here contains solely aligned facial images. In Phase 4, ViT failed on general scenes synthesized by Google Imagen 3 (`test_scene_diffusion_001`, $P(\text{fake}) = 34.24\%$) and localized composite splicing (`test_scene_manip_001`, $P(\text{fake}) = 4.91\%$). ViT-priority will inherit these general-scene limitations unless coupled with a domain router. | |
| 3. **Recommendation for Benchmark Expansion**: Prior to claiming general-scene deepfake detection reliability, an independently sourced, multi-generator benchmark (e.g., RAISE camera photography, Stable Diffusion 3.5, Midjourney v6, Flux.1, and localized inpainting datasets) must be curated and verified. | |
| --- | |
| ## 9. Phase 5 Deliverables Index | |
| All required files have been created and verified: | |
| - `src/decision_fusion.py`: Complete decision fusion engine with 4 strategies and `FusionResult` schema. | |
| - `scripts/evaluate_phase5_fusion.py`: Automated validation evaluation harness with subprocess isolation and held-out protocol. | |
| - `tests/test_decision_fusion.py`: Focused unit test suite covering all failure modes and tie-breaking contracts (15/15 passed). | |
| - `benchmark/results/phase5_validation_metrics.json`: Complete hierarchical metrics document with separation margins and sample transition catalogs. | |
| - `benchmark/results/phase5_val_raw_predictions.csv`: 150 individual detector inference records across 50 validation samples. | |
| - `docs/PHASE5_FUSION_FEASIBILITY_REPORT.md`: This comprehensive feasibility report. | |
| --- | |
| *Report completed and held-out test evaluation verified under explicit user approval.* | |
| --- | |
| ## 10. Post-Approval Held-Out Test Evaluation Results | |
| Upon receiving explicit user approval (--confirm), the frozen it_priority strategy was evaluated once against the 105-sample held_out_test partition using the reproducible protocol command: | |
| `powershell | |
| python scripts/evaluate_phase5_fusion.py --eval-test --strategy vit_priority --confirm | |
| ` | |
| ### 10.1 Empirical Performance on Held-Out Test Split (=105$) | |
| | Partition / Domain | Total Samples | Valid Samples | Accuracy | 95% Wilson CI | Precision | Recall | Specificity | F1 Score | False Positives | False Negatives | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | |
| | **Overall Classification** | 103 | 103 | **98.06%** (101/103) | [93.19%, 99.47%] | **100.00%** (50/50) | **96.15%** (50/52) | **100.00%** (51/51) | **0.9804** | 0 | 2 (diffusion_001, manip_001) | | |
| | **Facial Sub-Benchmark** | 100 | 100 | **100.00%** (100/100) | [96.30%, 100.00%] | **100.00%** (50/50) | **100.00%** (50/50) | **100.00%** (50/50) | **1.0000** | 0 | 0 | | |
| | **Scene Diagnostic Probes** | 3 | 3 | **33.33%** (1/3) | [6.15%, 79.23%] | **0.00%** (0/0) | **0.00%** (0/2) | **100.00%** (1/1) | **0.0000** | 0 | 2 (diffusion_001, manip_001) | | |
| | **Rejection Controls** | 2 | 2 | **100.00%** (2/2) | [34.24%, 100.00%] | N/A | N/A | N/A | N/A | 0 | 0 | | |
| ### 10.2 Comparison with Phase 4 Baseline Detectors | |
| | Model / Fusion Strategy | Overall Accuracy (N=103) | Facial Accuracy (N=100) | Scene Diagnostics (N=3) | Controls Rejected (N=2) | Execution Latency | | |
| | :--- | :---: | :---: | :---: | :---: | :---: | | |
| | **ConvNeXt/DRCT** | 41.75% (43/103) | 42.00% (42/100) | 33.33% (1/3) | 2/2 (100%) | ~140s | | |
| | **UnivFD (CLIP ViT-L/14)** | 59.22% (61/103) | 60.00% (60/100) | 33.33% (1/3) | 2/2 (100%) | ~120s | | |
| | **ViT Alone (dima806)** | 98.06% (101/103) | 100.00% (100/100) | 33.33% (1/3) | 2/2 (100%) | ~85s | | |
| | **Majority Vote (Simulated)** | ~71.84% (74/103) | ~74.00% (74/100) | 33.33% (1/3) | 2/2 (100%) | ~345s | | |
| | **ViT-Priority Fusion (Frozen)** | **98.06% (101/103)** | **100.00% (100/100)** | **33.33% (1/3)** | **2/2 (100%)** | Evaluated via Phase 4 cache | | |
| ### 10.3 Key Scientific Conclusions | |
| 1. **Fusion Feasibility Verdict**: Democratic ensembling (majority voting or mean probability averaging) severely harms facial deepfake detection performance when combining specialized detectors with general-purpose, face-insensitive detectors. | |
| 2. **ViT-Priority Value**: ViT-priority achieves the highest possible reliability among the explored fusion strategies by preventing weaker detectors from outvoting ViT on facial media while introducing an autonomous, software-level fallback chain when ViT fails or is missing. | |
| 3. **General Scene Limitation**: All three detectors share common blindspots on modern latent diffusion (Google Imagen 3) and spliced composites at default thresholds. Expanding detection capability across open-domain scenes requires adding an upstream domain classifier or specialized localization detectors rather than naive score blending. | |