Spaces:
Running on Zero
Running on Zero
| # Robustness Evaluation Pack | |
| This document explains the robustness evaluation tools used in AuralGuard-AASIST++. | |
| --- | |
| ## 1. Purpose | |
| Standard accuracy is not enough to understand whether an audio deepfake detector is reliable. | |
| A model can have high accuracy but still fail in specific real-world situations. | |
| The robustness evaluation pack tests questions such as: | |
| - Does the model falsely flag real accented speech? | |
| - Does the model behave differently for male and female speakers? | |
| - Does audio length affect the prediction? | |
| - Are the model probabilities believable? | |
| - Can the model approximately locate partial fake regions? | |
| --- | |
| ## 2. Main Evaluations | |
| ### False-Alarm Evaluation | |
| Checks how often real speech is wrongly predicted as fake. This is the most important evaluation for the project. | |
| ### Gender Bias Diagnosis | |
| Checks whether false-fake rates differ across gender groups when metadata is available. This is reported as a diagnostic, not as a complete fairness solution. | |
| ### Audio Length Robustness | |
| Checks whether predictions change when the same type of audio is evaluated with different durations. | |
| ### Calibration | |
| Checks whether the model’s confidence scores are meaningful. | |
| ### Partial Fake Localization | |
| Checks whether sliding-window inference can point near manipulated regions in simulated partial-fake examples. | |
| --- | |
| ## 3. Main Scripts | |
| | Script | Purpose | | |
| |---|---| | |
| | `evaluate_false_alarms.py` | Tests false fake rates on real speech | | |
| | `evaluate_gender_bias.py` | Compares subgroup false-alarm behavior | | |
| | `evaluate_length_robustness.py` | Tests different input lengths | | |
| | `evaluate_calibration.py` | Measures probability calibration | | |
| | `create_partial_fake_set.py` | Creates simulated partial fake examples | | |
| | `evaluate_partial_localization.py` | Tests timestamp localization quality | | |
| | `build_final_research_tables.py` | Builds summary result tables | | |
| --- | |
| ## 4. Important Results | |
| | Test | Result | | |
| |---|---:| | |
| | EdAcc false fake rate improved | 71.33% to 1.67% | | |
| | GLOBE false fake rate improved | 89.90% to 1.01% | | |
| | Calibration ECE | 0.0249 | | |
| | Partial fake hit rate | 71.67% | | |
| | Mean center error | 0.477 seconds | | |
| --- | |
| ## 5. Why This Matters | |
| These tests make the project stronger because they show more than simple training performance. | |
| They show how the system behaves under realistic conditions and whether the model output should be trusted, reviewed, or treated carefully. | |
| --- | |
| ## 6. Summary | |
| The robustness evaluation pack supports the central claim of the project: AuralGuard-AASIST++ is not only a detector. It is a robustness-focused forensic decision-support pipeline. | |