docs/robustness_evaluation.md

Robustness Evaluation Pack

This document explains the robustness evaluation tools used in AuralGuard-AASIST++.

---

1. Purpose

Standard accuracy is not enough to understand whether an audio deepfake detector is reliable.

A model can have high accuracy but still fail in specific real-world situations.

The robustness evaluation pack tests questions such as:

• Does the model falsely flag real accented speech?
• Does the model behave differently for male and female speakers?
• Does audio length affect the prediction?
• Are the model probabilities believable?
• Can the model approximately locate partial fake regions?

---

2. Main Evaluations

False-Alarm Evaluation

Checks how often real speech is wrongly predicted as fake. This is the most important evaluation for the project.

Gender Bias Diagnosis

Checks whether false-fake rates differ across gender groups when metadata is available. This is reported as a diagnostic, not as a complete fairness solution.

Audio Length Robustness

Checks whether predictions change when the same type of audio is evaluated with different durations.

Calibration

Checks whether the model’s confidence scores are meaningful.

Partial Fake Localization

Checks whether sliding-window inference can point near manipulated regions in simulated partial-fake examples.

---

3. Main Scripts

Script	Purpose
evaluate_false_alarms.py	Tests false fake rates on real speech
evaluate_gender_bias.py	Compares subgroup false-alarm behavior
evaluate_length_robustness.py	Tests different input lengths
evaluate_calibration.py	Measures probability calibration
create_partial_fake_set.py	Creates simulated partial fake examples
evaluate_partial_localization.py	Tests timestamp localization quality
build_final_research_tables.py	Builds summary result tables

---

4. Important Results

Test	Result
EdAcc false fake rate improved	71.33% to 1.67%
GLOBE false fake rate improved	89.90% to 1.01%
Calibration ECE	0.0249
Partial fake hit rate	71.67%
Mean center error	0.477 seconds

---

5. Why This Matters

These tests make the project stronger because they show more than simple training performance.

They show how the system behaves under realistic conditions and whether the model output should be trusted, reviewed, or treated carefully.

---

6. Summary

The robustness evaluation pack supports the central claim of the project: AuralGuard-AASIST++ is not only a detector. It is a robustness-focused forensic decision-support pipeline.