| # Phase 3 Master Plan: Knowledge Distillation (KD) |
|
|
| ## 1. Executive Summary |
| Phase 3 focuses on "distilling" the spatial awareness and high-accuracy gaze features of a State-of-the-Art **Teacher (L2CS-Net)** into our lightweight **Student (LIPE V2)**. By training the Student to replicate the Teacher's output distribution, we aim to achieve professional-grade accuracy (< 4.2° error) at a fraction of the computational cost (~0.001 GFLOPs). |
|
|
| ## 2. Technical Architecture |
|
|
| ### A. The Teacher (Feature Provider) |
| * **Model:** L2CS-Net with ResNet backbone. |
| * **Primary Role:** Generate "Soft Labels" for every sample. |
| * **Optimization:** **Offline Distillation.** Labels are pre-computed to eliminate inference latency during the main training loop. |
|
|
| ### B. The Student (Feature Trainee) |
| * **Model:** `LIPEV2Student` (Refined Architecture). |
| * **Primary Role:** Learn to predict gaze using only 8x8 patches and zero-centered landmarks. |
|
|
| ### C. Hybrid Loss Strategy ($L_{total}$) |
| $L_{total} = w_1 \cdot L_{AW} + w_2 \cdot L_{KD}$ |
|
|
| #### Dynamic Weighting Schedule (Refined with Gradient Smoothing) |
| To prevent **Gradient Shock**, transitions between stages use Linear Annealing over a 10-epoch window: |
| * **Stage 0: Sanity Check (Hard-Threshold):** Fixed $w_1=1.0, w_2=0.5$ (Epochs 1-5). |
| * **Stage 1: Teacher Warm-up:** $w_1=0.2, w_2=1.0$ (Epochs 6-30). |
| * **Transition Phase:** Linearly interpolate $w_1 \to 1.0$ and $w_2 \to 0.1$ (Epochs 31-40). |
| * **Stage 2: GT Fine-tuning:** Fixed $w_1=1.0, w_2=0.1$ (Epochs 41-100). |
|
|
| ## 3. Advanced Training Techniques |
|
|
| ### A. Performance Optimization: Offline Distillation |
| To eliminate **On-the-fly Latency**, we shift from real-time Teacher inference to **Offline Label Caching**: |
| * A pre-processing script will run the Teacher model once on the entire dataset. |
| * Teacher "Soft Labels" (Pitch/Yaw logits) will be stored in the HDF5 files. |
|
|
| ### B. Fairness: Subject-Level Weighted Sampling |
| To mitigate **Subject Imbalance** (e.g., p03 vs p10): |
| * Implement a `WeightedRandomSampler` where each sample weight is $1/N_{participant\_samples}$. |
| * This ensures each participant contributes equally to the gradient updates. |
|
|
| ### C. Gradient Control with AW Loss |
| We will tune the AW Loss parameters ($\alpha, \omega, \epsilon$) specifically for 8x8 patches to handle optical aliasing and prevent gradient explosion. |
|
|
| ## 4. Validation Strategy: LOPO (Leave-One-Person-Out) |
| 15-fold cross-validation to ensure subject-independent performance. |
|
|
| ## 5. Execution Roadmap |
|
|
| ### Step 1: Teacher Integration & Offline Caching |
| - [x] Load pre-trained L2CS-Net weights. |
| - [x] **New Task:** Develop `src/data/generate_teacher_labels.py`. |
| - [x] Pre-compute and save soft labels into `data/processed/pXX.h5`. |
|
|
| ### Step 2: Training Loop Implementation (`src/train.py`) |
| - [x] **Data Pipeline:** Connect `GazeDataset` with `WeightedRandomSampler`. |
| - [x] **Granular Logging:** Separate tracking for AW Loss, KD Loss, and MAE. |
| - [x] **Scheduler Integration:** Implement the Linear Annealing weight transition. |
| - [x] **Checkpointing:** Save best `.pt` weights based on Angular Error. |
| - [x] **Early Stopping:** Implemented patience-based stopping to optimize LOPO duration. |
|
|
| ### Step 3: Intermediate Verification & Quality Control |
| - [x] **Teacher Label Audit:** Verified keys in HDF5 files. |
| - [x] **Metric Validation:** Unit tested in previous sessions. |
| - [x] **Convergence Monitoring:** Verified smooth transition and improvement during epochs 31-40. |
|
|
| ### Step 4: Benchmarking & Finalization |
| - [ ] **LOPO Automation:** Created `src/lopo_train.py`. |
| - [ ] Compute mean 3D **Angular Error** ($\mathcal{E}_{angular}$) across all 15 LOPO folds. |
| - [ ] Final weight export (Target size < 2 MB). |
| - [ ] **Hardware Profiling:** Verify inference speed (FPS) and peak RAM on a standard CPU. |
| |
| |
