| # LIPE V2 Student Model Specification & Tracking |
|
|
| ## 1. Architecture Overview |
| The Student model is an asymmetric dual-branch network designed for high-efficiency gaze estimation on CPU/iGPU. |
|
|
| ### Branch A: Mini Conv-Embedder (Appearance) |
| * **Purpose:** Extract appearance features from eye patches. |
| * **Input:** `(Batch, 4, 8, 8)` - 4 grayscale patches of 8x8 pixels. |
| * **Layers:** |
| * Shared CNN Backbone: |
| * Conv2d(1 -> 16, k=3, p=0) + ReLU (Out: 6x6) |
| * Conv2d(16 -> 32, k=3, p=0) + ReLU (Out: 4x4) |
| * Conv2d(32 -> 64, k=3, p=0) + ReLU (Out: 2x2) |
| * Global Average Pooling (GAP) (Out: 64) |
| * Flatten & Reshape: $4 \times 64 = 256$ features. |
|
|
| ### Branch B: Geometric MLP (Coarse) |
| * **Purpose:** Extract geometric features from facial landmarks. |
| * **Input:** `(Batch, 956)` - 478 landmarks (x, y) flattened and Zero-Centered. |
| * **Layers:** |
| * Linear(956 -> 256) + LayerNorm + ReLU + Dropout(0.05) |
| * Linear(256 -> 256) + ReLU |
|
|
| ### Fusion & Regression Heads |
| * **Fusion:** Residual Addition (Appearance [256] + Geometry [256] = 256). |
| * **Pitch Head:** Linear(256 -> 64) -> ReLU -> Linear(64 -> 1) |
| * **Yaw Head:** Linear(256 -> 64) -> ReLU -> Linear(64 -> 1) |
|
|
| --- |
|
|
| ## 2. Technical Targets |
| | Metric | Target Value | Current Status | |
| | :--- | :--- | :--- | |
| | **Computational Cost** | < 0.12 GFLOPs | ~0.001 GFLOPs (Verified) | |
| | **RAM Usage** | < 45 MB | ~1.5 MB (Weights only) | |
| | **Inference Speed** | >= 30 FPS (CPU) | TBD (Estimated >> 100 FPS) | |
| | **Angular Error** | < 4.2° | TBD | |
|
|
| --- |
|
|
| ## 3. Implementation Checklist |
| - [x] Define `LIPEV2Student` class in `src/models/student.py`. |
| - [x] Implement `Mini Conv-Embedder` with shared weights. |
| - [x] Implement `Geometric MLP` branch. |
| - [x] Implement `forward` method with State A/B switching logic (Residual Addition). |
| - [x] Implement `AdaptiveWingLoss` in `src/models/loss.py`. |
| - [x] Weight initialization (Kaiming/Xavier). |
| - [x] Verify forward pass with dummy tensors. |
| - [ ] Estimate GFLOPs using `thop` or manual calculation. |
|
|
| --- |
|
|
| ## 4. Input/Output Specs |
| * **Input Patches:** `torch.Tensor` of shape `(N, 4, 8, 8)`, dtype `float32`, range `[0, 1]`. |
| * **Input Landmarks:** `torch.Tensor` of shape `(N, 956)`, dtype `float32`, Zero-Centered. |
| * **Output:** `torch.Tensor` of shape `(N, 2)` representing `[Pitch, Yaw]` in Radians. |
|
|
