Add 8-way classical geometric Mixture-of-Rotations optimizer ablation: README.md

README.md

@@ -56,15 +56,37 @@ Instead of routing tokens through multiple separate dense `nn.Linear` feed-forwa
 3. **Zero-Sum Walsh–Hadamard Upcycling & Multi-Domain Specialization (`PairAdam`)**:
    Initialized from `MatterWaveAI/grt-30m` via zero-sum Walsh–Hadamard angular perturbations (`Σ_e Δ^(e) = 0`) and specialized on a 4-domain corpus (`rotational_geometry`, `science_and_physics`, `code_and_algorithms`, `reasoning_and_language`) on an Intel Arc 140T XPU (`xpu:0`) with `PairAdam` and Switch-style load balancing:
 
-| Metric | Initial Upcycle | After MoR Specialization (`PairAdam`) |
-| :--- | :---: | :---: |
-| **Held-Out Validation Loss** | `5.8467` | **`0.5115`** |
-| **Held-Out Validation Perplexity** | `346.10` | **`1.67`** |
-| **Domain Loss — `code_and_algorithms`** | `5.4092` | **`0.4334`** |
-| **Domain Loss — `rotational_geometry`** | `6.6421` | **`0.4991`** |
-| **Domain Loss — `science_and_physics`** | `5.3337` | **`0.5293`** |
-| **Domain Loss — `reasoning_and_language`** | `6.1720` | **`0.6442`** |
-| **Mean Expert Routing Load (`E = 4`)** | `[0.25, 0.25, 0.25, 0.25]` | **`[0.250, 0.248, 0.256, 0.245]`** (`entropy = 1.386 / 1.386`) |
+| Metric | Initial Upcycle | After MoR Specialization (`PairAdam`, XPU) | After MoR Specialization (`PairAdam`, ROCm `gfx1151`) |
+| :--- | :---: | :---: | :---: |
+| **Held-Out Validation Loss** | `5.8467` | `0.5115` | **`0.4829`** |
+| **Held-Out Validation Perplexity** | `346.10` | `1.67` | **`1.62`** |
+| **Domain Loss — `code_and_algorithms`** | `5.4092` | `0.4334` | **`0.3669`** |
+| **Domain Loss — `rotational_geometry`** | `6.6421` | `0.4991` | **`0.4772`** (`0.4747` w/ `PolarNaturalAdam`) |
+| **Domain Loss — `science_and_physics`** | `5.3337` | `0.5293` | **`0.5155`** |
+| **Domain Loss — `reasoning_and_language`** | `6.1720` | `0.6442` | **`0.6466`** (w/ `PolarNaturalAdam`) |
+| **Mean Expert Routing Load (`E = 4`)** | `[0.25, 0.25, 0.25, 0.25]` | `[0.250, 0.248, 0.256, 0.245]` | **`[0.250, 0.250, 0.254, 0.246]`** (`entropy = 1.9999 / 2.0000 bits`) |
+
+---
+
+## 8-Way Classical Geometric Optimizer Ablation (`Spark ROCm gfx1151`)
+
+We benchmarked 8 classical geometric, symplectic, and gyroscopic optimizers from `transformers.models.grt.optimization_grt` on `MatterWaveAI/grt-30m-mor` (`80` Phase-2 specialization steps on `blocks[8:12]` + `final_norm` from an identical Phase-1 cached prefix state, `batch_size = 36`, `lr = 3.5e-3`, `max_angular_step = 0.08`, `seed = 42`):
+
+| Rank | Optimizer (`optimizer_type`) | Held-Out Val Loss | Held-Out Val PPL | `code_and_algorithms` Loss | `rotational_geometry` Loss | `science_and_physics` Loss | `reasoning_and_language` Loss | Routing Entropy (bits / `2.0000`) | Top-4 Expert Angle L2 Div | All-Layer Angle L2 Div | Time (s) |
+| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **1** | **`PairAdam` (`pair_adam`)** | **`0.4829`** | **`1.6208`** | **`0.3669`** | `0.4772` | **`0.5155`** | `0.6553` | `1.9999` (`1.3862` nats) | `2.5347` | `2.5386` | `180.15s` |
+| **2** | **`PolarNaturalAdam` (`polar_natural_adam`)** | `0.4848` | `1.6239` | `0.3744` | **`0.4747`** | `0.5226` | **`0.6466`** | `1.9999` (`1.3862` nats) | `2.5349` | `2.5387` | `180.75s` |
+| **3** | **`CyclideAdam` (`cyclide_adam`)** | `0.5006` | `1.6498` | `0.3758` | `0.4986` | `0.5426` | `0.6718` | `1.9999` (`1.3862` nats) | `2.5789` | `2.5534` | `180.09s` |
+| **4** | **`HuygensAdam` (`huygens_adam`)** | `0.5225` | `1.6862` | `0.3957` | `0.5107` | `0.5696` | `0.7041` | `1.9999` (`1.3862` nats) | **`2.6216`** | **`2.5676`** | `176.07s` |
+| **5** | **`HuygensSymplecticAdam` (`huygens_symplectic_adam`)** | `0.5230` | `1.6871` | `0.4091` | `0.5202` | `0.5570` | `0.6859` | `1.9999` (`1.3862` nats) | `2.5351` | `2.5388` | **`176.01s`** |
+| **6** | **`HessianPairAdam` (`hessian_pair_adam`)** | `0.5403` | `1.7165` | `0.4082` | `0.5330` | `0.5916` | `0.7201` | `1.9999` (`1.3862` nats) | `2.5349` | `2.5387` | `180.09s` |
+| **7** | **`SymplecticAdam` (`symplectic_adam`)** | `0.5567` | `1.7448` | `0.4185` | `0.5530` | `0.6082` | `0.7417` | `1.9999` (`1.3862` nats) | `2.5345` | `2.5386` | `180.20s` |
+| **8** | **`GyricAdam` (`gyric_adam`)** | `0.5745` | `1.7763` | `0.4279` | `0.5672` | `0.6412` | `0.7609` | `1.9999` (`1.3862` nats) | `2.5350` | `2.5387` | `177.29s` |
+
+### Key Geometric Findings on Mixture-of-Rotations (`ffn_n_experts = 4`)
+- **Huygens Circular Order Parameter Splay Torque Prevents Expert Collapse**: On 3D MoR expert tensors (`expert_expand_angles [4, 21, 512]` and `expert_contract_angles [4, 21, 512]`), `HuygensAdam` computes the circular synchronization order parameter $R_p = |\frac{1}{4}\sum_{e=0}^3 e^{i\theta_{e,k,p}}|$ directly across dimension `0` (`E = 4` experts) and applies a zero-mean splay restoring torque whenever experts over-synchronize, achieving the **highest pairwise expert angle L2 divergence (`2.6216` in top-4 MoR blocks, `2.5676` all-layer mean)** while maintaining `1.9999 / 2.0000` bits of routing entropy.
+- **Polar Riemannian Metric Excels on Conceptual & Geometric Domains**: `PolarNaturalAdam` achieves **#1 validation loss on `rotational_geometry` (`0.4747`)** and **#1 validation loss on `reasoning_and_language` (`0.6466`)**, finishing within `0.0019` overall loss of `PairAdam` (`0.4848` vs. `0.4829`).
+- **Curriculum Advantage of `HuygensSymplecticAdam`**: Combining early-stage Huygens expert splay repulsion with late-stage Gromov–Liouville symplectic non-squeezing (`HuygensSymplecticAdam`, `0.5230` loss / `1.6871` PPL) outperforms standalone `SymplecticAdam` (`0.5567`), `HessianPairAdam` (`0.5403`), and `GyricAdam` (`0.5745`).
 
 ---