grt-30m-reranker / README.md
davidhouseholter's picture
Add 8-way classical geometric cross-encoder optimizer ablation: README.md
016dd34 verified
|
Raw History Blame Contribute Delete
11.5 kB
---
license: apache-2.0
library_name: sentence-transformers
pipeline_tag: text-ranking
tags:
- sentence-transformers
- cross-encoder
- reranker
- grt
- rotational-transformer
- givens-rotations
- clifford-algebra
- matterwave
base_model: MatterWaveAI/grt-30m-embedding
---
# GRT-30M-Reranker (`MatterWaveAI/grt-30m-reranker`)
**GRT-30M-Reranker** (`GRTForSequenceClassification`) is a **100% Native Rotational Cross-Encoder Reranker** (`29,305,759` parameters, **zero `nn.Linear` layers**) compatible with both Hugging Face `AutoModelForSequenceClassification` and `sentence-transformers` `CrossEncoder`.
Unlike conventional cross-encoders that apply dense linear classification projections (`nn.Linear`), `GRTForSequenceClassification` pairs a 12-block Givens/Clifford rotational backbone (`GRTModel`) with a zero-`nn.Linear` **`GRTCosineClassificationHead`** (`reranker.score`):
1. **Hermitian Planar-Pair Phase Agreement (`u* v`)**: Circular-pooled query and document representations `(u, v)` over `D / 2 = 128` planar `(x, y)` pairs are combined via the Hermitian complex conjugate product `(ux*vx + uy*vy, ux*vy - uy*vx)` and scored against learned planar `phase_angles` and `pair_scales`.
2. **Multi-Vector Rotational Late Interaction (`MaxSim` / `MinSim`)**: Token-level rotational cosine similarity matrices between query and document token trajectories extract bidirectional `MaxSim` (`q→d` and `d→q`), worst-case token alignment `MinSim` (`q→d`), and lexical polar circular agreement (`lex_cos`), weighted by `late_weights` and calibrated by `output_scale` + `bias`.
---
## Reranking Benchmark Results (`AllNLI` & `STS-B`)
| Model / Pipeline | Architecture | AllNLI `200×10` MRR@10 | AllNLI `200×10` NDCG@10 | Recall@1 | Recall@3 | STS-B Test Spearman (`ρ`) | Throughput (pairs/s) |
| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| `MatterWaveAI/grt-30m` (Base Unaligned) | `GRTModel` (circular pool) | `37.99` | `52.12` | `20.5%` | `41.0%` | `32.51` | `120` |
| `MatterWaveAI/grt-30m-embedding` (Bi-Encoder) | `SentenceTransformer` (`circular`) | `52.56` | `63.74` | `32.5%` | `68.0%` | `62.42` | `152` |
| **`MatterWaveAI/grt-30m-reranker` (Head-Only Calibrated Checkpoint)** | `GRTForSequenceClassification` (`CrossEncoder`) | `56.93` | `67.13` | `38.5%` | `70.0%` | `61.75` | `123` |
| **Two-Stage Hybrid (`grt-30m-embedding` + Head-Only `grt-30m-reranker`)** | Bi-Encoder + Cross-Encoder Fusion | `57.05` | `67.21` | `38.5%` | `70.0%` | `62.94` | `123` |
| **`MatterWaveAI/grt-30m-reranker` (Top-Block Joint `HessianPairAdam` / `CyclideAdam`)** | `GRTForSequenceClassification` (`blocks[-1]` + `score`) | **`63.50`** | **`72.26`** | **`46.0%`** | **`76.5%`** | **`63.12`** | `123` |
| **Two-Stage Hybrid (`grt-30m-embedding` + Top-Block Joint `grt-30m-reranker`)** | Bi-Encoder + Cross-Encoder Fusion | **`62.14`** | **`71.18`** | **`44.0%`** | **`76.0%`** | **`64.09`** | `123` |
---
## 8-Way Classical Geometric Optimizer Ablation (`AllNLI Reranking + STS-B Cross-Encoder`)
To evaluate how classical geometric, Riemannian, conic, gyroscopic, Kuramoto–Huygens synchronization, and Gromov–Liouville symplectic optimizers (`transformers.models.grt.optimization_grt`) shape cross-encoder token-level `MaxSim`/`MinSim` and Hermitian phase alignment, we ran a controlled **8-way optimizer ablation** on **`AMD Radeon 8060S Graphics` (`ROCm / gfx1151`)** with fused HIP Givens/Clifford kernels:
- **Trainable Parameters (`812,169` / `0` `nn.Linear` layers)**: Top rotational transformer block `reranker.model.blocks[-1]` (`811,651` parameters: `[11, 128]` Q/K/V/O Givens cascades, `[1, 128]` Clifford mixer cascade, `[21, 512]` FFN expand/contract butterfly cascades, paired radial softplus scales, and `Cl(2,0)` bivector coupling tensors) + `final_norm` (`128` parameters) + `GRTCosineClassificationHead` (`262` parameters: `phase_angles`, `pair_scales`, `late_weights`, `output_scale`, `bias`), with Theorem 2.1 Cauchy–Riemann scale-partner pairing active on both `blocks[-1]` and `score.phase_angles`.
- **Controlled Schedule**: `200` steps (`batch_size = 128` AllNLI hard-negative triplets + `128` STS-B graded pairs per step, `backbone_lr = 1.5e-4`, `seed = 42`), evaluated on **AllNLI `200×10` Hard-Negative Reranking**, **AllNLI `100×20` Extended Candidate Reranking**, **STS-B Validation (`1,500` pairs)**, **STS-B Test (`1,379` pairs)**, and **Two-Stage Hybrid (`Bi-Encoder + Cross-Encoder`) Fusion**:
| Rank (`200×10` MRR) | Optimizer (`optimization_grt.py`) | AllNLI `200×10` MRR@10 | AllNLI `200×10` NDCG@10 | `200×10` Recall@1 / @3 | AllNLI `100×20` MRR@10 | AllNLI `100×20` NDCG@10 | STS-B Val Spearman (`×100`) | STS-B Test Spearman / Pearson (`×100`) | Hybrid `100×20` MRR@10 | Hybrid STS-B Test Spearman (`×100`) | ROCm Time (`200` Steps) |
| :---: | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **1** | **`HessianPairAdam` (`Sophia-Torus` Curvature)** | **`63.50` (BEST)** | **`72.26` (BEST)** | **`46.0%` (BEST)** / `76.0%` | `57.03` | `67.45` | `71.16` | `62.61` / `63.47` | `54.97` | `64.02` | `93.8s` |
| **2** | **`HuygensSymplecticAdam` (Two-Stage Curriculum)** | **`63.27` (#2)** | **`72.09` (#2)** | **`45.5%` (#2)** / `76.0%` | `57.38` (#3) | `67.70` (#3) | `71.18` | `62.49` / `63.40` | **`56.95` (BEST)** | `63.95` | `94.0s` |
| **3** | **`SymplecticAdam` (Non-Squeezing Floor + Cavity)** | **`63.13` (#3)** | **`72.00` (#3)** | **`45.0%` (#3)** / `76.0%` | `56.79` | `67.25` | `71.23` | `62.43` / `63.37` | `54.80` | `63.93` | `93.8s` |
| **4** | **`CyclideAdam` (Dupin Cyclide Conic + Twist)** | **`63.06` (#4)** | `71.91` (#4) | **`45.0%` (#3)** / **`76.5%` (BEST)** | **`57.44` (BEST)** | **`67.75` (BEST)** | **`71.85` (#2)** | **`63.12` (BEST)** / `64.07` | `56.30` (#2) | **`64.09` (BEST)** | `93.8s` |
| **5** | **`GyricAdam` (Poinsot–Landé + Thomas Precession)** | `62.75` | `71.68` | `44.5%` / **`76.5%` (BEST)** | `57.14` | `67.50` | **`71.87` (BEST)** | `63.08` / `64.09` (#2) | `55.76` | `64.01` | `93.9s` |
| **6** | **`HuygensAdam` (Circular Order Parameter `R_p`)** | `62.50` | `71.50` | `44.0%` / `76.0%` | `56.07` | `66.71` | `71.78` | **`63.11` (#2)** / `64.09` (#2) | `56.27` (#3) | **`64.08` (#2)** | `93.7s` |
| **7** | **`PolarNaturalAdam` (Riemannian Polar Metric)** | `62.41` | `71.42` | `44.0%` / **`76.5%` (BEST)** | `57.24` | `67.58` | `71.83` (#3) | **`63.11` (#2)** / **`64.12` (BEST)** | `55.82` | `64.06` | `93.7s` |
| **8** | **`PairAdam` (Factored Toroidal Baseline)** | `62.25` | `71.31` | `43.5%` / **`76.5%` (BEST)** | `57.41` (#2) | `67.71` (#2) | `71.73` | **`63.11` (#2)** / `64.07` | `55.77` | `64.06` | **`93.3s`** |
| — | **Head-Only `PairAdam` (Published Checkpoint)** | `56.93` | `67.13` | `38.5%` / `70.0%` | — | — | — | `61.75` / `62.99` | — | `62.94` | — |
| — | **Bi-Encoder `grt-30m-embedding` (`circular` pool)** | `52.56` | `63.74` | `32.5%` / `68.0%` | `45.63` | `57.33` | `71.79` | `62.42` / `63.30` | — | — | — |
### Key Geometric Insights for Cross-Encoder Reranking
1. **Joint Top-Block Rotational Adaptation (`+6.57` MRR@10 & `+7.5%` Recall@1 over Head-Only)**: Unfreezing `blocks[-1]` (`[11, 128]` QKV/O and `[21, 512]` FFN butterfly cascades) alongside `GRTCosineClassificationHead` allows token-level `MaxSim`/`MinSim` late-interaction matrices to sharpen hard-negative separation, lifting AllNLI `200×10` `MRR@10` from `56.93` to **`63.50`** (`Recall@1` from `38.5%` to **`46.0%`**) and `STS-B Test Spearman` from `61.75` to **`63.12`** (`64.09` Hybrid).
2. **`HessianPairAdam` (`Sophia-Torus`) Leads Top-1 Hard-Negative Discrimination (`63.50` MRR@10 / `72.26` NDCG@10 / `46.0%` Recall@1)**: Preconditioning both `blocks[-1]` cascade angles and `score.phase_angles` with exact Theorem 2.1 Cauchy–Riemann polar curvature (`c_theta`) and antipodal saddle-escape kicks achieves the highest `200×10` `MRR@10` (`+1.25` over `PairAdam`) and `Recall@1` (`+2.5%` over `PairAdam`).
3. **`HuygensSymplecticAdam` & `SymplecticAdam` Dominate Two-Stage Curriculum & Hybrid `100×20` Reranking (`56.95` Hybrid `100×20` MRR@10 [BEST], `63.27` `200×10` MRR@10 [#2])**: Annealing early Kuramoto–Huygens circular order-parameter splay synchronization into the Gromov–Liouville symplectic non-squeezing floor and Helmholtz–Casimir wave-cavity Laplacian yields the #2 standalone `200×10` reranker (`63.27` MRR@10, `45.5%` Recall@1) and the **#1 Two-Stage Hybrid `100×20` reranker (`56.95` MRR@10, `+1.18` over `PairAdam`)**.
4. **`CyclideAdam`, `GyricAdam` & `PolarNaturalAdam` Lead Semantic Calibration & Deep Candidate Pools (`57.44` `100×20` MRR@10 [BEST], `63.12` STS-B Test Spearman [BEST], `64.12` Test Pearson [BEST])**: Dupin Cyclide conic metrics with trace-free deviatoric twist (`CyclideAdam`) achieve the best 20-candidate reranking (`57.44` MRR@10, `70.0%` Recall@3) and best STS-B Test Spearman (`63.12` standalone / `64.09` hybrid), while `GyricAdam` wins STS-B Validation (`71.87` Spearman) and `PolarNaturalAdam` wins STS-B Test Pearson (`64.12`).
---
## Quickstart (`sentence-transformers` `CrossEncoder`)
```python
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("MatterWaveAI/grt-30m-reranker")
query = "How do Givens rotations replace dense linear projections in a Transformer?"
documents = [
"Cascaded 2D Givens rotations and butterfly permutations mix planar coordinate pairs with O(D log D) steps and zero nn.Linear layers.",
"The stock market closed slightly higher on Tuesday after retail earnings reports.",
"Standard Transformers rely on dense matrix multiplications (nn.Linear) in QKV and FFN projections.",
]
results = reranker.rank(query, documents, return_documents=True)
for r in results:
print(f"{r['score']:.4f} | {r['text']}")
```
---
## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face
1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
8. **Rotational Text-to-Speech (`GRTTTS`)**: [`MatterWaveAI/grt-30m-tts`](https://huggingface.co/MatterWaveAI/grt-30m-tts)
9. **Multimodal Vision-Audio-Language-Speech (`GRTOmni`)**: [`MatterWaveAI/grt-30m-omni`](https://huggingface.co/MatterWaveAI/grt-30m-omni)
- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
---
## License
Apache 2.0
---
## Citation
```bibtex
@techreport{householter2026grt,
title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
author = {Householter, David},
institution = {MatterWave AI},
year = {2026},
url = {https://matterwave.ai}
}
```