reports/ABLATION.md

# Ablation — baseline / +TTA / +TTA+SL(4) / +TTA+SL(4)+QV

_Date: 2026-04-24. All 15 jobs completed (22233980 + 22234159-22234177)._
_Source dirs under `/gpfs/work2/0/prjs0824/qi_proj/`._

Four-cell ablation over **7 benchmark datasets** to isolate the
contribution of each pipeline component.

| row | tag | description |
|---|---|---|
| A | baseline | SLAMFormer paper ckpt-10, no TTA, no PGO |
| B | +TTA | adds test-time Adam on scorer/desc heads against live layer-1 Q·V KL |
| C | +TTA+SL(4) | +TTA, then offline SL(4) projective PGO (`--use_indep`) |
| D | +TTA+SL(4)+QV | +TTA+SL(4), adds Q·V-loop-closure edges filtered at z-score τ = 1.0 |

Indoor uses bn=10, outdoor bn=15. All ATE in meters, Sim3-aligned.

---

## 1. Indoor — per-scene ATE (bn_every = 10)

### 1.1 TUM RGB-D — 9 scenes

| scene | A baseline | B +TTA | C +TTA+SL4 | D +TTA+SL4+QV |
|---|---:|---:|---:|---:|
| 360         | 0.175 | 0.076 | 0.080 | 0.083 |
| desk        | 0.068 | 0.018 | 0.142 | 0.131 |
| desk2       | 0.076 | 0.023 | 0.095 | 0.047 |
| floor       | 0.684 | 0.248 | 0.232 | **0.138** |
| plant       | 0.092 | 0.023 | 0.031 | 0.070 |
| room        | 0.579 | 0.105 | 0.152 | 0.151 |
| rpy         | 0.034 | 0.020 | 0.022 | 0.023 |
| teddy       | 0.253 | 0.069 | **0.042** | 0.083 |
| xyz         | 0.032 | 0.010 | 0.011 | 0.013 |
| **MEAN**    | **0.222** | **0.066** | **0.090** | **0.082** |
| **Δ vs A**  | — | **−70 %** | **−59 %** | **−63 %** |

**Finding.** +TTA carries the indoor win (−70 % ATE). +SL4 regresses
on 6/9 scenes; +QV rescues 3 of those (desk2, floor, room) but net
averages worse than +TTA-alone. Per-scene picks: +SL4+QV is best on
floor (loop-rich), +SL4 best on teddy.

### 1.2 7-Scenes — 17 sequences

| sequence | A | B +TTA | C +TTA+SL4 | D +QV |
|---|---:|---:|---:|---:|
| chess_sequence3       | 0.268 | 0.042 | 0.064 | 0.083 |
| chess_sequence5       | 0.222 | 0.038 | 0.045 | 0.053 |
| fire_sequence3        | 0.138 | 0.025 | 0.042 | 0.067 |
| fire_sequence4        | 0.113 | 0.025 | 0.027 | 0.038 |
| heads_sequence1       | 0.059 | 0.017 | 0.043 | 0.044 |
| office_sequence2      | 0.238 | 0.075 | 0.061 | 0.065 |
| office_sequence6      | 0.353 | 0.087 | 0.111 | 0.108 |
| office_sequence7      | 0.411 | 0.099 | 0.096 | 0.109 |
| pumpkin_sequence1     | 0.218 | 0.067 | 0.083 | 0.083 |
| pumpkin_sequence7     | 0.233 | 0.060 | 0.088 | 0.083 |
| redkitchen_sequence12 | 0.287 | 0.062 | 0.088 | 0.092 |
| redkitchen_sequence14 | 0.276 | 0.053 | 0.080 | 0.080 |
| redkitchen_sequence3  | 0.047 | 0.049 | 0.050 | 0.050 |
| redkitchen_sequence4  | 0.187 | 0.050 | 0.050 | 0.050 |
| redkitchen_sequence6  | 0.161 | 0.076 | 0.082 | 0.079 |
| stairs_sequence1      | 0.027 | 0.027 | 0.041 | 0.041 |
| stairs_sequence4      | 0.034 | 0.035 | 0.034 | 0.034 |
| **MEAN**              | **0.193** | **0.052** | **0.064** | **0.068** |
| **Δ vs A**            | — | **−73 %** | **−67 %** | **−65 %** |

**Finding.** +TTA best on 16 of 17 sequences. SL4/QV regress uniformly.
No QV edges fire above z=1.0 on most scenes → scale-only inter edges
just drift freely.

### 1.3 NRGBD — 8 scenes

| scene | A | B +TTA | C +TTA+SL4 | D +QV |
|---|---:|---:|---:|---:|
| breakfast_room    | 0.121 | 0.019 | 0.052 | 0.081 |
| complete_kitchen  | 0.362 | 0.039 | 0.299 | 0.102 |
| green_room        | 0.235 | 0.022 | 0.057 | 0.076 |
| grey_white_room   | 0.117 | 0.048 | 0.057 | 0.138 |
| kitchen           | 0.471 | 0.064 | 0.166 | 0.172 |
| morning_apartment | 0.088 | 0.016 | 0.019 | 0.027 |
| staircase         | 0.210 | 0.021 | 0.024 | 0.138 |
| whiteroom         | 0.442 | 0.029 | 0.158 | 0.141 |
| **MEAN**          | **0.256** | **0.032** | **0.104** | **0.109** |
| **Δ vs A**        | — | **−87 %** | **−59 %** | **−57 %** |

**Finding.** Cleanest ablation: TTA is the single decisive lever on
NRGBD. Raw SL4 regresses 3×; QV pulls back complete_kitchen to 0.102
(the one scene with real revisits in the submap sequence).

### 1.4 ETH3D — 13 scenes

| scene | A | B +TTA | C +TTA+SL4 | D +QV |
|---|---:|---:|---:|---:|
| courtyard     | 1.363 | 0.469 | 1.126 | 1.126 |
| delivery_area | 1.029 | 0.252 | 0.251 | 0.272 |
| electro       | 0.721 | 0.236 | 0.256 | 0.271 |
| facade        | 2.463 | 0.340 | 1.661 | 1.070 |
| kicker        | 0.169 | 0.046 | 0.047 | 0.075 |
| meadow        | 0.210 | 0.248 | 0.215 | 0.215 |
| office        | 0.085 | 0.069 | 0.069 | 0.072 |
| pipes         | 0.065 | 0.060 | 0.076 | 0.076 |
| playground    | 0.282 | 0.157 | 0.195 | 0.208 |
| relief        | 0.147 | 0.150 | 0.120 | 0.120 |
| relief_2      | 0.124 | 0.119 | 0.114 | 0.114 |
| terrace       | 0.113 | 0.120 | 0.113 | 0.113 |
| terrains      | 0.281 | 0.061 | 0.178 | 0.232 |
| **MEAN**      | **0.542** | **0.179** | **0.340** | **0.305** |
| **Δ vs A**    | — | **−67 %** | **−37 %** | **−44 %** |

**Finding.** +TTA again wins net. +QV rescues facade (−36 % vs +SL4)
because facade has a real outdoor loop. But on 9/13 scenes +QV ≥ +SL4.

---

## 2. Outdoor — per-scene ATE (bn_every = 15)

### 2.1 Oxford Spires — 2 GT-testable scenes (paper ckpt vs V4)

| scene | A baseline | B +TTA | C +TTA+SL4 | D +TTA+SL4+QV |
|---|---:|---:|---:|---:|
| **paper ckpt:**
| christ-church-01 | 24.19 | 13.87 | 14.65 | 14.48 |
| keble-college-02 | 28.40 | 10.77 | 10.05 | **9.98** |
| MEAN (paper)     | 26.29 | **12.32** | 12.35 | **12.23** |
| **Δ vs A**       | — | −53 % | −53 % | **−53 %** |
| **V4 ckpt:**
| christ-church-01 | 20.52 | 20.52 | 23.24 | 20.99 |
| keble-college-02 | 28.07 | 28.07 | 26.78 | 28.04 |
| MEAN (V4)        | 24.29 | 24.29 | 25.01 | 24.51 |
| **Paper vs V4**  | paper 8 % better | paper 49 % better | paper 51 % better | paper 50 % better |

**Finding.** Paper ckpt + QV wins at 12.23 m. QV's 0.7 % improvement
on keble-02 confirms it's picking up the real indoor loop. **V4 loses
to paper everywhere** — the V4-B'' ckpt-1 overlap curriculum did NOT
transfer to Oxford outdoor.

### 2.2 KITTI Odometry — 11 sequences (paper vs V4)

| seq | A | B +TTA paper | C paper +SL4 | D paper +QV | B +TTA V4 | C V4 +SL4 | D V4 +QV |
|---|---:|---:|---:|---:|---:|---:|---:|
| 00 | 196.1 | 171.5 | 180.9 | 161.4 | 185.8 | 195.4 | 180.5 |
| 01 | 752.6 | 629.8 | 615.8 | 651.5 | 662.9 | 656.8 | 656.8 |
| 02 | 306.0 | 297.9 | 267.2 | 269.4 | 291.7 | 287.8 | 282.1 |
| 03 | 161.1 | 106.4 | 103.2 | 109.7 | 86.8 | 82.7 | 92.4 |
| 04 | 110.5 |  58.9 |  59.5 |  61.4 | 175.6 | 175.6 | — |
| 05 | 159.7 | 156.5 | 161.1 | 135.5 | 157.2 | 156.1 | 156.1 |
| 06 | 130.7 | 117.0 | 115.0 | 118.8 | 118.5 | 118.4 | 117.5 |
| 07 |  83.5 |  59.1 |  63.4 |  61.7 |  65.6 |  72.5 |  71.9 |
| 08 | 272.5 | 271.1 | 146.7 | 255.5 | 264.4 | 243.0 | 260.8 |
| 09 | 219.9 | 200.9 | 195.5 | 202.7 | 197.7 | 193.5 | 195.0 |
| 10 | 208.4 | 198.6 | 181.3 | 199.1 | 182.5 | 193.9 | 184.9 |
| **MEAN** | **236.5** | **206.1** | **189.9** | 202.4 | 217.1 | 216.0 | 219.8 |
| **Δ vs A (paper)** | — | −13 % | **−20 %** | −14 % | −8 % | −9 % | −7 % |

**Finding.** Paper ckpt + SL4 is the best KITTI recipe at mean 189.9 m —
**SL4 provides a consistent −7 % over +TTA-alone on KITTI.** QV hurts
(sparse feature matches outdoor don't give good z-score signal). V4 is
worse than paper on all 11 sequences — vKITTI2 training doesn't help.

### 2.3 Tanks & Temples — 6 Training scenes

| scene | A | B +TTA paper | C paper +SL4 | D paper +QV | B +TTA V4 |
|---|---:|---:|---:|---:|---:|
| Barn         | 1.302 | 0.192 | 0.393 | 0.292 | 0.568 |
| Caterpillar  | 0.340 | 0.160 | 0.236 | 0.236 | 0.626 |
| Church       | 3.215 | 0.731 | 1.049 | 1.087 | 3.192 |
| Ignatius     | 0.770 | 0.345 | 0.373 | 0.736 | 0.552 |
| Meetingroom  | 1.434 | 0.304 | 0.550 | 0.438 | 0.750 |
| Truck        | 3.682 | 3.616 | 3.621 | 3.621 | 3.467 |
| **MEAN**     | **1.790** | **0.891** | 1.037 | 1.068 | 1.526 |
| **Δ vs A**   | — | **−50 %** | −42 % | −40 % | −15 % |

**Finding.** +TTA wins on T&T. SL4 regresses; QV no help. V4 is 71 %
worse than paper.

---

## 3. Summary of all 4 rows across all 7 datasets

### Indoor (bn=10, paper ckpt)

| dataset | A baseline | B +TTA | C +SL4 | D +QV | best |
|---|---:|---:|---:|---:|:---:|
| TUM (9)     | 0.222 | **0.066** | 0.090 | 0.082 | B |
| 7Sc (17)    | 0.193 | **0.052** | 0.064 | 0.068 | B |
| NRGBD (8)   | 0.256 | **0.032** | 0.104 | 0.109 | B |
| ETH3D (13)  | 0.542 | **0.179** | 0.340 | 0.305 | B |

### Outdoor (bn=15)

| dataset | ckpt | A | B | C | D | best |
|---|---|---:|---:|---:|---:|:---:|
| Oxford (2)  | paper | 26.29 | 12.32 | 12.35 | **12.23** | D |
| Oxford (2)  | V4    | 24.29 | 24.29 | 25.01 | 24.51 | A=B |
| KITTI (11)  | paper | 236.5 | 206.1 | **189.9** | 202.4 | C |
| KITTI (11)  | V4    | 219.8 | 217.1 | **216.0** | 219.8 | C |
| T&T (6)     | paper | 1.790 | **0.891** | 1.037 | 1.068 | B |
| T&T (6)     | V4    | 1.790 | **1.526** | 1.766 | 1.619 | B |

---

## 4. AUC@3° and AUC@30° — TTA is what lifts rotation

| dataset | AUC@3° A | AUC@3° B | AUC@3° C | AUC@3° D |
|---|---:|---:|---:|---:|
| TUM     | 0.009 | **0.374** | 0.384 | 0.380 |
| 7Sc     | 0.047 | **0.238** | 0.236 | 0.236 |
| NRGBD   | 0.069 | **0.756** | 0.743 | 0.741 |
| ETH3D   | 0.263 | **0.486** | 0.486 | 0.489 |
| T&T     | 0.047 | **0.128** | 0.129 | 0.134 |
| Oxford  | 0.000 | 0.000 | 0.000 | 0.000 |
| KITTI   | 0.002 | 0.001 | 0.001 | 0.001 |

**TTA gives the AUC@3° win across 5 of 7 datasets.** Oxford and KITTI
remain at 0 because absolute-scale drift saturates the rotation
alignment window.

---

## 5. Reconstruction (F1, Acc, Comp in m)

Only +TTA row shown — SL4 and QV don't change recon (fusion-limited).

| dataset | F1 (normalized ×100) | Acc | Comp |
|---|---:|---:|---:|
| NRGBD | **82.3** (GCA: 64.3) | 0.922 | 0.751 |
| ETH3D | 31.3 (GCA: 98.98)  | 0.437 | 0.290 |
| 7-Scenes | 1.5 (GCA: 80.4) | 0.395 | 0.008 |
| T&T | 0.4 | 0.006 | 0.003 |

**NRGBD is the single dataset where our recon F1 beats GCA.** Others
need a fusion head.

---

## 6. Headline takeaways

1. **TTA is the single biggest win**: −67 % to −87 % ATE on all 4 indoor
   datasets and T&T. AUC@3° jumps 5–10×.
2. **SL(4) PGO helps only on KITTI** (−8 % paper ckpt, −1 % V4). On every
   other dataset it regresses by 30–230 %.
3. **QV loop-closure helps marginally on Oxford (−0.7 %)** and on a
   handful of indoor scenes with real loops (TUM floor −44 %, desk2
   −51 %). On no-loop scenes the z=1.0 threshold still admits noise.
4. **V4-B'' ckpt-1 loses to paper ckpt-10 on every outdoor metric.**
   The overlap-curriculum training on vKITTI2 did not transfer to real
   outdoor geometry.

## 7. Production recipes

| dataset | recipe | why |
|---|---|---|
| TUM, 7-Scenes, NRGBD, ETH3D | `bn=10 TTA=1 PGO=0` | SL4/QV regress on low-drift indoor |
| Oxford | `bn=15 TTA=1 PGO=1 QV z=1.0` paper ckpt | QV captures keble indoor loop; V4 hurts |
| KITTI | `bn=15 TTA=1 PGO=1 QV=0` paper ckpt | SL4 helps; QV hurts on sparse outdoor |
| T&T | `bn=15 TTA=1 PGO=0` paper ckpt | SL4 regresses; no real loops in training scenes |