Update TODO.md: corrected scoring, new roadmap based on V3 results

own-solver/TODO.md

@@ -1,108 +1,109 @@
 # NeuroGolf Solver — Roadmap
 
-> Current: V90 · 400/400 tasks · LB 5322.64 · Target: >6043
-> Philosophy: **Score optimization is now the primary lever.**
+> Current: V3 optimizer · 400/400 tasks · LB 5560.70 · Target: >6043
+> Philosophy: **Reduce intermediate tensor count + weight memory. That's it.**
 > Rule: **NEVER claim a feature works without Kaggle submission confirmation.**
-> Updated: 2026-05-21 — V90 confirmed. 400 tasks submitted, 385 solved + 15 fallback.
+> Updated: 2026-05-22 — V3 confirmed. Merge + stage1 = +238 pts over V90.
 
 ---
 
-## The Gap: 5322 → 6043 = +721 pts needed
+## The Gap: 5560.70 → 6043 = +483 pts needed
 
-Score = log2(1,509,949 / file_size). To gain 721 pts across 400 tasks = avg +1.8 pts/task.
-Each 1 pt gain = halving the file size. So we need to halve file sizes on average.
+Score = max(1.0, 25.0 - ln(memory + params)).
+memory = sum of all intermediate tensor bytes (ORT profiler).
+params = sum of all weight element counts.
 
-Current avg: ~13.3 pts/task. Target avg: ~15.1 pts/task.
++483 pts across 400 tasks = avg +1.2 pts/task.
+Each +1 pt ≈ reducing cost by factor of e (2.718×).
 
 ---
 
-## Score Optimization Plan (Priority Order)
-
-### Phase 1: fp16 weight conversion (est. +50-100 pts)
-- Convert all float32 initializers to float16
-- Add Cast(input, to=FLOAT16) at model entry
-- Halves weight storage for tasks with large float32 weights
-- `fp16_optimizer.py` already exists — run at submission level
-- **Target**: All 23 tasks with >1KB fp32 weights
-
-### Phase 2: Rebuild conv_var with smaller kernels (est. +100-200 pts)
-- conv_var tasks score 8.1-11.2 pts (files 4-90 KB)
-- 17 tasks use conv_var → if each gains 3 pts avg = +51 pts
-- Strategies:
-  - Try kernel_size=5 instead of full-grid kernels
-  - Factored convolutions (depthwise separable)
-  - Iterative small-kernel approach
-  - Prune near-zero weights before saving
-
-### Phase 3: Weight quantization for MatMul tasks (est. +50-100 pts)
-- Tasks with large MatMul weights (182: 462KB, 158: 158KB, 277: 147KB)
-- Low-rank factorization: W = U @ V where U is Mx16, V is 16xN
-- int8 quantization with DequantizeLinear op
-- Could save 75% on largest tasks
-
-### Phase 4: Architecture pruning (est. +30-50 pts)
-- Remove redundant Cast/Pad/Slice chains in generated models
-- Merge consecutive same-type ops
-- Eliminate dead branches (outputs not connected to final output)
-- Strip unnecessary metadata from ONNX protobuf
-
-### Phase 5: Rebuild analytical solvers more compactly (est. +50-100 pts)
-- Many wave1-16 solvers use verbose ONNX graphs
-- Could merge into fewer ops with same semantics
-- Constants sharing across similar operations
-- Use fp16 for all intermediate computations
+## What Worked (V3 vs V90: +238 pts)
+
+1. **Merge best-of-both** (80 tasks from 5743): +200 pts
+   - 5743 has compact 15-70 node models for tasks where V90 had 100+ nodes
+   - Fewer nodes = fewer intermediate tensors = less memory
+   
+2. **Stage 1 trivial optimization** (6 tasks): +38 pts  
+   - transpose → 25 pts (was 14.5)
+   - color_perm → 21 pts (was 14.5)
+   - color_map → 18.8 pts (was 13.3)
 
 ---
 
-## Task Score Distribution (V90)
+## Optimization Plan (Priority Order)
+
+### Phase 1: Reduce intermediate count for ALL tasks [est. +150-250 pts]
+
+**The single biggest lever.** Each intermediate tensor (1,10,30,30) float32 = 36KB memory.
+Removing 3 unnecessary intermediates from 200 tasks = +0.8 pts × 200 = +160 pts.
+
+Techniques:
+- **OneHot replacement**: Replace 10×(Equal+Cast)+Concat with single OneHot op
+  - Saves 9-19 intermediate tensors per conv/analytical task
+  - Applies to: conv_var (18), conv_fixed (16), many wave22 tasks
+- **Direct output pattern**: Ensure last node outputs directly to graph 'output'
+- **Fuse Slice+Pad chains**: Consecutive Slice or Pad operations → merge into one
+- **Remove dead value_info entries**: Some models have unused intermediate declarations
+
+### Phase 2: fp16 weights WITHOUT Cast node [est. +50-100 pts]
+
+Convert float32 initializers to float16 but do NOT add Cast(input→fp16).
+The Cast creates a new intermediate (1,10,30,30) fp16 = 18KB — often MORE than the savings.
+
+Better approach: Store weights as fp16, let ORT handle mixed-precision inference.
+OR: Only add Cast when weight savings > 18KB (i.e., weights > 36KB float32).
 
-| Score Range | Tasks | Example | Action |
-|-------------|-------|---------|--------|
-| 15-16+ pts | ~80 | transpose (127B), rotate (588B) | Already optimal |
-| 13-15 pts | ~150 | Most analytical solvers | Minor gains possible |
-| 11-13 pts | ~100 | conv_fixed, medium wave22 | fp16 + pruning |
-| 8-11 pts | ~50 | conv_var, large wave22 | Major rebuild needed |
-| 1-8 pts | ~20 | Huge MatMul, conv_var large | Quantize/factorize |
+Target: 60+ tasks with >10KB weights.
+
+### Phase 3: SVD factorization for large weights [est. +20-50 pts]
+
+Tasks with 50-462 KB weight matrices:
+- 182 (462KB), 158 (158KB), 277 (147KB), 387 (101KB), 285 (99KB), 324 (93KB)
+- Factor W(M×N) → U(M×R) @ V(R×N), validate at each R
+
+### Phase 4: Sign-corrected Conv for conv_var [est. +8 pts]
+
+Already working for 6 tasks (127, 171, 220, 230, 258, 261).
+Replaces 15-node architecture with 3 nodes.
+Upload `sign_corrected_conv.py` to solver pipeline.
+
+### Phase 5: Solve more tasks with compact models [est. +50-100 pts]
+
+15 unsolved tasks currently use fallback reference models.
+Some could potentially be solved with custom compact solvers.
+Also: tasks currently using bloated approaches might have simpler solutions.
 
 ---
 
-## Quick Wins (implement first)
+## Quick Wins (implement next)
 
-1. **Run fp16_optimizer.py on full submission** — instant gains, already coded
-2. **Strip unused initializers** from all generated ONNX models
-3. **Reduce conv_var time budget** to prefer smaller kernels
-4. **Rebuild task 239** without TopK to avoid profiler warning
+1. ☐ **OneHot op replacement** in conv.py — replace `add_onehot_block()` with single OneHot node
+2. ☐ **fp16 weights without Cast** — modify fp16_optimizer to skip the Cast node
+3. ☐ **Integrate sign_corrected_conv** into solve_conv_variable as first attempt
+4. ☐ **Run optimizer on V91** (once V91 has above fixes integrated)
 
 ---
 
-## Remaining 15 "Fallback" Tasks
-
-These use reference models directly (not our solvers). They score based on file size only.
-Improving them requires either:
-- Building custom solvers from scratch (hard, unique puzzles)
-- Making the reference models smaller (fp16 conversion)
-
-| Task | Current Size | Potential fp16 Size | Score Gain |
-|------|-------------|--------------------:|-----------|
-| 133 | 287,675 B | ~144 KB | +1.0 |
-| 21 | 29,463 B | ~15 KB | +1.0 |
-| 80 | 15,132 B | ~8 KB | +0.9 |
-| 208 | 14,111 B | ~7 KB | +1.0 |
-| 323 | 11,147 B | ~6 KB | +0.9 |
-| 379 | 9,267 B | ~5 KB | +0.9 |
-| 5 | 8,024 B | ~4 KB | +0.9 |
-| 234 | 7,484 B | ~4 KB | +0.9 |
-| 397 | 6,194 B | ~3 KB | +0.9 |
-| Others | <5 KB | ~2.5 KB | +0.8 each |
+## Task Score Distribution (V3, estimated)
+
+| Score Range | Tasks | Optimization Target |
+|-------------|-------|-------------------|
+| 18-25 pts | ~6 | Already optimal (trivial tasks) |
+| 14-16 pts | ~50 | Minor gains from fp16/node reduction |
+| 13-14 pts | ~200 | Phase 1 target: reduce intermediates |
+| 12-13 pts | ~100 | Phase 1+2: intermediates + fp16 |
+| 10-12 pts | ~30 | Phase 2+3: fp16 + SVD for large weights |
+| <10 pts | ~14 | Fallback tasks, hard to improve |
 
 ---
 
 ## Research Queue
 
-1. [x] Wave 22: 143 tasks (complete)
-2. [x] Fallback zip for 15 unsolved (complete)
-3. [ ] fp16 weight optimization pass
-4. [ ] conv_var kernel size reduction
-5. [ ] MatMul weight factorization
-6. [ ] Architecture pruning pass
-7. [ ] Custom solvers for near-miss tasks (21, 80, 184, 202)
+1. [x] Merge best-of-both (V3: +238 pts)
+2. [x] Stage 1 trivial optimization (6 tasks)
+3. [x] Sign-corrected Conv (6 conv_var tasks)
+4. [ ] OneHot op replacement (conv + analytical tasks)
+5. [ ] fp16 weights without Cast overhead
+6. [ ] SVD factorization for large weights
+7. [ ] Solve/improve fallback tasks