Spaces:
Running
Running
| # DSPy: MIPROv2 and GEPA on the cited-answer step | |
| **Result: neither optimiser beat the hand-written prompt on held-out test by more | |
| than the noise.** MIPROv2 is +3.7 points [−2.5, +9.8]; GEPA is −3.7 [−10.1, +2.6]. | |
| Both improved the validation score they optimised against, and GEPA's gain did not | |
| transfer. | |
| Raw results: [`dspy_test.json`](https://github.com/nihardamera/RegRadar/blob/main/eval/results/dspy_test.json) (per-question test scores, so every | |
| interval here can be recomputed); programs: [`../dspy/`](https://github.com/nihardamera/RegRadar/blob/main/eval/dspy); code: | |
| [`regradar/optimize.py`](https://github.com/nihardamera/RegRadar/blob/main/regradar/optimize.py). | |
| ## Setup | |
| - **Program:** answer a golden question from its five retrieved chunks (section / | |
| hybrid, fixed per question in `eval/dspy/retrieval.json`) with claims that cite | |
| them; the citation checks of `regradar/answer.py` decide whether it stands (one | |
| retry with the problems, then withheld). Only the instructions and demonstrations | |
| differ between programs. | |
| - **Metric (no judge):** an answerable question scores the mean of citation recall | |
| and citation precision against the gold spans, and 0 if declined or withheld; | |
| declining scores 1 when the question is unanswerable or retrieval missed the | |
| answer. It also returns feedback in words (parts missed, which chunk had them, | |
| which check failed), which GEPA reads. | |
| - **Data:** 60 dev questions split 32 train / 28 validation by a hash of the ID; | |
| **90 test questions scored once per program at the end.** Never used for | |
| optimising. | |
| - **Models:** task Haiku 5.5 (effort low); MIPROv2's proposer and GEPA's reflection | |
| model Sonnet 5.5 (effort medium). MIPROv2 `auto="light"`, up to 2 bootstrapped | |
| demonstrations; GEPA 200 metric calls. All calls through the project's cached, | |
| capped client. | |
| ## Results | |
| | Program | Validation (n = 28) | **Test (n = 90)** | Test − baseline (paired) | | |
| | --- | --- | --- | --- | | |
| | Baseline: the hand-written prompt | 0.71 [0.56, 0.85] | **0.80 [0.73, 0.86]** | | | |
| | MIPROv2 (new instructions + 2 demonstrations) | 0.83 [0.69, 0.93] | **0.84 [0.78, 0.90]** | +3.7 [−2.5, +9.8] | | |
| | GEPA (rewritten instructions) | 0.83 [0.72, 0.93] | **0.77 [0.69, 0.84]** | −3.7 [−10.1, +2.6] | | |
| Scores are means of the per-question metric (0 to 1); differences in points. | |
| 95% percentile-bootstrap intervals, 10,000 resamples. | |
| ## Cost | |
| | | USD | | |
| | --- | --- | | |
| | MIPROv2 optimisation (Sonnet proposals, Haiku trials; includes a first attempt that stopped on a missing dependency, whose calls the second attempt replayed from cache) | 0.69 | | |
| | GEPA optimisation | 0.50 | | |
| | Scoring three programs on validation and test | 0.24 | | |
| | **Total** | **1.43** | | |
| ## What it says | |
| 1. **28 validation questions are too few to optimise a prompt against.** GEPA's | |
| instructions grew from 90 to 693 words and quote specific misses from its | |
| training questions ("Past misses: a question asked what is required of every | |
| digital payment..."). That is fitting the examples, and test fell. | |
| 2. **MIPROv2 points the right way but is within noise.** Its instructions add a | |
| step-by-step procedure and checks for near-duplicate and superseded documents, | |
| plus two demonstrations. The paired interval's half-width is about 6 points at | |
| n = 90; confirming +3.7 would take roughly 250 test questions at this variance. | |
| 3. **The hand-written prompt was already shaped by the same failures** (the number | |
| check, the not-found rule), and the code checks catch the worst outputs whatever | |
| the prompt. There was less left to find than in the papers' benchmarks. | |
| 4. The baseline here (0.80) is the DSPy port of the production prompt, not the | |
| production run itself: the adapter adds its own field markers, so its outputs | |
| differ slightly from `answers_test_*.json`. | |