org card: per-experiment hyperparameters and VQA/caption results

README.md

@@ -13,19 +13,43 @@ LoRA finetunes of **Qwen3-VL-8B-Instruct**, trained jointly on **visual
 question answering** and **image captioning** over overhead imagery. All model
 repos are private.
 
-The programme has run in **two rounds against two different answer styles**.
-They are not comparable to each other and are not scored on the same
-benchmark.
+## Start here
 
-| | Round 0XX | Round 1XX |
-|---|---|---|
-| runs | Aspaklarya-001 ... 021 (21, 20 evaluated) | Aspaklarya-101 ... 107 |
-| VQA column | `VQA` | `VQA_1126` |
-| reference answers | ~4 characters (`yes`, `3`, `river`) | ~300 characters, grounded reasoning |
-| model output | one word | 270-295 characters |
-| benchmark | 1,000-image union draw, 2,000 graded items | four-slice seed-1105 draw, 1,991 graded items |
-| best | Aspaklarya-020-step4500 | Aspaklarya-101-step4500 |
-| what it answered | hyperparameters are saturated, data is the lever | verbose supervision is worth +8 on DM VQA; data is still the only lever |
+| | |
+|---|---|
+| **best in-domain model** | `Aspaklarya-101-step4500` - MEAN4 **76.72** |
+| **best general model** | `Aspaklarya-020-step4500` - MEAN4 74.90, but wins 3 slices of 4 |
+| **best free option** | `Aspaklarya-106-soup` - MEAN4 76.36, no training cost |
+| **confirmed levers** | exactly one: **corpus size**, and specifically *more images* |
+| **closed levers** | LoRA rank, LoRA scale, caption loss weight, LR schedule, vision-tower freezing, decoding strategy, input resolution, QA depth, question paraphrasing, hard negatives, teacher distillation, curriculum ordering |
+| **noise floor** | MEAN4 sigma **0.31**; per-slice 0.88 / 0.74 / 0.25 / 0.52 |
+| **current ceiling** | 76.72, unmoved since round 1XX |
+
+**Nothing has beaten `101-step4500` since it was trained.** Three rounds and
+roughly 30 experiments later it is still the reference, and this page is mostly
+a record of what does *not* work. Two published results have been **retracted**
+after control errors were found; both retractions are kept visible below rather
+than edited away.
+
+The programme has run in **three rounds**. Rounds 0XX and 1XX trained against
+two different answer styles and are not scored on the same VQA benchmark;
+round 11X is a set of probes against a replicated control.
+
+| | Round 0XX | Round 1XX | Round 11X |
+|---|---|---|---|
+| runs | Aspaklarya-001 ... 021 (21, 20 evaluated) | Aspaklarya-101 ... 111 | exp112 ... 120 + 3 controls |
+| VQA column | `VQA` | `VQA_1126` | `VQA_1126` |
+| reference answers | ~4 characters (`yes`, `3`, `river`) | ~300 characters, grounded reasoning | as 1XX |
+| model output | one word | 270-295 characters | as 1XX |
+| benchmark | 1,000-image union draw, 2,000 graded items | four-slice seed-1105 draw, 1,991 graded items | same four-slice draw |
+| budget | full (4 epochs) | full (4 epochs) | short probe (1 epoch, steps 562/1124) |
+| best | Aspaklarya-020-step4500 | Aspaklarya-101-step4500 | **none - no arm beat the control** |
+| repos published | yes | yes | **none** |
+| what it answered | hyperparameters are saturated, data is the lever | verbose supervision is worth +8 on DM VQA; data is still the only lever | six more axes closed; the noise floor was measured properly for the first time |
+
+**Round 11X is at the bottom of this page** and is the most recent work. If you
+only read one section, read that one: it contains the sigma measurement that
+determines which differences on this page are real.
 
 Both rounds are now scored on the **same four-slice benchmark**, so the
 MEAN4 column below is comparable across rounds. The *legacy* and *new VQA*

index.html

@@ -39,20 +39,38 @@ a { color: #0969da; }
 <main>
 <h1>Aspaklarya - Qwen3-VL-8B for remote-sensing VQA + captioning</h1>
 <p>LoRA finetunes of <strong>Qwen3-VL-8B-Instruct</strong>, trained jointly on <strong>visual question answering</strong> and <strong>image captioning</strong> over overhead imagery. All model repos are private.</p>
-<p>The programme has run in <strong>two rounds against two different answer styles</strong>. They are not comparable to each other and are not scored on the same benchmark.</p>
+<h2>Start here</h2>
+<table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap"></th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap"></th>
+</tr></thead><tbody>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>best in-domain model</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><code>Aspaklarya-101-step4500</code> - MEAN4 <strong>76.72</strong></td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>best general model</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><code>Aspaklarya-020-step4500</code> - MEAN4 74.90, but wins 3 slices of 4</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>best free option</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><code>Aspaklarya-106-soup</code> - MEAN4 76.36, no training cost</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>confirmed levers</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">exactly one: <strong>corpus size</strong>, and specifically <em>more images</em></td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>closed levers</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">LoRA rank, LoRA scale, caption loss weight, LR schedule, vision-tower freezing, decoding strategy, input resolution, QA depth, question paraphrasing, hard negatives, teacher distillation, curriculum ordering</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>noise floor</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">MEAN4 sigma <strong>0.31</strong>; per-slice 0.88 / 0.74 / 0.25 / 0.52</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>current ceiling</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">76.72, unmoved since round 1XX</td></tr>
+</tbody></table>
+<p><strong>Nothing has beaten <code>101-step4500</code> since it was trained.</strong> Three rounds and roughly 30 experiments later it is still the reference, and this page is mostly a record of what does <em>not</em> work. Two published results have been <strong>retracted</strong> after control errors were found; both retractions are kept visible below rather than edited away.</p>
+<p>The programme has run in <strong>three rounds</strong>. Rounds 0XX and 1XX trained against two different answer styles and are not scored on the same VQA benchmark; round 11X is a set of probes against a replicated control.</p>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap"></th>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Round 0XX</th>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Round 1XX</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Round 11X</th>
 </tr></thead><tbody>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">runs</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-001 ... 021 (21, 20 evaluated)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-101 ... 107</td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">VQA column</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><code>VQA</code></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><code>VQA_1126</code></td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">reference answers</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">~4 characters (<code>yes</code>, <code>3</code>, <code>river</code>)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">~300 characters, grounded reasoning</td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">model output</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">one word</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">270-295 characters</td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">benchmark</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1,000-image union draw, 2,000 graded items</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">four-slice seed-1105 draw, 1,991 graded items</td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">best</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-020-step4500</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-101-step4500</td></tr>
-<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">what it answered</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">hyperparameters are saturated, data is the lever</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">verbose supervision is worth +8 on DM VQA; data is still the only lever</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">runs</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-001 ... 021 (21, 20 evaluated)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-101 ... 111</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">exp112 ... 120 + 3 controls</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">VQA column</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><code>VQA</code></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><code>VQA_1126</code></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><code>VQA_1126</code></td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">reference answers</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">~4 characters (<code>yes</code>, <code>3</code>, <code>river</code>)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">~300 characters, grounded reasoning</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">as 1XX</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">model output</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">one word</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">270-295 characters</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">as 1XX</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">benchmark</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1,000-image union draw, 2,000 graded items</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">four-slice seed-1105 draw, 1,991 graded items</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">same four-slice draw</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">budget</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">full (4 epochs)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">full (4 epochs)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">short probe (1 epoch, steps 562/1124)</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">best</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-020-step4500</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-101-step4500</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>none - no arm beat the control</strong></td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">repos published</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">yes</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">yes</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>none</strong></td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">what it answered</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">hyperparameters are saturated, data is the lever</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">verbose supervision is worth +8 on DM VQA; data is still the only lever</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">six more axes closed; the noise floor was measured properly for the first time</td></tr>
 </tbody></table>
+<p><strong>Round 11X is at the bottom of this page</strong> and is the most recent work. If you only read one section, read that one: it contains the sigma measurement that determines which differences on this page are real.</p>
 <p>Both rounds are now scored on the <strong>same four-slice benchmark</strong>, so the MEAN4 column below is comparable across rounds. The <em>legacy</em> and <em>new VQA</em> columns individually are not: The two benchmarks reward opposite answer styles; see <em>Why the legacy score drops</em> below.</p>
 <h2>Round 0XX - terse answers (Aspaklarya-001 ... 021)</h2>
 <p>The original 21-experiment programme, trained on the one-word <code>VQA</code> column and scored on the 1,000-image union draw. Everything below this heading refers to that round only.</p>