org card: per-experiment hyperparameters and VQA/caption results

README.md

@@ -9,7 +9,7 @@ pinned: false
 
 # Aspaklarya — Qwen3-VL-8B for remote-sensing VQA + captioning
 
-LoRA finetunes of **Qwen3-VL-8B-Instruct**, trained jointly on **visual question answering** and **image captioning** over overhead imagery. 15 experiments, 16 evaluated checkpoints. All model repos are private.
+LoRA finetunes of **Qwen3-VL-8B-Instruct**, trained jointly on **visual question answering** and **image captioning** over overhead imagery. 17 experiments, 17 evaluated recipes. All model repos are private.
 
 ## Setup
 
@@ -45,8 +45,10 @@ Winners are merged into the bf16 base and re-validated: adapters are evaluated o
 | Aspaklarya-013 | 16 / 16 | 0.0 | 1e-4 | linear | trained | 1c + 2q | 18,864 | low LR, long schedule |
 | Aspaklarya-014 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | effective batch 16 |
 | Aspaklarya-015 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | effective batch 64 |
+| Aspaklarya-016 | 16 / 16 | 0.0 | **4e-4** | linear | trained | 1c + 2q | 18,864 | batch 64 + linear LR scaling |
+| Aspaklarya-017 | 16 / 16 | 0.0 | **3e-4** | linear | trained | 1c + 2q | 18,864 | batch 64 + sqrt LR scaling (running) |
 
-Constant everywhere: effective batch 32 (except 014/015), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.
+Constant everywhere: effective batch 32 (except 014 at 16, 015-017 at 64), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.
 
 ## Results
 
@@ -74,6 +76,18 @@ Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with boo
 | Qwen3-VL-8B (base) | 68.37 ±0.89 | 58.94 ±1.50 | 63.66 ±0.92 | 65.68 ±0.66 | 56.01 ±1.01 | 60.84 ±0.62 | 63.07 ±0.94 | 53.18 ±1.46 | 58.13 ±0.91 |
 
 Every finetune beats both references overall: **+17.1** over the base model and **+8.7** over the in-house reference on the full set.
+## Learning rate at large batch
+
+Aspaklarya-015 gave the best captioning in the table at effective batch 64, but it was trained at 2e-4 — the rate tuned for batch 32. Two follow-ups tested whether the batch-64 recipe was simply under-stepped. Scores below come from a separate 500-image draw used for live checkpoint selection, so they compare only with each other.
+
+| LR at batch 64 | ep 1 | ep 2 | ep 3 | ep 4 | ep 5 | Outcome |
+|---|---|---|---|---|---|---|
+| 2e-4 (015) | 76.92 | 77.03 | 77.25 | 77.64 | **77.92** | peak at epoch 5 |
+| 4e-4 (016) | **77.67** | 76.93 | 76.42 | 76.89 | — | stopped at epoch 4, declining |
+| 3e-4 (017) | running | | | | | |
+
+Linear LR scaling is **wrong** for LoRA here: 4e-4 peaked at epoch 1 and then lost ground at every later checkpoint, ending below where 2e-4 started. The adapter's effective step size does not follow the full-finetune scaling rule, so raising LR with batch size overshoots rather than compensating.
+
 ## Error bars
 
 Bootstrap over images, 3000 resamples. Absolute standard error:
@@ -92,7 +106,7 @@ The top eight runs are statistically tied: every confidence interval against the
 
 ## Findings
 
-- **Optimization is saturated.** LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64) and schedule were all swept. None beat the noise floor.
+- **Optimization is saturated.** LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64) and LR scaling at large batch were all swept. None beat the noise floor.
 - **The slices disagree.** Ranking by in-domain score almost inverts the RSICD ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.
 - **Captioning is the real gain.** It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.
 - **The in-domain VQA win is not established.** The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.

index.html

@@ -38,7 +38,7 @@ a { color: #0969da; }
 <body>
 <main>
 <h1>Aspaklarya — Qwen3-VL-8B for remote-sensing VQA + captioning</h1>
-<p>LoRA finetunes of <strong>Qwen3-VL-8B-Instruct</strong>, trained jointly on <strong>visual question answering</strong> and <strong>image captioning</strong> over overhead imagery. 15 experiments, 16 evaluated checkpoints. All model repos are private.</p>
+<p>LoRA finetunes of <strong>Qwen3-VL-8B-Instruct</strong>, trained jointly on <strong>visual question answering</strong> and <strong>image captioning</strong> over overhead imagery. 17 experiments, 17 evaluated recipes. All model repos are private.</p>
 <h2>Setup</h2>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
 <th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Item</th>
@@ -81,8 +81,10 @@ a { color: #0969da; }
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-013</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1e-4</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">low LR, long schedule</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-014</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">2e-4</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">effective batch 16</td></tr>
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-015</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">2e-4</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">effective batch 64</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-016</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>4e-4</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">batch 64 + linear LR scaling</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-017</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>3e-4</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">batch 64 + sqrt LR scaling (running)</td></tr>
 </tbody></table>
-<p>Constant everywhere: effective batch 32 (except 014/015), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.</p>
+<p>Constant everywhere: effective batch 32 (except 014 at 16, 015-017 at 64), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.</p>
 <h2>Results</h2>
 <p>Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with bootstrap standard error. <strong>Overall</strong> = mean of caption and VQA. <strong>In-dom.</strong> = the target overhead-imagery subset (491 images); <strong>RSICD</strong> = the public RSICD remote-sensing benchmark subset (509 images). Sorted by in-domain overall; <strong>bold marks the best value in each column</strong>.</p>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
@@ -117,6 +119,22 @@ a { color: #0969da; }
 <tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Qwen3-VL-8B (base)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.37 ±0.89</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">58.94 ±1.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">63.66 ±0.92</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">65.68 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">56.01 ±1.01</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">60.84 ±0.62</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">63.07 ±0.94</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">53.18 ±1.46</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">58.13 ±0.91</td></tr>
 </tbody></table>
 <p>Every finetune beats both references overall: <strong>+17.1</strong> over the base model and <strong>+8.7</strong> over the in-house reference on the full set.</p>
+<h2>Learning rate at large batch</h2>
+<p>Aspaklarya-015 gave the best captioning in the table at effective batch 64, but it was trained at 2e-4 — the rate tuned for batch 32. Two follow-ups tested whether the batch-64 recipe was simply under-stepped. Scores below come from a separate 500-image draw used for live checkpoint selection, so they compare only with each other.</p>
+<table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">LR at batch 64</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">ep 1</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">ep 2</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">ep 3</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">ep 4</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">ep 5</th>
+<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">Outcome</th>
+</tr></thead><tbody>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">2e-4 (015)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">76.92</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.03</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.25</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>77.92</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">peak at epoch 5</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">4e-4 (016)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>77.67</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">76.93</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">76.42</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">76.89</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">—</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">stopped at epoch 4, declining</td></tr>
+<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">3e-4 (017)</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">running</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"></td></tr>
+</tbody></table>
+<p>Linear LR scaling is <strong>wrong</strong> for LoRA here: 4e-4 peaked at epoch 1 and then lost ground at every later checkpoint, ending below where 2e-4 started. The adapter&#x27;s effective step size does not follow the full-finetune scaling rule, so raising LR with batch size overshoots rather than compensating.</p>
 <h2>Error bars</h2>
 <p>Bootstrap over images, 3000 resamples. Absolute standard error:</p>
 <table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
@@ -134,7 +152,7 @@ a { color: #0969da; }
 <p>The top eight runs are statistically tied: every confidence interval against the leader straddles zero. <strong>VQA error is ~2.3x caption error</strong>, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps and the gaps to the two references are real. Doubling the eval set from 500 to 1000 images did not separate the field, and it reshuffled the order.</p>
 <h2>Findings</h2>
 <ul>
-<li><strong>Optimization is saturated.</strong> LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64) and schedule were all swept. None beat the noise floor.</li>
+<li><strong>Optimization is saturated.</strong> LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64) and LR scaling at large batch were all swept. None beat the noise floor.</li>
 <li><strong>The slices disagree.</strong> Ranking by in-domain score almost inverts the RSICD ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.</li>
 <li><strong>Captioning is the real gain.</strong> It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.</li>
 <li><strong>The in-domain VQA win is not established.</strong> The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.</li>