Spaces:
Running
Running
org card: per-experiment hyperparameters and VQA/caption results
Browse files- README.md +29 -5
- index.html +26 -5
README.md
CHANGED
|
@@ -47,6 +47,8 @@ Winners are merged into the bf16 base and re-validated: adapters are evaluated o
|
|
| 47 |
| Aspaklarya-015 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | effective batch 64 |
|
| 48 |
| Aspaklarya-016 | 16 / 16 | 0.0 | **4e-4** | linear | trained | 1c + 2q | 18,864 | batch 64 + linear LR scaling |
|
| 49 |
| Aspaklarya-017 | 16 / 16 | 0.0 | **3e-4** | linear | trained | 1c + 2q | 18,864 | batch 64 + sqrt LR scaling |
|
|
|
|
|
|
|
| 50 |
|
| 51 |
Constant everywhere: effective batch 32 (except 014 at 16, 015-017 at 64), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.
|
| 52 |
|
|
@@ -57,8 +59,10 @@ Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with boo
|
|
| 57 |
| Model | In-dom. cap | In-dom. vqa | In-dom. overall | All cap | All vqa | All overall | RSICD cap | RSICD vqa | RSICD overall |
|
| 58 |
|---|---|---|---|---|---|---|---|---|---|
|
| 59 |
| Aspaklarya-011 | 81.85 ±0.66 | 83.51 ±1.48 | **82.68 ±0.84** | 79.92 ±0.49 | 75.78 ±1.13 | 77.85 ±0.64 | 78.06 ±0.67 | 68.33 ±1.64 | 73.20 ±0.92 |
|
| 60 |
-
| Aspaklarya-
|
| 61 |
-
| Aspaklarya-
|
|
|
|
|
|
|
| 62 |
| Aspaklarya-006 | 81.40 ±0.71 | 82.95 ±1.50 | 82.18 ±0.90 | 79.81 ±0.50 | 75.79 ±1.12 | 77.80 ±0.65 | 78.26 ±0.68 | 68.88 ±1.63 | 73.57 ±0.91 |
|
| 63 |
| Aspaklarya-005 | 81.37 ±0.67 | 82.37 ±1.50 | 81.87 ±0.85 | 79.55 ±0.49 | 75.74 ±1.11 | 77.64 ±0.62 | 77.79 ±0.71 | 69.35 ±1.65 | 73.57 ±0.93 |
|
| 64 |
| Aspaklarya-012 | 81.68 ±0.66 | 82.05 ±1.53 | 81.86 ±0.86 | 79.73 ±0.48 | 75.02 ±1.13 | 77.37 ±0.64 | 77.85 ±0.71 | 68.23 ±1.63 | 73.04 ±0.90 |
|
|
@@ -109,19 +113,39 @@ The top eight runs are statistically tied: every confidence interval against the
|
|
| 109 |
|
| 110 |
## Findings
|
| 111 |
|
| 112 |
-
- **Optimization is saturated.** LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64)
|
| 113 |
- **The slices disagree.** Ranking by in-domain score almost inverts the RSICD ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.
|
| 114 |
- **Captioning is the real gain.** It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.
|
| 115 |
- **The in-domain VQA win is not established.** The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.
|
| 116 |
- **The vision tower matters.** Freezing it is the one ablation that consistently hurt.
|
| 117 |
- **Extra QA per image does not help.** 4-QA and 6-QA mixtures tie with 2-QA.
|
| 118 |
- **Larger batch trades generality for domain fit.** Effective batch 64 gives the best in-domain captioning in the table and the worst RSICD VQA.
|
|
|
|
| 119 |
- **Data is the remaining lever**, not hyperparameters.
|
| 120 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
## Status
|
| 122 |
|
| 123 |
-
The hyperparameter search is **closed**. Every axis available in this setup has been swept — rank, alpha, dropout, learning rate, schedule, weight decay, gradient clipping, vision-tower freezing, QA-per-image, caption prompt pool, batch geometry, effective batch size,
|
|
|
|
|
|
|
| 124 |
|
| 125 |
-
|
| 126 |
|
| 127 |
Any further gain has to come from the training corpus: more in-domain imagery, denser or higher-quality question-answer pairs, or a different caption source.
|
|
|
|
| 47 |
| Aspaklarya-015 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | effective batch 64 |
|
| 48 |
| Aspaklarya-016 | 16 / 16 | 0.0 | **4e-4** | linear | trained | 1c + 2q | 18,864 | batch 64 + linear LR scaling |
|
| 49 |
| Aspaklarya-017 | 16 / 16 | 0.0 | **3e-4** | linear | trained | 1c + 2q | 18,864 | batch 64 + sqrt LR scaling |
|
| 50 |
+
| Aspaklarya-018 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | **bf16 base** (no 4-bit) |
|
| 51 |
+
| Aspaklarya-019 | 16 / 16 | 0.05 | 2e-4 | linear | trained | 1c + 2q | 18,864 | **bf16 base** + 011 recipe |
|
| 52 |
|
| 53 |
Constant everywhere: effective batch 32 (except 014 at 16, 015-017 at 64), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.
|
| 54 |
|
|
|
|
| 59 |
| Model | In-dom. cap | In-dom. vqa | In-dom. overall | All cap | All vqa | All overall | RSICD cap | RSICD vqa | RSICD overall |
|
| 60 |
|---|---|---|---|---|---|---|---|---|---|
|
| 61 |
| Aspaklarya-011 | 81.85 ±0.66 | 83.51 ±1.48 | **82.68 ±0.84** | 79.92 ±0.49 | 75.78 ±1.13 | 77.85 ±0.64 | 78.06 ±0.67 | 68.33 ±1.64 | 73.20 ±0.92 |
|
| 62 |
+
| Aspaklarya-019 | **82.39 ±0.65** | 82.82 ±1.48 | 82.61 ±0.85 | 80.33 ±0.48 | 75.91 ±1.13 | 78.12 ±0.63 | 78.34 ±0.70 | 69.24 ±1.62 | 73.79 ±0.89 |
|
| 63 |
+
| Aspaklarya-001 | 81.84 ±0.66 | 83.23 ±1.44 | 82.54 ±0.81 | 79.97 ±0.48 | 75.94 ±1.12 | 77.95 ±0.63 | 78.17 ±0.69 | 68.89 ±1.66 | 73.53 ±0.93 |
|
| 64 |
+
| Aspaklarya-018 | 82.28 ±0.69 | 82.70 ±1.51 | 82.49 ±0.86 | **80.46 ±0.49** | 75.80 ±1.11 | **78.13 ±0.63** | 78.70 ±0.67 | 69.15 ±1.61 | 73.92 ±0.90 |
|
| 65 |
+
| Aspaklarya-015 | 82.14 ±0.64 | 82.76 ±1.46 | 82.45 ±0.82 | 80.07 ±0.49 | 74.73 ±1.13 | 77.40 ±0.63 | 78.07 ±0.72 | 66.98 ±1.65 | 72.52 ±0.92 |
|
| 66 |
| Aspaklarya-006 | 81.40 ±0.71 | 82.95 ±1.50 | 82.18 ±0.90 | 79.81 ±0.50 | 75.79 ±1.12 | 77.80 ±0.65 | 78.26 ±0.68 | 68.88 ±1.63 | 73.57 ±0.91 |
|
| 67 |
| Aspaklarya-005 | 81.37 ±0.67 | 82.37 ±1.50 | 81.87 ±0.85 | 79.55 ±0.49 | 75.74 ±1.11 | 77.64 ±0.62 | 77.79 ±0.71 | 69.35 ±1.65 | 73.57 ±0.93 |
|
| 68 |
| Aspaklarya-012 | 81.68 ±0.66 | 82.05 ±1.53 | 81.86 ±0.86 | 79.73 ±0.48 | 75.02 ±1.13 | 77.37 ±0.64 | 77.85 ±0.71 | 68.23 ±1.63 | 73.04 ±0.90 |
|
|
|
|
| 113 |
|
| 114 |
## Findings
|
| 115 |
|
| 116 |
+
- **Optimization is saturated.** LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64), LR scaling at large batch and base-weight precision were all swept. None beat the noise floor.
|
| 117 |
- **The slices disagree.** Ranking by in-domain score almost inverts the RSICD ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.
|
| 118 |
- **Captioning is the real gain.** It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.
|
| 119 |
- **The in-domain VQA win is not established.** The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.
|
| 120 |
- **The vision tower matters.** Freezing it is the one ablation that consistently hurt.
|
| 121 |
- **Extra QA per image does not help.** 4-QA and 6-QA mixtures tie with 2-QA.
|
| 122 |
- **Larger batch trades generality for domain fit.** Effective batch 64 gives the best in-domain captioning in the table and the worst RSICD VQA.
|
| 123 |
+
- **Training in bf16 instead of 4-bit is a small, consistent positive** — the only change in the whole sweep that never loses on any slice — but it is still inside the error bar.
|
| 124 |
- **Data is the remaining lever**, not hyperparameters.
|
| 125 |
|
| 126 |
+
## Base-weight precision
|
| 127 |
+
|
| 128 |
+
Runs 001-017 all trained their adapters against **4-bit quantized** base weights, while evaluation and the final merge both use the **bf16** base. That is a train/serve mismatch: the adapter was fitted to compensate for quantization error that is absent at inference time.
|
| 129 |
+
|
| 130 |
+
Runs **018** and **019** repeat the two strongest recipes with the base loaded in bf16 and nothing else changed. Against the reference, paired on 1000 images:
|
| 131 |
+
|
| 132 |
+
| run | overall | in-domain | RSICD |
|
| 133 |
+
|---|---|---|---|
|
| 134 |
+
| Aspaklarya-018 | **+0.17 ±0.42** | -0.05 ±0.59 | **+0.39 ±0.59** |
|
| 135 |
+
| Aspaklarya-019 | **+0.16 ±0.40** | **+0.07 ±0.57** | **+0.25 ±0.57** |
|
| 136 |
+
|
| 137 |
+
These are the first two runs in the sweep to post a positive overall delta, and **018** is the only run that is not worse than the reference on any slice. Both are still well inside the paired error bar (p = 0.34 that each is in fact worse), so neither is promoted.
|
| 138 |
+
|
| 139 |
+
The mechanism is not the expected one. Removing quantization error should mostly help the harder task, VQA; instead the gain is **entirely in captioning** (+0.48 and +0.35) while VQA is flat to slightly negative. That is the same caption-up / VQA-flat signature produced by every other lever in the sweep, which suggests precision is acting through the same saturated channel rather than unlocking anything new.
|
| 140 |
+
|
| 141 |
+
Removing quantization also costs **~9 GB of GPU memory** during training (19 GB vs 10 GB), which is a real constraint on a shared 40 GB card.
|
| 142 |
+
|
| 143 |
## Status
|
| 144 |
|
| 145 |
+
The hyperparameter search is **closed**. Every axis available in this setup has been swept — rank, alpha, dropout, learning rate, schedule, weight decay, gradient clipping, vision-tower freezing, QA-per-image, caption prompt pool, batch geometry, effective batch size, learning rate at large batch, and base-weight precision. Nineteen recipes were trained and evaluated under identical conditions, and none displaced the reference recipe by more than the paired error bar.
|
| 146 |
+
|
| 147 |
+
Batch size and learning rate illustrate the ceiling. Effective batch 16 came in at **-0.59** overall, batch 64 at 2e-4 at **-0.56**, and batch 64 at 3e-4 at **-0.34** overall and **-0.87** in-domain; batch 64 at 4e-4 was stopped at epoch 4 while declining. In each case captioning held steady or improved while VQA fell, which is the same trade seen across the whole sweep.
|
| 148 |
|
| 149 |
+
Base precision is the one axis that points the right way (**+0.17** overall, never negative on a slice) without reaching significance. It is the natural thing to re-test if the training corpus is ever enlarged, since a larger corpus would shrink the error bar that currently hides it.
|
| 150 |
|
| 151 |
Any further gain has to come from the training corpus: more in-domain imagery, denser or higher-quality question-answer pairs, or a different caption source.
|
index.html
CHANGED
|
@@ -83,6 +83,8 @@ a { color: #0969da; }
|
|
| 83 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-015</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">2e-4</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">effective batch 64</td></tr>
|
| 84 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-016</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>4e-4</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">batch 64 + linear LR scaling</td></tr>
|
| 85 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-017</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>3e-4</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">batch 64 + sqrt LR scaling</td></tr>
|
|
|
|
|
|
|
| 86 |
</tbody></table>
|
| 87 |
<p>Constant everywhere: effective batch 32 (except 014 at 16, 015-017 at 64), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.</p>
|
| 88 |
<h2>Results</h2>
|
|
@@ -100,8 +102,10 @@ a { color: #0969da; }
|
|
| 100 |
<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">RSICD overall</th>
|
| 101 |
</tr></thead><tbody>
|
| 102 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-011</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.85 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">83.51 ±1.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>82.68 ±0.84</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.92 ±0.49</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.78 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.85 ±0.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.06 ±0.67</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.33 ±1.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.20 ±0.92</td></tr>
|
| 103 |
-
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-
|
| 104 |
-
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-
|
|
|
|
|
|
|
| 105 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-006</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.40 ±0.71</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.95 ±1.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.18 ±0.90</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.81 ±0.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.79 ±1.12</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.80 ±0.65</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.26 ±0.68</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.88 ±1.63</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.57 ±0.91</td></tr>
|
| 106 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-005</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.37 ±0.67</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.37 ±1.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.87 ±0.85</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.55 ±0.49</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.74 ±1.11</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.64 ±0.62</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.79 ±0.71</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">69.35 ±1.65</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.57 ±0.93</td></tr>
|
| 107 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-012</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.68 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.05 ±1.53</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.86 ��0.86</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.73 ±0.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.02 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.37 ±0.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.85 ±0.71</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.23 ±1.63</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.04 ±0.90</td></tr>
|
|
@@ -154,18 +158,35 @@ a { color: #0969da; }
|
|
| 154 |
<p>The top eight runs are statistically tied: every confidence interval against the leader straddles zero. <strong>VQA error is ~2.3x caption error</strong>, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps and the gaps to the two references are real. Doubling the eval set from 500 to 1000 images did not separate the field, and it reshuffled the order.</p>
|
| 155 |
<h2>Findings</h2>
|
| 156 |
<ul>
|
| 157 |
-
<li><strong>Optimization is saturated.</strong> LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64)
|
| 158 |
<li><strong>The slices disagree.</strong> Ranking by in-domain score almost inverts the RSICD ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.</li>
|
| 159 |
<li><strong>Captioning is the real gain.</strong> It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.</li>
|
| 160 |
<li><strong>The in-domain VQA win is not established.</strong> The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.</li>
|
| 161 |
<li><strong>The vision tower matters.</strong> Freezing it is the one ablation that consistently hurt.</li>
|
| 162 |
<li><strong>Extra QA per image does not help.</strong> 4-QA and 6-QA mixtures tie with 2-QA.</li>
|
| 163 |
<li><strong>Larger batch trades generality for domain fit.</strong> Effective batch 64 gives the best in-domain captioning in the table and the worst RSICD VQA.</li>
|
|
|
|
| 164 |
<li><strong>Data is the remaining lever</strong>, not hyperparameters.</li>
|
| 165 |
</ul>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 166 |
<h2>Status</h2>
|
| 167 |
-
<p>The hyperparameter search is <strong>closed</strong>. Every axis available in this setup has been swept — rank, alpha, dropout, learning rate, schedule, weight decay, gradient clipping, vision-tower freezing, QA-per-image, caption prompt pool, batch geometry, effective batch size,
|
| 168 |
-
<p>
|
|
|
|
| 169 |
<p>Any further gain has to come from the training corpus: more in-domain imagery, denser or higher-quality question-answer pairs, or a different caption source.</p>
|
| 170 |
</main>
|
| 171 |
</body>
|
|
|
|
| 83 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-015</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">2e-4</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">effective batch 64</td></tr>
|
| 84 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-016</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>4e-4</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">batch 64 + linear LR scaling</td></tr>
|
| 85 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-017</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>3e-4</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">batch 64 + sqrt LR scaling</td></tr>
|
| 86 |
+
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-018</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.0</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">2e-4</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>bf16 base</strong> (no 4-bit)</td></tr>
|
| 87 |
+
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-019</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">16 / 16</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">0.05</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">2e-4</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">linear</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">trained</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">1c + 2q</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">18,864</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>bf16 base</strong> + 011 recipe</td></tr>
|
| 88 |
</tbody></table>
|
| 89 |
<p>Constant everywhere: effective batch 32 (except 014 at 16, 015-017 at 64), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.</p>
|
| 90 |
<h2>Results</h2>
|
|
|
|
| 102 |
<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">RSICD overall</th>
|
| 103 |
</tr></thead><tbody>
|
| 104 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-011</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.85 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">83.51 ±1.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>82.68 ±0.84</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.92 ±0.49</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.78 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.85 ±0.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.06 ±0.67</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.33 ±1.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.20 ±0.92</td></tr>
|
| 105 |
+
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-019</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>82.39 ±0.65</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.82 ±1.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.61 ±0.85</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">80.33 ±0.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.91 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.12 ±0.63</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.34 ±0.70</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">69.24 ±1.62</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.79 ±0.89</td></tr>
|
| 106 |
+
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-001</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.84 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">83.23 ±1.44</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.54 ±0.81</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.97 ±0.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.94 ±1.12</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.95 ±0.63</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.17 ±0.69</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.89 ±1.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.53 ±0.93</td></tr>
|
| 107 |
+
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-018</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.28 ±0.69</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.70 ±1.51</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.49 ±0.86</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>80.46 ±0.49</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.80 ±1.11</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>78.13 ±0.63</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.70 ±0.67</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">69.15 ±1.61</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.92 ±0.90</td></tr>
|
| 108 |
+
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-015</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.14 ±0.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.76 ±1.46</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.45 ±0.82</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">80.07 ±0.49</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">74.73 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.40 ±0.63</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.07 ±0.72</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">66.98 ±1.65</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">72.52 ±0.92</td></tr>
|
| 109 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-006</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.40 ±0.71</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.95 ±1.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.18 ±0.90</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.81 ±0.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.79 ±1.12</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.80 ±0.65</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">78.26 ±0.68</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.88 ±1.63</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.57 ±0.91</td></tr>
|
| 110 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-005</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.37 ±0.67</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.37 ±1.50</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.87 ±0.85</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.55 ±0.49</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.74 ±1.11</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.64 ±0.62</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.79 ±0.71</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">69.35 ±1.65</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.57 ±0.93</td></tr>
|
| 111 |
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-012</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.68 ±0.66</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">82.05 ±1.53</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">81.86 ��0.86</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">79.73 ±0.48</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">75.02 ±1.13</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.37 ±0.64</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">77.85 ±0.71</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">68.23 ±1.63</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">73.04 ±0.90</td></tr>
|
|
|
|
| 158 |
<p>The top eight runs are statistically tied: every confidence interval against the leader straddles zero. <strong>VQA error is ~2.3x caption error</strong>, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps and the gaps to the two references are real. Doubling the eval set from 500 to 1000 images did not separate the field, and it reshuffled the order.</p>
|
| 159 |
<h2>Findings</h2>
|
| 160 |
<ul>
|
| 161 |
+
<li><strong>Optimization is saturated.</strong> LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size (16 / 32 / 64), LR scaling at large batch and base-weight precision were all swept. None beat the noise floor.</li>
|
| 162 |
<li><strong>The slices disagree.</strong> Ranking by in-domain score almost inverts the RSICD ranking: the top three in-domain models place 11th, 2nd and 16th on the general set. The selected model wins by never being bad anywhere, not by leading any single slice.</li>
|
| 163 |
<li><strong>Captioning is the real gain.</strong> It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.</li>
|
| 164 |
<li><strong>The in-domain VQA win is not established.</strong> The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.</li>
|
| 165 |
<li><strong>The vision tower matters.</strong> Freezing it is the one ablation that consistently hurt.</li>
|
| 166 |
<li><strong>Extra QA per image does not help.</strong> 4-QA and 6-QA mixtures tie with 2-QA.</li>
|
| 167 |
<li><strong>Larger batch trades generality for domain fit.</strong> Effective batch 64 gives the best in-domain captioning in the table and the worst RSICD VQA.</li>
|
| 168 |
+
<li><strong>Training in bf16 instead of 4-bit is a small, consistent positive</strong> — the only change in the whole sweep that never loses on any slice — but it is still inside the error bar.</li>
|
| 169 |
<li><strong>Data is the remaining lever</strong>, not hyperparameters.</li>
|
| 170 |
</ul>
|
| 171 |
+
<h2>Base-weight precision</h2>
|
| 172 |
+
<p>Runs 001-017 all trained their adapters against <strong>4-bit quantized</strong> base weights, while evaluation and the final merge both use the <strong>bf16</strong> base. That is a train/serve mismatch: the adapter was fitted to compensate for quantization error that is absent at inference time.</p>
|
| 173 |
+
<p>Runs <strong>018</strong> and <strong>019</strong> repeat the two strongest recipes with the base loaded in bf16 and nothing else changed. Against the reference, paired on 1000 images:</p>
|
| 174 |
+
<table style="border-collapse:collapse;margin:1rem 0;font-size:.85rem;border:1px solid #d0d7de"><thead><tr>
|
| 175 |
+
<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">run</th>
|
| 176 |
+
<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">overall</th>
|
| 177 |
+
<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">in-domain</th>
|
| 178 |
+
<th style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;background:#f6f8fa;font-weight:600;white-space:nowrap">RSICD</th>
|
| 179 |
+
</tr></thead><tbody>
|
| 180 |
+
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-018</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>+0.17 ±0.42</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">-0.05 ±0.59</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>+0.39 ±0.59</strong></td></tr>
|
| 181 |
+
<tr><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap">Aspaklarya-019</td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>+0.16 ±0.40</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>+0.07 ±0.57</strong></td><td style="border:1px solid #d0d7de;padding:5px 10px;text-align:left;white-space:nowrap"><strong>+0.25 ±0.57</strong></td></tr>
|
| 182 |
+
</tbody></table>
|
| 183 |
+
<p>These are the first two runs in the sweep to post a positive overall delta, and <strong>018</strong> is the only run that is not worse than the reference on any slice. Both are still well inside the paired error bar (p = 0.34 that each is in fact worse), so neither is promoted.</p>
|
| 184 |
+
<p>The mechanism is not the expected one. Removing quantization error should mostly help the harder task, VQA; instead the gain is <strong>entirely in captioning</strong> (+0.48 and +0.35) while VQA is flat to slightly negative. That is the same caption-up / VQA-flat signature produced by every other lever in the sweep, which suggests precision is acting through the same saturated channel rather than unlocking anything new.</p>
|
| 185 |
+
<p>Removing quantization also costs <strong>~9 GB of GPU memory</strong> during training (19 GB vs 10 GB), which is a real constraint on a shared 40 GB card.</p>
|
| 186 |
<h2>Status</h2>
|
| 187 |
+
<p>The hyperparameter search is <strong>closed</strong>. Every axis available in this setup has been swept — rank, alpha, dropout, learning rate, schedule, weight decay, gradient clipping, vision-tower freezing, QA-per-image, caption prompt pool, batch geometry, effective batch size, learning rate at large batch, and base-weight precision. Nineteen recipes were trained and evaluated under identical conditions, and none displaced the reference recipe by more than the paired error bar.</p>
|
| 188 |
+
<p>Batch size and learning rate illustrate the ceiling. Effective batch 16 came in at <strong>-0.59</strong> overall, batch 64 at 2e-4 at <strong>-0.56</strong>, and batch 64 at 3e-4 at <strong>-0.34</strong> overall and <strong>-0.87</strong> in-domain; batch 64 at 4e-4 was stopped at epoch 4 while declining. In each case captioning held steady or improved while VQA fell, which is the same trade seen across the whole sweep.</p>
|
| 189 |
+
<p>Base precision is the one axis that points the right way (<strong>+0.17</strong> overall, never negative on a slice) without reaching significance. It is the natural thing to re-test if the training corpus is ever enlarged, since a larger corpus would shrink the error bar that currently hides it.</p>
|
| 190 |
<p>Any further gain has to come from the training corpus: more in-domain imagery, denser or higher-quality question-answer pairs, or a different caption source.</p>
|
| 191 |
</main>
|
| 192 |
</body>
|