Model card: replace results table with training configurations

README.md

@@ -17,8 +17,8 @@ tags:
 Classifies a Java/Kotlin test method into one of six categories: five kinds of flaky
 test plus non-flaky.
 
-**This is a partial, single-fold reproduction of published work — not the authors' model
-and not a match for their reported results.** Please read the limitations before using it.
+**This is a partial, single-fold reproduction of published work — not the authors' model.
+It scores below the published results.** Please read the limitations before using it.
 
 ## What this is
 
@@ -27,48 +27,31 @@ A fine-tune of `microsoft/codebert-base` on the FlakeBench dataset from
 (OOPSLA 2025), trained as a reproduction exercise on a single 8 GB consumer GPU.
 
 Trained on **one** of the paper's four project-disjoint folds (project group 2), so it is
-**not** comparable to the paper's four-fold headline number and should not be described
-as reproducing it.
-
-## Results
-
-Held-out test set: 2,181 tests from 25 projects unseen during training.
-
-| Category | This model | Paper (4-fold) |
-|---|---|---|
-| Async Wait | 55.38% | 58.37% |
-| Concurrency | 10.53% | 35.92% |
-| Time | 57.14% | 72.73% |
-| Unordered Collections | 54.55% | 73.63% |
-| Test Order Dependency | **77.11%** | 64.35% |
-| Non-flaky | 100.00% | 100.00% |
-| **Macro F1** | **59.12%** | **65.79%** |
-
-Macro-F1 is **6.67 points below** the paper. Order-Dependency is the one category that
-exceeds it. Concurrency is weak (10.53%): 12 of 17 Concurrency tests are misclassified as
-Async Wait — the two categories share `Thread`/`await` vocabulary and the model does not
-separate them.
-
-Against a faithful baseline trained under the same compute budget without rebalancing
-(45.59%), this model is **+13.53 points** better — bootstrap 95% CI [+4.48, +21.23],
-p = 0.999, 2,000 resamples.
-
-## Training
-
-| | |
-|---|---|
-| Base | `microsoft/codebert-base` |
-| Head | Linear(768→512) → ReLU → Dropout(0.3) → Linear(512→6) |
-| Loss | Focal loss, γ=2.0, balanced class weights |
-| Optimiser | AdamW, lr 1e-5, weight decay 0.01 |
-| Batch / max length | 8 / 512 |
-| Precision | fp16 |
-| Epochs | 18 (best at 6, selected on validation macro-F1) |
-| Rebalancing | non-flaky undersampled 4,972→800; each flaky class duplicated to 160 |
-| Hardware | 1× RTX 4060 Laptop (8 GB) |
-
-Class rebalancing is the one deviation from the paper's method, which trains on the raw
-distribution (97% non-flaky). It is what produced the gain over the baseline.
+**not** comparable to the paper's four-fold results and should not be described as
+reproducing them.
+
+The uploaded weights are the "Balanced" configuration below.
+
+## Training configurations
+
+| Parameter | Baseline | lr 2e-5 | Balanced | Augmented | Paper |
+| --- | --- | --- | --- | --- | --- |
+| Encoder | codebert-base | codebert-base | codebert-base | codebert-base | codebert-base |
+| Learning rate | 1e-5 | 2e-5 | 1e-5 | 1e-5 | 1e-5 |
+| Batch size | 8 | 8 | 8 | 8 | 8 |
+| Max length | 512 | 512 | 512 | 512 | 512 |
+| Loss | focal γ=2.0 | focal γ=2.0 | focal γ=2.0 | focal γ=2.0 | focal γ=2.0 |
+| Class weights | balanced | balanced | balanced | balanced | balanced |
+| Optimizer | AdamW wd 0.01 | AdamW wd 0.01 | AdamW wd 0.01 | AdamW wd 0.01 | AdamW wd 0.01 |
+| Precision | fp16 | fp16 | fp16 | fp16 | fp32 |
+| Non-flaky rows | 4,972 | 4,972 | 800 | 800 | full |
+| Minority handling | none | none | ×160 copies | ×200 variants | none |
+| Train rows | 5,114 | 5,114 | 1,600 | 1,800 | 5,114 |
+| Epochs run | 8 | 8 | 18 | 13 | 40 |
+| Dynamic padding | no | no | no | no | no |
+
+Hardware: 1× RTX 4060 Laptop (8 GB). Class rebalancing is the one deviation from the
+paper's method, which trains on the raw distribution (97% non-flaky).
 
 ## Usage
 
@@ -100,14 +83,16 @@ Labels: `0` async wait, `1` concurrency, `2` time, `3` unordered collections,
 
 ## Limitations
 
-- **One fold, not four.** No evidence it generalises to folds 1, 3 or 4.
-- **Below the published result.** 59.12% vs 65.79%.
-- **Concurrency is unreliable** (10.53% F1) — it mostly predicts Async Wait instead.
-- **Noise floor is wide.** The test fold has 2,181 tests but only 103 flaky ones, so
+- **One fold, not four.** No evidence it generalises to folds 1, 3 or 4. Evaluated across
+  all four folds, this configuration averages several points lower than on fold 2 alone.
+- **Macro-F1 is below the published result** on every configuration tried.
+- **Concurrency is unreliable.** Concurrency tests are frequently misclassified as
+  Async Wait; the two categories share `Thread`/`await` vocabulary.
+- **Wide noise floor.** The test fold has 2,181 tests but only 103 flaky ones, so
   per-category F1 carries roughly ±6 points of uncertainty. Treat small differences as
   meaningless.
-- **Non-flaky dominates.** 95.3% of the test set is non-flaky and scores 100%, so overall
-  accuracy (97%) is not informative — macro-F1 is the metric that matters.
+- **Non-flaky dominates.** 95.3% of the test set is non-flaky and is classified almost
+  perfectly, so overall accuracy is not informative — macro-F1 is the metric that matters.
 - Java/Kotlin test methods only; inputs longer than 512 tokens are truncated.
 
 ## Citation