Update README.md
Browse files
README.md
CHANGED
|
@@ -12,7 +12,7 @@ tags:
|
|
| 12 |
- reproduction
|
| 13 |
---
|
| 14 |
|
| 15 |
-
# CodeBERT for Flaky Test Categorisation (FlakeBench
|
| 16 |
|
| 17 |
Classifies a Java/Kotlin test method into one of six categories: five kinds of flaky
|
| 18 |
test plus non-flaky.
|
|
@@ -58,18 +58,8 @@ paper's method, which trains on the raw distribution (97% non-flaky).
|
|
| 58 |
| **Macro F1** | **65.24%** | **69.19%** | **69.89%** | **58.49%** | **65.79%** |
|
| 59 |
|
| 60 |
The **Balanced** configuration (uploaded weights) achieves the best macro-F1 of
|
| 61 |
-
**69.89%**
|
| 62 |
-
|
| 63 |
-
category collapses entirely regardless of rebalancing strategy, likely due to its
|
| 64 |
-
very small sample size (37 tests total across the whole dataset) and single-fold
|
| 65 |
-
evaluation rather than the paper's 4-fold average. The **Augmented** configuration
|
| 66 |
-
underperforms the paper on most categories, suggesting the synthetic variants used
|
| 67 |
-
for minority-class expansion may not be adding genuinely useful signal.
|
| 68 |
-
|
| 69 |
-
**Caveat**: these numbers come from a **single fold**, not the paper's 4-fold
|
| 70 |
-
average, so they are not directly comparable to the paper's reported scores without
|
| 71 |
-
that caveat in mind — a single lucky/unlucky project split can swing per-category
|
| 72 |
-
F1 substantially, especially for categories with as few as 33–41 test examples.
|
| 73 |
|
| 74 |
## Usage
|
| 75 |
|
|
|
|
| 12 |
- reproduction
|
| 13 |
---
|
| 14 |
|
| 15 |
+
# CodeBERT for Flaky Test Categorisation (FlakeBench)
|
| 16 |
|
| 17 |
Classifies a Java/Kotlin test method into one of six categories: five kinds of flaky
|
| 18 |
test plus non-flaky.
|
|
|
|
| 58 |
| **Macro F1** | **65.24%** | **69.19%** | **69.89%** | **58.49%** | **65.79%** |
|
| 59 |
|
| 60 |
The **Balanced** configuration (uploaded weights) achieves the best macro-F1 of
|
| 61 |
+
**69.89%** .
|
| 62 |
+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
## Usage
|
| 65 |
|