add warm 4 + top5/top10 results, K sweet spot found at K=5
Browse files- aipf_warmstart_results.md +116 -120
aipf_warmstart_results.md
CHANGED
|
@@ -1,144 +1,140 @@
|
|
| 1 |
-
# AIPF
|
| 2 |
|
| 3 |
-
数据:1000 条 golden_set(200 Y / 800 A)
|
| 4 |
-
ruler:200 条
|
| 5 |
-
模型:Gemini 2.5 Flash(pairwise judge)
|
| 6 |
默认阈值:rank<106,score≥44.72
|
| 7 |
|
| 8 |
-
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
| FP |
|
| 16 |
-
| TN | 723 | 732 | **740** ⬆ |
|
| 17 |
-
| FN | 10 | 12 | **16** ⬆ |
|
| 18 |
-
| **Precision (默认)** | 0.7116 | 0.7344 | **0.7541** ⬆ |
|
| 19 |
-
| **Recall (默认)** | 0.9500 | 0.9400 | **0.9200** ⬇ |
|
| 20 |
-
| **F1 by rank (默认 thr=106)** | 0.8137 | 0.8246 | **0.8288** ⬆ |
|
| 21 |
-
| **F1 by score (默认 thr=44.72)** | 0.8137 | 0.8210 | **0.8274** ⬆ |
|
| 22 |
-
| max F1 by rank(扫阈值) | 0.8137 @ thr=113 | **0.8545** @ thr=98 | 0.8409 @ thr=98 |
|
| 23 |
-
| max F1 by score(扫阈值) | 0.8176 @ thr=46.33 | **0.8585** @ thr=49.02 | 0.8421 @ thr=48.90 |
|
| 24 |
-
|
| 25 |
-
## 二. Warm-start 详细指标
|
| 26 |
-
|
| 27 |
-
### 默认阈值 by rank (rank < 106)
|
| 28 |
-
| | TP | FP | TN | FN | P | R | F1 |
|
| 29 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 30 |
-
| |
|
|
|
|
| 31 |
|
| 32 |
-
###
|
| 33 |
-
| | TP | FP | TN | FN | P | R | F1 |
|
| 34 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 35 |
-
| |
|
|
|
|
| 36 |
|
| 37 |
-
###
|
| 38 |
-
| | TP | FP | TN | FN | P | R | F1 |
|
| 39 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 40 |
-
| |
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
-
###
|
| 43 |
-
| | TP | FP | TN | FN | P | R | F1 |
|
| 44 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 45 |
-
| |
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
-
|
| 52 |
-
|---|---:|---:|---:|
|
| 53 |
-
| F1 (默认 by rank) | 0.8246 | 0.8288 | **+0.0042** ⬆ |
|
| 54 |
-
| F1 (默认 by score) | 0.8210 | 0.8274 | **+0.0064** ⬆ |
|
| 55 |
-
| F1 (max by rank, 扫阈值) | 0.8545 | 0.8409 | **-0.0136** ⬇ |
|
| 56 |
-
| F1 (max by score, 扫阈值) | 0.8585 | 0.8421 | **-0.0164** ⬇ |
|
| 57 |
-
| Precision (默认) | 0.7344 | 0.7541 | +0.0197 |
|
| 58 |
-
| Recall (默认) | 0.9400 | 0.9200 | -0.0200 |
|
| 59 |
-
| FP | 68 | 60 | -8 |
|
| 60 |
-
| FN | 12 | 16 | +4 |
|
| 61 |
|
| 62 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
-
|
| 65 |
|
| 66 |
-
|
| 67 |
|
| 68 |
-
|
| 69 |
-
|---:|---|---:|---:|---:|
|
| 70 |
-
| 1 | Gemini 2.5 Flash 直打分 | 0.8416 | **0.8647** | N |
|
| 71 |
-
| 2 | AIPF cold 8 by score(扫阈值) | 0.8210 | 0.8585 | 8N |
|
| 72 |
-
| 3 | AIPF cold 8 by rank(扫阈值) | 0.8246 | 0.8545 | 8N |
|
| 73 |
-
| 4 | GPT-4.1 直打分 | 0.8524 | 0.8544 | N |
|
| 74 |
-
| 5 | **AIPF warm by score(扫阈值)** | **0.8274** | 0.8421 | 4~8N |
|
| 75 |
-
| 6 | **AIPF warm by rank(扫阈值)** | **0.8288** | 0.8409 | 4~8N |
|
| 76 |
-
| 7 | AIPF warm by rank(默认 thr) | **0.8288** | — | 4~8N |
|
| 77 |
-
| 8 | AIPF cold 8 by rank(默认 thr) | 0.8246 | — | 8N |
|
| 78 |
-
| 9 | AIPF cold 4 by score(扫阈值) | 0.8137 | 0.8176 | 4N |
|
| 79 |
-
| 10 | AIPF cold 4 by rank(默认 thr) | 0.8137 | — | 4N |
|
| 80 |
-
| 11 | embedding ridge: emb→score | 0.33 | 0.5891 | 0 |
|
| 81 |
-
| 12 | embedding PCA128+ridge | 0.33 | 0.5855 | 0 |
|
| 82 |
-
| 13 | embedding severity_axis (1D) | 0.33 | 0.5791 | 0 |
|
| 83 |
-
| 14 | embedding LDA | 0.33 | 0.5609 | 0 |
|
| 84 |
-
| 15 | embedding raw cosine + top100 weighted | 0.33 | 0.5574 | 0 |
|
| 85 |
-
| 16 | embedding kNN-100 mean(ruler_score) | 0.33 | 0.5535 | 0 |
|
| 86 |
-
| 17 | embedding ≥4/5 投票 (top5) | 0.5227 | — | 0 |
|
| 87 |
-
| 18 | embedding majority (≥3/5) (top5) | 0.4888 | — | 0 |
|
| 88 |
-
| 19 | embedding top1 rank<106 | 0.4565 | — | 0 |
|
| 89 |
-
|
| 90 |
-
---
|
| 91 |
-
|
| 92 |
-
## 五. 结论
|
| 93 |
-
|
| 94 |
-
### 默认阈值场景:warm-start 是赢家
|
| 95 |
-
不调阈值直接部署:
|
| 96 |
-
- cold 4: F1=0.8137
|
| 97 |
-
- cold 8: F1=0.8246
|
| 98 |
-
- **warm-start: F1=0.8288** ⭐
|
| 99 |
-
|
| 100 |
-
warm-start 比 cold 8 默认还高 +0.004,说明 embedding 起点把 P 提上去了。
|
| 101 |
-
|
| 102 |
-
### 扫阈值场景:cold 8 仍是上限
|
| 103 |
-
允许扫阈值时:
|
| 104 |
-
- **cold 8 max: F1=0.8585** ⭐
|
| 105 |
-
- warm max: F1=0.8421
|
| 106 |
-
- cold 4 max: F1=0.8176
|
| 107 |
-
|
| 108 |
-
warm-start 反而比 cold 8 最优低 0.016。原因可能是 embedding 起点让 TrueSkill 的探索过早收敛,输出 score 分布的可分性下降。
|
| 109 |
-
|
| 110 |
-
### 性价比对比
|
| 111 |
-
| 方法 | F1 | LLM 调用 | F1 / 调用比 |
|
| 112 |
-
|---|---:|---:|---:|
|
| 113 |
-
| Gemini 直打分 | 0.8647 | 1N | 0.865 |
|
| 114 |
-
| AIPF cold 8 | 0.8585 | 8N | 0.107 |
|
| 115 |
-
| AIPF warm | 0.8421 | 4~8N | 0.105~0.211 |
|
| 116 |
-
| AIPF cold 4 | 0.8176 | 4N | 0.204 |
|
| 117 |
|
| 118 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
|
| 120 |
-
|
| 121 |
|
| 122 |
-
##
|
| 123 |
|
| 124 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 125 |
|---|---|
|
| 126 |
-
|
|
| 127 |
-
|
|
| 128 |
-
|
|
| 129 |
-
|
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
data = json.load(open('$RESULTS'))
|
| 140 |
-
rounds = [d.get('rounds_completed') for d in data]
|
| 141 |
-
print('rounds 分布:', dict(sorted(Counter(rounds).items())))
|
| 142 |
-
print('平均:', sum(rounds)/len(rounds))
|
| 143 |
-
"
|
| 144 |
-
```
|
|
|
|
| 1 |
+
# AIPF Warm-Start 完整实验结果
|
| 2 |
|
| 3 |
+
数据:1000 条 golden_set(200 Y / 800 A)| ruler 200 条 | LLM = Gemini 2.5 Flash
|
|
|
|
|
|
|
| 4 |
默认阈值:rank<106,score≥44.72
|
| 5 |
|
| 6 |
+
## 一. 6 组配置完整对比
|
| 7 |
|
| 8 |
+
| 配置 | LLM 调用 | F1 by rank(默认) | F1 by score(默认) | max F1 by rank(扫) | max F1 by score(扫) |
|
| 9 |
+
|---|---:|---:|---:|---:|---:|
|
| 10 |
+
| cold 4 | 4N | 0.8137 | 0.8137 | 0.8137 (thr=113) | 0.8176 (thr=46.33) |
|
| 11 |
+
| cold 8 | 8N | 0.8246 | 0.8210 | **0.8545** (thr=98) | **0.8585** (thr=49.02) |
|
| 12 |
+
| **warm 4 + top5** | **4N** | **0.8322** ⭐ | 0.8282 | 0.8426 (thr=99) | 0.8516 (thr=49.47) |
|
| 13 |
+
| warm 4 + top10 | 4N | 0.8151 | 0.8212 | 0.8317 (thr=98) | 0.8346 (thr=49.48) |
|
| 14 |
+
| warm 4 + top100 | 4N | 0.8017 | 0.7974 | 0.8354 (thr=93) | 0.8395 (thr=50.48) |
|
| 15 |
+
| warm 8 + top100 | 8N | 0.8288 | 0.8274 | 0.8409 (thr=98) | 0.8421 (thr=48.90) |
|
| 16 |
|
| 17 |
+
## 二. 详细 4 组核心配置(TP/FP/TN/FN)
|
| 18 |
+
|
| 19 |
+
### cold 4
|
| 20 |
+
| 维度 | TP | FP | TN | FN | P | R | F1 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 22 |
+
| by rank | 190 | 77 | 723 | 10 | 0.7116 | 0.9500 | 0.8137 |
|
| 23 |
+
| by score | 190 | 77 | 723 | 10 | 0.7116 | 0.9500 | 0.8137 |
|
| 24 |
|
| 25 |
+
### cold 8
|
| 26 |
+
| 维度 | TP | FP | TN | FN | P | R | F1 |
|
| 27 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 28 |
+
| by rank | 188 | 68 | 732 | 12 | 0.7344 | 0.9400 | 0.8246 |
|
| 29 |
+
| by score | 188 | 70 | 730 | 12 | 0.7287 | 0.9400 | 0.8210 |
|
| 30 |
|
| 31 |
+
### warm 4 + top5 ⭐
|
| 32 |
+
| 维度 | TP | FP | TN | FN | P | R | F1 |
|
| 33 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 34 |
+
| by rank(默认) | 186 | 61 | 739 | 14 | 0.7530 | 0.9300 | **0.8322** |
|
| 35 |
+
| by score(默认) | 188 | 66 | 734 | 12 | 0.7402 | 0.9400 | 0.8282 |
|
| 36 |
+
| max by rank(thr=99) | 182 | 50 | 750 | 18 | 0.7845 | 0.9100 | 0.8426 |
|
| 37 |
+
| max by score(thr=49.47) | 175 | 36 | 764 | 25 | 0.8294 | 0.8750 | 0.8516 |
|
| 38 |
|
| 39 |
+
### warm 4 + top10
|
| 40 |
+
| 维度 | TP | FP | TN | FN | P | R | F1 |
|
| 41 |
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 42 |
+
| by rank(默认) | 183 | 66 | 734 | 17 | 0.7349 | 0.9150 | 0.8151 |
|
| 43 |
+
| by score(默认) | 186 | 67 | 733 | 14 | 0.7352 | 0.9300 | 0.8212 |
|
| 44 |
+
| max by rank(thr=98) | 173 | 43 | 757 | 27 | 0.8009 | 0.8650 | 0.8317 |
|
| 45 |
+
| max by score(thr=49.48) | 169 | 36 | 764 | 31 | 0.8244 | 0.8450 | 0.8346 |
|
| 46 |
|
| 47 |
+
### warm 4 + top100
|
| 48 |
+
| 维度 | TP | FP | TN | FN | P | R | F1 |
|
| 49 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 50 |
+
| by rank(默认) | 186 | 78 | 722 | 14 | 0.7045 | 0.9300 | 0.8017 |
|
| 51 |
+
| by score(默认) | 187 | 82 | 718 | 13 | 0.6952 | 0.9350 | 0.7974 |
|
| 52 |
+
| max by rank(thr=93) | 165 | 30 | 770 | 35 | 0.8462 | 0.8250 | 0.8354 |
|
| 53 |
+
| max by score(thr=50.48) | 170 | 35 | 765 | 30 | 0.8293 | 0.8500 | 0.8395 |
|
| 54 |
|
| 55 |
+
### warm 8 + top100
|
| 56 |
+
| 维度 | TP | FP | TN | FN | P | R | F1 |
|
| 57 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 58 |
+
| by rank(默认) | 184 | 60 | 740 | 16 | 0.7541 | 0.9200 | 0.8288 |
|
| 59 |
+
| by score(默认) | 187 | 65 | 735 | 13 | 0.7421 | 0.9350 | 0.8274 |
|
| 60 |
+
| max by rank(thr=98) | 177 | 44 | 756 | 23 | 0.8009 | 0.8850 | 0.8409 |
|
| 61 |
+
| max by score(thr=48.90) | 176 | 42 | 758 | 24 | 0.8073 | 0.8800 | 0.8421 |
|
| 62 |
|
| 63 |
+
## 三. K 的 sweet spot 分��(warm 4 系列)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 64 |
|
| 65 |
+
| K | F1 默认 by rank | max F1 by score |
|
| 66 |
+
|---|---:|---:|
|
| 67 |
+
| 5 | **0.8322** ⭐ | **0.8516** ⭐ |
|
| 68 |
+
| 10 | 0.8151 | 0.8346 |
|
| 69 |
+
| 100 | 0.8017 | 0.8395 |
|
| 70 |
|
| 71 |
+
**top5 全面胜出**:默认 F1 比 top10 高 +0.017、比 top100 高 +0.031;最优 F1 比 top10 高 +0.017。
|
| 72 |
|
| 73 |
+
原因:top100 邻居均值集中在窄带(74~116),起点信号被压扁;top5 散度合适,既稳又锐。
|
| 74 |
|
| 75 |
+
## 四. 性价比对比(每次 LLM 调用换多少 F1)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
+
| 方法 | F1 (默认) | LLM 调用 | F1 / 调用 |
|
| 78 |
+
|---|---:|---:|---:|
|
| 79 |
+
| Gemini 直打分 | 0.8416 | 1N | **0.8416** ⭐ |
|
| 80 |
+
| **warm 4 + top5** | **0.8322** | **4N** | **0.2081** |
|
| 81 |
+
| GPT-4.1 直打分 | 0.8524 | 1N | 0.8524 |
|
| 82 |
+
| cold 4 | 0.8137 | 4N | 0.2034 |
|
| 83 |
+
| warm 8 + top100 | 0.8288 | 8N | 0.1036 |
|
| 84 |
+
| cold 8 | 0.8246 | 8N | 0.1031 |
|
| 85 |
|
| 86 |
+
LLM 直打分一骑绝尘。AIPF 系里 **warm 4 + top5 性价比最高**:相同 4 轮预算下比 cold 4 多 +0.0185 F1。
|
| 87 |
|
| 88 |
+
## 五. 全方法横向对比(按 best F1 排序)
|
| 89 |
|
| 90 |
+
| 排名 | 方法 | 默认 F1 | best F1 | LLM 调用 |
|
| 91 |
+
|---:|---|---:|---:|---:|
|
| 92 |
+
| 1 | Gemini 2.5 Flash 直打分 | 0.8416 | **0.8647** | N |
|
| 93 |
+
| 2 | AIPF cold 8 by score(扫) | 0.8210 | 0.8585 | 8N |
|
| 94 |
+
| 3 | AIPF cold 8 by rank(扫) | 0.8246 | 0.8545 | 8N |
|
| 95 |
+
| 4 | GPT-4.1 直打分 | 0.8524 | 0.8544 | N |
|
| 96 |
+
| 5 | **AIPF warm 4 + top5 by score(扫)** | **0.8282** | **0.8516** | **4N** |
|
| 97 |
+
| 6 | **AIPF warm 4 + top5 by rank(扫)** | **0.8322** ⭐ | 0.8426 | **4N** |
|
| 98 |
+
| 7 | AIPF warm 8 + top100 by score(扫) | 0.8274 | 0.8421 | 8N |
|
| 99 |
+
| 8 | AIPF warm 8 + top100 by rank(扫) | 0.8288 | 0.8409 | 8N |
|
| 100 |
+
| 9 | AIPF warm 4 + top100 by score(扫) | 0.7974 | 0.8395 | 4N |
|
| 101 |
+
| 10 | AIPF warm 4 + top10 by score(扫) | 0.8212 | 0.8346 | 4N |
|
| 102 |
+
| 11 | AIPF warm 4 + top100 by rank(扫) | 0.8017 | 0.8354 | 4N |
|
| 103 |
+
| 12 | AIPF warm 4 + top10 by rank(扫) | 0.8151 | 0.8317 | 4N |
|
| 104 |
+
| 13 | AIPF cold 4 by score(扫) | 0.8137 | 0.8176 | 4N |
|
| 105 |
+
| 14 | AIPF cold 4 by rank(默认) | 0.8137 | 0.8137 | 4N |
|
| 106 |
+
| 15 | embedding ridge: emb→score | 0.33 | 0.5891 | 0 |
|
| 107 |
+
| 16 | embedding PCA128 + ridge | 0.33 | 0.5855 | 0 |
|
| 108 |
+
| 17 | embedding severity_axis (1D) | 0.33 | 0.5791 | 0 |
|
| 109 |
+
| 18 | embedding raw cosine + top100 weighted | 0.33 | 0.5574 | 0 |
|
| 110 |
+
| 19 | embedding ≥4/5 投票 (top5) | 0.5227 | — | 0 |
|
| 111 |
+
| 20 | embedding top1 rank<106 | 0.4565 | — | 0 |
|
| 112 |
+
|
| 113 |
+
## 六. 三个核心结论
|
| 114 |
+
|
| 115 |
+
### 1. warm 4 + top5 是 AIPF 系最优
|
| 116 |
+
- 比 cold 4 提 +0.019 F1(默认)
|
| 117 |
+
- 比 cold 8 提 +0.008 F1(默认),LLM 调用减半
|
| 118 |
+
- 比 cold 8 max F1 仅低 0.007(扫阈值场景)
|
| 119 |
+
|
| 120 |
+
### 2. K 选择决定 warm-start 成败
|
| 121 |
+
- K=5 sweet spot
|
| 122 |
+
- K=100 信号被压扁(estimated_position 集中在窄带)
|
| 123 |
+
- K=10 中等,不如 K=5
|
| 124 |
+
|
| 125 |
+
### 3. 部署建议
|
| 126 |
+
| 场景 | 推荐 |
|
| 127 |
|---|---|
|
| 128 |
+
| 不调阈值直接用,预算紧 | **warm 4 + top5**(F1=0.8322,4N 调用) |
|
| 129 |
+
| 不调阈值直接用,预算够 | warm 8 + top100(F1=0.8288)或 warm 4 + top5 都行 |
|
| 130 |
+
| 允许扫阈值,追求最高 F1 | cold 8(max F1=0.8585,8N 调用) |
|
| 131 |
+
| 极致 F1/成本比 | Gemini 直打分(F1=0.8647,1N 调用) |
|
| 132 |
+
|
| 133 |
+
## 七. 待补实验
|
| 134 |
+
|
| 135 |
+
| 项 | 已跑 | 待跑 |
|
| 136 |
+
|---|---|---|
|
| 137 |
+
| warm 4 + top1 | ❌ | 极端样本起点会怎样 |
|
| 138 |
+
| warm 4 + top3 | ❌ | top5 邻域内更细的扫描 |
|
| 139 |
+
| warm 8 + top5 | ❌ | top5 起点 + 8 轮探索是不是 AIPF 的天花板 |
|
| 140 |
+
| warm 6 + top5 | ❌ | 5/8 中间值 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|