warm-start experiment results vs cold 4/8
Browse files- aipf_warmstart_results.md +144 -0
aipf_warmstart_results.md
ADDED
|
@@ -0,0 +1,144 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# AIPF Youth 严重度分类 — Warm-Start 实验结果
|
| 2 |
+
|
| 3 |
+
数据:1000 条 golden_set(200 Y / 800 A)
|
| 4 |
+
ruler:200 条
|
| 5 |
+
模型:Gemini 2.5 Flash(pairwise judge)
|
| 6 |
+
默认阈值:rank<106,score≥44.72
|
| 7 |
+
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
## 一. 三组核心配置对比
|
| 11 |
+
|
| 12 |
+
| 维度 | cold 4 轮 | cold 8 轮 | **warm-start** |
|
| 13 |
+
|---|---:|---:|---:|
|
| 14 |
+
| TP(默认 by rank) | 190 | 188 | **184** |
|
| 15 |
+
| FP | 77 | 68 | **60** ⬇ |
|
| 16 |
+
| TN | 723 | 732 | **740** ⬆ |
|
| 17 |
+
| FN | 10 | 12 | **16** ⬆ |
|
| 18 |
+
| **Precision (默认)** | 0.7116 | 0.7344 | **0.7541** ⬆ |
|
| 19 |
+
| **Recall (默认)** | 0.9500 | 0.9400 | **0.9200** ⬇ |
|
| 20 |
+
| **F1 by rank (默认 thr=106)** | 0.8137 | 0.8246 | **0.8288** ⬆ |
|
| 21 |
+
| **F1 by score (默认 thr=44.72)** | 0.8137 | 0.8210 | **0.8274** ⬆ |
|
| 22 |
+
| max F1 by rank(扫阈值) | 0.8137 @ thr=113 | **0.8545** @ thr=98 | 0.8409 @ thr=98 |
|
| 23 |
+
| max F1 by score(扫阈值) | 0.8176 @ thr=46.33 | **0.8585** @ thr=49.02 | 0.8421 @ thr=48.90 |
|
| 24 |
+
|
| 25 |
+
## 二. Warm-start 详细指标
|
| 26 |
+
|
| 27 |
+
### 默认阈值 by rank (rank < 106)
|
| 28 |
+
| | TP | FP | TN | FN | P | R | F1 |
|
| 29 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 30 |
+
| | 184 | 60 | 740 | 16 | **0.7541** | 0.9200 | **0.8288** |
|
| 31 |
+
|
| 32 |
+
### 默认阈值 by score (score ≥ 44.72)
|
| 33 |
+
| | TP | FP | TN | FN | P | R | F1 |
|
| 34 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 35 |
+
| | 187 | 65 | 735 | 13 | 0.7421 | 0.9350 | 0.8274 |
|
| 36 |
+
|
| 37 |
+
### 扫阈值最优 by rank (best thr=98)
|
| 38 |
+
| | TP | FP | TN | FN | P | R | F1 |
|
| 39 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 40 |
+
| | 177 | 44 | 756 | 23 | **0.8009** | 0.8850 | **0.8409** |
|
| 41 |
+
|
| 42 |
+
### 扫阈值最优 by score (best thr=48.90)
|
| 43 |
+
| | TP | FP | TN | FN | P | R | F1 |
|
| 44 |
+
|---|---:|---:|---:|---:|---:|---:|---:|
|
| 45 |
+
| | 176 | 42 | 758 | 24 | **0.8073** | 0.8800 | **0.8421** |
|
| 46 |
+
|
| 47 |
+
---
|
| 48 |
+
|
| 49 |
+
## 三. 关键变化(warm vs cold-8)
|
| 50 |
+
|
| 51 |
+
| 项 | cold 8 | warm | Δ |
|
| 52 |
+
|---|---:|---:|---:|
|
| 53 |
+
| F1 (默认 by rank) | 0.8246 | 0.8288 | **+0.0042** ⬆ |
|
| 54 |
+
| F1 (默认 by score) | 0.8210 | 0.8274 | **+0.0064** ⬆ |
|
| 55 |
+
| F1 (max by rank, 扫阈值) | 0.8545 | 0.8409 | **-0.0136** ⬇ |
|
| 56 |
+
| F1 (max by score, 扫阈值) | 0.8585 | 0.8421 | **-0.0164** ⬇ |
|
| 57 |
+
| Precision (默认) | 0.7344 | 0.7541 | +0.0197 |
|
| 58 |
+
| Recall (默认) | 0.9400 | 0.9200 | -0.0200 |
|
| 59 |
+
| FP | 68 | 60 | -8 |
|
| 60 |
+
| FN | 12 | 16 | +4 |
|
| 61 |
+
|
| 62 |
+
**Warm-start 的 trade-off**:少误判 8 个 A(FP -8),但漏判 4 个 Y(FN +4),整体偏向"更保守"。
|
| 63 |
+
|
| 64 |
+
---
|
| 65 |
+
|
| 66 |
+
## 四. 全方法横向对比(更新版,按 best F1 排序)
|
| 67 |
+
|
| 68 |
+
| 排名 | 方法 | 默认 F1 | best F1 | LLM 调用 |
|
| 69 |
+
|---:|---|---:|---:|---:|
|
| 70 |
+
| 1 | Gemini 2.5 Flash 直打分 | 0.8416 | **0.8647** | N |
|
| 71 |
+
| 2 | AIPF cold 8 by score(扫阈值) | 0.8210 | 0.8585 | 8N |
|
| 72 |
+
| 3 | AIPF cold 8 by rank(扫阈值) | 0.8246 | 0.8545 | 8N |
|
| 73 |
+
| 4 | GPT-4.1 直打分 | 0.8524 | 0.8544 | N |
|
| 74 |
+
| 5 | **AIPF warm by score(扫阈值)** | **0.8274** | 0.8421 | 4~8N |
|
| 75 |
+
| 6 | **AIPF warm by rank(扫阈值)** | **0.8288** | 0.8409 | 4~8N |
|
| 76 |
+
| 7 | AIPF warm by rank(默认 thr) | **0.8288** | — | 4~8N |
|
| 77 |
+
| 8 | AIPF cold 8 by rank(默认 thr) | 0.8246 | — | 8N |
|
| 78 |
+
| 9 | AIPF cold 4 by score(扫阈值) | 0.8137 | 0.8176 | 4N |
|
| 79 |
+
| 10 | AIPF cold 4 by rank(默认 thr) | 0.8137 | — | 4N |
|
| 80 |
+
| 11 | embedding ridge: emb→score | 0.33 | 0.5891 | 0 |
|
| 81 |
+
| 12 | embedding PCA128+ridge | 0.33 | 0.5855 | 0 |
|
| 82 |
+
| 13 | embedding severity_axis (1D) | 0.33 | 0.5791 | 0 |
|
| 83 |
+
| 14 | embedding LDA | 0.33 | 0.5609 | 0 |
|
| 84 |
+
| 15 | embedding raw cosine + top100 weighted | 0.33 | 0.5574 | 0 |
|
| 85 |
+
| 16 | embedding kNN-100 mean(ruler_score) | 0.33 | 0.5535 | 0 |
|
| 86 |
+
| 17 | embedding ≥4/5 投票 (top5) | 0.5227 | — | 0 |
|
| 87 |
+
| 18 | embedding majority (≥3/5) (top5) | 0.4888 | — | 0 |
|
| 88 |
+
| 19 | embedding top1 rank<106 | 0.4565 | — | 0 |
|
| 89 |
+
|
| 90 |
+
---
|
| 91 |
+
|
| 92 |
+
## 五. 结论
|
| 93 |
+
|
| 94 |
+
### 默认阈值场景:warm-start 是赢家
|
| 95 |
+
不调阈值直接部署:
|
| 96 |
+
- cold 4: F1=0.8137
|
| 97 |
+
- cold 8: F1=0.8246
|
| 98 |
+
- **warm-start: F1=0.8288** ⭐
|
| 99 |
+
|
| 100 |
+
warm-start 比 cold 8 默认还高 +0.004,说明 embedding 起点把 P 提上去了。
|
| 101 |
+
|
| 102 |
+
### 扫阈值场景:cold 8 仍是上限
|
| 103 |
+
允许扫阈值时:
|
| 104 |
+
- **cold 8 max: F1=0.8585** ⭐
|
| 105 |
+
- warm max: F1=0.8421
|
| 106 |
+
- cold 4 max: F1=0.8176
|
| 107 |
+
|
| 108 |
+
warm-start 反而比 cold 8 最优低 0.016。原因可能是 embedding 起点让 TrueSkill 的探索过早收敛,输出 score 分布的可分性下降。
|
| 109 |
+
|
| 110 |
+
### 性价比对比
|
| 111 |
+
| 方法 | F1 | LLM 调用 | F1 / 调用比 |
|
| 112 |
+
|---|---:|---:|---:|
|
| 113 |
+
| Gemini 直打分 | 0.8647 | 1N | 0.865 |
|
| 114 |
+
| AIPF cold 8 | 0.8585 | 8N | 0.107 |
|
| 115 |
+
| AIPF warm | 0.8421 | 4~8N | 0.105~0.211 |
|
| 116 |
+
| AIPF cold 4 | 0.8176 | 4N | 0.204 |
|
| 117 |
+
|
| 118 |
+
LLM 直打分一骑绝尘。AIPF warm 如果用的是 4 轮,**每次 LLM 调用 F1 收益和 cold 4 相当但绝对值高 +0.024**,是 AIPF 套路下的最优解。
|
| 119 |
+
|
| 120 |
+
---
|
| 121 |
+
|
| 122 |
+
## 六. 待确认 / 待补实验
|
| 123 |
+
|
| 124 |
+
| 项 | 还没确认 |
|
| 125 |
+
|---|---|
|
| 126 |
+
| Warm-start 用了几轮(4 / 8) | 看 `rounds_completed` 字段 |
|
| 127 |
+
| Warm-start 用了什么 K(top1 / top5 / top100)的 estimated_position | 看跑前 `add_estimated_position.py --k` 参数 |
|
| 128 |
+
| Warm 4 vs cold 4 直接对照 | 需要再跑一组 num_rounds=4 + warm |
|
| 129 |
+
| Warm 2 vs cold 4 对照 | 极限省调用实验 |
|
| 130 |
+
|
| 131 |
+
诊断 warm 真实使用轮数:
|
| 132 |
+
|
| 133 |
+
```bash
|
| 134 |
+
RUN_DIR=$(ls -td example/yss_ruler_eval/runs/20260429/*/ | head -1)
|
| 135 |
+
RESULTS=$(ls $RUN_DIR/outputs/find_positions/*/results_*.json | head -1)
|
| 136 |
+
python3 -c "
|
| 137 |
+
import json
|
| 138 |
+
from collections import Counter
|
| 139 |
+
data = json.load(open('$RESULTS'))
|
| 140 |
+
rounds = [d.get('rounds_completed') for d in data]
|
| 141 |
+
print('rounds 分布:', dict(sorted(Counter(rounds).items())))
|
| 142 |
+
print('平均:', sum(rounds)/len(rounds))
|
| 143 |
+
"
|
| 144 |
+
```
|