FlameLedger-Encoder / scorecard.md
rarfileexe's picture
Upload 9 files
cab21aa verified
|
Raw History Blame Contribute Delete
3.72 kB

synthetic TEST, held-out papers only (n=150 queries)

method recall@1 recall@5 recall@10 MRR@10
BM25 0.800 0.967 1.000 0.874
base dense 768 0.713 0.967 0.993 0.811
base dense 512 0.693 0.967 0.993 0.807
base dense 256 0.720 0.947 0.987 0.814
base dense 128 0.687 0.900 0.987 0.790
base + BM25 (RRF) 0.787 0.987 1.000 0.873
fine-tuned dense 768 0.793 0.993 1.000 0.871
fine-tuned dense 512 0.807 0.987 1.000 0.880
fine-tuned dense 256 0.800 0.973 0.993 0.871
fine-tuned dense 128 0.800 0.967 0.993 0.865
fine-tuned + BM25 (RRF) 0.813 1.000 1.000 0.891

synthetic TEST, full archive (n=150 queries)

method recall@1 recall@5 recall@10 MRR@10
BM25 0.647 0.900 0.940 0.752
base dense 768 0.527 0.840 0.893 0.652
base dense 512 0.513 0.840 0.900 0.646
base dense 256 0.527 0.793 0.907 0.641
base dense 128 0.453 0.767 0.847 0.589
base + BM25 (RRF) 0.673 0.913 0.967 0.776
fine-tuned dense 768 0.647 0.920 0.953 0.747
fine-tuned dense 512 0.653 0.907 0.967 0.752
fine-tuned dense 256 0.627 0.893 0.960 0.734
fine-tuned dense 128 0.627 0.833 0.920 0.720
fine-tuned + BM25 (RRF) 0.713 0.940 0.987 0.807

GOLD (teacher-drafted, UNREVIEWED) (n=23 queries)

method recall@1 recall@5 recall@10 MRR@10
BM25 0.304 0.609 0.739 0.436
base dense 768 0.391 0.609 0.739 0.506
base dense 512 0.391 0.696 0.696 0.504
base dense 256 0.348 0.609 0.696 0.476
base dense 128 0.391 0.565 0.565 0.454
base + BM25 (RRF) 0.261 0.565 0.696 0.422
fine-tuned dense 768 0.478 0.609 0.696 0.537
fine-tuned dense 512 0.435 0.652 0.696 0.523
fine-tuned dense 256 0.435 0.565 0.739 0.509
fine-tuned dense 128 0.391 0.522 0.609 0.454
fine-tuned + BM25 (RRF) 0.391 0.696 0.696 0.505

Fine-tuned minus base (paired bootstrap, 95% CI)

query set comparison metric delta 95% CI
synthetic TEST, held-out papers only base dense 768 -> fine-tuned dense 768 recall@5 +0.027 [+0.007, +0.060]
synthetic TEST, held-out papers only base dense 768 -> fine-tuned dense 768 mrr@10 +0.060 [+0.031, +0.092]
synthetic TEST, held-out papers only base + BM25 (RRF) -> fine-tuned + BM25 (RRF) recall@5 +0.013 [+0.000, +0.033]
synthetic TEST, held-out papers only base + BM25 (RRF) -> fine-tuned + BM25 (RRF) mrr@10 +0.018 [-0.002, +0.037]
synthetic TEST, full archive base dense 768 -> fine-tuned dense 768 recall@5 +0.080 [+0.033, +0.127]
synthetic TEST, full archive base dense 768 -> fine-tuned dense 768 mrr@10 +0.095 [+0.053, +0.141]
synthetic TEST, full archive base + BM25 (RRF) -> fine-tuned + BM25 (RRF) recall@5 +0.027 [+0.000, +0.060]
synthetic TEST, full archive base + BM25 (RRF) -> fine-tuned + BM25 (RRF) mrr@10 +0.031 [-0.001, +0.065]
GOLD (teacher-drafted, UNREVIEWED) base dense 768 -> fine-tuned dense 768 recall@5 +0.000 [-0.130, +0.130]
GOLD (teacher-drafted, UNREVIEWED) base dense 768 -> fine-tuned dense 768 mrr@10 +0.031 [-0.021, +0.099]
GOLD (teacher-drafted, UNREVIEWED) base + BM25 (RRF) -> fine-tuned + BM25 (RRF) recall@5 +0.130 [+0.000, +0.261]
GOLD (teacher-drafted, UNREVIEWED) base + BM25 (RRF) -> fine-tuned + BM25 (RRF) mrr@10 +0.083 [+0.014, +0.164]