### synthetic TEST, held-out papers only (n=150 queries) | method | recall@1 | recall@5 | recall@10 | MRR@10 | |---|---|---|---|---| | BM25 | 0.800 | 0.967 | 1.000 | 0.874 | | base dense 768 | 0.713 | 0.967 | 0.993 | 0.811 | | base dense 512 | 0.693 | 0.967 | 0.993 | 0.807 | | base dense 256 | 0.720 | 0.947 | 0.987 | 0.814 | | base dense 128 | 0.687 | 0.900 | 0.987 | 0.790 | | base + BM25 (RRF) | 0.787 | 0.987 | 1.000 | 0.873 | | fine-tuned dense 768 | 0.793 | 0.993 | 1.000 | 0.871 | | fine-tuned dense 512 | 0.807 | 0.987 | 1.000 | 0.880 | | fine-tuned dense 256 | 0.800 | 0.973 | 0.993 | 0.871 | | fine-tuned dense 128 | 0.800 | 0.967 | 0.993 | 0.865 | | fine-tuned + BM25 (RRF) | 0.813 | 1.000 | 1.000 | 0.891 | ### synthetic TEST, full archive (n=150 queries) | method | recall@1 | recall@5 | recall@10 | MRR@10 | |---|---|---|---|---| | BM25 | 0.647 | 0.900 | 0.940 | 0.752 | | base dense 768 | 0.527 | 0.840 | 0.893 | 0.652 | | base dense 512 | 0.513 | 0.840 | 0.900 | 0.646 | | base dense 256 | 0.527 | 0.793 | 0.907 | 0.641 | | base dense 128 | 0.453 | 0.767 | 0.847 | 0.589 | | base + BM25 (RRF) | 0.673 | 0.913 | 0.967 | 0.776 | | fine-tuned dense 768 | 0.647 | 0.920 | 0.953 | 0.747 | | fine-tuned dense 512 | 0.653 | 0.907 | 0.967 | 0.752 | | fine-tuned dense 256 | 0.627 | 0.893 | 0.960 | 0.734 | | fine-tuned dense 128 | 0.627 | 0.833 | 0.920 | 0.720 | | fine-tuned + BM25 (RRF) | 0.713 | 0.940 | 0.987 | 0.807 | ### GOLD (teacher-drafted, UNREVIEWED) (n=23 queries) | method | recall@1 | recall@5 | recall@10 | MRR@10 | |---|---|---|---|---| | BM25 | 0.304 | 0.609 | 0.739 | 0.436 | | base dense 768 | 0.391 | 0.609 | 0.739 | 0.506 | | base dense 512 | 0.391 | 0.696 | 0.696 | 0.504 | | base dense 256 | 0.348 | 0.609 | 0.696 | 0.476 | | base dense 128 | 0.391 | 0.565 | 0.565 | 0.454 | | base + BM25 (RRF) | 0.261 | 0.565 | 0.696 | 0.422 | | fine-tuned dense 768 | 0.478 | 0.609 | 0.696 | 0.537 | | fine-tuned dense 512 | 0.435 | 0.652 | 0.696 | 0.523 | | fine-tuned dense 256 | 0.435 | 0.565 | 0.739 | 0.509 | | fine-tuned dense 128 | 0.391 | 0.522 | 0.609 | 0.454 | | fine-tuned + BM25 (RRF) | 0.391 | 0.696 | 0.696 | 0.505 | ### Fine-tuned minus base (paired bootstrap, 95% CI) | query set | comparison | metric | delta | 95% CI | |---|---|---|---|---| | synthetic TEST, held-out papers only | base dense 768 -> fine-tuned dense 768 | recall@5 | +0.027 | [+0.007, +0.060] | | synthetic TEST, held-out papers only | base dense 768 -> fine-tuned dense 768 | mrr@10 | +0.060 | [+0.031, +0.092] | | synthetic TEST, held-out papers only | base + BM25 (RRF) -> fine-tuned + BM25 (RRF) | recall@5 | +0.013 | [+0.000, +0.033] | | synthetic TEST, held-out papers only | base + BM25 (RRF) -> fine-tuned + BM25 (RRF) | mrr@10 | +0.018 | [-0.002, +0.037] | | synthetic TEST, full archive | base dense 768 -> fine-tuned dense 768 | recall@5 | +0.080 | [+0.033, +0.127] | | synthetic TEST, full archive | base dense 768 -> fine-tuned dense 768 | mrr@10 | +0.095 | [+0.053, +0.141] | | synthetic TEST, full archive | base + BM25 (RRF) -> fine-tuned + BM25 (RRF) | recall@5 | +0.027 | [+0.000, +0.060] | | synthetic TEST, full archive | base + BM25 (RRF) -> fine-tuned + BM25 (RRF) | mrr@10 | +0.031 | [-0.001, +0.065] | | GOLD (teacher-drafted, UNREVIEWED) | base dense 768 -> fine-tuned dense 768 | recall@5 | +0.000 | [-0.130, +0.130] | | GOLD (teacher-drafted, UNREVIEWED) | base dense 768 -> fine-tuned dense 768 | mrr@10 | +0.031 | [-0.021, +0.099] | | GOLD (teacher-drafted, UNREVIEWED) | base + BM25 (RRF) -> fine-tuned + BM25 (RRF) | recall@5 | +0.130 | [+0.000, +0.261] | | GOLD (teacher-drafted, UNREVIEWED) | base + BM25 (RRF) -> fine-tuned + BM25 (RRF) | mrr@10 | +0.083 | [+0.014, +0.164] |