Out-of-domain calibration

#1
by motazalqaoud - opened

Really nice that the card splits in-domain and out-of-domain results. Reading A.1, the gap to liquid-d1-ultra is about 17 points in domain (87.8 vs 71.2) but about 1 point on the 9 out-of-domain sets (70.6 vs 69.3), and ECE-15 goes from 0.021 to 0.109 there while liquid-d1-ultra stays at 0.035. Since the in-domain sets are held-out groups of sources MANTA trained on, I'd guess that's where most of the headline gap comes from. Have you tried something like temperature scaling on a small sample from a new dataset? That out-of-domain calibration seems like the number that matters most for anyone trying it on their own data.

Sign up or log in to comment