ProbabilityBench: Model Evaluation and Calibration with Exact Ground-Truth Uncertainty

Decision models like Jev are designed to predict class probabilities directly, but how can we tell whether their predictions are actually correct?

Suppose a decision model predicts a 70% probability that an instance belongs to the positive class in a binary classification task. Even after observing the true binary label, we still don’t know whether 70% was the right probability. Standard calibration metrics don’t fully answer this question either, because they evaluate predictions in aggregate rather than for individual instances. Proper scoring rules such as log loss are useful for evaluating probabilistic predictions, but without knowing the irreducible loss, we cannot determine how much of the observed loss comes from inaccurate probability estimates.

In our new paper, ProbabilityBench, we introduce a benchmark where the true probabilities are known exactly. We design simulated worlds with full knowledge of the data- and label-generating processes, allowing us to compute the exact class probabilities for each instance. Models must infer these probabilities from natural-language descriptions without access to the underlying generating processes. This allows us to directly measure how close their predictions are to the ground-truth probabilities.

We evaluate 16 models, including decision models such as Jev and Perplexity Decider, and find that decision models often achieve strong trade-offs between probability accuracy and cost. We also show how ProbabilityBench can help evaluate existing evaluation methods, including calibration metrics and Bayes-error estimators.

Until now, evaluating predictions against exact class probabilities has often meant relying on simple synthetic settings like Gaussian distributions. If you've ever wished for something more realistic and interesting, we hope you'll give ProbabilityBench a try!