| --- |
| title: README |
| emoji: 📊 |
| colorFrom: indigo |
| colorTo: red |
| sdk: static |
| pinned: false |
| --- |
| |
| # JevBench |
|
|
| **Metamorphic coherence testing for typed probabilistic decision models.** |
|
|
| Typed decision models answer questions about a text with probability distributions over declared answers: yes or no, |
| one of several options, or a level on an ordered scale. JevBench tests whether those probabilities stay coherent when |
| a question is reworded, negated, logically combined with others, offered a different set of options, or asked |
| together with other questions: 50 metamorphic relations in five dimensions, no gold labels needed, every test frozen |
| in advance. |
|
|
| | | | |
| |---|---| |
| | 📄 Technical report | [alphaXiv](https://www.alphaxiv.org/abs/2610.jevbench-metamorphic-coherence-testing) | |
| | 🗂️ Dataset | [JevBench/jevbench](https://huggingface.co/datasets/JevBench/jevbench): 240 cases, 2,240 questions, frozen test suites and every evaluation report | |
| | 🏆 Leaderboard | [jevbench.github.io](https://jevbench.github.io): 21 openly released models | |
| | 💻 Code | [github.com/JevBench/jevbench](https://github.com/JevBench/jevbench) · `pip install jevbench` | |
|
|
| Maintained by [Chen Feng](https://mrchenfeng.github.io/), [ML Lab](https://qub-ml.github.io/), Queen's University Belfast. |
|
|
