Note EVAL · human labels. 22 public classification/NLI/rating sets recast as choice/noul/score. Some configs (e.g. chaosnli, 100 annotators per item) carry human label distributions, so you can test calibration, not only accuracy.

Note EVAL · synthetic, soft labels. The most-used shared benchmark: one state, several typed questions answered together, gold as a probability distribution. 4 workflows, 1.2k train / 400 test. PT/ES translation: telepatia-ai/typed-decisions-pt-es.

Note EVAL/TRAIN · exact rule labels. Procedurally generated states where every answer is computed from rules stated in the state, so labels have no noise. Good for checking reasoning over state, not guessing.

Note EVAL · human gold (Mind2Web). Web-agent decisions: pick the right page element, plus a yes/no per candidate. Test and OOD splits hold out whole websites.

Note TRAIN · human labels. 500k rows from tasksource classification and multiple-choice sets, recast as decisions with paraphrased questions. Flat schema: state, question, kind, options, target.

Note TRAIN · controlled synthetic, rule labels. 12 frozen configs (routing, extraction, browser actions…) with calibration and OOD splits and detailed provenance. Same flat schema as tasksource-jev.

Note TRAIN · mixture. The Open-Jev 27B training mix: adds 101k WANLI NLI rows (human labels) and 51k new controlled rows in 8 domains.

Note TRAIN · LLM-teacher soft labels. 741k templated operational scenarios across 53 domains, for distilling small encoders. Same flat schema. Large, but states are template-generated.

Note TRAIN/EVAL · images. The only image decision set so far: 12.8k images, 8.2k of them synthetic scenes with exact labels.

Note COMPARISON · per-item predictions from Jev, Laya and others on the same 10k items. Rerun significance tests without API cost.

Note COMPARISON · Jev vs Laya on benchmarks where Jev has published numbers, run by a third party.

Note COMPARISON · Jev as a RAG reranker vs a cross-encoder and no reranking (XQuAD, SciFact), with paired CIs and calibration tables.

Note SOURCE · human labels. Not Jev-framed. 433k premise/hypothesis pairs: support, contradict or neutral. A common base for a choice question over a passage.

Note SOURCE · human labels. Not Jev-framed. 16k yes/no questions answered from a passage. Maps directly to a noul (yes/no) question.

Note SOURCE · human labels. Not Jev-framed. 13k banking queries, 77 intents. Tests choice over many fine-grained options. Parquet copy of PolyAI/banking77 (the original has no viewer).

Note SOURCE · human labels. Not Jev-framed. 128k news items in 4 topics. A simple choice question for mixes.

Note SOURCE · human labels. Not Jev-framed. 11k movie-review sentences on a 5-level sentiment scale. Useful for score or ordinal choice questions.