Post
143
Jev-style decisions that read images: Jev-Style-2B-Decision-v4
A state (text, JSON, and now images) and typed questions in (pick one, yes/no, score); a calibrated probability for every option out, in one pass.
Try it in your browser:
chaoliangUNSW/jev-style-2b
• Decision Index 0.3 public index: 44.37, up from 34.05 for our 2B v3 on the board (one complete self-run with the kit's own runner and scorer: 140,178 requests, all answered). Every area is up. Submitted for review: https://github.com/apolinario/decision-index/pull/319
• Decision Index Vision public index: 57.55 (one complete self-run, 12,210 rows, all answered). The first Jev-Style model that reads images: photos, screenshots, receipts, charts.
• JevBench v1.4.1, 231 public items (self-run with the official harness, not an official board entry): 74.9% (173/231); v3 73.6%.
• Same answer when the options are listed in reverse order: 94.0% of 705 pairs from our held-out rows (v3: 88.0% of 643).
• 32,768 tokens per call (v3: 25,600), image tokens included; nothing truncated.
• 16 ms median per request (750 Decision Index requests, one at a time, A100, bfloat16 with CUDA graphs).
The board's headline Full score comes from the maintainers' private tests, so these public numbers are not a board rank. Several benchmarks' official train splits are in the training data (listed on the card), so those results are not zero-shot. Trained in part on soft labels distilled from TypeSafe Jev 1.14 via its public API. Needs the shipped runtimes (PyTorch; the GGUF and MLX builds are text only).
Models (Apache-2.0; some training data has restrictive or unclear terms and includes OpenAI GPT and Anthropic Claude outputs, see "Training data and licences" on the main card):
chaoliangUNSW/Jev-Style-2B-Decision-v4
All v4 builds: https://huggingface.co/collections/chaoliangUNSW/jev-style-decision-v4-2b
Not affiliated with TypeSafe or Jev.
A state (text, JSON, and now images) and typed questions in (pick one, yes/no, score); a calibrated probability for every option out, in one pass.
Try it in your browser:
chaoliangUNSW/jev-style-2b
• Decision Index 0.3 public index: 44.37, up from 34.05 for our 2B v3 on the board (one complete self-run with the kit's own runner and scorer: 140,178 requests, all answered). Every area is up. Submitted for review: https://github.com/apolinario/decision-index/pull/319
• Decision Index Vision public index: 57.55 (one complete self-run, 12,210 rows, all answered). The first Jev-Style model that reads images: photos, screenshots, receipts, charts.
• JevBench v1.4.1, 231 public items (self-run with the official harness, not an official board entry): 74.9% (173/231); v3 73.6%.
• Same answer when the options are listed in reverse order: 94.0% of 705 pairs from our held-out rows (v3: 88.0% of 643).
• 32,768 tokens per call (v3: 25,600), image tokens included; nothing truncated.
• 16 ms median per request (750 Decision Index requests, one at a time, A100, bfloat16 with CUDA graphs).
The board's headline Full score comes from the maintainers' private tests, so these public numbers are not a board rank. Several benchmarks' official train splits are in the training data (listed on the card), so those results are not zero-shot. Trained in part on soft labels distilled from TypeSafe Jev 1.14 via its public API. Needs the shipped runtimes (PyTorch; the GGUF and MLX builds are text only).
Models (Apache-2.0; some training data has restrictive or unclear terms and includes OpenAI GPT and Anthropic Claude outputs, see "Training data and licences" on the main card):
chaoliangUNSW/Jev-Style-2B-Decision-v4
All v4 builds: https://huggingface.co/collections/chaoliangUNSW/jev-style-decision-v4-2b
Not affiliated with TypeSafe or Jev.