Image-Text-to-Text
Transformers
Safetensors
English
qwen3_5
jev
system-one
system-two
typed-decisions
decision-model
calibrated-probabilities
multimodal
vision
zero-shot
decision-index
recommendation
lora
vllm
conversational
Eval Results
Instructions to use autotrust/AutoTrust-Decider-Pro-Vision-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use autotrust/AutoTrust-Decider-Pro-Vision-27B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="autotrust/AutoTrust-Decider-Pro-Vision-27B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("autotrust/AutoTrust-Decider-Pro-Vision-27B") model = AutoModelForMultimodalLM.from_pretrained("autotrust/AutoTrust-Decider-Pro-Vision-27B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use autotrust/AutoTrust-Decider-Pro-Vision-27B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "autotrust/AutoTrust-Decider-Pro-Vision-27B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "autotrust/AutoTrust-Decider-Pro-Vision-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/autotrust/AutoTrust-Decider-Pro-Vision-27B
- SGLang
How to use autotrust/AutoTrust-Decider-Pro-Vision-27B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "autotrust/AutoTrust-Decider-Pro-Vision-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "autotrust/AutoTrust-Decider-Pro-Vision-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "autotrust/AutoTrust-Decider-Pro-Vision-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "autotrust/AutoTrust-Decider-Pro-Vision-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use autotrust/AutoTrust-Decider-Pro-Vision-27B with Docker Model Runner:
docker model run hf.co/autotrust/AutoTrust-Decider-Pro-Vision-27B
Add blog text
Browse files- blog/JEV-27B-VL.md +110 -0
blog/JEV-27B-VL.md
ADDED
|
@@ -0,0 +1,110 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# JEV-27B-VL: a decision model that learned to see without a single image of training
|
| 2 |
+
|
| 3 |
+
*AutoTrust · 30 September 2026*
|
| 4 |
+
|
| 5 |
+
Today we are releasing **[autotrust/JEV-27B-VL](https://huggingface.co/autotrust/JEV-27B-VL)**, JEV-27B with vision. Show it
|
| 6 |
+
images and text, ask a question, and it answers in one forward pass with a calibrated probability for every option. When a
|
| 7 |
+
question needs more thought, the same model thinks it through step by step, and it can look at the images while it does.
|
| 8 |
+
|
| 9 |
+
Its decision head never saw an image during training. It still makes image decisions well enough to compete with a
|
| 10 |
+
recommender built from tens of thousands of users' behaviour.
|
| 11 |
+
|
| 12 |
+
**TL;DR**
|
| 13 |
+
|
| 14 |
+
* **Zero-shot image recommendation:** looking only at video covers, JEV-27B-VL ranks what a user will watch next as well as
|
| 15 |
+
collaborative filtering learned from **59,045 users' watch histories** (AUC 0.727 vs 0.728), with a higher top-5 hit rate
|
| 16 |
+
(59% vs 49%), and without any interaction data.
|
| 17 |
+
* **Pictures beat words:** covers alone score far better than titles alone (AUC 0.727 vs 0.649).
|
| 18 |
+
* **Everything JEV-27B does on text, it still does:** it gives the same text decisions. That includes a **RewardBench score of
|
| 19 |
+
89.9** (ahead of GPT-4o, Gemini 1.5 Pro and Claude 3.5 Sonnet used as judges), catching its own hallucinations (71% → 96%
|
| 20 |
+
accuracy on the questions it chooses to answer) and zero-shot news recommendation that beats rankers trained on the data.
|
| 21 |
+
* **Open weights**, Apache-2.0, one command to serve.
|
| 22 |
+
|
| 23 |
+
## One model, two ways to answer, now with eyes
|
| 24 |
+
|
| 25 |
+
| | what you get | typical time |
|
| 26 |
+
|---|---|---|
|
| 27 |
+
| **System 1** | a yes/no, pick-one or 0-5 decision over text **and images**, as a calibrated probability for every option | about 0.1 s |
|
| 28 |
+
| **System 2** | the full 27B model thinking step by step, with image input | seconds |
|
| 29 |
+
|
| 30 |
+
Nothing to parse and no prompt gymnastics: you ask *"Is this scenario one where the user clicks this video?"* and get
|
| 31 |
+
`{"true": 0.83, "false": 0.17}`.
|
| 32 |
+
|
| 33 |
+
## It sees: zero-shot image recommendation
|
| 34 |
+
|
| 35 |
+
We tested it on [MicroLens](https://github.com/westlake-repl/MicroLens), a public dataset of real users of a short-video app
|
| 36 |
+
with the original cover images. For each of 200 users we hid the video they actually watched next among 19 videos other people
|
| 37 |
+
were watching at the same time, which is what a feed would have shown. JEV-27B-VL looked at the covers of the last five videos
|
| 38 |
+
the user watched and at each candidate cover, and ranked the 20 candidates.
|
| 39 |
+
|
| 40 |
+

|
| 41 |
+
|
| 42 |
+
| method | uses interaction data? | AUC | hit in top 5 |
|
| 43 |
+
|---|---|---:|---:|
|
| 44 |
+
| random order | no | 0.489 | 21% |
|
| 45 |
+
| title similarity | no | 0.602 | 39% |
|
| 46 |
+
| JEV-27B-VL reading titles only | no | 0.649 | 46% |
|
| 47 |
+
| **JEV-27B-VL looking at covers only** | **no** | **0.727** | **59%** |
|
| 48 |
+
| collaborative filtering | yes, 59,045 users | 0.728 | 49% |
|
| 49 |
+
|
| 50 |
+
Three things stand out:
|
| 51 |
+
|
| 52 |
+
* **It matches collaborative filtering with zero behaviour data.** Collaborative filtering needs to know who watched what. JEV
|
| 53 |
+
only needs the pictures.
|
| 54 |
+
* **New videos are covered from the first second.** A video nobody has watched yet has no co-watch history, but it has a cover.
|
| 55 |
+
* **Looking is better than reading.** The covers carry more of a user's taste than the titles do (+0.078 AUC, statistically
|
| 56 |
+
significant).
|
| 57 |
+
|
| 58 |
+
A typical example: a user had been watching manga-commentary and dubbing clips, including part 1.6 of one manga-commentary
|
| 59 |
+
series. Among 20 candidates, JEV-27B-VL put part 1.9 of the same series first, with a click probability of 1.00. That is the
|
| 60 |
+
video the user watched next. It recognised the series from the cover art alone, and scored all 20 covers in 2.6 seconds.
|
| 61 |
+
|
| 62 |
+
The same one-pass decisions work for any question about an image. A cover of a big plate of food gets *"shows food or
|
| 63 |
+
cooking" = 1.00*; a game screenshot gets *"video game" = 1.00* out of four categories.
|
| 64 |
+
|
| 65 |
+
## It still does everything JEV-27B does
|
| 66 |
+
|
| 67 |
+
JEV-27B-VL gives the same text decisions as [JEV-27B](https://huggingface.co/autotrust/JEV-27B) (same top option on 24 of 24
|
| 68 |
+
reference decisions). These results were measured with JEV-27B:
|
| 69 |
+
|
| 70 |
+
**A judge that beats frontier models used as judges.** On RewardBench, 2,985 human-verified pairs of a better and a worse
|
| 71 |
+
answer, System 1 scores **89.9** in one forward pass per ordering, ahead of the published results for Gemini 1.5 Pro (88.2),
|
| 72 |
+
GPT-4o (86.7) and Claude 3.5 Sonnet (84.2). It is as accurate as the same model thinking before it judges (0.900 vs 0.902
|
| 73 |
+
on a 400-pair sample), and about six times faster.
|
| 74 |
+
|
| 75 |
+

|
| 76 |
+
|
| 77 |
+
**It knows when an answer is made up.** System 2 answers 1,000 trivia questions and gets 71% right. System 1 then checks each
|
| 78 |
+
answer. Answering only the half it trusts most gives **96% accuracy**; ranking by the model's own stated confidence gets only
|
| 79 |
+
87%. When System 1 says 0.97 or more, the answer was right 99% of the time.
|
| 80 |
+
|
| 81 |
+

|
| 82 |
+
|
| 83 |
+
**Zero-shot news recommendation.** On real MSN News users (MIND), reading the headlines a user clicked, JEV scores an AUC of
|
| 84 |
+
**0.642** with no training, above every zero-shot baseline (best 0.606) and above LightGBM rankers trained on the dataset
|
| 85 |
+
(0.590 and 0.616). Added to a trained ranker as one feature, it is worth as much as four times more training data.
|
| 86 |
+
|
| 87 |
+

|
| 88 |
+
|
| 89 |
+
**Search re-ranking** above a dedicated re-ranker: nDCG@10 **0.858** on TREC-COVID, against 0.793 for bge-reranker-v2-m3 and
|
| 90 |
+
0.623 for BM25.
|
| 91 |
+
|
| 92 |
+
**Thinks only when it needs to.** System 1 answers when it is confident and hands the rest to System 2. With a 0.70
|
| 93 |
+
threshold, 70% of questions are answered in 0.11 s and accuracy rises from 0.792 to **0.892**, close to thinking on every
|
| 94 |
+
question (0.917).
|
| 95 |
+
|
| 96 |
+

|
| 97 |
+
|
| 98 |
+
## Try it
|
| 99 |
+
|
| 100 |
+
```bash
|
| 101 |
+
hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL
|
| 102 |
+
bash JEV-27B-VL/serve.sh # vLLM, one GPU with 80 GB or more
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
The [model card](https://huggingface.co/autotrust/JEV-27B-VL) has copy-paste code for image decisions and image chat. All the
|
| 106 |
+
experiments above, with code and an interactive web app (image recommendation, news recommendation, judge, hallucination
|
| 107 |
+
guard and more), are in **[JEV-27B-DEMO](https://github.com/yuhai-china/JEV-27B-DEMO)**.
|
| 108 |
+
|
| 109 |
+
Image decisions are zero-shot, and the recommendation result above comes from one dataset of 200 users. We would love to hear
|
| 110 |
+
what JEV-27B-VL does on your images.
|