cloudyu commited on
Commit
4c67fe1
·
verified ·
1 Parent(s): 95c0041

Add blog text

Browse files
Files changed (1) hide show
  1. blog/JEV-27B-VL.md +110 -0
blog/JEV-27B-VL.md ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # JEV-27B-VL: a decision model that learned to see without a single image of training
2
+
3
+ *AutoTrust · 30 September 2026*
4
+
5
+ Today we are releasing **[autotrust/JEV-27B-VL](https://huggingface.co/autotrust/JEV-27B-VL)**, JEV-27B with vision. Show it
6
+ images and text, ask a question, and it answers in one forward pass with a calibrated probability for every option. When a
7
+ question needs more thought, the same model thinks it through step by step, and it can look at the images while it does.
8
+
9
+ Its decision head never saw an image during training. It still makes image decisions well enough to compete with a
10
+ recommender built from tens of thousands of users' behaviour.
11
+
12
+ **TL;DR**
13
+
14
+ * **Zero-shot image recommendation:** looking only at video covers, JEV-27B-VL ranks what a user will watch next as well as
15
+ collaborative filtering learned from **59,045 users' watch histories** (AUC 0.727 vs 0.728), with a higher top-5 hit rate
16
+ (59% vs 49%), and without any interaction data.
17
+ * **Pictures beat words:** covers alone score far better than titles alone (AUC 0.727 vs 0.649).
18
+ * **Everything JEV-27B does on text, it still does:** it gives the same text decisions. That includes a **RewardBench score of
19
+ 89.9** (ahead of GPT-4o, Gemini 1.5 Pro and Claude 3.5 Sonnet used as judges), catching its own hallucinations (71% → 96%
20
+ accuracy on the questions it chooses to answer) and zero-shot news recommendation that beats rankers trained on the data.
21
+ * **Open weights**, Apache-2.0, one command to serve.
22
+
23
+ ## One model, two ways to answer, now with eyes
24
+
25
+ | | what you get | typical time |
26
+ |---|---|---|
27
+ | **System 1** | a yes/no, pick-one or 0-5 decision over text **and images**, as a calibrated probability for every option | about 0.1 s |
28
+ | **System 2** | the full 27B model thinking step by step, with image input | seconds |
29
+
30
+ Nothing to parse and no prompt gymnastics: you ask *"Is this scenario one where the user clicks this video?"* and get
31
+ `{"true": 0.83, "false": 0.17}`.
32
+
33
+ ## It sees: zero-shot image recommendation
34
+
35
+ We tested it on [MicroLens](https://github.com/westlake-repl/MicroLens), a public dataset of real users of a short-video app
36
+ with the original cover images. For each of 200 users we hid the video they actually watched next among 19 videos other people
37
+ were watching at the same time, which is what a feed would have shown. JEV-27B-VL looked at the covers of the last five videos
38
+ the user watched and at each candidate cover, and ranked the 20 candidates.
39
+
40
+ ![Zero-shot image recommendation on MicroLens](https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/blog/image_recommendation.png)
41
+
42
+ | method | uses interaction data? | AUC | hit in top 5 |
43
+ |---|---|---:|---:|
44
+ | random order | no | 0.489 | 21% |
45
+ | title similarity | no | 0.602 | 39% |
46
+ | JEV-27B-VL reading titles only | no | 0.649 | 46% |
47
+ | **JEV-27B-VL looking at covers only** | **no** | **0.727** | **59%** |
48
+ | collaborative filtering | yes, 59,045 users | 0.728 | 49% |
49
+
50
+ Three things stand out:
51
+
52
+ * **It matches collaborative filtering with zero behaviour data.** Collaborative filtering needs to know who watched what. JEV
53
+ only needs the pictures.
54
+ * **New videos are covered from the first second.** A video nobody has watched yet has no co-watch history, but it has a cover.
55
+ * **Looking is better than reading.** The covers carry more of a user's taste than the titles do (+0.078 AUC, statistically
56
+ significant).
57
+
58
+ A typical example: a user had been watching manga-commentary and dubbing clips, including part 1.6 of one manga-commentary
59
+ series. Among 20 candidates, JEV-27B-VL put part 1.9 of the same series first, with a click probability of 1.00. That is the
60
+ video the user watched next. It recognised the series from the cover art alone, and scored all 20 covers in 2.6 seconds.
61
+
62
+ The same one-pass decisions work for any question about an image. A cover of a big plate of food gets *"shows food or
63
+ cooking" = 1.00*; a game screenshot gets *"video game" = 1.00* out of four categories.
64
+
65
+ ## It still does everything JEV-27B does
66
+
67
+ JEV-27B-VL gives the same text decisions as [JEV-27B](https://huggingface.co/autotrust/JEV-27B) (same top option on 24 of 24
68
+ reference decisions). These results were measured with JEV-27B:
69
+
70
+ **A judge that beats frontier models used as judges.** On RewardBench, 2,985 human-verified pairs of a better and a worse
71
+ answer, System 1 scores **89.9** in one forward pass per ordering, ahead of the published results for Gemini 1.5 Pro (88.2),
72
+ GPT-4o (86.7) and Claude 3.5 Sonnet (84.2). It is as accurate as the same model thinking before it judges (0.900 vs 0.902
73
+ on a 400-pair sample), and about six times faster.
74
+
75
+ ![RewardBench](https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/blog/judge_rewardbench.png)
76
+
77
+ **It knows when an answer is made up.** System 2 answers 1,000 trivia questions and gets 71% right. System 1 then checks each
78
+ answer. Answering only the half it trusts most gives **96% accuracy**; ranking by the model's own stated confidence gets only
79
+ 87%. When System 1 says 0.97 or more, the answer was right 99% of the time.
80
+
81
+ ![Hallucination guard](https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/blog/hallucination_guard.png)
82
+
83
+ **Zero-shot news recommendation.** On real MSN News users (MIND), reading the headlines a user clicked, JEV scores an AUC of
84
+ **0.642** with no training, above every zero-shot baseline (best 0.606) and above LightGBM rankers trained on the dataset
85
+ (0.590 and 0.616). Added to a trained ranker as one feature, it is worth as much as four times more training data.
86
+
87
+ ![News recommendation](https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/blog/news_recommendation.png)
88
+
89
+ **Search re-ranking** above a dedicated re-ranker: nDCG@10 **0.858** on TREC-COVID, against 0.793 for bge-reranker-v2-m3 and
90
+ 0.623 for BM25.
91
+
92
+ **Thinks only when it needs to.** System 1 answers when it is confident and hands the rest to System 2. With a 0.70
93
+ threshold, 70% of questions are answered in 0.11 s and accuracy rises from 0.792 to **0.892**, close to thinking on every
94
+ question (0.917).
95
+
96
+ ![System 1 to System 2](https://huggingface.co/autotrust/JEV-27B-VL/resolve/main/blog/system1_to_system2.png)
97
+
98
+ ## Try it
99
+
100
+ ```bash
101
+ hf download autotrust/JEV-27B-VL --local-dir JEV-27B-VL
102
+ bash JEV-27B-VL/serve.sh # vLLM, one GPU with 80 GB or more
103
+ ```
104
+
105
+ The [model card](https://huggingface.co/autotrust/JEV-27B-VL) has copy-paste code for image decisions and image chat. All the
106
+ experiments above, with code and an interactive web app (image recommendation, news recommendation, judge, hallucination
107
+ guard and more), are in **[JEV-27B-DEMO](https://github.com/yuhai-china/JEV-27B-DEMO)**.
108
+
109
+ Image decisions are zero-shot, and the recommendation result above comes from one dataset of 200 users. We would love to hear
110
+ what JEV-27B-VL does on your images.