Instructions to use vantu-fit/Joe0.1-27B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use vantu-fit/Joe0.1-27B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="vantu-fit/Joe0.1-27B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("vantu-fit/Joe0.1-27B") model = AutoModelForMultimodalLM.from_pretrained("vantu-fit/Joe0.1-27B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use vantu-fit/Joe0.1-27B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "vantu-fit/Joe0.1-27B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vantu-fit/Joe0.1-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/vantu-fit/Joe0.1-27B
- SGLang
How to use vantu-fit/Joe0.1-27B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "vantu-fit/Joe0.1-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vantu-fit/Joe0.1-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "vantu-fit/Joe0.1-27B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vantu-fit/Joe0.1-27B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use vantu-fit/Joe0.1-27B with Docker Model Runner:
docker model run hf.co/vantu-fit/Joe0.1-27B
Joe0.1-27B
TL;DR
- What it is: a 27B vision-language model that answers multiple-choice questions about images. It returns a probability for every option from one forward pass (no text generation). Built for the vision board of the Jev Decision Index.
- Base model: Qwen/Qwen3.8-27B (Apache-2.0), revision
1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, read in non-thinking mode. 27B parameters, bf16, 12 safetensors shards (about 54 GB). - Score: public vision index 76.64 on the Decision Index kit (all 12,210 public rows, our own run on 1 x H200).
- Licence: research and noncommercial use only (see "Licence").
How to run it
The inference code ships in this repo: jev_vision.py holds the model, prompt and readout, and jev_vision_engine.py
is the Decision Index kit engine.
pip install -r requirements.txt
from PIL import Image
from jev_vision import JevVision
m = JevVision("vantu-fit/Joe0.1-27B") # or a local dir; about 54 GB of bf16 weights on one GPU
probs = m.decide([Image.open("photo.jpg")], "Which object is closer to the camera?", ["the chair", "the lamp"])
Decision Index kit, vision suite. This is the command of the submitted run, one request at a time:
git clone https://github.com/apolinario/decision-index && git -C decision-index checkout f5a6e4f
pip install --no-deps -e decision-index && pip install httpx pyarrow
decision-index suite rebuild --vision --work work --font "<Arial Bold .ttf>" && decision-index vision verify --dir work/vision
PYTHONPATH=<this repo dir> decision-index vision pipeline --dir work/vision \
--engine jev_vision_engine:JevVisionEngine --option model=<this repo dir or id> --workers 1 --out runs/Joe0.1-27B
The run's status.json / environment.json record the engine as jev_base_lora_engine:JevVisionEngine, a thin
module in our research repository that only re-exports jev_vision_engine:JevVisionEngine; the SHA-256 of
jev_vision.py and jev_vision_engine.py in environment.json equal the files in this repo, so a run with
jev_vision_engine:JevVisionEngine executes the same code.
What one request does
- Prompt. One fixed rendering for every question: the model's chat template with thinking turned off, the images,
the question, the options as
A. ...,B. ...lines, and a fixed start of the answer:<|im_start|>user\n{<|vision_start|><|image_pad|><|vision_end|> per image}{question}\nA. {option}\nB. {option}\n... Answer with the option's letter from the given choices directly.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\nThe answer is ( - Images. Converted to RGB. An image over 1,638,400 pixels (1.6 MP) is downscaled to fit, keeping the aspect ratio (LANCZOS). Smaller images are used at their native size.
- Attention ("noncausal reading"). The softmax-attention layers see the whole prompt in both directions (a padding
mask only); the Gated DeltaNet layers are unchanged.
base_lora.jsonswitches this on. - Readout. The last hidden state is scored against the single-token letters A..Z (in fp32), and a softmax over the first n letters (the shown options) gives the probabilities. No sampling, no temperature.
Declared limits
- Task:
choiceonly (multiple choice with lettered options). Free-form answers are not supported. - Options: 2 to 26 per question. No truncation.
- Images: each image is capped at 1.6 MP (1,638,400 px). The kit suite uses up to 2 images per row; more images work as long as they fit in GPU memory.
Evaluation
All numbers are from our own machines. The board's maintainers run the private half and measure latency themselves.
Decision Index kit, public vision suite. Kit commit f5a6e4fd, edition 0.3, engine
jev_vision_engine:JevVisionEngine, one request at a time, 1 x H200. The run is complete: 12,210 of 12,210 rows ok,
vision verify passed, 0 missing rows. Public vision index: 76.64. Calibration on the public suite (ECE over 10
confidence bins): 0.025.
| Benchmark | Rows | Accuracy | Skill | Weight |
|---|---|---|---|---|
| CV-Bench | 2,038 | 88.37 | 80.82 | 1 |
| BLINK | 961 | 83.45 | 74.39 | 1 |
| RealWorldQA | 589 | 82.51 | 71.42 | 1 |
| CharXiv | 532 | 86.65 | 82.21 | 1 |
| InfographicVQA | 244 | 99.18 | 98.91 | 1 |
| Mind2Web | 973 | 93.53 | 91.37 | 1 |
| Winoground | 800 | 93.13 | 86.25 | 1 |
| KIE (CORD+FUNSD) | 2,465 | 94.77 | 91.38 | 1 |
| Moderation (Hateful Memes) | 1,213 | 71.39 | 42.79 | 1 |
| R-Bench-M | 665 | 43.76 | 32.50 | 0.5 |
| MMMU-Pro vision | 1,730 | 52.66 | 45.70 | 0.25 |
| Public vision index | 12,210 | 76.64 |
Skill = (accuracy โ chance) / (1 โ chance) x 100: 0 is guessing, 100 is perfect. The index is the board's weighted mean
of the skills, clipped to [0, 100]. Results files: vantu-fit/Joe0.1-27B-results.
Latency (kit total_wall_ms, one request at a time, 1 x H200, 8 CPU cores, 12,210 rows): median 168 ms,
mean 217 ms, p80 337 ms, p95 449 ms, max 9.9 s. Two-image rows (BLINK): median 467 ms, p80 548 ms.
base_lora.json and the results' environment.json keep absolute paths from our training cluster; they are
informational and not read at inference (the engine loads weights from this repo).
Limitations
- Multiple choice only. The model reads one letter after
The answer is (. Free-form generation was not trained or evaluated, and noncausal reading is defined over a complete prompt, not for token-by-token decoding. - Weak benchmarks: Moderation (Hateful Memes) skill 42.8; R-Bench-M 32.5; MMMU-Pro vision 45.7.
- Numerical noise. Changing the weights by one bf16 unit in the last place changed about 100 of the 12,210 kit answers and the index by about 0.1 in our tests, so expect this much variation across GPUs and kernels.
- Unmeasured: the board's private half; latency on the board's RTX PRO 6000; GPU peak memory.
- Image cap. Images above 1.6 MP are downscaled.
- Not for safety-critical decisions. Moderation and other judgements are benchmark-level only.
Disclosures
- Benchmark images in training data. 138 training items use images that match BLINK (val / test) and CV-Bench images; 17 of these images are in the kit's public vision suite (14 CV-Bench, 3 BLINK val). The training questions on them are our own, not the benchmarks'. Some of the BLINK test images may also be in the board's private BLINK set.
- Benchmark training split. The training data includes rows from the Mind2Web train split; the board uses the
test_websitesplit, which has different websites.
Licence
- Base weights: Qwen3.8-27B is Apache-2.0; its licence text is included as
LICENSE.base-Qwen3.8-27B. - Fine-tuned weights: released for research and noncommercial use only, as is. Some training data carries
noncommercial or share-alike terms, and LLM-generated data whose providers' terms may restrict training competing
models was used. See
LICENSE_NOTICE.md. This is not legal advice.
- Downloads last month
- 77