| --- |
| license: apache-2.0 |
| language: [en, zh] |
| tags: [decision-model, judge, reranker, calibration, system-one] |
| pipeline_tag: text-classification |
| --- |
| # llav — the open decision model (v0.4 text · v0.5-preview image & audio) |
|
|
| **llav** turns a fuzzy `if` in software into a typed call: give it a *state* (ticket, passage, image, audio clip) and a yes/no *question*, get back a calibrated probability. It speaks the System One protocol (drop-in for Jev-style clients) and is served at **https://llav.app** (free tier: 10,000 decisions / day / key). |
|
|
| This card describes the models and publishes the benchmark numbers. **Weights are not in this repository yet**: the text judge weights (`llav-0.4-round0`, `llav-0.4-rounds`) and the image/audio adapters open with the v0.5 release. Code, training and evaluation scripts are open now at **https://github.com/llachat/llav**. |
|
|
| ## Models |
| | Model | Base | What it judges | |
| |---|---|---| |
| | llav-0.4 text judge | Qwen3-Reranker-4B (LoRA, weight-averaged) | evidence / relevance / yes-no questions about text | |
| | llav image judge (0.5-preview) | Qwen3-VL-Reranker-8B (LoRA) | yes/no questions about an image (one or two photos) | |
| | llav audio judge (0.5-preview) | Qwen2.5-Omni-3B thinker (LoRA, bf16) | emotion / intent / sound questions about a clip | |
|
|
| ## Numbers (measured by us, same candidates, same day; details and caveats at https://llav.app/benchmarks) |
| | Benchmark | llav | Jev 1.14 | Gemini Decision 1 | |
| |---|---|---|---| |
| | Text judge, MuSiQue held-out q50–200 (AUC) | **0.970** single 4B · 0.977 five-model ensemble | 0.969 | 0.966 | |
| | Calibration, round 0 (Brier / ECE, lower is better) | **0.0095 / 0.004** | 0.0113 / 0.015 | 0.0118 / 0.031 | |
| | Image judge, blind set (DocVQA / ChartQA / MMStar, 3,120 items, AUC) | 0.830 | **0.984** | 0.962 | |
| | Audio judge, blind set (CREMA-D emotion, 1,200 clips, AUC) | 0.697 | 0.455 | **0.714** | |
|
|
| Image and audio numbers are blind sets (tasks llav never saw in training). On sets drawn from llav's own training sources it scores 0.950 (image) and 0.952 (audio). |
|
|
| ## Use |
| ```bash |
| curl https://api.llav.app/v1/systemone -H "Authorization: Bearer $LLAV_KEY" -H "Content-Type: application/json" \ |
| -d '{"state":"Customer: the parcel never arrived and I want my money back.","questions":{"refund":{"type":"noul","instructions":"Is the customer asking for a refund?"}}}' |
| ``` |
| Self-hosting (MLX on Apple silicon, or one NVIDIA GPU) is documented in the GitHub repo; weights follow with v0.5. |
|
|
| ## License |
| Apache-2.0 for code and the released weights. Benchmarks reuse public datasets under their own licenses. |
|
|
