llav model card (public, no weights)
Browse files
README.md
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language: [en, zh]
|
| 4 |
+
tags: [decision-model, judge, reranker, calibration, system-one]
|
| 5 |
+
pipeline_tag: text-classification
|
| 6 |
+
---
|
| 7 |
+
# llav — the open decision model (v0.4 text · v0.5-preview image & audio)
|
| 8 |
+
|
| 9 |
+
**llav** turns a fuzzy `if` in software into a typed call: give it a *state* (ticket, passage, image, audio clip) and a yes/no *question*, get back a calibrated probability. It speaks the System One protocol (drop-in for Jev-style clients) and is served at **https://llav.app** (free tier: 10,000 decisions / day / key).
|
| 10 |
+
|
| 11 |
+
This card describes the models and publishes the benchmark numbers. **Weights are not in this repository yet**: the text judge weights (`llav-0.4-round0`, `llav-0.4-rounds`) and the image/audio adapters open with the v0.5 release. Code, training and evaluation scripts are open now at **https://github.com/llachat/llav**.
|
| 12 |
+
|
| 13 |
+
## Models
|
| 14 |
+
| Model | Base | What it judges |
|
| 15 |
+
|---|---|---|
|
| 16 |
+
| llav-0.4 text judge | Qwen3-Reranker-4B (LoRA, weight-averaged) | evidence / relevance / yes-no questions about text |
|
| 17 |
+
| llav image judge (0.5-preview) | Qwen3-VL-Reranker-8B (LoRA) | yes/no questions about an image (one or two photos) |
|
| 18 |
+
| llav audio judge (0.5-preview) | Qwen2.5-Omni-3B thinker (LoRA, bf16) | emotion / intent / sound questions about a clip |
|
| 19 |
+
|
| 20 |
+
## Numbers (measured by us, same candidates, same day; details and caveats at https://llav.app/benchmarks)
|
| 21 |
+
| Benchmark | llav | Jev 1.14 | Gemini Decision 1 |
|
| 22 |
+
|---|---|---|---|
|
| 23 |
+
| Text judge, MuSiQue held-out (AUC / R@10) | **0.977 / 0.942** | 0.969 / 0.924 | 0.941 / 0.845 |
|
| 24 |
+
| Calibration, round 0 (Brier / ECE, lower is better) | **0.0095 / 0.004** | 0.0113 / 0.015 | 0.0118 / 0.031 |
|
| 25 |
+
| Image judge, 2,458 items (AUC) | 0.950 | **0.955** | 0.942 |
|
| 26 |
+
| Audio judge, 1,200 clips (AUC) | **0.952** | 0.533 | 0.842 |
|
| 27 |
+
| SAG2 end-to-end, MuSiQue (R@10) | 0.812 | **0.933** | 0.895 |
|
| 28 |
+
|
| 29 |
+
Image and audio benches are in-distribution for llav (trained on the same source datasets, different splits); blind-set results are being added.
|
| 30 |
+
|
| 31 |
+
## Use
|
| 32 |
+
```bash
|
| 33 |
+
curl https://api.llav.app/v1/systemone -H "Authorization: Bearer $LLAV_KEY" -H "Content-Type: application/json" \
|
| 34 |
+
-d '{"state":"Customer: the parcel never arrived and I want my money back.","questions":{"refund":{"type":"noul","instructions":"Is the customer asking for a refund?"}}}'
|
| 35 |
+
```
|
| 36 |
+
Self-hosting (MLX on Apple silicon, or one NVIDIA GPU) is documented in the GitHub repo; weights follow with v0.5.
|
| 37 |
+
|
| 38 |
+
## License
|
| 39 |
+
Apache-2.0 for code and the released weights. Benchmarks reuse public datasets under their own licenses.
|