Blame for README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: Qwen/Qwen3.5-0.8B
base_model_relation: finetune
tags:
- clef
- systemone
- decision-model
- distillation
- qwen3.5
- image-text-to-typed-output
- multimodal
- structured-output
- classification
- custom-code
- fluidinference
---

# clef-vision-0.8b

A 0.92B-parameter **image decision model**, distilled from Cloudflare's
[Clef-flash](https://huggingface.co/Cloudflare/clef-flash) (Qwen3.5-9B) into a
[Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) backbone with Clef's joint schema head.
Same contract and API as Clef / Jev / SystemOne: a `state` (text or JSON), optional images, and typed questions
(`noul` true/false, `choice`, ordered `score`) go in; **one probability for every allowed option of every
question** comes out of a single forward pass. Nothing is generated.

This is the PyTorch release (any CUDA / CPU device, e.g. Jetson). The Apple-silicon build is
[FluidInference/clef-vision-0.8b-coreml](https://huggingface.co/FluidInference/clef-vision-0.8b-coreml).

## Files

The layout matches Cloudflare's Clef releases, so Clef's own code loads it unchanged.

| File | Purpose |
|---|---|
| `model.safetensors`, `config.json`, `generation_config.json` | Qwen3.5-0.8B backbone with the distillation LoRA merged in, including the vision encoder |
| `joint_head.safetensors`, `joint_head_config.json` | Joint schema head (hidden 1024, width 1024, 2 routing + 2 layers, 16 heads; 69M params) |
| `joint_schema_model.py` | Cloudflare's Clef module, unchanged (Apache-2.0): record encoding, batching, `load_release_model`, `systemone` |
| `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`, `processor_config.json` | Tokenizer and image processor |
| `LICENSE` | Apache-2.0 |

## Usage

Needs `torch`, `transformers` ≥ 5.10 (Qwen3.5) and `pillow`. On CUDA, install `causal-conv1d` and
`flash-linear-attention` for the fast Gated DeltaNet kernels; without them transformers falls back to slower
reference code.

```python
import sys

import torch
from huggingface_hub import snapshot_download
from PIL import Image

path = snapshot_download("FluidInference/clef-vision-0.8b")
sys.path.insert(0, path)
from joint_schema_model import collate_records, encode_record, load_release_model

model, processor = load_release_model(path, device="cuda")

record = {
    "state": {"task": "Check the camera frame."},
    "images": [Image.open("frame.jpg").convert("RGB")],
    # The student was trained at this image size; use the same on inference.
    "media_kwargs": {"size": {"shortest_edge": 16384, "longest_edge": 200704}},
    "questions": {
        "person": {"type": "noul", "instructions": "Is there a person in the image?"},
        "scene": {
            "type": "choice",
            "instructions": "Where was this taken?",
            "criteria": {"indoor": "Inside a building.", "outdoor": "Outside."},
        },
    },
}

encoded = encode_record(processor.tokenizer, record, processor=processor)
batch = collate_records([encoded], processor.tokenizer.pad_token_id, torch.device("cuda"))
with torch.inference_mode():
    logits = model(batch)[0]

for question, question_logits in zip(encoded.questions, logits):
    probabilities = question_logits.float().softmax(-1).tolist()
    print(question.question_id, dict(zip(question.option_ids, probabilities)))
```

`systemone(model, processor, request)` from the same module takes a Jev/SystemOne `POST /v1/systemone` request
body and returns the response body; see the [Clef-flash card](https://huggingface.co/Cloudflare/clef-flash) for
the full input format.

Verified by loading this exact folder with `load_release_model` (CPU, fp32): token ids identical to the reference
encoder on 4 image records, 8/8 argmax agreement and max probability difference 4.3e-3 against the shipped
student. On Apple silicon, use the Core ML build; loading this checkpoint onto MPS through `device_map` crashed
in our tests.

## Quality

Held-out test, 1,500 records / 4,094 questions over Oxford-IIIT Pets, Food-101 and COCO val images
(classification, reference matching, tile grids):

| | Teacher agreement | Gold accuracy | KL to teacher |
|---|---:|---:|---:|
| this model (0.92B) | 92.5% | **92.2%** | 0.095 |
| teacher Clef-flash 9B (MLX 8-bit) | — | 94.1% | — |

Per task (gold): COCO object presence 98%, Food-101 dish 97%, pets breed 95%, reference-match grids 96–100%,
tile grids 88–93%. The training images cover pets, food and COCO objects; expect lower accuracy on other domains
(e.g. surveillance or industrial camera feeds) until you check it on your own data.

## Training

14,246 records (train 5,648 / dev 2,167 / test 6,431, hash-split by image) labelled by Clef-flash running locally
(MLX 8-bit backbone, exact port of the head). Student: LoRA r64 on the language model (merged into this
checkpoint) + a fresh joint schema head; loss = KL(T=2) to the teacher + 0.3 × label-smoothed CE on dataset gold;
530 steps × 16 records on a 24 GB Apple M5 Pro (6.2 h). Vision tower frozen.

## Licenses

Apache-2.0 (this model, Qwen3.5-0.8B, Clef and `joint_schema_model.py` © Cloudflare). Training images:
Oxford-IIIT Pet (CC BY-SA 4.0), Food-101 (research use), COCO 2017 val (CC BY 4.0).