| --- |
| license: apache-2.0 |
| base_model: Cloudflare/clef-flash |
| tags: |
| - system-one |
| - decision-model |
| - bitsandbytes |
| - 4-bit |
| --- |
| |
| # clef-flash-4bit |
|
|
| A pre-quantized 4-bit copy of [Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash), a 9B System One decision model (Qwen3.5-9B backbone + joint schema head) with a Jev/SystemOne-compatible interface. It exists so that GPUs without bf16 and with little memory (for example a Colab T4) can download about a third of the original size and load the model without quantizing it on the fly. |
|
|
| - **Backbone:** quantized with bitsandbytes NF4, compute dtype float16 (the quantization settings are stored in `config.json`). |
| - **Joint schema head, loader code (`joint_schema_model.py`), processor and tokenizer:** copied unchanged from the original repository. |
| - **License:** Apache-2.0, like the original. All credit for the model goes to Cloudflare; see the [original model card](https://huggingface.co/Cloudflare/clef-flash) for its description, evaluations and limitations. |
|
|
| ## Usage |
|
|
| Exactly as the original, without passing a quantization config: |
|
|
| ```python |
| import sys, torch |
| from huggingface_hub import snapshot_download |
| |
| path = snapshot_download("irodkin/clef-flash-4bit") |
| sys.path.insert(0, path) |
| from joint_schema_model import load_release_model, systemone |
| |
| model, processor = load_release_model(path, device="cuda", dtype=torch.float16) |
| response = systemone(model, processor, { |
| "model": "clef-flash", |
| "state": "Customer says the parcel arrived damaged and wants their money back.", |
| "questions": {"refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"}}, |
| }) |
| print(response["answers"]) |
| ``` |
|
|
| Needs `transformers` ≥ 5.10, `bitsandbytes`, `torchvision` and `pillow`. |
|
|
| ## Check |
|
|
| Against quantizing the original on the fly with the same settings, the pre-quantized checkpoint gives identical decisions (maximum probability difference 0.0000 on 16 noul/choice questions) and loads with 7.9 GiB of GPU memory. Download size: about 8.5 GB instead of 19 GB (the embeddings and the vision encoder stay in 16-bit). Used as the retrieval controller in a course lab, it matched on-the-fly 4-bit quantization exactly (accuracy 1.00 / 1.00 / 0.87 on single-hop / 2-hop / 3-hop questions). |
|
|
