Instructions to use moondream/clef-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use moondream/clef-lora with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Tighten model card
Browse files
README.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
-
base_model:
|
|
|
|
|
|
|
| 4 |
library_name: peft
|
| 5 |
tags:
|
| 6 |
- clef
|
|
@@ -10,111 +12,77 @@ tags:
|
|
| 10 |
|
| 11 |
# clef-lora
|
| 12 |
|
| 13 |
-
The
|
| 14 |
-
[Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash), recovered from their merged weights.
|
| 15 |
|
| 16 |
-
| Path | Model | Base | Adapter |
|
| 17 |
-
|---|---|---|---|
|
| 18 |
-
| `/`
|
| 19 |
-
| `clef-flash/` | Clef-flash (9B) | `Qwen/Qwen3.5-9B` |
|
| 20 |
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
## Clef
|
| 24 |
-
|
| 25 |
-
Cloudflare describes Clef as Qwen3.8-27B with a frozen backbone, rank-256 LoRA adapters trained jointly with a
|
| 26 |
-
joint schema head, and the adapters merged into the released bf16 weights. This repo is that adapter, taken back
|
| 27 |
-
out, so Clef can run as a base model plus an adapter (for example on a quantized base, or next to other adapters).
|
| 28 |
-
|
| 29 |
-
| | |
|
| 30 |
-
|---|---|
|
| 31 |
-
| Source | `Cloudflare/clef` @ `ed3eed331870db2eff4b0db01237128ede8a00ce` |
|
| 32 |
-
| Base | `Qwen/Qwen3.8-27B` @ `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` |
|
| 33 |
-
| Format | PEFT LoRA, bf16, `r=256`, `lora_alpha=256` (scale 1); the 36 Gated DeltaNet `in_proj_a`/`in_proj_b` modules have `r=48` via `rank_pattern`/`alpha_pattern` |
|
| 34 |
-
| Modules | 186 linears in layers 40–63 only: `linear_attn.{in_proj_qkv,in_proj_z,in_proj_a,in_proj_b,out_proj}`, `self_attn.{q,k,v,o}_proj`, `mlp.{gate,up,down}_proj` |
|
| 35 |
-
|
| 36 |
-
## How it was extracted
|
| 37 |
-
|
| 38 |
-
`delta = W_clef - W_base` for every tensor (fp32). 186 of 1,184 tensors differ, all in layers 40–63; layers 0–39,
|
| 39 |
-
embeddings, `lm_head`, norms, GDN `conv1d`/`A_log`/`dt_bias` and the whole vision tower are byte-identical to the
|
| 40 |
-
base. Every delta has effective rank exactly 256 (48 for the rank-48 gate projections) with the 257th singular value
|
| 41 |
-
at the predicted bf16 merge-rounding floor (median 1.08×, max 2.04×), so the adapter is recoverable essentially
|
| 42 |
-
exactly: `lora_B = U sqrt(S)`, `lora_A = sqrt(S) V^T` from the top-256 SVD (exact, fp32, on GPU). The rank-256 fit leaves 0.2–4.0% relative
|
| 43 |
-
residual per tensor (median 2.1%), which matches the expected merge rounding noise (residual / expected noise:
|
| 44 |
-
median 0.97, max 1.01). `extraction.json` has the per-module residuals.
|
| 45 |
-
|
| 46 |
-
## Verification
|
| 47 |
-
|
| 48 |
-
Qwen3.8-27B (bf16) + this adapter + Clef's own head, against Cloudflare/clef, paired on the same requests:
|
| 49 |
-
|
| 50 |
-
| | Answers changed | Records fully matching | Max KL |
|
| 51 |
-
|---|---:|---:|---:|
|
| 52 |
-
| Decision Index 0.2.1 dev subset (6,373 requests, 30,950 questions) | 0.09% | 99.6% | 0.0035 |
|
| 53 |
-
| Vision suite (11 benchmarks, 2,400 rows) | 0.12% | 99.9% | 0.0024 |
|
| 54 |
-
|
| 55 |
-
That is bf16 noise: identical arithmetic in a different order (merged weight vs unmerged adapter).
|
| 56 |
|
| 57 |
## Usage
|
| 58 |
|
| 59 |
-
The adapter
|
| 60 |
-
and its encoding code (`joint_schema_model.py`) come from `Cloudflare/clef`:
|
| 61 |
|
| 62 |
```python
|
| 63 |
-
import json, sys
|
| 64 |
from pathlib import Path
|
| 65 |
from huggingface_hub import snapshot_download
|
| 66 |
from peft import PeftModel
|
| 67 |
from safetensors.torch import load_file
|
| 68 |
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
|
| 69 |
-
import torch
|
| 70 |
|
| 71 |
clef = Path(snapshot_download("Cloudflare/clef", revision="ed3eed331870db2eff4b0db01237128ede8a00ce",
|
| 72 |
allow_patterns=["*.py", "*.json", "*.jinja", "joint_head.safetensors"]))
|
| 73 |
sys.path.insert(0, str(clef))
|
| 74 |
import joint_schema_model as jsm
|
| 75 |
|
| 76 |
-
backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
|
| 77 |
-
|
| 78 |
backbone = PeftModel.from_pretrained(backbone, "moondream/clef-lora")
|
| 79 |
head = jsm.JointSchemaHead(**json.loads((clef / "joint_head_config.json").read_text()))
|
| 80 |
head.load_state_dict(load_file(clef / "joint_head.safetensors"))
|
| 81 |
model = jsm.ClefModel(backbone, head.to("cuda", torch.bfloat16)).eval()
|
| 82 |
processor = AutoProcessor.from_pretrained(clef)
|
|
|
|
| 83 |
print(jsm.systemone(model, processor, {"model": "clef", "state": "Checkout is down.",
|
| 84 |
"questions": {"outage": {"type": "noul", "instructions": "Is a service down?"}}}))
|
| 85 |
```
|
| 86 |
|
| 87 |
-
|
| 88 |
-
|
| 89 |
|
| 90 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 91 |
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
progress.
|
| 96 |
|
| 97 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
| Base | `Qwen/Qwen3.5-9B` @ `c202236235762e1c871ad0ccb60c8ee5ba337b9a` |
|
| 103 |
-
| Format | PEFT LoRA, bf16, `r=256`, `lora_alpha=256` (scale 1); the 48 Gated DeltaNet `in_proj_a`/`in_proj_b` modules have `r=32` (their smaller dimension) via `rank_pattern`/`alpha_pattern` |
|
| 104 |
-
| Modules | 248 linears in all 32 layers: `linear_attn.{in_proj_qkv,in_proj_z,in_proj_a,in_proj_b,out_proj}`, `self_attn.{q,k,v,o}_proj`, `mlp.{gate,up,down}_proj` |
|
| 105 |
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
matching the expected rounding noise (residual / expected noise: median 0.96, max 0.99). Embeddings, `lm_head`,
|
| 109 |
-
norms and the vision tower are unchanged. (The base stores its 24 GDN gated-norm weights in float32 and Clef-flash
|
| 110 |
-
stores them in bf16; they are identical after casting, so they are not part of the adapter.)
|
| 111 |
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
prompts are mostly low-confidence. Usage is as for Clef, with `Cloudflare/clef-flash` @ `8b2e5fd1…` for the head and
|
| 116 |
-
code, `Qwen/Qwen3.5-9B` @ `c2022362…` as the backbone, and `subfolder="clef-flash"`.
|
| 117 |
|
| 118 |
## License
|
| 119 |
|
| 120 |
-
Apache-2.0,
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
base_model:
|
| 4 |
+
- Qwen/Qwen3.8-27B
|
| 5 |
+
- Qwen/Qwen3.5-9B
|
| 6 |
library_name: peft
|
| 7 |
tags:
|
| 8 |
- clef
|
|
|
|
| 12 |
|
| 13 |
# clef-lora
|
| 14 |
|
| 15 |
+
The LoRA adapters inside Cloudflare's Clef models, extracted from their merged weights.
|
|
|
|
| 16 |
|
| 17 |
+
| Path | Model | Source | Base | Adapter |
|
| 18 |
+
|---|---|---|---|---|
|
| 19 |
+
| `/` | Clef (27B) | `Cloudflare/clef@ed3eed33` | `Qwen/Qwen3.8-27B@1d4bf0f2` | 186 linears, layers 40–63 |
|
| 20 |
+
| `clef-flash/` | Clef-flash (9B) | `Cloudflare/clef-flash@8b2e5fd1` | `Qwen/Qwen3.5-9B@c2022362` | 248 linears, all 32 layers |
|
| 21 |
|
| 22 |
+
Both are PEFT LoRAs in bf16, `r=256`, `lora_alpha=256`. The Gated DeltaNet `in_proj_a`/`in_proj_b` modules use
|
| 23 |
+
their full width instead (48 for Clef, 32 for Clef-flash), set via `rank_pattern`/`alpha_pattern`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
|
| 25 |
## Usage
|
| 26 |
|
| 27 |
+
The adapter replaces the backbone only. The decision head and encoding code come from the Cloudflare repo.
|
|
|
|
| 28 |
|
| 29 |
```python
|
| 30 |
+
import json, sys, torch
|
| 31 |
from pathlib import Path
|
| 32 |
from huggingface_hub import snapshot_download
|
| 33 |
from peft import PeftModel
|
| 34 |
from safetensors.torch import load_file
|
| 35 |
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
|
|
|
|
| 36 |
|
| 37 |
clef = Path(snapshot_download("Cloudflare/clef", revision="ed3eed331870db2eff4b0db01237128ede8a00ce",
|
| 38 |
allow_patterns=["*.py", "*.json", "*.jinja", "joint_head.safetensors"]))
|
| 39 |
sys.path.insert(0, str(clef))
|
| 40 |
import joint_schema_model as jsm
|
| 41 |
|
| 42 |
+
backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
|
| 43 |
+
"Qwen/Qwen3.8-27B", revision="1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", dtype=torch.bfloat16, device_map={"": "cuda"})
|
| 44 |
backbone = PeftModel.from_pretrained(backbone, "moondream/clef-lora")
|
| 45 |
head = jsm.JointSchemaHead(**json.loads((clef / "joint_head_config.json").read_text()))
|
| 46 |
head.load_state_dict(load_file(clef / "joint_head.safetensors"))
|
| 47 |
model = jsm.ClefModel(backbone, head.to("cuda", torch.bfloat16)).eval()
|
| 48 |
processor = AutoProcessor.from_pretrained(clef)
|
| 49 |
+
|
| 50 |
print(jsm.systemone(model, processor, {"model": "clef", "state": "Checkout is down.",
|
| 51 |
"questions": {"outage": {"type": "noul", "instructions": "Is a service down?"}}}))
|
| 52 |
```
|
| 53 |
|
| 54 |
+
For Clef-flash, use `Cloudflare/clef-flash@8b2e5fd1`, `Qwen/Qwen3.5-9B@c2022362` and `subfolder="clef-flash"`.
|
| 55 |
+
Install `flash-linear-attention` and `causal-conv1d`; without them transformers uses a much slower path.
|
| 56 |
|
| 57 |
+
## Extraction
|
| 58 |
+
|
| 59 |
+
- `delta = W_finetuned - W_base` per tensor in fp32, with the base cast to the fine-tune's dtype first.
|
| 60 |
+
- Only the linears listed above change. Embeddings, `lm_head`, norms and the vision tower match the base.
|
| 61 |
+
- Each delta is exactly rank 256 above the bf16 merge-rounding floor. An exact SVD gives `B = U√S`, `A = √S Vᵀ`.
|
| 62 |
+
- The rank-256 fit leaves 0.2–4% relative error per tensor, matching the expected rounding noise.
|
| 63 |
+
`extraction.json` lists it per module.
|
| 64 |
|
| 65 |
+
## Verification
|
| 66 |
+
|
| 67 |
+
Base + adapter + the release's own head against the released model, on the same requests:
|
|
|
|
| 68 |
|
| 69 |
+
| Model | Test set | Answers changed | Max KL |
|
| 70 |
+
|---|---|---:|---:|
|
| 71 |
+
| Clef | Decision Index 0.2.1, 6,373 requests (30,950 questions) | 0.09% | 0.0035 |
|
| 72 |
+
| Clef | Vision, 11 benchmarks (2,400 rows) | 0.12% | 0.0024 |
|
| 73 |
+
| Clef-flash | 120 synthetic requests (231 questions) | 0.43% (1) | 0.0059 |
|
| 74 |
|
| 75 |
+
This is bf16 rounding noise: the same arithmetic as the merged weights, done in a different order.
|
| 76 |
+
|
| 77 |
+
## On an FP8 base
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
+
The Clef adapter, unchanged, on `Qwen/Qwen3.8-27B-FP8` with FP8 weights and activations emulated in bf16 on H200.
|
| 80 |
+
Paired on 27,231 Decision Index requests (114,220 gold-scored questions):
|
|
|
|
|
|
|
|
|
|
| 81 |
|
| 82 |
+
- Accuracy: no significant change (net −0.012% of questions; mean −0.05 points per benchmark).
|
| 83 |
+
- Calibration: unchanged.
|
| 84 |
+
- 1.15% of answers flip, almost all near-ties, in both directions.
|
|
|
|
|
|
|
| 85 |
|
| 86 |
## License
|
| 87 |
|
| 88 |
+
Apache-2.0, following the source models and their bases.
|