vikhyatk commited on
Commit
fa67d07
·
verified ·
1 Parent(s): 44d180f

Tighten model card

Browse files
Files changed (1) hide show
  1. README.md +41 -73
README.md CHANGED
@@ -1,6 +1,8 @@
1
  ---
2
  license: apache-2.0
3
- base_model: Qwen/Qwen3.8-27B
 
 
4
  library_name: peft
5
  tags:
6
  - clef
@@ -10,111 +12,77 @@ tags:
10
 
11
  # clef-lora
12
 
13
- The LoRAs inside [Cloudflare/clef](https://huggingface.co/Cloudflare/clef) and
14
- [Cloudflare/clef-flash](https://huggingface.co/Cloudflare/clef-flash), recovered from their merged weights.
15
 
16
- | Path | Model | Base | Adapter |
17
- |---|---|---|---|
18
- | `/` (repo root) | Clef (27B) | `Qwen/Qwen3.8-27B` | rank 256, 186 modules, layers 40–63 |
19
- | `clef-flash/` | Clef-flash (9B) | `Qwen/Qwen3.5-9B` | rank 256, 248 modules, all 32 layers |
20
 
21
- Load Clef-flash's with `PeftModel.from_pretrained(backbone, "moondream/clef-lora", subfolder="clef-flash")`.
22
-
23
- ## Clef
24
-
25
- Cloudflare describes Clef as Qwen3.8-27B with a frozen backbone, rank-256 LoRA adapters trained jointly with a
26
- joint schema head, and the adapters merged into the released bf16 weights. This repo is that adapter, taken back
27
- out, so Clef can run as a base model plus an adapter (for example on a quantized base, or next to other adapters).
28
-
29
- | | |
30
- |---|---|
31
- | Source | `Cloudflare/clef` @ `ed3eed331870db2eff4b0db01237128ede8a00ce` |
32
- | Base | `Qwen/Qwen3.8-27B` @ `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` |
33
- | Format | PEFT LoRA, bf16, `r=256`, `lora_alpha=256` (scale 1); the 36 Gated DeltaNet `in_proj_a`/`in_proj_b` modules have `r=48` via `rank_pattern`/`alpha_pattern` |
34
- | Modules | 186 linears in layers 40–63 only: `linear_attn.{in_proj_qkv,in_proj_z,in_proj_a,in_proj_b,out_proj}`, `self_attn.{q,k,v,o}_proj`, `mlp.{gate,up,down}_proj` |
35
-
36
- ## How it was extracted
37
-
38
- `delta = W_clef - W_base` for every tensor (fp32). 186 of 1,184 tensors differ, all in layers 40–63; layers 0–39,
39
- embeddings, `lm_head`, norms, GDN `conv1d`/`A_log`/`dt_bias` and the whole vision tower are byte-identical to the
40
- base. Every delta has effective rank exactly 256 (48 for the rank-48 gate projections) with the 257th singular value
41
- at the predicted bf16 merge-rounding floor (median 1.08×, max 2.04×), so the adapter is recoverable essentially
42
- exactly: `lora_B = U sqrt(S)`, `lora_A = sqrt(S) V^T` from the top-256 SVD (exact, fp32, on GPU). The rank-256 fit leaves 0.2–4.0% relative
43
- residual per tensor (median 2.1%), which matches the expected merge rounding noise (residual / expected noise:
44
- median 0.97, max 1.01). `extraction.json` has the per-module residuals.
45
-
46
- ## Verification
47
-
48
- Qwen3.8-27B (bf16) + this adapter + Clef's own head, against Cloudflare/clef, paired on the same requests:
49
-
50
- | | Answers changed | Records fully matching | Max KL |
51
- |---|---:|---:|---:|
52
- | Decision Index 0.2.1 dev subset (6,373 requests, 30,950 questions) | 0.09% | 99.6% | 0.0035 |
53
- | Vision suite (11 benchmarks, 2,400 rows) | 0.12% | 99.9% | 0.0024 |
54
-
55
- That is bf16 noise: identical arithmetic in a different order (merged weight vs unmerged adapter).
56
 
57
  ## Usage
58
 
59
- The adapter only replaces the backbone weights. Clef's decision head (`joint_head.safetensors`, `joint_head_config.json`)
60
- and its encoding code (`joint_schema_model.py`) come from `Cloudflare/clef`:
61
 
62
  ```python
63
- import json, sys
64
  from pathlib import Path
65
  from huggingface_hub import snapshot_download
66
  from peft import PeftModel
67
  from safetensors.torch import load_file
68
  from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
69
- import torch
70
 
71
  clef = Path(snapshot_download("Cloudflare/clef", revision="ed3eed331870db2eff4b0db01237128ede8a00ce",
72
  allow_patterns=["*.py", "*.json", "*.jinja", "joint_head.safetensors"]))
73
  sys.path.insert(0, str(clef))
74
  import joint_schema_model as jsm
75
 
76
- backbone = Qwen3_5ForConditionalGeneration.from_pretrained("Qwen/Qwen3.8-27B", revision="1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0",
77
- dtype=torch.bfloat16, device_map={"": "cuda"})
78
  backbone = PeftModel.from_pretrained(backbone, "moondream/clef-lora")
79
  head = jsm.JointSchemaHead(**json.loads((clef / "joint_head_config.json").read_text()))
80
  head.load_state_dict(load_file(clef / "joint_head.safetensors"))
81
  model = jsm.ClefModel(backbone, head.to("cuda", torch.bfloat16)).eval()
82
  processor = AutoProcessor.from_pretrained(clef)
 
83
  print(jsm.systemone(model, processor, {"model": "clef", "state": "Checkout is down.",
84
  "questions": {"outage": {"type": "noul", "instructions": "Is a service down?"}}}))
85
  ```
86
 
87
- Requires the `flash-linear-attention` and `causal-conv1d` kernels for speed (transformers falls back to a slow torch
88
- path without them).
89
 
90
- ## On an FP8 base
 
 
 
 
 
 
91
 
92
- With this adapter unchanged on `Qwen/Qwen3.8-27B-FP8` (A100 emulation of FP8 weights and activations), accuracy stays
93
- within noise on the dev subsets (Decision Index benchmarks −0.34 points on average, vision index 71.40 vs 71.44), but
94
- 1.2% of text and 2.75% of vision answers flip, mostly near-ties on hard benchmarks. A full-suite comparison is in
95
- progress.
96
 
97
- ## Clef-flash
 
 
 
 
98
 
99
- | | |
100
- |---|---|
101
- | Source | `Cloudflare/clef-flash` @ `8b2e5fd17c09fd49bc2880b805d5323ec7ff4ff3` |
102
- | Base | `Qwen/Qwen3.5-9B` @ `c202236235762e1c871ad0ccb60c8ee5ba337b9a` |
103
- | Format | PEFT LoRA, bf16, `r=256`, `lora_alpha=256` (scale 1); the 48 Gated DeltaNet `in_proj_a`/`in_proj_b` modules have `r=32` (their smaller dimension) via `rank_pattern`/`alpha_pattern` |
104
- | Modules | 248 linears in all 32 layers: `linear_attn.{in_proj_qkv,in_proj_z,in_proj_a,in_proj_b,out_proj}`, `self_attn.{q,k,v,o}_proj`, `mlp.{gate,up,down}_proj` |
105
 
106
- Same method as Clef. Unlike Clef, every layer is adapted (relative delta norm about 7.8% median). Every delta is
107
- rank 256 (32 for the gates) above the bf16 merge-rounding floor; the rank-256 fit leaves 0.2–3.7% relative residual,
108
- matching the expected rounding noise (residual / expected noise: median 0.96, max 0.99). Embeddings, `lm_head`,
109
- norms and the vision tower are unchanged. (The base stores its 24 GDN gated-norm weights in float32 and Clef-flash
110
- stores them in bf16; they are identical after casting, so they are not part of the adapter.)
111
 
112
- Verification: Qwen3.5-9B (bf16) + this adapter + Clef-flash's own head against the released Clef-flash, on 120
113
- synthetic requests (231 questions): 1 answer flipped (a near-tie), mean |Δp| 0.003, max |Δp| 0.052, KL mean 0.0002,
114
- p99 0.0036, max 0.0059. That is bf16 noise, slightly above Clef's because every layer is adapted and the synthetic
115
- prompts are mostly low-confidence. Usage is as for Clef, with `Cloudflare/clef-flash` @ `8b2e5fd1…` for the head and
116
- code, `Qwen/Qwen3.5-9B` @ `c2022362…` as the backbone, and `subfolder="clef-flash"`.
117
 
118
  ## License
119
 
120
- Apache-2.0, as Cloudflare/clef, Cloudflare/clef-flash, Qwen/Qwen3.8-27B and Qwen/Qwen3.5-9B.
 
1
  ---
2
  license: apache-2.0
3
+ base_model:
4
+ - Qwen/Qwen3.8-27B
5
+ - Qwen/Qwen3.5-9B
6
  library_name: peft
7
  tags:
8
  - clef
 
12
 
13
  # clef-lora
14
 
15
+ The LoRA adapters inside Cloudflare's Clef models, extracted from their merged weights.
 
16
 
17
+ | Path | Model | Source | Base | Adapter |
18
+ |---|---|---|---|---|
19
+ | `/` | Clef (27B) | `Cloudflare/clef@ed3eed33` | `Qwen/Qwen3.8-27B@1d4bf0f2` | 186 linears, layers 40–63 |
20
+ | `clef-flash/` | Clef-flash (9B) | `Cloudflare/clef-flash@8b2e5fd1` | `Qwen/Qwen3.5-9B@c2022362` | 248 linears, all 32 layers |
21
 
22
+ Both are PEFT LoRAs in bf16, `r=256`, `lora_alpha=256`. The Gated DeltaNet `in_proj_a`/`in_proj_b` modules use
23
+ their full width instead (48 for Clef, 32 for Clef-flash), set via `rank_pattern`/`alpha_pattern`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
 
25
  ## Usage
26
 
27
+ The adapter replaces the backbone only. The decision head and encoding code come from the Cloudflare repo.
 
28
 
29
  ```python
30
+ import json, sys, torch
31
  from pathlib import Path
32
  from huggingface_hub import snapshot_download
33
  from peft import PeftModel
34
  from safetensors.torch import load_file
35
  from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
 
36
 
37
  clef = Path(snapshot_download("Cloudflare/clef", revision="ed3eed331870db2eff4b0db01237128ede8a00ce",
38
  allow_patterns=["*.py", "*.json", "*.jinja", "joint_head.safetensors"]))
39
  sys.path.insert(0, str(clef))
40
  import joint_schema_model as jsm
41
 
42
+ backbone = Qwen3_5ForConditionalGeneration.from_pretrained(
43
+ "Qwen/Qwen3.8-27B", revision="1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0", dtype=torch.bfloat16, device_map={"": "cuda"})
44
  backbone = PeftModel.from_pretrained(backbone, "moondream/clef-lora")
45
  head = jsm.JointSchemaHead(**json.loads((clef / "joint_head_config.json").read_text()))
46
  head.load_state_dict(load_file(clef / "joint_head.safetensors"))
47
  model = jsm.ClefModel(backbone, head.to("cuda", torch.bfloat16)).eval()
48
  processor = AutoProcessor.from_pretrained(clef)
49
+
50
  print(jsm.systemone(model, processor, {"model": "clef", "state": "Checkout is down.",
51
  "questions": {"outage": {"type": "noul", "instructions": "Is a service down?"}}}))
52
  ```
53
 
54
+ For Clef-flash, use `Cloudflare/clef-flash@8b2e5fd1`, `Qwen/Qwen3.5-9B@c2022362` and `subfolder="clef-flash"`.
55
+ Install `flash-linear-attention` and `causal-conv1d`; without them transformers uses a much slower path.
56
 
57
+ ## Extraction
58
+
59
+ - `delta = W_finetuned - W_base` per tensor in fp32, with the base cast to the fine-tune's dtype first.
60
+ - Only the linears listed above change. Embeddings, `lm_head`, norms and the vision tower match the base.
61
+ - Each delta is exactly rank 256 above the bf16 merge-rounding floor. An exact SVD gives `B = U√S`, `A = √S Vᵀ`.
62
+ - The rank-256 fit leaves 0.2–4% relative error per tensor, matching the expected rounding noise.
63
+ `extraction.json` lists it per module.
64
 
65
+ ## Verification
66
+
67
+ Base + adapter + the release's own head against the released model, on the same requests:
 
68
 
69
+ | Model | Test set | Answers changed | Max KL |
70
+ |---|---|---:|---:|
71
+ | Clef | Decision Index 0.2.1, 6,373 requests (30,950 questions) | 0.09% | 0.0035 |
72
+ | Clef | Vision, 11 benchmarks (2,400 rows) | 0.12% | 0.0024 |
73
+ | Clef-flash | 120 synthetic requests (231 questions) | 0.43% (1) | 0.0059 |
74
 
75
+ This is bf16 rounding noise: the same arithmetic as the merged weights, done in a different order.
76
+
77
+ ## On an FP8 base
 
 
 
78
 
79
+ The Clef adapter, unchanged, on `Qwen/Qwen3.8-27B-FP8` with FP8 weights and activations emulated in bf16 on H200.
80
+ Paired on 27,231 Decision Index requests (114,220 gold-scored questions):
 
 
 
81
 
82
+ - Accuracy: no significant change (net −0.012% of questions; mean −0.05 points per benchmark).
83
+ - Calibration: unchanged.
84
+ - 1.15% of answers flip, almost all near-ties, in both directions.
 
 
85
 
86
  ## License
87
 
88
+ Apache-2.0, following the source models and their bases.