georgep123 commited on
Commit
da263b1
·
verified ·
1 Parent(s): 36d240f

Card standard: add routing, modality scope, provenance and not-measured sections

Browse files

Structural and qualitative only. No benchmark figure added, removed or altered; frontmatter, H1, code blocks, spec/benchmark/GGUF tables and citation preserved byte-identically.

Files changed (1) hide show
  1. README.md +171 -0
README.md CHANGED
@@ -33,6 +33,8 @@ flash-1-mini is a 4-billion-parameter model fine-tuned from Qwen3.5-4B for Canad
33
  - **Modalities:** Text + image input → text output
34
  - **Code & examples:** [github.com/getsimpledirect/flash-1-mini](https://github.com/getsimpledirect/flash-1-mini)
35
 
 
 
36
  | Spec | Value |
37
  |---|---|
38
  | Parameters | 4.54B |
@@ -43,6 +45,87 @@ flash-1-mini is a 4-billion-parameter model fine-tuned from Qwen3.5-4B for Canad
43
  | Precision | bfloat16 |
44
  | Tied embeddings | Yes |
45
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
  ## Highlights
47
 
48
  Measured against its base model under identical conditions (same prompts, same scoring):
@@ -132,6 +215,14 @@ ollama run flash-1-mini
132
 
133
  GGUF inference of this architecture (Qwen3.5 hybrid linear-attention / Gated DeltaNet) requires a recent `llama.cpp` build with support for these layers. The multi-token-prediction (MTP) head is excluded from the GGUF (not used at inference). To run the bf16 weights in lower precision instead, load them with `bitsandbytes` 4-bit/8-bit via `BitsAndBytesConfig`.
134
 
 
 
 
 
 
 
 
 
135
  ## Benchmarks
136
 
137
  All figures are flash-1-mini vs the Qwen3.5-4B base under identical conditions (same prompts, few-shot counts, scoring, greedy decoding). See the SimpleDirect benchmarking methodology and CBLRE eval-set documentation for full protocol.
@@ -146,6 +237,43 @@ All figures are flash-1-mini vs the Qwen3.5-4B base under identical conditions (
146
  | General knowledge (MMLU) | 69.8% | 69.8% |
147
  | Privacy-compliance bilingual parity (FR/EN) | — | **1.00** |
148
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
149
  ### Where it is weaker
150
 
151
  Specialization carried measurable costs, reported here in full:
@@ -157,6 +285,34 @@ Specialization carried measurable costs, reported here in full:
157
 
158
  If your workload is primarily retrieval-grounded QA or tool/function-calling orchestration, evaluate carefully against these numbers.
159
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
160
  ## Training
161
 
162
  flash-1-mini is a supervised fine-tune of Qwen3.5-4B using parameter-efficient adapters (LoRA with DoRA, rank 32 / alpha 64, RS-LoRA), with the vision tower frozen, on a bilingual Canadian legal corpus weighted toward citation production and Quebec civil-law content. The trained adapter was merged into the base weights and the checkpoint canonicalized for serving. The architecture is unchanged from the base.
@@ -168,6 +324,21 @@ flash-1-mini is a supervised fine-tune of Qwen3.5-4B using parameter-efficient a
168
  - **Bilingual, not omniscient in French.** Parity is strong on tested tracks but French professional-law MCQ regressed; do not assume uniform French superiority.
169
  - **Hallucination.** Like all LLMs, it can produce confident, incorrect output.
170
  - **Quebec register.** The model is evaluated for legal correctness, not certified for Quebec-French dialectal register.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
171
 
172
  ## License and attribution
173
 
 
33
  - **Modalities:** Text + image input → text output
34
  - **Code & examples:** [github.com/getsimpledirect/flash-1-mini](https://github.com/getsimpledirect/flash-1-mini)
35
 
36
+ ## Model at a glance
37
+
38
  | Spec | Value |
39
  |---|---|
40
  | Parameters | 4.54B |
 
45
  | Precision | bfloat16 |
46
  | Tied embeddings | Yes |
47
 
48
+ Context length, vocabulary size and precision are configuration values read from this
49
+ repository's `config.json`. They describe what the model accepts, not a measured result at
50
+ those limits. Card last reviewed 2026-09-21.
51
+
52
+ ## Should you use this model?
53
+
54
+ A decision table rather than a pitch. The left column is what this model was specialized for;
55
+ the right is where something else will serve you better. Every row is a pair on purpose — if
56
+ only one column fits your situation, that is the answer.
57
+
58
+ | Reach for this model when… | Use something else when… |
59
+ |---|---|
60
+ | You are drafting, summarizing or checking Canadian legal and regulatory text, and a qualified person reviews the output before it is used | You want an answer you can act on without a lawyer reading it — this model does not give legal advice and nothing on this card supports unreviewed use |
61
+ | The task is producing or checking legal citations, the one behaviour with a headline measured gain over the base (see Benchmarks) | You need citations you can rely on without checking — the measured accuracy is far from perfect, so most outputs still need verification against primary sources |
62
+ | You need one model that handles both Canadian legal traditions, common law and Québec civil law, in English and Canadian French | Your work sits outside Canada — the specialization corpus and the citation conventions it was trained on are Canadian, and nothing here measures other jurisdictions |
63
+ | The material cannot leave your infrastructure: a 4.54B bf16 model runs on a single modest accelerator, with no hosted API and no data leaving your hardware | You are comfortable with a hosted frontier model and want the broadest general capability available rather than a domain specialization |
64
+ | You need image input alongside text — scanned filings, judgments, forms, screenshots of regulatory pages | You need the image path to carry a measured accuracy claim — see **Modality scope**: every figure on this card is a text benchmark |
65
+ | You want open weights under Apache-2.0 that you can pin to a revision, diff, and re-run indefinitely | You need a certified, accredited, insured or professionally supervised legal product — this is a set of weights, not a service |
66
+ | Your inputs are long and you want headroom in the context window | Your workload is primarily retrieval-grounded QA or tool / function-calling orchestration — both regressed against the base, and the numbers are in **Where it is weaker** |
67
+
68
+ **A realistic place it fits.** A clerk or associate pastes a draft memo and the sources it leans
69
+ on, and asks the model to check that each citation is correctly formed and points at the right
70
+ instrument. The model flags the ones that look wrong. A qualified person opens the primary
71
+ sources and decides. The model shortens the distance between "here is a draft" and "here is a
72
+ list of things to check" — it does not close that loop, and it is not meant to.
73
+
74
+ 🔴 **Legal work has consequences for real people, so be blunt about what this is.** flash-1-mini
75
+ is a drafting and analysis aid. It is not a source of legal advice, it does not practise law, it
76
+ has no duty to any client, and it cannot tell you when it is wrong. Every output that reaches a
77
+ filing, an opinion, a client or a regulator must be reviewed by a qualified person who is
78
+ accountable for it. A fluent, confident, correctly-formatted citation can still point at nothing.
79
+
80
+ ## What makes it distinct
81
+
82
+ Five properties, each labelled by what kind of thing it is. **Measured** means a number appears
83
+ in the Benchmarks table below, with the provenance caveats recorded there. **Design intent**
84
+ means it describes what the model was built and trained to do, with no measurement on this card
85
+ establishing that the intent was met.
86
+
87
+ - **Citation integrity — measured.** The largest reported gain over the base is on legal-citation
88
+ integrity, and it is the result the specialization was aimed at. It is also the result most
89
+ worth reading with the provenance note below in hand: the figure is reported without a harness,
90
+ version, item count or date.
91
+ - **Bilingual English / Canadian French — measured, in both directions.** The card reports a
92
+ bilingual parity result on a privacy-compliance track and a regression on French
93
+ professional-law multiple choice. Both are below. Read them together: parity on one tested
94
+ track is not parity in general, and this card does not claim uniform French competence.
95
+ - **Both legal traditions — design intent, with one measured point against it.** The training
96
+ corpus was weighted toward Québec civil-law content alongside common-law material. The one
97
+ measured civil-law figure on this card moved the wrong way (CBLRE Québec civil law, in **Where
98
+ it is weaker**). Coverage of both traditions describes what was trained for; it is not a claim
99
+ that both are equally well served.
100
+ - **Image input — architectural, not measured here.** The vision tower is inherited from the base
101
+ and was frozen during training. See **Modality scope**.
102
+ - **Long context — a configuration value, not a measurement.** The context length in the table
103
+ above is what the configuration declares. No long-context retrieval or reasoning result is
104
+ reported on this card at any length.
105
+
106
+ ## Modality scope
107
+
108
+ flash-1-mini accepts **text and images** and emits **text**. The architecture is
109
+ `Qwen3_5ForConditionalGeneration`; the released processor is a `Qwen3VLProcessor` over a
110
+ `Qwen2VLImageProcessorFast`, and the configuration declares image and vision special tokens.
111
+ Image input is intended for working alongside a text instruction — reading a scanned judgment,
112
+ a filed form, a regulatory page or a table, and answering questions or drafting from it.
113
+
114
+ Three limits on how far to carry that, and the second is the one readers get wrong:
115
+
116
+ - **The vision tower was frozen during training.** Image handling is inherited from the base
117
+ rather than adapted to Canadian legal documents. Nothing in the specialization corpus taught
118
+ it to read a Québec land-register extract or a bilingual statutory layout.
119
+ - 🔴 **Every benchmark figure on this card is text-only.** CBLRE, IFEval, MMLU, BBH,
120
+ Global-MMLU FR, BFCL and the retrieval evaluation are all text benchmarks. **Do not read any
121
+ of those scores as evidence about the image path.** A text score describes the text path; it
122
+ says nothing about how the model handles a scanned page, and the two are not interchangeable.
123
+ - **The GGUF builds drop the vision tower entirely.** They are text-only, as stated in the GGUF
124
+ section above. If you need image input, use the bf16 safetensors.
125
+
126
+ If your workload depends on document images, treat the image path as something you must
127
+ evaluate yourself on your own material, because this card gives you nothing to go on.
128
+
129
  ## Highlights
130
 
131
  Measured against its base model under identical conditions (same prompts, same scoring):
 
215
 
216
  GGUF inference of this architecture (Qwen3.5 hybrid linear-attention / Gated DeltaNet) requires a recent `llama.cpp` build with support for these layers. The multi-token-prediction (MTP) head is excluded from the GGUF (not used at inference). To run the bf16 weights in lower precision instead, load them with `bitsandbytes` 4-bit/8-bit via `BitsAndBytesConfig`.
217
 
218
+ 🔴 **No GGUF tier has been evaluated, and the numbers on this card do not describe these files.**
219
+ Every figure in Benchmarks below was measured on the bf16 safetensors. A quantized copy is a
220
+ different artifact, and how far each tier drifts from the bf16 weights on Canadian legal
221
+ citation work has not been measured. The bit-depth guidance in the paragraph above is the
222
+ general expectation for quantization, not a measured result for this model — it is a reasonable
223
+ default for choosing a tier, and it is not evidence. Validate the tier you intend to ship on
224
+ your own cases before relying on it.
225
+
226
  ## Benchmarks
227
 
228
  All figures are flash-1-mini vs the Qwen3.5-4B base under identical conditions (same prompts, few-shot counts, scoring, greedy decoding). See the SimpleDirect benchmarking methodology and CBLRE eval-set documentation for full protocol.
 
237
  | General knowledge (MMLU) | 69.8% | 69.8% |
238
  | Privacy-compliance bilingual parity (FR/EN) | — | **1.00** |
239
 
240
+ ### How these figures are recorded, and what that costs you
241
+
242
+ 🔴 **These figures are reported without the provenance needed to reproduce them.** We would
243
+ rather state that plainly here than let a reader assume it exists somewhere.
244
+
245
+ **What this card records:** the two arms being compared (this model and the Qwen3.5-4B base),
246
+ that both arms ran under identical conditions — same prompts, same few-shot counts, same
247
+ scoring, greedy decoding — the benchmark names, and the resulting percentages.
248
+
249
+ **What this card does not record:**
250
+
251
+ - the **date** the evaluations were run;
252
+ - the **harness and its version** — nothing here names the evaluation code or pins it;
253
+ - the **exact revisions** of either arm, by commit or digest;
254
+ - the **item counts** (`n`) behind each percentage, so no delta can be sized;
255
+ - the **protocol details**: shots, dtype, batch size, seed, generation caps;
256
+ - any **paired analysis** — per-example records, discordant counts, confidence intervals or a
257
+ significance test — so equal and near-equal scores are single figures with no statement of how
258
+ closely the two arms actually agree;
259
+ - how many **comparisons were run in total** before these were selected for the table, so no
260
+ multiplicity correction is possible.
261
+
262
+ **The consequence, stated directly: a figure without a recorded harness and version is not
263
+ reproducible as stated.** You cannot re-run these from this card and expect to land on the same
264
+ numbers, and if you get different ones, neither of us can say which difference is the model and
265
+ which is the setup. Treat every percentage here as a reported result whose protocol is not
266
+ recoverable from this document, rather than as a citable measurement.
267
+
268
+ 🔴 **`flash-1-mini-20260602` is a model version string, not an evaluation date.** The digits
269
+ identify the checkpoint. Do not read them as the day any benchmark was run — no evaluation date
270
+ is recorded anywhere on this card.
271
+
272
+ We are not guessing at the missing fields to close the gap. When the runs are re-done with a
273
+ pinned harness, a recorded date and per-example records, the figures will be republished with
274
+ that provenance attached, and any that do not survive will be corrected in place with the
275
+ history kept visible.
276
+
277
  ### Where it is weaker
278
 
279
  Specialization carried measurable costs, reported here in full:
 
285
 
286
  If your workload is primarily retrieval-grounded QA or tool/function-calling orchestration, evaluate carefully against these numbers.
287
 
288
+ ## What is not measured yet
289
+
290
+ Named specifically, because a gap that is not named reads like a claim.
291
+
292
+ - **No GGUF tier is evaluated.** None of the three quantized files in `gguf/` has a published
293
+ score, on any benchmark, at any tier. The numbers above were measured on the bf16 safetensors
294
+ and do not transfer.
295
+ - **The image path.** This card reports **no** vision, document-understanding or OCR benchmark —
296
+ every figure in the table is a text benchmark. **This card does not state whether the image
297
+ path has been benchmarked elsewhere.** On the evidence presented here, it is unmeasured, and
298
+ that is all this document can tell you.
299
+ - **Long context.** No retrieval, recall or reasoning result is reported at long input lengths.
300
+ The declared context length says what the model accepts, not how it behaves near the limit.
301
+ - **Statistical strength of every reported difference.** With no item counts, paired tests or
302
+ intervals, none of the deltas above — in either direction — has an established size. The
303
+ directions are what is on record; the magnitudes are not.
304
+ - **Any comparison outside its own base.** The model has not been evaluated here against any
305
+ third-party model other than Qwen3.5-4B, and this card makes no comparative claim about any
306
+ other model.
307
+ - **Safety, bias and robustness.** No toxicity, bias, jailbreak, refusal or adversarial-prompt
308
+ evaluation is reported. Legal material carries loaded and jurisdictionally sensitive content,
309
+ and none of that has been characterized here.
310
+ - **French coverage beyond the two tracks reported.** Canadian French legal drafting, Québec
311
+ civil-law reasoning and fr-CA register are measured on the tracks listed above and nowhere
312
+ else.
313
+ - **Real-world outcomes.** Nothing here measures drafting quality in practice, error rates in
314
+ deployed use, or time saved. A benchmark score is not a fitness-for-purpose finding.
315
+
316
  ## Training
317
 
318
  flash-1-mini is a supervised fine-tune of Qwen3.5-4B using parameter-efficient adapters (LoRA with DoRA, rank 32 / alpha 64, RS-LoRA), with the vision tower frozen, on a bilingual Canadian legal corpus weighted toward citation production and Quebec civil-law content. The trained adapter was merged into the base weights and the checkpoint canonicalized for serving. The architecture is unchanged from the base.
 
324
  - **Bilingual, not omniscient in French.** Parity is strong on tested tracks but French professional-law MCQ regressed; do not assume uniform French superiority.
325
  - **Hallucination.** Like all LLMs, it can produce confident, incorrect output.
326
  - **Quebec register.** The model is evaluated for legal correctness, not certified for Quebec-French dialectal register.
327
+ - **Fabricated citations look exactly like correct ones.** Improved citation integrity means
328
+ fewer malformed or invented references, not none. A fluent citation in the right format can
329
+ still point at an instrument that does not exist or does not say what the model claims.
330
+ - **The law changes and the corpus does not.** The training corpus has a cutoff that this card
331
+ does not state. Statutes are amended, regulations are replaced and judgments are overturned;
332
+ the model has no way to know what has moved since it was trained, and will not tell you.
333
+ - **Confidentiality and privilege are your responsibility.** Running locally keeps material on
334
+ your own hardware, which is a deployment choice, not compliance. Self-hosting does not by
335
+ itself establish privilege, professional-secrecy obligations under Québec law, PIPEDA or
336
+ provincial privacy compliance, or data residency.
337
+ - **Unmeasured paths are not safe paths.** Image input, long context and the quantized GGUF
338
+ tiers all carry no measurement on this card. Absence of a reported problem is not evidence of
339
+ absence — see **What is not measured yet**.
340
+ - **No certification is implied.** This is an open-weight release for independent evaluation. No
341
+ law society, regulator or government body has reviewed, certified or endorsed it.
342
 
343
  ## License and attribution
344