georgep123 commited on
Commit
b4c15c5
·
verified ·
1 Parent(s): da263b1

Card: correct unsupported general-reasoning retention claim; add 2026-09-21 paired re-measurement

Browse files

The opening description asserted the model 'retains the full general-reasoning and vision capability of its base model'. The general-reasoning half is not supported: an independent paired re-measurement against Qwen/Qwen3.5-4B (lm-evaluation-harness 0.4.11, bf16, batch 8, seed 0, greedy, 0-shot except GSM8K 5-shot at max_gen_toks=512 on both arms; exact McNemar on per-example records aligned by doc_id and verified by doc_hash; 90% Clopper-Pearson; Holm-Bonferroni across six tasks; measured 2026-09-21) establishes gains on ARC-Challenge (+4.69 pp) and HellaSwag (+1.59 pp) and losses on MMLU (-2.19 pp) and GSM8K (-9.25 pp). WinoGrande and PIQA not established.

That clause is replaced with an accurate statement (capability redistributed, not retained). The vision half is untouched and preserved as an inheritance statement; nothing here tests the image path.

A new Benchmarks subsection adds the paired results with item counts, intervals, the multiplicity correction, and two caveats: MMLU is loglikelihood-scored and the cleanest row; GSM8K is generation-budget-sensitive and directional only.

No pre-existing figure on the card was altered, removed or recomputed. These results do not revise the existing Benchmarks table -- different protocols. Verified by re-download at the pushed revision: all 160 pre-existing numeric tokens unchanged, frontmatter/H1/code blocks/citation byte-identical.

Files changed (1) hide show
  1. README.md +76 -1
README.md CHANGED
@@ -23,7 +23,7 @@ tags:
23
 
24
  **A compact, bilingual, vision-capable model specialized for Canadian legal and regulatory work — in English and Canadian French.**
25
 
26
- flash-1-mini is a 4-billion-parameter model fine-tuned from Qwen3.5-4B for Canadian legal tasks. It is built for the parts of legal work that have to be right: producing correctly-formatted legal citations and following detailed instructions, across both of Canada's official languages and both of its legal traditions (common law and Quebec civil law). It retains the full general-reasoning and vision capability of its base model.
27
 
28
  - **Version:** `flash-1-mini-20260602`
29
  - **Developed by:** Alpine Pacific Trading Inc. (operating as SimpleDirect®)
@@ -285,6 +285,81 @@ Specialization carried measurable costs, reported here in full:
285
 
286
  If your workload is primarily retrieval-grounded QA or tool/function-calling orchestration, evaluate carefully against these numbers.
287
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
288
  ## What is not measured yet
289
 
290
  Named specifically, because a gap that is not named reads like a claim.
 
23
 
24
  **A compact, bilingual, vision-capable model specialized for Canadian legal and regulatory work — in English and Canadian French.**
25
 
26
+ flash-1-mini is a 4-billion-parameter model fine-tuned from Qwen3.5-4B for Canadian legal tasks. It is built for the parts of legal work that have to be right: producing correctly-formatted legal citations and following detailed instructions, across both of Canada's official languages and both of its legal traditions (common law and Quebec civil law). Its general-reasoning capability is redistributed rather than retained: an independent paired re-measurement against the base, dated 2026-09-21, establishes gains on some general benchmarks and losses on others (see **Independent paired re-measurement against the base**). Its vision capability is inherited from the base model, whose vision tower was frozen during training; no figure on this card measures it.
27
 
28
  - **Version:** `flash-1-mini-20260602`
29
  - **Developed by:** Alpine Pacific Trading Inc. (operating as SimpleDirect®)
 
285
 
286
  If your workload is primarily retrieval-grounded QA or tool/function-calling orchestration, evaluate carefully against these numbers.
287
 
288
+ ### Independent paired re-measurement against the base (2026-09-21)
289
+
290
+ This is a **separate, later measurement** from the figures above. It was run to check one
291
+ sentence that used to stand in this card's opening description, which asserted that the model
292
+ inherited its base model's general-reasoning capability in full, alongside its vision
293
+ capability. The general-reasoning half of that claim is not supported and has been replaced.
294
+ Nothing in this section bears on the vision half, which was not tested.
295
+
296
+ 🔴 **These numbers do not revise the figures in the Benchmarks table above.** The two sets used
297
+ different protocols, different task selections and different scoring, and the older set's
298
+ protocol is not recoverable (see *How these figures are recorded*). They are two measurements
299
+ of different things, not an old value and a corrected one. The Highlights bullet "No loss of
300
+ general capability" belongs to that older, unrecorded set; this re-measurement neither confirms
301
+ nor revises it.
302
+
303
+ **Provenance.**
304
+
305
+ - **Arms:** `simpledirect/flash-1-mini` against its base `Qwen/Qwen3.5-4B`, bf16 safetensors on
306
+ the Transformers path. The exact revision digests of the two arms are not recorded on this
307
+ card.
308
+ - **Harness:** `lm-evaluation-harness` 0.4.11.
309
+ - **Protocol:** bf16, batch size 8, seed 0, greedy decoding, no chat template, 0-shot on every
310
+ task except GSM8K, which is 5-shot. Generation was capped at `max_gen_toks=512` **on both
311
+ arms** for GSM8K; the other five tasks are loglikelihood-scored and generate nothing.
312
+ - **Analysis:** exact McNemar on the per-example records, aligned by `doc_id` with item identity
313
+ verified by `doc_hash`. Intervals are 90% Clopper-Pearson. Multiplicity correction is
314
+ **Holm-Bonferroni** across this model's six tasks — the family is those six tasks on this one
315
+ model, and the question it answers is which single differences can be relied on.
316
+ - **Date:** measured 2026-09-21.
317
+ - The per-example discordant counts (`b_only` / `c_only`) behind each row live in the run
318
+ records and are not reproduced on this card.
319
+ - Measured on the bf16 safetensors. No GGUF tier was evaluated here either.
320
+
321
+ **Established after Holm-Bonferroni correction — four of the six tasks.** Gains and losses are
322
+ in the same table, in task order, with no row emphasised:
323
+
324
+ | Task | Shots | n | flash-1-mini − base | 90% CI | Holm-corrected |
325
+ |---|---|---|---|---|---|
326
+ | ARC-Challenge | 0 | 1,172 | +4.69 pp | [+3.08, +6.14] | p < 1e-5 — established |
327
+ | HellaSwag | 0 | 10,042 | +1.59 pp | [+1.17, +2.00] | p < 1e-5 — established |
328
+ | MMLU | 0 | 14,042 | −2.19 pp | [−2.63, −1.75] | p < 1e-5 — established |
329
+ | GSM8K | 5 | 1,319 | −9.25 pp | [−11.15, −7.20] | p < 1e-5 — established |
330
+
331
+ So the direction is not uniform: capability is redistributed by the specialization rather than
332
+ preserved by it. Two tasks move up, two move down, and the largest single movement in either
333
+ direction is a loss (GSM8K, −9.25 pp) — which is also the least stable of the four, for the
334
+ reason given below.
335
+
336
+ **Not established — the remaining two tasks.** WinoGrande (−0.40 pp, p = 0.77) and PIQA
337
+ (−0.11 pp, p = 0.92) do not reach significance even before correction. Two things to be clear
338
+ about: "not established" is not "no difference", and this card does not record the item counts
339
+ or intervals for these two, so **no upper bound on either difference is stated here** and
340
+ neither point estimate should be cited as a sized delta. What is on record is that neither
341
+ survived.
342
+
343
+ **Two caveats, at equal prominence with the results.**
344
+
345
+ - **MMLU is the cleanest row here.** It is loglikelihood-scored — the model generates nothing —
346
+ so none of the generation-budget sensitivity that affects GSM8K applies to it. It also
347
+ reproduced exactly across two independent runs (0.7211 for flash-1-mini against 0.7431 for
348
+ the base, both times). Be precise about what that second fact buys: under greedy decoding on a
349
+ fixed item set, repeated runs reproducing is a statement about harness determinism and is
350
+ **not** evidence that the two arms differ. The evidence that they differ is the paired McNemar
351
+ test and the interval in the table, and nothing else.
352
+ - **GSM8K is the least stable row here.** The generation cap alone moves GSM8K scores by up to
353
+ 10.7 pp on this base, and each model over-generates by a different amount, so the cap
354
+ reweights the comparison rather than cancelling out of it. Read −9.25 pp as directional and
355
+ protocol-dependent: it is what this cap produced, and a different budget could produce a
356
+ materially different number. Both arms ran at the same cap, and the cap is stated above —
357
+ that is the most that can be said for it.
358
+
359
+ **What this section does not cover.** These are six general-capability text benchmarks. They say
360
+ nothing about Canadian legal citation work, nothing about French, and — like every other figure
361
+ on this card — nothing about the image path.
362
+
363
  ## What is not measured yet
364
 
365
  Named specifically, because a gap that is not named reads like a claim.