Align documentation with the verified Swedish-form evidence

README.md

@@ -18,12 +18,14 @@ datasets:
   - precisit/one-pass-sv-forms-synthetic
 ---
 
-# One-Pass SV-Forms (SV0) — a Swedish one-pass form specialist (research checkpoint)
+# One-Pass SV-Forms (SV0): a Swedish one-pass form specialist (research checkpoint)
 
 **This is a research checkpoint, not a product.** It is a 706 048-parameter model that scores a
-supplied list of options in **one forward pass** — the "Jev"/System One shape — trained from
-scratch on **synthetic** Swedish forms. It generates no text, needs no tokenizer, and answers in
-about a millisecond on an Apple device (Core ML: 1.4 MB fp16, 787 KB int8).
+supplied list of options in **one forward pass**, using a System One-style interface.
+It was trained from scratch on **synthetic** Swedish forms. It generates no text, needs no tokenizer, and answers in
+a measured median of 1.314 ms per prediction on an Apple M4 with `CPU_AND_NE`
+(Core ML packages: about 1,472 KiB fp16, 787 KiB int8). This excludes input encoding,
+model loading and warm-up; it is not an end-to-end form completion time.
 
 | | |
 | --- | --- |
@@ -31,26 +33,29 @@ about a millisecond on an Apple device (Core ML: 1.4 MB fp16, 787 KB int8).
 | Architecture | byte-level embeddings + 2-layer Transformer encoder (width 128, 4 heads) + option-attention head; `tinyx` |
 | Parameters | 706 048 |
 | Context / option budget | 224 bytes of context, 96 bytes per option, **up to 40 options** |
-| Training data | 10 000 synthetic Swedish form episodes (234 921 decisions); **no real form, person or customer data** |
+| Training data | 10 000 synthetic Swedish form episodes (234 921 decisions); **generated rather than collected from people or customers** |
 | Licence | MIT (weights and code) |
-| Lineage | independent implementation of the same interface as [Cua's CUA-S1](https://huggingface.co/cua-ai/cua-s1-forms) and [jevlike](https://github.com/vinnylarouge/jevlike); trained from scratch, not a fine-tune of either |
+| Lineage | uses vendored [Cua-S1](https://github.com/trycua/cua/tree/main/libs/cua-s1) architecture and training code, with an option-attention head credited to [jevlike](https://github.com/vinnylarouge/jevlike); new weights trained from scratch, not fine-tuned from Cua weights |
 
 ## Measured results (our harness, Apple M4)
 
-| Run | decisions | top-1 | majority-class baseline | ECE | silently skipped a required fill |
+| Run | decisions | top-1 | majority-class baseline | ECE | silently skipped an expected fill |
 | --- | ---: | ---: | ---: | ---: | ---: |
-| Held-out synthetic test (form-signature disjoint) | 2 310 | **99.29 %** | 47.36 % | 0.0014 | **1** |
-| Hand-written out-of-distribution demo | 50 | **100.00 %** | 64.00 % | — | 0 |
-| Shuffled-context control (same test rows) | 2 310 | 31.99 % | 47.36 % | — | — |
-
-The shuffled-context control is the number to look at first: rotating contexts between rows drops
-the model to 31.99 %, well below the majority baseline, so it is reading the element label and the
-document rather than exploiting option statistics. Without that control a 100 % demo set means
-nothing.
-
-For comparison, the released English checkpoint of the same family scores **20.75 %** on the same
-29 839 rows (`results-english-reference.json`) — far below the 47.36 % majority baseline, with
-5 412 of 13 844 required fills silently skipped. The language was the barrier, not the interface.
+| Held-out synthetic test (form-signature disjoint) | 29 839 | **99.29 %** | 47.36 % | 0.0014 | **1** |
+| Hand-written out-of-distribution demo | 50 | **100.00 %** | 64.00 % | n/a | 0 |
+| Shuffled-context control (same test rows) | 29 839 | 31.99 % | 47.36 % | n/a | n/a |
+
+The shuffled-context control rotates contexts between rows while retaining their options
+and labels. Accuracy falls to 31.99%, showing that the model uses the context rather than
+only option statistics. It does not identify which context clues the model has learned.
+The synthetic test contains 13,844 expected fill decisions; one is answered skip. This
+count excludes choosing a wrong fill value. Fill accuracy is 99.00%.
+
+For comparison, the unchanged released English checkpoint scores **20.75%** on the same
+29,839 rows, with 5,412 of 13,844 expected fills skipped. Labels, action strings and the
+generated distribution all change; the result does not isolate language as the cause.
+The 50 handwritten decisions comprise 32 fills, 12 skips, three checks and three clicks.
+They are a small diagnostic set, not a broad test of Swedish forms.
 
 ### Data-size curve (same 4-epoch schedule, same seed, corrected generator)
 
@@ -60,51 +65,48 @@ For comparison, the released English checkpoint of the same family scores **20.7
 | 4 000 | 94 405 | 98.94 % | 6 | 0.0014 | 56 min |
 | **10 000** | **234 921** | **99.29 %** | **1** | 0.0014 | 163 min |
 
-This checkpoint is the 10 000-episode point. The curve is published because it is the useful
-part: at 900 episodes the same pipeline sits at 77 %, so a single small run would have been
-misread as a statement about the method rather than about the data.
+This checkpoint is the 10,000-episode point. Each run has its own generated test set.
+At a fixed four epochs, the larger training sets also receive more optimization steps.
+These runs do not isolate the effect of data size or establish convergence.
 
 ## Changelog
 
-- **v2 (2026-09-21)** — this release. Two fixes and a scale-up, all three visible above:
-  1. **Corpus correctness.** The Swedish catalogue's `personnummer` generator emitted nine digits
-     formatted `DDDDDDDD-DD` and `organisationsnummer` emitted `DDDDD-DDDDD`; both are now the
-     real Swedish shape `YYMMDD-XXXX`. As a side effect the models can no longer identify those
-     two concepts from the *shape of the value* alone — the bug had been an accidental
-     giveaway — so per-action accuracy on `fill` is the honest kind now.
-  2. **Scale.** 900 → 10 000 episodes, which moved top-1 from 77 % to 99.29 % and silent skips
-     from 127 to 1.
-  3. **Correction of our own first explanation.** An intermediate 900-episode run on the
-     corrected corpus scored 83.02 % (v1) versus 77.10 % (corrected), and we initially attributed
-     that six-point drop to the removed giveaway. The 4 000- and 10 000-episode points show the
-     dominant factor was data size, not corpus correctness. The measurement is recorded here
-     because we published the wrong explanation for part of a working day.
-  4. **Correction of our Core ML claim.** We published that 4-bit palettisation destroys this
-     model. Scored per compute-unit path, 4-bit weights keep 99.27 % top-1 with 32 changed
-     decisions in 29 839 on the **CPU**, while the **Neural Engine** path for the same package
-     returns 49.92 % with 15 017 changes and reports `ANECCompile() FAILED`. The collapse was an
-     artefact of scoring int4 only with `CPU_AND_NE`: a broken accelerator path looked like a
-     broken quantiser. int8 still ships, for the better reason that it is identical to fp16 on
-     both paths, and the follow-up investigation is in the vault at
-     `ideas/2026-09-21-int4-investigation-one-pass.md`.
-- **v1 (2026-09-21)** — first release: 900 synthetic episodes, 83.02 % top-1, 7.5 % silent skips,
-  787 KB int8. Superseded; kept in the history of this repository rather than hidden.
+- **v2 (2026-09-21)**: corrected synthetic identifier formats and increased the corpus
+  to 10,000 episodes. The generator previously emitted malformed personnummer- and
+  organisationsnummer-shaped values. Tests now check the intended formats and checksums.
+  Such formatting differences could provide shortcuts, but the runs do not measure how
+  much the model used them. Comparing the earlier 83.02% run with the corrected
+  900-episode run at 77.10% does not establish a cause for the difference: the generated
+  training and test examples changed. Larger subsequent runs also used more training steps.
+- **Core ML clarification**: int4 was initially evaluated with `CPU_AND_NE` and described
+  as losing too much weight precision. The same package reaches 99.27% on CPU, versus
+  49.92% with CPU and Neural Engine allowed. Actual operation placement and the root
+  cause remain unknown. Int8 and fp16 are not prediction-identical. See the public
+  [execution investigation](https://github.com/precisit/one-pass-specialists/blob/9240818b2f9221ecce9e27af11eab9816dfc51f9/examples/sv-forms/COREML-EXECUTION.md) for the full M4 comparison, M1 Max follow-up,
+  decompression control and remaining tests.
+- **v1 (2026-09-21)**: earlier 900-episode release, 83.02% top-1 and a reported 7.5%
+  silent-skip rate. Its artifacts remain in repository history. Use each revision's
+  own result files and denominators when comparing releases.
 
 ## What it is NOT
 
-- **Not trained or tested on real Swedish forms.** The corpus is synthetic by construction:
-  fictional names, `.invalid` e-mail domains, generated personnummer-shaped identifiers that pass
-  their checksum but belong to nobody. Two sets carry the non-synthetic evidence, and both are
-  small: a 50-decision set we wrote by hand, and a 270-decision set built from two of our own
-  shipped forms (Kanslist's and Pratsam's lead forms, real labels and option lists, synthetic
-  values, hand-written entity-to-field mapping). **On those real forms the model scores 52.8 % and
-  77.3 % top-1**, against 99.29 % on synthetic held-out rows — the gap is the most important number
-  in this card. The failure mode is abstention rather than invention (on the Kanslist form it
-  answers "skip" for 57 of 65 required fills), and the cause is vocabulary and label length: the
-  training catalogue's element labels are short concept names, these forms' labels are sentences.
+- **Not validated for real form completion.** Training data are generated, not collected
+  from people or customer submissions. Names are constructed and email addresses use
+  `.invalid`. Generated identifiers are not checked against registries; coincidental
+  matches with real identifiers cannot be ruled out. The external checks are small:
+  50 handwritten decisions and 270 decisions in simplified fixtures based on Kanslist
+  and Pratsam forms. Their labels and controls differ from the current sites. They use
+  synthetic scenarios and hand-mapped expected actions, not real submissions or browser runs.
+  The fixture results are **52.8% and 77.3%**, with 57 of 65 expected Kanslist fills
+  answered skip. These are expected fills, not necessarily HTML-required fields.
+  Pratsam's fixture also has a gold-label defect: its scenario uses `mig själv`, while
+  the radio mapping checks for `ja`, labelling all 15 radio rows skip. Those results
+  are provisional. Vocabulary, input lengths, roles and fixture conventions need
+  investigation; the current evidence does not isolate one cause.
 - **Not an agent.** It does not generate text, cannot invent a value it was not given, and does
-  not decide execution order — the option list and the sequence come from ordinary code around it.
-- **Not safe unsupervised.** One required fill out of 29 839 was still answered "skip": a silent
+  not decide execution order: the option list and the sequence come from ordinary code around it.
+- **Not safe unsupervised.** One expected fill out of 13,844 was answered "skip" in the
+  synthetic test: a silent
   failure. Any real integration must check outcomes outside the model (fail closed, dry run, one
   submit, human review before consequential actions).
 - **Not tuned for throughput.** One decision per `predict` call is what the latency numbers
@@ -112,30 +114,43 @@ misread as a statement about the method rather than about the data.
 
 ## Core ML export (same checkpoint, coremltools 9.0, Apple M4)
 
-| Variant | package | CPU top-1 | CPU parity vs PyTorch (mismatches) | ANE top-1 | ANE parity (mismatches) |
+| Variant | package | CPU top-1 | CPU parity vs PyTorch (mismatches) | CPU + NE allowed top-1 | CPU + NE allowed parity (mismatches) |
 | --- | ---: | ---: | ---: | ---: | ---: |
-| fp16 | 1.44 MB | 99.28 % | 0.999531 (14) | 99.29 % | 0.999497 (15) |
-| **int8** | **787 KB** | **99.31 %** | 0.999598 (12) | 99.29 % | 0.999564 (13) |
-| int4 | 481 KB | **99.27 %** | 0.998928 (32) | **49.92 %** | 0.496732 (15 017) |
-
-Scored per compute-unit path on the same 29 839 held-out decisions, same encoding, same masked
-argmax. The reading matters more than the table:
-
-- **4-bit weights are nearly lossless on the CPU path** (32 changed decisions in 29 839) and the
-  package is 481 KB.
-- **The Neural Engine path for that palettised graph is silently wrong**: 49.92 % top-1, 15 017
-  changed decisions, while the graph also reports `ANECCompile() FAILED`. It does not raise an
-  error, it substitutes a different computation — which is why this model's large margins (median
-  12.1 logits) did not protect it.
-- **We first published this as "int4 destroys the model".** That was our measurement artefact:
-  int4 had only ever been scored with `CPU_AND_NE`, so a broken accelerator path looked like a
-  broken quantiser. Corrected here, and the wrong explanation is kept in the changelog rather than
-  deleted.
-- **int8 ships** because it is identical to fp16 on *both* paths — not because it survives
-  quantisation better.
-- **int4 + ANE is achievable for this architecture**: the released English conversion of the same
-  model runs 4-bit on the Neural Engine at 99.93 % in our own verification, so the difference is in
-  the export. Levers and open questions: `ideas/2026-09-21-int4-investigation-one-pass.md`.
+| fp16 | 1,472 KiB | 99.28 % | 0.999531 (14) | 99.29 % | 0.999497 (15) |
+| **int8** | **787 KiB** | **99.31 %** | 0.999598 (12) | 99.29 % | 0.999564 (13) |
+| int4 | 481 KiB | **99.27 %** | 0.998928 (32) | **49.92 %** | 0.496732 (15 017) |
+
+All 29,839 held-out decisions are included, with the same encoding and masked argmax.
+`CPU_AND_NE` allows CPU and Neural Engine use; it does not report which operations ran
+on which processor. Package sizes sum the files, not runtime memory. One KiB is 1,024 bytes.
+
+The int8 recipe uses a uniform 256-entry palette per tensor with float16 computation.
+Int4 uses grouped-channel K-means palettes, groups of 32, and a different deployment
+target: iOS 18, versus iOS 17 for fp16 and int8. These are different export recipes,
+not only different bit widths.
+
+The same int4 package preserves 99.27% top-1 on CPU but falls to 49.92% when CPU and
+Neural Engine are allowed. That is an execution-dependent discrepancy, not evidence
+that these rounded weights cannot represent the task. Int8's rounded top-1 matches
+fp16 under `CPU_AND_NE`, but it changes 13 choices versus PyTorch; it is not lossless
+or prediction-identical to fp16.
+
+A follow-up on M1 Max, macOS 26.5.1, used 512 deterministic samples spread across the
+test set. Int4 scored 511/512 on CPU and 266/512 with `CPU_AND_NE`. Decompressing its
+palettes to dense constants retained exactly those choices under each setting. This
+removes the palette-expansion operations but keeps the rounded weights and iOS 18
+target. It also runs conversion passes, so it is not proof of an otherwise identical graph.
+`ANECCompile() FAILED` messages were observed, but the buffered output could not be
+attributed to individual calls. Neither these messages nor the predictions establish
+operation placement, fallback or a faulty operation.
+
+See the [public investigation](https://github.com/precisit/one-pass-specialists/blob/9240818b2f9221ecce9e27af11eab9816dfc51f9/examples/sv-forms/COREML-EXECUTION.md) for pinned data, scripts, artifact hashes
+and the remaining target-matched export and graph-inspection work. The chip and OS
+both differ between the M4 and M1 Max tests; their effects are not isolated.
+
+The original M4 latency run excluded three warm-up rows, leaving 29,836 predictions.
+Int8 with `CPU_AND_NE` had median 1.314 ms and p95 1.411 ms; fp16 on CPU had median
+1.727 ms. These time `MLModel.predict`, not preprocessing, model loading or a full form.
 
 ## Usage
 
@@ -152,15 +167,18 @@ hf_hub_download("precisit/one-pass-sv-forms", "sv0-forms.json", local_dir=weight
 
 The context string and options must be built exactly as in training (byte ids = UTF-8 byte + 1,
 zero padded; `UPPGIFT fyll i formuläret från dokumentet och skicka sedan in` / `FORM <titel>` /
-`ELEMENT <roll> "<etikett>" value="…"`). A mismatch there is the most likely cause of poor output —
-the generator, the catalogue and the scoring harness are in the toolkit repository.
+`ELEMENT <roll> "<etikett>" value="…"`). An encoding mismatch is one possible source of poor output; vocabulary, truncation,
+fixture labels and runtime behavior also need checking. The generator, catalogue and
+scoring harness are in the toolkit repository.
 
 ## Building your own
 
 [`precisit/one-pass-specialists`](https://github.com/precisit/one-pass-specialists) is the public
 toolkit that produced this checkpoint: catalogue, synthetic generator with form-disjoint splits,
 trainer, evaluation (including the shuffled-context control and the silent-skip count), Core ML
-export with a bit-identity proof before anything is written, and the SV0 recipe with all receipts.
+export with a 512-example check of the export-friendly eager implementation against
+its trace, followed by Core ML decision comparisons, and the SV0 recipe. The trace
+check does not establish bit identity with the original unmodified forward pass.
 The synthetic corpus itself is published as
 [`precisit/one-pass-sv-forms-synthetic`](https://huggingface.co/datasets/precisit/one-pass-sv-forms-synthetic).
 
@@ -170,5 +188,5 @@ MIT. The architecture, training loop and evaluation metrics are vendored unmodif
 MIT-licensed [`libs/cua-s1`](https://github.com/trycua/cua/tree/main/libs/cua-s1) (which credits
 `jevlike` for the option-attention head design). The Swedish catalogue, synthetic generator,
 training runs, measurements and Core ML tooling are Precisit's. **No TypeSafe AI code, weights,
-data or API output was used** — "Jev" and "System One" are their names for a similar interface, and
+data or API output was used**: "Jev" and "System One" are their names for a similar interface, and
 this is an independent implementation. See `THIRD_PARTY_NOTICES.md`.