| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.8-27B |
| base_model_relation: adapter |
| library_name: peft |
| pipeline_tag: text-classification |
| tags: |
| - lora |
| - document-question-answering |
| - structured-decisions |
| - calibration |
| - synthetic-evaluation |
| --- |
| |
| <!-- Generated by the release packager from release/solomon-release.json and the measured artifacts. |
| Do not edit this file by hand: it is hashed in MANIFEST.json and an edit trips enforcement. --> |
| |
| # Solomon |
|
|
| A LoRA adapter and trained answer heads for **Qwen/Qwen3.8-27B** that turn a document plus a set of |
| structured questions into one probability per decision, each with a retrieval pointer to where the |
| support for it plausibly sits in the source. It does not generate text. |
|
|
| > ### Read this before you trust a number on this page |
| > |
| > **v1.1 was measured on real documents, and the headline gain is not statistically established.** On a |
| > held-out panel of 802 questions over |
| > 54 real documents it answers |
| > **706** whole |
| > questions right against 679 |
| > for v1.0: +3.4 points, 95% document-bootstrap interval |
| > [-1.3, +7.4]. |
| > The interval includes zero. |
| > |
| > **The evaluation labels are AI-generated and have not been checked by a human.** See **Evaluation labels**. |
| > |
| > Nothing on this page is a certified error rate and nothing is guaranteed. |
|
|
| ## What it is, and what it is for |
|
|
| Give it a document once and ask structured questions against it. Each answer comes back as a |
| probability. On request it also returns **ranked pointers** — the three sentences an experimental relevance |
| head scores highest — as a place to start reading, not as the reason for the answer (**Evidence**, below). |
| There is no chat, no reasoning trace and no sampling: every answer is read from letter logits at a fixed |
| position through trained linear heads, so the same document and the same question return the same |
| numbers every time. |
|
|
| Four answer types: |
|
|
| | Type | Question shape | What comes back | |
| |---|---|---| |
| | Yes / no | does the document establish X? | one probability | |
| | Single choice | which of these does it state? | one probability over the listed options | |
| | Ordered choice | which threshold does it state? | one probability over the ordered options | |
| | Multi-label | which of these apply? | **one probability per candidate** | |
|
|
| **v1.1 removed the entity answer type.** "Which of these parties is the X?" is a yes/no question with the |
| party written in: ask one yes/no question per candidate, or send the parties as multi-label candidates. An |
| old entity request (`candidate_kind: "entity"`, or a `{candidate}` placeholder) is answered with a 400 that |
| says exactly this. |
|
|
| **It is for** turning documents into structured, machine-readable answers where you need a number |
| attached to each one, and where determinism and a refusal to drift matter more than fluency. |
|
|
| **It is not for** general knowledge question-answering, chat, generation, or summarisation. It is not |
| for any setting where a wrong answer is costly and cannot be checked: see **Limitations**. |
|
|
| ## How it works |
|
|
| | | | |
| |---|---| |
| | Base model | `Qwen/Qwen3.8-27B`, Apache-2.0, pinned revision `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0`. **Not redistributed here** | |
| | Adaptation | LoRA, rank 64, **question-side** placement, float32 | |
| | Answer projection | trained linear heads (`adapter/heads.npz`), not the language-model head | |
| | Readout | structured letter logits; yes/no-shaped units collapse to a binary log-odds before temperature | |
| | Calibration | one scalar per answer type; **all 1.0 (unscaled) in v1.1** | |
| | Runtime identity | a 21-key binding (BF16; one binding per precision) that refuses to load if the engine is not the one measured | |
|
|
| **The document is prefilled once; the questions branch off it.** The document goes through the model a |
| single time and becomes a reusable state (`POST /states`). Every question is then answered as an |
| isolated branch off that prefix. Two consequences are worth stating because they are tested on real |
| hardware and not merely intended: asking the same questions in a different order returns the same |
| answers, and answering from the cached document state matches a full forward pass with the same decision and |
| within 0.05 in probability (the v1.1 acceptance runs: BF16 full, int8 subset; largest difference |
| 0.0108 in BF16, and 0.0097 in an earlier BF16 |
| run). That limit is looser than the 1e-3 used for fp32, because BF16 differences of about 0.01 are expected; the |
| fp32 configuration was not re-run through acceptance for v1.1. |
|
|
| **The adapter is off while the document is read, and on from the question onward.** That is what |
| question-side placement means. Applying it across the whole sequence gives a different model to |
| the one that was measured, and the identity binding exists partly to stop that happening by accident. |
|
|
| **The readout is structured, not generated.** Rather than sampling an answer and parsing it, the model |
| is asked to commit at a fixed position and the letter logits at that position are read through trained |
| heads (readout mode `four_collapsed`). Yes/no-shaped units — a yes/no question, and each individual candidate |
| inside a multi-label answer — are read through **one merged yes/no head** and collapsed to a single binary |
| log-odds, `p = sigmoid(z / T)` with `z = log P(yes)/P(no)`, before the temperature is applied. The head is |
| shared, and results are reported per type. Choice questions apply the temperature to the listed slice, |
| `softmax(logits[:n] / T)`. **v1.1 serves every type at T = 1.0** (boolean 1.0 · multilabel 1.0 · single 1.0 · ordered 1.0): per-type temperatures were fitted on |
| the real dev panel and did not improve held-out calibration, so none is applied (see **Calibration**). |
|
|
| A branch whose `head_key` is not in the calibration artifact's map is **refused, not served at an |
| assumed 1.0**. |
|
|
| **The temperatures live inside the runtime binding**, covered by its checksum. A calibration fitted on |
| one model may not be served on another: the loader refuses by name rather than serving scalars that |
| mean nothing. Serving at temperature 1.0 everywhere is always permitted, on any model. |
|
|
| ### What the returned score means |
|
|
| `ordering_score` means two different things depending on the question, and the difference matters. |
|
|
| - **Single-unit** — a yes/no question, a single or ordered choice, and **every per-candidate value** |
| inside a multi-label answer. Here the score *is* the readout probability (unscaled in v1.1); how well that |
| magnitude holds on real documents is measured under **Reliability on real documents**, below. |
| - **Multi-unit** — the rolled-up question-level score for a multi-label question with more |
| than one candidate. It is the **product** of the per-candidate probabilities, which assumes those |
| candidates are independent. **That assumption has never been validated as a joint probability.** It |
| orders such questions well; it is a heuristic ordering, not a calibrated joint. If you need a |
| magnitude for one of these, read the per-candidate numbers. |
|
|
| The field is not called `probability` because that would be accurate for the first case and an |
| overclaim for the second. |
|
|
| There is **no abstention**. Every question is answered. The service will not emit a field named |
| `abstain`, `confidence`, `threshold` or `certified_error_rate`; it raises rather than return one. If |
| you want to decline low-confidence answers, that is your policy, made on your population, and this |
| release makes no claim about where to put the line. |
|
|
| ## How to run it |
|
|
| The package is a library, not a daemon: you build the engine, wrap it in the serving layer and start the |
| HTTP surface in four lines. **The reference CUDA configuration is BF16**: base weights in bfloat16 with the |
| linear-attention recurrence promoted to float32 (`precision='bf16'`, the configuration the model was trained |
| in). The v1.1 BF16 and fp32 measurements on this card were run on NVIDIA B200 GPUs. Two alternatives are selectable, each with |
| its own measured numbers below and its own identity: `precision='fp32'` (float32 weights and attention, |
| float64 recurrence; roughly twice the memory) and `precision='int8'` (weight-only 8-bit decoder linears via |
| torchao, compute in bf16; the smallest footprint, not faster: on an NVIDIA RTX A6000 |
| (48 GB) it held 29.2 GiB after load and peaked at |
| 32.8 GiB on the real test text documents and |
| 40.4 GiB on page images). The binding pins the precision, so a |
| configuration can only be served against numbers measured on it. **The Apple-silicon MLX package in `mlx/` has |
| not been updated for v1.1**: it is pinned to the v1.0 revision of this repository and loads the v1.0 adapter, |
| heads and calibration, not the files described on this card. It is experimental, and none of the numbers on this |
| card describe it. |
|
|
| ```sh |
| pip install -r requirements.lock |
| |
| huggingface-cli download DoccyHealth/Solomon --local-dir ./solomon |
| huggingface-cli download Qwen/Qwen3.8-27B --revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 --local-dir ./solomon/base |
| |
| cd solomon |
| export PYTHONPATH=$PWD/src |
| ``` |
|
|
| Run from inside that directory: the engine looks for the base model in `base/`, and the serving layer |
| reads `serving/selection.json` and the binding it names, both relative to the package root. |
|
|
| ```python |
| from solomon import api, service |
| from solomon.serving import ServiceEngine |
| |
| engine = ServiceEngine('adapter/adapter.safetensors', 'adapter/heads.npz') # precision='bf16' (default) | 'fp32' | 'int8' |
| layer = service.service('./store', engine) # './store' holds cached document states |
| # fp32 / int8: build the engine with that precision AND pass its binding, e.g. |
| # engine = ServiceEngine(..., precision='int8') |
| # layer = service.service('./store', engine, binding='serving/serving-binding-int8.json') |
| server = api.serve(layer) # GET /health, POST /states, POST /v1/decide |
| print('http://127.0.0.1:%d' % server.server_port) |
| server.serve_forever() # or skip the server and call layer.decide(...) directly |
| ``` |
|
|
| `ServiceEngine` also takes `expected_adapter=` and `expected_heads=`, the two sha256 values printed |
| below; pass them and the engine refuses to start on a file that is not the one this card describes. |
|
|
| ### Asking questions |
|
|
| One request carries a document (or a saved `state_id`) and any number of questions. The four shapes: |
|
|
| ```jsonc |
| POST /v1/decide |
| { |
| "state": "…the document text…", |
| "evidence": "support", // none | support | sufficiency | removal |
| "questions": { |
| "certified": "Is Rookwood Ltd certified to supply produce?", // yes / no |
| "growers": {"type": "noul", "instructions": "Which growers may supply produce?", |
| "candidates": ["Rookwood Ltd", "Ostler Ltd"]}, // multi-label |
| "scheme": {"type": "choice", "instructions": "Which certification scheme applies?", |
| "options": ["Freshcare", "SQF", "GlobalG.A.P."]}, // single choice |
| "risk": {"type": "choice", "instructions": "What supply risk rating is recorded?", |
| "options": ["low", "medium", "high"], "ordered": true} // ordered choice |
| } |
| } |
| ``` |
|
|
| ```jsonc |
| { |
| "answers": { |
| "certified": {"type": "noul", "noul": 0.97, "ordering_score": 0.97, "temperature": 1.0, |
| "evidence_method": "trained_relevance_head_ranked", "evidence": [{"start": 212, "end": 256, "text": "…", "score": 0.83, "rank": 1}, …], "evidence_suppressed": false}, |
| "growers": {"type": "noul", "candidate_kind": "label", |
| "candidates": {"Rookwood Ltd": 0.96, "Ostler Ltd": 0.04}, |
| "candidate_ordering_scores": {"Rookwood Ltd": 0.96, "Ostler Ltd": 0.96}, |
| "candidate_evidence": {"Rookwood Ltd": [{"start": 212, "end": 256, "text": "…", "score": 0.91, "rank": 1}, …], |
| "Ostler Ltd": [{"start": 257, "end": 309, "text": "…", "score": 0.77, "rank": 1}, …]}, |
| "candidate_evidence_suppressed": {"Rookwood Ltd": false, "Ostler Ltd": false}}, |
| "scheme": {"type": "choice", "answer": "Freshcare", "probabilities": {"Freshcare": 0.91, "SQF": 0.06, "GlobalG.A.P.": 0.03}}, |
| "risk": {"type": "choice", "ordered": true, "answer": "medium", "probabilities": {"low": 0.12, "medium": 0.81, "high": 0.07}} |
| } |
| } |
| ``` |
|
|
| (Illustrative numbers, not measurements.) The entity form `{"instructions": "Is {candidate} a grower?", |
| "candidates": [...]}` was removed in v1.1: write `"growers"` above, or one yes/no question per party. |
|
|
| **Evidence.** At `evidence: "support"` the response carries up to three ranked sentence pointers with scores |
| from an experimental relevance head, per answer branch. `evidence_method` names the selector: |
| `trained_relevance_head_ranked`, or `lexical_overlap_fallback` (word overlap, used for page images and whenever |
| the head is not configured, with `evidence_fallback_reason` saying why). Neither establishes that the |
| answer was caused by the span (`evidence_faithfulness_established: false`). See **Evidence (experimental)**. |
|
|
| `GET /health` reports the contract, the readout mode and the runtime identity. If the engine you built |
| differs from the one the numbers were measured on — a different torch build, a different adapter, a |
| different arithmetic mode — **the binding refuses to load rather than quietly serving different |
| numbers**. That is intended behaviour; do not work around it. |
|
|
| ### What is in this repository |
|
|
| ``` |
| README.md this file |
| LICENSE Apache-2.0 |
| NOTICE attribution and the Apache-2.0 4(b)/4(c) notices |
| MODIFICATIONS.md statement of changes, third-party text scan, maintainer notes |
| MANIFEST.json every file, its size and its sha256 |
| requirements.lock the pins of the image this was qualified on, and dependency licences |
| licenses/Qwen-Apache-2.0.txt the upstream licence, verbatim |
| adapter/adapter.safetensors the LoRA adapter |
| adapter/heads.npz the trained answer heads |
| adapter/config.json base repo, pinned revision, checksums, placement |
| serving/serving-binding.json runtime identity binding (bf16, default) and the frozen temperatures |
| serving/serving-binding-fp32.json runtime identity binding for precision=fp32 |
| serving/serving-binding-int8.json runtime identity binding for precision=int8 |
| serving/evidence-head.safetensors relevance head weights (float32) |
| serving/evidence-head.json relevance head config incl. lexical_residual alpha |
| serving/evidence-policy.json ranked-pointer serving policy (top 3, suppressed when not stated) |
| serving/readout-temperature-v3.json the calibration artifact, standalone |
| serving/selection.json readout mode and model identity |
| serving/service-export.json qualification envelope hashes |
| src/solomon/__init__.py solomon: a document plus structured questions in, one probability per decision out |
| src/solomon/api.py HTTP surface for the Solomon layer (solomon/service.py) |
| src/solomon/binding.py serving identity: which readout is served, and proof that it is the one that was measured |
| src/solomon/calibration.py the readout calibration: one positive scalar per answer type, and nothing else |
| src/solomon/engine.py question-only CUDA runtime for immutable semantic-head checkpoints |
| src/solomon/engine_contract.py the answer contract engine: contract v3 on the reference engine's float32 cached path |
| src/solomon/engine_cuda.py CUDA contract-v3 engine |
| src/solomon/engine_numerics.py versioned CUDA repair: bounded FP32 attention, FP64 recurrent accumulation |
| src/solomon/engine_reasoning.py the confidence layer text reasoning on immutable answer-contract base-prefix states |
| src/solomon/engine_reference.py the reference engine: task-agnostic document prefix, float32 arithmetic, chunked prefill |
| src/solomon/evidence.py deterministic source references and explicit evidence interventions |
| src/solomon/evidence_head.py v1.1 evidence head: the ONE forward shared by the trainer (the training tooling via |
| src/solomon/evidence_packages.py evidence packages (v3): selected spans plus source-derived governing context, per question unit |
| src/solomon/evidence_selector.py evidence selection for the Solomon layer (v1.1): the trained relevance head, with word overlap as a labelled fallback |
| src/solomon/heads.py final normalized feature extraction, preserving the qualified CUDA engine |
| src/solomon/prompts_two_letter.py two-letter (Noul) prompts and block conversions |
| src/solomon/readout.py contract v3 readouts: branch jobs for every answer type, predictions from letter logits, and metrics |
| src/solomon/reliability.py what the readout says about its own answer, with nothing fitted behind it |
| src/solomon/retrieval.py inference-only source candidates and explicitly labelled retrieval baselines |
| src/solomon/routing.py real callback-driven escalation |
| src/solomon/semantics.py answer semantics (design note, not distributed) |
| src/solomon/service.py decision layer: the Solomon serving contract over the pinned readout chain |
| src/solomon/service_answers.py confidence-aware five-task service, reusing immutable answer-contract input persistence |
| src/solomon/service_checked.py backend-neutral contract-v3 service; restart replays immutable inputs, not tensors |
| src/solomon/service_evidence.py optional source-grounded evidence around the existing confidence service |
| src/solomon/service_heads.py trained-head fast/views service; unchanged decoder and separate stage confidence |
| src/solomon/service_packages.py evidence packages (v3) in the answer service: per-unit, source-grounded, page-referenced |
| src/solomon/service_states.py local contract-v3 prototype: task-neutral text/image states and five answer types |
| src/solomon/serving.py preserve five-task trained-head confidence/routing with optional evidence |
| src/solomon/units.py v1.1 shared sentence/list-item splitter |
| mlx/ the optional Apple-silicon package: its own library, tests and notices |
| ``` |
|
|
| **What is deliberately not here:** no training data, no evaluation panels, no document corpora, no |
| datasets, no cached states, no rendered images, no score archives, no test fixtures, no logs and no |
| base model weights. The package is default-deny: a manifest names every permitted file and the build |
| fails if anything else is present. |
|
|
| The files under `serving/` carry identity only: hashes, the readout mode, the served design and the |
| per-type temperatures. The loader verifies the binding's checksum and every runtime key it carries. The |
| measured provenance behind those hashes (which panels, which fit, which qualification run) is held in |
| the maintainer's records and is not distributed. |
|
|
| ## Evidence (experimental) |
|
|
| **Evidence here is a set of ranked pointers, not an explanation.** At `evidence: "support"` the service returns |
| the **top 3 sentences** of the document with a relevance score each (`evidence_method: |
| "trained_relevance_head_ranked"`). The scores come from a small relevance head fitted after training, on the |
| model's layer-42 states plus a word-overlap term. It reads states the answer already computed; it |
| makes no extra model call and **cannot change an answer**: with the head on and off, all 3,230 real |
| test answer branches were identical (maximum logit difference 0.0). |
|
|
| - `evidence_faithfulness_established` is `false`. Nothing shows the pointed-to sentence caused the answer. |
| - **No pointers are returned when the answer is "not stated"** (the answer's collapsed state is not-stated, or a |
| choice answer resolved to the reserved not-stated option). That state is the absence signal; the head itself |
| has no reliable "no evidence" signal. |
| - On the development panel the head put a labelled supporting sentence in its top 3 more often than plain word |
| overlap: hit@3 0.817 against 0.669 |
| over 753 labelled rows. That comparison was used to choose the head, so it is |
| optimistic, and **no test-panel measurement exists**. The precision and recall targets for evidence were not established. |
| - **Word overlap is the labelled fallback** (`lexical_overlap_fallback`, with `evidence_fallback_reason`), used for |
| page-image documents and whenever the head is not configured. |
|
|
| Treat pointers as a place to start reading and verify them yourself. |
|
|
| ## What it scores |
|
|
| Every figure in this section was measured on the adapter this repository ships |
| (`d122466d430a…`) in the BF16 reference configuration unless stated. The scoring runs |
| loaded a heads file (`96ea51416bbe…`) that is the shipped `adapter/heads.npz` |
| (`f766d752d776…`) plus two legacy slots that were never read; every array the two files share is |
| byte-identical, so the served logits are the measured logits. The packager checks those hashes against the measurement records and refuses to |
| build if they differ. Figures for v1.0 are that model re-scored on the same panels, for comparison only. |
|
|
| ### Real documents |
|
|
| Real public documents (government notices, policies, agreements, correspondence, minutes and similar), split by |
| document into training, dev and test. Test documents were never trained on and never used for any fit. |
|
|
| | Panel | Questions / documents | This model | v1.0 | External reference¹ | v1.1 − v1.0, points [95% CI] | |
| |---|---|---|---|---|---| |
| | Test | 802 / 54 | **706 (88.0%)** | 679 (84.7%) | 690 (86.0%) | +3.4 [-1.3, +7.4] | |
| | Dev | 624 / 15 | **555 (88.9%)** | 529 (84.8%) | 553 (88.6%) | +4.2 [+1.0, +7.5] | |
|
|
| ¹ a commercial structured-decision API (external reference), scored on the same questions and labels. |
|
|
| Intervals are document-cluster bootstrap (2,000 resamples). **On test the interval includes zero**: the release |
| rule asked for a lower bound of −1.0 points and the measured bound is |
| -1.3. The owner accepted this miss for v1.1; see **Release decisions**. |
|
|
| Per answer type, real test: |
|
|
| | Answer type | Questions | This model | v1.0 | External reference | v1.1 − v1.0, points [95% CI] | |
| |---|---|---|---|---|---| |
| | Yes / no | 149 | 132 / 149 | 132 / 149 | 135 / 149 | +0.0 [-10.1, +7.2] | |
| | Party-role questions (formerly entity; now asked as yes/no per party) | 213 | 185 / 213 | 165 / 213 | 180 / 213 | +9.4 [+0.0, +19.2] | |
| | Multi-label | 226 | 191 / 226 | 190 / 226 | 184 / 226 | +0.4 [-4.8, +5.9] | |
| | Ordered choice | 109 | 95 / 109 | 90 / 109 | 91 / 109 | +4.6 [-1.9, +10.7] | |
| | Single choice | 105 | 103 / 105 | 102 / 105 | 100 / 105 | +1.0 [+0.0, +3.0] | |
| | Multi-label, per candidate (slots) | 1,204 | 96.5% | 95.8% | not tallied | +0.7 [-1.0, +2.3] | |
| | Party-role, per party (slots) | 1,329 | 97.2% | 94.9% | not tallied | +2.3 [+0.4, +4.1] | |
|
|
| Counts are whole questions right, with the same question definition applied to all three models. |
|
|
| ### Page images against text |
|
|
| The same 802 real test questions, asked from rendered page images instead |
| of extracted text: 713 |
| whole questions right from images against 706 |
| from text, with 98.5% of 2,936 |
| answer branches agreeing between the two. |
|
|
| ### Natural images |
|
|
| A panel of photographs and pictures with structured questions (no document text). Two populations are reported |
| and they are different numbers: |
|
|
| - **Per answer unit** (731 units): this model 97.3%, |
| v1.0 96.2%, base Qwen 95.5%. |
| Answers stated at 0.99 or above that were wrong: this model 0.0% |
| of 293, base Qwen 1.5% of |
| 401. |
| - **Whole questions** (415): this model 95.2%, v1.0 |
| 93.3%, base Qwen 92.0%. |
|
|
| ### General knowledge, with no document (out of domain) |
|
|
| 800 multiple-choice items, 400 from MMLU and 400 from MMLU-Pro, every model on the same items and prompt. |
|
|
| | Model | Accuracy (800) | Answers stated ≥ 0.99 | Of those, wrong | ECE (top label, 15 bins) | |
| |---|---|---|---|---| |
| | **This model (v1.1)** | 72.9% | 24 | 0 (0.0%) | 0.052 | |
| | v1.0 | 72.8% | 401 | 20 (5.0%) | 0.146 | |
| | External reference¹ | 87.1% | 314 | 5 (1.6%) | 0.040 | |
| | Base Qwen, same prompt | 71.8% | 187 | 4 (2.1%) | 0.049 | |
|
|
| All four rows come from one computation on the same 800 items. Probabilities are unscaled (T = 1) for every row, |
| which is how v1.1 serves them. |
|
|
| v1.0 stated half of its answers at 0.99 or above and was wrong on 1 in 20 of them. This model almost never |
| claims 0.99 on general knowledge. MMLU moved +1.5 points and |
| MMLU-Pro -1.25 points against v1.0; the release rule allowed at |
| most 1 point either way, so **MMLU-Pro missed it**. The owner accepted this miss for v1.1. |
|
|
| ### Reliability on real documents |
|
|
| Real test, text, every answer unit (each option or candidate scored against its label): of the probabilities |
| stated in each band, the share that were actually right. A calibrated model's column would track the band. |
|
|
| | Stated P(yes) | this model | external reference | |
| |---|---|---| |
| | 0.00–0.01 | 0.0% (n=2,095) | 0.1% (n=2,619) | |
| | 0.01–0.02 | 0.8% (n=1,038) | 0.7% (n=305) | |
| | 0.02–0.05 | 3.2% (n=569) | 1.8% (n=325) | |
| | 0.05–0.10 | 14.4% (n=132) | 3.0% (n=202) | |
| | 0.10–0.20 | 46.0% (n=87) | 7.5% (n=213) | |
| | 0.20–0.30 | 51.6% (n=31) | 18.1% (n=116) | |
| | 0.30–0.40 | 18.2% (n=22) | 28.6% (n=63) | |
| | 0.40–0.50 | 66.7% (n=18) | 30.4% (n=56) | |
| | 0.50–0.60 | 47.1% (n=17) | 39.5% (n=43) | |
| | 0.60–0.70 | 61.5% (n=13) | 55.6% (n=63) | |
| | 0.70–0.80 | 53.1% (n=32) | 68.6% (n=70) | |
| | 0.80–0.90 | 83.7% (n=43) | 82.9% (n=111) | |
| | 0.90–0.95 | 89.3% (n=84) | 91.0% (n=89) | |
| | 0.95–0.98 | 97.3% (n=295) | 92.5% (n=106) | |
| | 0.98–0.99 | 99.6% (n=485) | 95.6% (n=90) | |
| | 0.99–1.00 | 100.0% (n=126) | 99.5% (n=616) | |
|
|
| The top end is at or above the external reference. **The low end under-calls**: answers stated at 5–20% are yes |
| more often than stated. (This table is the unscaled readout, which is what v1.1 serves.) |
|
|
| ### Calibration |
|
|
| Per-type temperatures were fitted on the real dev panel (boolean 0.8175 · multilabel 0.8423 · single 1.1077 · ordered 1.2562) and checked on held-out real test. |
| **They did not improve held-out calibration**: test ECE got worse in 8 of 10 answer-type × modality |
| cells, and the question-weighted ECE across all cells was 0.0212 with the fitted temperatures against |
| 0.0199 unscaled. Only multi-label improved. **v1.1 therefore ships unscaled probabilities (T = 1.0 for |
| every type).** Real test ECE per cell (10 bins): |
|
|
| | Answer type | Input | Units | ECE, unscaled (served) | ECE, dev-fitted temperature | Meets 0.03 target | |
| |---|---|---|---|---|---| |
| | yes/no | image | 149 | **0.055** | 0.068 | **no** | |
| | yes/no | text | 149 | **0.067** | 0.081 | **no** | |
| | party-role (per party) | image | 1,329 | **0.009** | 0.020 | yes | |
| | party-role (per party) | text | 1,329 | **0.011** | 0.013 | yes | |
| | multi-label (per candidate) | image | 1,204 | **0.020** | 0.012 | yes | |
| | multi-label (per candidate) | text | 1,204 | **0.024** | 0.013 | yes | |
| | ordered | image | 109 | **0.073** | 0.086 | **no** | |
| | ordered | text | 109 | **0.054** | 0.077 | **no** | |
| | single | image | 105 | **0.022** | 0.053 | yes | |
| | single | text | 105 | **0.014** | 0.040 | yes | |
|
|
| The 0.03 target is **missed for yes/no** (0.067 text, |
| 0.055 image) **and ordered choice** (0.054 text, |
| 0.073 image). Those cells have only |
| 105–149 questions each (yes/no, single and ordered). Party-role rows are |
| the former entity questions, now asked as one yes/no question per party. |
|
|
| ### Precision configurations |
|
|
| Real test, text, 3,230 answer branches. Accuracy is whole |
| questions; flips are served decisions that differ from BF16. |
|
|
| | Configuration | Status | Whole-question accuracy | vs BF16, points [95% CI] | Decisions flipped vs BF16 | ECE (unscaled) | |
| |---|---|---|---|---|---| |
| | `bf16` | **default, reference** | 88.03% | — | — | 0.018 | |
| | `fp32` | comparison | 87.66% | -0.37 [-0.77, +0.00] | 0.10% | 0.016 | |
| | `int8` | option | 87.53% | -0.50 [-1.02, +0.00] | 0.27% | 0.016 | |
|
|
| `int8` is weight-only 8-bit (torchao) with bf16 compute. **`int8` page-image accuracy has not been scored against |
| the labels**; a decision-agreement run against fp32 on page images agreed on |
| 99.71% of served decisions. On page |
| images `fp32` scored 88.78% against BF16 |
| 88.90%. No speed claim is made for any configuration. |
|
|
| ### Failure modes (synthetic probes) |
|
|
| 6,000 generated questions across 38 targeted failure modes, paired |
| against v1.0. Most modes are flat. Modes whose interval excludes zero: |
|
|
| | Mode | Questions | v1.0 | This model | Difference, points [95% CI] | |
| |---|---|---|---|---| |
| | indirect reference | 160 | 80.6% | 72.5% | -8.1 [-13.1, -3.8] | |
| | opposite polarity question | 135 | 94.1% | 97.0% | +3.0 [+0.7, +5.9] | |
|
|
| - **Paraphrase agreement** on yes/no questions: 0.951 (v1.0 |
| 0.946); the target was 0.98 and is not met. |
| - **Adversarial confident flips** (answer changed at ≥ 0.9 by an injected instruction, false summary or |
| self-classifying text): 2.0% (v1.0 2.1%); |
| the target was 1% and is not met. |
| - **Per-type calibration on real test** misses the 0.03 ECE target for yes/no and ordered questions (see |
| **Calibration**, below; reported, not blocking). |
|
|
| ## Limitations |
|
|
| - **Evaluation labels are not human-verified.** Every real-document reference label was produced by AI |
| labellers: two blind passes plus adjudication, with 99% agreement on binary slots between the passes. The owner |
| decided to release v1.1 without a human label review. Some measured errors may be label errors, and some |
| measured successes may share a labeller's mistake. |
| - **The headline improvement is not significant** (interval includes zero) and the real panels are small: |
| 54 test documents. |
| - **The low end of the probability scale under-calls** on real documents (see the reliability table). |
| - **Indirect references regressed** on the synthetic probes (table above). |
| - **The multi-candidate roll-up is an ordering, not a joint probability.** Read per-candidate values if you |
| need a magnitude. |
| - **Nothing here is a certified error rate.** No threshold is enforced anywhere on the serving path. |
| - **Page images:** measured on the real test panel in BF16 and fp32 only. |
|
|
| ## Release decisions |
|
|
| Two release rules were missed and **both were accepted by the owner for v1.1**: |
|
|
| 1. Real test, whole questions: lower bound of the 95% interval -1.3 |
| points against a rule of −1.0. |
| 2. General knowledge: MMLU-Pro -1.25 points against a limit |
| of 1 point (MMLU +1.5). |
|
|
| ## Evaluation labels |
|
|
| Reference labels on the real panels are AI-generated (two blind passes plus adjudication) and **have not been |
| reviewed by a human**. No Claude or GPT output is used anywhere as training input. |
|
|
| ## Training data (v1.1) |
|
|
| **Real documents.** 200 real public documents: 160 collected for this release plus |
| 40 from an earlier evaluation panel (those 40 are test-only). Split by document: 113 train, |
| 24 dev, 63 test (the evaluation panels above use the labelled subset). Licences of the |
| 160 collected documents, as recorded at collection: 59 Australian government pages under CC BY 4.0, |
| 76 UK Crown copyright under the Open Government Licence v3.0, 25 US federal government works (public domain). |
| No document is distributed here. |
|
|
| **Where the training labels came from.** |
|
|
| - Real-document training labels: Qwen3.8 2.4T (open weights), called through OpenRouter and routed to |
| third-party hosts serving full-precision weights, not the Alibaba API. |
| - Anchor targets from the unmodified base Qwen model, so general behaviour does not drift. |
| - Code generators for the synthetic documents and targeted failure-mode questions. |
| - Replay of the v1.0 training data (whose third-party sources are listed below). |
|
|
| **No Claude or GPT output is ever training input or a training label.** The build enforces this with an |
| allow-list of row producers. |
|
|
| ## Training-data provenance (third-party texts in the synthetic documents) |
|
|
| **No training document, panel, corpus or source text is distributed in this repository.** |
| The adapter was trained on synthetic documents, and some of those documents were produced by **editing |
| third-party natural texts**. Those texts are listed here so that their provenance is on the record, and |
| so that a reviewer doing lawful-sourcing diligence can see what was used without having to ask. |
|
|
| The model card discloses the provenance of the third-party texts the training panel was edited from, and attributes them. This is provenance disclosure and attribution as good practice, and it supports documented-lawful-sourcing procurement review. It is NOT a concession that the trained weights are a derivative work or an adaptation of those texts; that question is open and nothing in this package answers it. |
|
|
| 13 of the 42 reviewed sources were used in the training panel: |
|
|
| | | Source | URL | Licence, as recorded | |
| |---|---|---|---| |
| | | Django's security policies | [https://docs.djangoproject.com/en/dev/internals/security/](https://docs.djangoproject.com/en/dev/internals/security/) | BSD 3-Clause (Django project LICENSE, which covers the documentation in the django/django repository) | |
| | **CC BY-SA** | Wikipedia:Arbitration/Policy (English Wikipedia arbitration policy) | [https://en.wikipedia.org/wiki/Wikipedia:Arbitration/Policy](https://en.wikipedia.org/wiki/Wikipedia:Arbitration/Policy) | CC BY-SA 4.0 (Wikipedia text; attribution: English Wikipedia contributors, 'Wikipedia:Arbitration/Policy') | |
| | | Common Paper Mutual Non-Disclosure Agreement, Version 1.0 – Standard Terms | [https://commonpaper.com/standards/mutual-nda/1.0/](https://commonpaper.com/standards/mutual-nda/1.0/) | CC BY 4.0 (stated in the agreement footer and in the CommonPaper/Mutual-NDA repository README: 'free to use and modify under CC BY 4.0') | |
| | | 36 CFR Part 2 (National Park Service) - Resource Protection, Public Use and Recreation: sections 2.10, 2.13, 2.14, 2.15, 2.16, 2.21 and 2.22 | [https://www.ecfr.gov/current/title-36/chapter-I/part-2](https://www.ecfr.gov/current/title-36/chapter-I/part-2) | US Government work (public domain): Code of Federal Regulations text, not subject to copyright (17 U.S.C. 105) | |
| | | NSF Proposal & Award Policies & Procedures Guide (PAPPG, NSF 24-1), Chapter IV: Non-Award Decisions and Transactions | [https://www.nsf.gov/policies/pappg/24-1/ch-4-non-award-decisions-transactions](https://www.nsf.gov/policies/pappg/24-1/ch-4-non-award-decisions-transactions) | US Government work (public domain): U.S. National Science Foundation policy guide, not subject to copyright (17 U.S.C. 105) | |
| | | GOV.UK: Make a court claim for money | [https://www.gov.uk/api/content/make-court-claim-for-money](https://www.gov.uk/api/content/make-court-claim-for-money) | Open Government Licence v3.0 | |
| | | 42 CFR 68: NIH Loan Repayment Programs (2025 edition) | [https://www.govinfo.gov/content/pkg/CFR-2025-title42-vol1/xml/CFR-2025-title42-vol1-part68.xml](https://www.govinfo.gov/content/pkg/CFR-2025-title42-vol1/xml/CFR-2025-title42-vol1-part68.xml) | US Government work (public domain) | |
| | **CC BY-SA** | WordPress.com (Automattic) Terms of Service, last updated April 10, 2026 | [https://github.com/Automattic/legalmattic/blob/master/Terms%20of%20Service/WordPress.com/EN-Terms-of-Service.md](https://github.com/Automattic/legalmattic/blob/master/Terms%20of%20Service/WordPress.com/EN-Terms-of-Service.md) | CC BY-SA 4.0 (Automattic/legalmattic LICENSE.txt and README; the Terms themselves state they are available under a Creative Commons Sharealike license) | |
| | **CC BY-SA** | GitLab Handbook: Global Travel and Expense Policy (sections 1 to 3) | [https://handbook.gitlab.com/handbook/finance/expenses/](https://handbook.gitlab.com/handbook/finance/expenses/) | CC BY-SA 4.0 (GitLab handbook content; licence badge in the footer of handbook.gitlab.com pages) | |
| | | Sourcegraph Handbook: Spending company money | [https://github.com/sourcegraph/handbook/blob/main/content/benefits-pay-perks/benefits-perks/spending-company-money.md](https://github.com/sourcegraph/handbook/blob/main/content/benefits-pay-perks/benefits-perks/spending-company-money.md) | Apache License 2.0 (LICENSE of the public sourcegraph/handbook repository) | |
| | | Project Jupyter Governance: Executive Council | [https://raw.githubusercontent.com/jupyter/governance/main/docs/executive_council.md](https://raw.githubusercontent.com/jupyter/governance/main/docs/executive_council.md) | CC0 1.0 Universal (jupyter/governance repository LICENSE.md) | |
| | | 29 CFR 1904.30–1904.34: establishment records and annual summaries (2025 edition) | [https://www.govinfo.gov/content/pkg/CFR-2025-title29-vol5/xml/CFR-2025-title29-vol5-part1904.xml](https://www.govinfo.gov/content/pkg/CFR-2025-title29-vol5/xml/CFR-2025-title29-vol5-part1904.xml) | US Government work (public domain) | |
| | | 31 CFR 1.2, 1.4 and 1.6: Treasury FOIA requests and appeals (2025 edition) | [https://www.govinfo.gov/content/pkg/CFR-2025-title31-vol1/xml/CFR-2025-title31-vol1-part1.xml](https://www.govinfo.gov/content/pkg/CFR-2025-title31-vol1/xml/CFR-2025-title31-vol1-part1.xml) | US Government work (public domain) | |
|
|
| **3 of these are recorded as CC BY-SA 4.0** — *Wikipedia:Arbitration/Policy (English Wikipedia arbitration policy)*, *WordPress.com (Automattic) Terms of Service, last updated April 10, 2026*, *GitLab Handbook: Global Travel and Expense Policy (sections 1 to 3)*. Share-alike is the one term |
| attribution cannot cure. Whether a share-alike obligation can propagate through training into model |
| weights **is legally unsettled**; there is no authority settling it in either direction, and the |
| project's own licence review explicitly declines to infer one. The owner of this release |
| **accepted that residual risk on 2026-09-21 rather than resolving |
| it**, and kept this package under Apache-2.0. A reader should treat the question as open, not answered. |
|
|
| **On the strength of this evidence.** The licences above are **as recorded by the person who collected |
| each source**, from the source's own stated terms at the time of collection. The review records |
| `evidence_level: "authoring metadata assertion, not archived governing licence text"` and |
| `upstream_terms_independently_verified: false` for every row. No governing licence text was archived |
| alongside most of these sources, and this listing should not be read as a licence audit. |
|
|
| Listing these sources is **provenance disclosure and attribution as good practice**. It is not a |
| statement that the trained weights are a derivative work, an adaptation, or a copy of any of these |
| texts. |
|
|
|
|
| ## Third-party dependency licences |
|
|
| `requirements.lock` names the packages the serving layer needs. **None of them is redistributed in |
| this repository** — you install them yourself from their own publishers — so Apache-2.0 section 4(a) |
| imposes no bundled-notice obligation here and no dependency licence text is packaged. This summary |
| exists because a reviewer will ask for one. |
|
|
| Each licence in the table is the one declared in that distribution's OWN package metadata -- the `METADATA` file of an installed wheel -- read from a copy on the maintainer's machine. Where no copy existed, the row says NOT VERIFIED instead of guessing. |
|
|
| | Package | Pinned as | Licence, as declared by the distribution itself | |
| |---|---|---| |
| | `torch` | 2.13.0 | BSD-3-Clause — read from version 2.8.0 | |
| | `torchvision` | 0.28.0 | NOT VERIFIED | |
| | `transformers` | 5.17.0 | Apache 2.0 License | |
| | `flash-linear-attention` | 0.5.2 | NOT VERIFIED | |
| | `safetensors` | unpinned in the qualified image | Apache Software License — read from version 0.8.0 | |
| | `accelerate` | unpinned in the qualified image | Apache (Apache Software License) — read from version 1.15.0 | |
| | `numpy` | unpinned in the qualified image | BSD-3-Clause AND 0BSD AND MIT AND Zlib AND CC0-1.0 — read from version 2.5.3 | |
| | `scipy` | unpinned in the qualified image | BSD License (classifier); the METADATA `License` field carries the Enthought / SciPy Developers copyright line rather than an SPDX identifier — read from version 1.18.1 | |
| | `pillow` | unpinned in the qualified image | MIT-CMU — read from version 12.3.0 | |
|
|
| **Two rows say NOT VERIFIED and mean it.** A licence read from one installed version is evidence about that version only. Where the table names a version different from the pin, that is the version whose metadata was actually read, and the pinned version's own metadata could differ. Treat this table as a starting point for |
| your own review, not as a legal opinion, and re-check the distributions you actually install. |
|
|
| Full evidence paths for each row are in the release descriptor |
| (`release/solomon-release.json` → `dependency_licences`), which is not distributed; the same |
| information is repeated in the comments of `requirements.lock`. |
|
|
|
|
| ## Licence and attribution |
|
|
| Copyright 2026 Doccy Pty Ltd. |
|
|
| This repository is licensed **Apache-2.0** — the adapter and head weights, the calibration artifact, |
| the serving code and the documentation alike. See `LICENSE` and `NOTICE`. |
|
|
| It is a **derivative work** of `Qwen/Qwen3.8-27B`, Copyright 2026 Alibaba Cloud, licensed under Apache-2.0. |
| The upstream licence text is reproduced verbatim at `licenses/Qwen-Apache-2.0.txt` (sha256 |
| `bbedc3fda3305820b977265f01b8619d87570a6739de3a5582c3464840f1e57a`). `NOTICE` carries the attribution required by section 4(c) and |
| `MODIFICATIONS.md` the statement of changes required by section 4(b). |
|
|
| No `NOTICE` file exists in the upstream repository at the pinned revision (HTTP 404, checked |
| 2026-09-18), so section 4(d) carries nothing forward and no upstream attribution text has been |
| invented. |
|
|
| "Qwen" and "Alibaba Cloud" are used nominatively to identify the base model. Apache-2.0 section 6 |
| grants no trademark rights and none are claimed. No endorsement or affiliation is implied. |
|
|
| ## Verify what you downloaded |
|
|
| ```sh |
| shasum -a 256 adapter/adapter.safetensors # d122466d430a058bb6457d919f811160e97fbd20149f4f24ca455c5d83e360a0 |
| shasum -a 256 adapter/heads.npz # f766d752d7768a419a9657155cf27f042834d9de29392cf7470d8725130e67ab |
| shasum -a 256 serving/readout-temperature-v3.json |
| # -> 1a2285d8fd56d17ee1d06a1e9fce866cc0d3b0263730754babb11deea5f1f7c9 |
| ``` |
|
|
| `MANIFEST.json` carries the size and sha256 of every file in this repository. |
|
|
| The calibration artifact has **two legitimate and different hashes**, and confusing them makes a sound |
| provenance chain look tampered with. `1a2285d8…` is the *file* hash, what `shasum` |
| returns. `945bad44…` is the artifact's own internal `sha256` field, computed over its |
| contents with that field removed — a self-referential field cannot hash the file containing it. The |
| loader verifies the payload hash; use the file hash to check the file you were given. Both are recorded |
| in the serving binding's `provenance`, under those names. |
|
|
| ## Release record |
|
|
| Machine-readable identity for citation and pinning. The model identity, the calibration and the runtime |
| binding move together; pin by revision. |
|
|
| | | | |
| |---|---| |
| | Repository | `DoccyHealth/Solomon` | |
| | Release | `1.1.0`, 2026-09-21 | |
| | Serving contract | `solomon-v1` | |
| | Adapter sha256 | `d122466d430a058bb6457d919f811160e97fbd20149f4f24ca455c5d83e360a0` | |
| | Heads sha256 | `f766d752d7768a419a9657155cf27f042834d9de29392cf7470d8725130e67ab` | |
| | Base model | `Qwen/Qwen3.8-27B` at `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` | |
| | Runtime binding sha256 (payload) | `0add0efda28902180db757a12160953e0f5d8dc303decc27b2cfda84db900da8` | |
| | Calibration sha256 (file / payload) | `1a2285d8fd56d17ee1d06a1e9fce866cc0d3b0263730754babb11deea5f1f7c9` / `945bad449b7f5ffc88e597277d632fbab81c3c8729e22c8babd3f4a45fe1378b` | |
| | Runtime fingerprint | `7d529382321e0e14131a643920d7522b85d1b521d249db89fba839b3fd0f8bea` | |
| | Readout | `four_collapsed` | |
| | Served temperatures | boolean 1.0 · multilabel 1.0 · single 1.0 · ordered 1.0 | |
|
|
