Model card, figures and technical report for the 9B release

Browse files

Files changed (13) hide show

.gitattributes +1 -0
NOTICE +9 -2
README.md +129 -77
assets/banner.png +0 -0
assets/calibration_gold.png +2 -2
assets/calibration_silver.png +3 -0
assets/confusion_treatment_lev.png +2 -2
assets/headline_gold.png +0 -0
assets/headline_silver.png +0 -0
assets/latency.png +0 -0
assets/per_task_gold.png +2 -2
assets/per_task_silver.png +2 -2
lev-technical-report.pdf +2 -2

.gitattributes CHANGED Viewed

@@ -39,3 +39,4 @@ assets/confusion_treatment_lev.png filter=lfs diff=lfs merge=lfs -text
 assets/per_task_gold.png filter=lfs diff=lfs merge=lfs -text
 assets/per_task_silver.png filter=lfs diff=lfs merge=lfs -text
 lev-technical-report.pdf filter=lfs diff=lfs merge=lfs -text

 assets/per_task_gold.png filter=lfs diff=lfs merge=lfs -text
 assets/per_task_silver.png filter=lfs diff=lfs merge=lfs -text
 lev-technical-report.pdf filter=lfs diff=lfs merge=lfs -text
+assets/calibration_silver.png filter=lfs diff=lfs merge=lfs -text

NOTICE CHANGED Viewed

@@ -1,4 +1,11 @@
-d3-flash (Decision 3.0)
 Copyright 2026 vLLM Semantic Router Team
-Built on Qwen/Qwen3.5-9B (Apache-2.0).

+Lev
+Copyright 2026 Jonathan Kleiman
+Lev is a fine-tune of d3-mini. It contains the d3-mini runtime and weights with merged adapters and a tuned readout.
+--- Upstream notice ---
+d3-mini (Decision 3.0)
 Copyright 2026 vLLM Semantic Router Team
+Built on Qwen/Qwen3.5-4B (Apache-2.0).

README.md CHANGED Viewed

@@ -4,7 +4,7 @@ language:
 - en
 library_name: transformers
 pipeline_tag: zero-shot-classification
-base_model: vllm-sr/d3-mini
 tags:
 - legal
 - decision-model
@@ -19,80 +19,130 @@ tags:
 # Lev
-Lev is a 4B decision model for case law. You give it an excerpt of a court decision and one or more typed questions, and it returns a probability for every answer option. It does not generate text. It was built for citator work: deciding how a decision treats a case it cites, whether the cited case is an earlier stage of the same proceeding, and what happened to it on appeal. It also answers ten other classification questions about decisions and passages.
-Lev is a fine-tune of [d3-mini](https://huggingface.co/vllm-sr/d3-mini) (Qwen3.5-4B backbone, Apache-2.0) and loads with the same runtime.
 | | |
 | --- | --- |
-| **Parameters** | 4.54B (the d3-mini checkpoint with merged adapters and a tuned readout) |
 | **Input** | Text: an excerpt of a decision, up to a few thousand tokens |
 | **Questions** | Choice (named options with definitions) and yes/no |
-| **Trained for** | 13 questions on Canadian court decisions in English |
 | **License** | Apache-2.0 |
 ## Results
-Scores are on a held-out test set of 1428 excerpts from decisions dated after the training data ends, with 5367 judgments in all. Every system got the same excerpt, instructions and option definitions. The gold set is the 4674 judgments on which two independent annotators agreed; the silver set is every judgment, labelled by the first annotator only. Both annotators are models, not people. Details and caveats are in the [technical report](lev-technical-report.pdf).
-**Gold (two annotators agree), accuracy %**
-| Task | n | **Lev** | Jev 1.13 | OpenAI Decisions (gpt-6-luna) | d3-mini (base) |
 |---|---|---|---|---|---|
-| Treatment of the cited case | 407 | **82.8** | 79.9 | 69.0 | 68.8 |
-| Appeal history | 590 | 93.2 | **93.9** | 86.4 | 86.4 |
-| Reported negative history | 596 | 95.6 | **97.7** | 95.8 | 91.9 |
-| Cited in dissent | 598 | 96.0 | 95.5 | **97.0** | 83.8 |
-| *Mean of the four citator tasks* | 2191 | **91.9** | 91.7 | 87.1 | 82.7 |
-| *Mean of seven decision-level tasks* | 2123 | 94.8 | 95.8 | **96.0** | 91.5 |
-| *Mean of two passage-level tasks* | 360 | **89.3** | 87.9 | 85.6 | 83.1 |
-| *Mean of all 13* | 4674 | 93.1 | **93.4** | 91.7 | 87.5 |
-**Silver (first annotator), accuracy %**
-| Task | n | **Lev** | Jev 1.13 | OpenAI Decisions (gpt-6-luna) | d3-mini (base) |
 |---|---|---|---|---|---|
-| Treatment of the cited case | 623 | **68.7** | 57.8 | 53.1 | 52.2 |
-| Appeal history | 623 | **90.5** | 89.7 | 82.2 | 82.5 |
-| Reported negative history | 623 | 93.1 | **94.2** | 93.4 | 88.4 |
-| Cited in dissent | 623 | **94.4** | 92.6 | 94.1 | 82.5 |
-| *Mean of the four citator tasks* | 2492 | **86.7** | 83.6 | 80.7 | 76.4 |
-| *Mean of seven decision-level tasks* | 2415 | **89.7** | 89.0 | 89.2 | 84.9 |
-| *Mean of two passage-level tasks* | 460 | **76.1** | 70.9 | 67.9 | 70.4 |
-| *Mean of all 13* | 5367 | **86.7** | 84.5 | 83.3 | 80.1 |
-![Accuracy by task on the gold set](assets/per_task_gold.png)
-On the gold set Lev and Jev are level over all 13 tasks (93.1% and 93.4%; the 95% interval on the difference is -1.2 to +0.3 points), and Lev is ahead of the OpenAI Decisions API by 1.6 points (+0.7 to +2.6). On the silver set Lev is ahead of both, by 2.2 and 3.7 points. Silver labels come from the same annotator as Lev's training labels, so that margin flatters it; the gold comparison is the fairer one.
-Where Lev is better and worse:
-- **Citation treatment** is its best task: 82.8% on gold against 79.9% (Jev) and 69.0% (OpenAI), with macro-F1 of 70.5% against 60.9% and 60.2%.
-- It assigns a treatment to a case that is only quoted or only argued by a party less often than the hosted systems (7.7% of such citations on gold, against 13.8% and 12.3%; there are only 65 of them, so treat the ordering as tentative).
-- It is behind on **area of law** (85.7% against 93.2% and 92.5%). For that question alone, use a hosted model.
-- **Negative treatment needs a threshold.** If you take the top answer as the flag, Lev catches 71% of the 38 negative treatments in the gold set, against 97% for Jev and 84% for OpenAI. If you flag a citation whenever the probabilities of `questioned`, `not_followed`, `overruled` and `superseded_by_statute` sum to 0.2 or more, it catches 95% at 88% precision (Jev 100% at 68%, OpenAI 97% at 90%). Do not use Lev's top answer as a negative-treatment flag.
-```python
-NEGATIVE = ("questioned", "not_followed", "overruled", "superseded_by_statute")
-p = result["answers"]["treatment"]["probabilities"]
-flag_for_review = sum(p[k] for k in NEGATIVE) >= 0.2
-```
-### Public, human-labelled tasks
-None of these tasks were used in training. The Australian set has treatment labels written by the deciding judges; the others are LegalBench tasks about judicial decisions.
-| Task | n | **Lev** | Jev 1.13 | OpenAI Decisions (gpt-6-luna) | d3-mini (base) |
 |---|---|---|---|---|---|
-| Treatment, Federal Court of Australia (judge-labelled) | 818 | **42.8** | 40.8 | 38.0 | 40.0 |
-| Overruling (LegalBench) | 300 | 95.0 | 93.3 | 92.0 | **95.3** |
-| Tax Court of Canada outcomes (LegalBench) | 244 | 98.0 | **99.2** | 96.7 | 98.4 |
-| Function of decision section (LegalBench) | 300 | **47.3** | 44.0 | 45.7 | 46.7 |
-| Oral argument question purpose (LegalBench) | 300 | 45.0 | 44.3 | **47.7** | 39.7 |
-| SCALR holdings (LegalBench) | 300 | 76.3 | 79.7 | **83.6** | 75.0 |
-| *Mean of six* |  | **67.4** | 66.9 | 67.3 | 65.8 |
-Fine-tuning did not cost Lev its general ability: its mean over the six tasks is 67.4%, against 65.8% for the base model, and all four systems are within two points of each other. The citator skill transfers to Australia only slightly (42.8% against 40.0% for the base model, a difference that is not statistically reliable). Every system is right about 86% of the time on whether the Australian court limited or rejected the case, and poor at telling which positive label the judge chose (recall on "followed" is under 20% for Lev and both hosted systems). The report discusses why.
 ## Quickstart
@@ -134,7 +184,7 @@ result = model.system_one(
 print(result["answers"]["treatment"]["choice"], result["answers"]["treatment"]["probabilities"])
 ```
-The 13 questions Lev was trained on, with the exact instructions and option definitions, are in [`questions.py`](questions.py). Use that wording for the trained tasks. You can also ask your own questions with your own options, as with d3-mini; the public-task table above gives a sense of how far that carries.
 ### How to build the excerpt
@@ -166,48 +216,50 @@ Three of the treatment options exist to keep a citator honest. `inside_quotation
 ## Calibration
-A single temperature (1.2) was fitted on held-out data drawn at natural label frequencies and is stored in `decision_config.json`. Averaged over the 13 tasks, expected calibration error on the gold set is 2.9% for Lev, 4.1% for Jev and 3.7% for OpenAI. Answers Lev gives with at least 90% confidence are right 99% of the time; in the 80 to 90% band, 89%. Ranking its citator answers by confidence, Lev can decide 89% of the gold judgments before accuracy on the accepted set drops below 98% (Jev 87%, OpenAI 73%).
-![Reliability on the gold set](assets/calibration_gold.png)
 ## Training
 Lev was trained on a private corpus of Canadian court decisions with model-assisted labels; test labels were adjudicated by a second, independent model. Training decisions are dated before 1 July 2024 and test decisions on or after it.
-- Low-rank adapters (rank 32) on every linear layer of the upper 16 of 32 decoder layers, plus the readout, trained through a reimplementation of the d3 forward pass, since the released runtime is inference-only.
-- Three stages, 89,718 training instances in all: a class-balanced round split over three GPUs and averaged, a second class-balanced round, and a short final stage on data sampled with square-root label frequencies so that the model stops over-predicting rare answers.
-- The order of the options is shuffled for every training example, so the model has to read the definitions.
-- Training took about 4.4 A100 GPU-hours in total.
-- Adapters are merged into the bfloat16 weights. The merged model scored the same as the unmerged one on held-out data.
-The adapters and the code are in [jfloydian/lev-lora](https://huggingface.co/jfloydian/lev-lora), and the [technical report](lev-technical-report.pdf) has the details.
 ## Limitations
-- **Both annotators of the main test set are models.** "Gold" means two independent model annotators agree. No person reviewed the labels. The public tasks above have human labels.
-- **The test set over-samples rare labels.** The scores describe how each system handles each kind of case, not how often it will be right on a random citation.
 - **Canadian decisions in English with numbered paragraphs.** Other jurisdictions, French-language decisions and differently built excerpts are outside what was tested, apart from the public tasks above.
-- **Excerpts hide things.** Lev sees the paragraphs around a citation, not the decision. Positive treatment in particular is often only visible from how the rest of the decision uses the case, and two careful readers of the same decision agree that a case was followed or applied only about half the time.
 - **It classifies an excerpt; it is not a citator.** It does not find citations, resolve them to cases, or check that the excerpt is complete. Treat its output as a lead to check against the decision. It is not legal advice.
 - **Images and video** are supported by the d3 runtime it inherits, but Lev was trained and tested on text only.
-- **One training decision was prompted by the test set.** The final training stage, which corrects the model's over-prediction of rare answers, was added after test scores of an earlier checkpoint showed the problem. It used training data only and the released model was chosen on held-out development data, but the test set is not untouched by design decisions.
-- **Hosted baselines change.** Jev 1.13 and the OpenAI Decisions API (in public beta) were called on a single day.
 ## Files
-- `model-0000{1,2}-of-00002.safetensors`, `readout.safetensors`, `decision_config.json`: the model in d3 package format.
-- `d3_*.py`, `modeling_d3.py`, `pipeline_d3.py`: the d3 runtime, unchanged from d3-mini.
 - `questions.py`, `excerpts.py`: the trained questions and the excerpt builders.
 - `lev-technical-report.pdf`: the technical report.
 - The LoRA adapters and the training and evaluation code are in [jfloydian/lev-lora](https://huggingface.co/jfloydian/lev-lora).
 ## License and attribution
-Apache-2.0. Lev is built on d3-mini by the vLLM Semantic Router team, which is built on Qwen3.5-4B; their licence and notice files are included. If you use Lev, please credit Jonathan Kleiman.
 ```bibtex
 @misc{kleiman2026lev,
-  title  = {Lev: a 4B open decision model for citator and case-law classification},
   author = {Kleiman, Jonathan},
   year   = {2026},
   howpublished = {\url{https://huggingface.co/jfloydian/lev}}

 - en
 library_name: transformers
 pipeline_tag: zero-shot-classification
+base_model: vllm-sr/d3-flash
 tags:
 - legal
 - decision-model
 # Lev
+Lev is a 9B decision model for case law. You give it an excerpt of a court decision and one or more typed questions, and it returns a probability for every answer option. It does not generate text. It was built for citator work: deciding how a decision treats a case it cites, whether the cited case is an earlier stage of the same proceeding, and what happened to it on appeal. It also answers ten other classification questions about decisions and passages.
+Lev is a fine-tune of [d3-flash](https://huggingface.co/vllm-sr/d3-flash) (Qwen3.5-9B backbone, Apache-2.0) and loads with the same runtime.
+- **Treatment of a cited case: 90.8%** on agreed labels, against 80.7% for Jev 1.13 and 66.8% for the OpenAI Decisions API, on 238 held-out citations from decisions dated after the training data ends.
+- **Four citator questions: 96.4%** (Jev 94.2%, OpenAI 89.2%, d3-flash 90.4%); **all 13 questions: 96.4%** (Jev 95.2%, OpenAI 93.1%, d3-flash 91.9%).
+- **Negative treatment:** at a 0.2 threshold on the summed negative probability, recall 100% at precision 72% (Jev 100% at 57%, OpenAI 77% at 83%).
+- **Invented treatments:** 1.9% of citations that are not the court's own receive a treatment (Jev 15.1%, OpenAI 7.5%).
+- **Calibration:** expected calibration error 3.3% (Jev 4.1%, OpenAI 3.5%); 97% of citator answers can be accepted at 98% accuracy (Jev 96%, OpenAI 84%).
+- **Area of law (49 options): 93.2%** (Jev 92.6%, OpenAI 91.3%). The difference from Jev is within the 95% interval (-2.5 to +3.7 points).
+- **Held-out legal tasks:** 87.8% mean over 20 held-out LegalBench tasks from other areas of law (d3-flash 86.9%, Jev 83.7%, OpenAI 82.4%).
+- **Speed:** a median of 591 ms to answer all questions about one excerpt on one A100 through the stock runtime.
+- **Open:** Apache-2.0 weights, adapters, training and evaluation code, and a [technical report](lev-technical-report.pdf).
 | | |
 | --- | --- |
+| **Parameters** | 8.39B (the d3-flash checkpoint with merged adapters and a tuned readout) |
 | **Input** | Text: an excerpt of a decision, up to a few thousand tokens |
 | **Questions** | Choice (named options with definitions) and yes/no |
+| **Trained for** | 13 questions on Canadian court decisions in English, plus public legal classification tasks |
 | **License** | Apache-2.0 |
 ## Results
+Scores are on a held-out test set of 1,028 excerpts from decisions dated after the training data ends, with 4,550 judgments in all. No version of Lev was scored on it until the weights were frozen, and the frozen model was scored once. Every system got the same excerpt, instructions and option definitions. The first table uses the 4,030 judgments on which two independent annotators agreed; the second uses every judgment, labelled by the first annotator only. Both annotators are models, not people. Details and caveats are in the [technical report](lev-technical-report.pdf).
+**Agreed labels (two annotators agree), accuracy %**
+| Task | n | **Lev** | Jev 1.13 | OpenAI Decisions (gpt-6-luna) | d3-flash (base) |
 |---|---|---|---|---|---|
+| Treatment of the cited case | 238 | **90.8** | 80.7 | 66.8 | 75.2 |
+| Appeal history | 361 | 97.2 | **97.5** | 92.5 | 94.2 |
+| Reported negative history | 371 | 97.6 | **98.7** | 97.6 | 97.0 |
+| Cited in dissent | 386 | **100.0** | **100.0** | **100.0** | 95.3 |
+| *Mean of the four citator tasks* | 1356 | **96.4** | 94.2 | 89.2 | 90.4 |
+| Disposition | 297 | 89.9 | 90.2 | **91.6** | 88.2 |
+| Decision type | 362 | **98.1** | 94.5 | 97.0 | 87.8 |
+| Proceeding | 366 | 96.2 | **97.5** | 93.4 | 94.3 |
+| Area of law (49 options) | 311 | **93.2** | 92.6 | 91.3 | 80.7 |
+| Publication ban | 389 | **99.7** | 99.5 | 99.5 | 99.5 |
+| Self-represented party | 371 | **99.2** | 98.4 | 97.6 | 98.7 |
+| Separate opinions | 393 | **100.0** | **100.0** | 99.7 | **100.0** |
+| *Mean of seven decision-level tasks* | 2489 | **96.6** | 96.1 | 95.7 | 92.7 |
+| Standard of review | 77 | 94.8 | **96.1** | 94.8 | 93.5 |
+| Test-element finding | 108 | **96.3** | 91.7 | 88.0 | 90.7 |
+| *Mean of two passage-level tasks* | 185 | **95.6** | 93.9 | 91.4 | 92.1 |
+| *Mean of all 13* | 4030 | **96.4** | 95.2 | 93.1 | 91.9 |
+**First annotator's labels, accuracy %**
+| Task | n | **Lev** | Jev 1.13 | OpenAI Decisions (gpt-6-luna) | d3-flash (base) |
 |---|---|---|---|---|---|
+| Treatment of the cited case | 386 | **74.1** | 58.3 | 50.0 | 55.2 |
+| Appeal history | 386 | **93.3** | **93.3** | 87.8 | 89.9 |
+| Reported negative history | 386 | 95.1 | **95.6** | **95.6** | 94.8 |
+| Cited in dissent | 386 | **100.0** | **100.0** | **100.0** | 95.3 |
+| *Mean of the four citator tasks* | 1544 | **90.6** | 86.8 | 83.4 | 83.8 |
+| Disposition | 394 | **81.2** | 76.4 | 75.1 | 73.4 |
+| Decision type | 394 | 92.4 | 89.6 | **92.9** | 83.8 |
+| Proceeding | 394 | 91.6 | **92.4** | 89.8 | 89.3 |
+| Area of law (49 options) | 394 | **79.9** | 76.4 | 77.7 | 70.6 |
+| Publication ban | 394 | **98.7** | 98.5 | 98.5 | 98.5 |
+| Self-represented party | 394 | **94.7** | 93.7 | 93.1 | 93.7 |
+| Separate opinions | 394 | **99.7** | **99.7** | 99.5 | **99.7** |
+| *Mean of seven decision-level tasks* | 2758 | **91.2** | 89.5 | 89.5 | 87.0 |
+| Standard of review | 128 | **78.9** | 62.5 | 60.9 | 59.4 |
+| Test-element finding | 120 | **90.8** | 85.8 | 80.8 | 83.3 |
+| *Mean of two passage-level tasks* | 248 | **84.9** | 74.2 | 70.9 | 71.4 |
+| *Mean of all 13* | 4550 | **90.0** | 86.3 | 84.8 | 83.6 |
+![Accuracy by task on the agreed labels](assets/per_task_gold.png)
+Averaged over the 13 questions on agreed labels, Lev scores 96.4%, Jev 95.2% and the OpenAI Decisions API 93.1%; the base model before fine-tuning scores 91.9%. Pooled over all agreed judgments, Lev is +0.9 points from Jev (95% interval +0.2 to +1.5) and +2.6 from OpenAI (+1.9 to +3.4). On the four citator questions the differences are +1.4 (+0.3 to +2.6) and +5.5 (+4.0 to +6.9).
+- **Treatment of the cited case** is where Lev is strongest: 90.8% against 80.7% for Jev and 66.8% for OpenAI.
+- **Area of law**: Lev scores 93.2%, ahead of Jev by 0.6 points and ahead of OpenAI by 1.9 points.
+- **Negative treatment** (questioned, not followed, overruled, superseded by statute): flagging a citation when the four negative options sum to a probability of 0.2 or more, Lev catches 100% of the 13 negative treatments at 72% precision; Jev 100% at 57%; OpenAI 77% at 83%. Use the threshold, not the top answer, and set it on your own data.
+- **Invented treatments**: for citations that are not the court's own (inside a quotation, in a party's argument, or not named), Lev assigns a treatment 1.9% of the time, Jev 15.1% and OpenAI 7.5%.
+### Held-out legal tasks
+20 LegalBench tasks from other areas of law (contract clauses, privacy policies, legal-aid triage, supply chain disclosures, federal jurisdiction, statutory entailment), 1,890 items, each asked as a yes/no question with one plain instruction. The suite was assembled before the released model was trained, its tasks were left out of the training data, and the released model was scored on it once.
+| LegalBench task | n | **Lev** | Jev 1.13 | OpenAI Decisions (gpt-6-luna) | d3-flash (base) |
+|---|---|---|---|---|---|
+| `contract_nli_confidentiality_of_agreement` | 82 | **95.1** | 91.5 | 84.1 | **95.1** |
+| `contract_nli_explicit_identification` | 100 | **92.0** | **92.0** | 86.0 | 91.0 |
+| `contract_nli_notice_on_compelled_disclosure` | 100 | 98.0 | **99.0** | 97.0 | 97.0 |
+| `contract_nli_permissible_copy` | 87 | 79.3 | 77.0 | **80.5** | **80.5** |
+| `contract_nli_survival_of_obligations` | 100 | 95.0 | 92.0 | 94.0 | **96.0** |
+| `cuad_anti-assignment` | 100 | **94.0** | 85.0 | 85.0 | 89.0 |
+| `cuad_audit_rights` | 100 | **99.0** | 90.0 | 87.0 | 92.0 |
+| `cuad_insurance` | 100 | **95.0** | 88.0 | 87.0 | 94.0 |
+| `cuad_license_grant` | 100 | 91.0 | 90.0 | **92.0** | 90.0 |
+| `cuad_termination_for_convenience` | 100 | 95.0 | **97.0** | 96.0 | 96.0 |
+| `diversity_6` | 100 | 71.0 | 66.0 | 39.0 | **74.0** |
+| `learned_hands_benefits` | 66 | **90.9** | 87.9 | 83.3 | **90.9** |
+| `learned_hands_consumer` | 100 | **83.0** | 78.0 | 72.0 | **83.0** |
+| `learned_hands_health` | 100 | 83.0 | 87.0 | **89.0** | 81.0 |
+| `learned_hands_immigration` | 100 | **99.0** | 98.0 | **99.0** | **99.0** |
+| `legal_reasoning_causality` | 55 | 81.8 | 60.0 | 80.0 | **83.6** |
+| `opp115_data_retention` | 100 | **76.0** | 66.0 | 74.0 | 71.0 |
+| `opp115_user_choice_control` | 100 | **82.0** | 79.0 | **82.0** | 79.0 |
+| `sara_entailment` | 100 | 77.0 | 77.0 | **78.0** | **78.0** |
+| `supply_chain_disclosure_best_practice_certification` | 100 | **78.0** | 74.0 | 64.0 | 77.0 |
+| *Mean* |  | **87.8** | 83.7 | 82.4 | 86.9 |
+Lev's mean is 87.8%, against 86.9% for its base model, 83.7% for Jev and 82.4% for OpenAI.
+### Judicial decisions from other courts
+The Australian set has treatment labels written by the deciding judges; the others are LegalBench tasks about judicial decisions. None were used in training. This suite was also scored for several candidate models during development.
+| Task | n | **Lev** | Jev 1.13 | OpenAI Decisions (gpt-6-luna) | d3-flash (base) |
 |---|---|---|---|---|---|
+| Treatment, Federal Court of Australia (judge-labelled) | 818 | **44.4** | 40.8 | 38.0 | 39.7 |
+| Overruling (LegalBench) | 300 | 94.7 | 93.3 | 92.0 | **95.3** |
+| Tax Court of Canada outcomes (LegalBench) | 244 | 96.7 | **99.2** | 96.7 | 98.4 |
+| Function of decision section (LegalBench) | 300 | 40.7 | 44.0 | **45.7** | 41.7 |
+| Oral argument question purpose (LegalBench) | 300 | 43.0 | 44.3 | **47.7** | 42.0 |
+| SCALR holdings (LegalBench) | 300 | 75.3 | 79.7 | **83.6** | 75.3 |
+| *Mean of six* |  | 65.8 | 66.9 | **67.3** | 65.4 |
+Nobody does well on the Australian labels. Every system is right about 85% of the time on whether the Australian court limited or rejected the case, and close to guessing on which positive label the judge chose. The report explains why: two careful readers of the same decision agree that a case was followed or applied only about half the time.
 ## Quickstart
 print(result["answers"]["treatment"]["choice"], result["answers"]["treatment"]["probabilities"])
 ```
+The 13 questions Lev was trained on, with the exact instructions and option definitions, are in [`questions.py`](questions.py). Use that wording for the trained tasks. You can also ask your own questions with your own options, as with d3-flash; the public-task table above gives a sense of how far that carries.
 ### How to build the excerpt
 ## Calibration
+A single temperature (1.6) was fitted on held-out data drawn at natural label frequencies and is stored in `decision_config.json`. Averaged over the 13 tasks, expected calibration error on agreed labels is 3.3% for Lev, 4.1% for Jev and 3.5% for OpenAI. Answers Lev gives with at least 90% confidence are right 100% of the time; in the 80 to 90% band, 88%. Ranking its citator answers by confidence, Lev can decide 97% of the agreed judgments before accuracy on the accepted set drops below 98% (Jev 96%, OpenAI 84%).
+![Reliability on the agreed labels](assets/calibration_gold.png)
 ## Training
 Lev was trained on a private corpus of Canadian court decisions with model-assisted labels; test labels were adjudicated by a second, independent model. Training decisions are dated before 1 July 2024 and test decisions on or after it.
+- Low-rank adapters (rank 64) on every linear layer of the upper 16 of 32 decoder layers, plus the readout, trained through a reimplementation of the d3 forward pass, since the released runtime is inference-only.
+- 127,888 examples of Lev's own questions, 78,300 examples from public legal classification datasets (LegalBench tasks that are in none of the evaluation suites, ContractNLI, MAUD, ECtHR, LEDGAR, CaseHOLD, Supreme Court issue areas, unfair terms of service), and 17458 replay prompts answered beforehand by the base model and used as soft targets, so that the fine-tune keeps the base model's behaviour on tasks it was not tuned for.
+- Yes/no questions are drawn near their real rates and many-option questions close to balanced. The order of the options is shuffled for every training example, so the model has to read the definitions.
+- Six rounds. In each round several adapters are trained in parallel on different parts of the data from the same starting point and their weight updates are averaged; the next round starts from the average. Later rounds add examples drawn from training decisions not used before, with more weight on area of law, and keep each area-of-law example with a probability equal to how often two annotators agree on its label. The arms that make up the released model saw 287,032 examples in all.
+- Training took about 20 A100 GPU-hours.
+- Adapters are merged into the bfloat16 weights.
+- Several of the held-out public tasks reported above come from source datasets (ContractNLI, CUAD, MAUD, OPP-115, terms of service) from which other examples were used in training. The tasks were held out and exact texts were removed; the source documents are not guaranteed unseen.
+The adapters and the code are in [jfloydian/lev-lora](https://huggingface.co/jfloydian/lev-lora), and the [technical report](lev-technical-report.pdf) has the details, including the experiments that did not help.
 ## Limitations
+- **Both annotators of the test set are models.** "Agreed" means two independent model annotators gave the same answer. No person reviewed the labels. The public tasks above have human labels.
+- **The test set over-samples rare labels**, and it is thin in the very rarest: it has no citation in a dissent and few reported negative histories. The scores describe how each system handles each kind of case, not how often it will be right on a random citation.
+- **The model was tuned against held-out data more than once.** Two earlier evaluation sets were used to find and fix weaknesses, and a new set was drawn each time for the next measurement. The numbers above are from the last set, scored once.
 - **Canadian decisions in English with numbered paragraphs.** Other jurisdictions, French-language decisions and differently built excerpts are outside what was tested, apart from the public tasks above.
+- **Excerpts hide things.** Lev sees the paragraphs around a citation, not the decision. Positive treatment in particular is often only visible from how the rest of the decision uses the case.
 - **It classifies an excerpt; it is not a citator.** It does not find citations, resolve them to cases, or check that the excerpt is complete. Treat its output as a lead to check against the decision. It is not legal advice.
 - **Images and video** are supported by the d3 runtime it inherits, but Lev was trained and tested on text only.
+- **Hosted baselines change.** Jev 1.13 and the OpenAI Decisions API (in public beta) were called over two days.
 ## Files
+- `model-0000{1,2,3,4}-of-00004.safetensors`, `readout.safetensors`, `decision_config.json`: the model in d3 package format.
+- `d3_*.py`, `modeling_d3.py`, `pipeline_d3.py`: the d3 runtime, unchanged from d3-flash.
 - `questions.py`, `excerpts.py`: the trained questions and the excerpt builders.
 - `lev-technical-report.pdf`: the technical report.
 - The LoRA adapters and the training and evaluation code are in [jfloydian/lev-lora](https://huggingface.co/jfloydian/lev-lora).
 ## License and attribution
+Apache-2.0. Lev is built on d3-flash by the vLLM Semantic Router team, which is built on Qwen3.5-9B; their licence and notice files are included. If you use Lev, please credit Jonathan Kleiman.
 ```bibtex
 @misc{kleiman2026lev,
+  title  = {Lev: a 9B open decision model for citator and case-law classification},
   author = {Kleiman, Jonathan},
   year   = {2026},
   howpublished = {\url{https://huggingface.co/jfloydian/lev}}