southern-lev
_.---------._
|_____________|
_____|_____________|_____
'-------------------------'
/ _ _ \
| (o) (o) |
| __ |
\ .~~~~~~~~. / "Why, I do declare:
\ `------' / that precedent has been
'.__________.' *distinguished*."
___/ >()< \___
/ \ / \ / \
/ | \_/ () \_/ | \
/ | () | \
(_) | () | (_)_
|________________| |
| || | |
| || | |
| || | |
|______||______| |
(_______)(_______) _|_
A citation-treatment classifier for U.S. Supreme Court opinions. The name is a nod to Lev.
Given excerpts of a Supreme Court opinion around one of its citations to an earlier Supreme Court decision, the model predicts how the opinion treats that decision. There are ten labels:
| Label | Meaning (short) |
|---|---|
follows |
applies the cited decision as controlling |
supports |
cites it for a proposition, without applying it as controlling |
explains |
describes or discusses it |
background |
mentions it in passing |
history |
it is the same case's earlier stage |
distinguishes |
says how it differs, as a reason it does not decide this case (or decides less of it) |
questions |
doubts or criticises it |
limits |
narrows it |
overrules |
overrules it |
unknown |
the excerpts do not say |
The four negative treatments are distinguishes, questions, limits and overrules.
This is a student model: its labels come from another model, not from lawyers. It
was fine-tuned from answerdotai/ModernBERT-large to reproduce labels that Claude
Sonnet 5.5 gave through the Anthropic Batch API. Training was one run: two epochs, batch
8, learning rate 2e-5, inputs up to 1,024 tokens, bf16 autocast on one NVIDIA L4. The
checkpoint kept is the one with the best validation macro F1, which was the final step.
Training data
There are 68,636 citing-to-cited pairs from CourtListener: 54,851 for training, 6,952 for validation and 6,833 for test. Every pair is a U.S. Supreme Court opinion citing an earlier Supreme Court decision. The pairs are split by a hash of the citing opinion, so no opinion's text is on both sides. The labels were collected in rounds, and the mix is not the population's:
| Labels | Prompt, effort | How the pairs were chosen |
|---|---|---|
| 25,500 | prompt v3, medium | at random |
| 19,552 | prompt v5, medium | mostly pairs a cheaper model (Claude Haiku 5.5) had flagged as possibly negative |
| 10,183 | prompt v5, xhigh | the pairs from the row above that Sonnet answered below 0.7 confidence, re-read; this label replaces the medium one |
| 4,606 + 3,094 | prompt v5, medium / xhigh | the unread pairs an earlier student ranked most likely negative; those below 0.7 confidence re-read at xhigh |
| 9,500 | prompt v5, medium | the unread pairs an earlier student ranked most likely questions, limits or overrules (few turned out to be) |
Those rounds give 72,435 labels. Two groups are excluded, which leaves the 68,636:
- the 500 hand-gold pairs below
- 3,299 pairs from slip opinions whose stored text changed after their requests were built
Two prompt versions are mixed in. v5 widens distinguishes to include soft
distinctions and cf. citations followed by the difference; the 25,500 random v3 labels
predate that.
Training-split label counts:
| Label | Count |
|---|---|
| supports | 20,057 |
| distinguishes | 10,766 |
| follows | 7,079 |
| background | 6,742 |
| explains | 6,167 |
| questions | 2,135 |
| history | 843 |
| limits | 721 |
| overrules | 205 |
| unknown | 136 |
Input format
The model reads a shortened version of the request the teacher was sent:
- the header lines: citing opinion, cited decision, whether the citing court can overrule the cited one, the citation signals, and, where present, CourtListener's parentheticals
- each marked occurrence of the citation (
[[n: …]]), cut to about 2,400 characters around the mark (two thirds before it, one third after) - up to 2,000 characters of the cited decision's text at the pinned page
The median input is about 590 tokens and the 90th percentile about 850, so few are cut by the 1,024-token limit. For example (shortened):
Citing opinion: Mohawk Industries, Inc. v. Carpenter, 558 U.S. 100 (2009), concurrence.
Cited decision: Ashcroft v. Iqbal, 556 U.S. 662 (2009).
Court: the citing court can overrule the cited decision (the same court).
Signals: See; See, e. g.,
Occurrences:
[1] … In taking this path, the Court needlessly perpetuates a judicial policy that we for many years have criticized and struggled to limit. See, e. g., [[7: Ashcroft v. Iqbal, 556 U. S. 662, 671-675 (2009)]]; Will v. Hallock, 546 U. S. 345, 349 (2006); …
[2] … As we recognized last Term, however, our attempts to contain the Cohen doctrine have not all been successful or persuasive. See [[26: Ashcroft, supra, at 672]] (“As a general matter, the collateral-order doctrine may have expanded …”). …
Cited passage(pinned page 671):
II
We first address whether the Court of Appeals had subject-matter jurisdiction …
Text built any other way (whole paragraphs, no marks, no header) is out of distribution. This function builds it from the full request message:
import re
MARK = re.compile(r"\[\[\d+: ")
AROUND = 2400
def student_text(message):
head, _, rest = message.partition("\nOccurrences:\n")
occurrences = rest.split("\nCited passage")[0]
parts = re.split(r"(?m)^\[(\d+)\] ", occurrences)
kept = []
for i in range(1, len(parts) - 1, 2):
body = parts[i + 1]
m = MARK.search(body)
if m:
end = body.find("]]", m.start())
end = len(body) if end < 0 else end + 2
span = AROUND - (end - m.start())
left = max(0, m.start() - max(span * 2 // 3, 200))
right = min(len(body), end + max(span // 3, 120))
body = ("…" if left else "") + body[left:right] + ("…" if right < len(body) else "")
kept.append(f"[{parts[i]}] {body.strip()}")
cited = rest.partition("\nCited passage")[2].strip()[:2000]
return (head.strip() + "\nOccurrences:\n" + "\n".join(kept)
+ ("\nCited passage" + cited if cited else "")).strip()
Evaluation
All the numbers below are from a single training run, with no confidence intervals. On 494 pairs, a 95% interval on an accuracy near 0.65 is about ±4 points.
1. Hand gold set (494 pairs, none of them seen in training)
The gold set is 506 Supreme Court pairs. Each label was decided by a Claude Opus 5.5 agent reading the full citing opinion, not by a lawyer. Four pairs have no label and eight could not be built as requests, so 494 are scored. The gold set's dev half was used while developing the prompts, so for the Claude rows the held-out half (248 pairs) is the cleaner comparison. This model was not tuned on either half.
| This model | Sonnet 5.5 medium (teacher) | Sonnet 5.5 xhigh | |
|---|---|---|---|
| Exact label, all 494 | 0.678 | 0.694 | 0.711 |
| Exact label, held-out half | 0.657 | — | 0.698 |
| Negatives found (of 86) | 46, at 92% precision (46 of 50 flagged) | 46 | 52, at 93% precision |
| AUROC of P(negative), negatives against the rest | 0.892 | — | — |
On the held-out half alone, the model finds 23 of 41 negatives, with 25 flagged.
The per-class scores of this model on gold are below. The rows above were scored in fp32; this table comes from the training script's bf16 scoring, which puts exact accuracy at 0.680 instead of 0.678.
| Class | n | Precision | Recall | F1 |
|---|---|---|---|---|
| supports | 176 | 0.66 | 0.83 | 0.74 |
| follows | 124 | 0.70 | 0.69 | 0.70 |
| background | 88 | 0.90 | 0.52 | 0.66 |
| distinguishes | 58 | 0.82 | 0.53 | 0.65 |
| explains | 12 | 0.26 | 0.83 | 0.39 |
| questions | 11 | 0.80 | 0.36 | 0.50 |
| limits | 9 | 0.50 | 0.11 | 0.18 |
| overrules | 8 | 0.80 | 0.50 | 0.62 |
| history | 8 | 0.80 | 1.00 | 0.89 |
Macro F1 over these nine classes is 0.59. The rare classes have 8 to 12 gold examples each,
so their scores are very noisy. The commonest errors are background read as supports
(36) and confusion between follows and supports (46 either way).
2. Independent third-party labels: Demir & Canbaz (435 pairs, out of domain)
mmikaildemir/negative_treatment (MIT) is Paul Hellyer's reading of federal circuit and district citations that a commercial citator (Shepard's, KeyCite or BCite) flagged as negative.
- 195 of them are negative by his codes.
- 198 controls were drawn from other published federal opinions citing the same cases. They were never flagged, but no one read them, so they are only presumed neutral.
These are not Supreme Court opinions, unlike the training data, and Hellyer's definitions are broader than this taxonomy's. No Claude model labelled this set, so it is the check least shared with the teacher.
| This model | Sonnet 5.5 medium | Sonnet 5.5 xhigh | Opus 5.5 xhigh | |
|---|---|---|---|---|
| Hellyer's negatives called negative (195) | 59% (116) | 61% | 67% | 71% |
His "distinguished" called distinguishes or limits (131) |
72% (94) | 71% | 74% | 78% |
| Controls called negative (198) | 7% (13) | 8% | 8% | 11% |
| AUROC of P(negative), negatives against controls | 0.863 | — | — | — |
3. Agreement with the teacher on the test split (6,833 pairs)
This measures agreement with Sonnet's labels, not correctness, on the same skewed mix as training. Accuracy is 0.772 and macro F1 is 0.663.
| Class | n | F1 |
|---|---|---|
| distinguishes | 1,419 | 0.84 |
| supports | 2,478 | 0.83 |
| history | 103 | 0.83 |
| background | 888 | 0.77 |
| follows | 852 | 0.70 |
| unknown | 16 | 0.67 |
| explains | 718 | 0.64 |
| questions | 237 | 0.61 |
| limits | 106 | 0.47 |
| overrules | 16 | 0.28 |
Compared with smaller students
The same labels, trained into smaller models or with a shorter input (about 700 characters around each mark, no cited passage, 512 tokens):
| Gold exact | Gold held-out | Gold negatives | D&C negatives | D&C AUROC | |
|---|---|---|---|---|---|
| ModernBERT-base, short input | 0.630 | 0.601 | 39 of 86 | 50% | 0.828 |
| ModernBERT-large, short input | 0.652 | 0.617 | 46 of 86 | 53% | 0.824 |
| ModernBERT-base, long input | 0.664 | 0.637 | 37 of 86 | 52% | 0.856 |
| This model (large, long input) | 0.678 | 0.657 | 46 of 86 | 59% | 0.863 |
Full numbers for this model are in eval-report.json.
What it is good for, and what it is not
- Good for ranking. The model can rank citations by how likely they are to be negative (sum the four negative probabilities), for example to choose which ones a person or a stronger model should read. When it flags a negative it is usually right (92% on gold), but it misses about 45% of them.
- Not an authoritative label. It should not be trusted as the label for any one
citation, especially for
limits(1 of 9 found on gold),questionsandoverrules. It is about 1.5 points below the model it learned from on gold, and that model is itself well short of a careful reading of the whole opinion. - Supreme Court only. It was trained only on Supreme Court opinions citing Supreme Court decisions. On lower-court citations (the D&C set) it still separates negatives from controls (AUROC 0.86), but every model compared there misses more of Hellyer's negatives than it does on gold.
- Its own taxonomy. The ten labels are close to, but not the same as, a commercial citator's red and yellow flags.
- The gold set shares a model family with the teacher. Gold was labelled by Claude Opus agents and the training labels by Claude Sonnet, so agreement on gold may overstate accuracy against a lawyer's reading. The D&C set has no such overlap.
Use
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
repo = "ferguscooney/southern-lev"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
with torch.no_grad():
probs = torch.softmax(model(**tok(text, truncation=True, max_length=1024, return_tensors="pt")).logits, -1)[0]
label = model.config.id2label[int(probs.argmax())]
p_negative = float(sum(probs[model.config.label2id[n]] for n in ("distinguishes", "questions", "limits", "overrules")))
text is the output of student_text above.
Provenance
- Opinion text: CourtListener bulk data.
- Labels: Claude Sonnet 5.5 (Anthropic) through the Batch API, at medium and xhigh effort.
- Gold: Claude Opus 5.5 agents reading the full opinions.
- Third-party check: Demir & Canbaz (MIT).
- Base model:
answerdotai/ModernBERT-large(Apache-2.0).
- Downloads last month
- 19
Model tree for ferguscooney/southern-lev
Base model
answerdotai/ModernBERT-large