LawGranite / README.md
xanderwick's picture
Point to the 4-bit MLX version
6728cdd verified
|
Raw History Blame Contribute Delete
18.7 kB
---
base_model: ibm-granite/granite-4.2-8b
base_model_relation: finetune
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- legal
- granite
- dpo
---
# LawGranite
*Version 1.0*
A fine-tune of [IBM Granite 4.2 8B](https://huggingface.co/ibm-granite/granite-4.2-8b) for drafting
US litigation prose, such as briefs and memoranda, in the register of practicing attorneys.
> **⚠️ Do not trust its citations.** In my measurements it often cites cases that cannot be
> found, and citations that do resolve to a real reporter page frequently point to a *different
> case* than the one it names, far more often than in attorney-written briefs. A citation that
> resolves is not proof it is right: the page may belong to another case. Every citation must be
> checked against the source before any use. This is a drafting aid, not legal research, and it
> gives no legal advice.
## Designed to be paired, not used alone
LawGranite does one job: write persuasive litigation prose that reads like an attorney wrote it.
It was designed with four goals in mind:
- **Persuasive drafting:** it is trained on real briefs to argue in the register courts expect.
- **Lower cost:** drafting is the longest, most token-heavy part of the work. Handing it to a small
local model saves token usage on the more capable (and more expensive) models in the system.
- **Client confidentiality:** it runs entirely on your own hardware, so the facts and documents
you give it for drafting never leave your machine. (A cloud model it is paired with still sees
whatever you send that model, so decide what goes to it.)
- **No watermarking:** LawGranite does not watermark its output. I built it in part out of
concern about the growth of hidden text watermarking, such as Google's SynthID, and the lack of
transparency around it. These schemes embed a statistical signal in the words a model chooses.
The writer cannot see it, and in practice only the vendor that controls the detector can check
for it or say what a detection means. For attorney work product, a hidden, vendor-readable mark
in a filing is a confidentiality and transparency problem in its own right. LawGranite's
decoding is ordinary temperature and top-p sampling, set out in full under Usage, with no
watermarking step. No watermark does not mean no disclosure: see **Disclosure** under
"Intended use and limits".
LawGranite is **not** designed to find, choose, or verify legal authority, and it should not be
deployed as a standalone legal assistant. Use it in one of two ways:
- **With LawGoose**, my companion program, which is built to address weaknesses present in many
local models. LawGoose gives LawGranite only the drafting role. A more capable model does the
research, using LawGoose's CourtListener search and citation-verification tools, and LawGoose
checks every draft before an attorney sees it:
- **Authority list enforcement:** any case citation not on the authorities the attorney supplied,
including its short and *Id.* cites, is replaced with `[AUTHORITY NEEDED]`. An optional strict
setting does the same when no authorities are supplied.
- **Wrong-case check:** every case citation is resolved through CourtListener, and the case at
that reporter page is compared with the case the draft names. A mismatch is flagged as a
wrong-case citation; a citation whose case name cannot be checked is marked inline, so none
passes silently.
- **Fact check:** names, dates, amounts, durations and statutes the drafter was never given are
flagged inline for the attorney (flagged, not removed).
- **Confidentiality:** LawGoose is designed to keep client facts on the firm's own hardware or
private network from start to finish, including running LawGranite on a private rented GPU for
firms without Apple Silicon.
- **Lower zero-data-retention spend:** LawGoose is intended to reduce the spend on
zero-data-retention AI services, by handing the longest, most token-heavy work (drafting) to
LawGranite running locally.
- **Under a more capable general model**, which does the research and planning and checks every
citation and factual claim against the source, and calls LawGranite only to turn the result into
attorney-style prose.
Either way, the model's text is a draft for an attorney to verify, not a finished filing.
Unless a figure says otherwise, every number in this card measures LawGranite **by itself**, with
no checks applied after drafting.
## Enterprise and practice ready
- **Signed weights:** every file in the release is listed in a cryptographically signed SHA-256
manifest, so your security team can verify that the weights are exactly the ones I published
and have not been altered (see Signed release, below).
- **Runs on your own infrastructure:** client data never has to leave your network. LawGranite runs
on Apple Silicon, on NVIDIA GPUs, or on a private cloud instance.
- **Deployment help:** I am happy to help set up LawGranite and LawGoose for an enterprise
deployment.
## Signed release
The weights and every other file in this release are cryptographically signed. `SHA256SUMS` lists
the SHA-256 hash of each file (weights, config, tokenizer, chat template). `SHA256SUMS.p7s` is a
signature over that list, signed with a certificate tied to the Federal Bridge. If any file is
altered, its hash no longer matches the signed list.
To verify a download:
```bash
cd ~/.omlx/models/LawGranite
# 1. the signature is valid and was made by my certificate
# (identrust-igc-chain.pem, included in this repository: IdenTrust's IGC root and intermediate certificates)
openssl cms -verify -binary -inform DER -in SHA256SUMS.p7s -content SHA256SUMS \
-CAfile identrust-igc-chain.pem -purpose any -out /dev/null
# 2. every file matches the signed list
shasum -a 256 -c SHA256SUMS
```
The model card (`README.md`) is not covered by the signature, so it can be updated without
re-signing the weights.
## What you need to run it
This model is designed to be more accessible than many others, but it does require specialized
hardware. Any of the following can run it:
| machine | memory |
|---|---|
| Apple Silicon Mac | **64 GB or more** unified memory (96–128 GB is comfortable) |
| Workstation or server with one NVIDIA GPU | **48 GB or more** of GPU memory (e.g. RTX 6000 Ada, L40S, A6000, A100 80 GB, H100) |
| Two NVIDIA GPUs | 24 GB each or more (e.g. 2× RTX 4090) |
| Cloud | Any instance with one of the GPUs above |
Also needed:
- **Disk:** about 36 GB free for the download.
- **System memory (RAM):** at least 48 GB on GPU machines, because the model passes through RAM while loading.
- **Software:** on a Mac, oMLX (see Usage). Elsewhere, Python with `transformers` and PyTorch (I
tested `transformers` 4.57.6), or vLLM for serving.
Running on a CPU alone, without a GPU or Apple chip, works if the machine has 64 GB or more of RAM,
but it is too slow for practical drafting. The weights are stored at full precision (float32) on
purpose: the last training step made very small changes that lower-precision formats round away.
This full-precision release is the reference model.
**Smaller hardware:** [LawGranite 4-bit (MLX)](https://huggingface.co/xanderwick/LawGranite-4bit-MLX)
is a 4.6 GB version for Apple Silicon Macs with 16 GB or more of memory. It matches this model on
reading as human-written, writing style and instruction following, but it relies on a supplied list
of authorities somewhat less: 79% of its case citations come from the list, against 88% here. Its
card gives the full comparison.
## Usage
The recommended way to run LawGranite on a Mac is [oMLX](https://github.com/jundot/omlx), a local
server for Apple Silicon. It serves the model on your own machine through an OpenAI-compatible
API, so other tools can connect to it, and no text leaves the computer.
**1. Install oMLX and the Hugging Face CLI**
These commands use [Homebrew](https://brew.sh), the standard package manager for macOS. If
`brew --version` in Terminal says "command not found", install Homebrew first:
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```
When it finishes, run the two "Next steps" commands it prints, which add `brew` to your PATH. Then:
```bash
brew install jundot/omlx/omlx
brew install hf # the Hugging Face CLI
```
**2. Download the model into oMLX's model folder** (about 35 GB)
```bash
hf auth login # access requires approval; log in with an approved account
hf download xanderwick/LawGranite --local-dir ~/.omlx/models/LawGranite
```
oMLX finds every model folder under `~/.omlx/models/`. The folder name, `LawGranite`, is the
model name you use in requests.
**3. Set the model's options in oMLX, then start the server**
In oMLX's settings for `LawGranite`, set:
- **Thinking: off** (`enable_thinking: false`). Granite's template turns reasoning on by default,
and this model was never trained with reasoning traces.
- **Temperature 0.7, top-p 0.95.** LawGranite's evaluations below used these values. Greedy decoding
(temperature 0) reads far more machine-like on the detector.
- **Max tokens 3,072 or more**, so full drafts are not cut off.
- **No quantization in oMLX.** Keep these weights at full precision (float32); quantizing them on
load loses part of the final training. For a smaller model, use
[LawGranite 4-bit](https://huggingface.co/xanderwick/LawGranite-4bit-MLX), which was quantized
with that loss measured.
- **Free memory:** LawGranite needs about 35 GB of free memory. If another large model is pinned
or loaded in oMLX, unpin or unload it first, or oMLX will refuse to load LawGranite.
```bash
omlx serve # serves on http://127.0.0.1:8000
```
**4. Send a drafting request**
```bash
curl http://127.0.0.1:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "LawGranite",
"messages": [{"role": "user", "content": "Draft a short argument that the complaint fails to state a claim for breach of fiduciary duty under Colorado law."}],
"max_tokens": 3072
}'
```
The evaluations also used a repetition penalty of 1.1. If your client lets you set it, use the same value.
**Not on a Mac?** The weights are standard Hugging Face safetensors and load with `transformers`
(`dtype="auto"`) or vLLM (`--dtype float32 --default-chat-template-kwargs '{"enable_thinking": false}'`)
on the NVIDIA hardware listed above.
## Training
| stage | data | method |
|---|---|---|
| continued pretraining | US court opinions and briefs from [CourtListener](https://www.courtlistener.com/) / RECAP (opinions:briefs 70:30) | LoRA, merged |
| SFT | ~58M tokens: drafting tasks built from held-out briefs (outline→draft, facts→statement, body→summary), instruction and constraint following, multi-turn | LoRA r32 on all 40 blocks, merged |
| DPO | Two runs from the same starting point, each on 1,084 on-policy pairs (detector pairs preferring samples that an AI-text detector scored as human-written, plus constraint pairs where the chosen draft obeys a length or format cap) and grounded-drafting pairs that prefer drafts citing only the supplied authorities (418 and 748 pairs) | LoRA r16 each, β 0.1, 1 epoch; the two updates blended 35:65 and merged in fp32 (this repo) |
All training ran on public court filings. **Those filings contain real names of
parties, attorneys, and judges**, and the model can reproduce them.
## How it learned to sound like an attorney, not an AI
Most AI models write in a recognizable "AI voice": predictable phrasing, stock transitions, an even,
generic tone. To train that out of LawGranite, I built a tool that spots AI-written legal text, and
used it to coach the model.
**The AI-writing detector.** It was taught on 56,608 legal documents, half written by attorneys
and half by AI systems. Given a passage, it estimates how likely it is that AI wrote it.
- **Accurate:** tested on text from AI systems it had never seen, it told AI from human writing
correctly almost every time (a score of 0.9987, where 1.0 is perfect and 0.5 is guessing).
- **Few false alarms:** I checked it against 1,200 briefs filed before 2022, written before modern
AI existed. It flagged none of their full-length passages as AI-written. It does not judge
passages under 120 words, because there is too little text to tell.
**How it shaped the model.** The detector was used twice:
1. **Choosing what the model learned from.** 55,139 recent briefs were screened, and passages that
looked AI-written were left out of the training examples, so the model learned from human
writing only.
2. **Coaching on its own drafts.** The model wrote several drafts for each request. The detector
scored them, and the model was trained to favor the drafts that read as human-written. Repeated
over many requests, this steers it away from the habits that make text sound like AI.
**Result.** Before this coaching, 85.4% of the model's drafts read as human-written to the detector.
After it, **99.5%** do (Evaluation, below).
**Limits.**
- The model was graded by the same detector that coached it, so the 99.5% shows it satisfies
*this* detector. Other detectors, such as Pangram, and experienced readers may judge differently.
- The detector learned mostly from one kind of drafting task and one family of AI systems, so it is
most reliable on that kind of text.
- See **Disclosure** under "Intended use and limits": this is about the quality of the writing,
not about hiding AI involvement where disclosure is required.
**Technical details** (for readers who want them)
- **Detector:** a ModernBERT sequence classifier, fine-tuned on 56,608 documents (28,304 human,
28,304 AI). It scores text in windows of up to 510 tokens and outputs P(AI).
- **Generalization:** zero-shot AUROC 0.9987 mean (0.9961 minimum) across three splits that held
out entire AI model families. With citations masked, AUROC is 0.9991, so it is not relying on
citation formatting.
- **Operating point:** a window reads as human-written when P(AI) < 0.05. Windows under 120 words
are abstained on. Against 1,200 pre-2022 briefs, the false-positive rate on full-length windows
was 0 of 1,200; nearly all false positives occurred on short windows, which is why the floor exists.
- **Data screening:** 55,139 post-2022 briefs were screened at the excerpt level before building the
supervised fine-tuning (SFT) data.
- **Preference training:** Direct Preference Optimization (DPO) on on-policy pairs, with the chosen
draft scoring low P(AI) and the rejected draft high, mixed with constraint pairs (1,084 pairs) and
grounded-drafting pairs. Two LoRA runs (rank 16, β 0.1, 1 epoch) were blended 35:65 and merged
in float32.
- **Measurement:** 200 held-out prompts, temperature 0.7, top-p 0.95, repetition penalty 1.1. The
share of drafts below 0.05 rose from 85.4% (SFT model) to 99.5% (released model).
**What the detector builds on.** The detector stands on published research and open resources:
- **ModernBERT** (Answer.AI and LightOn, 2024; `answerdotai/ModernBERT-base`): the open encoder
model the detector is fine-tuned from, chosen for its long context, since legal filings run long.
- **RAID** (Dugan et al., ACL 2024), **M4** (Wang et al., EACL 2024) and **HC3** (Guo et al.,
2023): public AI-text detection benchmarks used during development. RAID's practice of testing
on AI systems held out from training is the basis for the "never seen in training" result above.
- **CourtListener and RECAP** (Free Law Project): the source of the attorney-written briefs.
These public benchmarks contain little or no legal writing, so the final detector was trained on
matched pairs instead: each AI-written document was generated from the outline of a real brief
and paired with that brief. The two sides share topic, structure and length, so the detector has
to separate them on *how* they are written, not *what* they are about.
## Evaluation
LawGranite drafts text that reads far more like an attorney wrote it than the base IBM Granite 4.2 8B
does, and it follows drafting instructions somewhat better. Both models answered the same 200 test
requests, none of which LawGranite saw during training, at the same settings (those under Usage).
| | Granite 4.2 8B | **LawGranite** |
|---|---|---|
| Drafts that read as human-written on the project's AI-text detector | 0.5% (1 of 190) | **99.5%** (190 of 191) |
| Distance from real attorney writing style (lower is closer) | 0.84 | **0.39** |
| Instruction following: 81 tests of word limits, edits, Q&A, system prompts, multi-turn | 61.7% | **66.7%** |
## Intended use and limits
- **Intended:** the drafting component of a larger system (LawGoose, or a more capable model
that handles research and citation checking), producing first drafts for attorneys who will
verify every authority, rewrite, and take responsibility for the filing.
- **Supplied authorities:** even when the prompt gives a list of authorities, the model by itself
cites cases that are not on it in about 43% of drafts. Keep the list enforced by the surrounding
system, as LawGoose does: on 116 grounded drafting tasks, LawGoose's enforcement (strict
setting) cut drafts citing an unlisted case from 56 to 18, without removing any supplied
authority. A citation whose case name LawGoose cannot identify is left in place and flagged
for the attorney instead.
- **Not intended:** legal advice, legal research, unsupervised filing, or use by
non-lawyers as a substitute for counsel.
- **Disclosure:** part of this model's preference training rewarded output that an AI-text
detector scores as human-written. That training is why it reads as human 99.5% of the time on the
project's detector. Do not use it to conceal AI involvement where a court, rule, or client
requires disclosure. Many courts now require it.
- **Third-party AI detectors:** results against independent AI-text detectors, including Pangram,
are mixed. The 99.5% "reads as human-written" figure comes from this project's own detector,
which the model was trained against; other detectors may flag some of its output as AI-written.
- **Coverage:** the data skews toward US federal courts. State-specific practice, local
rules, and recent changes in law are not reliably represented.
- **Long drafts:** when output was capped at 1,536 tokens (roughly 1,100 words), 70% of full
drafts written from an outline were cut off. Allow 3,072 or more output tokens for full drafts.