| --- |
| license: apache-2.0 |
| tags: |
| - coreml |
| - legal |
| - citation-verification |
| - hallucination-detection |
| - apple-neural-engine |
| - hand-written-weights |
| datasets: |
| - ai-law-society-lab/Legal_Phantom_Citation |
| - harvard-lil/cold-cases |
| --- |
| |
| # CiteNet: a legal-citation checker whose weights were written by hand |
|
|
| A 397,250-parameter network (13,871 nonzero) that flags legal citations that cannot exist: an invented reporter series (`F.5th`, `S.W.4th`), a court the reporter never prints (`88 F. Supp. 3d 412 (5th Cir. 2015)`), a state the regional reporter doesn't cover, or a year that doesn't fit the volume (`912 F.3d 455 (9th Cir. 1998)`; vol 912 is from 2018). |
|
|
| **No training.** Claude (Anthropic) wrote `citenet/spec.py`: the reporters and their real series, which courts each reporter prints, the West regional map, and approximate volume→year timelines. It also wrote `citenet/net.py`, which compiles each entry into specific weights: |
|
|
| - a conv filter that fires only when the characters spell `<digit> F.3d<space>` |
| - digit decoders with weight k·10^i on digit k |
| - a shared ReLU "volume thermometer" that every volume→year curve reads from |
| - AND gates like `relu(8·[federal appellate reporter] + [state court] − 8)` |
|
|
| Every nonzero weight has a name, and there was no gradient step. |
|
|
| ## Results (LePhantomCite, held-out eval split, citation level) |
|
|
| [ai-law-society-lab/Legal_Phantom_Citation](https://huggingface.co/datasets/ai-law-society-lab/Legal_Phantom_Citation) (Liu, Stammbach & Henderson 2026, CC BY 4.0). The network was designed using only the `aux_train` split. `eval` was scored once with frozen weights. |
|
|
| | | phantom citations caught | real citations flagged | |
| |---|---|---| |
| | v1.0, first and only blind run | 31 / 31 | 3 / 983 | |
| | v1.5 (this release; later fixes were not blind) | 31 / 31 | 1 / 983 | |
|
|
| - **The remaining flag** is `652 A.2d 634 (D.C. Cir. 1995)`. A.2d never prints D.C. Circuit opinions, so it is likely an error in the source brief. |
| - **Out of scope:** other hallucination types (case-name mismatch, wrong pincite, misquote) need the case text. They are flagged 2 / 276 times. |
| - **Hand-written vs trained:** trained from scratch, the same architecture (v1.1) ties on this benchmark, with 0 / 983 false flags. It caught 1 of 5 planted fakes of kinds the training data never shows, against 5 / 5 for the hand-written net. That 5-fake set is tiny and I wrote it, so treat it as illustrative. A naive fine-tune starting from the hand-written weights made it worse. See `results/`. |
|
|
| **Limits.** A string-only network can tell that a citation *could* exist, not that it does. LePhantomCite's non-existent citations are structurally impossible, so it catches all of them. Real LLM fakes, such as the six citations ChatGPT invented in *Mata v. Avianca* (e.g. `925 F.3d 1339 (11th Cir. 2019)`), copy valid formats and pass. Use this as a fast first filter in front of a case-law lookup. |
|
|
| **Real case law.** On 3.30 GB of US court opinions (291,339 decisions, COLD Cases shard 0), 0.134% of citations are flagged. Most flags look like real errors in the opinions, e.g. `387 U.S. 1 (1887)` for In re Gault (1967). They are unverified, and some are the model's own mistakes. |
|
|
| ## Speed (Core ML, Apple M5 Pro, native Swift) |
|
|
| | | | |
| |---|---| |
| | 1 citation (`CiteNet_b1`, CPU) | ~50 µs | |
| | 32 citations (`CiteNet_b32`, Neural Engine) | 170 µs | |
| | bulk, Neural Engine (`CiteNet_lean_b512`) | ~430k citations/s | |
| | bulk, GPU (`CiteNet_lean_b8192`) | ~1.38M citations/s | |
| | Neural Engine + GPU together | **~1.82M citations/s** | |
| | raw text, end to end: native scanner + ANE + GPU | **3.30 GB of opinions in 2.3 s** (1.44 GB/s, 1.68M citations/s) | |
|
|
| Every build is fp16, and the labels match the fp32 PyTorch model on all 4,383 LePhantomCite citations. Fixed batch shapes matter: with `EnumeratedShapes`, Core ML ran everything on the CPU. |
|
|
| ## Files |
|
|
| - `models/`: `CiteNet_b{1,32,512}.mlpackage` output `p_fake`, the 7 `evidence` values and the named `concepts`. `CiteNet_lean_b{512,8192}.mlpackage` output only `p_fake` and `evidence`, for bulk work. `meta.json` names every neuron; `weights.npz` holds the raw weights. |
| - `citenet/`: the spec, the weight compiler and the tokenizer (`text.py`: citation extraction and normalization to 64 char codes). |
| - `bench/`: |
| - `CiteScan.swift`: a byte-level Swift port of the tokenizer, byte-identical to Python (`scripts/scan_parity.py`). |
| - `stream/main.swift`: the multithreaded raw-text pipeline. |
| - `firehose.swift`, `bench.swift`: benchmarks. |
| - `demo/`: a local web demo (`python demo/server.py`, localhost:8025). |
| - a live checker |
| - `/wall`: a scrolling text wall that highlights citations as they arrive, with a planted-fakes mode that reports planted, caught and missed counts |
| - `/live`: the full-speed scan |
| - `/inspect`: the neuron and filter view |
| - `results/`: the v1.0 blind eval log, v1.5 eval logs, the training comparison and benchmark logs. |
|
|
| ## Use |
|
|
| ```python |
| import numpy as np, coremltools as ct |
| from citenet import text |
| |
| m = ct.models.MLModel("models/CiteNet_b32.mlpackage") |
| cites = [c for _, _, c in text.extract("See Henley v. Northwind, 412 F.5th 1187 (2d Cir. 2023); " |
| "Bell Atl. Corp. v. Twombly, 550 U.S. 544, 570 (2007).")] |
| x = np.zeros((32, 1, 1, 64), np.float16) |
| x[:len(cites)] = text.encode(cites) |
| p = m.predict({"codes": x})["p_fake"][:len(cites), 0] |
| print(list(zip(cites, p.round(3)))) # F.5th -> ~0.99, Twombly -> ~0.02 |
| ``` |
|
|
| To rebuild every model from the spec: `python scripts/convert.py` (torch + coremltools). The demo server expects the models in `build/`. |
|
|
| Data: LePhantomCite (CC BY 4.0) and COLD Cases (CC0) are not included; download them from the Hub. |
|
|
