jaymz0/2Can-v0.1-0.75B

2Can

Two models have to agree, or it says "I don't know."

2Can checks whether a claim is supported by a source text. It runs two independent encoders, an NLI model and a fact-checking model, and only answers when both agree strongly:

decision	when
SUPPORTED	both models ≥ pass_min (0.90)
UNSUPPORTED	both models ≤ fail_max (0.03)
NOUL	anything else: the models disagree, or neither is confident

Every scored claim also gets p_supported, a single calibrated probability that combines both models. 2Can only classifies and never generates text, so it cannot invent content, although it can still be wrong (see the error rates below). It is small (about 0.75B parameters across the two verify encoders): in our runs on an RTX 5060 Ti 16 GB it took roughly 60–85 ms per claim, batched, depending on source length, and CPU is much slower.

This repo ships no new weights. 2Can is a policy plus a calibration layer over three published models (not fine-tuned or merged), which are downloaded from the Hub on first run:

role	model	license
verify (NLI)	tasksource/ModernBERT-large-nli	Apache-2.0
verify (fact-check)	lytang/MiniCheck-RoBERTa-Large	MIT
choose / score (zero-shot)	MoritzLaurer/ModernBERT-large-zeroshot-v2.0	Apache-2.0

Quick start

pip install -e .
2can-serve                      # binds 127.0.0.1:8766; GPU if available, else CPU

curl -s localhost:8766/v1/verify -H 'Content-Type: application/json' -d '{
  "source": "The Golden Gate Bridge opened to traffic in 1937.",
  "claims": ["The bridge opened in 1937.", "The bridge opened in 1952.", "The Golden Gate Bridge is in Paris."]}'

claim	decision	p_supported
The bridge opened in 1937.	SUPPORTED	0.966
The bridge opened in 1952.	NOUL (uncertain)	0.122
The Golden Gate Bridge is in Paris.	NOUL (uncertain)	0.052

UNSUPPORTED is deliberately strict: both models must put the claim at or below 0.03, which only about 10% of LLM-AggreFact claims reach. To reject claims, threshold p_supported. Treat SUPPORTED as the high-precision tier: about 6% of SUPPORTED claims were still false on the held-out set.

Each result carries decision, noul_reason, p_supported, both raw model probabilities (p_modernbert, p_minicheck), and confidence (the weaker model's margin on a decision).

Other endpoints: POST /v1/choose (zero-shot pick among options, or NOUL when none fits or two are too close), POST /v1/score (0-1 score of text against a criterion, NOUL in the middle band), POST /v1/embed, GET /v1/health, GET /v1/policy.

To bind a non-loopback address, set a bearer token: TWOCAN_TOKEN_FILE=/path/to/token (at least 32 characters). The server refuses to listen publicly without one. Without a token it accepts only loopback Host headers and JSON bodies, so a web page in your browser can't drive it. The rest of the configuration is described at the top of two_can/server.py.

How the numbers were fit

Everything in two_can/policy.json was fit on public human labels, the dev split of LLM-AggreFact, and evaluated on its held-out test split. (The splits share 110 of 2,849 test source documents, which matters little for a 3-parameter fit.) At most 400 rows were sampled per sub-dataset (seed 0), so no single source dominates. LLM-AggreFact is gated and licensed CC BY-ND 4.0, so only the fitted parameters are distributed here, never its rows. scripts/fit_calibration.py reproduces both fits.

Thresholds use a fixed-sequence search that keeps the 95% Clopper-Pearson upper bound on the false-accept and false-reject rates at or below 10% on dev, counting at most 3 claims per source document.

Held-out results on the LLM-AggreFact test sample (4,358 claims, of which 4,347 were scored; the other 11 were too long or not a claim). The numbers come from the GPU path (bf16/fp16). The CPU path runs fp32. Its probabilities typically differ by less than 0.01, and about 1% of decisions that sit near a threshold can flip.

claims decided (not NOUL)	35.5% of 4,358
false accepts among SUPPORTED	68 / 1,114 (6.1%); 95% upper bound 9.5% (document-capped)
false rejects among UNSUPPORTED	36 / 433 (8.3%); 95% upper bound 10.9% (document-capped), just above the 10% target
calibration error (ECE) of p_supported	0.016 (averaging the raw model probabilities gives 0.101)
AUROC of p_supported	0.871

Limitations

• Calibration depends on the text you feed it. p_supported was fit on LLM-AggreFact. Across its sub-datasets the test ECE ranges from 0.03 to 0.27 (ExpertQA and Reveal are worst), and it can be worse again on text unlike LLM-AggreFact. If you have labelled data from your own domain, refit with scripts/fit_calibration.py.
• It abstains a lot, on purpose. About two thirds of LLM-AggreFact claims come back NOUL. The design pairs 2Can with a slower judge (a person or an LLM) for the claims it won't decide.
• Contradictions usually come back NOUL, not UNSUPPORTED (see the quick-start table), including a changed number or a plain falsehood. p_supported still ranks them low.
• Long sources of more than 32 windows of 512 tokens return NOUL (source_too_long).
• English only, as far as the upstream models go.
• Commercial use: 2Can's own code and policy are MIT. The upstream models carry their own licenses (above), and some of them were trained on research datasets with non-commercial terms, so check the upstream model cards before commercial use.

Citation

If you use 2Can, please cite the upstream models and LLM-AggreFact: Tang, Laban & Durrett, MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (2024).

License

MIT © 2026 James Yeung