Instructions to use Encevra/Athena-27B-Preview with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Encevra/Athena-27B-Preview with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Athena-27B-Preview
Evidence-grounded document reasoning from Encevra
Introduction
Athena-27B-Preview is Encevra's research preview for document-grounded reasoning: given a document and a question, it answers from the document, supports the answer with verbatim quotes from the source, and abstains when the evidence is insufficient. It is a LoRA adapter for Qwen/Qwen3.8-27B at revision 37561816e66b5b9153b1b64c10968c18c1e6f4a0. This repository contains only the adapter weights.
This release is Preview v0.3 (selected checkpoint step-1500, adapter sha256 a29d0eac3198c9af6a7466a7fef5811386768d90ed5fb37caf0dee0b2774bf13). It continues QLoRA training from Preview v0.2. The previous public adapters remain available at revisions v0.2 and v0.1.
Highlights
- Continues v0.2 with official ContractNLI train, CUAD v1 train, long-document, and cross-clause data. Only v0.3 was trained on the official ContractNLI train split. v0.2 was not. The 17 ContractNLI hypothesis wordings are shared across train, dev, and test by design; this is not zero-shot on unseen hypotheses.
- The released checkpoint is step 1500, chosen with a pre-registered rule on development data only (0.2-dev abstain and clause bars, then best ContractNLI dev macro-F1). Held-out and official test sets were not used for selection.
- On fair held-out CUAD, long-document, and cross-clause checks, v0.3 gains over v0.2 and the paired 95% confidence intervals all clear zero.
- On official ContractNLI test, paper-style accuracy is 0.885, F1(Contradiction) 0.419, and F1(Entailment) 0.848, on par with the best published fine-tuned result. This is not a claim of state of the art.
- The v0.2 document-label / generic-title limitation was retested: v0.3 scores 25/25 on the generic-title set.
- A hosted public demo is not part of this release.
v0.3 results
Intervals below are percentile bootstrap 95% CIs, seed 27038, 10,000 draws. Paired intervals are v0.3 minus v0.2, aligned by item id. A tie with no discordant pairs is not evidence of equivalence. An interval whose low end is exactly 0 does not clear zero. The primary ContractNLI rule maps ABSTAIN to NotMentioned.
Held-out and Athena-style sets
| Set | v0.3 step-1500 | v0.2 | Paired Δ, 95% CI |
|---|---|---|---|
| CUAD held-out (n=200) | 169/200 (0.845) | 127/200 (0.635) | +0.210 [0.145, 0.275]; 54 discordant; clears zero |
| Long-document held-out (n=40) | 33/40 (0.825) | 18/40 (0.450) | +0.375 [0.225, 0.525]; 15 discordant; clears zero |
| Cross-clause held-out (n=40) | 39/40 (0.975) | 30/40 (0.750) | +0.225 [0.100, 0.350]; 9 discordant; clears zero |
| Holdout-50 | 49/50 | 49/50 | Δ 0; 0 discordant; interval not formed; not evidence of equivalence |
| Renewal-5 | 5/5 | 5/5 | Δ 0; 0 discordant; interval not formed; not evidence of equivalence |
| Generic-title 25 | 25/25 | 25/25 | Δ 0; 0 discordant; interval not formed; not evidence of equivalence |
ContractNLI official test
Paper-style metrics follow Koreeda & Manning 2021 / contract-nli-bert macro_label_micro_doc: accuracy and binary F1(C) / F1(E) averaged over the 17 hypotheses. Because every hypothesis appears in all 123 test documents, this accuracy equals instance-level micro accuracy (1,850/2,091 = 0.8847, reported as 0.885). Athena is a generative model emitting hard labels, not the paper's span classifier, so this is context only, not a leaderboard claim. Do not compare the instance-level macro-F1 (0.847) or evidence F1 (0.793) to the published paper figures.
Only v0.3 was trained on official ContractNLI train. The v0.2 column is shown for completeness and is not a trained-versus-trained comparison.
| Metric | v0.3 | v0.2 (not trained on ContractNLI) | Published context (different models and, for the LLM rows, different protocols) |
|---|---|---|---|
| Paper-style accuracy | 0.885 [0.871, 0.898] | 0.690 [0.670, 0.710] | DeBERTa-xlarge+CUAD 0.892; Span NLI BERT-large 0.875; Gemma-4 26B zero-shot 0.812; Qwen-3.5 27B zero-shot 0.793 |
| Paper-style F1(Contradiction) | 0.419 | 0.295 | DeBERTa-xlarge+CUAD 0.405; Span NLI BERT-large 0.357 |
| Paper-style F1(Entailment) | 0.848 | 0.517 | DeBERTa-xlarge+CUAD 0.859; Span NLI BERT-large 0.834 |
Sources: Koreeda & Manning 2021, arXiv:2110.01799; 2026 AACL zero-shot open LLMs, aclanthology.org/2026.aacl-short.57. On these paper-style metrics, v0.3 is on par with the best published fine-tuned result. It is not state of the art: DeBERTa-xlarge+CUAD remains slightly ahead on accuracy (0.892) and F1(E) (0.859); v0.3 is slightly ahead on F1(C) (0.419 vs 0.405).
Paired v0.3 minus v0.2 micro accuracy is +0.195 [0.176, 0.215] and clears zero, but that gain includes the fact that only v0.3 was trained on this dataset. Official ContractNLI dev (v0.3 only, n=1,037) is accuracy 0.870 [0.850, 0.890] and macro-F1 0.822, consistent with the pre-registered selection of step 1500.
FACTS Grounding (unofficial public-200 sample)
This is not an official Kaggle or FACTS leaderboard score, and it is not a Kaggle rank. It does not beat, match, or sit on the official leaderboard.
The comparison uses a 200-item sample of the public split only (seed 27038; 67 short, 67 medium, 66 long), not the private set and not all public items. Judges are Gemini 2.5 Flash and GPT-5 gpt-5-2025-08-07 with the public v1 implicit_span_level template. The official eligibility / quality filter was not run. Official Kaggle scores use a three-judge ensemble, the v2 prompts, the eligibility filter, and the average of the public and private sets (1,719 examples). See the FACTS Grounding leaderboard.
| Protocol | v0.3 | v0.2 | Paired Δ, 95% CI |
|---|---|---|---|
| Primary (unparsed verdicts counted inaccurate), after one cheap retry of empty GPT-5 verdicts | 85.75% [82.0%, 89.25%] | 81.5% [77.0%, 85.5%] | +4.25 pp [0.0, 8.5]; 57 discordant; does not clear zero (the lower bound is exactly 0.0) |
| First pass only (38 empty GPT-5 verdicts counted inaccurate) | 84.25% [80.5%, 88.0%] | 81.5% [77.0%, 85.5%] | +2.75 pp [−1.5, 7.0]; does not clear zero |
| Skip-unparsed sensitivity, after retry (31 empty GPT-5 verdicts remain) | 93.00% [89.75%, 95.75%] | 87.75% [83.5%, 91.5%] | +5.25 pp [1.75, 9.0]; clears zero. This is a sensitivity, not the primary score. |
Seven of 200 prompts were truncated to the 16,384-token window. After the retry, 31 GPT-5 verdicts were still empty and count as inaccurate in the primary score. Do not line these unofficial percentages up against official leaderboard rows.
Quickstart
The verified example is examples/infer.py. Install with pip install transformers peft unsloth huggingface_hub (Unsloth's 4-bit path needs an NVIDIA GPU), load Qwen/Qwen3.8-27B at the revision pinned in the snippet, and attach this adapter with peft.PeftModel. To load a previous public adapter, pass revision="v0.2" or revision="v0.1" to snapshot_download.
Recommended settings: greedy decoding, thinking off, the passage included in the user message, and at most 512 new tokens for short questions (the unofficial FACTS run used a 1,024-token cap). Athena is trained to reply as ANSWER: <final answer> and QUOTE: <exact words from the passage>, or ANSWER: ABSTAIN when the passage does not support an answer.
Python (Unsloth, 4-bit)
import os
from pathlib import Path
from huggingface_hub import snapshot_download
from peft import PeftModel
from unsloth import FastModel
REVISION = "37561816e66b5b9153b1b64c10968c18c1e6f4a0"
BASE = "Qwen/Qwen3.8-27B"
REPO_ID = "Encevra/Athena-27B-Preview"
STOP = ("<|im_end|>", "<|endoftext|>", "<|im_start|>")
token = os.environ.get("HF_TOKEN")
root = snapshot_download(REPO_ID, allow_patterns=["adapter/*"], token=token)
# Previous public adapters:
# snapshot_download(REPO_ID, revision="v0.2", allow_patterns=["adapter/*"], token=token)
# snapshot_download(REPO_ID, revision="v0.1", allow_patterns=["adapter/*"], token=token)
adapter = Path(root) / "adapter"
model, tokenizer = FastModel.from_pretrained(
BASE,
revision=REVISION,
max_seq_length=4096,
load_in_4bit=True,
full_finetuning=False,
)
model = PeftModel.from_pretrained(model, str(adapter), is_trainable=False)
FastModel.for_inference(model)
passage = (
"The river froze on the first of January, 1887, and remained "
"closed to boats until the twelfth of March."
)
question = "On what date did the river freeze?"
user = f"""Use only the passage below. Do not use outside knowledge.
If the passage does not state the answer, reply on its own line:
ANSWER: ABSTAIN
If it does, end with the answer and at least one quote copied from the passage:
ANSWER: <final answer>
QUOTE: <exact words from the passage>
PASSAGE:
{passage}
QUESTION:
{question}
"""
# Then tokenize, generate greedily with thinking off, and stop on STOP.
Training
v0.3 continues the v0.2 adapter (sha256 f09e207b0419cd738d4b21f1d833c53b48199c8f9f6233bf2952fbe13339cfa2) with QLoRA, LoRA rank 8 and alpha 8, on a private mix of official ContractNLI train, CUAD v1 train contracts, long-document, and cross-clause examples, plus a replay of the 0.2 mix. The training mix is not published and raw contract text is not included in this repository. No Kimi outputs or other closed-API outputs were used as training targets. Selection used development data only, as described above.
Limitations
- ContractNLI is not zero-shot on unseen hypotheses. Training used the official train split, which is the fair protocol for that test set, but the 17 hypothesis wordings appear in train, dev, and test by design. Report v0.3 as trained on official train, not as zero-shot. v0.2 was not trained on ContractNLI, so the test-set gap versus v0.2 is not a trained-versus-trained comparison.
- Paper-style metrics only for published comparisons. Instance-level macro-F1 (0.847) and evidence F1 (0.793) are our own metrics and should not be compared to the 2021 paper or to the 2026 AACL zero-shot figures.
- Not state of the art on ContractNLI. "On par with the best published fine-tuned result" is the intended wording. DeBERTa-xlarge+CUAD remains slightly ahead on paper-style accuracy and F1(E).
- FACTS numbers are unofficial. They are a 200-item public sample, two judges, a v1 template, and no eligibility filter. The primary paired interval versus v0.2 touches zero. Skip-unparsed is a sensitivity only. Do not claim a Kaggle rank or that Athena beats the official leader.
- Generic-title limitation, updated. v0.2 noted that some clause-exception answers were sensitive to generic document titles. That 25-item generic-title set was rerun for v0.3: 25/25. The 5/5 renewal and 49/50 holdout results are exact ties with v0.2 at or near ceiling and are not evidence of equivalence.
- Synthetic and constructed held-out sets. Holdout-50, renewal-5, and generic-title 25 are synthetic or constructed instruments, not a naturally occurring contract corpus. CUAD held-out uses official CUAD test/held-out contracts. Long-document and cross-clause held-out sets are constructed. These should not be read as in-the-wild contract performance.
- Small evaluation sets. Several slices have 25 to 50 items. Intervals on small slices are wide.
- Context window. Evaluation used a 16,384-token window. Seven unofficial FACTS prompts were truncated to fit. Longer official FACTS documents can exceed that window.
- Presentation. The terse
ANSWER/QUOTEformat can read as mechanical. - Not for consequential use. Athena is meant for research on document-grounded question answering, and it works best when a person can read the source and check the quotes. It is not for medical, legal, financial, or other consequential decisions about people; unattended use; harmful or criminal assistance; or merged-weight use (merging was not tested).
Earlier v0.1 and v0.2 evaluation tables, including the v0.1 document-grounding and general-ability charts still in assets/, were measured on those earlier checkpoints. They are not scores for v0.3.
License
The Athena-27B-Preview adapter is licensed under the Apache License, Version 2.0. The full terms are in LICENSE. Attribution and third-party notices are in NOTICE.
The base model, Qwen/Qwen3.8-27B, is also licensed under Apache-2.0. A copy of its license file at the pinned revision is in BASE_MODEL_LICENSE.txt.
ContractNLI. Official document text, span boundaries, and NLI labels used in training come from Koreeda and Manning, “ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts,” Findings of EMNLP 2021. The dataset is distributed by Hitachi America, Ltd. under Creative Commons Attribution 4.0 International (CC BY 4.0) and accompanying terms. CC BY 4.0 requires attribution and does not allow relicensing that source text or those labels as Apache-2.0. Raw ContractNLI documents are not included in this repository.
CUAD v1. Official contract text and clause labels used in training come from Hendrycks, Burns, Chen, and Ball, “CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review,” arXiv:2103.06268, curated by The Atticus Project, Inc., and licensed under CC BY 4.0. Raw CUAD contracts are not included in this repository.
@inproceedings{koreeda-manning-2021-contractnli,
title = "ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts",
author = "Koreeda, Yuta and Manning, Christopher D.",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
year = "2021",
publisher = "Association for Computational Linguistics"
}
@article{hendrycks2021cuad,
title={CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review},
author={Dan Hendrycks and Collin Burns and Anya Chen and Spencer Ball},
journal={arXiv preprint arXiv:2103.06268},
year={2021}
}
Citation
@misc{encevra2026athena27bpreview,
title = {Athena-27B-Preview: Evidence-Grounded Document Reasoning},
author = {{Encevra}},
year = {2026},
url = {https://huggingface.co/Encevra/Athena-27B-Preview}
}
- Downloads last month
- -
Model tree for Encevra/Athena-27B-Preview
Base model
Qwen/Qwen3.8-27B

