ClauseQwen - Qwen3 Legal Contract Clause Analysis

QLoRA fine-tuning of Qwen3-1.7B for Legal commercial contract-clause analysis using CUAD and LegalBench / ContractNLI.

Legal Commercial contracts contain dense legal language that can make clause review and structured extraction slow when performed manually. ClauseQwen is designed to reduce that effort by identifying clause types, detecting whether contractual concepts are present, extracting relevant language, and returning natural language.

Why QLoRa

QLoRA adds 4-bit base model quantization to reduce memory usage, making it more optimal (on the memory side) than vanilla LoRa.

Model

• Base model: Qwen/Qwen3-1.7B
• Fine-tuning: QLoRA / LoRA
• Quantization: 4-bit NF4
• Training: TRL SFTTrainer
• Adapters: PEFT LoRA

Datasets

CUAD

theatticusproject/cuad

Used for:

• clause-category classification
• clause-presence detection
• relevant-language extraction

LegalBench / ContractNLI

nguha/legalbench

Adds contract reasoning tasks covering concepts such as confidentiality, limited use, survival of obligations, third-party sharing, compelled disclosure, return of confidential information, and related contractual obligations.

Training Run

Metric	Result
Training examples	31,395
Validation examples	2,730
Epochs	2
Training steps	1,824
Final training loss	0.01686
Final evaluation loss	0.00751
Evaluation mean token accuracy	99.74%
Training runtime	~16h 57m

training was completed locally on an NVIDIA GeForce RTX 3050 with 6 GB VRAM using 4-bit QLoRA

Adapter output:

outputs/legal-qwen3-1.7b-qlora

Base vs Fine-Tuned Evaluation

held-out evaluation of 100 examples compared the original Qwen3-1.7B against Qwen3-1.7B with the legal QLoRA adapter.

Metric	Base Model (qwen3-1.7b)	Fine-Tuned Model (ClauseQwen)	Improvements
JSON validity	100%	100%	0 pp
Clause-type accuracy	15%	92%	+77 pp
Clause-presence accuracy	76%	91%	+15 pp
Extraction F1	0.7275	0.9059	+0.1784

largest improvement was in clause-type classification, increasing from 15% to 92%. Extraction F1 increased from 0.7275 to 0.9059 while maintaining 100% valid JSON output.

ContractNLI note: The reported 100-example benchmark contained CUAD extraction and classification examples only. ContractNLI classification accuracy was therefore not evaluated in this benchmark and should be measured separately with a stratified ContractNLI evaluation.

HF Imports

pipeline:

from transformers import pipeline

pipe = pipeline("text-generation", model="AaronTekle/legal-qwen3-1.7b-qlora")

load model directly:

from transformers import AutoModel
model = AutoModel.from_pretrained("AaronTekle/legal-qwen3-1.7b-qlora", device_map="auto")