thurstonpowers/xll-spo-verifier

XLL SPO Verifier — v1.1 actor-relation verification

Task-specialized fine-tune of Laya (421M, ModernBERT-large backbone) for verifying v1.1 actor-oriented SPO assertions extracted from United States statutes and regulations. Given a source section and a candidate assertion (actor → relationship → target), it returns three calibrated noul (true/false probability) verdicts. It is designed to pair with XLL SPO Nemotron 4B, which extracts the candidate relationships. The verifier checks:

1. direction_correct — are the semantic roles (subject, predicate, target, recipient) assigned as the source actually writes them?
2. claim_supported — does the source support the complete proposed relationship, including roles, action, modality, negation, and target meaning?
3. scope_preserved — does the candidate keep all source restrictions governing the relationship (conditions, exceptions, applicability, quantities, AND/OR alternatives)?

This is the v1.1 three-question actor contract — a breaking change from the single claim_stated verifier previously shipped in this repository. Campaigns citing thurstonpowers/xll-laya-verifier (checkpoint laya-2ep) refer to that earlier revision, retrievable via git history.

Contract

The frozen question texts ship in questions.json (canonical sha256 d24107da8afca6260de1366b7a17e6a857b1e3a9e06b436ce9fffacfed993962) and are byte-identical to training. The canonical JSON SHA-256 above hashes json.dumps(questions, sort_keys=True, separators=(",", ":")).encode("utf-8"); it is not the raw-file checksum. The raw-file SHA-256 is 1de33a92f9a0a9b33a757d308620484cba6982980395d9b86c61d8ed13737469. Both checks are shown below. State shape:

{
  "source_identifier": "ct/regulations/sec-4-157-6",
  "source_text": "...full section text...",
  "candidate": { "subject": {...}, "predicate_text": "...", "object_literal": "...", "quote": {...}, "negated": false, "qualifiers": [], "participants": [] },
  "reading": {
    "subject": "The commissioner",
    "predicate": "may order",
    "target": "a pre-hearing conference",
    "qualifiers": [],
    "participants": []
  }
}

reading uses resolved_to → raw → citation in that order, with object_literal as the target fallback.

Usage

import hashlib
import json
from pathlib import Path
from huggingface_hub import HfApi, snapshot_download
from laya import load

repo_id = "thurstonpowers/xll-spo-verifier"
revision = HfApi().model_info(repo_id).sha  # resolve once; retain this commit for reproducibility
snapshot = snapshot_download(repo_id, revision=revision)
question_bytes = (Path(snapshot) / "questions.json").read_bytes()
assert hashlib.sha256(question_bytes).hexdigest() == "1de33a92f9a0a9b33a757d308620484cba6982980395d9b86c61d8ed13737469"
questions = json.loads(question_bytes)
canonical = json.dumps(questions, sort_keys=True, separators=(",", ":")).encode("utf-8")
assert hashlib.sha256(canonical).hexdigest() == "d24107da8afca6260de1366b7a17e6a857b1e3a9e06b436ce9fffacfed993962"
agent = load(snapshot)  # model and questions from the same pinned revision
result = agent.predict(state, questions)   # all three questions, one forward pass
print({k: round(v["noul"], 3) for k, v in result["answers"].items()})

Runs on CPU. Verification bands used in the XLL campaign: fail if any noul < 0.5, flag if any < 0.65, else pass (heuristics, not calibrated decision thresholds).

Training

Fine-tuned on the DGX Spark fleet with thermally paced full-parameter training (pause 87°C / resume 80°C), micro-batch 1, effective batch 64:

Setting	Value
Rows	61,279 train (34,193 unique) / 2,104 dev
Epochs	1 completed of 2 planned (operator early stop after epoch 1; checkpoint then calibrate + evaluate only)
Calibration	4,767 items, source-group disjoint
Finished	2026-10-12

Training data: thurstonpowers/xll-spo-verifier, v1.1/laya/ (train/dev splits disjoint by CFR part group; exact payload dedup; contradictory gold rejected).

Evaluation

Calibrated held-out evaluation (DEV_EVAL.json, 6,312 items, source-group disjoint):

• Accuracy 0.949, mean confidence 0.968
• ECE 0.019, Brier 0.091 (top-1 0.0455)

Calibration caveat: the fitted noul temperature clamped at the top of its [0.5, 5] range, and the laya runtime reports some checkpoint temperature entries outside the valid interval (substituted with 0.5). Treat confidence values as approximately calibrated; the recommended decision rule is threshold banding, not raw-confidence interpretation.

Limitations

• Verifies submitted claims only; it cannot detect relationships that were never submitted (assertion support ≠ section completeness).
• Trained on US federal sources (eCFR/GPO teacher data); Connecticut is used downstream, not in training.
• Scope/role verdicts are model judgments on quoted evidence; adjudication workflows should treat flags and fails as review queues, not ground truth.

Provenance

See WEIGHTS_PROVENANCE.json for weight hashes, question-set binding, and the verification chain.

Replication dataset

Training and evaluation data: thurstonpowers/xll-spo-verifier. Use the model-specific contract and splits described in the dataset card.

For the current actor-relation checkpoint, use the v1.1/ files, not the legacy provision-oriented root files. The dataset includes exact model-bound splits, hashes, original reports and replication code.