thurstonpowers/xll-laya-verifier

XLL Laya Verifier — v1.1 actor-relation fact checking

Task-specialized fine-tune of Laya (421M, ModernBERT-large backbone) for verifying v1.1 actor-oriented SPO assertions extracted from United States statutes and regulations. Given a source section and a candidate assertion (actor → relationship → target), it returns three calibrated noul (true/false probability) verdicts:

1. direction_correct — are the semantic roles (subject, predicate, target, recipient) assigned as the source actually writes them?
2. claim_supported — does the source support the complete proposed relationship, including roles, action, modality, negation, and target meaning?
3. scope_preserved — does the candidate keep all source restrictions governing the relationship (conditions, exceptions, applicability, quantities, AND/OR alternatives)?

This is the v1.1 three-question actor contract — a breaking change from the single claim_stated verifier previously shipped in this repository. Campaigns citing thurstonpowers/xll-laya-verifier (checkpoint laya-2ep) refer to that earlier revision, retrievable via git history.

Contract

The frozen question texts ship in questions.json (canonical sha256 d24107da8afca6260de1366b7a17e6a857b1e3a9e06b436ce9fffacfed993962) and are byte-identical to training. State shape:

{
  "source_identifier": "ct/regulations/sec-4-157-6",
  "source_text": "...full section text...",
  "candidate": { "subject": {...}, "predicate_text": "...", "object_literal": "...", "quote": {...}, "negated": false, "qualifiers": [], "participants": [] },
  "reading": {
    "subject": "The commissioner",
    "predicate": "may order",
    "target": "a pre-hearing conference",
    "qualifiers": [],
    "participants": []
  }
}

reading uses resolved_to → raw → citation in that order, with object_literal as the target fallback.

Usage

import json
from laya import load

agent = load("thurstonpowers/xll-laya-verifier")
questions = json.load(open("questions.json"))
result = agent.predict(state, questions)   # all three questions, one forward pass
print({k: round(v["noul"], 3) for k, v in result["answers"].items()})

Runs on CPU. Verification bands used in the XLL campaign: fail if any noul < 0.5, flag if any < 0.65, else pass (heuristics, not calibrated decision thresholds).

Training

Fine-tuned on the DGX Spark fleet with thermally paced full-parameter training (pause 87°C / resume 80°C), micro-batch 1, effective batch 64:

Setting	Value
Rows	61,279 train (34,193 unique) / 2,104 dev
Epochs	1 completed of 2 planned (operator early stop after epoch 1; checkpoint then calibrate + evaluate only)
Calibration	4,767 items, source-group disjoint
Finished	2026-10-12

Training data: thurstonpowers/xll-spo-dataset, v1.1/laya/ (train/dev splits disjoint by CFR part group; exact payload dedup; contradictory gold rejected).

Evaluation

Calibrated held-out evaluation (DEV_EVAL.json, 6,312 items, source-group disjoint):

• Accuracy 0.949, mean confidence 0.968
• ECE 0.019, Brier 0.091 (top-1 0.0455)

Calibration caveat: the fitted noul temperature clamped at the top of its [0.5, 5] range, and the laya runtime reports some checkpoint temperature entries outside the valid interval (substituted with 0.5). Treat confidence values as approximately calibrated; the recommended decision rule is threshold banding, not raw-confidence interpretation.

Limitations

• Verifies submitted claims only; it cannot detect relationships that were never submitted (assertion support ≠ section completeness).
• Trained on US federal sources (eCFR/GPO teacher data); Connecticut is used downstream, not in training.
• Scope/role verdicts are model judgments on quoted evidence; adjudication workflows should treat flags and fails as review queues, not ground truth.

Provenance

See WEIGHTS_PROVENANCE.json for weight hashes, question-set binding, and the verification chain.