Joint training checkpoint step 0

README.md

@@ -9,8 +9,8 @@ tags: [tabular-data, masked-lm, token-classification, experimental]
 
 Experimental budget-limited mmBERT-base joint fine-tuning checkpoint.
 
-One shared encoder, 18 category BIO outputs, insertion outputs, and the pretrained MLM vocabulary head. Correction is trained in place: the selected corrupted span is replaced by a variable length mask buffer inside its original cell, with no appended JSON request. The MLM head emits replacement tokens followed by `END_EDIT` and visible padding tokens.
-Completed optimizer updates: **800**. This checkpoint is a research baseline, not a validated automatic cleaning system.
+One shared encoder, 2 residual category BIO/insertion outputs (`text.encoding`, `text.spelling`), and the pretrained MLM vocabulary head. Declared schema rules detect deterministic violations before neural classification. Correction generates a complete cell inside `<replacement>` beside the visible `<original>` in the same cell. Valid copies are training targets too. The MLM head emits replacement tokens followed by `END_EDIT` and visible padding tokens.
+Completed optimizer updates: **0**. This checkpoint is a research baseline, not a validated automatic cleaning system.
 
 ## Use and resume
 
@@ -22,41 +22,102 @@ The recoverable state includes model, tokenizer, optimizer and random generators
 
 Canonical labels refer to decoded Unicode cell coordinates. Prediction is token-granular. Disabled categories are filtered before edits; overlapping proposals and linked structural repairs require abstention/group handling. Suggestions are not automatically applied. Missing values need not have recoverable intended contents. Real-source unlabelled cells are excluded from negative supervision.
 
-Dataset revision: `7da5aa80bc2a68a375930f3bd3ed2f7fdda0152f`. Train/validation formats are held out; test data was not used during training or checkpoint selection. A frozen test diagnostic is reported below. This dataset is mostly synthetic. The detector/correction task weight is `ln(18)/ln(256000) = 0.2321`; the optional conditional-missing pair weight is recorded in `run_config.json` and `metrics.json`. The run records periodic held-out validation losses for both tasks in `validation_history`, then a final validation report. These are development diagnostics, not deployment precision or end-to-end repair accuracy. No probability calibration or latency guarantee is claimed. See `metrics.json` for sample sizes, loss, operating-point diagnostics and exact-repair outcomes.
+Dataset revision: `602b8a454ca448713d8a5658650d4feaad8d579c`. Train/validation formats are held out; test data was not consumed. This dataset is mostly synthetic. The detector/correction task weight is `ln(2)/ln(256000) = 0.055661`; valid-copy correction examples are sampled with probability 0.30. The run records periodic held-out validation losses for both tasks in `validation_history`, then a final validation report. These are development diagnostics, not deployment precision or end-to-end repair accuracy. Conditional pseudo-perplexity is scored with each response token masked in turn, original visible; an acceptance threshold is fitted on the reported validation sample. Its sample size and observed errors limit its reliability. No latency guarantee is claimed. See `metrics.json` for sample sizes, loss, operating-point diagnostics and exact-repair outcomes.
 
-## Licensing
-
-TabFix fine-tuning contributions are CC BY-NC 4.0; commercial use requires the owner's permission. The underlying mmBERT material remains under its MIT terms and attribution, attributed in BASE_MODEL_CARD.md for pretrained checkpoints. Dataset source terms and USDA attribution remain documented in the dataset card.
-
-## Training controls
-
-The learning rate warms up for 100 updates, then follows cosine decay to 10% of its peak over a fixed 1600-update horizon. A fixed, stratified validation panel of up to 252 records is evaluated before training and every 200 updates or 600 seconds. Selection minimizes detection loss plus 0.232104 times correction loss (equal averages of fully and partially masked views). Training stops after 4 checks without an improvement of at least 0.001. Final publication restores the best complete checkpoint; recoverable latest and best revisions are retained as tags in this repository. These controls reduce risk; they do not guarantee generalization. The test split is not used for checkpoint selection.
-
-## Measured validation and end-to-end results
+## Validation results
 
-Training started from pretrained mmBERT. The selected checkpoint is update **800**, restored after training reached update 1523 and its time budget. Model and optimizer/RNG hashes match the saved best-step-800 checkpoint. Selection used a fixed 252-example validation panel with 393 eligible correction edits.
+Selected step: 0. Stop reason: training in progress.
 
-| Fixed validation panel | Initial | Selected (800) | Last evaluated (1523) |
-|---|---:|---:|---:|
-| Detection loss | 1.69733 | 0.20521 | 0.33725 |
-| Correction loss | 9.67505 | 0.45218 | 0.44362 |
-| Weighted selection score | 3.94294 | 0.31017 | 0.44021 |
+```json
+{}
+```
 
-![Validation curves](evaluation/validation_curves.png)
-
-The final gold-span validation diagnostic repaired 19/28 references exactly. This supplies the correct error location and is **not end-to-end accuracy**.
-
-### Frozen test diagnostic (2026-09-20)
-
-The complete pipeline scanned 55 synthetic test tables spanning 18 error categories, seven languages and 14 source families, with threshold 0.9 fixed before evaluation. All 55 results use L40S BF16. Model revision evaluated: `044567585e7943aab9252339cfe7bb1c2c44509b`; subsequent documentation commits do not change its weights.
-
-- Faulty cells exactly repaired: **23/45 (51.1%)**.
-- Originally valid cells changed incorrectly: **55/6,193 (0.89%)**.
-- Detection precision (cell + category): **35/161 (21.7%)**; recall: **35/46 (76.1%)**.
-- Hard-negative tables harmed: **11/18**.
+## Licensing
 
-This pipeline is **not ready for automatic correction**. On this diagnostic it harmed more valid cells than it repaired faulty cells. Date formatting accounts for 28 harmed cells, required missing values 12, spelling 10, temporal consistency 4 and unit consistency 1. Relationship errors were missed in the small positive sample. Missing-value generation can invent content even when a replacement is not inferable.
+TabFix fine-tuning contributions are CC BY-NC 4.0; commercial use requires the owner's permission. The underlying mmBERT material remains under its MIT terms and attribution, attributed in BASE_MODEL_CARD.md for pretrained checkpoints. Dataset source terms and USDA attribution remain documented in the dataset card.
 
-The sample is deliberately stratified and small; these are diagnostic counts, not production performance estimates. Category-level denominators are small, and compound cells can count under multiple categories. The current policy refuses overlapping repairs and does not apply structural shifts. Test results have not been used to tune thresholds or weights.
+## Training controls
 
-[Detailed category findings](evaluation/test/FINDINGS.md) · [Raw counts](evaluation/test/summary.json) · [Frozen sample](evaluation/test/selection.json) · [Provenance](evaluation/test/provenance.json). Per-case inputs, outputs and references are retained in `evaluation/test/case_*.json`. Full training and validation measurements remain in [metrics.json](metrics.json).
+The learning rate warms up for 100 updates, then follows cosine decay to 10% of its peak over a fixed 5000-update horizon. A fixed, stratified validation panel of up to 28 records is evaluated before training and every 300 updates or 600 seconds. Selection minimizes detection loss plus 0.055661 times correction loss (equal averages of fully and partially masked views). Training stops after 4 checks without an improvement of at least 0.001. Final publication restores the best complete checkpoint; recoverable latest and best revisions are retained as tags in this repository. These controls reduce risk; they do not guarantee generalization. The test split is not used for checkpoint selection.
+
+## Recorded run results
+
+```json
+{
+  "tiny_smoke": false,
+  "dataset_revision": "602b8a454ca448713d8a5658650d4feaad8d579c",
+  "train_records": 27948,
+  "correction_eligible_records": 27805,
+  "correction_request_version": 4,
+  "batch_size": 4,
+  "start_step": 0,
+  "device": "cuda",
+  "parameters": 307794699,
+  "validation_history": [
+    {
+      "step": 0,
+      "detection_loss": 1.5472150828157152,
+      "correction_loss": 6.555269168309259,
+      "correction_full_mask_loss": 7.438174559519841,
+      "correction_partial_mask_loss": 5.672363777098676,
+      "selection_score": 1.9120902847241386,
+      "examples": 28,
+      "correction_edits": 52,
+      "correction_category_counts": {
+        "copy": 10,
+        "text.spelling": 7,
+        "format.boolean": 7,
+        "format.date": 3,
+        "text.encoding": 5,
+        "schema.category": 3,
+        "schema.identifier": 4,
+        "schema.type": 2,
+        "consistency.dependency": 6,
+        "consistency.temporal": 3,
+        "consistency.unit": 1,
+        "format.case": 1
+      },
+      "token_and_gap_tp_fp_fn_at_0.9": [
+        1,
+        234,
+        69
+      ],
+      "sample_ids": [
+        "7cc2b2069f314821114a2a2b09543c680923083980692c6a95b3371b316390d9-open-clean",
+        "2c30950b5459afe6967673e8b23c222adab49967bf80e72af0ef1d4fb96296ca-open",
+        "bd4bba00c41c73249e4fd9fe97876df42dbaefbe6aabeec4430d2acb63926318-open",
+        "0121e99febc488b49db6e39b07ef71255f5a94d9ef0292f7a34a705e59271bbe-open",
+        "4d74e65f621bbb1dfc2848cdb8aefbd32befa5042d8482c3340de2d9833c06ec-open",
+        "d120e4d54be4c9692441a06d9cc4bf1db02bd205c6a317a1a7ac4548c09666cc-open",
+        "d6b7a9c8fc0552cfcf0d26ce0ce1be2663d5aa86d80527fc77c4ccc02a036984-open",
+        "05f04c447b2d5e98f170add5fcdde123cb8db18946eb5e21ea8573fdfdd66ffe-open",
+        "b68fd4277af10beef7958e8d2f6b31805d635db888e60b38ae2306bf138ea584-open",
+        "c2710a557bc21c7f5d4c38d580e9d54042923e32d22848cc3f025caa864aa51c-open",
+        "da8d1a36641b16437fd72e9db23a60fbb6f185acdceec0f217491c232d3602b5-open-clean",
+        "55e92ca80f64845e208d3cda2f1e8fc3cb620ca57c66fce22d9bb9a86321af4a-open",
+        "53820fc3b14267390c15dfad0a80f83c9d0ed59c95e40c96c2283e4f3b167e09-open",
+        "d5f74ad40dca51caabef5cfcfc1845e3b40fcd71608304e4036786b6e27bfad6-open",
+        "6481cc1b3a6148ca5e8df1783aa3c15bd97b4224ea44ec304e2448bd613c3a07",
+        "8a1d338e2b1a352eac2c0834432588239c231b43df14e68402999151d62a70d8",
+        "04434b82c7685003521e12204c5af32606d402dd454be5d270c2b39c5a3c2b55",
+        "7aeaf4538d564e7bfc4a20cac6b97e42655170cd4ac40c98db0d24ff056ba388",
+        "9d7d44d25b2bd975829249dc3cd673c9a972f47ad8b1fa1bf0206f527d27934c",
+        "b01405d247c32e01b05e977fe963ca8ab440e1559f1c7f8adf0a99928d29b997",
+        "89f4e9132a30351d7836acfa80b56de26924f82d1b2403e48f1c265b4053ab99",
+        "fbe61f1cc2096a02443b68b5dbf6da5694e8cb60ec5fb445ca5bc29fc147b0e9",
+        "4dee644acbc9859035ebc82ce0d74ac45065b429f74ec833d9ae5dda3daf02a0",
+        "86e7d7c56f4e30ae4e63da4327d92c7a9ff2e18d47611dcba9e794b11a222f93",
+        "d515d8316e03100e37bb3be520d62c30463d7d30a235e0921c7ff2d88921c714",
+        "b4a926b6d1c343e53f4fd04dc327fb0fd86d3f4532b964fea22340ef73ca68de",
+        "3e8dff1e9e6db8d09c388b141194864e22c347550db958abeae4500015fa8ecb",
+        "522ef97370ddbad950ea41bb8000b46488a7db7a05af8ee602117f2c0e2c400f"
+      ]
+    }
+  ],
+  "selection": {
+    "best_score": 1.9120902847241386,
+    "best_step": 0,
+    "bad_checks": 0
+  }
+}
+```

categories.json

@@ -1,20 +1,4 @@
 [
   "text.encoding",
-  "text.invisible",
-  "text.spelling",
-  "format.number",
-  "format.date",
-  "format.boolean",
-  "format.case",
-  "format.whitespace",
-  "format.unit",
-  "schema.type",
-  "schema.identifier",
-  "schema.category",
-  "consistency.unit",
-  "consistency.dependency",
-  "consistency.temporal",
-  "missing.required",
-  "missing.conditional",
-  "structure.shift"
+  "text.spelling"
 ]

checkpoint.json

@@ -1,19 +1,19 @@
 {
   "version": 1,
-  "step": 800,
-  "dataset_revision": "7da5aa80bc2a68a375930f3bd3ed2f7fdda0152f",
+  "step": 0,
+  "dataset_revision": "602b8a454ca448713d8a5658650d4feaad8d579c",
   "base_revision": "c5955035435e2bf121cde7f3c8863ef52ff35d82",
   "files": {
-    "model.safetensors": "940f08b9dfd115245a4f2455de8f7304535577a6704f627dcb94a65cd02b6226",
+    "model.safetensors": "90b750926e48ae595b38bf48d9b890758a27de4ba5d1f0f00706bbf94b20e18d",
     "config.json": "9e105ce4ed25ed2d6be2443eeb69932a530668a248ce3211d43a89a136d42531",
     "tokenizer_config.json": "51d83f5e4c1174f8dad62610076998730a57a9ac7025680653c720bca2a9b0a1",
     "tokenizer.json": "47e408472c05385fea5fb9f8b02bbd838d55da8f15a22c863a99fd31d89243f5",
-    "training_state.pt": "6e70e38863b991a90700c7dcdc4db2d774332fdc48dec046d6968ab86d898b02",
-    "run_config.json": "eec4d8db620354de7883275f2c7282ee3bdd72328c3ff829001e248aeea49962",
-    "metrics.json": "78d483e3ef5107dee73faaf6c1bbb6d34f6efc7bb67814a9cc527bd8f792a036",
-    "categories.json": "d95af1bcb70bc78d53297d4e57909776773fff76329098ece751e0734f03b29b",
-    "train_tabfix.py": "8712d7afc651f2d3c55210f3a062ad6a19a94b6185b60d8b5593cacbdf31f7be",
-    "README.md": "e7acd7f995d46f0efee4036271ec7d6f2100b2b7280257312d9cca161d9c350a",
+    "training_state.pt": "cd9e1448a8ac294f6f6d25dc2d5a1fc00faf3ff28d70475cf1942077b0ca5aae",
+    "run_config.json": "d9e42266559b78f89751de2b1f9e9b27229cbd7646c126ea00fd691144495c1f",
+    "metrics.json": "10b5cd912f0fa835b3542364cc6ceb7a6a5232babdecfea8e1095dab06b6eab5",
+    "categories.json": "da00946a1a8dfd24dd3bd01f47a4493abe6ade281d6a8571ae9728e0406d4aab",
+    "train_tabfix.py": "b16a43dbb276461d319a05bd4719de7520e0612fa23f7d83fb5fba58bebbc24f",
+    "README.md": "4b63d21bde39281a638ad6333175ebe4b8290646b9bf6434c0bff21c15673fcf",
     "LICENSE": "b6c1c7bb5aae03bb6b3396fc7c65cbe7306c422ed424b0b7b687d9a50cc15412",
     "BASE_MODEL_CARD.md": "35724415037ee3159aa733c16050620cc19a76219112a6ceeb3949572e0b37e9"
   },

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:940f08b9dfd115245a4f2455de8f7304535577a6704f627dcb94a65cd02b6226
-size 1231390724
+oid sha256:90b750926e48ae595b38bf48d9b890758a27de4ba5d1f0f00706bbf94b20e18d
+size 1231193844

run_config.json

@@ -1,28 +1,26 @@
 {
   "repo_id": "Antix5/tabfix-preview",
-  "output": "tabfix_run",
+  "output": "/tmp/tabfix-v2",
   "local_data": "",
   "tokenizer": "",
   "resume": "",
   "resume_revision": "main",
   "tiny": false,
-  "steps": 1600,
-  "train_seconds": 2100,
-  "save_seconds": 300,
+  "steps": 100000,
+  "train_seconds": 12000,
+  "save_seconds": 600,
   "lr": 2e-05,
   "seed": 20260913,
-  "validation_examples": 252,
+  "validation_examples": 28,
   "validation_repairs": 28,
   "validation_seconds": 600,
-  "validation_steps": 200,
+  "validation_steps": 300,
   "patience": 4,
   "min_delta": 0.001,
   "warmup_steps": 100,
-  "schedule_steps": 1600,
+  "schedule_steps": 5000,
   "max_length": 8192,
-  "correction_weight": 0.23210369975344178,
-  "batch_size": 2,
-  "verify_only": false,
-  "conditional_pair_weight": 0.25,
-  "allow_objective_change": false
+  "correction_weight": 0.055661360737461876,
+  "batch_size": 4,
+  "verify_only": false
 }

train_tabfix.py

@@ -22,6 +22,7 @@ from collections import Counter
 from contextlib import nullcontext
 from dataclasses import asdict, dataclass
 from decimal import Decimal, InvalidOperation
+from datetime import date, datetime
 from functools import lru_cache
 import hashlib
 import html
@@ -47,10 +48,10 @@ from tokenizers import Tokenizer, pre_tokenizers
 from transformers import AutoTokenizer, ModernBertConfig, ModernBertForMaskedLM, PreTrainedTokenizerFast
 
 DATASET = "Antix5/tabular-errors-v1"
-REVISION = "7da5aa80bc2a68a375930f3bd3ed2f7fdda0152f"
+REVISION = "602b8a454ca448713d8a5658650d4feaad8d579c"
 BASE = "jhu-clsp/mmBERT-base"
 BASE_REVISION = "c5955035435e2bf121cde7f3c8863ef52ff35d82"
-CATEGORIES = (
+BUSINESS_CATEGORIES = (
     "text.encoding", "text.invisible", "text.spelling", "format.number",
     "format.date", "format.boolean", "format.case", "format.whitespace",
     "format.unit", "schema.type", "schema.identifier", "schema.category",
@@ -58,6 +59,7 @@ CATEGORIES = (
     "missing.required", "missing.conditional", "structure.shift",
 )
 CONTROL = ("<|tabfix_repair|>", "<|tabfix_end|>", "<|tabfix_pad|>")
+CATEGORIES = ("text.encoding", "text.spelling")
 K = len(CATEGORIES)
 stop_requested = False
 
@@ -104,6 +106,191 @@ def emit(event: str, **fields: object) -> None:
     print(json.dumps({"event": event, **fields}, ensure_ascii=False), flush=True)
 
 
+@dataclass(frozen=True)
+class RuleFinding:
+    row: int
+    columns: tuple[int, ...]
+    category: str
+    rule: str
+    status: str
+    evidence: str
+
+
+def xml_json(element: ET.Element, name: str, default: object) -> object:
+    text = element.findtext(name)
+    return cast(object, json.loads(text)) if text else default
+
+
+def column_checks(value: str, column: ET.Element) -> list[tuple[str, str, bool | None]]:
+    """Check declared contracts only; None means unsupported/undetermined."""
+    rules = mapping(xml_json(column, "constraints", {}))
+    formats = [string(x) for x in sequence(xml_json(column, "accepted_formats", []))]
+    nulls = sequence(xml_json(column, "null_markers", []))
+    if not value or value in nulls:
+        return [("missing.required", "nullability", column.get("nullable") != "forbidden")]
+    result: list[tuple[str, str, bool | None]] = []
+    for key in ("enum", "lexicon"):
+        if rules.get(key):
+            result.append(("format.boolean" if column.get("type") == "boolean" else "schema.category", key, value in sequence(rules[key])))
+    if isinstance(rules.get("pattern"), str):
+        try:
+            result.append(("schema.identifier", "pattern", re.fullmatch(string(rules["pattern"]), value) is not None))
+        except re.error:
+            result.append(("schema.identifier", "pattern", None))
+    for key in ("upper", "lower"):
+        if rules.get("case") == key:
+            result.append(("format.case", "case", value == (value.upper() if key == "upper" else value.lower())))
+    if rules.get("trimmed") is True:
+        result.append(("format.whitespace", "trimmed", value == value.strip()))
+    forbidden = rules.get("forbidden_characters")
+    if isinstance(forbidden, list):
+        result.append(("text.invisible", "forbidden_characters", all(string(c) not in value for c in sequence(cast(object, forbidden)))))
+    kind = column.get("type")
+    if value in sequence(rules.get("accepted_representations", [])):
+        return result
+    if kind in ("integer", "decimal", "numeric_syntax"):
+        patterns: list[str] = []
+        if kind == "integer":
+            patterns = [r"[+-]?[0-9]+"]
+        elif kind == "numeric_syntax":
+            patterns = [r"[+-]?(?=[0-9., \u00a0\u202f]*[0-9])[0-9., \u00a0\u202f]+(?:[eE][+-]?[0-9]+)?"]
+        else:
+            for fmt in formats:
+                if fmt.startswith("decimal_"):
+                    sep = "," if "comma" in fmt else r"\."
+                    patterns.append(r"[+-]?[0-9]+" + (sep + r"[0-9]{2}" if "2dp" in fmt else "(?:" + sep + r"[0-9]+)?"))
+        valid = any(re.fullmatch(p, value) for p in patterns) if patterns else None
+        result.append(("schema.type" if kind == "integer" else "format.number", "numeric_format", bool(valid) if valid is not None else None))
+        if valid and kind != "numeric_syntax":
+            parsed = Decimal(value.replace(",", "."))
+            for key in ("minimum", "maximum"):
+                bound = rules.get(key)
+                if isinstance(bound, (int, float)):
+                    result.append(("schema.type", key, parsed >= Decimal(str(bound)) if key == "minimum" else parsed <= Decimal(str(bound))))
+    elif kind == "date":
+        fmts = {"YYYY-MM-DD": "%Y-%m-%d", "DD/MM/YYYY": "%d/%m/%Y", "MM/DD/YYYY": "%m/%d/%Y"}
+        valid_date = False
+        supported = bool(formats) and all(f in fmts for f in formats)
+        for fmt in formats:
+            if fmt in fmts:
+                try:
+                    dt = datetime.strptime(value, fmts[fmt])
+                    valid_date |= dt.strftime(fmts[fmt]) == value
+                except ValueError:
+                    pass
+        result.append(("format.date", "calendar_format", valid_date if supported else None))
+    return result
+
+
+def deterministic_findings(xml: str) -> list[RuleFinding]:
+    root = ET.fromstring(xml)
+    columns = root.findall("./schema/column")
+    schema = root.find("schema")
+    if schema is None:
+        return []
+    keys = {c.get("key", ""): i for i, c in enumerate(columns) if c.get("key")}
+    relations = sequence(xml_json(schema, "relations", []))
+    dependencies = mapping(xml_json(schema, "functional_dependencies", {}))
+    findings: list[RuleFinding] = []
+    for row in root.findall("./rows/row"):
+        ri = int(row.attrib["index"])
+        values = [c.text or "" for c in row.findall("cell")]
+        if len(values) != len(columns):
+            continue
+        checks = [column_checks(v, c) for v, c in zip(values, columns)]
+        for ci, (value, col, cc) in enumerate(zip(values, columns, checks)):
+            for cat, rule, valid in cc:
+                findings.append(RuleFinding(ri, (ci,), cat, rule, "unknown" if valid is None else "valid" if valid else "invalid", "declared column contract"))
+            rules = mapping(xml_json(col, "constraints", {}))
+            condition = rules.get("required_when")
+            if isinstance(condition, dict):
+                cond = mapping(cast(object, condition))
+                determinant = cond.get("column")
+                valid_condition = isinstance(determinant, int) and 0 <= determinant < len(values)
+                known = valid_condition and bool(values[cast(int, determinant)]) and all(v is True for _, _, v in checks[cast(int, determinant)])
+                required: bool | None = None
+                if known:
+                    dv = values[cast(int, determinant)]
+                    if "equals" in cond:
+                        required = dv == cond["equals"]
+                    elif "not_equals" in cond:
+                        required = dv != cond["not_equals"]
+                present = bool(value) and value not in sequence(xml_json(col, "null_markers", []))
+                findings.append(RuleFinding(ri, (ci,), "missing.conditional", "required_when", "unknown" if required is None else "valid" if present or not required else "invalid", "declared condition; determinant must be valid"))
+        for raw in relations:
+            if not isinstance(raw, list) or not raw:
+                continue
+            rel = [string(x) for x in sequence(cast(object, raw))]
+            kind, args = rel[0], rel[1:]
+            if any(key not in keys for key in args):
+                continue
+            ix = tuple(keys[key] for key in args)
+            cat = "consistency.unit" if kind in ("unit_equivalence", "pack_mass", "unit_factor") else "consistency.temporal" if kind == "interval" else "consistency.dependency"
+            if kind == "conditional":
+                continue  # Executed through required_when.
+            status = "unknown"
+            if all(values[i] and all(v is True for _, _, v in checks[i]) for i in ix):
+                vv = [values[i] for i in ix]
+                try:
+                    valid: bool | None = None
+                    if kind == "lookup" and len(ix) == 2:
+                        reference = mapping(dependencies.get(args[1], {})).get(vv[0])
+                        valid = vv[1] == reference if reference is not None else None
+                    elif kind == "interval" and len(ix) == 3:
+                        start, end = date.fromisoformat(vv[0]), date.fromisoformat(vv[1])
+                        valid = end >= start and Decimal(vv[2]) == (end - start).days
+                    elif kind == "unit_equivalence" and len(ix) == 4:
+                        factors = {"kg": Decimal(1000), "g": Decimal(1), "mg": Decimal(".001")}
+                        if vv[1] in factors and vv[3] in factors:
+                            valid = Decimal(vv[0].replace(",", ".")) * factors[vv[1]] == Decimal(vv[2].replace(",", ".")) * factors[vv[3]]
+                    elif kind in ("difference", "multiply", "balance", "le"):
+                        nums = [Decimal(v.replace(",", ".")) for v in vv]
+                        if kind == "le" and len(nums) == 2:
+                            valid = nums[0] <= nums[1]
+                        elif len(nums) == 3:
+                            expected = nums[0] * nums[1] if kind == "multiply" else nums[0] - nums[1]
+                            # Compare at the explicitly declared output precision.
+                            formats = sequence(xml_json(columns[ix[-1]], "accepted_formats", []))
+                            if any("2dp" in string(f) for f in formats):
+                                expected = expected.quantize(Decimal(".01"))
+                            valid = expected == nums[2]
+                    status = "unknown" if valid is None else "valid" if valid else "invalid"
+                except (ValueError, InvalidOperation):
+                    status = "unknown"
+            findings.append(RuleFinding(ri, ix, cat, kind, status, "relation contradiction identifies a group, not its faulty member"))
+    return findings
+
+
+def deterministic_repair(value: str, column: ET.Element) -> str | None:
+    """Only unambiguous canonicalization, never fill missing information."""
+    if not value:
+        return None
+    rules = mapping(xml_json(column, "constraints", {}))
+    candidate = value.strip() if rules.get("trimmed") is True else value
+    if rules.get("case") == "upper":
+        candidate = candidate.upper()
+    elif rules.get("case") == "lower":
+        candidate = candidate.lower()
+    enum = sequence(rules.get("enum", []))
+    equivalent = [string(v) for v in enum if string(v).casefold() == candidate.casefold()]
+    if candidate not in enum and len(equivalent) == 1 and column.get("type") == "boolean":
+        candidate = equivalent[0]
+    checks = column_checks(candidate, column)
+    if candidate != value and checks and all(ok is True for _, _, ok in checks):
+        return candidate
+    return None
+
+
+def replace_cell_xml(xml: str, row: int, column: int, value: str) -> str:
+    cell = next(c for c in cells(xml) if (c.row, c.column) == (row, column))
+    escaped = html.escape(value, quote=False).replace("\r", "&#13;").replace("\n", "&#10;").replace("\t", "&#9;") if value else "<empty/>"
+    return xml[:cell.anchor] + escaped + xml[cell.close:]
+
+
+def invalid_cells(xml: str) -> set[tuple[int, int]]:
+    return {(f.row, c) for f in deterministic_findings(xml) if f.status == "invalid" for c in f.columns}
+
+
 @dataclass(frozen=True)
 class Edit:
     row: int
@@ -252,7 +439,10 @@ class Consumer:
             for at in boundaries:
                 gaps[anchor_token(c, offsets, at)] = [0 if trusted else -100] * K
         collisions = 0
-        for e in record.edits:
+        neural_indices = record.metadata.get("neural_error_indices")
+        for ei, e in enumerate(record.edits):
+            if e.category not in CATEGORIES or (isinstance(neural_indices, list) and ei not in neural_indices):
+                continue
             c = next(c for c in cc if (c.row, c.column) == (e.row, e.column))
             hit, _, _ = interval(c, offsets, e)
             category = CATEGORIES.index(e.category)
@@ -264,6 +454,14 @@ class Consumer:
                     target[category] = 1 if j == 0 else 2
             else:
                 gaps.setdefault(anchor_token(c, offsets, e.start), [-100] * K)[category] = 1
+        # Deterministic cells are not part of the residual neural objective.
+        blocked: set[tuple[int, int]] = invalid_cells(record.xml) if record.metadata.get("task_version") == 2 else set()
+        for c in cc:
+            if (c.row, c.column) in blocked:
+                for t in tokens_for(c, offsets):
+                    labels.pop(t, None)
+                for at in range(len(c.value) + 1):
+                    gaps.pop(anchor_token(c, offsets, at), None)
         pos = sorted(labels)
         gp = sorted(gaps)
         return Prepared(record, ids, offsets, cc, pos, [labels[p] for p in pos], gp,
@@ -272,28 +470,18 @@ class Consumer:
     def correction(self, p: Prepared, e: Edit, rng: random.Random, training: bool,
                    capacity: int | None = None, mask_all: bool = False) -> tuple[list[int], list[int], list[int], list[bool], str]:
         c = next(c for c in p.cells if (c.row, c.column) == (e.row, e.column))
-        hit, lo, hi = interval(c, p.offsets, e)
-        # Do not apply two differently categorized edits through the same token.
-        if training and any(other != e and other.row == e.row and other.column == e.column
-               and other.start < hi and other.end > lo for other in p.record.edits):
-            raise ValueError("Overlapping token-expanded repairs require explicit group handling")
-        target = c.value[lo:e.start] + e.replacement + c.value[e.end:hi]
-        gold = self.literal(target)
-        n = len(hit)
-        budget = capacity or min(64, max(8, n + max(2, math.ceil(.2 * n))))
-        if training and rng.random() < .25:
-            budget = min(64, budget * 2)
-        if training and len(gold) > budget:
-            budget = min(64, budget * 2)
-            self.stats["larger_training_buffer"] += 1
+        # Whole-cell target, original retained locally. The edit is supplied by
+        # correction_candidates; its replacement is never put into inference input.
+        target = c.value[:e.start] + e.replacement + c.value[e.end:]
+        gold = self.literal(target) if training else []
+        n = len(self.literal(c.value))
+        budget = capacity or min(256, max(8, n + max(2, math.ceil(.2 * n))))
         if training and len(gold) > budget:
-            raise ValueError("Repair exceeds bounded buffer")
-        char_start = c.bounds[lo][0] if lo < len(c.bounds) else c.close
-        char_end = c.bounds[hi-1][1] if hi > lo else char_start
-        if not c.value:
-            char_start, char_end = c.anchor, c.close  # Replace the whole <empty/> marker.
+            raise ValueError("Whole-cell repair exceeds source-derived buffer")
         mask_id = integer(self.tokenizer.mask_token_id)
-        skeleton = p.record.xml[:char_start] + CONTROL[0] + string(self.tokenizer.mask_token) * (budget + 1) + p.record.xml[char_end:]
+        original = p.record.xml[c.anchor:c.close]
+        local = "<original>" + original + "</original><replacement>" + string(self.tokenizer.mask_token) * (budget + 1) + "</replacement>"
+        skeleton = p.record.xml[:c.anchor] + local + p.record.xml[c.close:]
         ids = self.tokenizer.encode(skeleton, add_special_tokens=True)  # pyright: ignore[reportUnknownMemberType] — third-party typing boundary
         positions = [i for i, token in enumerate(ids) if token == mask_id]
         if len(positions) != budget + 1:
@@ -318,6 +506,11 @@ class Consumer:
 
 
 def eligible(record: Record) -> list[Edit]:
+    """Whole-cell targets, including certified valid copies, stored in release v2."""
+    candidates = record.metadata.get("correction_candidates")
+    if isinstance(candidates, list):
+        return [Edit.parse(e) for e in sequence(cast(object, candidates))]
+    # Local callers/tests can supply one selected edit directly.
     result: list[Edit] = []
     for value in sequence(record.metadata.get("repair_policies", [])):
         policy = mapping(value)
@@ -326,129 +519,6 @@ def eligible(record: Record) -> list[Edit]:
     return result
 
 
-@dataclass
-class ConditionalPair:
-    """Same empty field/context; only the declared determinant value changes."""
-    negative: Record
-    positive: Record
-    row: int
-    column: int
-
-
-def conditional_pairs(record: Record) -> list[ConditionalPair]:
-    # Only certified clean training records; do not invent labels for real sources.
-    if record.edits or record.metadata.get("label_scope") != "validated_synthetic_contracts":
-        return []
-    schema = ET.fromstring(record.xml).find("schema")
-    if schema is None:
-        return []
-    columns = schema.findall("column")
-    cc = cells(record.xml)
-    result: list[ConditionalPair] = []
-    targets = {integer(x) for x in sequence(record.metadata["target_rows"])}
-    supervised = {integer(x) for x in sequence(record.metadata["supervised_columns"])}
-    for col, definition in enumerate(columns):
-        rules_node = definition.find("constraints")
-        rules = mapping(cast(object, json.loads(rules_node.text or "{}"))) if rules_node is not None else {}
-        required = rules.get("required_when")
-        if not isinstance(required, dict) or col not in supervised:
-            continue
-        rule = mapping(cast(object, required))
-        determinant = rule.get("column")
-        if not isinstance(determinant, int) or isinstance(determinant, bool) or not 0 <= determinant < len(columns) or determinant == col:
-            continue
-        operators = [key for key in ("equals", "not_equals") if key in rule]
-        if len(operators) != 1:
-            continue
-        node = columns[determinant].find("constraints")
-        constraints = mapping(cast(object, json.loads(node.text or "{}"))) if node is not None else {}
-        allowed = [string(x) for x in sequence(constraints.get("enum", []))]
-        op = operators[0]
-        positive = [v for v in allowed if (v == rule[op]) == (op == "equals")]
-        negative = [v for v in allowed if (v == rule[op]) != (op == "equals")]
-        if not positive or not negative:
-            continue
-        for row in sorted(targets):
-            target = next((c for c in cc if (c.row, c.column) == (row, col)), None)
-            status = next((c for c in cc if (c.row, c.column) == (row, determinant)), None)
-            if target is None or status is None:
-                continue
-            for yes in positive:
-                for no in negative:
-                    def view(value: str, target: Cell = target, status: Cell = status) -> Record:
-                        xml = record.xml
-                        replacements = [(target.anchor, target.close, "<empty/>"),
-                                        (status.anchor, status.close, html.escape(value))]
-                        for lo, hi, replacement in sorted(replacements, reverse=True):
-                            xml = xml[:lo] + replacement + xml[hi:]
-                        return Record(record.id, xml, [], {"target_rows": [], "supervised_columns": [], "label_scope": "pair_only"})
-                    result.append(ConditionalPair(view(no), view(yes), row, col))
-    return result
-
-
-def conditional_pair_loss(model: TabFix, consumer: Consumer, pair: ConditionalPair,
-                          device: torch.device) -> Tensor:
-    """Proper binary CE on one matched negative/positive pair; no category dilution."""
-    losses: list[Tensor] = []
-    for record, label in ((pair.negative, 0.), (pair.positive, 1.)):
-        p = consumer.prepare(record)
-        cell = next(c for c in p.cells if (c.row, c.column) == (pair.row, pair.column))
-        position = anchor_token(cell, p.offsets, 0)
-        hidden = model.hidden(p.ids, device)
-        logit = model.insert.forward(hidden[position])[CATEGORIES.index("missing.conditional")].float()
-        losses.append(F.binary_cross_entropy_with_logits(logit, torch.tensor(label, device=device)))
-    return torch.stack(losses).mean()
-
-
-def diversify_arithmetic(records: list[Record]) -> list[Record]:
-    """Replace single arithmetic corruptions with bounded signed offsets; preserve sampling mass."""
-    output: list[Record] = []
-    for record in records:
-        if len(record.edits) != 1 or record.edits[0].category != "consistency.dependency" or record.metadata.get("label_scope") != "validated_synthetic_contracts":
-            output.append(record)
-            continue
-        e = record.edits[0]
-        schema = ET.fromstring(record.xml).find("schema")
-        if schema is None:
-            output.append(record)
-            continue
-        columns = schema.findall("column")
-        keys = {c.get("key"): i for i, c in enumerate(columns)}
-        node = schema.find("relations")
-        relations = sequence(cast(object, json.loads(node.text or "[]"))) if node is not None else []
-        if not any(sequence(rel)[0] in ("multiply", "difference", "balance") and keys.get(string(sequence(rel)[-1])) == e.column for rel in relations):
-            output.append(record)
-            continue
-        cell = next(c for c in cells(record.xml) if (c.row, c.column) == (e.row, e.column))
-        clean = cell.value[:e.start] + e.replacement + cell.value[e.end:]
-        try:
-            number_value = Decimal(clean.replace(",", "."))
-        except InvalidOperation:
-            output.append(record)
-            continue
-        node = columns[e.column].find("constraints")
-        rules = mapping(cast(object, json.loads(node.text or "{}"))) if node is not None else {}
-        deltas = [-10, -3, -2, -1, 1, 2, 3, 10]
-        random.Random(record.id).shuffle(deltas)
-        values: list[str] = []
-        for delta in deltas:
-            value = number_value + delta
-            if ("minimum" in rules and value < Decimal(str(rules["minimum"]))) or ("maximum" in rules and value > Decimal(str(rules["maximum"]))):
-                continue
-            rendered = str(value).replace(".", ",") if "," in clean else str(value)
-            values.append(rendered)
-        if not values:
-            output.append(record)
-            continue
-        for i, value in enumerate(values):
-            meta = dict(record.metadata)
-            meta["sampling_weight"] = number(meta["sampling_weight"]) / len(values)
-            xml = record.xml[:cell.anchor] + html.escape(value) + record.xml[cell.close:]
-            output.append(Record(record.id + f"-arithmetic-{i}", xml,
-                                 [Edit(e.row, e.column, 0, len(value), clean, e.category)], meta))
-    return output
-
-
 class Batch(Protocol):
     def to_pylist(self) -> list[object]: ...
 
@@ -500,11 +570,11 @@ def detection_loss(model: TabFix, p: Prepared, device: torch.device) -> Tensor:
 
 def detection_from_hidden(model: TabFix, p: Prepared, h: Tensor) -> Tensor:
     device = h.device
-    labels = torch.tensor(p.labels, device=device, dtype=torch.long)
+    labels = torch.tensor(p.labels, device=device, dtype=torch.long).reshape(-1, K)
     logits = model.bio.forward(h[p.positions]).reshape(-1, K, 3)
     loss = F.cross_entropy(logits.reshape(-1, 3).float(), labels.reshape(-1), ignore_index=-100, reduction="none").reshape(-1, K)
     bio = mean_parts(loss, labels > 0, labels != -100)
-    gl = torch.tensor(p.gap_labels, device=device, dtype=torch.float32)
+    gl = torch.tensor(p.gap_labels, device=device, dtype=torch.float32).reshape(-1, K)
     gi = model.insert.forward(h[p.gaps])
     il = F.binary_cross_entropy_with_logits(gi.float(), gl.clamp_min(0), reduction="none")
     return bio + .5 * mean_parts(il, gl > 0, gl != -100)
@@ -548,11 +618,9 @@ class RunConfig:
     warmup_steps: int = 100
     schedule_steps: int = 2400
     max_length: int = 8192
-    correction_weight: float = math.log(18) / math.log(256000)
+    correction_weight: float = math.log(K) / math.log(256000)
     batch_size: int = 2
     verify_only: bool = False
-    conditional_pair_weight: float = 0.0
-    allow_objective_change: bool = False
 
 
 def learning_rate(cfg: RunConfig, step: int) -> float:
@@ -623,6 +691,9 @@ def load_checkpoint(path: Path, device: torch.device) -> tuple[TabFix, PreTraine
     config._attn_implementation = "sdpa"  # pyright: ignore[reportPrivateUsage] — third-party typing boundary
     mlm = ModernBertForMaskedLM(config)
     model = TabFix(mlm)
+    saved_categories = cast(object, json.loads((path / "categories.json").read_text()))
+    if saved_categories != list(CATEGORIES):
+        raise ValueError("Checkpoint uses another label map; use its bundled training script")
     # Store tied embedding once; the duplicate decoder weight shares storage.
     weights = load_file(str(path / "model.safetensors"))
     weights["mlm.decoder.weight"] = weights["mlm.model.embeddings.tok_embeddings.weight"]
@@ -685,7 +756,7 @@ tags: [tabular-data, masked-lm, token-classification, experimental]
 
 {'Temporary random tiny smoke-test model, not a pretrained fine-tune.' if cfg.tiny else 'Experimental budget-limited mmBERT-base joint fine-tuning checkpoint.'}
 
-One shared encoder, 18 category BIO outputs, insertion outputs, and the pretrained MLM vocabulary head. Correction is trained in place: the selected corrupted span is replaced by a variable length mask buffer inside its original cell, with no appended JSON request. The MLM head emits replacement tokens followed by `END_EDIT` and visible padding tokens.
+One shared encoder, {K} residual category BIO/insertion outputs (`text.encoding`, `text.spelling`), and the pretrained MLM vocabulary head. Declared schema rules detect deterministic violations before neural classification. Correction generates a complete cell inside `<replacement>` beside the visible `<original>` in the same cell. Valid copies are training targets too. The MLM head emits replacement tokens followed by `END_EDIT` and visible padding tokens.
 Completed optimizer updates: **{step}**. This checkpoint is a research baseline, not a validated automatic cleaning system.
 
 ## Use and resume
@@ -698,7 +769,15 @@ The recoverable state includes model, tokenizer, optimizer and random generators
 
 Canonical labels refer to decoded Unicode cell coordinates. Prediction is token-granular. Disabled categories are filtered before edits; overlapping proposals and linked structural repairs require abstention/group handling. Suggestions are not automatically applied. Missing values need not have recoverable intended contents. Real-source unlabelled cells are excluded from negative supervision.
 
-Dataset revision: `{REVISION}`. Train/validation formats are held out; test data was not consumed. This dataset is mostly synthetic. The detector/correction task weight is `ln(18)/ln(256000) = 0.2321`; the optional conditional-missing pair weight is recorded in `run_config.json` and `metrics.json`. The run records periodic held-out validation losses for both tasks in `validation_history`, then a final validation report. These are development diagnostics, not deployment precision or end-to-end repair accuracy. No probability calibration or latency guarantee is claimed. See `metrics.json` for sample sizes, loss, operating-point diagnostics and exact-repair outcomes.
+Dataset revision: `{REVISION}`. Train/validation formats are held out; test data was not consumed. This dataset is mostly synthetic. The detector/correction task weight is `ln({K})/ln(256000) = {cfg.correction_weight:.6f}`; valid-copy correction examples are sampled with probability 0.30. The run records periodic held-out validation losses for both tasks in `validation_history`, then a final validation report. These are development diagnostics, not deployment precision or end-to-end repair accuracy. Conditional pseudo-perplexity is scored with each response token masked in turn, original visible; an acceptance threshold is fitted on the reported validation sample. Its sample size and observed errors limit its reliability. No latency guarantee is claimed. See `metrics.json` for sample sizes, loss, operating-point diagnostics and exact-repair outcomes.
+
+## Validation results
+
+Selected step: {report.get('selected_step', step)}. Stop reason: {report.get('stop_reason', 'training in progress')}.
+
+```json
+{json.dumps({key: value for key, value in mapping(report.get('validation', {})).items() if key != 'prediction_sample'}, ensure_ascii=False, indent=2)}
+```
 
 ## Licensing
 
@@ -719,50 +798,98 @@ The learning rate warms up for {cfg.warmup_steps} updates, then follows cosine d
 @torch.no_grad()
 def generate_repair(model: TabFix, consumer: Consumer, p: Prepared, edit: Edit, device: torch.device) -> str | None:
     model.eval()
-    for capacity in (None, 64):
+    for capacity in (None, 256):
         ids, positions, _, _, _ = consumer.correction(p, edit, random.Random(0), False, capacity)
         end = integer(consumer.tokenizer.convert_tokens_to_ids(CONTROL[1]))
+        pad = integer(consumer.tokenizer.convert_tokens_to_ids(CONTROL[2]))
         mask = integer(consumer.tokenizer.mask_token_id)
         limit = len(positions)
         finish: int | None = None
-        for _ in range(limit):
+        # PAD hypotheses before the selected END may need one additional pass.
+        for _ in range(2 * limit):
             active = [j for j, pos in enumerate(positions) if ids[pos] == mask and (finish is None or j < finish)]
             if not active:
                 break
             with autocast(device):
                 logits = model.vocab(model.hidden(ids, device)[[positions[j] for j in active]]).float()
             for special in consumer.tokenizer.all_special_ids:
-                if special != end:
+                if special not in (end, pad):
                     logits[:, special] = -torch.inf
             if finish is not None:
-                logits[:, end] = -torch.inf
+                logits[:, end] = logits[:, pad] = -torch.inf
             probs = logits.softmax(-1)
-            confidence, prediction = probs.max(-1)
-            # The reserved final slot may predict END_EDIT, never ordinary text.
-            # Retain its original probability instead of renormalizing it to 1.
+            # Keep original probabilities when restricting boundary candidates.
+            candidates = probs.clone()
             for i, j in enumerate(active):
+                if finish is None and j == 0:
+                    candidates[i, pad] = -1
                 if j == limit - 1:
-                    confidence[i] = probs[i, end]
-                    prediction[i] = end
+                    candidates[i] = -1
+                    candidates[i, end] = probs[i, end]
+                    candidates[i, pad] = probs[i, pad]
+            confidence, prediction = candidates.max(-1)
             confidence = torch.nan_to_num(confidence, nan=-1.0)
             pick = int(confidence.argmax())
             if float(confidence[pick]) < 0:
                 break
-            j = active[pick]
-            token = int(prediction[pick])
-            if token == end:
-                # Never discard committed non-padding tokens to its right.
-                if any(ids[positions[k]] != mask for k in range(j + 1, limit)):
-                    logits[pick, end] = -torch.inf
-                    token = int(logits[pick].argmax())
-                else:
-                    finish = j
+            j, token = active[pick], int(prediction[pick])
             ids[positions[j]] = token
-        if finish is not None and all(ids[positions[j]] != mask for j in range(finish)):
+            if token == end:
+                finish = j
+                # END defines the replacement length, including immediate deletion.
+                # Earlier hypotheses to its right must not prevent termination.
+                for k, pos in enumerate(positions):
+                    if k > j:
+                        ids[pos] = pad
+                    elif k < j and ids[pos] == pad:
+                        ids[pos] = mask
+        if finish is not None and all(ids[positions[j]] not in (mask, pad, end) for j in range(finish)):
             return consumer.literal_tokenizer.decode([ids[positions[j]] for j in range(finish)], skip_special_tokens=False)
     return None
 
 
+@torch.no_grad()
+def conditional_perplexity(model: TabFix, consumer: Consumer, p: Prepared, edit: Edit,
+                           answer: str, device: torch.device) -> float:
+    tokens = consumer.literal(answer)
+    end = integer(consumer.tokenizer.convert_tokens_to_ids(CONTROL[1]))
+    pad = integer(consumer.tokenizer.convert_tokens_to_ids(CONTROL[2]))
+    mask = integer(consumer.tokenizer.mask_token_id)
+    ids, positions, _, _, _ = consumer.correction(p, edit, random.Random(0), False,
+                                                max(8, len(tokens)))
+    target = tokens + [end]
+    for j, pos in enumerate(positions):
+        ids[pos] = target[j] if j < len(target) else pad
+    losses: list[float] = []
+    model.eval()
+    for start in range(0, len(target), 2):
+        indices = list(range(start, min(start + 2, len(target))))
+        views = [ids.copy() for _ in indices]
+        for view, j in zip(views, indices):
+            view[positions[j]] = mask
+        with autocast(device):
+            hidden = model.hidden_batch(views, device)
+            selected = torch.stack([hidden[k, positions[j]] for k, j in enumerate(indices)])
+            logits = model.vocab(selected).float()
+        losses.extend(cast(list[float], F.cross_entropy(logits, torch.tensor([target[j] for j in indices], device=device), reduction="none").cpu().tolist()))  # pyright: ignore[reportUnknownMemberType] — torch typing boundary
+    return math.exp(min(80., sum(losses) / len(losses)))
+
+
+def calibrate_perplexity(repairs: list[dict[str, object]]) -> dict[str, object]:
+    scored = sorted((number(r["perplexity"]), bool(r["exact"])) for r in repairs if r.get("perplexity") is not None)
+    threshold = 0.0
+    accepted = 0
+    # Strict empirical policy: accept no observed incorrect validation repair.
+    # Tied scores are considered together; tiny samples are explicitly reported.
+    for score in sorted({score for score, _ in scored}):
+        group = [(s, ok) for s, ok in scored if s <= score]
+        if not all(ok for _, ok in group):
+            break
+        threshold, accepted = score, len(group)
+    return {"threshold": threshold, "scored": len(scored), "accepted": accepted,
+            "observed_errors_at_threshold": 0, "policy": "largest validation prefix with zero observed exact-match errors; not a statistical guarantee"}
+
+
 @torch.no_grad()
 def predict_edits(model: TabFix, consumer: Consumer, p: Prepared, device: torch.device,
                   enabled: set[str], threshold: float = .9) -> list[Edit]:
@@ -773,7 +900,15 @@ def predict_edits(model: TabFix, consumer: Consumer, p: Prepared, device: torch.
         probs = model.bio.forward(h).reshape(-1, K, 3).float().softmax(-1)
         insert = model.insert.forward(h).float().sigmoid()
     proposed: list[Edit] = []
+    blocked = invalid_cells(p.record.xml)
+    columns = ET.fromstring(p.record.xml).findall("./schema/column")
     for c in p.cells:
+        if (c.row, c.column) in blocked:
+            continue
+        if c.column < len(columns):
+            rules = mapping(xml_json(columns[c.column], "constraints", {}))
+            if rules.get("enum") or rules.get("lexicon"):
+                continue
         tok = tokens_for(c, p.offsets)
         for category, name in enumerate(CATEGORIES):
             if name not in enabled:
@@ -843,25 +978,24 @@ def monitor(model: TabFix, consumer: Consumer, records: list[Record], cfg: RunCo
     partial: list[float] = []
     category_counts: Counter[str] = Counter()
     confusion = [0, 0, 0]
-    pair_losses: list[float] = []
-    for r in monitoring_sample(records, cfg.validation_examples, cfg.seed):
+    residual = [r for r in records if r.metadata.get("neural_view")]
+    general = [r for r in records if not r.metadata.get("neural_view")]
+    monitored = (monitoring_sample(residual, max(1, cfg.validation_examples // 2), cfg.seed)
+                 + monitoring_sample(general, max(1, cfg.validation_examples // 2), cfg.seed)) if residual and general else monitoring_sample(records, cfg.validation_examples, cfg.seed)
+    for r in monitored:
         p = consumer.prepare(r)
-        pairs = conditional_pairs(r)
-        if pairs:
-            with autocast(device):
-                pair_losses.append(float(conditional_pair_loss(model, consumer, pairs[0], device)))
         with autocast(device):
             h = model.hidden(p.ids, device)
             det.append(float(detection_from_hidden(model, p, h)))
             bio = model.bio.forward(h[p.positions]).reshape(-1, K, 3).float().softmax(-1)[:, :, 1:].sum(-1)
             gap = model.insert.forward(h[p.gaps]).float().sigmoid()
         for probs, raw_labels in ((bio, p.labels), (gap, p.gap_labels)):
-            labels = torch.tensor(raw_labels, device=device)
+            labels = torch.tensor(raw_labels, device=device).reshape(-1, K)
             valid, actual, predicted = labels != -100, labels > 0, probs >= .9
             confusion[0] += int((predicted & actual & valid).sum())
             confusion[1] += int((predicted & ~actual & valid).sum())
             confusion[2] += int((~predicted & actual & valid).sum())
-        for e in eligible(r):
+        for e in eligible(r)[:2]:
             category_counts[e.category] += 1
             for all_masked, output in ((True, full), (False, partial)):
                 view = consumer.correction(p, e, random.Random(r.id + str(e)), True, mask_all=all_masked)
@@ -876,10 +1010,8 @@ def monitor(model: TabFix, consumer: Consumer, records: list[Record], cfg: RunCo
             "selection_score": d + cfg.correction_weight * c,
             "examples": len(det), "correction_edits": len(full),
             "correction_category_counts": dict(category_counts),
-            "conditional_pair_loss": sum(pair_losses) / len(pair_losses) if pair_losses else None,
-            "conditional_pairs": len(pair_losses),
             "token_and_gap_tp_fp_fn_at_0.9": confusion,
-            "sample_ids": [r.id for r in monitoring_sample(records, cfg.validation_examples, cfg.seed)]}
+            "sample_ids": [r.id for r in monitored]}
 
 
 @torch.no_grad()
@@ -896,7 +1028,7 @@ def evaluate(model: TabFix, consumer: Consumer, records: list[Record], cfg: RunC
             losses.append(float(detection_loss(model, p, device)))
             h = model.hidden(p.ids, device)
             prob = model.bio.forward(h[p.positions]).reshape(-1, K, 3).float().softmax(-1)[:, :, 1:].sum(-1)
-        labels = torch.tensor(p.labels, device=device)
+        labels = torch.tensor(p.labels, device=device).reshape(-1, K)
         valid = labels != -100
         actual = labels > 0
         for threshold, v in counts.items():
@@ -915,14 +1047,15 @@ def evaluate(model: TabFix, consumer: Consumer, records: list[Record], cfg: RunC
                     e = eligible_edits[0]
                     _, _, _, _, target = consumer.correction(p, e, random.Random(0), False)
                     answer = generate_repair(model, consumer, p, e, device)
-                    repairs.append({"id": r.id, "category": e.category, "target": target, "prediction": answer, "exact": answer == target})
+                    score = conditional_perplexity(model, consumer, p, e, answer, device) if answer is not None else math.inf
+                    repairs.append({"id": r.id, "category": e.category, "target": target, "prediction": answer, "exact": answer == target, "perplexity": score if math.isfinite(score) else None})
             except ValueError as exc:
                 consumer.stats[str(exc)] += 1
     return {"sample_policy": "seeded language x positive/clean round-robin validation sample",
             "validation_examples": len(losses), "detection_loss": sum(losses) / max(1, len(losses)),
             "correction_examples": len(correction_losses), "correction_loss": sum(correction_losses) / max(1, len(correction_losses)),
             "token_category_confusion_tp_fp_fn_tn": counts,
-            "gold_span_repairs": repairs, "prediction_sample": predictions,
+            "gold_cell_repairs": repairs, "perplexity_calibration": calibrate_perplexity(repairs), "prediction_sample": predictions,
             "limitations": "Token diagnostics, not cell/span calibration; limited gold-span repair sample; test untouched."}
 
 
@@ -941,9 +1074,6 @@ def main(cfg: RunConfig) -> None:
             emit("verified", step=manifest["step"], parameters=sum(p.numel() for p in model.parameters()))
             return
         old = read_json(resume_path / "run_config.json")
-        previous_pair_weight = number(old.get("conditional_pair_weight", 0.0))
-        if previous_pair_weight != cfg.conditional_pair_weight and not cfg.allow_objective_change:
-            raise ValueError("Conditional pair weight changed on resume; retain the saved weight or explicitly pass --allow-objective-change for an ablation")
         for name in ("seed", "lr", "max_length", "correction_weight", "batch_size", "tiny", "warmup_steps", "schedule_steps", "validation_examples", "patience", "min_delta"):
             if old.get(name) != asdict(cfg)[name]:
                 raise ValueError(f"Resume config mismatch: {name}")
@@ -965,25 +1095,23 @@ def main(cfg: RunConfig) -> None:
         torch.set_rng_state(cast(Tensor, state["torch_rng"]))
         if device.type == "cuda":
             torch.cuda.set_rng_state_all(cast(list[Tensor], state["cuda_rng"]))
-    train = diversify_arithmetic(read_records("train", cfg))
+    train = read_records("train", cfg)
     validation = read_records("validation", cfg)
     weights = [number(r.metadata["sampling_weight"]) for r in train]
     repair_indices = [i for i, r in enumerate(train) if eligible(r)]
-    repair_weights = [weights[i] for i in repair_indices]
+    repair_only_indices = [i for i in repair_indices if any(e.category != "copy" for e in eligible(train[i]))]
+    copy_indices = [i for i in repair_indices if any(e.category == "copy" for e in eligible(train[i]))]
+    neural_indices = [i for i, r in enumerate(train) if r.metadata.get("neural_view")]
+    if not neural_indices:
+        neural_indices = list(range(len(train)))  # Tiny local smoke fixtures.
+    neural_weights = [weights[i] for i in neural_indices]
     if cfg.batch_size < 1:
         raise ValueError("Batch size must be positive")
-    if cfg.conditional_pair_weight < 0:
-        raise ValueError("Conditional pair weight must be nonnegative")
-    pair_pool = [pair for r in train for pair in conditional_pairs(r)] if cfg.conditional_pair_weight else []
-    # Deduplicate identical paired contexts so repeated negative records do not increase exposure.
-    pair_pool = list({(p.negative.xml, p.positive.xml, p.row, p.column): p for p in pair_pool}.values())
-    if cfg.conditional_pair_weight and not pair_pool:
-        raise ValueError("No certified conditional pairs available in training split")
     api = HfApi() if cfg.repo_id else None
     if api:
         # A fresh objective intentionally replaces the existing preview repo;
         # the upload commits remain atomic and the local checkpoint is recoverable.
-        _ = api.create_repo(cfg.repo_id, private=False, exist_ok=True)
+        _ = api.repo_info(cfg.repo_id)
     @lru_cache(maxsize=64)
     def prepared(i: int) -> Prepared:
         return consumer.prepare(train[i])
@@ -993,7 +1121,7 @@ def main(cfg: RunConfig) -> None:
         validation_history.extend(entry for item in sequence(prior_report.get("validation_history", [])) if integer((entry := mapping(item))["step"]) <= step)
     report: dict[str, object] = {"tiny_smoke": cfg.tiny, "dataset_revision": REVISION,
         "train_records": len(train), "correction_eligible_records": len(repair_indices),
-        "correction_request_version": 3, "batch_size": cfg.batch_size, "conditional_pair_weight": cfg.conditional_pair_weight, "conditional_pairs": len(pair_pool),
+        "correction_request_version": 4, "batch_size": cfg.batch_size,
         "start_step": step, "device": str(device), "parameters": sum(p.numel() for p in model.parameters()),
         "validation_history": validation_history}
     emit("ready", **report)
@@ -1008,7 +1136,7 @@ def main(cfg: RunConfig) -> None:
         selection = Selection(number(state_selection["best_score"]), integer(state_selection["best_step"]), integer(state_selection["bad_checks"]))
         if not best_path.exists():
             if cfg.repo_id:
-                source_best = Path(snapshot_download(cfg.repo_id, revision=f"best-step-{selection.best_step}"))
+                source_best = Path(snapshot_download(cfg.repo_id, revision=f"v2-best-step-{selection.best_step}"))
             elif selection.best_step == step:
                 source_best = resume_path
             else:
@@ -1031,7 +1159,7 @@ def main(cfg: RunConfig) -> None:
             shutil.copytree(current, best_path, dirs_exist_ok=True)
             if api:
                 uploaded = read_json(Path(cfg.output) / "last_upload.json")
-                api.create_tag(repo_id=cfg.repo_id, tag=f"best-step-{step}", revision=string(uploaded["commit"]), exist_ok=True)
+                api.create_tag(repo_id=cfg.repo_id, tag=f"v2-best-step-{step}", revision=string(uploaded["commit"]), exist_ok=True)
         last_validation_at = saved_at = time.monotonic()
 
     if not resume_path:
@@ -1044,7 +1172,7 @@ def main(cfg: RunConfig) -> None:
         optimizer.zero_grad(set_to_none=True)
         for group in optimizer.param_groups:
             group["lr"] = learning_rate(cfg, step)
-        batch = [prepared(rng.choices(range(len(train)), weights=weights, k=1)[0]) for _ in range(cfg.batch_size)]
+        batch = [prepared(rng.choices(neural_indices, weights=neural_weights, k=1)[0]) for _ in range(cfg.batch_size)]
         with autocast(device):
             hidden = model.hidden_batch([p.ids for p in batch], device)
             det = torch.stack([detection_from_hidden(model, p, hidden[i]) for i, p in enumerate(batch)]).mean()
@@ -1053,9 +1181,12 @@ def main(cfg: RunConfig) -> None:
         det.backward()  # pyright: ignore[reportUnknownMemberType] — third-party typing boundary
         views: list[tuple[list[int], list[int], list[int], list[bool], str]] = []
         for _ in range(cfg.batch_size * 20):
-            rp = prepared(rng.choices(repair_indices, weights=repair_weights, k=1)[0])
+            copy_task = bool(copy_indices) and (not repair_only_indices or rng.random() < .3)
+            pool = copy_indices if copy_task else repair_only_indices
+            rp = prepared(rng.choice(pool))
+            choices = [e for e in eligible(rp.record) if (e.category == "copy") == copy_task]
             try:
-                views.append(consumer.correction(rp, rng.choice(eligible(rp.record)), rng, True))
+                views.append(consumer.correction(rp, rng.choice(choices), rng, True))
             except ValueError as exc:
                 consumer.stats[str(exc)] += 1
             if len(views) == cfg.batch_size:
@@ -1070,15 +1201,6 @@ def main(cfg: RunConfig) -> None:
             raise FloatingPointError("Nonfinite correction loss")
         cor.backward()  # pyright: ignore[reportUnknownMemberType] — third-party typing boundary
         cor_value = float(unweighted_cor.detach())
-        pair_value: float | None = None
-        if pair_pool:
-            with autocast(device):
-                auxiliary = conditional_pair_loss(model, consumer, rng.choice(pair_pool), device)
-            if not bool(torch.isfinite(auxiliary)):
-                raise FloatingPointError("Nonfinite conditional pair loss")
-            (cfg.conditional_pair_weight * auxiliary).backward()  # pyright: ignore[reportUnknownMemberType] — third-party typing boundary
-            pair_value = float(auxiliary.detach())
-        report["last_conditional_pair_loss"] = pair_value
         _ = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0, error_if_nonfinite=True)
         optimizer.step()  # pyright: ignore[reportUnknownMemberType] — third-party typing boundary
         step += 1
@@ -1111,7 +1233,7 @@ def main(cfg: RunConfig) -> None:
             check_validation()
         if api:
             last_upload = read_json(Path(cfg.output) / "last_upload.json")
-            api.create_tag(repo_id=cfg.repo_id, tag=f"latest-step-{step}", revision=string(last_upload["commit"]), exist_ok=True)
+            api.create_tag(repo_id=cfg.repo_id, tag=f"v2-latest-step-{step}", revision=string(last_upload["commit"]), exist_ok=True)
         report["last_training_step"] = step
         report["selected_step"] = selection.best_step
         report["selection_at_stop"] = asdict(selection)
@@ -1153,7 +1275,7 @@ def parse_args() -> RunConfig:
         save_seconds=integer(args["save_seconds"]), lr=number(args["lr"]), seed=integer(args["seed"]),
         validation_examples=integer(args["validation_examples"]), validation_repairs=integer(args["validation_repairs"]), validation_seconds=integer(args["validation_seconds"]),
         validation_steps=integer(args["validation_steps"]), patience=integer(args["patience"]), min_delta=number(args["min_delta"]), warmup_steps=integer(args["warmup_steps"]), schedule_steps=integer(args["schedule_steps"]),
-        max_length=integer(args["max_length"]), correction_weight=number(args["correction_weight"]), batch_size=integer(args["batch_size"]), conditional_pair_weight=number(args["conditional_pair_weight"]), allow_objective_change=bool(args["allow_objective_change"]), verify_only=bool(args["verify_only"]))
+        max_length=integer(args["max_length"]), correction_weight=number(args["correction_weight"]), batch_size=integer(args["batch_size"]), verify_only=bool(args["verify_only"]))
 
 
 def handle_signal(_signum: int, _frame: object) -> None:

training_state.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6e70e38863b991a90700c7dcdc4db2d774332fdc48dec046d6968ab86d898b02
-size 2462881846
+oid sha256:cd9e1448a8ac294f6f6d25dc2d5a1fc00faf3ff28d70475cf1942077b0ca5aae
+size 11293