Mirror laya-typed-decisions-ce-1024 (CE-only, 1024/256) + model card

README.md

@@ -0,0 +1,183 @@
+---
+license: apache-2.0
+base_model: convaiinnovations/laya
+library_name: pytorch
+pipeline_tag: text-classification
+tags:
+- calibration
+- decision-making
+- typed-decisions
+- non-autoregressive
+- modernbert
+- proper-scoring-rules
+datasets:
+- LocalLLaMA/typed-decisions
+language:
+- en
+metrics:
+- accuracy
+- brier_score
+- nll
+model-index:
+- name: laya-typed-decisions-ce-1024
+  results:
+  - task:
+      type: text-classification
+      name: Typed decisions
+    dataset:
+      type: LocalLLaMA/typed-decisions
+      name: LocalLLaMA/typed-decisions
+      split: test
+    metrics:
+    - type: accuracy
+      value: 0.7885
+      name: Accuracy
+    - type: brier_score
+      value: 0.0495
+      name: Brier (vs soft targets)
+    - type: nll
+      value: 0.8581
+      name: NLL (vs soft targets)
+---
+
+# laya-typed-decisions-ce-1024
+
+A ModernBERT-large decision model fine-tuned on `LocalLLaMA/typed-decisions` with **plain cross-entropy** against the teacher's soft targets, at the documented 1024-token context / 256-token option budget.
+
+It is the best-performing checkpoint from the reproduction study *Dissecting RLCD*, where I took apart the training method behind TypeSafe's Jev and its open reproduction, Laya. The short version of that study: Laya's RL term is a noise-smoothed cross-entropy gradient, and on this benchmark plain CE matches or beats it on every proper score.
+
+**Test-set results (2,000 decisions, 400 cases):**
+
+| Metric | Value |
+|---|---|
+| Accuracy (vs hard label) | **0.7885** |
+| Brier (vs soft targets) | **0.0495** |
+| NLL (vs soft targets) | **0.8581** |
+| Soft accuracy | 0.5236 |
+| Mean confidence | 0.6444 |
+| Mean target max | 0.6589 |
+| Score MAE (ordinal questions) | 0.2115 |
+| Fitted temperature (choice / score / noul) | 1.116 / 1.070 / 1.120 |
+
+For reference, the `laya-typed-decisions` checkpoint reports 0.766 accuracy and 0.062 Brier; TypeSafe's published Jev 1.13.0 figure is 0.727.
+
+## Model description
+
+A bidirectional encoder plus a small transformer head that reads one logit per masked option marker:
+
+```
+[CLS] <type> question: <instr> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP]
+                                                        |
+                                  ModernBERT-large encoder (bidirectional)
+                                                        |
+                                  h += type_emb(qtype)
+                                                        |
+                                  2 x TransformerEncoderLayer (pre-norm)
+                                                        |
+                                  gather h at [MASK] positions -> scorer MLP -> 1 logit/option
+```
+
+One input row per typed question. Question types are `choice` (pick one of K named options), `score` (ordinal), and `noul` (binary yes/no). Option count is per-row, not fixed.
+
+- Encoder: `answerdotai/ModernBERT-large` (395M)
+- Head: 2 pre-norm transformer layers, dropout 0.1
+- Scorer: LayerNorm, Linear, GELU, Linear(->1)
+- Total: ~421M parameters
+- Sequence budget: 1024 tokens total, 256 for the question + option block
+
+The output is a probability distribution over the row's options. For the `score` type, the expected level is `sum i * p_i`.
+
+## Intended use
+
+Structured, closed-set probabilistic decisions over short English text: routing, triage, classification with calibrated confidence, and ordinal rating tasks where you control the option set.
+
+**Out of scope:** open-ended generation (this model does not generate text), option sets larger than ~255, languages other than English (the base is English; a multilingual Laya variant exists), and any use where the option wording and the state distribution differ sharply from the training workflows.
+
+## How to use
+
+This is a custom architecture, not a `transformers` `AutoModel`. Load it with the code from the reproduction repo:
+
+```bash
+git clone https://github.com/LakoreAI/rlcd-reverse-engineering
+cd rlcd-reverse-engineering && uv sync
+```
+
+```python
+from pathlib import Path
+import torch
+from huggingface_hub import snapshot_download
+
+import sys
+sys.path.insert(0, "rlcd-reverse-engineering")  # or pip install -e it
+
+from src.pipelines.infer import load_model, infer
+
+d = snapshot_download("minhleduc/laya-typed-decisions-ce-1024")
+model, cfg, tokenizer = load_model(Path(d) / "model.safetensors", torch.device("cuda"))
+
+result = infer(
+    Path(d) / "model.safetensors",
+    state='{"task": "triage", "ticket": "customer cannot log in after password reset"}',
+    question='{"type": "choice", "instructions": "Which queue should this go to?",'
+             ' "criteria": {"billing": "payment or invoice", "auth": "login or account access",'
+             ' "network": "connectivity"}}',
+)
+print(result["predicted_key"], result["probabilities"])
+```
+
+`load_model` reads `model_config.json` to rebuild the architecture and `temperatures.json` to apply the fitted per-(type, K-bucket) temperature at inference, so the reported probabilities are already calibration-scaled.
+
+## Training data
+
+`LocalLLaMA/typed-decisions` (Apache-2.0): 1,200 train cases and 400 test cases across four workflows (agent-trace observability, customer service, invoice processing, security incidents), flattened to ~5,400 training decisions. Targets are the dataset's soft teacher distributions, not the hard labels.
+
+## Training procedure
+
+- Initialised from `convaiinnovations/laya`.
+- Loss: soft cross-entropy against the teacher distributions only (no RL term).
+- 4 epochs, effective batch 64 (micro-batch 8 x 4 accumulation on a single 24 GB GPU), AdamW, lr 2.5e-5 encoder / 1e-4 head, weight decay 0.01, cosine decay to 1e-6, gradient clipping 1.0, bf16.
+- No early stopping; the final-epoch weights are evaluated.
+- Post-hoc temperature fitted per (question type, option-count bucket) by NLL on a held-out 10% slice of the training cases.
+
+## Evaluation
+
+Full test-set breakdown by question type (2,000 decisions):
+
+| Type | n | Accuracy | Brier | NLL | Mean conf. | Mean target max |
+|---|---|---|---|---|---|---|
+| choice | 600 | 0.7550 | 0.0611 | 0.9818 | 0.6194 | 0.6345 |
+| score | 800 | 0.7588 | 0.0511 | 1.0331 | 0.5817 | 0.5929 |
+| noul | 600 | 0.8617 | 0.0356 | 0.5009 | 0.7532 | 0.7714 |
+| overall | 2000 | 0.7885 | 0.0495 | 0.8581 | 0.6444 | 0.6589 |
+
+**A calibration caveat that matters here.** On this benchmark the gold hard label is the teacher's argmax on 98.4% of rows, so hard-label ECE is not a valid calibration measure: a model that reproduces the teacher perfectly still scores 0.326 hard-label ECE, and over-sharpening *lowers* it. The raw hard-label ECE of this checkpoint is 0.144 (0.162 after temperature). Judge calibration against the soft targets instead, with Brier and NLL above. The fitted temperatures (all > 1) are the model's own signal that the raw logits were slightly over-sharp before scaling.
+
+## Relation to RLCD
+
+This checkpoint is deliberately RL-free. In the accompanying study, adding Laya's RL term (a score-function estimate of a noise-smoothed proper score) did not improve accuracy and made the fitted temperature rise with the training noise scale. CE-only was at least as good on every proper score. See the linked repo for the full ablation, the estimator derivation, and the per-run result files.
+
+## Limitations
+
+- Single benchmark; the four workflows are synthetic and teacher-generated.
+- English only.
+- Option sets above ~20 options degrade, because options share a fixed token budget.
+- Accuracy differences of ~1 point are within seed noise on a dataset this size; three seeds gave 0.782 +/- 0.004 for the 512-token variant.
+- The base checkpoint was already fine-tuned by the Laya authors; this model is a further fine-tune on the same benchmark's train split.
+
+## Links
+
+- Code, configs, and per-run results: [LakoreAI/rlcd-reverse-engineering](https://github.com/LakoreAI/rlcd-reverse-engineering)
+- All ablation checkpoints and metrics: [minhleduc/rlcd-e2-checkpoints](https://huggingface.co/minhleduc/rlcd-e2-checkpoints)
+- Base model: [convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya)
+- Dataset: [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)
+
+## Citation
+
+```bibtex
+@misc{leduc2026dissectingrlcd,
+  title  = {Dissecting RLCD: What Reinforcement Learning Does (and Doesn't) Do for Calibrated Typed Decisions},
+  author = {Le Duc Minh},
+  year   = {2026},
+  url    = {https://github.com/LakoreAI/rlcd-reverse-engineering}
+}
+```

config.json

@@ -0,0 +1,78 @@
+{
+  "model": {
+    "encoder_name": "answerdotai/ModernBERT-large",
+    "head_layers": 2,
+    "head_dropout": 0.1,
+    "num_qtypes": 3,
+    "cls_query": false,
+    "slot_emb": false,
+    "max_len": 1024,
+    "head_max_len": 256,
+    "k_buckets": [
+      2,
+      5,
+      10
+    ]
+  },
+  "training": {
+    "dataset_name": "LocalLLaMA/typed-decisions",
+    "dataset_config": "all",
+    "calib_fraction": 0.1,
+    "seed": 42,
+    "max_examples": null,
+    "augment_permute": false,
+    "ckpt_dir": "checkpoints",
+    "result_dir": "results",
+    "run_name": "e2_ce_only_1024_seed42",
+    "epochs": 4,
+    "batch_size": 4,
+    "accum_steps": 16,
+    "num_workers": 4,
+    "pin_memory": true,
+    "amp": true,
+    "amp_dtype": "bf16",
+    "gradient_checkpointing": true,
+    "lr": 2.5e-05,
+    "lr_head": 0.0001,
+    "weight_decay": 0.01,
+    "grad_clip": 1.0,
+    "log_every": 10,
+    "eval_every": 1,
+    "ckpt_every": 4,
+    "w_rl": 0.0,
+    "w_ce": 1.0,
+    "num_noise_samples": 4,
+    "sigma_start": 0.4,
+    "sigma_end": 0.1,
+    "anneal_sigma": true,
+    "reward_w_spherical": 0.75,
+    "reward_w_rps": 1.0,
+    "arch": {
+      "encoder_name": "answerdotai/ModernBERT-large",
+      "head_layers": 2,
+      "max_len": 1024,
+      "head_max_len": 256
+    },
+    "init_from": "convaiinnovations/laya",
+    "log_grad_diagnostics": true,
+    "lr_scheduler": {
+      "type": "cosine",
+      "t_max": "auto",
+      "eta_min": 1e-06
+    },
+    "early_stopping": {
+      "enabled": false
+    },
+    "save_best": false,
+    "best_metric": "raw_ece",
+    "best_mode": "min",
+    "wandb": {
+      "enabled": true,
+      "project": "rlcd-reverse-engineering",
+      "entity": null,
+      "group": "e2-acc",
+      "log_artifacts": false
+    },
+    "resume_from": null
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be8aae6b150491cfe3d8049d69fce35dc06dfe4aac1db1d71a321b19432a9f9f
+size 1684141044

model_config.json

@@ -0,0 +1,18 @@
+{
+  "encoder_name": "answerdotai/ModernBERT-large",
+  "head_layers": 2,
+  "head_dropout": 0.1,
+  "num_qtypes": 3,
+  "cls_query": false,
+  "slot_emb": false,
+  "max_len": 1024,
+  "head_max_len": 256,
+  "k_buckets": [
+    2,
+    5,
+    10
+  ],
+  "source_checkpoint": "epoch_4.pt",
+  "epoch": 4,
+  "step": 340
+}

temperatures.json

@@ -0,0 +1,24 @@
+{
+  "k_buckets": [
+    2,
+    5,
+    10
+  ],
+  "temperatures": [
+    {
+      "qtype": 0,
+      "bucket": 1,
+      "T": 1.116391658782959
+    },
+    {
+      "qtype": 1,
+      "bucket": 1,
+      "T": 1.0700936317443848
+    },
+    {
+      "qtype": 2,
+      "bucket": 0,
+      "T": 1.1198104619979858
+    }
+  ]
+}

test_eval.json

@@ -0,0 +1,99 @@
+{
+  "raw": {
+    "ece": 0.14405169022083278,
+    "brier": 0.049473914223986594,
+    "nll": 0.8580627283966169,
+    "accuracy": 0.7885,
+    "soft_accuracy": 0.5236424660123885,
+    "mean_confidence": 0.6444483368843794,
+    "mean_target_max": 0.6589462240487337,
+    "score_mae": 0.21151374208275228,
+    "n": 2000,
+    "by_type": {
+      "choice": {
+        "ece": 0.1381368541220824,
+        "brier": 0.0610927829263998,
+        "nll": 0.9818144658269982,
+        "accuracy": 0.755,
+        "soft_accuracy": 0.48078609759608903,
+        "mean_confidence": 0.6193696513772011,
+        "mean_target_max": 0.6345381752649943,
+        "score_mae": null,
+        "n": 600
+      },
+      "score": {
+        "ece": 0.1779335317760706,
+        "brier": 0.05114634511056351,
+        "nll": 1.03313795868773,
+        "accuracy": 0.75875,
+        "soft_accuracy": 0.4435949847009033,
+        "mean_confidence": 0.5816603348776698,
+        "mean_target_max": 0.5928848401084542,
+        "score_mae": 0.21151374208275228,
+        "n": 800
+      },
+      "noul": {
+        "ece": 0.10842230906089145,
+        "brier": 0.03562513767280417,
+        "nll": 0.5008773505780846,
+        "accuracy": 0.8616666666666667,
+        "soft_accuracy": 0.6732288095106681,
+        "mean_confidence": 0.7532443584005037,
+        "mean_target_max": 0.7714361180861791,
+        "score_mae": null,
+        "n": 600
+      }
+    }
+  },
+  "post_temperature": {
+    "ece": 0.16220938038825988,
+    "brier": 0.04794188590972476,
+    "nll": 0.8542706172754988,
+    "accuracy": 0.7885,
+    "soft_accuracy": 0.5147140698879957,
+    "mean_confidence": 0.6262906277328729,
+    "mean_target_max": 0.6589462240487337,
+    "score_mae": 0.21261174298007973,
+    "n": 2000,
+    "by_type": {
+      "choice": {
+        "ece": 0.16171381801366805,
+        "brier": 0.058598528968896064,
+        "nll": 0.9764881309494376,
+        "accuracy": 0.755,
+        "soft_accuracy": 0.46792497878273326,
+        "mean_confidence": 0.5932861614724,
+        "mean_target_max": 0.6345381752649943,
+        "score_mae": null,
+        "n": 600
+      },
+      "score": {
+        "ece": 0.19320389255881307,
+        "brier": 0.04962007356784852,
+        "nll": 1.0289345176471398,
+        "accuracy": 0.75875,
+        "soft_accuracy": 0.4367176624760032,
+        "mean_confidence": 0.5665761560201645,
+        "mean_target_max": 0.5928848401084542,
+        "score_mae": 0.21261174298007973,
+        "n": 800
+      },
+      "noul": {
+        "ece": 0.12275225609540938,
+        "brier": 0.035047659306388444,
+        "nll": 0.49916790310603876,
+        "accuracy": 0.8616666666666667,
+        "soft_accuracy": 0.6654983708759149,
+        "mean_confidence": 0.7389143896102905,
+        "mean_target_max": 0.7714361180861791,
+        "score_mae": null,
+        "n": 600
+      }
+    }
+  },
+  "fitted_temperature": {
+    "type0_bucket1": 1.116391658782959,
+    "type2_bucket0": 1.1198104619979858,
+    "type1_bucket1": 1.0700936317443848
+  }
+}

train_log.json

@@ -0,0 +1,70 @@
+[
+  {
+    "epoch": 1,
+    "step": 85,
+    "loss": 0.9641271437353948,
+    "sigma": 0.4,
+    "epoch_seconds": 228.8010711669922,
+    "train_rows": 5400,
+    "raw_ece": 0.10835536271333696,
+    "raw_brier": 0.07556118707742442,
+    "raw_nll": 0.8722028755644957,
+    "raw_accuracy": 0.7066666666666667,
+    "raw_soft_accuracy": 0.4981412665049235,
+    "raw_mean_confidence": 0.5995895755290985,
+    "raw_mean_target_max": 0.6725333379705747,
+    "raw_score_mae": 0.29329004988539964,
+    "raw_n": 600
+  },
+  {
+    "epoch": 2,
+    "step": 170,
+    "loss": 0.8357182100488275,
+    "sigma": 0.30000000000000004,
+    "epoch_seconds": 224.5660684108734,
+    "train_rows": 5400,
+    "raw_ece": 0.16046804825464883,
+    "raw_brier": 0.05988674723912672,
+    "raw_nll": 0.8413767837701986,
+    "raw_accuracy": 0.7716666666666666,
+    "raw_soft_accuracy": 0.5137750238180161,
+    "raw_mean_confidence": 0.615313962996006,
+    "raw_mean_target_max": 0.6725333379705747,
+    "raw_score_mae": 0.24656671059007446,
+    "raw_n": 600
+  },
+  {
+    "epoch": 3,
+    "step": 255,
+    "loss": 0.8000329575373554,
+    "sigma": 0.2,
+    "epoch_seconds": 223.96768260002136,
+    "train_rows": 5400,
+    "raw_ece": 0.1488726359605789,
+    "raw_brier": 0.05451012486705442,
+    "raw_nll": 0.8328132717187205,
+    "raw_accuracy": 0.78,
+    "raw_soft_accuracy": 0.5247553873838235,
+    "raw_mean_confidence": 0.6352111848195394,
+    "raw_mean_target_max": 0.6725333379705747,
+    "raw_score_mae": 0.22168685695699727,
+    "raw_n": 600
+  },
+  {
+    "epoch": 4,
+    "step": 340,
+    "loss": 0.7846576284726753,
+    "sigma": 0.09999999999999998,
+    "epoch_seconds": 222.44736337661743,
+    "train_rows": 5400,
+    "raw_ece": 0.13711758901675544,
+    "raw_brier": 0.052573612098253716,
+    "raw_nll": 0.8299004402291029,
+    "raw_accuracy": 0.7766666666666666,
+    "raw_soft_accuracy": 0.5293207626231015,
+    "raw_mean_confidence": 0.6415973689158757,
+    "raw_mean_target_max": 0.6725333379705747,
+    "raw_score_mae": 0.21637691374926363,
+    "raw_n": 600
+  }
+]