README.md

---
license: apache-2.0
base_model: convaiinnovations/laya
library_name: pytorch
pipeline_tag: text-classification
tags:
- calibration
- decision-making
- typed-decisions
- non-autoregressive
- modernbert
- proper-scoring-rules
datasets:
- LocalLLaMA/typed-decisions
language:
- en
metrics:
- accuracy
- brier_score
- nll
model-index:
- name: sev
  results:
  - task:
      type: text-classification
      name: Typed decisions
    dataset:
      type: LocalLLaMA/typed-decisions
      name: LocalLLaMA/typed-decisions
      split: test
    metrics:
    - type: accuracy
      value: 0.7885
      name: Accuracy
    - type: brier_score
      value: 0.0495
      name: Brier (vs soft targets)
    - type: nll
      value: 0.8581
      name: NLL (vs soft targets)
---

# Sev — typed-decisions, CE-only, 1024/256

A ModernBERT-large decision model fine-tuned on `LocalLLaMA/typed-decisions` with **plain cross-entropy** against the teacher's soft targets, at the documented 1024-token context / 256-token option budget.

It is the best-performing checkpoint from the **Sev** study (*Dissecting RLCD*), where I took apart the training method behind TypeSafe's Jev and its open reproduction, Laya. The short version of that study: Laya's RL term is a noise-smoothed cross-entropy gradient, and on this benchmark plain CE matches or beats it on every proper score.

**Test-set results (2,000 decisions, 400 cases):**

| Metric | Value |
|---|---|
| Accuracy (vs hard label) | **0.7885** |
| Brier (vs soft targets) | **0.0495** |
| NLL (vs soft targets) | **0.8581** |
| Soft accuracy | 0.5236 |
| Mean confidence | 0.6444 |
| Mean target max | 0.6589 |
| Score MAE (ordinal questions) | 0.2115 |
| Fitted temperature (choice / score / noul) | 1.116 / 1.070 / 1.120 |

For reference, the `laya-typed-decisions` checkpoint reports 0.766 accuracy and 0.062 Brier; TypeSafe's published Jev 1.13.0 figure is 0.727.

## Model description

A bidirectional encoder plus a small transformer head that reads one logit per masked option marker:

```
[CLS] <type> question: <instr> [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] <state> [SEP]
                                                        |
                                  ModernBERT-large encoder (bidirectional)
                                                        |
                                  h += type_emb(qtype)
                                                        |
                                  2 x TransformerEncoderLayer (pre-norm)
                                                        |
                                  gather h at [MASK] positions -> scorer MLP -> 1 logit/option
```

One input row per typed question. Question types are `choice` (pick one of K named options), `score` (ordinal), and `noul` (binary yes/no). Option count is per-row, not fixed.

- Encoder: `answerdotai/ModernBERT-large` (395M)
- Head: 2 pre-norm transformer layers, dropout 0.1
- Scorer: LayerNorm, Linear, GELU, Linear(->1)
- Total: ~421M parameters
- Sequence budget: 1024 tokens total, 256 for the question + option block

The output is a probability distribution over the row's options. For the `score` type, the expected level is `sum i * p_i`.

## Intended use

Structured, closed-set probabilistic decisions over short English text: routing, triage, classification with calibrated confidence, and ordinal rating tasks where you control the option set.

**Out of scope:** open-ended generation (this model does not generate text), option sets larger than ~255, languages other than English (the base is English; a multilingual Laya variant exists), and any use where the option wording and the state distribution differ sharply from the training workflows.

## How to use

This is a custom architecture, not a `transformers` `AutoModel`. Load it with the code from the reproduction repo:

```bash
git clone https://github.com/LakoreAI/sev
cd rlcd-reverse-engineering && uv sync
```

```python
from pathlib import Path
import torch
from huggingface_hub import snapshot_download

import sys
sys.path.insert(0, "rlcd-reverse-engineering")  # or pip install -e it

from src.pipelines.infer import load_model, infer

d = snapshot_download("LakoreAI/sev")
model, cfg, tokenizer = load_model(Path(d) / "model.safetensors", torch.device("cuda"))

result = infer(
    Path(d) / "model.safetensors",
    state='{"task": "triage", "ticket": "customer cannot log in after password reset"}',
    question='{"type": "choice", "instructions": "Which queue should this go to?",'
             ' "criteria": {"billing": "payment or invoice", "auth": "login or account access",'
             ' "network": "connectivity"}}',
)
print(result["predicted_key"], result["probabilities"])
```

`load_model` reads `model_config.json` to rebuild the architecture and `temperatures.json` to apply the fitted per-(type, K-bucket) temperature at inference, so the reported probabilities are already calibration-scaled.

## Training data

`LocalLLaMA/typed-decisions` (Apache-2.0): 1,200 train cases and 400 test cases across four workflows (agent-trace observability, customer service, invoice processing, security incidents), flattened to ~5,400 training decisions. Targets are the dataset's soft teacher distributions, not the hard labels.

## Training procedure

- Initialised from `convaiinnovations/laya`.
- Loss: soft cross-entropy against the teacher distributions only (no RL term).
- 4 epochs, effective batch 64 (micro-batch 8 x 4 accumulation on a single 24 GB GPU), AdamW, lr 2.5e-5 encoder / 1e-4 head, weight decay 0.01, cosine decay to 1e-6, gradient clipping 1.0, bf16.
- No early stopping; the final-epoch weights are evaluated.
- Post-hoc temperature fitted per (question type, option-count bucket) by NLL on a held-out 10% slice of the training cases.

## Evaluation

Full test-set breakdown by question type (2,000 decisions):

| Type | n | Accuracy | Brier | NLL | Mean conf. | Mean target max |
|---|---|---|---|---|---|---|
| choice | 600 | 0.7550 | 0.0611 | 0.9818 | 0.6194 | 0.6345 |
| score | 800 | 0.7588 | 0.0511 | 1.0331 | 0.5817 | 0.5929 |
| noul | 600 | 0.8617 | 0.0356 | 0.5009 | 0.7532 | 0.7714 |
| overall | 2000 | 0.7885 | 0.0495 | 0.8581 | 0.6444 | 0.6589 |

**A calibration caveat that matters here.** On this benchmark the gold hard label is the teacher's argmax on 98.4% of rows, so hard-label ECE is not a valid calibration measure: a model that reproduces the teacher perfectly still scores 0.326 hard-label ECE, and over-sharpening *lowers* it. The raw hard-label ECE of this checkpoint is 0.144 (0.162 after temperature). Judge calibration against the soft targets instead, with Brier and NLL above. The fitted temperatures (all > 1) are the model's own signal that the raw logits were slightly over-sharp before scaling.

## Relation to RLCD

This checkpoint is deliberately RL-free. In the accompanying study, adding Laya's RL term (a score-function estimate of a noise-smoothed proper score) did not improve accuracy and made the fitted temperature rise with the training noise scale. CE-only was at least as good on every proper score. See the linked repo for the full ablation, the estimator derivation, and the per-run result files.

## Limitations

- Single benchmark; the four workflows are synthetic and teacher-generated.
- English only.
- Option sets above ~20 options degrade, because options share a fixed token budget.
- Accuracy differences of ~1 point are within seed noise on a dataset this size; three seeds gave 0.782 +/- 0.004 for the 512-token variant.
- The base checkpoint was already fine-tuned by the Laya authors; this model is a further fine-tune on the same benchmark's train split.

## Links

- Code, configs, and per-run results: [LakoreAI/sev](https://github.com/LakoreAI/sev)
- All ablation checkpoints and metrics: [minhleduc/rlcd-e2-checkpoints](https://huggingface.co/minhleduc/rlcd-e2-checkpoints)
- Base model: [convaiinnovations/laya](https://huggingface.co/convaiinnovations/laya)
- Dataset: [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)

## Citation

```bibtex
@misc{leduc2026dissectingrlcd,
  title  = {Dissecting RLCD: What Reinforcement Learning Does (and Doesn't) Do for Calibrated Typed Decisions},
  author = {Le Duc Minh},
  year   = {2026},
  url    = {https://github.com/LakoreAI/sev}
}
```