README.md

---
license: apache-2.0
language:
- en
library_name: transformers
base_model: jhu-clsp/ettin-encoder-150m
pipeline_tag: zero-shot-classification
tags:
- decision-model
- system-one
- falcondec
- lightdec
- calibrated-decisions
- multiple-choice
- intent-classification
- customer-support
- natural-language-inference
- code
- guardrails
- agents
- selective-prediction
- falconsai
- model-surgeon
- attested-lineage
---

> Source model card: `Falconsai/LightDec` @ `main`, carried verbatim below. Its licence is the repository's. The Model Surgeon record follows it.


# Falconsai/LightDec

**[View in Model Surgeon](https://surgeon.falcons.ai/?hub=Falconsai/LightDec)**

**A lightweight, single-pass, typed, calibrated decision model for agentic systems.** Give it a **state** (text, code or JSON), one or more **typed questions** (`choice`, `noul` yes/no, `score` ordinal) and a closed set of options. It returns a calibrated probability for every option, from one encoder pass per question.

LightDec is the FalconDec architecture trained with **FalconDec notebook V2** on the **`standard`** data preset. That run adds agent-specific decisions (AgentTrek, Counsel, HotpotQA) to a balanced mix of 58 test tasks across 9 domains. This page is both the **model card** and the **developer guide**: how to load it, test it, and use it as a decision component in an agentic system.

> **At a glance.** Test accuracy **0.725** (micro and task-macro) on 17,498 decisions from 58 tasks, with ECE **0.025**. At a 0.70 confidence threshold, LightDec answers **56%** of decisions at **89.6%** accuracy and defers the rest. Weights: **319 MB** fp16, **161 MB** int8. It is strongest on support routing, code understanding, intents, guardrails and agent-step checks, and weakest on multi-step arithmetic, date and table reasoning, and very wide label sets. Evaluate it on your own traffic (§6.4) before acting on its answers.

## Contents

1. [Model summary](#1-model-summary)
2. [Results](#2-results)
3. [Intended and out-of-scope uses](#3-intended-and-out-of-scope-uses)
4. [The Hub repository](#4-the-hub-repository)
5. [Install, load and read a result](#5-install-load-and-read-a-result)
6. [Testing the model](#6-testing-the-model)
7. [Using it in an agentic system](#7-using-it-in-an-agentic-system)
8. [Tuning for your domain](#8-tuning-for-your-domain)
9. [Architecture](#9-architecture)
10. [Training data](#10-training-data)
11. [Training procedure and calibration](#11-training-procedure-and-calibration)
12. [Operational notes](#12-operational-notes)
13. [Bias, risks and limitations](#13-bias-risks-and-limitations)
14. [Versioning and lineage](#14-versioning-and-lineage)
15. [API reference](#15-api-reference)
16. [Citation and references](#16-citation-and-references)

---

## 1. Model summary

| | |
|---|---|
| **Model** | LightDec: the FalconDec architecture, notebook V2, `standard` preset. The checkpoint's own config reports `FalconDec` version `1.0.0` |
| **Task** | Closed-set decisions: given a state, a question and 2–N options, return a calibrated probability per option |
| **Question types** | `choice` (pick one), `noul` (yes/no, returns P(true)), `score` (ordinal rubric, returns the expected level) |
| **Backbone** | [`jhu-clsp/ettin-encoder-150m`](https://huggingface.co/jhu-clsp/ettin-encoder-150m) (ModernBERT-style encoder), fully fine-tuned |
| **Decision head** | Option-marker scoring plus a permutation-equivariant set-transformer head (§9) |
| **Parameters** | ≈160M (Hub reports 0.2B) |
| **Weights** | fp16 **319 MB** (`model.safetensors`) · per-channel int8 **161 MB** (`compact-int8/model_int8.safetensors`) |
| **Context** | 512 tokens; automatically 2,048 for questions with more than 24 options; a tournament above 96 options |
| **Calibration** | One temperature per (question type × option-count bucket), stored in the checkpoint and applied automatically |
| **Inference cost** | One encoder pass per question. The same architecture (`Falconsai/proof_v3`) measured 15.7 ms p50 for one question on a GPU; LightDec's own latency is not in its report (measure with §6.3) |
| **Output** | Probabilities, the chosen option, confidence, a `defer` flag, `p_true` (noul) and `expected_level` (score) |
| **Language** | English, plus code in Python, Java, JavaScript, PHP, Ruby, Go and C |
| **Custom code** | `falcondec_modeling.py` ships with the weights and holds the model and all inference logic. Load it with `importlib` (§5); `AutoModel.from_pretrained` alone won't build the decision head |
| **License** | Apache-2.0 for the weights and code. Check each training dataset's license before redistributing derived data |

The model has no generative component. It can only rank the options you give it, so it cannot produce text outside that set.

---

## 2. Results

All numbers come from this checkpoint's `falcondec_report.json`: one run, `standard` preset, 2 epochs, seed 42, `MODE="scratch"`, notebook V2.

### 2.1 At a glance

| Metric | Value |
|---|---|
| Test decisions / tasks | 17,498 / 58 (10 of them held out) |
| Test accuracy, micro | **0.725** |
| Test accuracy, task-macro | **0.725** |
| Held-out tasks, task-macro (10 tasks never trained on) | **0.567** |
| Expected calibration error (ECE, 15 bins) | **0.025** |
| Negative log-likelihood / Brier score | 0.652 / 0.358 |
| Area under the risk–coverage curve (AURC, lower is better) | **0.097** |
| Ordinal (`score`) mean absolute error, in levels | 0.572 |
| Coverage / accuracy at confidence ≥ 0.70 | **56.4% / 0.896** |
| Weights | fp16 319 MB · int8 161 MB |

**Selective prediction is the headline.** Calibration is good (ECE 0.025), so the confidence score is a reliable gate. Acting only on decisions with confidence ≥ 0.70 covers 56% of traffic at 89.6% accuracy, against 72.5% accuracy when answering everything. That is the property an agent loop needs: answer the easy majority locally, and hand the rest to an LLM or a human.

### 2.2 Per domain

| Domain | Test decisions | Task-macro accuracy |
|---|---|---|
| support | 600 | **0.997** |
| code | 2,350 | **0.864** |
| intents | 1,200 | **0.773** |
| guardrails | 1,094 | **0.768** |
| agentic | 717 | **0.754** |
| workflows | 2,000 | **0.689** |
| policy | 2,700 | **0.669** |
| reasoning | 5,637 | **0.667** |
| classification | 1,200 | **0.584** |

### 2.3 Per task

*Held-out tasks were never used for training, calibration or model selection. "proof_v2 (card)" lists proof_v2's published score for the same source and task (different samples; indicative only).*

| Domain | Task | n | Chance | **Accuracy** | ECE | proof_v2 (card) |
|---|---|---|---|---|---|---|
| support | `bitext/route` | 300 | 0.200 | **1.000** | 0.002 | 0.958 |
| support | `bitext/category` | 300 | 0.172 | **0.993** | 0.008 |  |
| code | `codexglue/lang_id` | 300 | 0.235 | **1.000** | 0.004 | 0.997 |
| code | `mbpp/solution` | 300 | 0.250 | **0.987** | 0.014 | 0.992 |
| code | `codexglue/code_to_doc` | 300 | 0.256 | **0.977** | 0.016 | 0.969 |
| code | `codexglue/doc_to_code` | 300 | 0.274 | **0.973** | 0.022 | 0.961 |
| code | `codexglue/func_name` | 275 | 0.263 | **0.938** | 0.021 | 0.901 |
| code | `bigclonebench/clone` | 300 | 0.500 | **0.863** | 0.098 | 0.383 |
| code | `humaneval/completion` *(held out)* | 119 | 0.394 | **0.756** | 0.152 | 0.575 |
| code | `mbpp/bugspot` | 156 | 0.413 | **0.731** | 0.086 | 0.475 |
| code | `devign/vulnerability` | 300 | 0.500 | **0.553** | 0.026 | 0.542 |
| intents | `banking77/intent` *(held out)* | 300 | 0.317 | **0.923** | 0.037 | 0.883 |
| intents | `massive_en/intent` | 300 | 0.122 | **0.907** | 0.046 |  |
| intents | `clinc150/intent` | 300 | 0.122 | **0.793** | 0.081 | 0.850 |
| intents | `banking77/intent_77` *(held out)* | 300 | 0.013 | **0.470** | 0.200 |  |
| guardrails | `jailbreak/detect` | 262 | 0.500 | **0.966** | 0.018 |  |
| guardrails | `civil_comments/toxic` | 300 | 0.500 | **0.807** | 0.051 |  |
| guardrails | `agentharm/refuse` *(held out)* | 416 | 0.500 | **0.654** | 0.178 |  |
| guardrails | `prompt_injections/detect` *(held out)* | 116 | 0.500 | **0.647** | 0.272 |  |
| agentic | `hotpotqa/retrieve` | 298 | 0.168 | **0.842** | 0.078 |  |
| agentic | `hotpotqa/comparison_yes_no` | 17 | 0.500 | **0.824** | 0.185 |  |
| agentic | `counsel/step_has_error` | 201 | 0.500 | **0.791** | 0.093 |  |
| agentic | `counsel/critique_quality` | 201 | 0.333 | **0.557** | 0.094 |  |
| workflows | `typed_decisions/customer_service` | 500 | 0.280 | **0.720** | 0.099 |  |
| workflows | `typed_decisions/security_incidents` | 500 | 0.340 | **0.712** | 0.140 |  |
| workflows | `typed_decisions/agent_trace_observability` | 500 | 0.300 | **0.696** | 0.106 |  |
| workflows | `typed_decisions/invoice_processing` | 500 | 0.350 | **0.628** | 0.101 |  |
| policy | `policy/access_control_transfer` | 300 | 0.333 | **1.000** | 0.001 |  |
| policy | `policy/return_window_transfer` | 300 | 0.333 | **1.000** | 0.019 |  |
| policy | `policy/free_shipping_transfer` | 300 | 0.500 | **0.850** | 0.067 |  |
| policy | `policy/sla_urgency_transfer` | 300 | 0.250 | **0.713** | 0.214 |  |
| policy | `policy/refund_approval_transfer` | 300 | 0.333 | **0.710** | 0.219 |  |
| policy | `policy/count_threshold_transfer` | 300 | 0.179 | **0.523** | 0.085 |  |
| policy | `policy/invoice_total_transfer` | 300 | 0.500 | **0.523** | 0.020 |  |
| policy | `policy/invoice_overdue_transfer` | 300 | 0.500 | **0.407** | 0.364 |  |
| policy | `policy/table_extreme_transfer` | 300 | 0.240 | **0.290** | 0.036 |  |
| reasoning | `qasc/mcq` | 300 | 0.125 | **0.983** | 0.007 |  |
| reasoning | `snli/must_be_true` | 300 | 0.333 | **0.970** | 0.029 | 0.908 |
| reasoning | `snli/contradicts` | 300 | 0.333 | **0.967** | 0.035 | 0.892 |
| reasoning | `scitail/support` | 300 | 0.500 | **0.957** | 0.030 |  |
| reasoning | `sciq/mcq` | 300 | 0.250 | **0.950** | 0.023 | 0.692 |
| reasoning | `snli/nli` | 594 | 0.333 | **0.837** | 0.035 |  |
| reasoning | `mnli/claim` | 300 | 0.333 | **0.807** | 0.072 | 0.492 |
| reasoning | `boolq/yes_no` | 300 | 0.500 | **0.783** | 0.071 | 0.717 |
| reasoning | `gsm8k/math` | 300 | 0.250 | **0.637** | 0.050 | 0.275 |
| reasoning | `commonsense_qa/mcq` | 296 | 0.200 | **0.611** | 0.058 | 0.442 |
| reasoning | `arc_easy/mcq` *(held out)* | 300 | 0.250 | **0.553** | 0.057 | 0.425 |
| reasoning | `openbookqa/mcq` | 300 | 0.250 | **0.550** | 0.068 | 0.292 |
| reasoning | `winogrande/blank` | 300 | 0.500 | **0.540** | 0.089 |  |
| reasoning | `arc_challenge/mcq` *(held out)* | 300 | 0.250 | **0.423** | 0.085 | 0.308 |
| reasoning | `anli/nli` | 300 | 0.333 | **0.393** | 0.185 |  |
| reasoning | `mmlu/mcq` *(held out)* | 300 | 0.250 | **0.393** | 0.094 |  |
| reasoning | `hellaswag/continuation` | 300 | 0.250 | **0.383** | 0.171 |  |
| reasoning | `aqua_rat/math` | 247 | 0.200 | **0.259** | 0.071 |  |
| classification | `ag_news/topic` | 300 | 0.250 | **0.847** | 0.051 |  |
| classification | `yelp/score` | 300 | 0.200 | **0.640** | 0.069 |  |
| classification | `emotion/6way` *(held out)* | 300 | 0.167 | **0.480** | 0.049 |  |
| classification | `sst5/score` *(held out)* | 300 | 0.200 | **0.370** | 0.086 |  |

### 2.4 Comparison with proof_v2 (indicative)

On the 22 tasks that both this report and proof_v2's model card cover, LightDec's task-macro accuracy is **0.807 vs 0.679**, and it scores higher on **20 of 22**. The largest gains are on the tasks proof_v2 reported as weak:

| Task | proof_v2 (card) | LightDec |
|---|---|---|
| BigCloneBench clone detection | 0.383 | **0.863** |
| GSM8K (4-option numeric) | 0.275 | **0.637** |
| MultiNLI claim | 0.492 | **0.807** |
| OpenBookQA | 0.292 | **0.550** |
| MBPP bug spotting | 0.475 | **0.731** |
| HumanEval completion *(held out)* | 0.575 | **0.756** |
| ARC-Challenge *(held out)* | 0.308 | **0.423** |

LightDec is lower on CLINC150 (0.793 vs 0.850) and MBPP task→solution (0.987 vs 0.992).

These are **different test samples and, for some tasks, different question formats**. For example, LightDec's MultiNLI task is three-way NLI, and its bug-spotting mutants are verified to fail the unit tests. The like-for-like head-to-head, which runs proof_v2 on identical decisions (notebook cell 23), **did not run** for this checkpoint (`head_to_head: null`).

### 2.5 Comparison with Laya and TypeSafe Jev (indicative)

| Benchmark | LightDec | Laya | TypeSafe Jev 1.13.0 |
|---|---|---|---|
| typed-decisions test (2,000 decisions, 4 workflows) | 0.689 | 0.766 (fine-tuned) · 0.362 (zero-shot) | 0.727 |
| AG News (4 labels) | 0.847 | 0.950 | 0.910 |
| DAIR Emotion, 6 labels *(held out for LightDec)* | 0.480 | 0.595 | 0.480 |
| Banking77, all 77 labels in one question *(held out)* | 0.470 | 0.425 | 0.870 (72 labels) |
| SST-5 (ordinal) *(held out)* | 0.370 | 0.372 | — |

Laya's numbers are from its own benchmark report; Jev's are third-party published. LightDec was trained on the typed-decisions training split, like the fine-tuned Laya checkpoint. **On typed-decisions LightDec trails both** (the teacher-agreement ceiling is 0.735 and the majority-class baseline 0.461). It matches Jev on Emotion, edges Laya on all-77 Banking77, and trails both on AG News. LightDec is 2.6× smaller than Laya's 421M English checkpoint.

---

## 3. Intended and out-of-scope uses

### Intended

| Use | Measured evidence |
|---|---|
| **Support and ticket routing** | Bitext route 1.000, Bitext category 0.993; Banking77 (held out, 2–5 options) 0.923; MASSIVE 0.907 |
| **Code understanding against a menu** | Language ID 1.000; code↔description 0.973–0.977; task→solution 0.987; function naming 0.938 |
| **Guardrails** | Jailbreak detection 0.966; toxicity 0.807. Held out: prompt-injection 0.647, AgentHarm refusal 0.654, so recalibrate and test on your own traffic |
| **Agent loops** | Retrieval routing (HotpotQA) 0.842; "does this agent step contain an error?" (Counsel) 0.791 |
| **Statement verification** | SNLI must-be-true / contradicts 0.970 / 0.967; SciTail 0.957; SciQ 0.950 |
| **Selective automation** | 89.6% accuracy on the 56% of decisions with confidence ≥ 0.70 |

### Out of scope

- **Multi-step arithmetic and quantitative reasoning.** AQuA 0.259 (chance 0.20); counting and summing policies 0.52; comparing values in a table 0.290. GSM8K reaches 0.637 only because it is posed as 4-option multiple choice with near-miss distractors. Route real math to an LLM or code.
- **Date reasoning in unfamiliar formats.** The invoice-overdue transfer test (ISO dates, whereas training used "Month DD, YYYY") scores **0.407, below chance, with ECE 0.364**. It is confidently wrong there. Normalise dates before asking, or compute them in code.
- **Very wide label sets.** All 77 Banking77 intents in one question score 0.470. Pre-filter to a shortlist of about 20 options (§7.1).
- **Hard commonsense and exam knowledge**: HellaSwag 0.383, ANLI 0.393, MMLU 0.393.
- **Code security and correctness gating**: Devign 0.553 is near chance. Don't use it to approve code.
- **Open-ended questions**: it always picks one of your options. Add "None of the above" when appropriate; it was trained with that option.
- **Non-English text**, and **high-stakes decisions without human oversight**.

---

## 4. The Hub repository

| File | Contents |
|---|---|
| `model.safetensors` | fp16 weights (319 MB): encoder, decision head and the temperature buffer |
| `falcondec_config.json` | Layout (sequence lengths, option budgets), special-token ids, temperatures, defer threshold, version, lineage |
| `encoder/` | Backbone configuration (the encoder is rebuilt from this, then the weights are loaded) |
| `tokenizer/` | Tokenizer files |
| `falcondec_modeling.py` | `FalconDec`, `load_falcondec`, `decide`, `score_items`, `save_falcondec` and the int8 codec |
| `falcondec_report.json` | Training configuration, data counts, history, temperatures and all test results |
| `compact-int8/` | The same model with per-channel int8 weights (`model_int8.safetensors`, 161 MB); a complete, self-contained directory with its own config, tokenizer and modeling file |
| `README.md` | This card |

**Pin a revision in production.** The repo can change, so pass a commit hash when you load.

---

## 5. Install, load and read a result

```bash
pip install torch "transformers>=4.48" safetensors huggingface_hub numpy
```

### 5.1 Load

Save this helper as `lightdec.py` next to your code. Every example below uses it.

```python
# lightdec.py
import importlib.util, json, shutil
from pathlib import Path
from huggingface_hub import snapshot_download


def load_lightdec(repo="Falconsai/LightDec", revision=None, variant="fp16", device=None, dtype=None):
    """Returns (fdm, model, tokenizer). variant: "fp16" (319 MB) or "int8" (161 MB, dequantised on load)."""
    path = Path(repo) if Path(repo).exists() else Path(snapshot_download(repo, revision=revision))
    if variant == "int8":
        path = path / "compact-int8"
    fc = json.loads((path / "falcondec_config.json").read_text(encoding="utf-8"))
    expected = fc.get("weights", "model.safetensors")
    if not (path / expected).exists():                  # e.g. a renamed weight file in a processed copy
        cands = sorted(path.glob("*.safetensors"))
        if not cands:
            raise FileNotFoundError(f"no .safetensors weights in {path}")
        local = Path("lightdec_local") / variant
        shutil.copytree(path, local, dirs_exist_ok=True)
        shutil.copy(cands[0], local / expected)
        path = local
    spec = importlib.util.spec_from_file_location("falcondec_modeling", str(path / "falcondec_modeling.py"))
    fdm = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(fdm)
    model, tok = fdm.load_falcondec(str(path), device=device, dtype=dtype)   # cuda if available, else cpu
    return fdm, model, tok
```

```python
from lightdec import load_lightdec

fdm, model, tok = load_lightdec()                      # or load_lightdec(revision="<commit>", variant="int8")
print(model.fcfg["name"], model.fcfg["version"], round(model.num_parameters() / 1e6, 1), "M params")
```

If loading prints `[FalconDec] load warning: missing=… unexpected=…`, the weights didn't match the architecture. Treat that as a failed load (§6.1 checks for it).

### 5.2 Decide

`decide()` takes one state and any number of typed questions: either a list, or a Jev/Laya-style dict keyed by name.

```python
state = {"from": "user@acme.com", "subject": "Duplicate charge on invoice #4411",
         "body": "We were billed twice for March. Please refund the duplicate today or we will cancel our plan."}

out = fdm.decide(model, tok, state, {
    "department": {"type": "choice", "instructions": "Which department should handle this request?",
                   "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",
                                "sales": "pricing, contracts", "other": "everything else"}},
    "urgency":    {"type": "score", "instructions": "How urgent is this request?",
                   "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
    "churn_risk": {"type": "noul", "instructions": "Does the user threaten to cancel or leave?"},
})
a = out["answers"]
print(a["department"]["choice"], round(a["department"]["confidence"], 3), a["department"]["defer"])
print("urgency level", round(a["urgency"]["expected_level"], 2), "of", 2)
print("P(churn)", round(a["churn_risk"]["p_true"], 3))
```

Plain options work too: `{"question": "Which team?", "options": ["Accounts", "Billing", "Shipping"]}`.

### 5.3 Reading a result

Each item in `out["results"]` (and `out["answers"][key]`) contains:

| Field | Meaning |
|---|---|
| `key` | The question's name (dict input) or `None` |
| `type` | `choice`, `noul` or `score` |
| `choice` | The chosen key: the criteria key, the option text, `True`/`False` for `noul`, or the level index for `score` |
| `choice_text` | The option text the model saw |
| `confidence` | Calibrated probability of `choice` |
| `probs` | The full distribution, keyed by `str(key)` |
| `defer` | `True` when `confidence` is below the defer threshold (default 0.70, stored in the config): don't act on it (§7.3) |
| `p_true` | `noul` only: calibrated P(yes) |
| `expected_level` | `score` only: probability-weighted level (0 … k−1); better than the argmax for ordinal rubrics (test MAE 0.57 levels) |

How to read them:

- **Low confidence, spread probabilities**: the state doesn't support any option clearly. Defer, or add "None of the above".
- **`noul` near 0.5**: genuinely ambiguous. Ask for more information rather than guessing.
- **`score`**: use `expected_level` for thresholds ("escalate if ≥ 1.5") rather than `choice`.

---

## 6. Testing the model

Tests 6.1–6.3 need no labelled data, so run them in CI whenever you change the revision. Test 6.4 is the one that tells you whether to ship.

### 6.1 Integrity and determinism

Save as `check_lightdec.py` and run `python check_lightdec.py [revision]`.

```python
import contextlib, io, sys
import numpy as np
from lightdec import load_lightdec

rev = sys.argv[1] if len(sys.argv) > 1 else None
log = io.StringIO()
with contextlib.redirect_stdout(log):
    fdm, model, tok = load_lightdec(revision=rev)
assert "load warning" not in log.getvalue(), log.getvalue()

fc = model.fcfg
assert fc["name"] == "FalconDec", fc["name"]           # LightDec checkpoints use the FalconDec architecture name
T = model.temperature.float().cpu().numpy()
assert T.shape == (3, 4) and (T > 0).all(), T
q = {"team": {"question": "Which team?", "options": ["recover password", "shipping", "invoicing"]}}
a = fdm.decide(model, tok, "I forgot my password and can't sign in.", q)["answers"]["team"]
b = fdm.decide(model, tok, "I forgot my password and can't sign in.", q)["answers"]["team"]
assert all(abs(a["probs"][k] - b["probs"][k]) < 1e-4 for k in a["probs"]), "non-deterministic"
assert abs(sum(a["probs"].values()) - 1) < 1e-3
print(f"OK  LightDec (FalconDec v{fc['version']}, notebook {fc.get('notebook_version')})  choice={a['choice']}  "
      f"conf={a['confidence']:.3f}  defer_threshold={fc.get('defer_threshold')}")
```

The stored temperatures should read approximately `[[1.707, 1.352, 1.466, 1.349], [1.402 ×4], [1.453 ×4]]` (§11.2).

### 6.2 Behavioural tests (pytest)

Save as `tests/test_lightdec.py` and run `pytest -q`. Set `LIGHTDEC_REVISION` to test a pinned commit.

```python
import os, random
import pytest
from lightdec import load_lightdec


@pytest.fixture(scope="session")
def fd():
    return load_lightdec(revision=os.environ.get("LIGHTDEC_REVISION"))


def ask(fd, state, question, options, **kw):
    fdm, model, tok = fd
    return fdm.decide(model, tok, state, [dict(question=question, options=options, **kw)])["results"][0]


def test_probabilities_are_valid(fd):
    r = ask(fd, "The build failed on main.", "What next?", ["Revert", "Ignore", "Retry"])
    assert all(0 <= p <= 1 for p in r["probs"].values()) and abs(sum(r["probs"].values()) - 1) < 1e-3


def test_typed_outputs(fd):
    fdm, model, tok = fd
    out = fdm.decide(model, tok, "I was charged twice. Refund me or I'm leaving.", {
        "refund": {"type": "noul", "instructions": "Does the user ask for a refund?"},
        "urgency": {"type": "score", "instructions": "How urgent?", "criteria": ["low", "medium", "high"]}})["answers"]
    assert 0 <= out["refund"]["p_true"] <= 1 and out["refund"]["choice"] in (True, False)
    assert 0 <= out["urgency"]["expected_level"] <= 2


def test_support_routing(fd):
    r = ask(fd, "I forgot my password and the reset email never arrived.", "Which team should handle this?",
            ["recover password", "billing and payment", "delivery information"])
    assert r["choice"] == "recover password"


def test_fanout_matches_single_questions(fd):
    # Batching changes padding; under bf16 that moves probabilities slightly, never the substance.
    fdm, model, tok = fd
    state = "I forgot my password and can't sign in."
    qs = [{"question": "Team?", "options": ["recover password", "shipping", "invoicing"]},
          {"question": "Urgent?", "options": ["Yes", "No"]}]
    together = fdm.decide(model, tok, state, qs)["results"]
    for q, t in zip(qs, together):
        alone = fdm.decide(model, tok, state, [q])["results"][0]
        assert all(abs(alone["probs"][k] - t["probs"][k]) < 2e-2 for k in alone["probs"])


def test_option_order_is_mostly_irrelevant(fd):
    # The head is order-equivariant, but the encoder sees positions; training reshuffled options every epoch.
    state, q = "Where is my parcel? It's three days late.", "What should support do?"
    opts = ["Give the delivery status", "Start a refund", "Book an appointment"]
    base = ask(fd, state, q, opts)["choice"]
    same = sum(ask(fd, state, q, random.Random(s).sample(opts, len(opts)))["choice"] == base for s in range(5))
    assert same >= 4


def test_many_options_use_the_tournament(fd):
    opts = [f"topic number {i}" for i in range(119)] + ["reset my password"]
    r = ask(fd, "I can't log in, I need to reset my password.", "What does the user want?", opts)
    assert len(r["probs"]) == 120 and abs(sum(r["probs"].values()) - 1) < 1e-3


def test_int8_agrees_with_fp16(fd):
    fdm8, m8, tok8 = load_lightdec(revision=os.environ.get("LIGHTDEC_REVISION"), variant="int8")
    fdm, model, tok = fd
    items = [dict(state=s, question="Which team?", options=["billing", "shipping", "accounts", "technical"])
             for s in ["I was double charged", "Where is my parcel?", "Change my email", "The app crashes on start",
                       "Refund the duplicate payment", "Package never arrived", "Reset my login", "Error 500 on checkout"]]
    a = [p.argmax() for p in fdm.score_items(model, tok, items)]
    b = [p.argmax() for p in fdm8.score_items(m8, tok8, items)]
    assert sum(x == y for x, y in zip(a, b)) >= len(items) - 1
```

### 6.3 Latency

```python
import time, numpy as np, torch
from lightdec import load_lightdec

fdm, model, tok = load_lightdec()
q = [{"question": "Route?", "options": ["billing and payment", "shipping", "recover password"]}]
for _ in range(5):
    fdm.decide(model, tok, "I was charged twice.", q)
t = []
for _ in range(100):
    if torch.cuda.is_available(): torch.cuda.synchronize()
    t0 = time.perf_counter(); fdm.decide(model, tok, "I was charged twice.", q)
    if torch.cuda.is_available(): torch.cuda.synchronize()
    t.append((time.perf_counter() - t0) * 1000)
print(f"p50 {np.percentile(t, 50):.1f} ms   p95 {np.percentile(t, 95):.1f} ms   on {model.device}")
```

For CPU serving, load the int8 variant with `device="cpu", dtype=torch.float32`, and optionally apply `torch.ao.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)` for int8 matrix multiplies.

### 6.4 Accuracy on your own labelled data

Write 50–500 decisions that look like your real traffic, one JSON object per line. `expected` may be a letter, a 0-based index or the option text; `type` is optional.

```json
{"id": "t1", "tag": "support", "state": "…", "question": "…", "options": ["…", "…"], "expected": "B", "type": "choice"}
```

```python
import json, string
import numpy as np
from lightdec import load_lightdec

fdm, model, tok = load_lightdec()
rows = [json.loads(l) for l in open("my_eval.jsonl", encoding="utf-8") if l.strip()]

def idx(v, opts):
    if isinstance(v, int): return v
    v = str(v).strip()
    if len(v) == 1 and v.upper() in string.ascii_uppercase[:len(opts)]: return string.ascii_uppercase.index(v.upper())
    return [o.lower() for o in opts].index(v.lower())

items = [dict(state=r["state"], question=r.get("question", ""), options=r["options"], type=r.get("type", "choice"))
         for r in rows]
probs = fdm.score_items(model, tok, items, batch_size=64)
gold = np.array([idx(r["expected"], r["options"]) for r in rows])
pred = np.array([p.argmax() for p in probs]); conf = np.array([p.max() for p in probs]); ok = pred == gold

def ece(c, k, bins=15):
    e = 0.0
    for lo in np.linspace(0, 1, bins, endpoint=False):
        m = (c > lo) & (c <= lo + 1 / bins)
        if m.any(): e += m.mean() * abs(c[m].mean() - k[m].mean())
    return e

print(f"accuracy {ok.mean():.3f} | ECE {ece(conf, ok):.3f}")
for tag in sorted({r.get("tag", "all") for r in rows}):
    m = np.array([r.get("tag", "all") == tag for r in rows]); print(f"  {tag:12s} n={m.sum():4d} acc={ok[m].mean():.3f}")
for t in (0.5, 0.6, 0.7, 0.8, 0.9):
    m = conf >= t
    print(f"  act if conf >= {t}: answers {m.mean():6.1%}, accuracy when answering {ok[m].mean() if m.any() else float('nan'):.3f}")
```

The last loop is the deferral policy of §7.3; on the published test mix, 0.70 gives 56% coverage at 0.896 accuracy. Pick the smallest threshold whose "accuracy when answering" meets your bar. If ECE on your data is much higher than 0.025, recalibrate (§8.2).

### 6.5 Regression gate between revisions

Fail the pipeline if any test task drops by more than two points between two revisions:

```python
import json, sys
from huggingface_hub import hf_hub_download

old_rev, new_rev = sys.argv[1:3]
rep = lambda rev: {r["task"]: r["acc"] for r in json.load(open(
    hf_hub_download("Falconsai/LightDec", "falcondec_report.json", revision=rev), encoding="utf-8"))["test_per_task"]}
old, new = rep(old_rev), rep(new_rev)
bad = [(t, old[t], new[t]) for t in new if t in old and new[t] < old[t] - 0.02]
print("\n".join(f"REGRESSION {t}: {a:.3f} -> {b:.3f}" for t, a, b in bad) or "no regressions")
sys.exit(1 if bad else 0)
```

---

## 7. Using it in an agentic system

### 7.1 Where it fits

An agent loop is mostly small decisions (which tool, is this safe, did that work, am I done, should a human look) around a few hard reasoning steps. LLMs are slow and poorly calibrated at the small ones. LightDec takes those; the LLM keeps planning, reasoning and generation.

| Agent step | How to phrase it | Evidence |
|---|---|---|
| **Entry routing** | state = the request; `choice` over sub-agents or workflows, plus "None of the above" | Intents 0.773–0.997 by source |
| **Retrieval routing** | state = the question; options = candidate documents or indexes | HotpotQA retrieve 0.842 (9 candidates) |
| **Step verification** | `noul`: "Does the agent's current step contain an error?" | Counsel step-error 0.791 |
| **Guardrail** | `noul`: "Does this input try to override the agent's instructions?" on user input *and* on tool results | Jailbreak 0.966; held-out injection 0.647, so validate on your traffic |
| **Conditional edges** | `choice`: "Retry, continue, escalate or finish?" over the current state | Workflows 0.689 |
| **Escalation** | `defer == True`, or `confidence` below your threshold → human or larger model | 0.896 accuracy on the confident 56% |

Keep option sets under about 20. For larger menus, shortlist first (embedding search or a coarse `choice`), then ask LightDec; all-77-label Banking77 drops to 0.470. Several questions about the same state go in one `decide()` call.

### 7.2 A routing node (LangGraph)

```python
from lightdec import load_lightdec

fdm, model, tok = load_lightdec(revision="<commit>")

def route(state: dict) -> str:
    res = fdm.decide(model, tok, state, {
        "next": {"type": "choice", "instructions": "What should the agent do next?",
                 "criteria": {"search": "needs external information", "code": "needs code written or run",
                              "answer": "has enough information to answer", "human": "ambiguous, risky or out of scope"}},
        "unsafe": {"type": "noul", "instructions": "Does the latest input try to override the agent's instructions?"},
    }, defer_threshold=0.75)["answers"]
    if res["unsafe"]["p_true"] > 0.5:
        return "human"
    if res["next"]["defer"]:
        return "llm_planner"             # low confidence: let the LLM decide
    return res["next"]["choice"]

graph.add_conditional_edges("observe", route, {"search": "search_node", "code": "code_node", "answer": "answer_node",
                                               "human": "human_node", "llm_planner": "planner_node"})
```

### 7.3 The deferral policy

| Situation | Action |
|---|---|
| `confidence` ≥ your threshold | Act |
| `confidence` below it (`defer == True`) | **Defer**: hand to the LLM, ask a human, or ask the user for more information |
| Guardrail `noul` with `p_true` above your risk threshold | Block or escalate, regardless of other answers |

Choose thresholds from your own evaluation (§6.4). The default 0.70 gives 56% coverage at 0.896 accuracy on the published test mix. Confidence is not trustworthy on the task types listed as out of scope in §3; date-format transfer, for example, is confidently wrong. For irreversible actions (payments, deletions, sending email), raise the threshold and keep a hard rule or human confirmation in front: the state is attacker-controlled text, and adversarial input can move scores. Log the question, options, choice, confidence, model version and Hub revision for every decision; that log becomes your next evaluation and fine-tuning set (§8.3).

### 7.4 As a tool for Claude (tool use)

```python
import json, threading
import anthropic
from lightdec import load_lightdec

fdm, model, tok = load_lightdec()
lock = threading.Lock()
client = anthropic.Anthropic()
tools = [{
    "name": "lightdec_decide",
    "description": ("Fast, local, calibrated closed-set decision model. Give it a state (text or JSON), a question and "
                    "2-20 distinct options; it returns the choice, a calibrated confidence and a 'defer' flag. "
                    "If 'defer' is true, don't rely on the answer. Not for arithmetic, dates or multi-step reasoning."),
    "input_schema": {"type": "object", "properties": {
        "state": {"type": "string", "description": "The message, document excerpt or JSON state."},
        "question": {"type": "string"},
        "options": {"type": "array", "items": {"type": "string"}, "minItems": 2},
        "type": {"type": "string", "enum": ["choice", "score"], "description": "score = options are ordered levels"}},
        "required": ["state", "question", "options"]},
}]

def run_tool(inp):
    with lock:
        r = fdm.decide(model, tok, inp["state"], [{"question": inp["question"], "options": inp["options"],
                                                   "type": inp.get("type", "choice")}])["results"][0]
    return {k: r[k] for k in ("choice", "confidence", "defer", "probs") if k in r}

messages = [{"role": "user", "content": "Triage: 'I forgot my password and the reset email never arrived.' "
                                        "Teams: Accounts, Billing, Shipping."}]
while True:
    resp = client.messages.create(model="claude-sonnet-5", max_tokens=1024, tools=tools, messages=messages)
    if resp.stop_reason != "tool_use":
        print("".join(b.text for b in resp.content if b.type == "text"))
        break
    messages.append({"role": "assistant", "content": resp.content})
    results = []
    for block in resp.content:
        if block.type == "tool_use" and block.name == "lightdec_decide":
            try:
                results.append({"type": "tool_result", "tool_use_id": block.id, "content": json.dumps(run_tool(block.input))})
            except Exception as exc:
                results.append({"type": "tool_result", "tool_use_id": block.id, "content": str(exc), "is_error": True})
    messages.append({"role": "user", "content": results})
```

### 7.5 As an MCP server

`pip install mcp`, then save `lightdec_mcp.py` next to `lightdec.py`:

```python
import os, threading
from mcp.server.fastmcp import FastMCP
from lightdec import load_lightdec

fdm, model, tok = load_lightdec(revision=os.environ.get("LIGHTDEC_REVISION"),
                                variant=os.environ.get("LIGHTDEC_VARIANT", "fp16"))
MIN_CONF = float(os.environ.get("LIGHTDEC_MIN_CONF", "0.7"))
lock = threading.Lock()
mcp = FastMCP("lightdec")


@mcp.tool()
def decide(state: str, question: str, options: list[str], type: str = "choice") -> dict:
    """Choose one of 2-20 distinct options for a question about a state. type="score" means ordered levels.
    Returns the choice, a calibrated confidence and 'defer' (true = not reliable enough to act on)."""
    with lock:
        r = fdm.decide(model, tok, state, [{"question": question, "options": options, "type": type}],
                       defer_threshold=MIN_CONF)["results"][0]
    return {k: r[k] for k in ("choice", "confidence", "defer", "probs", "expected_level") if k in r}


@mcp.tool()
def decide_many(state: str, questions: dict) -> dict:
    """Several typed questions about one state: {name: {"type": "choice"|"noul"|"score",
    "instructions": str, "criteria": {key: description} | [levels]}}."""
    with lock:
        return fdm.decide(model, tok, state, questions, defer_threshold=MIN_CONF)["answers"]


if __name__ == "__main__":
    mcp.run()
```

Register it in Claude Desktop's `claude_desktop_config.json`:

```json
{
  "mcpServers": {
    "lightdec": {
      "command": "C:\\path\\to\\python.exe",
      "args": ["C:\\path\\to\\lightdec_mcp.py"],
      "env": { "LIGHTDEC_REVISION": "<commit>", "LIGHTDEC_VARIANT": "int8" }
    }
  }
}
```

### 7.6 As an HTTP microservice

`pip install fastapi uvicorn`, then save `serve_lightdec.py`:

```python
import threading
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from lightdec import load_lightdec

fdm, model, tok = load_lightdec()
lock = threading.Lock()
app = FastAPI(title="LightDec")


class Decide(BaseModel):
    state: str | dict
    questions: dict | list
    defer_threshold: float = 0.7


@app.get("/health")
def health():
    return {"ok": True, "model": "LightDec", "version": model.fcfg.get("version")}


@app.post("/v1/decide")
def decide(req: Decide):
    try:
        with lock:
            return fdm.decide(model, tok, req.state, req.questions, defer_threshold=req.defer_threshold)
    except (ValueError, KeyError, TypeError) as exc:
        raise HTTPException(422, str(exc))
```

Run it with `uvicorn serve_lightdec:app --port 9904 --workers 1`. Each worker holds its own copy of the model; scale out with more processes.

### 7.7 Without an LLM: a support intake step

```python
from lightdec import load_lightdec

fdm, model, tok = load_lightdec()
QUEUES = {"billing": "charges, invoices, payments, refunds", "returns": "returning or exchanging items",
          "delivery": "shipping status, late or missing parcels", "accounts": "login, password, profile",
          "human": "complaints or requests to speak to a person"}

def intake(message: str) -> dict:
    a = fdm.decide(model, tok, message, {
        "queue": {"type": "choice", "instructions": "Which team should handle this message?", "criteria": QUEUES},
        "urgent": {"type": "noul", "instructions": "Does this need a reply within the hour?"},
    })["answers"]
    if a["queue"]["defer"]:
        return {"action": "human_review", "suggestion": a["queue"]["choice"],
                "reason": f"low confidence ({a['queue']['confidence']:.0%})"}
    return {"action": "enqueue", "queue": a["queue"]["choice"],
            "priority": "high" if a["urgent"]["p_true"] >= 0.5 else "normal",
            "evidence": {"confidence": round(a["queue"]["confidence"], 3), "model": "LightDec"}}
```

---

## 8. Tuning for your domain

### 8.1 Change the deferral threshold

Pass `defer_threshold=` to `decide()`, or set `model.fcfg["defer_threshold"]`. This changes nothing in the model and is usually enough.

### 8.2 Recalibrate on your data

If ECE on your traffic (§6.4) is noticeably worse than 0.025, fit one extra temperature on top of the stored ones and save a recalibrated copy:

```python
import numpy as np, torch
# probs, gold: from §6.4 (probabilities already include the stored temperatures)
logp = [np.log(np.clip(p, 1e-12, 1)) for p in probs]
def nll(s):
    return -np.mean([(lp / s)[g] - np.log(np.exp(lp / s).sum()) for lp, g in zip(logp, gold)])
s = min(np.linspace(0.5, 3.0, 51), key=nll)
print("extra temperature", s)
with torch.no_grad():
    model.temperature.mul_(float(s))
fdm.save_falcondec(model, tok, "lightdec_recalibrated")            # add int8=True for the compact variant
```

Fit on one labelled set and measure on another.

### 8.3 Fine-tune on your own decisions

Use the FalconDec training notebook (V2):

1. Export your logged and corrected decisions as JSONL: `{"state", "question", "options", "answer", "type"?, "task"?}`.
2. In cell 2, set `MODE="finetune"`, `FINETUNE_FROM="Falconsai/LightDec"`, `CUSTOM_DATA_JSONL="your_file.jsonl"`, and a preset.
3. Run the notebook. The version bumps automatically (1.0.0 → 1.0.1), and the report records the lineage.

Your tasks are up-weighted (`CUSTOM_WEIGHT`), while the public tasks keep the model general. Adding decisions with ISO-format dates and numeric tables is the most direct fix for the transfer weaknesses in §3. Check the regression gate (§6.5) before publishing.

### 8.4 Save and publish

```python
fdm.save_falcondec(model, tok, "lightdec_out")                     # fp16
fdm.save_falcondec(model, tok, "lightdec_out/compact-int8", int8=True)

from huggingface_hub import HfApi                                   # needs a write token: huggingface-cli login
HfApi().upload_folder(folder_path="lightdec_out", repo_id="Falconsai/LightDec", commit_message="LightDec v1.0.x")
```

---

## 9. Architecture

```
[CLS] question [SEP] [MASK] option₁ [MASK] option₂ … [MASK] optionₖ [SEP] state [SEP]
        │
  Ettin-150M encoder (22 layers, hidden 768)
        │
  hidden state at each [MASK]  +  CLS context  +  question-type embedding
        │
  set transformer: 2 layers, 8 heads, no positional encoding → options attend to each other, order-equivariant
        │
  MLP → one logit per option → ÷ temperature[type, option-count bucket] → softmax
```

- **Option markers** (the approach Laya uses): every option is read at its own `[MASK]` token, so all options are scored in **one** pass.
- **Question first, state last**: when the input is too long, the tail of the state is truncated, never the options.
- **Adaptive option budget**: up to 24 tokens per option within a 192-token head budget that grows with the option count, and 2,048-token sequences above 24 options, so labels stay distinct. Above 96 options, `decide()` runs a tournament.
- **Typed primitives**: `noul` is rendered as a neutral two-option Yes/No choice; `score` keeps its level order and reports an expected level.
- **Calibration** lives in the model: a 3 × 4 temperature table (question type × option-count bucket: ≤2, 3–5, 6–12, >12).
- **int8 storage**: per-output-channel symmetric int8 for every weight matrix, fp16 elsewhere, dequantised on load.

---

## 10. Training data

**155,747 training, 13,148 validation and 17,498 test decisions** from 58 tasks. No source failed to load (`skipped_builders` is empty). Every example is a **decision**: `state`, `question`, `options`, `answer`, plus type, task and domain.

| Domain | Sources | Decisions built |
|---|---|---|
| Support | Bitext customer support | Intent routing and category routing; 30% of messages wrapped as JSON program state |
| Intents | CLINC150, MASSIVE (en), Banking77 *(held out)* | Intents as runtime-defined options, 2–48 per question; Banking77 also as one 77-option question |
| Code | CodeXGLUE code-to-text (6 languages), Devign, BigCloneBench, MBPP, HumanEval *(held out)* | Language ID, code↔description, function naming, vulnerability, clones, task→solution; bug spotting against single-fault mutants **verified to fail the unit tests** |
| Guardrails | Jailbreak classification, Civil Comments; deepset prompt-injections and AgentHarm *(held out)* | `noul` detection and refusal |
| Agentic | Counsel (human meta-evaluations of agent-step critiques), HotpotQA, AgentTrek | Step-error and critique-quality, retrieval routing and comparison yes/no; AgentTrek next-action type and finish-now. AgentTrek loaded without errors, but none of its decisions appear in the test split, so its contribution isn't measured |
| Workflows | LocalLLaMA/typed-decisions | `choice`/`noul`/`score` with the teacher's soft probabilities; test = the benchmark's 2,000-decision test split |
| Policy | Synthetic, executable (in-notebook) | Return windows, approval tiers, AND/OR eligibility, overdue invoices, table look-ups, counting, SLA urgency, access control, invoice totals; the test split uses **transfer** wording, currencies and date formats |
| Reasoning | ARC-Easy/Challenge and MMLU *(held out)*, OpenBookQA, SciQ, CommonsenseQA, QASC, HellaSwag, WinoGrande, MMLU auxiliary-train, BoolQ, GSM8K, AQuA-RAT, SNLI, MultiNLI, ANLI, SciTail | Multiple choice, yes/no, NLI, numeric answers with near-miss distractors |
| Classification | AG News, Yelp (ordinal); DAIR Emotion and SST-5 *(held out)* | Topic; 5-level `score` sentiment |

**Augmentation**: options are reshuffled every epoch (ordinal levels keep their order). In 8% of choice questions the gold answer is removed and "None of the above" becomes correct; in another 4% it is added as a distractor. **Balance**: tasks are sampled with p ∝ n^0.5 each epoch. **Leak guard**: training decisions whose (state, question) appears in validation or test were removed. Mind2Web was excluded (opt-in in the notebook). Held-out sources were never used for training, calibration or model selection.

Check each dataset's card for its license before redistributing derived data. AgentHarm is used only as a held-out evaluation, in line with its intended use.

---

## 11. Training procedure and calibration

### 11.1 Setup

| Setting | Value |
|---|---|
| Mode / preset | `scratch` from the pretrained backbone / `standard` |
| Data caps | ≤4,000 train, ≤300 validation, ≤300 test decisions per source split |
| Epochs | 2 (best: epoch 2) |
| Objective | Strictly proper scoring rules: log score + 0.5 × spherical score, + 1.0 × ranked probability score for `score` questions; soft targets (50/50 with the hard label) where a teacher distribution exists. These are the RLCD rewards, optimised with exact gradients |
| Optional RLCD stage | Off |
| Optimiser | AdamW (β 0.9/0.98, weight decay 0.01), encoder LR 4e-5 with layer-wise decay 0.9, head LR 3e-4, 6% warm-up, cosine decay, gradient clipping 1.0 |
| Batching | Token-budget batches (16,384 tokens, ≤32 decisions), length-bucketed |
| Weights kept | EMA of the weights (decay 0.999), best validation task-macro accuracy |
| Precision | bf16 autocast, TF32 matmuls; attention `auto` (FlashAttention-2 if installed, else SDPA) |
| Sequence | 512 tokens; 2,048 above 24 options; question ≤96 tokens; ≤24 tokens per option (more for code options) |
| Hardware / time | NVIDIA GeForce RTX 5090 Laptop GPU · **46 minutes** |
| Software | Python 3.14.4 · PyTorch 2.11.0+cu128 · transformers 5.17.0 |
| Seed | 42 |

| Epoch | Train loss | Train acc | Val macro | Val micro | Val NLL |
|---|---|---|---|---|---|
| 1 | 0.868 | 0.654 | 0.731 | 0.738 | 0.565 |
| 2 | 0.540 | 0.802 | **0.763** | 0.775 | 0.520 |

Validation was still improving at epoch 2, so a longer schedule (the `full` preset) is likely to help.

### 11.2 Fitted temperatures

| Question type | k ≤ 2 | k 3–5 | k 6–12 | k > 12 |
|---|---|---|---|---|
| choice | 1.707 | 1.352 | 1.466 | 1.349 |
| noul | 1.402 | 1.402 | 1.402 | 1.402 |
| score | 1.453 | 1.453 | 1.453 | 1.453 |

All temperatures are above 1, so the raw model was over-confident, as Laya's checkpoints are. `noul` and `score` use a single per-type temperature because their questions fall into one option-count bucket (2 options for `noul`, mostly 3–5 levels for `score`).

---

## 12. Operational notes

- **Concurrency.** `decide()` isn't internally locked. Serialise calls with a lock per process (as in §7.4–7.6) and scale out with processes.
- **Hardware.** On a GPU, expect tens of milliseconds per call; measure yours with §6.3. On CPU, use the int8 variant (loaded as fp32) and optionally dynamic int8 quantisation.
- **Input length.** 512 tokens by default. The question and options come first, so an over-long state loses its *end*. Put the decisive information early, or summarise.
- **Determinism.** Repeated identical calls give identical results on the same hardware and library versions. Under bf16 on GPU, batching different questions together changes padding and can move probabilities slightly (about 1e-2); near-ties can flip. Decisions with a clear margin don't change.
- **Traceability.** Log the model version, the Hub revision and the variant (fp16/int8) with every decision.
- **Offline use.** After the first download, set `HF_HUB_OFFLINE=1`, or save a local copy and load it by path.

---

## 13. Bias, risks and limitations

- **Quantitative and date reasoning.** Arithmetic (AQuA 0.259), table comparisons (0.290) and counting or summing (0.523) are near chance. Overdue-invoice checks with ISO dates score 0.407 with ECE 0.364, meaning confidently wrong.
- **Wide option sets.** Accuracy falls to 0.470 with all 77 Banking77 intents in one question; shortlist first.
- **Generalisation gap.** Held-out tasks average 0.567 against 0.725 overall. Expect lower accuracy on traffic unlike the training mix, and measure it (§6.4).
- **Guardrail transfer.** In-distribution jailbreak detection is 0.966, but held-out prompt-injection (0.647) and AgentHarm refusal (0.654) are much lower, with ECE 0.272 and 0.178. Don't rely on it as the only safety layer.
- **No completed head-to-head with proof_v2.** The comparisons in §2.4–2.5 use published numbers on different samples.
- **Closed world.** The model always picks one of your options. Add "None of the above" when appropriate.
- **Distribution-dependent calibration.** ECE was measured on this test mix; re-check it on your traffic and recalibrate if needed (§8.2).
- **Adversarial input.** The state is untrusted text. The model can't be instructed like an LLM, but crafted input can shift its scores. Don't make it the only safeguard before irreversible actions.
- **Data provenance.** Training data is English, largely crowd-sourced, templated, synthetic or scraped from public code and web tasks. Biases in these sources and in the backbone's pre-training can carry into decisions. Automated routing can systematically misroute users whose phrasing differs from the training data (dialects, non-native speakers, assistive phrasing); monitor misroutes by group where possible.
- **Oversight.** Not for high-stakes decisions without human review.

---

## 14. Versioning and lineage

| | proof_V_1 | proof_v2 | proof_v3 | **LightDec** |
|---|---|---|---|---|
| Model | falconsproof v1 | falconsproof v2.0.0 | FalconDec v1.0.0 (notebook V1) | **FalconDec v1.0.0 (notebook V2)** |
| Backbone | DistilBERT, 128 tokens | ModernBERT-base, 384 tokens | Ettin-150M, 512 / 2,048 | Ettin-150M, 512 / 2,048 |
| Preset / epochs | — | small / 1 per stage | small / 1 | **standard / 2** |
| Training decisions | — | 18,795 | ≤500 per task | **155,747** |
| Agentic data | — | — | — | AgentTrek, Counsel, HotpotQA |
| Test accuracy | — | 0.813 (23-task, code-heavy mix) | 0.518 micro / 0.535 macro | **0.725 / 0.725** (58-task mix) |
| ECE | — | 0.008 | 0.014 | 0.025 |
| Weights | — | ≈596 MB fp32 | 319 MB fp16 | 319 MB fp16 · 161 MB int8 |

LightDec is a fresh `scratch` run from the pretrained Ettin backbone (lineage: `jhu-clsp/ettin-encoder-150m`). It doesn't inherit proof_v3's or proof_v2's weights. Fine-tuning LightDec with the notebook bumps the patch version (1.0.0 → 1.0.1).

**Changelog.** LightDec 1.0.0: first release. Notebook V2, `standard` preset, 2 epochs, seed 42.

---

## 15. API reference

All functions live in `falcondec_modeling.py`.

| Function | Description |
|---|---|
| `load_falcondec(path, device=None, dtype=None, attn_implementation="sdpa")` | Loads a FalconDec directory (fp16 or int8) or Hub repo id. Returns `(model, tokenizer)`; cuda if available |
| `decide(model, tok, state, questions, defer_threshold=None, batch_size=32)` | Typed questions about one state. `questions` is a list or a `{key: question}` dict. Returns `{"results": [...], "answers": {key: result}}` |
| `score_items(model, tok, items, batch_size=32)` | Batch scoring. `items = [{"state", "question", "options", "type"?, "option_tokens"?, "seq_len"?}]`. Returns a calibrated probability array per item |
| `save_falcondec(model, tok, out_dir, int8=False, extra_files=None)` | Writes a self-contained directory (copies the modeling file) |
| `quantize_int8(state_dict)` / `dequantize_int8(state_dict)` | The per-channel int8 codec used for `compact-int8/` |
| `assemble(...)`, `collate_features(...)` | Low-level sequence building and batching |

**Question fields**: `type` (`choice` / `noul` / `score`, default `choice`); `question` or `instructions`; `options` (list) or `criteria` (dict `{key: description}` for choice, list of levels for score); `labels` (`{"true": …, "false": …}` wording for noul); `option_tokens` and `seq_len` (optional per-question budgets).

**Model attributes**: `model.fcfg` (the live config: layout, `special`, `defer_threshold`, `version`, `lineage`); `model.temperature` (3 × 4 tensor); `model.num_parameters()`; `model.device`.

---

## 16. Citation and references

```bibtex
@misc{falconsai_lightdec_2026,
  title        = {LightDec: a lightweight, single-pass, typed, calibrated decision model for agentic systems},
  author       = {{Falconsai}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Falconsai/LightDec}},
  note         = {FalconDec architecture, Ettin-150M backbone; successor to Falconsai/proof_v3}
}
```

**Methods.** Warner et al. (2024), *ModernBERT*. Weller et al. (2025), *Ettin* encoders. Gneiting and Raftery (2007), *Strictly Proper Scoring Rules*. Guo et al. (2017), *On Calibration of Modern Neural Networks*. Geifman and El-Yaniv (2017), *Selective Classification*. Zaheer et al. (2017), *Deep Sets*; Lee et al. (2019), *Set Transformer*. Williams (1992), REINFORCE; Shao et al. (2024), GRPO. Hinton, Vinyals and Dean (2015), *Distillation*. Related decision models: Laya (Convai Innovations), TypeSafe Jev, Together Tev1.

**Data.** ARC, OpenBookQA, SciQ, CommonsenseQA, QASC, HellaSwag, WinoGrande, MMLU, BoolQ, GSM8K, AQuA-RAT, SNLI, MultiNLI, ANLI, SciTail, CLINC150, MASSIVE, Bitext, Banking77, AG News, Yelp, DAIR Emotion, SST-5, jailbreak-classification, Civil Comments, deepset prompt-injections, AgentHarm, CodeXGLUE, MBPP, HumanEval, LocalLLaMA/typed-decisions, AgentTrek, Counsel, HotpotQA.

Report issues, misroutes or evaluation results through the Community tab of this repository.

---

This card is generated from the surgical record itself; the package's
`lineage.intoto.jsonl` is the signed source of truth (verify it free at
the Surgeon's public verifier or with the bundled `verify_attestation.py`).

## Architecture
- Identification: **NLP · Small Language Model (SLM)** (98% confidence)
- Source format: `safetensors` · Intended task: not declared
- `config.json`: synthesized from the anatomy (no source config.json); model_type omitted — no architecture name in the source (QA-F-126)
- Source license: apache-2.0
- Lineage chain: 1 surgery (no prior attestation reachable) · Falconsai/LightDec
- Post-surgery totals: 159,654,157 parameters ·
  168 tensors
- Compute estimate: 15.02439 GFLOPs (comparison
  metric, not a measurement)

## Provenance & operations
- Parents: Falconsai/LightDec/model.safetensors
- Operations performed: forensics×1, hub_push×1, imaging×3, load×1, test×1
- Weight merges recorded: 0
- Quantized tensors (F32→F16): 0

## Surgery Log (ordered)
1. **load** — hub:Falconsai/LightDec/model.safetensors (319.3 MB, safetensors)
2. **hub_push** — Falconsai/LightDec ← 8 file(s) (attestation included)
3. **test** — PASS
4. **forensics** — clean — no anomalies
5. **imaging** — synthetic 16 token ids → [1, 2304] [WEAK]
6. **imaging** — synthetic 16 token ids → [1, 2304] [WEAK]
7. **imaging** — 41 token ids (hash-mapped, vocab 50368) → [1, 2304] [WEAK]

## Validation
- Tissue imaging: WEAK
- Structural integrity is testable offline via the packaged
  `load_and_test.py`.

## Compliance note
The signed attestation + this card together document model composition,
modification history, and validation evidence — the record structure
technical-documentation obligations (e.g. EU AI Act Annex IV) ask for.
This is evidence, not legal advice.

---
*Operated with Model Surgeon — verify this package at https://surgeon.falcons.ai/verify*
*© 2026 FALCONS.AI — Model Surgeon record format. The model weights remain their owner's.*