---
license: apache-2.0
language: [en]
library_name: transformers
tags: [decision-model, system-one, calibrated-decisions, multiple-choice, zero-shot-classification, falcondec]
---
# LightDec v1.0.1 — Falcon Decision

Single-pass, typed (`choice` / `noul` / `score`), calibrated closed-set decision model. Successor to
[`Falconsai/proof_v2`](https://huggingface.co/Falconsai/proof_v2).

* Built with FalconDec notebook V2 · backbone `jhu-clsp/ettin-encoder-150m` · mode `finetune` · preset `full` · lineage: jhu-clsp/ettin-encoder-150m → FalconDec-v1.0.0
* Parameters: 159.7M · weights: fp16 305 MB, int8 153 MB
* Layout: `[CLS] question [SEP] [MASK] opt1 … [MASK] optk [SEP] state [SEP]`; set-transformer option head;
  temperature per (question type × option-count bucket).

## Usage

```python
import importlib.util
from huggingface_hub import snapshot_download
path = snapshot_download("<repo>")            # or a local FalconDec-v1.0.1 directory
spec = importlib.util.spec_from_file_location("falcondec_modeling", f"{path}/falcondec_modeling.py")
fdm = importlib.util.module_from_spec(spec); spec.loader.exec_module(fdm)
model, tok = fdm.load_falcondec(path)          # add "/compact-int8" for the int8 artefact
out = fdm.decide(model, tok, state={"body": "I was charged twice, refund me or I cancel."}, questions={
    "dept": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "payments", "tech": "bugs"}},
    "churn": {"type": "noul", "instructions": "Does the user threaten to leave?"},
    "urgency": {"type": "score", "instructions": "How urgent?", "criteria": ["low", "medium", "high"]}})
```

## Test results

Overall: micro acc **0.781**, task-macro acc **0.769**, ECE **0.020**,
NLL 0.529, Brier 0.290, AURC 0.064.

| task | domain | heldout | n | chance | acc | proof_v2 (card) | ece |
|---|---|---|---|---|---|---|---|
| counsel/critique_quality | agentic | False | 201 | 0.333 | 0.592 |  | 0.227 |
| counsel/step_has_error | agentic | False | 201 | 0.500 | 0.801 |  | 0.169 |
| hotpotqa/comparison_yes_no | agentic | False | 26 | 0.500 | 0.846 |  | 0.151 |
| hotpotqa/retrieve | agentic | False | 497 | 0.167 | 0.873 |  | 0.083 |
| sst5/score | classification | True | 500 | 0.200 | 0.416 |  | 0.046 |
| emotion/6way | classification | True | 500 | 0.167 | 0.502 |  | 0.082 |
| yelp/score | classification | False | 500 | 0.200 | 0.654 |  | 0.069 |
| ag_news/topic | classification | False | 500 | 0.250 | 0.888 |  | 0.040 |
| devign/vulnerability | code | False | 500 | 0.500 | 0.596 | 0.542 | 0.050 |
| humaneval/completion | code | True | 119 | 0.394 | 0.672 | 0.575 | 0.107 |
| mbpp/bugspot | code | False | 256 | 0.406 | 0.730 | 0.475 | 0.102 |
| bigclonebench/clone | code | False | 500 | 0.500 | 0.952 | 0.383 | 0.016 |
| codexglue/func_name | code | False | 467 | 0.255 | 0.953 | 0.901 | 0.023 |
| codexglue/doc_to_code | code | False | 504 | 0.279 | 0.976 | 0.961 | 0.010 |
| mbpp/solution | code | False | 500 | 0.250 | 0.982 | 0.992 | 0.015 |
| codexglue/code_to_doc | code | False | 504 | 0.262 | 0.984 | 0.969 | 0.012 |
| codexglue/lang_id | code | False | 504 | 0.235 | 1.000 | 0.997 | 0.001 |
| prompt_injections/detect | guardrails | True | 116 | 0.500 | 0.578 |  | 0.286 |
| agentharm/refuse | guardrails | True | 416 | 0.500 | 0.743 |  | 0.067 |
| civil_comments/toxic | guardrails | False | 500 | 0.500 | 0.906 |  | 0.035 |
| jailbreak/detect | guardrails | False | 262 | 0.500 | 0.954 |  | 0.023 |
| banking77/intent_77 | intents | True | 300 | 0.013 | 0.533 |  | 0.147 |
| massive_en/intent | intents | False | 500 | 0.130 | 0.906 |  | 0.022 |
| banking77/intent | intents | True | 500 | 0.321 | 0.930 | 0.883 | 0.040 |
| clinc150/intent | intents | False | 500 | 0.124 | 0.942 | 0.850 | 0.036 |
| policy/invoice_total_transfer | policy | False | 500 | 0.500 | 0.486 |  | 0.032 |
| policy/invoice_overdue_transfer | policy | False | 500 | 0.500 | 0.600 |  | 0.201 |
| policy/sla_urgency_transfer | policy | False | 500 | 0.250 | 0.698 |  | 0.212 |
| policy/free_shipping_transfer | policy | False | 500 | 0.500 | 0.766 |  | 0.101 |
| policy/refund_approval_transfer | policy | False | 500 | 0.333 | 0.780 |  | 0.186 |
| policy/count_threshold_transfer | policy | False | 500 | 0.178 | 0.846 |  | 0.109 |
| policy/table_extreme_transfer | policy | False | 500 | 0.237 | 0.912 |  | 0.014 |
| policy/return_window_transfer | policy | False | 500 | 0.333 | 0.998 |  | 0.020 |
| policy/access_control_transfer | policy | False | 500 | 0.333 | 1.000 |  | 0.000 |
| aqua_rat/math | reasoning | False | 247 | 0.200 | 0.275 |  | 0.042 |
| mmlu/mcq | reasoning | True | 500 | 0.250 | 0.356 |  | 0.109 |
| anli/nli | reasoning | False | 498 | 0.333 | 0.416 |  | 0.184 |
| arc_challenge/mcq | reasoning | True | 500 | 0.250 | 0.460 | 0.308 | 0.088 |
| hellaswag/continuation | reasoning | False | 500 | 0.250 | 0.580 |  | 0.073 |
| arc_easy/mcq | reasoning | True | 500 | 0.250 | 0.594 | 0.425 | 0.038 |
| openbookqa/mcq | reasoning | False | 500 | 0.250 | 0.610 | 0.292 | 0.092 |
| winogrande/blank | reasoning | False | 500 | 0.500 | 0.612 |  | 0.111 |
| commonsense_qa/mcq | reasoning | False | 493 | 0.200 | 0.639 | 0.442 | 0.044 |
| gsm8k/math | reasoning | False | 500 | 0.250 | 0.662 | 0.275 | 0.061 |
| boolq/yes_no | reasoning | False | 500 | 0.500 | 0.816 | 0.717 | 0.027 |
| mnli/claim | reasoning | False | 500 | 0.333 | 0.838 | 0.492 | 0.071 |
| snli/nli | reasoning | False | 988 | 0.333 | 0.880 |  | 0.050 |
| sciq/mcq | reasoning | False | 498 | 0.250 | 0.954 | 0.692 | 0.031 |
| scitail/support | reasoning | False | 500 | 0.500 | 0.964 |  | 0.020 |
| snli/must_be_true | reasoning | False | 500 | 0.333 | 0.980 | 0.908 | 0.029 |
| snli/contradicts | reasoning | False | 500 | 0.333 | 0.986 | 0.892 | 0.014 |
| qasc/mcq | reasoning | False | 500 | 0.125 | 0.990 |  | 0.006 |
| bitext/category | support | False | 500 | 0.161 | 1.000 |  | 0.002 |
| bitext/route | support | False | 500 | 0.190 | 1.000 | 0.958 | 0.001 |
| typed_decisions/agent_trace_observability | workflows | False | 500 | 0.300 | 0.712 |  | 0.120 |
| typed_decisions/security_incidents | workflows | False | 500 | 0.340 | 0.730 |  | 0.144 |
| typed_decisions/customer_service | workflows | False | 500 | 0.280 | 0.740 |  | 0.085 |
| typed_decisions/invoice_processing | workflows | False | 500 | 0.350 | 0.838 |  | 0.122 |


## Training

Strictly proper objective (log + 0.5·spherical + 1.0·RPS for ordinal), soft teacher targets where available,
task-balanced sampling (α = 0.5), NOTA augmentation (8%), per-epoch option reshuffling,
AdamW + LLRD 0.9, EMA 0.999, 2 epoch(s), RLCD stage off/reverted.
Training time 172.2 min on NVIDIA GeForce RTX 5090 Laptop GPU.

## Limitations

* Closed world: the model ranks the options you give it. Add "None of the above" when that is a valid answer (it was trained with it).
* Multi-step arithmetic (GSM8K / AQuA) is a System-2 task; route low-confidence answers to an LLM (`defer` flag, threshold 0.7).
* English only; code judgement (vulnerabilities, clones) is still weak; check calibration on your own traffic.

## References

Warner et al. 2024 (ModernBERT) · Weller et al. 2025 (Ettin) · Gneiting & Raftery 2007 (strictly proper scoring rules) ·
Guo et al. 2017 (temperature scaling) · Geifman & El-Yaniv 2017 (selective classification) · Shao et al. 2024 (GRPO) ·
Williams 1992 (REINFORCE) · Zaheer et al. 2017 / Lee et al. 2019 (Deep Sets / Set Transformer) · Hinton et al. 2015 (distillation) ·
Laya (convaiinnovations) · TypeSafe Jev · Together Tev1.