compact-int8/README.md

metadata

license: apache-2.0
language:
  - en
library_name: transformers
tags:
  - decision-model
  - system-one
  - calibrated-decisions
  - multiple-choice
  - zero-shot-classification
  - falcondec

LightDec v1.0.1 — Falcon Decision

Single-pass, typed (choice / noul / score), calibrated closed-set decision model. Successor to Falconsai/proof_v2.

• Built with FalconDec notebook V2 · backbone jhu-clsp/ettin-encoder-150m · mode finetune · preset full · lineage: jhu-clsp/ettin-encoder-150m → FalconDec-v1.0.0
• Parameters: 159.7M · weights: fp16 305 MB, int8 153 MB
• Layout: [CLS] question [SEP] [MASK] opt1 … [MASK] optk [SEP] state [SEP]; set-transformer option head; temperature per (question type × option-count bucket).

Usage

import importlib.util
from huggingface_hub import snapshot_download
path = snapshot_download("<repo>")            # or a local FalconDec-v1.0.1 directory
spec = importlib.util.spec_from_file_location("falcondec_modeling", f"{path}/falcondec_modeling.py")
fdm = importlib.util.module_from_spec(spec); spec.loader.exec_module(fdm)
model, tok = fdm.load_falcondec(path)          # add "/compact-int8" for the int8 artefact
out = fdm.decide(model, tok, state={"body": "I was charged twice, refund me or I cancel."}, questions={
    "dept": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "payments", "tech": "bugs"}},
    "churn": {"type": "noul", "instructions": "Does the user threaten to leave?"},
    "urgency": {"type": "score", "instructions": "How urgent?", "criteria": ["low", "medium", "high"]}})

Test results

Overall: micro acc 0.781, task-macro acc 0.769, ECE 0.020, NLL 0.529, Brier 0.290, AURC 0.064.

task	domain	heldout	n	chance	acc	proof_v2 (card)	ece
counsel/critique_quality	agentic	False	201	0.333	0.592		0.227
counsel/step_has_error	agentic	False	201	0.500	0.801		0.169
hotpotqa/comparison_yes_no	agentic	False	26	0.500	0.846		0.151
hotpotqa/retrieve	agentic	False	497	0.167	0.873		0.083
sst5/score	classification	True	500	0.200	0.416		0.046
emotion/6way	classification	True	500	0.167	0.502		0.082
yelp/score	classification	False	500	0.200	0.654		0.069
ag_news/topic	classification	False	500	0.250	0.888		0.040
devign/vulnerability	code	False	500	0.500	0.596	0.542	0.050
humaneval/completion	code	True	119	0.394	0.672	0.575	0.107
mbpp/bugspot	code	False	256	0.406	0.730	0.475	0.102
bigclonebench/clone	code	False	500	0.500	0.952	0.383	0.016
codexglue/func_name	code	False	467	0.255	0.953	0.901	0.023
codexglue/doc_to_code	code	False	504	0.279	0.976	0.961	0.010
mbpp/solution	code	False	500	0.250	0.982	0.992	0.015
codexglue/code_to_doc	code	False	504	0.262	0.984	0.969	0.012
codexglue/lang_id	code	False	504	0.235	1.000	0.997	0.001
prompt_injections/detect	guardrails	True	116	0.500	0.578		0.286
agentharm/refuse	guardrails	True	416	0.500	0.743		0.067
civil_comments/toxic	guardrails	False	500	0.500	0.906		0.035
jailbreak/detect	guardrails	False	262	0.500	0.954		0.023
banking77/intent_77	intents	True	300	0.013	0.533		0.147
massive_en/intent	intents	False	500	0.130	0.906		0.022
banking77/intent	intents	True	500	0.321	0.930	0.883	0.040
clinc150/intent	intents	False	500	0.124	0.942	0.850	0.036
policy/invoice_total_transfer	policy	False	500	0.500	0.486		0.032
policy/invoice_overdue_transfer	policy	False	500	0.500	0.600		0.201
policy/sla_urgency_transfer	policy	False	500	0.250	0.698		0.212
policy/free_shipping_transfer	policy	False	500	0.500	0.766		0.101
policy/refund_approval_transfer	policy	False	500	0.333	0.780		0.186
policy/count_threshold_transfer	policy	False	500	0.178	0.846		0.109
policy/table_extreme_transfer	policy	False	500	0.237	0.912		0.014
policy/return_window_transfer	policy	False	500	0.333	0.998		0.020
policy/access_control_transfer	policy	False	500	0.333	1.000		0.000
aqua_rat/math	reasoning	False	247	0.200	0.275		0.042
mmlu/mcq	reasoning	True	500	0.250	0.356		0.109
anli/nli	reasoning	False	498	0.333	0.416		0.184
arc_challenge/mcq	reasoning	True	500	0.250	0.460	0.308	0.088
hellaswag/continuation	reasoning	False	500	0.250	0.580		0.073
arc_easy/mcq	reasoning	True	500	0.250	0.594	0.425	0.038
openbookqa/mcq	reasoning	False	500	0.250	0.610	0.292	0.092
winogrande/blank	reasoning	False	500	0.500	0.612		0.111
commonsense_qa/mcq	reasoning	False	493	0.200	0.639	0.442	0.044
gsm8k/math	reasoning	False	500	0.250	0.662	0.275	0.061
boolq/yes_no	reasoning	False	500	0.500	0.816	0.717	0.027
mnli/claim	reasoning	False	500	0.333	0.838	0.492	0.071
snli/nli	reasoning	False	988	0.333	0.880		0.050
sciq/mcq	reasoning	False	498	0.250	0.954	0.692	0.031
scitail/support	reasoning	False	500	0.500	0.964		0.020
snli/must_be_true	reasoning	False	500	0.333	0.980	0.908	0.029
snli/contradicts	reasoning	False	500	0.333	0.986	0.892	0.014
qasc/mcq	reasoning	False	500	0.125	0.990		0.006
bitext/category	support	False	500	0.161	1.000		0.002
bitext/route	support	False	500	0.190	1.000	0.958	0.001
typed_decisions/agent_trace_observability	workflows	False	500	0.300	0.712		0.120
typed_decisions/security_incidents	workflows	False	500	0.340	0.730		0.144
typed_decisions/customer_service	workflows	False	500	0.280	0.740		0.085
typed_decisions/invoice_processing	workflows	False	500	0.350	0.838		0.122

Training

Strictly proper objective (log + 0.5·spherical + 1.0·RPS for ordinal), soft teacher targets where available, task-balanced sampling (α = 0.5), NOTA augmentation (8%), per-epoch option reshuffling, AdamW + LLRD 0.9, EMA 0.999, 2 epoch(s), RLCD stage off/reverted. Training time 172.2 min on NVIDIA GeForce RTX 5090 Laptop GPU.

Limitations

• Closed world: the model ranks the options you give it. Add "None of the above" when that is a valid answer (it was trained with it).
• Multi-step arithmetic (GSM8K / AQuA) is a System-2 task; route low-confidence answers to an LLM (defer flag, threshold 0.7).
• English only; code judgement (vulnerabilities, clones) is still weak; check calibration on your own traffic.

References

Warner et al. 2024 (ModernBERT) · Weller et al. 2025 (Ettin) · Gneiting & Raftery 2007 (strictly proper scoring rules) · Guo et al. 2017 (temperature scaling) · Geifman & El-Yaniv 2017 (selective classification) · Shao et al. 2024 (GRPO) · Williams 1992 (REINFORCE) · Zaheer et al. 2017 / Lee et al. 2019 (Deep Sets / Set Transformer) · Hinton et al. 2015 (distillation) · Laya (convaiinnovations) · TypeSafe Jev · Together Tev1.