Delete manifest.json

manifest.json

@@ -1,83 +0,0 @@
-{
-  "contract": "https://surgeon.falcons.ai/model-surgeon/manifest/v1",
-  "tool": "FALCONS.AI Model Surgeon V7.99",
-  "copyright": "\u00a9 2026 FALCONS.AI",
-  "exported": "2026-09-26T07:21:44Z",
-  "architecture": {
-    "family": "transformer_nlp",
-    "label": "NLP \u00b7 Small Language Model (SLM)",
-    "confidence": 0.98,
-    "score": 8.92,
-    "tags": [],
-    "runners_up": [
-      {
-        "family": "mlp",
-        "label": "MLP / Tabular",
-        "score": 2.5
-      },
-      {
-        "family": "ssm",
-        "label": "State Space Model (Mamba/S4)",
-        "score": 1.8
-      }
-    ],
-    "total_params": 159654157
-  },
-  "intended_task": null,
-  "execution_plan_source": "name_heuristic",
-  "source_format": "safetensors",
-  "totals": {
-    "params": 159654157,
-    "bytes": 319308314,
-    "tensors": 168,
-    "modules": 226
-  },
-  "removed_nodes": [],
-  "reparented": [],
-  "grafted_components": [],
-  "structure_only_tensors_zero_filled": [],
-  "merged_tensors": [],
-  "lineage": {
-    "parents": [
-      {
-        "role": "primary",
-        "file": "Falconsai/LightDec/model.safetensors",
-        "format": "safetensors"
-      }
-    ],
-    "merges": [],
-    "quantized": [],
-    "head_prunes": [],
-    "source": {
-      "kind": "hub",
-      "repo": "Falconsai/LightDec",
-      "license": "apache-2.0",
-      "file": "",
-      "config": null,
-      "card": {
-        "repo": "Falconsai/LightDec",
-        "revision": "main",
-        "text": "---\r\nlicense: apache-2.0\r\nlanguage:\r\n- en\r\nlibrary_name: transformers\r\nbase_model: jhu-clsp/ettin-encoder-150m\r\npipeline_tag: zero-shot-classification\r\ntags:\r\n- decision-model\r\n- system-one\r\n- falcondec\r\n- lightdec\r\n- calibrated-decisions\r\n- multiple-choice\r\n- intent-classification\r\n- customer-support\r\n- natural-language-inference\r\n- code\r\n- guardrails\r\n- agents\r\n- selective-prediction\r\n- falconsai\r\n---\r\n\r\n# Falconsai/LightDec\r\n\r\n**[View in Model Surgeon](https://surgeon.falcons.ai/?hub=Falconsai/LightDec)**\r\n\r\n**A lightweight, single-pass, typed, calibrated decision model for agentic systems.** Give it a **state** (text, code or JSON), one or more **typed questions** (`choice`, `noul` yes/no, `score` ordinal) and a closed set of options. It returns a calibrated probability for every option, from one encoder pass per question.\r\n\r\nLightDec is the FalconDec architecture trained with **FalconDec notebook V2** on the **`standard`** data preset. That run adds agent-specific decisions (AgentTrek, Counsel, HotpotQA) to a balanced mix of 58 test tasks across 9 domains. This page is both the **model card** and the **developer guide**: how to load it, test it, and use it as a decision component in an agentic system.\r\n\r\n> **At a glance.** Test accuracy **0.725** (micro and task-macro) on 17,498 decisions from 58 tasks, with ECE **0.025**. At a 0.70 confidence threshold, LightDec answers **56%** of decisions at **89.6%** accuracy and defers the rest. Weights: **319 MB** fp16, **161 MB** int8. It is strongest on support routing, code understanding, intents, guardrails and agent-step checks, and weakest on multi-step arithmetic, date and table reasoning, and very wide label sets. Evaluate it on your own traffic (\u00a76.4) before acting on its answers.\r\n\r\n## Contents\r\n\r\n1. [Model summary](#1-model-summary)\r\n2. [Results](#2-results)\r\n3. [Intended and out-of-scope uses](#3-intended-and-out-of-scope-uses)\r\n4. [The Hub repository](#4-the-hub-repository)\r\n5. [Install, load and read a result](#5-install-load-and-read-a-result)\r\n6. [Testing the model](#6-testing-the-model)\r\n7. [Using it in an agentic system](#7-using-it-in-an-agentic-system)\r\n8. [Tuning for your domain](#8-tuning-for-your-domain)\r\n9. [Architecture](#9-architecture)\r\n10. [Training data](#10-training-data)\r\n11. [Training procedure and calibration](#11-training-procedure-and-calibration)\r\n12. [Operational notes](#12-operational-notes)\r\n13. [Bias, risks and limitations](#13-bias-risks-and-limitations)\r\n14. [Versioning and lineage](#14-versioning-and-lineage)\r\n15. [API reference](#15-api-reference)\r\n16. [Citation and references](#16-citation-and-references)\r\n\r\n---\r\n\r\n## 1. Model summary\r\n\r\n| | |\r\n|---|---|\r\n| **Model** | LightDec: the FalconDec architecture, notebook V2, `standard` preset. The checkpoint's own config reports `FalconDec` version `1.0.0` |\r\n| **Task** | Closed-set decisions: given a state, a question and 2\u2013N options, return a calibrated probability per option |\r\n| **Question types** | `choice` (pick one), `noul` (yes/no, returns P(true)), `score` (ordinal rubric, returns the expected level) |\r\n| **Backbone** | [`jhu-clsp/ettin-encoder-150m`](https://huggingface.co/jhu-clsp/ettin-encoder-150m) (ModernBERT-style encoder), fully fine-tuned |\r\n| **Decision head** | Option-marker scoring plus a permutation-equivariant set-transformer head (\u00a79) |\r\n| **Parameters** | \u2248160M (Hub reports 0.2B) |\r\n| **Weights** | fp16 **319 MB** (`model.safetensors`) \u00b7 per-channel int8 **161 MB** (`compact-int8/model_int8.safetensors`) |\r\n| **Context** | 512 tokens; automatically 2,048 for questions with more than 24 options; a tournament above 96 options |\r\n| **Calibration** | One temperature per (question type \u00d7 option-count bucket), stored in the checkpoint and applied automatically |\r\n| **Inference cost** | One encoder pass per question. The same architecture (`Falconsai/proof_v3`) measured 15.7 ms p50 for one question on a GPU; LightDec's own latency is not in its report (measure with \u00a76.3) |\r\n| **Output** | Probabilities, the chosen option, confidence, a `defer` flag, `p_true` (noul) and `expected_level` (score) |\r\n| **Language** | English, plus code in Python, Java, JavaScript, PHP, Ruby, Go and C |\r\n| **Custom code** | `falcondec_modeling.py` ships with the weights and holds the model and all inference logic. Load it with `importlib` (\u00a75); `AutoModel.from_pretrained` alone won't build the decision head |\r\n| **License** | Apache-2.0 for the weights and code. Check each training dataset's license before redistributing derived data |\r\n\r\nThe model has no generative component. It can only rank the options you give it, so it cannot produce text outside that set.\r\n\r\n---\r\n\r\n## 2. Results\r\n\r\nAll numbers come from this checkpoint's `falcondec_report.json`: one run, `standard` preset, 2 epochs, seed 42, `MODE=\"scratch\"`, notebook V2.\r\n\r\n### 2.1 At a glance\r\n\r\n| Metric | Value |\r\n|---|---|\r\n| Test decisions / tasks | 17,498 / 58 (10 of them held out) |\r\n| Test accuracy, micro | **0.725** |\r\n| Test accuracy, task-macro | **0.725** |\r\n| Held-out tasks, task-macro (10 tasks never trained on) | **0.567** |\r\n| Expected calibration error (ECE, 15 bins) | **0.025** |\r\n| Negative log-likelihood / Brier score | 0.652 / 0.358 |\r\n| Area under the risk\u2013coverage curve (AURC, lower is better) | **0.097** |\r\n| Ordinal (`score`) mean absolute error, in levels | 0.572 |\r\n| Coverage / accuracy at confidence \u2265 0.70 | **56.4% / 0.896** |\r\n| Weights | fp16 319 MB \u00b7 int8 161 MB |\r\n\r\n**Selective prediction is the headline.** Calibration is good (ECE 0.025), so the confidence score is a reliable gate. Acting only on decisions with confidence \u2265 0.70 covers 56% of traffic at 89.6% accuracy, against 72.5% accuracy when answering everything. That is the property an agent loop needs: answer the easy majority locally, and hand the rest to an LLM or a human.\r\n\r\n### 2.2 Per domain\r\n\r\n| Domain | Test decisions | Task-macro accuracy |\r\n|---|---|---|\r\n| support | 600 | **0.997** |\r\n| code | 2,350 | **0.864** |\r\n| intents | 1,200 | **0.773** |\r\n| guardrails | 1,094 | **0.768** |\r\n| agentic | 717 | **0.754** |\r\n| workflows | 2,000 | **0.689** |\r\n| policy | 2,700 | **0.669** |\r\n| reasoning | 5,637 | **0.667** |\r\n| classification | 1,200 | **0.584** |\r\n\r\n### 2.3 Per task\r\n\r\n*Held-out tasks were never used for training, calibration or model selection. \"proof_v2 (card)\" lists proof_v2's published score for the same source and task (different samples; indicative only).*\r\n\r\n| Domain | Task | n | Chance | **Accuracy** | ECE | proof_v2 (card) |\r\n|---|---|---|---|---|---|---|\r\n| support | `bitext/route` | 300 | 0.200 | **1.000** | 0.002 | 0.958 |\r\n| support | `bitext/category` | 300 | 0.172 | **0.993** | 0.008 |  |\r\n| code | `codexglue/lang_id` | 300 | 0.235 | **1.000** | 0.004 | 0.997 |\r\n| code | `mbpp/solution` | 300 | 0.250 | **0.987** | 0.014 | 0.992 |\r\n| code | `codexglue/code_to_doc` | 300 | 0.256 | **0.977** | 0.016 | 0.969 |\r\n| code | `codexglue/doc_to_code` | 300 | 0.274 | **0.973** | 0.022 | 0.961 |\r\n| code | `codexglue/func_name` | 275 | 0.263 | **0.938** | 0.021 | 0.901 |\r\n| code | `bigclonebench/clone` | 300 | 0.500 | **0.863** | 0.098 | 0.383 |\r\n| code | `humaneval/completion` *(held out)* | 119 | 0.394 | **0.756** | 0.152 | 0.575 |\r\n| code | `mbpp/bugspot` | 156 | 0.413 | **0.731** | 0.086 | 0.475 |\r\n| code | `devign/vulnerability` | 300 | 0.500 | **0.553** | 0.026 | 0.542 |\r\n| intents | `banking77/intent` *(held out)* | 300 | 0.317 | **0.923** | 0.037 | 0.883 |\r\n| intents | `massive_en/intent` | 300 | 0.122 | **0.907** | 0.046 |  |\r\n| intents | `clinc150/intent` | 300 | 0.122 | **0.793** | 0.081 | 0.850 |\r\n| intents | `banking77/intent_77` *(held out)* | 300 | 0.013 | **0.470** | 0.200 |  |\r\n| guardrails | `jailbreak/detect` | 262 | 0.500 | **0.966** | 0.018 |  |\r\n| guardrails | `civil_comments/toxic` | 300 | 0.500 | **0.807** | 0.051 |  |\r\n| guardrails | `agentharm/refuse` *(held out)* | 416 | 0.500 | **0.654** | 0.178 |  |\r\n| guardrails | `prompt_injections/detect` *(held out)* | 116 | 0.500 | **0.647** | 0.272 |  |\r\n| agentic | `hotpotqa/retrieve` | 298 | 0.168 | **0.842** | 0.078 |  |\r\n| agentic | `hotpotqa/comparison_yes_no` | 17 | 0.500 | **0.824** | 0.185 |  |\r\n| agentic | `counsel/step_has_error` | 201 | 0.500 | **0.791** | 0.093 |  |\r\n| agentic | `counsel/critique_quality` | 201 | 0.333 | **0.557** | 0.094 |  |\r\n| workflows | `typed_decisions/customer_service` | 500 | 0.280 | **0.720** | 0.099 |  |\r\n| workflows | `typed_decisions/security_incidents` | 500 | 0.340 | **0.712** | 0.140 |  |\r\n| workflows | `typed_decisions/agent_trace_observability` | 500 | 0.300 | **0.696** | 0.106 |  |\r\n| workflows | `typed_decisions/invoice_processing` | 500 | 0.350 | **0.628** | 0.101 |  |\r\n| policy | `policy/access_control_transfer` | 300 | 0.333 | **1.000** | 0.001 |  |\r\n| policy | `policy/return_window_transfer` | 300 | 0.333 | **1.000** | 0.019 |  |\r\n| policy | `policy/free_shipping_transfer` | 300 | 0.500 | **0.850** | 0.067 |  |\r\n| policy | `policy/sla_urgency_transfer` | 300 | 0.250 | **0.713** | 0.214 |  |\r\n| policy | `policy/refund_approval_transfer` | 300 | 0.333 | **0.710** | 0.219 |  |\r\n| policy | `policy/count_threshold_transfer` | 300 | 0.179 | **0.523** | 0.085 |  |\r\n| policy | `policy/invoice_total_transfer` | 300 | 0.500 | **0.523** | 0.020 |  |\r\n| policy | `policy/invoice_overdue_transfer` | 300 | 0.500 | **0.407** | 0.364 |  |\r\n| policy | `policy/table_extreme_transfer` | 300 | 0.240 | **0.290** | 0.036 |  |\r\n| reasoning | `qasc/mcq` | 300 | 0.125 | **0.983** | 0.007 |  |\r\n| reasoning | `snli/must_be_true` | 300 | 0.333 | **0.970** | 0.029 | 0.908 |\r\n| reasoning | `snli/contradicts` | 300 | 0.333 | **0.967** | 0.035 | 0.892 |\r\n| reasoning | `scitail/support` | 300 | 0.500 | **0.957** | 0.030 |  |\r\n| reasoning | `sciq/mcq` | 300 | 0.250 | **0.950** | 0.023 | 0.692 |\r\n| reasoning | `snli/nli` | 594 | 0.333 | **0.837** | 0.035 |  |\r\n| reasoning | `mnli/claim` | 300 | 0.333 | **0.807** | 0.072 | 0.492 |\r\n| reasoning | `boolq/yes_no` | 300 | 0.500 | **0.783** | 0.071 | 0.717 |\r\n| reasoning | `gsm8k/math` | 300 | 0.250 | **0.637** | 0.050 | 0.275 |\r\n| reasoning | `commonsense_qa/mcq` | 296 | 0.200 | **0.611** | 0.058 | 0.442 |\r\n| reasoning | `arc_easy/mcq` *(held out)* | 300 | 0.250 | **0.553** | 0.057 | 0.425 |\r\n| reasoning | `openbookqa/mcq` | 300 | 0.250 | **0.550** | 0.068 | 0.292 |\r\n| reasoning | `winogrande/blank` | 300 | 0.500 | **0.540** | 0.089 |  |\r\n| reasoning | `arc_challenge/mcq` *(held out)* | 300 | 0.250 | **0.423** | 0.085 | 0.308 |\r\n| reasoning | `anli/nli` | 300 | 0.333 | **0.393** | 0.185 |  |\r\n| reasoning | `mmlu/mcq` *(held out)* | 300 | 0.250 | **0.393** | 0.094 |  |\r\n| reasoning | `hellaswag/continuation` | 300 | 0.250 | **0.383** | 0.171 |  |\r\n| reasoning | `aqua_rat/math` | 247 | 0.200 | **0.259** | 0.071 |  |\r\n| classification | `ag_news/topic` | 300 | 0.250 | **0.847** | 0.051 |  |\r\n| classification | `yelp/score` | 300 | 0.200 | **0.640** | 0.069 |  |\r\n| classification | `emotion/6way` *(held out)* | 300 | 0.167 | **0.480** | 0.049 |  |\r\n| classification | `sst5/score` *(held out)* | 300 | 0.200 | **0.370** | 0.086 |  |\r\n\r\n### 2.4 Comparison with proof_v2 (indicative)\r\n\r\nOn the 22 tasks that both this report and proof_v2's model card cover, LightDec's task-macro accuracy is **0.807 vs 0.679**, and it scores higher on **20 of 22**. The largest gains are on the tasks proof_v2 reported as weak:\r\n\r\n| Task | proof_v2 (card) | LightDec |\r\n|---|---|---|\r\n| BigCloneBench clone detection | 0.383 | **0.863** |\r\n| GSM8K (4-option numeric) | 0.275 | **0.637** |\r\n| MultiNLI claim | 0.492 | **0.807** |\r\n| OpenBookQA | 0.292 | **0.550** |\r\n| MBPP bug spotting | 0.475 | **0.731** |\r\n| HumanEval completion *(held out)* | 0.575 | **0.756** |\r\n| ARC-Challenge *(held out)* | 0.308 | **0.423** |\r\n\r\nLightDec is lower on CLINC150 (0.793 vs 0.850) and MBPP task\u2192solution (0.987 vs 0.992).\r\n\r\nThese are **different test samples and, for some tasks, different question formats**. For example, LightDec's MultiNLI task is three-way NLI, and its bug-spotting mutants are verified to fail the unit tests. The like-for-like head-to-head, which runs proof_v2 on identical decisions (notebook cell 23), **did not run** for this checkpoint (`head_to_head: null`).\r\n\r\n### 2.5 Comparison with Laya and TypeSafe Jev (indicative)\r\n\r\n| Benchmark | LightDec | Laya | TypeSafe Jev 1.13.0 |\r\n|---|---|---|---|\r\n| typed-decisions test (2,000 decisions, 4 workflows) | 0.689 | 0.766 (fine-tuned) \u00b7 0.362 (zero-shot) | 0.727 |\r\n| AG News (4 labels) | 0.847 | 0.950 | 0.910 |\r\n| DAIR Emotion, 6 labels *(held out for LightDec)* | 0.480 | 0.595 | 0.480 |\r\n| Banking77, all 77 labels in one question *(held out)* | 0.470 | 0.425 | 0.870 (72 labels) |\r\n| SST-5 (ordinal) *(held out)* | 0.370 | 0.372 | \u2014 |\r\n\r\nLaya's numbers are from its own benchmark report; Jev's are third-party published. LightDec was trained on the typed-decisions training split, like the fine-tuned Laya checkpoint. **On typed-decisions LightDec trails both** (the teacher-agreement ceiling is 0.735 and the majority-class baseline 0.461). It matches Jev on Emotion, edges Laya on all-77 Banking77, and trails both on AG News. LightDec is 2.6\u00d7 smaller than Laya's 421M English checkpoint.\r\n\r\n---\r\n\r\n## 3. Intended and out-of-scope uses\r\n\r\n### Intended\r\n\r\n| Use | Measured evidence |\r\n|---|---|\r\n| **Support and ticket routing** | Bitext route 1.000, Bitext category 0.993; Banking77 (held out, 2\u20135 options) 0.923; MASSIVE 0.907 |\r\n| **Code understanding against a menu** | Language ID 1.000; code\u2194description 0.973\u20130.977; task\u2192solution 0.987; function naming 0.938 |\r\n| **Guardrails** | Jailbreak detection 0.966; toxicity 0.807. Held out: prompt-injection 0.647, AgentHarm refusal 0.654, so recalibrate and test on your own traffic |\r\n| **Agent loops** | Retrieval routing (HotpotQA) 0.842; \"does this agent step contain an error?\" (Counsel) 0.791 |\r\n| **Statement verification** | SNLI must-be-true / contradicts 0.970 / 0.967; SciTail 0.957; SciQ 0.950 |\r\n| **Selective automation** | 89.6% accuracy on the 56% of decisions with confidence \u2265 0.70 |\r\n\r\n### Out of scope\r\n\r\n- **Multi-step arithmetic and quantitative reasoning.** AQuA 0.259 (chance 0.20); counting and summing policies 0.52; comparing values in a table 0.290. GSM8K reaches 0.637 only because it is posed as 4-option multiple choice with near-miss distractors. Route real math to an LLM or code.\r\n- **Date reasoning in unfamiliar formats.** The invoice-overdue transfer test (ISO dates, whereas training used \"Month DD, YYYY\") scores **0.407, below chance, with ECE 0.364**. It is confidently wrong there. Normalise dates before asking, or compute them in code.\r\n- **Very wide label sets.** All 77 Banking77 intents in one question score 0.470. Pre-filter to a shortlist of about 20 options (\u00a77.1).\r\n- **Hard commonsense and exam knowledge**: HellaSwag 0.383, ANLI 0.393, MMLU 0.393.\r\n- **Code security and correctness gating**: Devign 0.553 is near chance. Don't use it to approve code.\r\n- **Open-ended questions**: it always picks one of your options. Add \"None of the above\" when appropriate; it was trained with that option.\r\n- **Non-English text**, and **high-stakes decisions without human oversight**.\r\n\r\n---\r\n\r\n## 4. The Hub repository\r\n\r\n| File | Contents |\r\n|---|---|\r\n| `model.safetensors` | fp16 weights (319 MB): encoder, decision head and the temperature buffer |\r\n| `falcondec_config.json` | Layout (sequence lengths, option budgets), special-token ids, temperatures, defer threshold, version, lineage |\r\n| `encoder/` | Backbone configuration (the encoder is rebuilt from this, then the weights are loaded) |\r\n| `tokenizer/` | Tokenizer files |\r\n| `falcondec_modeling.py` | `FalconDec`, `load_falcondec`, `decide`, `score_items`, `save_falcondec` and the int8 codec |\r\n| `falcondec_report.json` | Training configuration, data counts, history, temperatures and all test results |\r\n| `compact-int8/` | The same model with per-channel int8 weights (`model_int8.safetensors`, 161 MB); a complete, self-contained directory with its own config, tokenizer and modeling file |\r\n| `README.md` | This card |\r\n\r\n**Pin a revision in production.** The repo can change, so pass a commit hash when you load.\r\n\r\n---\r\n\r\n## 5. Install, load and read a result\r\n\r\n```bash\r\npip install torch \"transformers>=4.48\" safetensors huggingface_hub numpy\r\n```\r\n\r\n### 5.1 Load\r\n\r\nSave this helper as `lightdec.py` next to your code. Every example below uses it.\r\n\r\n```python\r\n# lightdec.py\r\nimport importlib.util, json, shutil\r\nfrom pathlib import Path\r\nfrom huggingface_hub import snapshot_download\r\n\r\n\r\ndef load_lightdec(repo=\"Falconsai/LightDec\", revision=None, variant=\"fp16\", device=None, dtype=None):\r\n    \"\"\"Returns (fdm, model, tokenizer). variant: \"fp16\" (319 MB) or \"int8\" (161 MB, dequantised on load).\"\"\"\r\n    path = Path(repo) if Path(repo).exists() else Path(snapshot_download(repo, revision=revision))\r\n    if variant == \"int8\":\r\n        path = path / \"compact-int8\"\r\n    fc = json.loads((path / \"falcondec_config.json\").read_text(encoding=\"utf-8\"))\r\n    expected = fc.get(\"weights\", \"model.safetensors\")\r\n    if not (path / expected).exists():                  # e.g. a renamed weight file in a processed copy\r\n        cands = sorted(path.glob(\"*.safetensors\"))\r\n        if not cands:\r\n            raise FileNotFoundError(f\"no .safetensors weights in {path}\")\r\n        local = Path(\"lightdec_local\") / variant\r\n        shutil.copytree(path, local, dirs_exist_ok=True)\r\n        shutil.copy(cands[0], local / expected)\r\n        path = local\r\n    spec = importlib.util.spec_from_file_location(\"falcondec_modeling\", str(path / \"falcondec_modeling.py\"))\r\n    fdm = importlib.util.module_from_spec(spec)\r\n    spec.loader.exec_module(fdm)\r\n    model, tok = fdm.load_falcondec(str(path), device=device, dtype=dtype)   # cuda if available, else cpu\r\n    return fdm, model, tok\r\n```\r\n\r\n```python\r\nfrom lightdec import load_lightdec\r\n\r\nfdm, model, tok = load_lightdec()                      # or load_lightdec(revision=\"<commit>\", variant=\"int8\")\r\nprint(model.fcfg[\"name\"], model.fcfg[\"version\"], round(model.num_parameters() / 1e6, 1), \"M params\")\r\n```\r\n\r\nIf loading prints `[FalconDec] load warning: missing=\u2026 unexpected=\u2026`, the weights didn't match the architecture. Treat that as a failed load (\u00a76.1 checks for it).\r\n\r\n### 5.2 Decide\r\n\r\n`decide()` takes one state and any number of typed questions: either a list, or a Jev/Laya-style dict keyed by name.\r\n\r\n```python\r\nstate = {\"from\": \"user@acme.com\", \"subject\": \"Duplicate charge on invoice #4411\",\r\n         \"body\": \"We were billed twice for March. Please refund the duplicate today or we will cancel our plan.\"}\r\n\r\nout = fdm.decide(model, tok, state, {\r\n    \"department\": {\"type\": \"choice\", \"instructions\": \"Which department should handle this request?\",\r\n                   \"criteria\": {\"billing\": \"invoices, payments, refunds\", \"technical\": \"bugs, outages\",\r\n                                \"sales\": \"pricing, contracts\", \"other\": \"everything else\"}},\r\n    \"urgency\":    {\"type\": \"score\", \"instructions\": \"How urgent is this request?\",\r\n                   \"criteria\": [\"not urgent\", \"soon\", \"critical deadline or blocking issue\"]},\r\n    \"churn_risk\": {\"type\": \"noul\", \"instructions\": \"Does the user threaten to cancel or leave?\"},\r\n})\r\na = out[\"answers\"]\r\nprint(a[\"department\"][\"choice\"], round(a[\"department\"][\"confidence\"], 3), a[\"department\"][\"defer\"])\r\nprint(\"urgency level\", round(a[\"urgency\"][\"expected_level\"], 2), \"of\", 2)\r\nprint(\"P(churn)\", round(a[\"churn_risk\"][\"p_true\"], 3))\r\n```\r\n\r\nPlain options work too: `{\"question\": \"Which team?\", \"options\": [\"Accounts\", \"Billing\", \"Shipping\"]}`.\r\n\r\n### 5.3 Reading a result\r\n\r\nEach item in `out[\"results\"]` (and `out[\"answers\"][key]`) contains:\r\n\r\n| Field | Meaning |\r\n|---|---|\r\n| `key` | The question's name (dict input) or `None` |\r\n| `type` | `choice`, `noul` or `score` |\r\n| `choice` | The chosen key: the criteria key, the option text, `True`/`False` for `noul`, or the level index for `score` |\r\n| `choice_text` | The option text the model saw |\r\n| `confidence` | Calibrated probability of `choice` |\r\n| `probs` | The full distribution, keyed by `str(key)` |\r\n| `defer` | `True` when `confidence` is below the defer threshold (default 0.70, stored in the config): don't act on it (\u00a77.3) |\r\n| `p_true` | `noul` only: calibrated P(yes) |\r\n| `expected_level` | `score` only: probability-weighted level (0 \u2026 k\u22121); better than the argmax for ordinal rubrics (test MAE 0.57 levels) |\r\n\r\nHow to read them:\r\n\r\n- **Low confidence, spread probabilities**: the state doesn't support any option clearly. Defer, or add \"None of the above\".\r\n- **`noul` near 0.5**: genuinely ambiguous. Ask for more information rather than guessing.\r\n- **`score`**: use `expected_level` for thresholds (\"escalate if \u2265 1.5\") rather than `choice`.\r\n\r\n---\r\n\r\n## 6. Testing the model\r\n\r\nTests 6.1\u20136.3 need no labelled data, so run them in CI whenever you change the revision. Test 6.4 is the one that tells you whether to ship.\r\n\r\n### 6.1 Integrity and determinism\r\n\r\nSave as `check_lightdec.py` and run `python check_lightdec.py [revision]`.\r\n\r\n```python\r\nimport contextlib, io, sys\r\nimport numpy as np\r\nfrom lightdec import load_lightdec\r\n\r\nrev = sys.argv[1] if len(sys.argv) > 1 else None\r\nlog = io.StringIO()\r\nwith contextlib.redirect_stdout(log):\r\n    fdm, model, tok = load_lightdec(revision=rev)\r\nassert \"load warning\" not in log.getvalue(), log.getvalue()\r\n\r\nfc = model.fcfg\r\nassert fc[\"name\"] == \"FalconDec\", fc[\"name\"]           # LightDec checkpoints use the FalconDec architecture name\r\nT = model.temperature.float().cpu().numpy()\r\nassert T.shape == (3, 4) and (T > 0).all(), T\r\nq = {\"team\": {\"question\": \"Which team?\", \"options\": [\"recover password\", \"shipping\", \"invoicing\"]}}\r\na = fdm.decide(model, tok, \"I forgot my password and can't sign in.\", q)[\"answers\"][\"team\"]\r\nb = fdm.decide(model, tok, \"I forgot my password and can't sign in.\", q)[\"answers\"][\"team\"]\r\nassert all(abs(a[\"probs\"][k] - b[\"probs\"][k]) < 1e-4 for k in a[\"probs\"]), \"non-deterministic\"\r\nassert abs(sum(a[\"probs\"].values()) - 1) < 1e-3\r\nprint(f\"OK  LightDec (FalconDec v{fc['version']}, notebook {fc.get('notebook_version')})  choice={a['choice']}  \"\r\n      f\"conf={a['confidence']:.3f}  defer_threshold={fc.get('defer_threshold')}\")\r\n```\r\n\r\nThe stored temperatures should read approximately `[[1.707, 1.352, 1.466, 1.349], [1.402 \u00d74], [1.453 \u00d74]]` (\u00a711.2).\r\n\r\n### 6.2 Behavioural tests (pytest)\r\n\r\nSave as `tests/test_lightdec.py` and run `pytest -q`. Set `LIGHTDEC_REVISION` to test a pinned commit.\r\n\r\n```python\r\nimport os, random\r\nimport pytest\r\nfrom lightdec import load_lightdec\r\n\r\n\r\n@pytest.fixture(scope=\"session\")\r\ndef fd():\r\n    return load_lightdec(revision=os.environ.get(\"LIGHTDEC_REVISION\"))\r\n\r\n\r\ndef ask(fd, state, question, options, **kw):\r\n    fdm, model, tok = fd\r\n    return fdm.decide(model, tok, state, [dict(question=question, options=options, **kw)])[\"results\"][0]\r\n\r\n\r\ndef test_probabilities_are_valid(fd):\r\n    r = ask(fd, \"The build failed on main.\", \"What next?\", [\"Revert\", \"Ignore\", \"Retry\"])\r\n    assert all(0 <= p <= 1 for p in r[\"probs\"].values()) and abs(sum(r[\"probs\"].values()) - 1) < 1e-3\r\n\r\n\r\ndef test_typed_outputs(fd):\r\n    fdm, model, tok = fd\r\n    out = fdm.decide(model, tok, \"I was charged twice. Refund me or I'm leaving.\", {\r\n        \"refund\": {\"type\": \"noul\", \"instructions\": \"Does the user ask for a refund?\"},\r\n        \"urgency\": {\"type\": \"score\", \"instructions\": \"How urgent?\", \"criteria\": [\"low\", \"medium\", \"high\"]}})[\"answers\"]\r\n    assert 0 <= out[\"refund\"][\"p_true\"] <= 1 and out[\"refund\"][\"choice\"] in (True, False)\r\n    assert 0 <= out[\"urgency\"][\"expected_level\"] <= 2\r\n\r\n\r\ndef test_support_routing(fd):\r\n    r = ask(fd, \"I forgot my password and the reset email never arrived.\", \"Which team should handle this?\",\r\n            [\"recover password\", \"billing and payment\", \"delivery information\"])\r\n    assert r[\"choice\"] == \"recover password\"\r\n\r\n\r\ndef test_fanout_matches_single_questions(fd):\r\n    # Batching changes padding; under bf16 that moves probabilities slightly, never the substance.\r\n    fdm, model, tok = fd\r\n    state = \"I forgot my password and can't sign in.\"\r\n    qs = [{\"question\": \"Team?\", \"options\": [\"recover password\", \"shipping\", \"invoicing\"]},\r\n          {\"question\": \"Urgent?\", \"options\": [\"Yes\", \"No\"]}]\r\n    together = fdm.decide(model, tok, state, qs)[\"results\"]\r\n    for q, t in zip(qs, together):\r\n        alone = fdm.decide(model, tok, state, [q])[\"results\"][0]\r\n        assert all(abs(alone[\"probs\"][k] - t[\"probs\"][k]) < 2e-2 for k in alone[\"probs\"])\r\n\r\n\r\ndef test_option_order_is_mostly_irrelevant(fd):\r\n    # The head is order-equivariant, but the encoder sees positions; training reshuffled options every epoch.\r\n    state, q = \"Where is my parcel? It's three days late.\", \"What should support do?\"\r\n    opts = [\"Give the delivery status\", \"Start a refund\", \"Book an appointment\"]\r\n    base = ask(fd, state, q, opts)[\"choice\"]\r\n    same = sum(ask(fd, state, q, random.Random(s).sample(opts, len(opts)))[\"choice\"] == base for s in range(5))\r\n    assert same >= 4\r\n\r\n\r\ndef test_many_options_use_the_tournament(fd):\r\n    opts = [f\"topic number {i}\" for i in range(119)] + [\"reset my password\"]\r\n    r = ask(fd, \"I can't log in, I need to reset my password.\", \"What does the user want?\", opts)\r\n    assert len(r[\"probs\"]) == 120 and abs(sum(r[\"probs\"].values()) - 1) < 1e-3\r\n\r\n\r\ndef test_int8_agrees_with_fp16(fd):\r\n    fdm8, m8, tok8 = load_lightdec(revision=os.environ.get(\"LIGHTDEC_REVISION\"), variant=\"int8\")\r\n    fdm, model, tok = fd\r\n    items = [dict(state=s, question=\"Which team?\", options=[\"billing\", \"shipping\", \"accounts\", \"technical\"])\r\n             for s in [\"I was double charged\", \"Where is my parcel?\", \"Change my email\", \"The app crashes on start\",\r\n                       \"Refund the duplicate payment\", \"Package never arrived\", \"Reset my login\", \"Error 500 on checkout\"]]\r\n    a = [p.argmax() for p in fdm.score_items(model, tok, items)]\r\n    b = [p.argmax() for p in fdm8.score_items(m8, tok8, items)]\r\n    assert sum(x == y for x, y in zip(a, b)) >= len(items) - 1\r\n```\r\n\r\n### 6.3 Latency\r\n\r\n```python\r\nimport time, numpy as np, torch\r\nfrom lightdec import load_lightdec\r\n\r\nfdm, model, tok = load_lightdec()\r\nq = [{\"question\": \"Route?\", \"options\": [\"billing and payment\", \"shipping\", \"recover password\"]}]\r\nfor _ in range(5):\r\n    fdm.decide(model, tok, \"I was charged twice.\", q)\r\nt = []\r\nfor _ in range(100):\r\n    if torch.cuda.is_available(): torch.cuda.synchronize()\r\n    t0 = time.perf_counter(); fdm.decide(model, tok, \"I was charged twice.\", q)\r\n    if torch.cuda.is_available(): torch.cuda.synchronize()\r\n    t.append((time.perf_counter() - t0) * 1000)\r\nprint(f\"p50 {np.percentile(t, 50):.1f} ms   p95 {np.percentile(t, 95):.1f} ms   on {model.device}\")\r\n```\r\n\r\nFor CPU serving, load the int8 variant with `device=\"cpu\", dtype=torch.float32`, and optionally apply `torch.ao.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)` for int8 matrix multiplies.\r\n\r\n### 6.4 Accuracy on your own labelled data\r\n\r\nWrite 50\u2013500 decisions that look like your real traffic, one JSON object per line. `expected` may be a letter, a 0-based index or the option text; `type` is optional.\r\n\r\n```json\r\n{\"id\": \"t1\", \"tag\": \"support\", \"state\": \"\u2026\", \"question\": \"\u2026\", \"options\": [\"\u2026\", \"\u2026\"], \"expected\": \"B\", \"type\": \"choice\"}\r\n```\r\n\r\n```python\r\nimport json, string\r\nimport numpy as np\r\nfrom lightdec import load_lightdec\r\n\r\nfdm, model, tok = load_lightdec()\r\nrows = [json.loads(l) for l in open(\"my_eval.jsonl\", encoding=\"utf-8\") if l.strip()]\r\n\r\ndef idx(v, opts):\r\n    if isinstance(v, int): return v\r\n    v = str(v).strip()\r\n    if len(v) == 1 and v.upper() in string.ascii_uppercase[:len(opts)]: return string.ascii_uppercase.index(v.upper())\r\n    return [o.lower() for o in opts].index(v.lower())\r\n\r\nitems = [dict(state=r[\"state\"], question=r.get(\"question\", \"\"), options=r[\"options\"], type=r.get(\"type\", \"choice\"))\r\n         for r in rows]\r\nprobs = fdm.score_items(model, tok, items, batch_size=64)\r\ngold = np.array([idx(r[\"expected\"], r[\"options\"]) for r in rows])\r\npred = np.array([p.argmax() for p in probs]); conf = np.array([p.max() for p in probs]); ok = pred == gold\r\n\r\ndef ece(c, k, bins=15):\r\n    e = 0.0\r\n    for lo in np.linspace(0, 1, bins, endpoint=False):\r\n        m = (c > lo) & (c <= lo + 1 / bins)\r\n        if m.any(): e += m.mean() * abs(c[m].mean() - k[m].mean())\r\n    return e\r\n\r\nprint(f\"accuracy {ok.mean():.3f} | ECE {ece(conf, ok):.3f}\")\r\nfor tag in sorted({r.get(\"tag\", \"all\") for r in rows}):\r\n    m = np.array([r.get(\"tag\", \"all\") == tag for r in rows]); print(f\"  {tag:12s} n={m.sum():4d} acc={ok[m].mean():.3f}\")\r\nfor t in (0.5, 0.6, 0.7, 0.8, 0.9):\r\n    m = conf >= t\r\n    print(f\"  act if conf >= {t}: answers {m.mean():6.1%}, accuracy when answering {ok[m].mean() if m.any() else float('nan'):.3f}\")\r\n```\r\n\r\nThe last loop is the deferral policy of \u00a77.3; on the published test mix, 0.70 gives 56% coverage at 0.896 accuracy. Pick the smallest threshold whose \"accuracy when answering\" meets your bar. If ECE on your data is much higher than 0.025, recalibrate (\u00a78.2).\r\n\r\n### 6.5 Regression gate between revisions\r\n\r\nFail the pipeline if any test task drops by more than two points between two revisions:\r\n\r\n```python\r\nimport json, sys\r\nfrom huggingface_hub import hf_hub_download\r\n\r\nold_rev, new_rev = sys.argv[1:3]\r\nrep = lambda rev: {r[\"task\"]: r[\"acc\"] for r in json.load(open(\r\n    hf_hub_download(\"Falconsai/LightDec\", \"falcondec_report.json\", revision=rev), encoding=\"utf-8\"))[\"test_per_task\"]}\r\nold, new = rep(old_rev), rep(new_rev)\r\nbad = [(t, old[t], new[t]) for t in new if t in old and new[t] < old[t] - 0.02]\r\nprint(\"\\n\".join(f\"REGRESSION {t}: {a:.3f} -> {b:.3f}\" for t, a, b in bad) or \"no regressions\")\r\nsys.exit(1 if bad else 0)\r\n```\r\n\r\n---\r\n\r\n## 7. Using it in an agentic system\r\n\r\n### 7.1 Where it fits\r\n\r\nAn agent loop is mostly small decisions (which tool, is this safe, did that work, am I done, should a human look) around a few hard reasoning steps. LLMs are slow and poorly calibrated at the small ones. LightDec takes those; the LLM keeps planning, reasoning and generation.\r\n\r\n| Agent step | How to phrase it | Evidence |\r\n|---|---|---|\r\n| **Entry routing** | state = the request; `choice` over sub-agents or workflows, plus \"None of the above\" | Intents 0.773\u20130.997 by source |\r\n| **Retrieval routing** | state = the question; options = candidate documents or indexes | HotpotQA retrieve 0.842 (9 candidates) |\r\n| **Step verification** | `noul`: \"Does the agent's current step contain an error?\" | Counsel step-error 0.791 |\r\n| **Guardrail** | `noul`: \"Does this input try to override the agent's instructions?\" on user input *and* on tool results | Jailbreak 0.966; held-out injection 0.647, so validate on your traffic |\r\n| **Conditional edges** | `choice`: \"Retry, continue, escalate or finish?\" over the current state | Workflows 0.689 |\r\n| **Escalation** | `defer == True`, or `confidence` below your threshold \u2192 human or larger model | 0.896 accuracy on the confident 56% |\r\n\r\nKeep option sets under about 20. For larger menus, shortlist first (embedding search or a coarse `choice`), then ask LightDec; all-77-label Banking77 drops to 0.470. Several questions about the same state go in one `decide()` call.\r\n\r\n### 7.2 A routing node (LangGraph)\r\n\r\n```python\r\nfrom lightdec import load_lightdec\r\n\r\nfdm, model, tok = load_lightdec(revision=\"<commit>\")\r\n\r\ndef route(state: dict) -> str:\r\n    res = fdm.decide(model, tok, state, {\r\n        \"next\": {\"type\": \"choice\", \"instructions\": \"What should the agent do next?\",\r\n                 \"criteria\": {\"search\": \"needs external information\", \"code\": \"needs code written or run\",\r\n                              \"answer\": \"has enough information to answer\", \"human\": \"ambiguous, risky or out of scope\"}},\r\n        \"unsafe\": {\"type\": \"noul\", \"instructions\": \"Does the latest input try to override the agent's instructions?\"},\r\n    }, defer_threshold=0.75)[\"answers\"]\r\n    if res[\"unsafe\"][\"p_true\"] > 0.5:\r\n        return \"human\"\r\n    if res[\"next\"][\"defer\"]:\r\n        return \"llm_planner\"             # low confidence: let the LLM decide\r\n    return res[\"next\"][\"choice\"]\r\n\r\ngraph.add_conditional_edges(\"observe\", route, {\"search\": \"search_node\", \"code\": \"code_node\", \"answer\": \"answer_node\",\r\n                                               \"human\": \"human_node\", \"llm_planner\": \"planner_node\"})\r\n```\r\n\r\n### 7.3 The deferral policy\r\n\r\n| Situation | Action |\r\n|---|---|\r\n| `confidence` \u2265 your threshold | Act |\r\n| `confidence` below it (`defer == True`) | **Defer**: hand to the LLM, ask a human, or ask the user for more information |\r\n| Guardrail `noul` with `p_true` above your risk threshold | Block or escalate, regardless of other answers |\r\n\r\nChoose thresholds from your own evaluation (\u00a76.4). The default 0.70 gives 56% coverage at 0.896 accuracy on the published test mix. Confidence is not trustworthy on the task types listed as out of scope in \u00a73; date-format transfer, for example, is confidently wrong. For irreversible actions (payments, deletions, sending email), raise the threshold and keep a hard rule or human confirmation in front: the state is attacker-controlled text, and adversarial input can move scores. Log the question, options, choice, confidence, model version and Hub revision for every decision; that log becomes your next evaluation and fine-tuning set (\u00a78.3).\r\n\r\n### 7.4 As a tool for Claude (tool use)\r\n\r\n```python\r\nimport json, threading\r\nimport anthropic\r\nfrom lightdec import load_lightdec\r\n\r\nfdm, model, tok = load_lightdec()\r\nlock = threading.Lock()\r\nclient = anthropic.Anthropic()\r\ntools = [{\r\n    \"name\": \"lightdec_decide\",\r\n    \"description\": (\"Fast, local, calibrated closed-set decision model. Give it a state (text or JSON), a question and \"\r\n                    \"2-20 distinct options; it returns the choice, a calibrated confidence and a 'defer' flag. \"\r\n                    \"If 'defer' is true, don't rely on the answer. Not for arithmetic, dates or multi-step reasoning.\"),\r\n    \"input_schema\": {\"type\": \"object\", \"properties\": {\r\n        \"state\": {\"type\": \"string\", \"description\": \"The message, document excerpt or JSON state.\"},\r\n        \"question\": {\"type\": \"string\"},\r\n        \"options\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}, \"minItems\": 2},\r\n        \"type\": {\"type\": \"string\", \"enum\": [\"choice\", \"score\"], \"description\": \"score = options are ordered levels\"}},\r\n        \"required\": [\"state\", \"question\", \"options\"]},\r\n}]\r\n\r\ndef run_tool(inp):\r\n    with lock:\r\n        r = fdm.decide(model, tok, inp[\"state\"], [{\"question\": inp[\"question\"], \"options\": inp[\"options\"],\r\n                                                   \"type\": inp.get(\"type\", \"choice\")}])[\"results\"][0]\r\n    return {k: r[k] for k in (\"choice\", \"confidence\", \"defer\", \"probs\") if k in r}\r\n\r\nmessages = [{\"role\": \"user\", \"content\": \"Triage: 'I forgot my password and the reset email never arrived.' \"\r\n                                        \"Teams: Accounts, Billing, Shipping.\"}]\r\nwhile True:\r\n    resp = client.messages.create(model=\"claude-sonnet-5\", max_tokens=1024, tools=tools, messages=messages)\r\n    if resp.stop_reason != \"tool_use\":\r\n        print(\"\".join(b.text for b in resp.content if b.type == \"text\"))\r\n        break\r\n    messages.append({\"role\": \"assistant\", \"content\": resp.content})\r\n    results = []\r\n    for block in resp.content:\r\n        if block.type == \"tool_use\" and block.name == \"lightdec_decide\":\r\n            try:\r\n                results.append({\"type\": \"tool_result\", \"tool_use_id\": block.id, \"content\": json.dumps(run_tool(block.input))})\r\n            except Exception as exc:\r\n                results.append({\"type\": \"tool_result\", \"tool_use_id\": block.id, \"content\": str(exc), \"is_error\": True})\r\n    messages.append({\"role\": \"user\", \"content\": results})\r\n```\r\n\r\n### 7.5 As an MCP server\r\n\r\n`pip install mcp`, then save `lightdec_mcp.py` next to `lightdec.py`:\r\n\r\n```python\r\nimport os, threading\r\nfrom mcp.server.fastmcp import FastMCP\r\nfrom lightdec import load_lightdec\r\n\r\nfdm, model, tok = load_lightdec(revision=os.environ.get(\"LIGHTDEC_REVISION\"),\r\n                                variant=os.environ.get(\"LIGHTDEC_VARIANT\", \"fp16\"))\r\nMIN_CONF = float(os.environ.get(\"LIGHTDEC_MIN_CONF\", \"0.7\"))\r\nlock = threading.Lock()\r\nmcp = FastMCP(\"lightdec\")\r\n\r\n\r\n@mcp.tool()\r\ndef decide(state: str, question: str, options: list[str], type: str = \"choice\") -> dict:\r\n    \"\"\"Choose one of 2-20 distinct options for a question about a state. type=\"score\" means ordered levels.\r\n    Returns the choice, a calibrated confidence and 'defer' (true = not reliable enough to act on).\"\"\"\r\n    with lock:\r\n        r = fdm.decide(model, tok, state, [{\"question\": question, \"options\": options, \"type\": type}],\r\n                       defer_threshold=MIN_CONF)[\"results\"][0]\r\n    return {k: r[k] for k in (\"choice\", \"confidence\", \"defer\", \"probs\", \"expected_level\") if k in r}\r\n\r\n\r\n@mcp.tool()\r\ndef decide_many(state: str, questions: dict) -> dict:\r\n    \"\"\"Several typed questions about one state: {name: {\"type\": \"choice\"|\"noul\"|\"score\",\r\n    \"instructions\": str, \"criteria\": {key: description} | [levels]}}.\"\"\"\r\n    with lock:\r\n        return fdm.decide(model, tok, state, questions, defer_threshold=MIN_CONF)[\"answers\"]\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    mcp.run()\r\n```\r\n\r\nRegister it in Claude Desktop's `claude_desktop_config.json`:\r\n\r\n```json\r\n{\r\n  \"mcpServers\": {\r\n    \"lightdec\": {\r\n      \"command\": \"C:\\\\path\\\\to\\\\python.exe\",\r\n      \"args\": [\"C:\\\\path\\\\to\\\\lightdec_mcp.py\"],\r\n      \"env\": { \"LIGHTDEC_REVISION\": \"<commit>\", \"LIGHTDEC_VARIANT\": \"int8\" }\r\n    }\r\n  }\r\n}\r\n```\r\n\r\n### 7.6 As an HTTP microservice\r\n\r\n`pip install fastapi uvicorn`, then save `serve_lightdec.py`:\r\n\r\n```python\r\nimport threading\r\nfrom fastapi import FastAPI, HTTPException\r\nfrom pydantic import BaseModel\r\nfrom lightdec import load_lightdec\r\n\r\nfdm, model, tok = load_lightdec()\r\nlock = threading.Lock()\r\napp = FastAPI(title=\"LightDec\")\r\n\r\n\r\nclass Decide(BaseModel):\r\n    state: str | dict\r\n    questions: dict | list\r\n    defer_threshold: float = 0.7\r\n\r\n\r\n@app.get(\"/health\")\r\ndef health():\r\n    return {\"ok\": True, \"model\": \"LightDec\", \"version\": model.fcfg.get(\"version\")}\r\n\r\n\r\n@app.post(\"/v1/decide\")\r\ndef decide(req: Decide):\r\n    try:\r\n        with lock:\r\n            return fdm.decide(model, tok, req.state, req.questions, defer_threshold=req.defer_threshold)\r\n    except (ValueError, KeyError, TypeError) as exc:\r\n        raise HTTPException(422, str(exc))\r\n```\r\n\r\nRun it with `uvicorn serve_lightdec:app --port 9904 --workers 1`. Each worker holds its own copy of the model; scale out with more processes.\r\n\r\n### 7.7 Without an LLM: a support intake step\r\n\r\n```python\r\nfrom lightdec import load_lightdec\r\n\r\nfdm, model, tok = load_lightdec()\r\nQUEUES = {\"billing\": \"charges, invoices, payments, refunds\", \"returns\": \"returning or exchanging items\",\r\n          \"delivery\": \"shipping status, late or missing parcels\", \"accounts\": \"login, password, profile\",\r\n          \"human\": \"complaints or requests to speak to a person\"}\r\n\r\ndef intake(message: str) -> dict:\r\n    a = fdm.decide(model, tok, message, {\r\n        \"queue\": {\"type\": \"choice\", \"instructions\": \"Which team should handle this message?\", \"criteria\": QUEUES},\r\n        \"urgent\": {\"type\": \"noul\", \"instructions\": \"Does this need a reply within the hour?\"},\r\n    })[\"answers\"]\r\n    if a[\"queue\"][\"defer\"]:\r\n        return {\"action\": \"human_review\", \"suggestion\": a[\"queue\"][\"choice\"],\r\n                \"reason\": f\"low confidence ({a['queue']['confidence']:.0%})\"}\r\n    return {\"action\": \"enqueue\", \"queue\": a[\"queue\"][\"choice\"],\r\n            \"priority\": \"high\" if a[\"urgent\"][\"p_true\"] >= 0.5 else \"normal\",\r\n            \"evidence\": {\"confidence\": round(a[\"queue\"][\"confidence\"], 3), \"model\": \"LightDec\"}}\r\n```\r\n\r\n---\r\n\r\n## 8. Tuning for your domain\r\n\r\n### 8.1 Change the deferral threshold\r\n\r\nPass `defer_threshold=` to `decide()`, or set `model.fcfg[\"defer_threshold\"]`. This changes nothing in the model and is usually enough.\r\n\r\n### 8.2 Recalibrate on your data\r\n\r\nIf ECE on your traffic (\u00a76.4) is noticeably worse than 0.025, fit one extra temperature on top of the stored ones and save a recalibrated copy:\r\n\r\n```python\r\nimport numpy as np, torch\r\n# probs, gold: from \u00a76.4 (probabilities already include the stored temperatures)\r\nlogp = [np.log(np.clip(p, 1e-12, 1)) for p in probs]\r\ndef nll(s):\r\n    return -np.mean([(lp / s)[g] - np.log(np.exp(lp / s).sum()) for lp, g in zip(logp, gold)])\r\ns = min(np.linspace(0.5, 3.0, 51), key=nll)\r\nprint(\"extra temperature\", s)\r\nwith torch.no_grad():\r\n    model.temperature.mul_(float(s))\r\nfdm.save_falcondec(model, tok, \"lightdec_recalibrated\")            # add int8=True for the compact variant\r\n```\r\n\r\nFit on one labelled set and measure on another.\r\n\r\n### 8.3 Fine-tune on your own decisions\r\n\r\nUse the FalconDec training notebook (V2):\r\n\r\n1. Export your logged and corrected decisions as JSONL: `{\"state\", \"question\", \"options\", \"answer\", \"type\"?, \"task\"?}`.\r\n2. In cell 2, set `MODE=\"finetune\"`, `FINETUNE_FROM=\"Falconsai/LightDec\"`, `CUSTOM_DATA_JSONL=\"your_file.jsonl\"`, and a preset.\r\n3. Run the notebook. The version bumps automatically (1.0.0 \u2192 1.0.1), and the report records the lineage.\r\n\r\nYour tasks are up-weighted (`CUSTOM_WEIGHT`), while the public tasks keep the model general. Adding decisions with ISO-format dates and numeric tables is the most direct fix for the transfer weaknesses in \u00a73. Check the regression gate (\u00a76.5) before publishing.\r\n\r\n### 8.4 Save and publish\r\n\r\n```python\r\nfdm.save_falcondec(model, tok, \"lightdec_out\")                     # fp16\r\nfdm.save_falcondec(model, tok, \"lightdec_out/compact-int8\", int8=True)\r\n\r\nfrom huggingface_hub import HfApi                                   # needs a write token: huggingface-cli login\r\nHfApi().upload_folder(folder_path=\"lightdec_out\", repo_id=\"Falconsai/LightDec\", commit_message=\"LightDec v1.0.x\")\r\n```\r\n\r\n---\r\n\r\n## 9. Architecture\r\n\r\n```\r\n[CLS] question [SEP] [MASK] option\u2081 [MASK] option\u2082 \u2026 [MASK] option\u2096 [SEP] state [SEP]\r\n        \u2502\r\n  Ettin-150M encoder (22 layers, hidden 768)\r\n        \u2502\r\n  hidden state at each [MASK]  +  CLS context  +  question-type embedding\r\n        \u2502\r\n  set transformer: 2 layers, 8 heads, no positional encoding \u2192 options attend to each other, order-equivariant\r\n        \u2502\r\n  MLP \u2192 one logit per option \u2192 \u00f7 temperature[type, option-count bucket] \u2192 softmax\r\n```\r\n\r\n- **Option markers** (the approach Laya uses): every option is read at its own `[MASK]` token, so all options are scored in **one** pass.\r\n- **Question first, state last**: when the input is too long, the tail of the state is truncated, never the options.\r\n- **Adaptive option budget**: up to 24 tokens per option within a 192-token head budget that grows with the option count, and 2,048-token sequences above 24 options, so labels stay distinct. Above 96 options, `decide()` runs a tournament.\r\n- **Typed primitives**: `noul` is rendered as a neutral two-option Yes/No choice; `score` keeps its level order and reports an expected level.\r\n- **Calibration** lives in the model: a 3 \u00d7 4 temperature table (question type \u00d7 option-count bucket: \u22642, 3\u20135, 6\u201312, >12).\r\n- **int8 storage**: per-output-channel symmetric int8 for every weight matrix, fp16 elsewhere, dequantised on load.\r\n\r\n---\r\n\r\n## 10. Training data\r\n\r\n**155,747 training, 13,148 validation and 17,498 test decisions** from 58 tasks. No source failed to load (`skipped_builders` is empty). Every example is a **decision**: `state`, `question`, `options`, `answer`, plus type, task and domain.\r\n\r\n| Domain | Sources | Decisions built |\r\n|---|---|---|\r\n| Support | Bitext customer support | Intent routing and category routing; 30% of messages wrapped as JSON program state |\r\n| Intents | CLINC150, MASSIVE (en), Banking77 *(held out)* | Intents as runtime-defined options, 2\u201348 per question; Banking77 also as one 77-option question |\r\n| Code | CodeXGLUE code-to-text (6 languages), Devign, BigCloneBench, MBPP, HumanEval *(held out)* | Language ID, code\u2194description, function naming, vulnerability, clones, task\u2192solution; bug spotting against single-fault mutants **verified to fail the unit tests** |\r\n| Guardrails | Jailbreak classification, Civil Comments; deepset prompt-injections and AgentHarm *(held out)* | `noul` detection and refusal |\r\n| Agentic | Counsel (human meta-evaluations of agent-step critiques), HotpotQA, AgentTrek | Step-error and critique-quality, retrieval routing and comparison yes/no; AgentTrek next-action type and finish-now. AgentTrek loaded without errors, but none of its decisions appear in the test split, so its contribution isn't measured |\r\n| Workflows | LocalLLaMA/typed-decisions | `choice`/`noul`/`score` with the teacher's soft probabilities; test = the benchmark's 2,000-decision test split |\r\n| Policy | Synthetic, executable (in-notebook) | Return windows, approval tiers, AND/OR eligibility, overdue invoices, table look-ups, counting, SLA urgency, access control, invoice totals; the test split uses **transfer** wording, currencies and date formats |\r\n| Reasoning | ARC-Easy/Challenge and MMLU *(held out)*, OpenBookQA, SciQ, CommonsenseQA, QASC, HellaSwag, WinoGrande, MMLU auxiliary-train, BoolQ, GSM8K, AQuA-RAT, SNLI, MultiNLI, ANLI, SciTail | Multiple choice, yes/no, NLI, numeric answers with near-miss distractors |\r\n| Classification | AG News, Yelp (ordinal); DAIR Emotion and SST-5 *(held out)* | Topic; 5-level `score` sentiment |\r\n\r\n**Augmentation**: options are reshuffled every epoch (ordinal levels keep their order). In 8% of choice questions the gold answer is removed and \"None of the above\" becomes correct; in another 4% it is added as a distractor. **Balance**: tasks are sampled with p \u221d n^0.5 each epoch. **Leak guard**: training decisions whose (state, question) appears in validation or test were removed. Mind2Web was excluded (opt-in in the notebook). Held-out sources were never used for training, calibration or model selection.\r\n\r\nCheck each dataset's card for its license before redistributing derived data. AgentHarm is used only as a held-out evaluation, in line with its intended use.\r\n\r\n---\r\n\r\n## 11. Training procedure and calibration\r\n\r\n### 11.1 Setup\r\n\r\n| Setting | Value |\r\n|---|---|\r\n| Mode / preset | `scratch` from the pretrained backbone / `standard` |\r\n| Data caps | \u22644,000 train, \u2264300 validation, \u2264300 test decisions per source split |\r\n| Epochs | 2 (best: epoch 2) |\r\n| Objective | Strictly proper scoring rules: log score + 0.5 \u00d7 spherical score, + 1.0 \u00d7 ranked probability score for `score` questions; soft targets (50/50 with the hard label) where a teacher distribution exists. These are the RLCD rewards, optimised with exact gradients |\r\n| Optional RLCD stage | Off |\r\n| Optimiser | AdamW (\u03b2 0.9/0.98, weight decay 0.01), encoder LR 4e-5 with layer-wise decay 0.9, head LR 3e-4, 6% warm-up, cosine decay, gradient clipping 1.0 |\r\n| Batching | Token-budget batches (16,384 tokens, \u226432 decisions), length-bucketed |\r\n| Weights kept | EMA of the weights (decay 0.999), best validation task-macro accuracy |\r\n| Precision | bf16 autocast, TF32 matmuls; attention `auto` (FlashAttention-2 if installed, else SDPA) |\r\n| Sequence | 512 tokens; 2,048 above 24 options; question \u226496 tokens; \u226424 tokens per option (more for code options) |\r\n| Hardware / time | NVIDIA GeForce RTX 5090 Laptop GPU \u00b7 **46 minutes** |\r\n| Software | Python 3.14.4 \u00b7 PyTorch 2.11.0+cu128 \u00b7 transformers 5.17.0 |\r\n| Seed | 42 |\r\n\r\n| Epoch | Train loss | Train acc | Val macro | Val micro | Val NLL |\r\n|---|---|---|---|---|---|\r\n| 1 | 0.868 | 0.654 | 0.731 | 0.738 | 0.565 |\r\n| 2 | 0.540 | 0.802 | **0.763** | 0.775 | 0.520 |\r\n\r\nValidation was still improving at epoch 2, so a longer schedule (the `full` preset) is likely to help.\r\n\r\n### 11.2 Fitted temperatures\r\n\r\n| Question type | k \u2264 2 | k 3\u20135 | k 6\u201312 | k > 12 |\r\n|---|---|---|---|---|\r\n| choice | 1.707 | 1.352 | 1.466 | 1.349 |\r\n| noul | 1.402 | 1.402 | 1.402 | 1.402 |\r\n| score | 1.453 | 1.453 | 1.453 | 1.453 |\r\n\r\nAll temperatures are above 1, so the raw model was over-confident, as Laya's checkpoints are. `noul` and `score` use a single per-type temperature because their questions fall into one option-count bucket (2 options for `noul`, mostly 3\u20135 levels for `score`).\r\n\r\n---\r\n\r\n## 12. Operational notes\r\n\r\n- **Concurrency.** `decide()` isn't internally locked. Serialise calls with a lock per process (as in \u00a77.4\u20137.6) and scale out with processes.\r\n- **Hardware.** On a GPU, expect tens of milliseconds per call; measure yours with \u00a76.3. On CPU, use the int8 variant (loaded as fp32) and optionally dynamic int8 quantisation.\r\n- **Input length.** 512 tokens by default. The question and options come first, so an over-long state loses its *end*. Put the decisive information early, or summarise.\r\n- **Determinism.** Repeated identical calls give identical results on the same hardware and library versions. Under bf16 on GPU, batching different questions together changes padding and can move probabilities slightly (about 1e-2); near-ties can flip. Decisions with a clear margin don't change.\r\n- **Traceability.** Log the model version, the Hub revision and the variant (fp16/int8) with every decision.\r\n- **Offline use.** After the first download, set `HF_HUB_OFFLINE=1`, or save a local copy and load it by path.\r\n\r\n---\r\n\r\n## 13. Bias, risks and limitations\r\n\r\n- **Quantitative and date reasoning.** Arithmetic (AQuA 0.259), table comparisons (0.290) and counting or summing (0.523) are near chance. Overdue-invoice checks with ISO dates score 0.407 with ECE 0.364, meaning confidently wrong.\r\n- **Wide option sets.** Accuracy falls to 0.470 with all 77 Banking77 intents in one question; shortlist first.\r\n- **Generalisation gap.** Held-out tasks average 0.567 against 0.725 overall. Expect lower accuracy on traffic unlike the training mix, and measure it (\u00a76.4).\r\n- **Guardrail transfer.** In-distribution jailbreak detection is 0.966, but held-out prompt-injection (0.647) and AgentHarm refusal (0.654) are much lower, with ECE 0.272 and 0.178. Don't rely on it as the only safety layer.\r\n- **No completed head-to-head with proof_v2.** The comparisons in \u00a72.4\u20132.5 use published numbers on different samples.\r\n- **Closed world.** The model always picks one of your options. Add \"None of the above\" when appropriate.\r\n- **Distribution-dependent calibration.** ECE was measured on this test mix; re-check it on your traffic and recalibrate if needed (\u00a78.2).\r\n- **Adversarial input.** The state is untrusted text. The model can't be instructed like an LLM, but crafted input can shift its scores. Don't make it the only safeguard before irreversible actions.\r\n- **Data provenance.** Training data is English, largely crowd-sourced, templated, synthetic or scraped from public code and web tasks. Biases in these sources and in the backbone's pre-training can carry into decisions. Automated routing can systematically misroute users whose phrasing differs from the training data (dialects, non-native speakers, assistive phrasing); monitor misroutes by group where possible.\r\n- **Oversight.** Not for high-stakes decisions without human review.\r\n\r\n---\r\n\r\n## 14. Versioning and lineage\r\n\r\n| | proof_V_1 | proof_v2 | proof_v3 | **LightDec** |\r\n|---|---|---|---|---|\r\n| Model | falconsproof v1 | falconsproof v2.0.0 | FalconDec v1.0.0 (notebook V1) | **FalconDec v1.0.0 (notebook V2)** |\r\n| Backbone | DistilBERT, 128 tokens | ModernBERT-base, 384 tokens | Ettin-150M, 512 / 2,048 | Ettin-150M, 512 / 2,048 |\r\n| Preset / epochs | \u2014 | small / 1 per stage | small / 1 | **standard / 2** |\r\n| Training decisions | \u2014 | 18,795 | \u2264500 per task | **155,747** |\r\n| Agentic data | \u2014 | \u2014 | \u2014 | AgentTrek, Counsel, HotpotQA |\r\n| Test accuracy | \u2014 | 0.813 (23-task, code-heavy mix) | 0.518 micro / 0.535 macro | **0.725 / 0.725** (58-task mix) |\r\n| ECE | \u2014 | 0.008 | 0.014 | 0.025 |\r\n| Weights | \u2014 | \u2248596 MB fp32 | 319 MB fp16 | 319 MB fp16 \u00b7 161 MB int8 |\r\n\r\nLightDec is a fresh `scratch` run from the pretrained Ettin backbone (lineage: `jhu-clsp/ettin-encoder-150m`). It doesn't inherit proof_v3's or proof_v2's weights. Fine-tuning LightDec with the notebook bumps the patch version (1.0.0 \u2192 1.0.1).\r\n\r\n**Changelog.** LightDec 1.0.0: first release. Notebook V2, `standard` preset, 2 epochs, seed 42.\r\n\r\n---\r\n\r\n## 15. API reference\r\n\r\nAll functions live in `falcondec_modeling.py`.\r\n\r\n| Function | Description |\r\n|---|---|\r\n| `load_falcondec(path, device=None, dtype=None, attn_implementation=\"sdpa\")` | Loads a FalconDec directory (fp16 or int8) or Hub repo id. Returns `(model, tokenizer)`; cuda if available |\r\n| `decide(model, tok, state, questions, defer_threshold=None, batch_size=32)` | Typed questions about one state. `questions` is a list or a `{key: question}` dict. Returns `{\"results\": [...], \"answers\": {key: result}}` |\r\n| `score_items(model, tok, items, batch_size=32)` | Batch scoring. `items = [{\"state\", \"question\", \"options\", \"type\"?, \"option_tokens\"?, \"seq_len\"?}]`. Returns a calibrated probability array per item |\r\n| `save_falcondec(model, tok, out_dir, int8=False, extra_files=None)` | Writes a self-contained directory (copies the modeling file) |\r\n| `quantize_int8(state_dict)` / `dequantize_int8(state_dict)` | The per-channel int8 codec used for `compact-int8/` |\r\n| `assemble(...)`, `collate_features(...)` | Low-level sequence building and batching |\r\n\r\n**Question fields**: `type` (`choice` / `noul` / `score`, default `choice`); `question` or `instructions`; `options` (list) or `criteria` (dict `{key: description}` for choice, list of levels for score); `labels` (`{\"true\": \u2026, \"false\": \u2026}` wording for noul); `option_tokens` and `seq_len` (optional per-question budgets).\r\n\r\n**Model attributes**: `model.fcfg` (the live config: layout, `special`, `defer_threshold`, `version`, `lineage`); `model.temperature` (3 \u00d7 4 tensor); `model.num_parameters()`; `model.device`.\r\n\r\n---\r\n\r\n## 16. Citation and references\r\n\r\n```bibtex\r\n@misc{falconsai_lightdec_2026,\r\n  title        = {LightDec: a lightweight, single-pass, typed, calibrated decision model for agentic systems},\r\n  author       = {{Falconsai}},\r\n  year         = {2026},\r\n  howpublished = {\\url{https://huggingface.co/Falconsai/LightDec}},\r\n  note         = {FalconDec architecture, Ettin-150M backbone; successor to Falconsai/proof_v3}\r\n}\r\n```\r\n\r\n**Methods.** Warner et al. (2024), *ModernBERT*. Weller et al. (2025), *Ettin* encoders. Gneiting and Raftery (2007), *Strictly Proper Scoring Rules*. Guo et al. (2017), *On Calibration of Modern Neural Networks*. Geifman and El-Yaniv (2017), *Selective Classification*. Zaheer et al. (2017), *Deep Sets*; Lee et al. (2019), *Set Transformer*. Williams (1992), REINFORCE; Shao et al. (2024), GRPO. Hinton, Vinyals and Dean (2015), *Distillation*. Related decision models: Laya (Convai Innovations), TypeSafe Jev, Together Tev1.\r\n\r\n**Data.** ARC, OpenBookQA, SciQ, CommonsenseQA, QASC, HellaSwag, WinoGrande, MMLU, BoolQ, GSM8K, AQuA-RAT, SNLI, MultiNLI, ANLI, SciTail, CLINC150, MASSIVE, Bitext, Banking77, AG News, Yelp, DAIR Emotion, SST-5, jailbreak-classification, Civil Comments, deepset prompt-injections, AgentHarm, CodeXGLUE, MBPP, HumanEval, LocalLLaMA/typed-decisions, AgentTrek, Counsel, HotpotQA.\r\n\r\nReport issues, misroutes or evaluation results through the Community tab of this repository.",
-        "blocks": []
-      }
-    },
-    "chain": {
-      "depth": 1,
-      "prior": null
-    },
-    "tool": "FALCONS.AI Model Surgeon V7.99",
-    "signature": "c3b008c863f9e0171f135129cffa11b8d1e07712c74857358a660a8ff8819e0a",
-    "copyright": "\u00a9 2026 FALCONS.AI"
-  },
-  "identification_probe": null,
-  "original_format_export": {
-    "file": null,
-    "note": "upload was already safetensors \u2014 model_edited.safetensors IS the original format"
-  },
-  "watermark": {
-    "text": "FALCONS.AI Model Surgeon V7.99 | session Zhl_Fu2B | 2026-09-26T07:21:44Z | NLP \u00b7 Small Language Model (SLM)",
-    "hmac_sha256": "8bbdc48dddbf3c3c14e612f411052cd9e0fb78fb2335469541aa050868618073",
-    "tensor": "_falconsai_",
-    "decode": "((values - 0.5) * 255) -> uint8 -> utf-8"
-  }
-}