| --- |
| license: apache-2.0 |
| library_name: gliner2 |
| pipeline_tag: text-classification |
| language: |
| - en |
| tags: |
| - decision-model |
| - calibration |
| - selective-prediction |
| - conformal-prediction |
| - abstention |
| - risk-aware |
| - security |
| - cybersecurity |
| - security-operations |
| - soc |
| - pentest |
| - red-team |
| - blue-team |
| - gliner2 |
| base_model: fastino/GLiNER2.5-Decide |
| metrics: |
| - accuracy |
| model-index: |
| - name: haruspex |
| results: |
| - task: |
| type: text-classification |
| name: Security decision (boolean / choice / score) |
| dataset: |
| type: custom |
| name: mobile pack test split (117 rows) |
| metrics: |
| - type: accuracy |
| value: 0.9231 |
| name: accuracy |
| - type: ece |
| value: 0.0598 |
| name: ece |
| - type: brier |
| value: 0.0598 |
| name: brier |
| --- |
| |
| # TyrusRC/haruspex (mobile) |
|
|
| **Haruspex** is a small, local **decision layer** for security operations. It reads a |
| situation and returns a *calibrated* typed verdict — **boolean** (yes/no + probability), |
| **choice** (pick one option), **score** (rate on a rubric) — and **abstains** when it is |
| not confident enough for your risk budget. |
|
|
| Haruspex **reuses** the published decision encoder |
| [`fastino/GLiNER2.5-Decide`](https://huggingface.co/fastino/GLiNER2.5-Decide) (Apache-2.0, non-autoregressive, CPU/GPU) and |
| specializes it on security decisions. It does not train a decision model from scratch. |
|
|
| - Base: `fastino/GLiNER2.5-Decide`. |
| - Added: a typed decision contract, security specialization data, a certified |
| **selective-calibration** layer (accept/abstain against a risk budget), conformal |
| **prediction sets**, and a **novelty gate** that abstains on unfamiliar input. |
| - Serves the TypeSafe `POST /v1/systemone` and native `/v1/evaluate` contracts. |
|
|
| ## Intended use (security, both sides) |
|
|
| - **Offensive (pentest / red-team):** is this template/probe relevant to the target? |
| is this scanner hit a real finding? finding severity; verdict discipline |
| (vulnerable / not-vulnerable / inconclusive); attack-path next step. |
| - **Defensive (SOC / blue-team):** alert and incident triage (true / benign / false |
| positive), alert severity/priority, detection-rule applicability, incident routing, CVE |
| applicability, ATT&CK tactic of Windows/Sysmon events. Raw Defender, Sentinel, Elastic |
| ECS and Windows event JSON is normalised before the model reads it. |
|
|
| It is a classifier for bounded decisions — not a chatbot, not an exploit generator. |
|
|
| ## Use |
|
|
| ```python |
| from haruspex import Haruspex |
| m = Haruspex.load("TyrusRC/haruspex", subfolder="mobile", device="cuda") # or "mps" / "cpu" |
| print(m.decide({"type":"boolean","state":"Target: WordPress 6.2, PHP 8.1", |
| "question":"Is a WordPress xmlrpc.php template relevant?", |
| "criteria":{"true":"WordPress target","false":"not WordPress"}})) |
| ``` |
|
|
| Or serve the decision API (keyless, self-hosted): |
|
|
| ```sh |
| pip install "haruspex[serve] @ git+https://github.com/TyrusRC/haruspex" |
| haruspex-serve --adapters mobile=TyrusRC/haruspex#mobile --port 3000 # POST /v1/systemone · /v1/evaluate |
| ``` |
|
|
| ## Calibration |
|
|
| Fitted calibrator: accept threshold `0.0`, target risk `0.2` at delta `0.1`, coverage on the calibration split `1.0` (n=78), temperature `1.0`. |
|
|
| Each verdict carries the full probability distribution over its options (temperature- |
| scaled). A Learn-then-Test selective layer accepts a verdict only when its confidence |
| clears a threshold certified on a held-out calibration split, so that with probability |
| 1 - delta the error among accepted verdicts stays within the target risk; otherwise it |
| abstains with `"route": "escalate"`. A conformal prediction set lists the options the |
| true answer is among, and a novelty gate abstains on inputs unlike the training data. |
| Fit with `python -m haruspex.calibrate` and `python -m haruspex.ood`; evaluate with |
| `python -m haruspex.eval`. |
|
|
| ## Evaluation |
|
|
| Held-out test sets, never used for training, model selection or calibration: |
|
|
| - **mobile pack test split (117 rows)**: accuracy 0.9231, ECE 0.0598, Brier 0.0598; with the calibrator it answers 1.0 of rows at accuracy 0.9391 and abstains on the rest. |
|
|
| Comparisons against general LLMs are in the repo's `docs/benchmark.md`. |
|
|
| ## Limitations |
|
|
| - A specialist for the decision types above; novel free-form judgments belong to a |
| general LLM. |
| - Calibrated on this domain's held-out split; recalibrate before relying on the |
| guarantee on your own data. |
| - Prediction-set coverage is marginal over the calibration mix, not per input. |
| - A calibrator fitted on a mix of domains holds its risk target on average over that |
| mix, not on each domain: check each test set's accepted accuracy under Evaluation, |
| and route a domain with its own adapter to that adapter. |
|
|
| Apache-2.0. Repo + recipe: https://github.com/TyrusRC/haruspex |
|
|
