⚖️ Nirnay — fast, deterministic decisions on CPU

Classify, rate and answer yes / no about any text or JSON — with calibrated probabilities, in one pass, no GPU, no text generation. Drop-in compatible with the Laya / Jev question format and /v1/systemone API.

Measured on a 2014 dual-Xeon workstation with no GPU, using Laya's own latency workload. Laya re-reads the text for every question; Nirnay reads it once — so 10 questions take 0.23 s instead of 1.4 s, and 50 take 0.65 s instead of 6.7 s.

Try it free in Colab (CPU runtime): nirnay_demo.ipynb · Code: github.com/nandukmelath/nirnay

New in v3.1 (default)

• Request routing 0.404 → 0.875 — "is this a coding, math, data, writing, factual or small-talk request?" — now ahead of every Laya checkpoint (best 0.659). Trained on code, math, SQL, writing, trivia and small-talk requests; the routing benchmark's own sources (gsm8k, mbpp) stay held out.
• SMS spam 0.670 → 0.790, emotion 0.550 → 0.595, toxicity 0.660 → 0.688, RAG relevance 0.655 → 0.680.
• New training data: request routing, short-message spam (no text from the held-out UCI SMS collection), request safety (NVIDIA Aegis 2.0, content-harm wording only), urgency asked a dozen ways, and 8 more phrasings of every question template.
• Against the best of Laya's three checkpoints: 5 wins, 2 ties, 5 losses (v2: 3 / 1 / 8).
• Trade-offs, measured: jailbreak detection 0.647 vs v2's 0.695 at the default threshold (its ranking quality is about the same: AUC 0.701 vs 0.713), and Banking77 0.598 vs 0.610. v2 stays available: nirnay.load("nandukmelath/nirnay", subfolder="v2").
• An unreleased intermediate (v3) phrased some safety questions about AI-assistant behaviour ("should an assistant decline this?") and learned to call every harmful prompt a jailbreak (AUC 0.564). v3.1 asks about content harm only, which fixed it. Same speed, same architecture, same API.

Why Nirnay

• One read per call — 1 question or 50 questions about the same text cost almost the same.
• No label limit — every option is encoded separately: 77-way intent detection beats Laya (0.598 vs ≤0.492), and hundreds of labels work.
• Order-proof and deterministic — reordering options can never change an answer (measured flip rate 0.000; Laya 0.14–0.19). Repeat calls are byte-identical; no decision changes across 1/4/8/16 CPU threads.
• Calibrated — answer_confidence is a probability you can gate on; per-type temperatures ship with the model.
• Adapts in minutes, without training — learn() with 16 examples per label takes Banking77 from 0.598 to 0.840 on a CPU.
• Small — 527 MB (v3.1) or 155 MB (v1, about 2.4x faster again). Laya's checkpoints are 614–804 MB.

Quick start

pip install git+https://github.com/nandukmelath/nirnay
# or: pip install https://huggingface.co/nandukmelath/nirnay/resolve/main/dist/nirnay-1.0.0-py3-none-any.whl

import nirnay

m = nirnay.load("nandukmelath/nirnay")                        # v3.1, most accurate (default)
# m = nirnay.load("nandukmelath/nirnay", subfolder="fast")    # v1, fastest
# m = nirnay.load("nandukmelath/nirnay", subfolder="v2")      # the previous default, pinned

r = m.predict(
    {"ticket": {"subject": "Payout failing", "body": "My payouts failed for 3 days, please help ASAP"}},
    {"team":     {"type": "choice", "instructions": "Which team should handle this?",
                  "criteria": {"billing": "payments", "technical": "bugs and integrations", "sales": "pricing"}},
     "urgent":   {"type": "noul",   "instructions": "Is this ticket urgent?"},
     "severity": {"type": "score",  "instructions": "How severe is the problem?",
                  "criteria": ["minor", "moderate", "severe"]}})

r["answers"]["team"]["choice"]        # 'billing'
r["answers"]["urgent"]["noul"]        # P(yes)
r["answers"]["severity"]["score"]     # expected level 0..2

m.predict_batch(states, questions)                              # many texts, same questions
m.learn("team", questions["team"], [(text, "billing"), ...])    # few-shot memory, no training

Every answer carries probabilities, confidence and answer_confidence (the calibrated probability of the reported answer); pass min_confidence= to get an abstained flag. Also included:

• python -m nirnay.server — HTTP server with POST /v1/systemone and /v1/systemone/batch (Jev / Laya wire format, so existing clients only change their base URL)
• python -m nirnay.mcp_server — MCP tools decide, classify, yes_no for local agents
• nirnay — command line

How it works

state (text / JSON)        question "<type> question: <instructions>"       options "<label>: <description>"
       |                               |                                            |
 encoder layers 1-8 (frozen) — every text encoded on its own; question/option results cached per schema
       |                               |                                            |
 encoder layers 9-12 (trained copies) over [ state ; questions ; options ] with an asymmetric mask:
     state rows    attend to the state only          -> computed ONCE per call, shared by every question
     question rows attend to state + own question
     option rows   attend to state + their question + own option   (never to other options)
       |
 score at each option's first token + learned cosine prior -> softmax over options -> per-bucket temperature

The top of the network is a cross-encoder (option tokens read the state at every layer), but the state never reads the questions — so one state encoding serves every question, and options are independent of each other. Training minimises strictly proper scoring rules (log score, plus ranked probability score for ratings) with exact gradients. Layers 1-8 are frozen, so only the top 4 layers + scorer train (29M parameters): v3.1 continued from v2 for 1.25 epochs on 324k typed questions, on a CPU (two sockets, data-parallel, ~9 hours). The repository also ships a GPU trainer (code/train/train_gpu.py, verified to reproduce the CPU pipeline) and a free-Kaggle-GPU job.

Benchmarks

Laya (laya 0.4.14 from PyPI, all three checkpoints, its own API and calibration) ran on the same machine on byte-identical cases: the 400-case suites of Laya's own application benchmark (seed 13) plus two held-out extras. Jev is a closed cloud API, so its column is the published third-party figure.

task	Nirnay v3.1	Nirnay v2	Nirnay v1	Laya en	Laya ml	Laya td	Jev (published)
AG News (4 topics)	0.925	0.925	0.882	0.950	0.935	0.953	0.910
DAIR emotion (6)	0.595	0.550	0.515	0.593	0.537	0.600	0.480
Banking77 (77 intents)	0.598	0.610	0.595	0.425	0.470	0.492	0.870 (72 labels, n=100)
support triage (10 queues)	0.407	0.415	0.417	0.502	0.540	0.505	—
Enron spam	0.995	0.995	0.968	0.998	0.998	0.968	—
phishing	0.973	0.975	0.948	0.980	0.993	0.940	—
jailbreak (toxic-chat)	0.647	0.695	0.787	0.708	0.805	0.762	—
toxicity (toxic-chat)	0.688	0.660	0.595	0.530	0.535	0.530	—
RAG passage relevance	0.680	0.655	0.613	0.625	0.672	0.625	—
model routing (domain)	0.875	0.404	0.446	0.639	0.441	0.659	—
SST-5 (5-level rating)	0.490	0.492	0.425	0.352	0.292	0.470	—
SMS spam	0.790	0.670	0.600	0.938	0.807	0.910	0.930 (independent)

Honest summary: against the best of Laya's three checkpoints (picked per task, after the fact) Nirnay v3.1 wins 5 tasks (Banking77, toxicity, RAG relevance, model routing, SST-5), ties 2 (emotion, Enron spam) and loses 5 (AG News, support triage, phishing, jailbreak, SMS spam); against Laya's default English checkpoint it is also 5 / 2 / 5. It beats Jev's published AG News and emotion figures. Laya is a 421M-parameter model fully fine-tuned on GPUs; Nirnay trains 29M parameters on a CPU.

Yes / no ranking quality (AUC, threshold-free; v3.1 vs v2): jailbreak 0.701 vs 0.713 · toxicity 0.787 vs 0.765 · SMS spam 0.919 vs 0.899 · RAG relevance 0.744 vs 0.727 · phishing 0.998 vs 0.997 · Enron spam 1.000 vs 1.000.

speed (warm p50)	1 question	10 questions	50 questions
Nirnay v3.1	129 ms	232 ms	655 ms
Nirnay v1 (fast)	54 ms	95 ms	258 ms
Laya at its fastest (any checkpoint, eager or its ONNX backend)	159 ms	1,389 ms	6,730 ms

few-shot learn(), Banking77, examples per intent	0	2	4	8	16
Nirnay v3.1	0.598	0.635	0.725	0.825	0.840
Nirnay v2	0.610	0.683	0.768	0.840	0.853

Determinism: repeat calls byte-identical; across 1/4/8/16 CPU threads no decision changed (probabilities differ by at most 0.0001). Cold start to first answer ~7 s (Laya 12–16 s). Full tables (calibration, footprint, ablations, v3 → v3.1): REPORT.md.

Limitations

• English only (bge-*-en backbones). For other languages use nirnay-multilingual: MASSIVE intents in 51 languages 0.386 vs Laya multilingual's 0.217 (higher in 49 of 51 languages).
• Wording still matters for concepts it rarely trained on. v3.1 learned urgency from a dozen phrasings; on a small check "Does the customer express urgency?" now gets 6 of 8 tickets right (v2: 4 of 8), and "Does this need immediate attention?" 8 of 8. Probabilities rank texts well even when a phrasing moves the yes/no threshold; fix the threshold with a few labelled examples via learn().
• Jailbreak detection is not a strength: jailbreak data is deliberately held out of training (as Laya does), so this is zero-shot; at the default threshold v3.1 over-flags some harmful but non-jailbreak prompts.
• It decides; it does not reason step by step or write text.
• Calibration is fitted on the training mixture; refit (or use learn()) on your own traffic before automated gating.

Training data

Public datasets listed in the metadata, converted to typed questions with varied wording, label subsets, yes/no reformulations and ordinal ratings (code/train/build_data.py). Held out of training, mirroring Laya's own policy: dair-ai/emotion, mteb/banking77, lmsys/toxic-chat, gsm8k / mbpp (the routing benchmark's sources), SetFit/sst5, ucirvine/sms_spam (and any text matching it), XNLI, MASSIVE, SIB-200 and all jailbreak / prompt-injection datasets; any text appearing in a benchmark case was removed by exact match.

New in v3.1: CodeAlpaca (CC-BY-4.0), HumanEval (MIT), SVAMP (MIT), AQuA-RAT (Apache-2.0), MATH (MIT), sql-create-context (CC-BY-4.0), Gretel synthetic text-to-SQL (Apache-2.0), WritingPrompts (MIT), Dolly 15k (CC-BY-SA-3.0), NQ-Open (CC-BY-SA-3.0), SODA (CC-BY-4.0), India SMS spam (MIT), ham-spam-scam (MIT, small sample), NVIDIA Aegis 2.0 (CC-BY-4.0). Some training datasets carry research-only or non-commercial terms (for example MS MARCO, RACE, Yahoo Answers, Yelp reviews, Quora question pairs). The code and weights are released under Apache-2.0; check those datasets' terms before commercial use.

Files

• nirnay_config.json, encoder.safetensors, head.safetensors, tokenizer.json — v3.1 (default)
• v2/ — the previous default · fast/ — v1 (bge-small)
• dist/nirnay-1.0.0-py3-none-any.whl — the inference package
• code/ — full source: package, data builder, CPU and GPU training, Kaggle job, benchmark harness, tests
• REPORT.md — every measurement

Built on BAAI/bge-base-en-v1.5 and BAAI/bge-small-en-v1.5 (MIT). The benchmark cases reproduce the application benchmark of Laya (source) by Nandakishor M (Convai Innovations), whose open model inspired this project.