Instructions to use eulogik/nirnay-450m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Laya
How to use eulogik/nirnay-450m with Laya:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Transformers
How to use eulogik/nirnay-450m with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="eulogik/nirnay-450m")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("eulogik/nirnay-450m", device_map="auto") - Notebooks
- Google Colab
- Kaggle
NIRNAY 450M: a small calibrated decision model that beats Jev on Banking77
NIRNAY (ΰ€¨ΰ€Ώΰ€°ΰ₯ΰ€£ΰ€―, "decision") is a 450M open-weight classifier for banking intent and typed decisions. One forward pass turns a state plus a question into calibrated probabilities. No text generation, no API key, no per-call bill. Apache-2.0, built by Eulogik.
Model Β· Live demo Β· Code Β· Paper Β· Evals
Banking77 keywords for search: banking intent classification, 77-way intent classifier, intent detection model, small language model for classification, calibrated decision model, system one model, Jev alternative, Laya fine-tune, on-device text classifier, Apache 2.0 classifier.
Numbers first
| Model | Banking77 test (n=3080) | Setup |
|---|---|---|
| NIRNAY phase_b | 0.8792 (Brier 0.208, ECE raw 0.089 / fitted 0.045) | fine-tuned, this repo |
| NIRNAY phase_a | 0.8656 (Brier 0.240, ECE raw 0.110 / fitted 0.033) | fine-tuned, this repo |
| Jev 1.13.0 | 0.803 (same 3,080 cases, recorded run) | zero-shot API (jevbench.xyz) |
| Julia-1 144M | 0.64 (their 72-label pilot, n=100, shortlist) | their card concedes grouped routing drops answers |
| Untrained baseline | 0.143 (our template, our measurement) | fresh weights |
Raw JSON: eval/banking77_phase_b.json, eval/banking77_phase_a.json. Same test split for every row above.
The honest caveat: we fine-tuned on the train split, Jev answered zero-shot. That is exactly the Laya thesis (a model you fine-tune on your data), and this repo proves it works: +7.6 points over the API on identical cases.
Same metric as Cloudflare's Decision Index table (macro-F1, measured by us on the same 3,080 cases, per-class scores in eval/banking77_phase_b_macro.json):
| Model | Banking77 macro-F1 | Setup |
|---|---|---|
| Clef 27B | 0.942 | their Decision Index 0.2.1 suite (self-reported) |
| Clef-Flash 9B | 0.909 | their Decision Index 0.2.1 suite (self-reported) |
| NIRNAY phase_b | 0.8785 | fine-tuned 450M, this repo |
| Kev 9B | 0.848 | their Decision Index 0.2.1 suite (self-reported) |
| Jev 1.13.0 | 0.797 | their Decision Index 0.2.1 suite (self-reported) |
| Laya | 0.143 | their Decision Index 0.2.1 suite (self-reported) |
Same metric family, different harnesses, so read it as a cross-check, not a leaderboard. We beat Jev by 8 and Kev by 3 at 60x and 20x fewer params, on a Mac instead of an H200.
| Benchmark / Metric | Jev 1.13.0 | NIRNAY phase_b |
|---|---|---|
| Banking77 (same 3,080) | 0.803 (recorded run) | 0.8792 (fine-tuned) |
| ECE, raw | 0.197 gap (jevals, their setup) | 0.089 (our test) |
| Latency, 1 decision | 310 to 478 ms (API medians) | 209 ms MPS / 361 ms CPU |
| Weights | closed API | Apache-2.0 |
| Cost | per-call billing | $0 self-hosted |
Jev figures above are third-party published, never measured here (no API access); sample sizes and prompts differ. Ours are measured in this repo.
Where Jev leads
- JevBench overall. 72.1 vs our 0.55 public: intelligence, sealed decisions, calibration at scale. Different league, honestly stated.
- Long documents. Our context caps at 512 tokens. Their API takes more.
- Zero setup. Their API works with a key. Ours needs a local install and a one-time 466MB checkpoint download, then it runs offline.
- Languages. We trained English banking. They serve many languages out of the box.
Speed, batch-1, measured 2026-09-30: 209 ms on M4 MPS, 361 ms on CPU. Faster than Jev API calls (310 to 478 ms in independent runs). Slower than Laya's 33 ms. ONNX ships (parity proven) but gave no CPU speedup; int8 is the open latency lever.
Try it
Live demo: huggingface.co/spaces/GautamKishore/nirnay-demo. Type a message, get the intent.
One line, your labels
pip install nirnay-450m
from nirnay_450m import Classifier
clf = Classifier() # downloads and caches the weights once
result = clf("My card was charged twice", {
"duplicate_charge": "charged twice",
"refund_status": "asking about a refund",
"card_arrival": "asking where the card is",
})
print(result.choice, result.confidence) # duplicate_charge 0.99
print(result.probabilities)
Labels can be a dict, a JSON file, or the string "banking77". The model
reads the descriptions, not just the names: on a 10-label Banking77 subset
it scores 0.9167 with the original names and 0.8833 with neutral keys
("A".."J") and the same descriptions, at 86-108ms per decision on M4.
New label sets work without retraining when they are in the same domain.
For a new domain, fine-tuning is the honest next step.
Your own labels, with examples
nirnay fit --data train.jsonl --labels labels.json --out my_model.pt --steps 200
nirnay classify "my ticket text" --labels labels.json \
--checkpoint my_model.pt --temps my_model.temps.json
train.jsonl is one {"state": "...", "label": "..."} per line. fit
freezes the encoder, trains the choice readout, early-stops on a held-out
split, fits a temperature, and writes a checkpoint plus a temps file. It
prints the held-out accuracy before and after. If you already have the
base checkpoint locally, add --base-checkpoint /path/phase_b.pt to skip
the download. This is the difference between a demo and a tool: your
labels, your data, minutes on a laptop.
CLI
nirnay classify "My card was charged twice" --labels banking77
nirnay classify --input tickets.jsonl --output decisions.jsonl --labels labels.json
nirnay serve --port 8000 # POST /v1/systemone, Jev-compatible
nirnay doctor
--threshold 0.9 marks low-confidence results as escalated. Batch mode
reads and writes JSONL. nirnay serve loads the trained checkpoint by
default, not the raw base model.
Advanced: score and yes/no
from nirnay_450m import NirnayAgent
agent = NirnayAgent(device="mps") # auto-downloads the checkpoint
out = agent.system_one(
"Ticket T-4411: outage started 02:10, still down, 40 agents idle.",
questions={
"urgency": {
"type": "score",
"instructions": "How urgent is this?",
"criteria": ["not urgent", "soon", "blocking"],
},
"page_oncall": {
"type": "noul",
"instructions": "Should the on-call be paged?",
},
},
)
print(out["answers"]["urgency"]["score"]) # expected level over the rubric
print(out["answers"]["page_oncall"]["noul"]) # P(true)
Full eval harness: scripts/eval_checkpoint.py. Server wire format is
documented in nirnay_450m/server.py.
How it was trained
Base is Laya 421M (Apache-2.0) plus ~30M of additions (concept bottleneck, deep supervision, coarse-to-fine pointer, byte path). 7,000 Phase A steps + 50 RLCD steps, all on one Mac. Three-group optimizer, seeded everything, hashes frozen before training.
Two training deaths taught us the fixes (both landed, both gated):
- Scale runaway. The concept encoder output grew unboundedly (healthy RMS 0.09, dead 13.5) while centroids sat still, so the VQ loss exploded 300x. Fix: affine-free LayerNorm on
zbefore quantize. Scale stops being a degree of freedom. Zero new params. - Usage collapse. With no pressure on code usage, all tokens fell into one code per chunk, the quantized states went constant, and accuracy flatlined at 1/77 with no loss spike to warn you. Fix: a load-balancing aux term (hard fractions times soft probs, linear so gradients never vanish).
Plus a safety net around training itself: heldout probes every 250 steps, best-checkpoint retention, and an abort that fires when accuracy halves. It caught four real collapses during development. Details: NIRNAY-Breakthrough-Plan.md (amendment 2026-09-26), MEMORY.md.
Calibration
We report raw and fitted ECE on every eval, always. Phase_b on Banking77 test:
Most mass sits above 0.9 confidence at 92% accuracy there. Per-bucket temperatures ship with the run.
Architecture
Bytes and token ids feed a frozen Laya encoder (LoRA adapters train). A concept bottleneck (layer-normed product-VQ, 4 chunks x 32 codes, mixture-of-slots) adds a learned residual. A 2-layer head scores dense markers; a coarse-to-fine pointer re-ranks the top 20 for 77-way decisions. Per-bucket temperatures calibrate the output. Deep supervision at layers 4/8/12 and RLCD exist only at training time.
Evals and honesty
eval/holds the raw result JSONs. Reproduce withscripts/eval_checkpoint.py --checkpoint <ckpt>.- JevBench public (231 items, shipped checkpoint): 0.550 overall, easy 0.875, original 0.569, hard 0.396. The hard tier (long policy docs, probability, temporal reasoning) is the gap. We publish it because the plan pre-registers pass/fail either way. Training data for that gap (10k programmatic reasoning items, ground truth by construction) is in
src/nirnay/data_synth_hard.py. - 19/19 gate checks green (
GATES.md). Zero-shot and fine-tuned numbers are never mixed.
Limits
- 512-token context. Long documents get head-truncated; the hard JevBench tier shows it (0.396). Longer context is v1.1 work.
- CPU latency 361 ms per decision in PyTorch. ONNX export ships under
onnx/on the model repo (parity proven 37/37, no speedup yet). No GGUF/Ollama build yet, LoRA-merged encoder weights are in progress for the converter path. - Banking77 is the proven lane (triage, routing, guardrails). Anything else, measure before trusting.
FAQ
What is NIRNAY? A 450M Apache-2.0 decision model: state plus typed question in, calibrated probabilities out. Choice, score, and yes/no questions.
How accurate is it? 87.9% on Banking77 test (3,080 cases), Brier 0.208, fitted ECE 0.045.
How does it compare to Jev? On identical Banking77 test cases: 0.879 (fine-tuned) vs 0.803 (Jev 1.13.0 zero-shot API). Ours runs local, no per-call cost.
How does it compare to Laya? Laya is our base (421M, Apache-2.0). We fixed its training instability, added the concept bottleneck and pointer, and fine-tuned it. Untrained on our template it scores 0.143 here.
How fast is it? 209 ms per decision on M4 GPU, 361 ms on CPU, batch-1, measured.
What license? Apache-2.0. Commercial use fine.
What hardware trained it? One Mac (M4, 16GB). Full run about 4 hours. Cloud spend: $0. Receipts in logs/ and MEMORY.md.
What is it bad at? Long documents over 512 tokens, and JevBench-hard style probability/temporal reasoning (0.396). See Limits.
Built by Eulogik
NIRNAY is built by Eulogik (GitHub, contact: info@eulogik.com), makers of pico-type, NanoForecast, TinyDoc-VLM, and KARN. Same house rules: Apache-2.0, measured numbers only, runs on your hardware.
License
Apache-2.0. Laya base (ConvAI Innovations, Apache-2.0). Banking77 data (PolyAI, CC-BY-4.0).
Links
- Model: https://huggingface.co/eulogik/nirnay-450m
- Code: https://github.com/eulogik/nirnay
- Paper: https://github.com/eulogik/nirnay/blob/main/papers/nirnay/main.pdf
- Evals: ./eval/banking77_phase_b.json
- Eulogik: https://eulogik.com (contact: info@eulogik.com)
- Downloads last month
- 9
Model tree for eulogik/nirnay-450m
Base model
convaiinnovations/layaSpace using eulogik/nirnay-450m 1
Evaluation results
- accuracy on Banking77 test splitself-reported0.879
- brier on Banking77 test splitself-reported0.208
- ECE (fitted) on Banking77 test splitself-reported0.045