Ashhad Ahsan Rehman

Great write-up. The step the post leaves to the application, turning noul: 0.83 into an action, is where things get hard in production. It's usually a threshold (if p > 0.9), and it quietly goes stale: raw probabilities are rarely calibrated, the right cut-off depends on what a mistake costs, and the outcome that says whether the decision was right (a chargeback, a reassigned ticket) arrives days or months later.

I'm building an open-source, self-hosted tool for that loop: trimset. It sits in front of any /v1/systemone server (so transformers serve or llama.cpp) with a one-line URL change. It logs every decision and joins the real outcome when it arrives. It calibrates per question and model, then compiles thresholds and an escalation band with a stated guarantee ("false passes ≤ 2% with 95% confidence"), which your app applies locally. It's pre-alpha; drift and silent-upgrade alerts come next.

Live demo on recorded data, threshold simulator included: https://demo.trimset.dev/#/questions/refund_fraud/simulator

For anyone running decision models in production: how do you pick your cut-offs today?