OpenJevX v0.5.2: model folder (8-bit ONNX, config, tokenizer), fine-tune checkpoint, card

README.md

@@ -3,7 +3,7 @@ license: apache-2.0
 base_model: convaiinnovations/laya
 tags: [decision-model, onnx, classification, jevx]
 ---
-# OpenJevX v0.5.0
+# OpenJevX v0.5.2
 
 A small, CPU-first decision model: yes/no, pick-one and 0–N rating questions about a state (JSON or text),
 answered on a laptop CPU. **It is meant to run in a container next to your own services**:
@@ -16,7 +16,7 @@ one container, the model inside, your requests never leave your machine.
 **The model is a folder:** `openjevx.w8.onnx` (8-bit weight-only, 598 MB; activations stay float, so answers
 don't depend on what else is in the request) + `config.json` (this model's own calibration temperatures) +
 `tokenizer.json`. The other files are the trainable checkpoint for fine-tuning further: `model.safetensors`,
-`encoder/`, `tokenizer/`, `rl_agent_config.json`. Older versions: git revision `v0.4.0`.
+`encoder/`, `tokenizer/`, `rl_agent_config.json`. Older versions: git revisions `v0.5.0` and `v0.4.0`.
 
 ## Run it in a container (recommended)
 
@@ -40,13 +40,15 @@ curl -s http://127.0.0.1:21118/v1/systemone -d '{
 
 Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
 expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
-Prefer a plain binary? macOS (Apple Silicon), Linux and Windows builds of the v0.5.1 server (ONNX Runtime 1.29)
-are on the [release page](https://github.com/muthuishere/openjevx/releases/latest).
+Prefer a plain binary? macOS (Apple Silicon), Linux and Windows builds of the v0.5.2 server (ONNX Runtime 1.29)
+are on the [release page](https://github.com/muthuishere/openjevx/releases/latest). The server can also load this
+model folder straight from S3 (`OPENJEVX_MODEL=s3://bucket/prefix/`); see the GitHub README.
 
 ## Or in Python
 
-Same answers as the server (a port of its prompt layout, markers, temperature and softmax; checked against the
-server on the same model: identical probabilities).
+Same answers as the server (a port of its prompt layout, markers, input length, temperature and softmax). Checked
+against the server on the same model: probabilities agree to within about 0.005; the rest is ONNX Runtime versions
+rounding differently.
 
 ```bash
 pip install onnxruntime tokenizers huggingface_hub
@@ -64,51 +66,58 @@ jev = openjevx.OpenJevX()   # downloads openjevx.w8.onnx + tokenizer.json + conf
 print(jev.decide({"age": 17}, {
     "adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
 }))
-# {'answers': {'adult': {'type': 'noul', 'noul': 0.04, ...}}}  (v0.5.0; about 0.04)
+# {'answers': {'adult': {'type': 'noul', 'noul': 0.08, ...}}}  (about 0.08)
 ```
 
 `openjevx.py` is a single file (also in the GitHub repo under `python/`); copy it into your project if you
 prefer. Request and answer shapes match `POST /v1/systemone`.
 
-## Release gate (v0.5.0)
+## Release gate
 
 The model folder on the Go server, on CPU. The gate sets share **no state** with training.
 "Right and confident" = right AND past jevx's default thresholds (yes >= 0.8, no <= 0.2, choice/score >= 0.6).
+Limits for the basics gates: right and confident >= 90%, confidently wrong <= 2%.
 
 | Gate set | Accuracy | Right and confident | Confidently wrong |
 |---|---|---|---|
-| Everyday basics (463) | 96.3% | 96.3% | 3.0% |
-| Rule-checking basics (300) | 94.7% | 94.3% | 3.7% |
+| Everyday basics (463) | 99.1% | 98.7% | 0.9% |
+| Rule-checking basics (300) | 100% | 100% | 0.0% |
+| Held-out rule-checking (11,949) | 98.9% | 98.8% | 1.0% |
+| Held-out software-work roles (9,159) | 98.5% | 97.8% | 1.0% |
+| Logs gate (900) | 92.0% | 91.9% | 7.8% |
 
-**Known miss:** confidently wrong is 3.0–3.7% on the basics gates, against our 2% target. It was accepted for this
-release; the fix is planned for the next model release. The rest of the gate (held-out test sets, logs gate, 13 fundamentals) is
-still being scored and will be added here.
+The 13 fundamentals through the jevx CLI with default thresholds: 13/13.
+
+**Known misses:** on log triage it is confidently wrong 7.8% of the time, and on real public incident postmortems
+(102 questions) 20.6%. Check those answers.
 
 **Latency** (v0.5.1 server, ONNX Runtime 1.29, CPU only, Apple M5 Pro, p50 end to end): 22 ms for a short
-request (36 tokens), 85 ms for a typical one (164 tokens), 540 ms near the 1024-token limit. Cold start is about
+request (36 tokens), 85 ms for a typical one (164 tokens), 540 ms for a long one (912 tokens). Inputs are cut at 512 tokens (`max_len` in
+`config.json`, the length this model was trained at). Cold start is about
 1 s once per process. ONNX Runtime 1.29 or newer is needed for this speed: it keeps the 8-bit weights 8-bit
 (`MatMulNBits`); in Python, `pip install -U onnxruntime`.
 
-## What v0.5.0 was trained on
+## What v0.5.2 was trained on
 
-734,795 decisions (408,505 rows). Labels come from evaluating rules or from real outcomes, never from another model's guesses.
+791,889 decisions (465,583 rows). Labels come from evaluating rules or from real outcomes, never from another model's guesses.
 
 | Area | Decisions | Share |
 |---|---|---|
-| Rule-checking across 38 business domains (retail pricing, recruiting, real estate, CI/CD, insurance claims, pharmacy stock, HR payroll, gaming, ...) | 200,139 | 27.2% |
-| Software-work roles (developers, testers, tech leads, managers, operations, everyone, agent checks) | 183,543 | 25.0% |
-| tasksource decision corpus | 146,567 | 19.9% |
-| Public sets with real labels (CVE fixes from bigvul, defect detection, code search, ms_marco relevance, HDFS and BGL operator log alerts) | 104,990 | 14.3% |
-| Log triage: application (Java/Python/Node/Go/nginx), database (Postgres/MySQL, real SQLSTATE codes), frontend (browser/Sentry) | 60,001 | 8.2% |
-| Everyday basics (driving licence age, store hours, parcel late, discount thresholds, fever, bag weight, ...) | 30,405 | 4.1% |
+| Rule-checking across 38 business domains (retail pricing, recruiting, real estate, CI/CD, insurance claims, pharmacy stock, HR payroll, gaming, ...) | 200,155 | 25.3% |
+| Software-work roles (developers, testers, tech leads, managers, operations, everyone, agent checks) | 183,543 | 23.2% |
+| tasksource decision corpus | 146,567 | 18.5% |
+| Public sets with real labels (CVE fixes from bigvul, defect detection, code search, ms_marco relevance, HDFS and BGL operator log alerts) | 104,990 | 13.3% |
+| Rule-reading drills: everyday rules with many thresholds, decimals, dates over 2020-2035, quarter-hour meetings, opening hours, missing/equal/contains checks | 75,321 | 9.5% |
+| Log triage: application (Java/Python/Node/Go/nginx), database (Postgres/MySQL, real SQLSTATE codes), frontend (browser/Sentry) | 60,001 | 7.6% |
+| Everyday basics (driving licence age, store hours, parcel late, discount thresholds, fever, bag weight, ...) | 12,162 | 1.5% |
 | Public typed-decisions | 9,150 | 1.2% |
 
-Run: one RTX 4090 on Vast.ai, 734,145 decisions after packaging, one full pass in 2.4 h at ~88 items/s, about $1.10 in total. Leakage-checked (22,592 test questions removed); every row checked against the trainer before renting.
+Run: one RTX 4090 on Vast.ai, 791,239 decisions after packaging, one full pass in 2.5 h at ~87 items/s, about $1.25. Leakage-checked (22,576 test questions removed); every row checked against the trainer before renting.
 
 ## Using it with jevx
 
 jevx caches answers by model name. After upgrading, run `jevx cache clear`, or use a versioned
-model name in your profile (for example `--model openjevx-v0.5.0`).
+model name in your profile (for example `--model openjevx-v0.5.2`).
 
 ## Fine-tune further
 
@@ -116,4 +125,4 @@ The whole pipeline (data generators, leakage check, one-job GPU training that re
 release gate) is in the GitHub repo under
 [`finetuning/`](https://github.com/muthuishere/openjevx/tree/main/finetuning). No training data is published.
 
-The trainable checkpoint is also available as one archive: [openjevx-finetune-v0.5.0.tar.gz](https://pub-8da821f06ff747cda688f8267ed2aa96.r2.dev/openjevx-finetune-v0.5.0.tar.gz).
+The trainable checkpoint is also available as one archive: [openjevx-finetune-v0.5.2.tar.gz](https://pub-8da821f06ff747cda688f8267ed2aa96.r2.dev/openjevx-finetune-v0.5.2.tar.gz).

config.json

@@ -1,12 +1,12 @@
 {
   "name": "openjevx",
-  "version": "0.5.0",
+  "version": "0.5.2",
   "temperature": {
-    "choice": 1.220432996749878,
-    "score": 1.161743402481079,
-    "noul": 1.2535336017608643
+    "choice": 1.1505310535430908,
+    "score": 1.1315456628799438,
+    "noul": 1.1007894277572632
   },
-  "max_len": 1024,
+  "max_len": 512,
   "head_max": 256,
   "special_ids": {
     "cls": 50281,
@@ -16,5 +16,5 @@
   },
   "quantization": "8-bit weight-only",
   "base_model": "convaiinnovations/laya",
-  "sha256": "109b2b1d3717ff9038da811d84f538262fc9d9bc1f958967d437e837f507e294"
+  "sha256": "fa1520c22876ba6a98f2b080b55c302e8f7ffbcf7630a1ff05d759bb21a18410"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:82d90be1947a6a2f393b115cc5c1e1dca9ed37c82cbea392b0ef1bdd17138760
+oid sha256:3f40ae2d9d1105d4b13f7d4dadc820d6451499076e57d2bf55c5d898d9a4e160
 size 842609220

openjevx.py

@@ -22,7 +22,7 @@ from tokenizers import Tokenizer
 
 REPO = "muthuishere/openjevx"
 CLS, SEP, MASK = 50281, 50282, 50284
-MAX_LEN, HEAD_MAX = 1024, 256
+MAX_LEN, HEAD_MAX = 1024, 256  # used only when config.json has no max_len / head_max (as the server does)
 
 
 def _criterion(v):
@@ -51,7 +51,9 @@ class OpenJevX:
         model = model or hf_hub_download(REPO, "openjevx.w8.onnx")
         tokenizer = tokenizer or hf_hub_download(REPO, "tokenizer.json")
         config = config or hf_hub_download(REPO, "config.json")  # this model's own calibration temperatures
-        t = json.load(open(config))["temperature"]
+        cfg = json.load(open(config))
+        t = cfg["temperature"]
+        self.max_len, self.head_max = cfg.get("max_len") or MAX_LEN, cfg.get("head_max") or HEAD_MAX
         self.temperature = {0: t["choice"], 1: t["score"], 2: t["noul"]}
         self.tok = Tokenizer.from_file(tokenizer)
         self.session = ort.InferenceSession(model, providers=providers or ["CPUExecutionProvider"])
@@ -63,19 +65,19 @@ class OpenJevX:
         qtype, keys, options = _parse(q)
         head = self._enc(q["type"] + " question: " + q["instructions"].replace("[MASK]", " "))
         opts = [[MASK] + self._enc(" " + o.replace("[MASK]", " "))[:48] for o in options]
-        budget = HEAD_MAX - sum(map(len, opts))
+        budget = self.head_max - sum(map(len, opts))
         if budget < 16:
-            per = max(4, (HEAD_MAX - 16) // max(1, len(opts)))
+            per = max(4, (self.head_max - 16) // max(1, len(opts)))
             opts = [o[:per] for o in opts]
-            budget = HEAD_MAX - sum(map(len, opts))
+            budget = self.head_max - sum(map(len, opts))
         head = head[:max(8, budget)]
         ids, markers = [CLS] + head + [SEP], []
         for o in opts:
             markers.append(len(ids))
             ids += o
         ids.append(SEP)
-        ids += state_ids[:max(0, MAX_LEN - len(ids) - 1)] + [SEP]
-        return ids[:MAX_LEN], [m for m in markers if m < MAX_LEN], qtype, keys
+        ids += state_ids[:max(0, self.max_len - len(ids) - 1)] + [SEP]
+        return ids[:self.max_len], [m for m in markers if m < self.max_len], qtype, keys
 
     def decide(self, state, questions):
         """state: dict/list (sent as JSON) or str; questions: {id: {type, instructions, criteria}}."""

openjevx.w8.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:109b2b1d3717ff9038da811d84f538262fc9d9bc1f958967d437e837f507e294
+oid sha256:fa1520c22876ba6a98f2b080b55c302e8f7ffbcf7630a1ff05d759bb21a18410
 size 598046561

openjevx.w8.onnx.sha256

@@ -1 +1 @@
-109b2b1d3717ff9038da811d84f538262fc9d9bc1f958967d437e837f507e294  openjevx.w8.onnx
+fa1520c22876ba6a98f2b080b55c302e8f7ffbcf7630a1ff05d759bb21a18410  openjevx.w8.onnx

rl_agent_config.json

@@ -11,9 +11,9 @@
   "amp_dtype": "bf16",
   "model_name": "openjevx",
   "temperature": [
-    1.220432996749878,
-    1.161743402481079,
-    1.2535336017608643
+    1.1505310535430908,
+    1.1315456628799438,
+    1.1007894277572632
   ],
   "training": {
     "updates": 7313,