File size: 8,016 Bytes
2abbcba 69619b6 2abbcba 6eacb9d 69619b6 353c1b7 3d7be4a 69619b6 3d7be4a 883bd72 6eacb9d 3d7be4a 6eacb9d 3d7be4a 6eacb9d 3d7be4a 883bd72 3d7be4a 6eacb9d 3d7be4a 69619b6 6eacb9d 353c1b7 883bd72 6eacb9d 353c1b7 883bd72 353c1b7 6eacb9d 353c1b7 6eacb9d 5399cb8 353c1b7 0098fc0 6eacb9d 0098fc0 353c1b7 6eacb9d 3d7be4a 6eacb9d 69619b6 883bd72 69619b6 6eacb9d 883bd72 6eacb9d 69619b6 353c1b7 6eacb9d 3d7be4a 69619b6 3d7be4a 353c1b7 6eacb9d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 | ---
license: apache-2.0
base_model: convaiinnovations/laya
tags: [decision-model, onnx, classification, jevx]
---
# OpenJevX v0.5.2
A small, CPU-first decision model: yes/no, pick-one and 0–N rating questions about a state (JSON or text),
answered on a laptop CPU. **It is meant to run in a container next to your own services**:
one container, the model inside, your requests never leave your machine.
- Site: [muthuishere.github.io/openjevx](https://muthuishere.github.io/openjevx/)
- Server, releases, Docker image, recipes: [github.com/muthuishere/openjevx](https://github.com/muthuishere/openjevx)
- CLI: [jevx](https://github.com/muthuishere/jevx)
**The model is a folder:** `openjevx.w8.onnx` (8-bit weight-only, 598 MB; activations stay float, so answers
don't depend on what else is in the request) + `config.json` (this model's own calibration temperatures) +
`tokenizer.json`. The other files are the trainable checkpoint for fine-tuning further: `model.safetensors`,
`encoder/`, `tokenizer/`, `rl_agent_config.json`. Older versions: git revisions `v0.5.0` and `v0.4.0`.
## Run it in a container (recommended)
```bash
git clone https://github.com/muthuishere/openjevx.git && cd openjevx
docker compose up -d --build # server + model on http://127.0.0.1:21118
```
Then ask it:
```bash
curl -s http://127.0.0.1:21118/v1/systemone -d '{
"state": {"age": 17},
"questions": {
"adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
"team": {"type": "choice", "instructions": "Which team should handle this ticket?",
"criteria": {"billing": "payments and charges", "engineering": "bugs and outages"}},
"sev": {"type": "score", "instructions": "How severe is this?", "criteria": ["low", "medium", "high"]}
}}'
```
Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
Prefer a plain binary? macOS (Apple Silicon), Linux and Windows builds of the v0.5.2 server (ONNX Runtime 1.29)
are on the [release page](https://github.com/muthuishere/openjevx/releases/latest). The server can also load this
model folder straight from S3 (`OPENJEVX_MODEL=s3://bucket/prefix/`); see the GitHub README.
## Or in Python
Same answers as the server (a port of its prompt layout, markers, input length, temperature and softmax). Checked
against the server on the same model: probabilities agree to within about 0.005; the rest is ONNX Runtime versions
rounding differently.
```bash
pip install onnxruntime tokenizers huggingface_hub
```
```python
from huggingface_hub import hf_hub_download
import importlib.util
path = hf_hub_download("muthuishere/openjevx", "openjevx.py")
spec = importlib.util.spec_from_file_location("openjevx", path); openjevx = importlib.util.module_from_spec(spec)
spec.loader.exec_module(openjevx)
jev = openjevx.OpenJevX() # downloads openjevx.w8.onnx + tokenizer.json + config.json once
print(jev.decide({"age": 17}, {
"adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
}))
# {'answers': {'adult': {'type': 'noul', 'noul': 0.08, ...}}} (about 0.08)
```
`openjevx.py` is a single file (also in the GitHub repo under `python/`); copy it into your project if you
prefer. Request and answer shapes match `POST /v1/systemone`.
## Release gate
The model folder on the Go server, on CPU. The gate sets share **no state** with training.
"Right and confident" = right AND past jevx's default thresholds (yes >= 0.8, no <= 0.2, choice/score >= 0.6).
Limits for the basics gates: right and confident >= 90%, confidently wrong <= 2%.
| Gate set | Accuracy | Right and confident | Confidently wrong |
|---|---|---|---|
| Everyday basics (463) | 99.1% | 98.7% | 0.9% |
| Rule-checking basics (300) | 100% | 100% | 0.0% |
| Held-out rule-checking (11,949) | 98.9% | 98.8% | 1.0% |
| Held-out software-work roles (9,159) | 98.5% | 97.8% | 1.0% |
| Logs gate (900) | 92.0% | 91.9% | 7.8% |
The 13 fundamentals through the jevx CLI with default thresholds: 13/13.
**Known misses:** on log triage it is confidently wrong 7.8% of the time, and on real public incident postmortems
(102 questions) 20.6%. On BGL supercomputer operator-log alerts (214, real labels) it is confidently wrong 35.0%: it
often answers no to `RAS KERNEL FATAL` lines. For that kind of log, use git revision `v0.5.0` (21.0% there). Check those answers.
## Benchmarks
Same server, CPU, jevx default thresholds. Full sets where n < 300, otherwise a fixed 300-row sample.
| Set | Kind | n | Accuracy | Right and confident | Confidently wrong |
|---|---|---|---|---|---|
| Dan Luu public postmortems | external, real labels | 102 | 68.6% | 62.7% | 20.6% |
| bigvul (CVE fixes) | external, held out | 201 | 45.8% | 0.0% | 0.0% |
| Defect detection | external, held out | 300 | 64.0% | 20.3% | 4.3% |
| Code search | external, held out | 300 | 94.0% | 90.0% | 4.0% |
| ms_marco relevance | external, held out | 300 | 72.3% | 32.0% | 6.7% |
| HDFS operator logs | external, held out | 118 | 68.6% | 18.6% | 0.0% |
| BGL operator logs | external, held out | 214 | 64.5% | 64.5% | 35.0% |
| Application/database/frontend logs | in-distribution | 1,224 | 95.3% | 95.3% | 4.5% |
It is not a general reasoner: on code and retrieval sets it is mostly unsure, so it rarely misleads there but rarely helps.
**Latency** (v0.5.1 server, ONNX Runtime 1.29, CPU only, Apple M5 Pro, p50 end to end): 22 ms for a short
request (36 tokens), 85 ms for a typical one (164 tokens), 540 ms for a long one (912 tokens). Inputs are cut at 512 tokens (`max_len` in
`config.json`, the length this model was trained at). Cold start is about
1 s once per process. ONNX Runtime 1.29 or newer is needed for this speed: it keeps the 8-bit weights 8-bit
(`MatMulNBits`); in Python, `pip install -U onnxruntime`.
## What v0.5.2 was trained on
791,889 decisions (465,583 rows). Labels come from evaluating rules or from real outcomes, never from another model's guesses.
| Area | Decisions | Share |
|---|---|---|
| Rule-checking across 38 business domains (retail pricing, recruiting, real estate, CI/CD, insurance claims, pharmacy stock, HR payroll, gaming, ...) | 200,155 | 25.3% |
| Software-work roles (developers, testers, tech leads, managers, operations, everyone, agent checks) | 183,543 | 23.2% |
| tasksource decision corpus | 146,567 | 18.5% |
| Public sets with real labels (CVE fixes from bigvul, defect detection, code search, ms_marco relevance, HDFS and BGL operator log alerts) | 104,990 | 13.3% |
| Rule-reading drills: everyday rules with many thresholds, decimals, dates over 2020-2035, quarter-hour meetings, opening hours, missing/equal/contains checks | 75,321 | 9.5% |
| Log triage: application (Java/Python/Node/Go/nginx), database (Postgres/MySQL, real SQLSTATE codes), frontend (browser/Sentry) | 60,001 | 7.6% |
| Everyday basics (driving licence age, store hours, parcel late, discount thresholds, fever, bag weight, ...) | 12,162 | 1.5% |
| Public typed-decisions | 9,150 | 1.2% |
Run: one RTX 4090 on Vast.ai, 791,239 decisions after packaging, one full pass in 2.5 h at ~87 items/s, about $1.25. Leakage-checked (22,576 test questions removed); every row checked against the trainer before renting.
## Using it with jevx
jevx caches answers by model name. After upgrading, run `jevx cache clear`, or use a versioned
model name in your profile (for example `--model openjevx-v0.5.2`).
## Fine-tune further
The whole pipeline (data generators, leakage check, one-job GPU training that returns the 8-bit ONNX, and the
release gate) is in the GitHub repo under
[`finetuning/`](https://github.com/muthuishere/openjevx/tree/main/finetuning). No training data is published.
The trainable checkpoint is also available as one archive: [openjevx-finetune-v0.5.2.tar.gz](https://pub-8da821f06ff747cda688f8267ed2aa96.r2.dev/openjevx-finetune-v0.5.2.tar.gz).
|