File size: 8,016 Bytes
2abbcba
 
 
69619b6
2abbcba
6eacb9d
69619b6
353c1b7
 
3d7be4a
69619b6
3d7be4a
 
 
 
883bd72
 
 
6eacb9d
3d7be4a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6eacb9d
 
 
3d7be4a
 
 
6eacb9d
 
 
3d7be4a
 
 
 
 
 
 
 
 
 
 
 
 
883bd72
3d7be4a
 
 
6eacb9d
3d7be4a
 
 
 
69619b6
6eacb9d
353c1b7
883bd72
 
6eacb9d
353c1b7
883bd72
353c1b7
6eacb9d
 
 
 
 
353c1b7
6eacb9d
 
 
5399cb8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
353c1b7
0098fc0
6eacb9d
 
0098fc0
 
353c1b7
6eacb9d
3d7be4a
6eacb9d
69619b6
883bd72
69619b6
6eacb9d
 
 
 
 
 
 
883bd72
 
6eacb9d
69619b6
353c1b7
 
 
6eacb9d
3d7be4a
69619b6
3d7be4a
 
 
 
353c1b7
6eacb9d
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
---
license: apache-2.0
base_model: convaiinnovations/laya
tags: [decision-model, onnx, classification, jevx]
---
# OpenJevX v0.5.2

A small, CPU-first decision model: yes/no, pick-one and 0–N rating questions about a state (JSON or text),
answered on a laptop CPU. **It is meant to run in a container next to your own services**:
one container, the model inside, your requests never leave your machine.

- Site: [muthuishere.github.io/openjevx](https://muthuishere.github.io/openjevx/)
- Server, releases, Docker image, recipes: [github.com/muthuishere/openjevx](https://github.com/muthuishere/openjevx)
- CLI: [jevx](https://github.com/muthuishere/jevx)

**The model is a folder:** `openjevx.w8.onnx` (8-bit weight-only, 598 MB; activations stay float, so answers
don't depend on what else is in the request) + `config.json` (this model's own calibration temperatures) +
`tokenizer.json`. The other files are the trainable checkpoint for fine-tuning further: `model.safetensors`,
`encoder/`, `tokenizer/`, `rl_agent_config.json`. Older versions: git revisions `v0.5.0` and `v0.4.0`.

## Run it in a container (recommended)

```bash
git clone https://github.com/muthuishere/openjevx.git && cd openjevx
docker compose up -d --build          # server + model on http://127.0.0.1:21118
```

Then ask it:

```bash
curl -s http://127.0.0.1:21118/v1/systemone -d '{
  "state": {"age": 17},
  "questions": {
    "adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
    "team":  {"type": "choice", "instructions": "Which team should handle this ticket?",
              "criteria": {"billing": "payments and charges", "engineering": "bugs and outages"}},
    "sev":   {"type": "score", "instructions": "How severe is this?", "criteria": ["low", "medium", "high"]}
  }}'
```

Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
Prefer a plain binary? macOS (Apple Silicon), Linux and Windows builds of the v0.5.2 server (ONNX Runtime 1.29)
are on the [release page](https://github.com/muthuishere/openjevx/releases/latest). The server can also load this
model folder straight from S3 (`OPENJEVX_MODEL=s3://bucket/prefix/`); see the GitHub README.

## Or in Python

Same answers as the server (a port of its prompt layout, markers, input length, temperature and softmax). Checked
against the server on the same model: probabilities agree to within about 0.005; the rest is ONNX Runtime versions
rounding differently.

```bash
pip install onnxruntime tokenizers huggingface_hub
```

```python
from huggingface_hub import hf_hub_download
import importlib.util

path = hf_hub_download("muthuishere/openjevx", "openjevx.py")
spec = importlib.util.spec_from_file_location("openjevx", path); openjevx = importlib.util.module_from_spec(spec)
spec.loader.exec_module(openjevx)

jev = openjevx.OpenJevX()   # downloads openjevx.w8.onnx + tokenizer.json + config.json once
print(jev.decide({"age": 17}, {
    "adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
}))
# {'answers': {'adult': {'type': 'noul', 'noul': 0.08, ...}}}  (about 0.08)
```

`openjevx.py` is a single file (also in the GitHub repo under `python/`); copy it into your project if you
prefer. Request and answer shapes match `POST /v1/systemone`.

## Release gate

The model folder on the Go server, on CPU. The gate sets share **no state** with training.
"Right and confident" = right AND past jevx's default thresholds (yes >= 0.8, no <= 0.2, choice/score >= 0.6).
Limits for the basics gates: right and confident >= 90%, confidently wrong <= 2%.

| Gate set | Accuracy | Right and confident | Confidently wrong |
|---|---|---|---|
| Everyday basics (463) | 99.1% | 98.7% | 0.9% |
| Rule-checking basics (300) | 100% | 100% | 0.0% |
| Held-out rule-checking (11,949) | 98.9% | 98.8% | 1.0% |
| Held-out software-work roles (9,159) | 98.5% | 97.8% | 1.0% |
| Logs gate (900) | 92.0% | 91.9% | 7.8% |

The 13 fundamentals through the jevx CLI with default thresholds: 13/13.

**Known misses:** on log triage it is confidently wrong 7.8% of the time, and on real public incident postmortems
(102 questions) 20.6%. On BGL supercomputer operator-log alerts (214, real labels) it is confidently wrong 35.0%: it
often answers no to `RAS KERNEL FATAL` lines. For that kind of log, use git revision `v0.5.0` (21.0% there). Check those answers.

## Benchmarks

Same server, CPU, jevx default thresholds. Full sets where n < 300, otherwise a fixed 300-row sample.

| Set | Kind | n | Accuracy | Right and confident | Confidently wrong |
|---|---|---|---|---|---|
| Dan Luu public postmortems | external, real labels | 102 | 68.6% | 62.7% | 20.6% |
| bigvul (CVE fixes) | external, held out | 201 | 45.8% | 0.0% | 0.0% |
| Defect detection | external, held out | 300 | 64.0% | 20.3% | 4.3% |
| Code search | external, held out | 300 | 94.0% | 90.0% | 4.0% |
| ms_marco relevance | external, held out | 300 | 72.3% | 32.0% | 6.7% |
| HDFS operator logs | external, held out | 118 | 68.6% | 18.6% | 0.0% |
| BGL operator logs | external, held out | 214 | 64.5% | 64.5% | 35.0% |
| Application/database/frontend logs | in-distribution | 1,224 | 95.3% | 95.3% | 4.5% |

It is not a general reasoner: on code and retrieval sets it is mostly unsure, so it rarely misleads there but rarely helps.

**Latency** (v0.5.1 server, ONNX Runtime 1.29, CPU only, Apple M5 Pro, p50 end to end): 22 ms for a short
request (36 tokens), 85 ms for a typical one (164 tokens), 540 ms for a long one (912 tokens). Inputs are cut at 512 tokens (`max_len` in
`config.json`, the length this model was trained at). Cold start is about
1 s once per process. ONNX Runtime 1.29 or newer is needed for this speed: it keeps the 8-bit weights 8-bit
(`MatMulNBits`); in Python, `pip install -U onnxruntime`.

## What v0.5.2 was trained on

791,889 decisions (465,583 rows). Labels come from evaluating rules or from real outcomes, never from another model's guesses.

| Area | Decisions | Share |
|---|---|---|
| Rule-checking across 38 business domains (retail pricing, recruiting, real estate, CI/CD, insurance claims, pharmacy stock, HR payroll, gaming, ...) | 200,155 | 25.3% |
| Software-work roles (developers, testers, tech leads, managers, operations, everyone, agent checks) | 183,543 | 23.2% |
| tasksource decision corpus | 146,567 | 18.5% |
| Public sets with real labels (CVE fixes from bigvul, defect detection, code search, ms_marco relevance, HDFS and BGL operator log alerts) | 104,990 | 13.3% |
| Rule-reading drills: everyday rules with many thresholds, decimals, dates over 2020-2035, quarter-hour meetings, opening hours, missing/equal/contains checks | 75,321 | 9.5% |
| Log triage: application (Java/Python/Node/Go/nginx), database (Postgres/MySQL, real SQLSTATE codes), frontend (browser/Sentry) | 60,001 | 7.6% |
| Everyday basics (driving licence age, store hours, parcel late, discount thresholds, fever, bag weight, ...) | 12,162 | 1.5% |
| Public typed-decisions | 9,150 | 1.2% |

Run: one RTX 4090 on Vast.ai, 791,239 decisions after packaging, one full pass in 2.5 h at ~87 items/s, about $1.25. Leakage-checked (22,576 test questions removed); every row checked against the trainer before renting.

## Using it with jevx

jevx caches answers by model name. After upgrading, run `jevx cache clear`, or use a versioned
model name in your profile (for example `--model openjevx-v0.5.2`).

## Fine-tune further

The whole pipeline (data generators, leakage check, one-job GPU training that returns the 8-bit ONNX, and the
release gate) is in the GitHub repo under
[`finetuning/`](https://github.com/muthuishere/openjevx/tree/main/finetuning). No training data is published.

The trainable checkpoint is also available as one archive: [openjevx-finetune-v0.5.2.tar.gz](https://pub-8da821f06ff747cda688f8267ed2aa96.r2.dev/openjevx-finetune-v0.5.2.tar.gz).