openjevx / README.md

Card: v0.5.2 benchmarks; BGL operator-log regression named

5399cb8 verified about 1 month ago

8.02 kB

---
license: apache-2.0
base_model: convaiinnovations/laya
tags: [decision-model, onnx, classification, jevx]
---
# OpenJevX v0.5.2

A small, CPU-first decision model: yes/no, pick-one and 0–N rating questions about a state (JSON or text),
answered on a laptop CPU. **It is meant to run in a container next to your own services**:
one container, the model inside, your requests never leave your machine.

- Site: [muthuishere.github.io/openjevx](https://muthuishere.github.io/openjevx/)
- Server, releases, Docker image, recipes: [github.com/muthuishere/openjevx](https://github.com/muthuishere/openjevx)
- CLI: [jevx](https://github.com/muthuishere/jevx)

**The model is a folder:** `openjevx.w8.onnx` (8-bit weight-only, 598 MB; activations stay float, so answers
don't depend on what else is in the request) + `config.json` (this model's own calibration temperatures) +
`tokenizer.json`. The other files are the trainable checkpoint for fine-tuning further: `model.safetensors`,
`encoder/`, `tokenizer/`, `rl_agent_config.json`. Older versions: git revisions `v0.5.0` and `v0.4.0`.

## Run it in a container (recommended)

```bash
git clone https://github.com/muthuishere/openjevx.git && cd openjevx
docker compose up -d --build          # server + model on http://127.0.0.1:21118
```

Then ask it:

```bash
curl -s http://127.0.0.1:21118/v1/systemone -d '{
  "state": {"age": 17},
  "questions": {
    "adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
    "team":  {"type": "choice", "instructions": "Which team should handle this ticket?",
              "criteria": {"billing": "payments and charges", "engineering": "bugs and outages"}},
    "sev":   {"type": "score", "instructions": "How severe is this?", "criteria": ["low", "medium", "high"]}
  }}'
```

Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
Prefer a plain binary? macOS (Apple Silicon), Linux and Windows builds of the v0.5.2 server (ONNX Runtime 1.29)
are on the [release page](https://github.com/muthuishere/openjevx/releases/latest). The server can also load this
model folder straight from S3 (`OPENJEVX_MODEL=s3://bucket/prefix/`); see the GitHub README.

## Or in Python

Same answers as the server (a port of its prompt layout, markers, input length, temperature and softmax). Checked
against the server on the same model: probabilities agree to within about 0.005; the rest is ONNX Runtime versions
rounding differently.

```bash
pip install onnxruntime tokenizers huggingface_hub
```

```python
from huggingface_hub import hf_hub_download
import importlib.util
path = hf_hub_download("muthuishere/openjevx", "openjevx.py")
spec = importlib.util.spec_from_file_location("openjevx", path); openjevx = importlib.util.module_from_spec(spec)
spec.loader.exec_module(openjevx)
jev = openjevx.OpenJevX()   # downloads openjevx.w8.onnx + tokenizer.json + config.json once
print(jev.decide({"age": 17}, {
    "adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
}))
# {'answers': {'adult': {'type': 'noul', 'noul': 0.08, ...}}}  (about 0.08)
```

`openjevx.py` is a single file (also in the GitHub repo under `python/`); copy it into your project if you
prefer. Request and answer shapes match `POST /v1/systemone`.

## Release gate

The model folder on the Go server, on CPU. The gate sets share **no state** with training.
"Right and confident" = right AND past jevx's default thresholds (yes >= 0.8, no <= 0.2, choice/score >= 0.6).
Limits for the basics gates: right and confident >= 90%, confidently wrong <= 2%.

| Gate set | Accuracy | Right and confident | Confidently wrong |
|---|---|---|---|
| Everyday basics (463) | 99.1% | 98.7% | 0.9% |
| Rule-checking basics (300) | 100% | 100% | 0.0% |
| Held-out rule-checking (11,949) | 98.9% | 98.8% | 1.0% |
| Held-out software-work roles (9,159) | 98.5% | 97.8% | 1.0% |
| Logs gate (900) | 92.0% | 91.9% | 7.8% |

The 13 fundamentals through the jevx CLI with default thresholds: 13/13.

**Known misses:** on log triage it is confidently wrong 7.8% of the time, and on real public incident postmortems
(102 questions) 20.6%. On BGL supercomputer operator-log alerts (214, real labels) it is confidently wrong 35.0%: it
often answers no to `RAS KERNEL FATAL` lines. For that kind of log, use git revision `v0.5.0` (21.0% there). Check those answers.

## Benchmarks

Same server, CPU, jevx default thresholds. Full sets where n < 300, otherwise a fixed 300-row sample.

| Set | Kind | n | Accuracy | Right and confident | Confidently wrong |
|---|---|---|---|---|---|
| Dan Luu public postmortems | external, real labels | 102 | 68.6% | 62.7% | 20.6% |
| bigvul (CVE fixes) | external, held out | 201 | 45.8% | 0.0% | 0.0% |
| Defect detection | external, held out | 300 | 64.0% | 20.3% | 4.3% |
| Code search | external, held out | 300 | 94.0% | 90.0% | 4.0% |
| ms_marco relevance | external, held out | 300 | 72.3% | 32.0% | 6.7% |
| HDFS operator logs | external, held out | 118 | 68.6% | 18.6% | 0.0% |
| BGL operator logs | external, held out | 214 | 64.5% | 64.5% | 35.0% |
| Application/database/frontend logs | in-distribution | 1,224 | 95.3% | 95.3% | 4.5% |
It is not a general reasoner: on code and retrieval sets it is mostly unsure, so it rarely misleads there but rarely helps.
**Latency** (v0.5.1 server, ONNX Runtime 1.29, CPU only, Apple M5 Pro, p50 end to end): 22 ms for a short
request (36 tokens), 85 ms for a typical one (164 tokens), 540 ms for a long one (912 tokens). Inputs are cut at 512 tokens (`max_len` in
`config.json`, the length this model was trained at). Cold start is about
1 s once per process. ONNX Runtime 1.29 or newer is needed for this speed: it keeps the 8-bit weights 8-bit
(`MatMulNBits`); in Python, `pip install -U onnxruntime`.

## What v0.5.2 was trained on

791,889 decisions (465,583 rows). Labels come from evaluating rules or from real outcomes, never from another model's guesses.

| Area | Decisions | Share |
|---|---|---|
| Rule-checking across 38 business domains (retail pricing, recruiting, real estate, CI/CD, insurance claims, pharmacy stock, HR payroll, gaming, ...) | 200,155 | 25.3% |
| Software-work roles (developers, testers, tech leads, managers, operations, everyone, agent checks) | 183,543 | 23.2% |
| tasksource decision corpus | 146,567 | 18.5% |
| Public sets with real labels (CVE fixes from bigvul, defect detection, code search, ms_marco relevance, HDFS and BGL operator log alerts) | 104,990 | 13.3% |
| Rule-reading drills: everyday rules with many thresholds, decimals, dates over 2020-2035, quarter-hour meetings, opening hours, missing/equal/contains checks | 75,321 | 9.5% |
| Log triage: application (Java/Python/Node/Go/nginx), database (Postgres/MySQL, real SQLSTATE codes), frontend (browser/Sentry) | 60,001 | 7.6% |
| Everyday basics (driving licence age, store hours, parcel late, discount thresholds, fever, bag weight, ...) | 12,162 | 1.5% |
| Public typed-decisions | 9,150 | 1.2% |
Run: one RTX 4090 on Vast.ai, 791,239 decisions after packaging, one full pass in 2.5 h at ~87 items/s, about $1.25. Leakage-checked (22,576 test questions removed); every row checked against the trainer before renting.
## Using it with jevx
jevx caches answers by model name. After upgrading, run `jevx cache clear`, or use a versioned
model name in your profile (for example `--model openjevx-v0.5.2`).
## Fine-tune further
The whole pipeline (data generators, leakage check, one-job GPU training that returns the 8-bit ONNX, and the
release gate) is in the GitHub repo under
[`finetuning/`](https://github.com/muthuishere/openjevx/tree/main/finetuning). No training data is published.
The trainable checkpoint is also available as one archive: [openjevx-finetune-v0.5.2.tar.gz](https://pub-8da821f06ff747cda688f8267ed2aa96.r2.dev/openjevx-finetune-v0.5.2.tar.gz).

	---
	license: apache-2.0
	base_model: convaiinnovations/laya
	tags: [decision-model, onnx, classification, jevx]
	---
	# OpenJevX v0.5.2

	A small, CPU-first decision model: yes/no, pick-one and 0–N rating questions about a state (JSON or text),
	answered on a laptop CPU. It is meant to run in a container next to your own services:
	one container, the model inside, your requests never leave your machine.

	- Site: [muthuishere.github.io/openjevx](https://muthuishere.github.io/openjevx/)
	- Server, releases, Docker image, recipes: [github.com/muthuishere/openjevx](https://github.com/muthuishere/openjevx)
	- CLI: [jevx](https://github.com/muthuishere/jevx)

	The model is a folder: `openjevx.w8.onnx` (8-bit weight-only, 598 MB; activations stay float, so answers
	don't depend on what else is in the request) + `config.json` (this model's own calibration temperatures) +
	`tokenizer.json`. The other files are the trainable checkpoint for fine-tuning further: `model.safetensors`,
	`encoder/`, `tokenizer/`, `rl_agent_config.json`. Older versions: git revisions `v0.5.0` and `v0.4.0`.

	## Run it in a container (recommended)

	```bash
	git clone https://github.com/muthuishere/openjevx.git && cd openjevx
	docker compose up -d --build # server + model on http://127.0.0.1:21118
	```

	Then ask it:

	```bash
	curl -s http://127.0.0.1:21118/v1/systemone -d '{
	"state": {"age": 17},
	"questions": {
	"adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
	"team": {"type": "choice", "instructions": "Which team should handle this ticket?",
	"criteria": {"billing": "payments and charges", "engineering": "bugs and outages"}},
	"sev": {"type": "score", "instructions": "How severe is this?", "criteria": ["low", "medium", "high"]}
	}}'
	```

	Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
	expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
	Prefer a plain binary? macOS (Apple Silicon), Linux and Windows builds of the v0.5.2 server (ONNX Runtime 1.29)
	are on the [release page](https://github.com/muthuishere/openjevx/releases/latest). The server can also load this
	model folder straight from S3 (`OPENJEVX_MODEL=s3://bucket/prefix/`); see the GitHub README.

	## Or in Python

	Same answers as the server (a port of its prompt layout, markers, input length, temperature and softmax). Checked
	against the server on the same model: probabilities agree to within about 0.005; the rest is ONNX Runtime versions
	rounding differently.

	```bash
	pip install onnxruntime tokenizers huggingface_hub
	```

	```python
	from huggingface_hub import hf_hub_download
	import importlib.util

	path = hf_hub_download("muthuishere/openjevx", "openjevx.py")
	spec = importlib.util.spec_from_file_location("openjevx", path); openjevx = importlib.util.module_from_spec(spec)
	spec.loader.exec_module(openjevx)

	jev = openjevx.OpenJevX() # downloads openjevx.w8.onnx + tokenizer.json + config.json once
	print(jev.decide({"age": 17}, {
	"adult": {"type": "noul", "instructions": "Is this person an adult (18 or older)?"},
	}))
	# {'answers': {'adult': {'type': 'noul', 'noul': 0.08, ...}}} (about 0.08)
	```

	`openjevx.py` is a single file (also in the GitHub repo under `python/`); copy it into your project if you
	prefer. Request and answer shapes match `POST /v1/systemone`.

	## Release gate

	The model folder on the Go server, on CPU. The gate sets share no state with training.
	"Right and confident" = right AND past jevx's default thresholds (yes >= 0.8, no <= 0.2, choice/score >= 0.6).
	Limits for the basics gates: right and confident >= 90%, confidently wrong <= 2%.

	\| Gate set \| Accuracy \| Right and confident \| Confidently wrong \|
	\|---\|---\|---\|---\|
	\| Everyday basics (463) \| 99.1% \| 98.7% \| 0.9% \|
	\| Rule-checking basics (300) \| 100% \| 100% \| 0.0% \|
	\| Held-out rule-checking (11,949) \| 98.9% \| 98.8% \| 1.0% \|
	\| Held-out software-work roles (9,159) \| 98.5% \| 97.8% \| 1.0% \|
	\| Logs gate (900) \| 92.0% \| 91.9% \| 7.8% \|

	The 13 fundamentals through the jevx CLI with default thresholds: 13/13.

	Known misses: on log triage it is confidently wrong 7.8% of the time, and on real public incident postmortems
	(102 questions) 20.6%. On BGL supercomputer operator-log alerts (214, real labels) it is confidently wrong 35.0%: it
	often answers no to `RAS KERNEL FATAL` lines. For that kind of log, use git revision `v0.5.0` (21.0% there). Check those answers.

	## Benchmarks

	Same server, CPU, jevx default thresholds. Full sets where n < 300, otherwise a fixed 300-row sample.

	\| Set \| Kind \| n \| Accuracy \| Right and confident \| Confidently wrong \|
	\|---\|---\|---\|---\|---\|---\|
	\| Dan Luu public postmortems \| external, real labels \| 102 \| 68.6% \| 62.7% \| 20.6% \|
	\| bigvul (CVE fixes) \| external, held out \| 201 \| 45.8% \| 0.0% \| 0.0% \|
	\| Defect detection \| external, held out \| 300 \| 64.0% \| 20.3% \| 4.3% \|
	\| Code search \| external, held out \| 300 \| 94.0% \| 90.0% \| 4.0% \|
	\| ms_marco relevance \| external, held out \| 300 \| 72.3% \| 32.0% \| 6.7% \|
	\| HDFS operator logs \| external, held out \| 118 \| 68.6% \| 18.6% \| 0.0% \|
	\| BGL operator logs \| external, held out \| 214 \| 64.5% \| 64.5% \| 35.0% \|
	\| Application/database/frontend logs \| in-distribution \| 1,224 \| 95.3% \| 95.3% \| 4.5% \|

	It is not a general reasoner: on code and retrieval sets it is mostly unsure, so it rarely misleads there but rarely helps.

	Latency (v0.5.1 server, ONNX Runtime 1.29, CPU only, Apple M5 Pro, p50 end to end): 22 ms for a short
	request (36 tokens), 85 ms for a typical one (164 tokens), 540 ms for a long one (912 tokens). Inputs are cut at 512 tokens (`max_len` in
	`config.json`, the length this model was trained at). Cold start is about
	1 s once per process. ONNX Runtime 1.29 or newer is needed for this speed: it keeps the 8-bit weights 8-bit
	(`MatMulNBits`); in Python, `pip install -U onnxruntime`.

	## What v0.5.2 was trained on

	791,889 decisions (465,583 rows). Labels come from evaluating rules or from real outcomes, never from another model's guesses.

	\| Area \| Decisions \| Share \|
	\|---\|---\|---\|
	\| Rule-checking across 38 business domains (retail pricing, recruiting, real estate, CI/CD, insurance claims, pharmacy stock, HR payroll, gaming, ...) \| 200,155 \| 25.3% \|
	\| Software-work roles (developers, testers, tech leads, managers, operations, everyone, agent checks) \| 183,543 \| 23.2% \|
	\| tasksource decision corpus \| 146,567 \| 18.5% \|
	\| Public sets with real labels (CVE fixes from bigvul, defect detection, code search, ms_marco relevance, HDFS and BGL operator log alerts) \| 104,990 \| 13.3% \|
	\| Rule-reading drills: everyday rules with many thresholds, decimals, dates over 2020-2035, quarter-hour meetings, opening hours, missing/equal/contains checks \| 75,321 \| 9.5% \|
	\| Log triage: application (Java/Python/Node/Go/nginx), database (Postgres/MySQL, real SQLSTATE codes), frontend (browser/Sentry) \| 60,001 \| 7.6% \|
	\| Everyday basics (driving licence age, store hours, parcel late, discount thresholds, fever, bag weight, ...) \| 12,162 \| 1.5% \|
	\| Public typed-decisions \| 9,150 \| 1.2% \|

	Run: one RTX 4090 on Vast.ai, 791,239 decisions after packaging, one full pass in 2.5 h at ~87 items/s, about $1.25. Leakage-checked (22,576 test questions removed); every row checked against the trainer before renting.

	## Using it with jevx

	jevx caches answers by model name. After upgrading, run `jevx cache clear`, or use a versioned
	model name in your profile (for example `--model openjevx-v0.5.2`).

	## Fine-tune further

	The whole pipeline (data generators, leakage check, one-job GPU training that returns the 8-bit ONNX, and the
	release gate) is in the GitHub repo under
	[`finetuning/`](https://github.com/muthuishere/openjevx/tree/main/finetuning). No training data is published.

	The trainable checkpoint is also available as one archive: [openjevx-finetune-v0.5.2.tar.gz](https://pub-8da821f06ff747cda688f8267ed2aa96.r2.dev/openjevx-finetune-v0.5.2.tar.gz).