Text Classification
MLX
Safetensors
qwen3_5
neojev
jev
system-one
typed-decisions
classification
calibration
vision
apple-silicon
4-bit precision
Instructions to use dowands/neojev-mlx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use dowands/neojev-mlx with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download dowands/neojev-mlx --local-dir neojev-mlx
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Add TypeSafe-compatible server.py, LRU schema cache; JevBench numbers via JevBench's own adapter
Browse files- README.md +17 -4
- decider.py +8 -2
- jevbench_via_server.py +32 -0
- server.py +115 -0
README.md
CHANGED
|
@@ -53,6 +53,18 @@ d.ask(
|
|
| 53 |
# "churn": {"value": 0.41, ...}, "_tokens": 43}
|
| 54 |
```
|
| 55 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
Question types:
|
| 57 |
|
| 58 |
| type | fields | returns |
|
|
@@ -142,12 +154,12 @@ On standard public benchmarks within the held-out set (500-item zero-shot subset
|
|
| 142 |
|
| 143 |
## Compared with other open Jev-style models (JevBench)
|
| 144 |
|
| 145 |
-
[JevBench](https://github.com/fstandhartinger/jevbench) (MIT, Benchmark Heaven) is the independent benchmark most Jev-class projects report on. We ran NeoJev on its **231 public items**
|
| 146 |
|
| 147 |
| system | base | weights (memory) | public accuracy (231) | p50 / p95 latency | where the latency was measured |
|
| 148 |
|---|---|---|---|---|---|
|
| 149 |
-
| **NeoJev (this model), CUDA** | Qwen3.8-27B, 56 layers | 25.8 GB peak GPU (FP8) | **0.
|
| 150 |
-
| **NeoJev (this model), MLX** | Qwen3.8-27B, 56 layers | 13.4 GB (4-bit) | **0.
|
| 151 |
| Jev 1.13.0 (TypeSafe AI, closed) | closed | n/a | 0.866 | 0.65 / 0.72 s | production API over the network |
|
| 152 |
| decider-2b (Mapika) | Qwen3.5-2B | 3.8 GB | 0.710 | 0.26 / 0.28 s | JevBench server, network |
|
| 153 |
| openJev Verdict 1.4 | ModernBERT-base 151M | about 0.6 GB (est.) | 0.576 | 0.31 / 0.92 s | CPU, 4 threads |
|
|
@@ -163,8 +175,9 @@ On standard public benchmarks within the held-out set (500-item zero-shot subset
|
|
| 163 |
|
| 164 |
How to read it:
|
| 165 |
|
| 166 |
-
- **Accuracy.** On the public items NeoJev
|
| 167 |
- **Speed and memory.** Among the 27B-class systems, NeoJev's p50 is 0.13 s on an L20. SimpleJev and reflex-27b show 1.0 s and 1.9 s in JevBench's runs (with a network hop, on other hardware, so treat this as an order of magnitude only). NeoJev also needs about half the memory of a BF16 27B, because it runs 56 of 64 layers of the FP8 checkpoint. Its p95 is dominated by the long documents of the hard tier; the easy and standard tiers take 0.11 s.
|
|
|
|
| 168 |
- **Latency is not directly comparable across rows.** Many JevBench rows include a network round trip from JevBench's server, and every row ran on different hardware.
|
| 169 |
- **This is not the official JevBench score.** The official score also needs the held-out and sealed items (which only JevBench runs; v1.4 adds 308 fresh sealed decisions), a judge tier, calibration and a cost figure.
|
| 170 |
- **Where the memory figures come from.** They are the published repository size where one exists (decider-2b, JevK5); otherwise they are an estimate of parameters × precision, marked "est.".
|
|
|
|
| 53 |
# "churn": {"value": 0.41, ...}, "_tokens": 43}
|
| 54 |
```
|
| 55 |
|
| 56 |
+
### Serving (TypeSafe-compatible API)
|
| 57 |
+
|
| 58 |
+
`server.py` exposes `POST /v1/systemone` in the TypeSafe request/response shape, so clients written for that API (and JevBench's `TypeSafeAdapter`) work unchanged:
|
| 59 |
+
|
| 60 |
+
```bash
|
| 61 |
+
python server.py --backend mlx --model dowands/NeoJev-Qwen3.8-27B-L56-MLX --port 8000
|
| 62 |
+
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{"state": "My card was charged twice.",
|
| 63 |
+
"questions": {"route": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "charges", "technical": "bugs"}}}}'
|
| 64 |
+
```
|
| 65 |
+
|
| 66 |
+
It serves one request at a time. `--layout state_first` (the default, used for the JevBench numbers) caches nothing; `--layout schema_first` caches the question prefix per schema (LRU, 64 schemas), which is faster when the same questions repeat.
|
| 67 |
+
|
| 68 |
Question types:
|
| 69 |
|
| 70 |
| type | fields | returns |
|
|
|
|
| 154 |
|
| 155 |
## Compared with other open Jev-style models (JevBench)
|
| 156 |
|
| 157 |
+
[JevBench](https://github.com/fstandhartinger/jevbench) (MIT, Benchmark Heaven) is the independent benchmark most Jev-class projects report on. We ran NeoJev on its **231 public items** the way JevBench runs TypeSafe-compatible systems: JevBench's own, unmodified `TypeSafeAdapter` sent every item to NeoJev's `server.py`, and JevBench's scoring code graded the answers (`jevbench_via_server.py` in this repo). The table below compares NeoJev with widely used open peers that JevBench measured. Peer numbers come from JevBench's published v1.4 results (repo commit `26eb72d`, 24 Sep 2026).
|
| 158 |
|
| 159 |
| system | base | weights (memory) | public accuracy (231) | p50 / p95 latency | where the latency was measured |
|
| 160 |
|---|---|---|---|---|---|
|
| 161 |
+
| **NeoJev (this model), CUDA** | Qwen3.8-27B, 56 layers | 25.8 GB peak GPU (FP8) | **0.861** | **0.132 / 1.33 s** | NVIDIA L20, local HTTP, serial |
|
| 162 |
+
| **NeoJev (this model), MLX** | Qwen3.8-27B, 56 layers | 13.4 GB (4-bit) | **0.853** | 0.52 / 9.7 s | Apple M5 Pro, local HTTP, serial |
|
| 163 |
| Jev 1.13.0 (TypeSafe AI, closed) | closed | n/a | 0.866 | 0.65 / 0.72 s | production API over the network |
|
| 164 |
| decider-2b (Mapika) | Qwen3.5-2B | 3.8 GB | 0.710 | 0.26 / 0.28 s | JevBench server, network |
|
| 165 |
| openJev Verdict 1.4 | ModernBERT-base 151M | about 0.6 GB (est.) | 0.576 | 0.31 / 0.92 s | CPU, 4 threads |
|
|
|
|
| 175 |
|
| 176 |
How to read it:
|
| 177 |
|
| 178 |
+
- **Accuracy.** On the public items NeoJev (0.861 CUDA, 0.853 MLX) is close to Jev 1.13.0 (0.866) and to the other Qwen3.8-27B systems (0.862 to 0.870). It is above the small models in the table (0.58 to 0.85; decider-2b 0.71, JevK5 0.85). Among open systems, only JevOne and NInfer Qwen3.8-Flash-Next (both 0.896, on 35B-class MoE models) and thinking-mode systems score higher. Frontier APIs (GPT, DeepSeek, Gemini) score higher as well.
|
| 179 |
- **Speed and memory.** Among the 27B-class systems, NeoJev's p50 is 0.13 s on an L20. SimpleJev and reflex-27b show 1.0 s and 1.9 s in JevBench's runs (with a network hop, on other hardware, so treat this as an order of magnitude only). NeoJev also needs about half the memory of a BF16 27B, because it runs 56 of 64 layers of the FP8 checkpoint. Its p95 is dominated by the long documents of the hard tier; the easy and standard tiers take 0.11 s.
|
| 180 |
+
- **Option order matters.** JevBench sends each choice question's options as a label-to-description map sorted by label, and NeoJev keeps the order it receives. When the options were instead put in each item's label order, NeoJev scored 0.866 (CUDA) and 0.870 (MLX); of the 119 choice items whose two orders differ, 8 flipped on MLX and 5 on CUDA. The table uses the JevBench path. Small or frozen decision models are generally order-sensitive, and JevBench documents this for other entrants too.
|
| 181 |
- **Latency is not directly comparable across rows.** Many JevBench rows include a network round trip from JevBench's server, and every row ran on different hardware.
|
| 182 |
- **This is not the official JevBench score.** The official score also needs the held-out and sealed items (which only JevBench runs; v1.4 adds 308 fresh sealed decisions), a judge tier, calibration and a cost figure.
|
| 183 |
- **Where the memory figures come from.** They are the published repository size where one exists (decider-2b, JevK5); otherwise they are an estimate of parameters × precision, marked "est.".
|
decider.py
CHANGED
|
@@ -29,6 +29,7 @@ How it works
|
|
| 29 |
"""
|
| 30 |
import json
|
| 31 |
import os
|
|
|
|
| 32 |
|
| 33 |
import mlx.core as mx
|
| 34 |
import mlx.nn as nn
|
|
@@ -98,7 +99,8 @@ class Decider:
|
|
| 98 |
self.W = self.W.astype(mx.float32)
|
| 99 |
self.norm_w = self.lm.model.norm.weight.astype(mx.float32)
|
| 100 |
self.eps = self.lm.model.norm.eps
|
| 101 |
-
self._schema_cache =
|
|
|
|
| 102 |
|
| 103 |
def _enc(self, s):
|
| 104 |
return self.tok.encode(s, add_special_tokens=False)
|
|
@@ -113,7 +115,11 @@ class Decider:
|
|
| 113 |
return h[0] # pre-norm residual stream of the last layer
|
| 114 |
|
| 115 |
def _prefix(self, key, text):
|
| 116 |
-
if key
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
cache = self.lm.make_cache()
|
| 118 |
self._run(self._enc(text), cache)
|
| 119 |
mx.eval([c.state for c in cache])
|
|
|
|
| 29 |
"""
|
| 30 |
import json
|
| 31 |
import os
|
| 32 |
+
from collections import OrderedDict
|
| 33 |
|
| 34 |
import mlx.core as mx
|
| 35 |
import mlx.nn as nn
|
|
|
|
| 99 |
self.W = self.W.astype(mx.float32)
|
| 100 |
self.norm_w = self.lm.model.norm.weight.astype(mx.float32)
|
| 101 |
self.eps = self.lm.model.norm.eps
|
| 102 |
+
self._schema_cache = OrderedDict() # LRU of cached question prefixes
|
| 103 |
+
self.max_schemas = 64
|
| 104 |
|
| 105 |
def _enc(self, s):
|
| 106 |
return self.tok.encode(s, add_special_tokens=False)
|
|
|
|
| 115 |
return h[0] # pre-norm residual stream of the last layer
|
| 116 |
|
| 117 |
def _prefix(self, key, text):
|
| 118 |
+
if key in self._schema_cache:
|
| 119 |
+
self._schema_cache.move_to_end(key)
|
| 120 |
+
else:
|
| 121 |
+
while len(self._schema_cache) >= self.max_schemas:
|
| 122 |
+
self._schema_cache.popitem(last=False)
|
| 123 |
cache = self.lm.make_cache()
|
| 124 |
self._run(self._enc(text), cache)
|
| 125 |
mx.eval([c.state for c in cache])
|
jevbench_via_server.py
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Run JevBench's own TypeSafe adapter (unmodified) against a NeoJev server on the 231 public items.
|
| 2 |
+
usage: jevbench_via_server.py <endpoint> <out.jsonl>"""
|
| 3 |
+
import json
|
| 4 |
+
import os
|
| 5 |
+
import sys
|
| 6 |
+
|
| 7 |
+
ROOT = os.path.join(os.path.dirname(__file__), "..")
|
| 8 |
+
JB = os.environ.get("JB_DIR", os.path.join(ROOT, "reviews", "jevbench"))
|
| 9 |
+
sys.path.insert(0, JB)
|
| 10 |
+
from jevbench.adapters.typesafe import TypeSafeAdapter # noqa: E402
|
| 11 |
+
from jevbench.scoring import score_task # noqa: E402
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
class T:
|
| 15 |
+
def __init__(self, r):
|
| 16 |
+
self.state, self.question, self.labels, self.expected = r["state"], r["question"], r["labels"], r["expected"]
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
ep, out = sys.argv[1], sys.argv[2]
|
| 20 |
+
ad = TypeSafeAdapter(endpoint=ep, model="neojev", key_env=None)
|
| 21 |
+
rows = []
|
| 22 |
+
for tier in ["easy", "original", "hard"]:
|
| 23 |
+
rows += [json.loads(l) for l in open(os.path.join(JB, "datasets", "public", f"{tier}.jsonl"))]
|
| 24 |
+
ok = 0
|
| 25 |
+
with open(out, "w") as f:
|
| 26 |
+
for r in rows:
|
| 27 |
+
res = ad.run(T(r))
|
| 28 |
+
s = score_task(res.probs, T(r)) if res.ok else {"correct": False, "valid": False}
|
| 29 |
+
ok += res.ok
|
| 30 |
+
f.write(json.dumps({"id": r["id"], "tier": r["id"].split("-")[0], "ok": res.ok, "error": res.error,
|
| 31 |
+
"latency_s": round(res.latency_s, 4), "correct": s["correct"], "probs": res.probs}) + "\n")
|
| 32 |
+
print("requests ok", ok, "/", len(rows))
|
server.py
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""TypeSafe-compatible HTTP server for NeoJev: POST /v1/systemone, GET /health (stdlib only).
|
| 2 |
+
|
| 3 |
+
python server.py --backend mlx --model dowands/NeoJev-Qwen3.8-27B-L56-MLX --port 8000
|
| 4 |
+
python server.py --backend cuda --model dowands/NeoJev-Qwen3.8-27B-L56 --port 8000
|
| 5 |
+
|
| 6 |
+
Request: {"state": str | object, "questions": {name: {"type": "choice"|"score"|"noul", "instructions": str,
|
| 7 |
+
"criteria": ...}}}
|
| 8 |
+
choice criteria: {label: description | null} (or a list of labels)
|
| 9 |
+
score criteria: [level description, ...] (labels are "0", "1", ...)
|
| 10 |
+
noul criteria: {"true": ..., "false": ...} (optional)
|
| 11 |
+
Response: {"model": ..., "answers": {name: {"type": ..., "choice"|"score"|"noul": ..., "probabilities": {...},
|
| 12 |
+
"confidence": ...}}, "usage": {"input_tokens": n}}
|
| 13 |
+
|
| 14 |
+
--layout state_first (default) puts the state before the questions and caches nothing; it is the layout used for
|
| 15 |
+
the published JevBench numbers. --layout schema_first caches the question prefix per schema (LRU), which is faster
|
| 16 |
+
when the same questions are asked about many states.
|
| 17 |
+
"""
|
| 18 |
+
import argparse
|
| 19 |
+
import json
|
| 20 |
+
from http.server import BaseHTTPRequestHandler, HTTPServer
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def translate(questions):
|
| 24 |
+
"""TypeSafe question dict -> (NeoJev question dict, per-question label list)."""
|
| 25 |
+
out, labels = {}, {}
|
| 26 |
+
for name, q in questions.items():
|
| 27 |
+
t, crit, ins = q["type"], q.get("criteria"), q.get("instructions", "")
|
| 28 |
+
if t == "choice":
|
| 29 |
+
if isinstance(crit, dict):
|
| 30 |
+
labs = list(crit)
|
| 31 |
+
opts = [f"{l}: {crit[l]}" if crit[l] else l for l in labs]
|
| 32 |
+
else:
|
| 33 |
+
labs = [str(x) for x in (crit or [])]
|
| 34 |
+
opts = labs
|
| 35 |
+
out[name] = {"type": "choice", "options": opts, "prompt": ins}
|
| 36 |
+
elif t == "noul":
|
| 37 |
+
c = crit or {}
|
| 38 |
+
labs = ["no", "yes"]
|
| 39 |
+
out[name] = {"type": "choice", "prompt": ins,
|
| 40 |
+
"options": [f"no: {c['false']}" if c.get("false") else "no",
|
| 41 |
+
f"yes: {c['true']}" if c.get("true") else "yes"]}
|
| 42 |
+
elif t == "score":
|
| 43 |
+
levels = [str(x) for x in (crit or [])]
|
| 44 |
+
labs = [str(i) for i in range(len(levels))]
|
| 45 |
+
out[name] = {"type": "score", "levels": levels, "prompt": ins}
|
| 46 |
+
else:
|
| 47 |
+
raise ValueError(f"unknown question type {t!r}")
|
| 48 |
+
labels[name] = (t, labs)
|
| 49 |
+
return out, labels
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def main():
|
| 53 |
+
ap = argparse.ArgumentParser()
|
| 54 |
+
ap.add_argument("--backend", choices=["mlx", "cuda"], required=True)
|
| 55 |
+
ap.add_argument("--model", required=True)
|
| 56 |
+
ap.add_argument("--port", type=int, default=8000)
|
| 57 |
+
ap.add_argument("--host", default="127.0.0.1")
|
| 58 |
+
ap.add_argument("--layout", choices=["state_first", "schema_first"], default="state_first")
|
| 59 |
+
ap.add_argument("--token-budget", type=int, default=10 ** 6, help="cap on tokens per call (default: no truncation)")
|
| 60 |
+
a = ap.parse_args()
|
| 61 |
+
if a.backend == "mlx":
|
| 62 |
+
from decider import Decider
|
| 63 |
+
d = Decider(a.model, token_budget=a.token_budget)
|
| 64 |
+
else:
|
| 65 |
+
from decider_cuda import Decider
|
| 66 |
+
d = Decider(a.model, token_budget=a.token_budget)
|
| 67 |
+
model_name = "neojev-qwen3.8-27b-l56"
|
| 68 |
+
|
| 69 |
+
class H(BaseHTTPRequestHandler):
|
| 70 |
+
def _send(self, code, obj):
|
| 71 |
+
b = json.dumps(obj).encode()
|
| 72 |
+
self.send_response(code)
|
| 73 |
+
self.send_header("Content-Type", "application/json")
|
| 74 |
+
self.send_header("Content-Length", str(len(b)))
|
| 75 |
+
self.end_headers()
|
| 76 |
+
self.wfile.write(b)
|
| 77 |
+
|
| 78 |
+
def log_message(self, *args):
|
| 79 |
+
pass
|
| 80 |
+
|
| 81 |
+
def do_GET(self):
|
| 82 |
+
self._send(200, {"ok": True, "model": model_name}) if self.path == "/health" else self._send(404, {"error": "not found"})
|
| 83 |
+
|
| 84 |
+
def do_POST(self):
|
| 85 |
+
if self.path != "/v1/systemone":
|
| 86 |
+
return self._send(404, {"error": "not found"})
|
| 87 |
+
try:
|
| 88 |
+
req = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0))))
|
| 89 |
+
state = req["state"] if isinstance(req["state"], str) else json.dumps(req["state"], ensure_ascii=False, indent=1)
|
| 90 |
+
qs, labels = translate(req["questions"])
|
| 91 |
+
res = d.ask(state, qs, cache_schema=(a.layout == "schema_first"))
|
| 92 |
+
except Exception as e: # malformed request
|
| 93 |
+
return self._send(400, {"error": f"{type(e).__name__}: {e}"})
|
| 94 |
+
answers = {}
|
| 95 |
+
for name, (t, labs) in labels.items():
|
| 96 |
+
p = res[name]["probs_list"]
|
| 97 |
+
probs = dict(zip(labs, p))
|
| 98 |
+
j = max(range(len(p)), key=lambda k: p[k])
|
| 99 |
+
ans = {"type": t, "probabilities": probs, "confidence": p[j]}
|
| 100 |
+
if t == "choice":
|
| 101 |
+
ans["choice"] = labs[j]
|
| 102 |
+
elif t == "noul":
|
| 103 |
+
ans["noul"] = probs["yes"]
|
| 104 |
+
else:
|
| 105 |
+
ans["score"] = sum(k * v for k, v in enumerate(p))
|
| 106 |
+
answers[name] = ans
|
| 107 |
+
self._send(200, {"model": model_name, "answers": answers, "usage": {"input_tokens": res["_tokens"]}})
|
| 108 |
+
|
| 109 |
+
print(f"NeoJev ({a.backend}, {a.layout}) on http://{a.host}:{a.port}/v1/systemone", flush=True)
|
| 110 |
+
# single-threaded on purpose: one GPU serves one request at a time, and MLX streams are bound to the main thread
|
| 111 |
+
HTTPServer((a.host, a.port), H).serve_forever()
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
if __name__ == "__main__":
|
| 115 |
+
main()
|