dowands commited on
Commit
c4eeba4
·
verified ·
1 Parent(s): 7f1800a

Add TypeSafe-compatible server.py, LRU schema cache; JevBench numbers via JevBench's own adapter

Browse files
Files changed (4) hide show
  1. README.md +17 -4
  2. decider.py +8 -2
  3. jevbench_via_server.py +32 -0
  4. server.py +115 -0
README.md CHANGED
@@ -53,6 +53,18 @@ d.ask(
53
  # "churn": {"value": 0.41, ...}, "_tokens": 43}
54
  ```
55
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  Question types:
57
 
58
  | type | fields | returns |
@@ -142,12 +154,12 @@ On standard public benchmarks within the held-out set (500-item zero-shot subset
142
 
143
  ## Compared with other open Jev-style models (JevBench)
144
 
145
- [JevBench](https://github.com/fstandhartinger/jevbench) (MIT, Benchmark Heaven) is the independent benchmark most Jev-class projects report on. We ran NeoJev on its **231 public items** with JevBench's own scoring code. Each question was translated the same way for every item, and the translation was fixed before the run (`scripts/run_jevbench.py`). The table below compares NeoJev with widely used open peers that JevBench measured. Peer numbers come from JevBench's published v1.4 results (repo commit `26eb72d`, 24 Sep 2026).
146
 
147
  | system | base | weights (memory) | public accuracy (231) | p50 / p95 latency | where the latency was measured |
148
  |---|---|---|---|---|---|
149
- | **NeoJev (this model), CUDA** | Qwen3.8-27B, 56 layers | 25.8 GB peak GPU (FP8) | **0.866** | **0.131 / 1.33 s** | NVIDIA L20, in-process, serial |
150
- | **NeoJev (this model), MLX** | Qwen3.8-27B, 56 layers | 13.4 GB (4-bit) | **0.870** | 0.51 / 8.4 s | Apple M5 Pro, in-process, serial |
151
  | Jev 1.13.0 (TypeSafe AI, closed) | closed | n/a | 0.866 | 0.65 / 0.72 s | production API over the network |
152
  | decider-2b (Mapika) | Qwen3.5-2B | 3.8 GB | 0.710 | 0.26 / 0.28 s | JevBench server, network |
153
  | openJev Verdict 1.4 | ModernBERT-base 151M | about 0.6 GB (est.) | 0.576 | 0.31 / 0.92 s | CPU, 4 threads |
@@ -163,8 +175,9 @@ On standard public benchmarks within the held-out set (500-item zero-shot subset
163
 
164
  How to read it:
165
 
166
- - **Accuracy.** On the public items NeoJev matches Jev 1.13.0 and the other Qwen3.8-27B systems (0.862 to 0.870). It is above the small models in the table (0.58 to 0.85; decider-2b 0.71, JevK5 0.85). Among open systems, only JevOne and NInfer Qwen3.8-Flash-Next (both 0.896, on 35B-class MoE models) and thinking-mode systems score higher. Frontier APIs (GPT, DeepSeek, Gemini) score higher as well.
167
  - **Speed and memory.** Among the 27B-class systems, NeoJev's p50 is 0.13 s on an L20. SimpleJev and reflex-27b show 1.0 s and 1.9 s in JevBench's runs (with a network hop, on other hardware, so treat this as an order of magnitude only). NeoJev also needs about half the memory of a BF16 27B, because it runs 56 of 64 layers of the FP8 checkpoint. Its p95 is dominated by the long documents of the hard tier; the easy and standard tiers take 0.11 s.
 
168
  - **Latency is not directly comparable across rows.** Many JevBench rows include a network round trip from JevBench's server, and every row ran on different hardware.
169
  - **This is not the official JevBench score.** The official score also needs the held-out and sealed items (which only JevBench runs; v1.4 adds 308 fresh sealed decisions), a judge tier, calibration and a cost figure.
170
  - **Where the memory figures come from.** They are the published repository size where one exists (decider-2b, JevK5); otherwise they are an estimate of parameters × precision, marked "est.".
 
53
  # "churn": {"value": 0.41, ...}, "_tokens": 43}
54
  ```
55
 
56
+ ### Serving (TypeSafe-compatible API)
57
+
58
+ `server.py` exposes `POST /v1/systemone` in the TypeSafe request/response shape, so clients written for that API (and JevBench's `TypeSafeAdapter`) work unchanged:
59
+
60
+ ```bash
61
+ python server.py --backend mlx --model dowands/NeoJev-Qwen3.8-27B-L56-MLX --port 8000
62
+ curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{"state": "My card was charged twice.",
63
+ "questions": {"route": {"type": "choice", "instructions": "Which team?", "criteria": {"billing": "charges", "technical": "bugs"}}}}'
64
+ ```
65
+
66
+ It serves one request at a time. `--layout state_first` (the default, used for the JevBench numbers) caches nothing; `--layout schema_first` caches the question prefix per schema (LRU, 64 schemas), which is faster when the same questions repeat.
67
+
68
  Question types:
69
 
70
  | type | fields | returns |
 
154
 
155
  ## Compared with other open Jev-style models (JevBench)
156
 
157
+ [JevBench](https://github.com/fstandhartinger/jevbench) (MIT, Benchmark Heaven) is the independent benchmark most Jev-class projects report on. We ran NeoJev on its **231 public items** the way JevBench runs TypeSafe-compatible systems: JevBench's own, unmodified `TypeSafeAdapter` sent every item to NeoJev's `server.py`, and JevBench's scoring code graded the answers (`jevbench_via_server.py` in this repo). The table below compares NeoJev with widely used open peers that JevBench measured. Peer numbers come from JevBench's published v1.4 results (repo commit `26eb72d`, 24 Sep 2026).
158
 
159
  | system | base | weights (memory) | public accuracy (231) | p50 / p95 latency | where the latency was measured |
160
  |---|---|---|---|---|---|
161
+ | **NeoJev (this model), CUDA** | Qwen3.8-27B, 56 layers | 25.8 GB peak GPU (FP8) | **0.861** | **0.132 / 1.33 s** | NVIDIA L20, local HTTP, serial |
162
+ | **NeoJev (this model), MLX** | Qwen3.8-27B, 56 layers | 13.4 GB (4-bit) | **0.853** | 0.52 / 9.7 s | Apple M5 Pro, local HTTP, serial |
163
  | Jev 1.13.0 (TypeSafe AI, closed) | closed | n/a | 0.866 | 0.65 / 0.72 s | production API over the network |
164
  | decider-2b (Mapika) | Qwen3.5-2B | 3.8 GB | 0.710 | 0.26 / 0.28 s | JevBench server, network |
165
  | openJev Verdict 1.4 | ModernBERT-base 151M | about 0.6 GB (est.) | 0.576 | 0.31 / 0.92 s | CPU, 4 threads |
 
175
 
176
  How to read it:
177
 
178
+ - **Accuracy.** On the public items NeoJev (0.861 CUDA, 0.853 MLX) is close to Jev 1.13.0 (0.866) and to the other Qwen3.8-27B systems (0.862 to 0.870). It is above the small models in the table (0.58 to 0.85; decider-2b 0.71, JevK5 0.85). Among open systems, only JevOne and NInfer Qwen3.8-Flash-Next (both 0.896, on 35B-class MoE models) and thinking-mode systems score higher. Frontier APIs (GPT, DeepSeek, Gemini) score higher as well.
179
  - **Speed and memory.** Among the 27B-class systems, NeoJev's p50 is 0.13 s on an L20. SimpleJev and reflex-27b show 1.0 s and 1.9 s in JevBench's runs (with a network hop, on other hardware, so treat this as an order of magnitude only). NeoJev also needs about half the memory of a BF16 27B, because it runs 56 of 64 layers of the FP8 checkpoint. Its p95 is dominated by the long documents of the hard tier; the easy and standard tiers take 0.11 s.
180
+ - **Option order matters.** JevBench sends each choice question's options as a label-to-description map sorted by label, and NeoJev keeps the order it receives. When the options were instead put in each item's label order, NeoJev scored 0.866 (CUDA) and 0.870 (MLX); of the 119 choice items whose two orders differ, 8 flipped on MLX and 5 on CUDA. The table uses the JevBench path. Small or frozen decision models are generally order-sensitive, and JevBench documents this for other entrants too.
181
  - **Latency is not directly comparable across rows.** Many JevBench rows include a network round trip from JevBench's server, and every row ran on different hardware.
182
  - **This is not the official JevBench score.** The official score also needs the held-out and sealed items (which only JevBench runs; v1.4 adds 308 fresh sealed decisions), a judge tier, calibration and a cost figure.
183
  - **Where the memory figures come from.** They are the published repository size where one exists (decider-2b, JevK5); otherwise they are an estimate of parameters × precision, marked "est.".
decider.py CHANGED
@@ -29,6 +29,7 @@ How it works
29
  """
30
  import json
31
  import os
 
32
 
33
  import mlx.core as mx
34
  import mlx.nn as nn
@@ -98,7 +99,8 @@ class Decider:
98
  self.W = self.W.astype(mx.float32)
99
  self.norm_w = self.lm.model.norm.weight.astype(mx.float32)
100
  self.eps = self.lm.model.norm.eps
101
- self._schema_cache = {}
 
102
 
103
  def _enc(self, s):
104
  return self.tok.encode(s, add_special_tokens=False)
@@ -113,7 +115,11 @@ class Decider:
113
  return h[0] # pre-norm residual stream of the last layer
114
 
115
  def _prefix(self, key, text):
116
- if key not in self._schema_cache:
 
 
 
 
117
  cache = self.lm.make_cache()
118
  self._run(self._enc(text), cache)
119
  mx.eval([c.state for c in cache])
 
29
  """
30
  import json
31
  import os
32
+ from collections import OrderedDict
33
 
34
  import mlx.core as mx
35
  import mlx.nn as nn
 
99
  self.W = self.W.astype(mx.float32)
100
  self.norm_w = self.lm.model.norm.weight.astype(mx.float32)
101
  self.eps = self.lm.model.norm.eps
102
+ self._schema_cache = OrderedDict() # LRU of cached question prefixes
103
+ self.max_schemas = 64
104
 
105
  def _enc(self, s):
106
  return self.tok.encode(s, add_special_tokens=False)
 
115
  return h[0] # pre-norm residual stream of the last layer
116
 
117
  def _prefix(self, key, text):
118
+ if key in self._schema_cache:
119
+ self._schema_cache.move_to_end(key)
120
+ else:
121
+ while len(self._schema_cache) >= self.max_schemas:
122
+ self._schema_cache.popitem(last=False)
123
  cache = self.lm.make_cache()
124
  self._run(self._enc(text), cache)
125
  mx.eval([c.state for c in cache])
jevbench_via_server.py ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Run JevBench's own TypeSafe adapter (unmodified) against a NeoJev server on the 231 public items.
2
+ usage: jevbench_via_server.py <endpoint> <out.jsonl>"""
3
+ import json
4
+ import os
5
+ import sys
6
+
7
+ ROOT = os.path.join(os.path.dirname(__file__), "..")
8
+ JB = os.environ.get("JB_DIR", os.path.join(ROOT, "reviews", "jevbench"))
9
+ sys.path.insert(0, JB)
10
+ from jevbench.adapters.typesafe import TypeSafeAdapter # noqa: E402
11
+ from jevbench.scoring import score_task # noqa: E402
12
+
13
+
14
+ class T:
15
+ def __init__(self, r):
16
+ self.state, self.question, self.labels, self.expected = r["state"], r["question"], r["labels"], r["expected"]
17
+
18
+
19
+ ep, out = sys.argv[1], sys.argv[2]
20
+ ad = TypeSafeAdapter(endpoint=ep, model="neojev", key_env=None)
21
+ rows = []
22
+ for tier in ["easy", "original", "hard"]:
23
+ rows += [json.loads(l) for l in open(os.path.join(JB, "datasets", "public", f"{tier}.jsonl"))]
24
+ ok = 0
25
+ with open(out, "w") as f:
26
+ for r in rows:
27
+ res = ad.run(T(r))
28
+ s = score_task(res.probs, T(r)) if res.ok else {"correct": False, "valid": False}
29
+ ok += res.ok
30
+ f.write(json.dumps({"id": r["id"], "tier": r["id"].split("-")[0], "ok": res.ok, "error": res.error,
31
+ "latency_s": round(res.latency_s, 4), "correct": s["correct"], "probs": res.probs}) + "\n")
32
+ print("requests ok", ok, "/", len(rows))
server.py ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """TypeSafe-compatible HTTP server for NeoJev: POST /v1/systemone, GET /health (stdlib only).
2
+
3
+ python server.py --backend mlx --model dowands/NeoJev-Qwen3.8-27B-L56-MLX --port 8000
4
+ python server.py --backend cuda --model dowands/NeoJev-Qwen3.8-27B-L56 --port 8000
5
+
6
+ Request: {"state": str | object, "questions": {name: {"type": "choice"|"score"|"noul", "instructions": str,
7
+ "criteria": ...}}}
8
+ choice criteria: {label: description | null} (or a list of labels)
9
+ score criteria: [level description, ...] (labels are "0", "1", ...)
10
+ noul criteria: {"true": ..., "false": ...} (optional)
11
+ Response: {"model": ..., "answers": {name: {"type": ..., "choice"|"score"|"noul": ..., "probabilities": {...},
12
+ "confidence": ...}}, "usage": {"input_tokens": n}}
13
+
14
+ --layout state_first (default) puts the state before the questions and caches nothing; it is the layout used for
15
+ the published JevBench numbers. --layout schema_first caches the question prefix per schema (LRU), which is faster
16
+ when the same questions are asked about many states.
17
+ """
18
+ import argparse
19
+ import json
20
+ from http.server import BaseHTTPRequestHandler, HTTPServer
21
+
22
+
23
+ def translate(questions):
24
+ """TypeSafe question dict -> (NeoJev question dict, per-question label list)."""
25
+ out, labels = {}, {}
26
+ for name, q in questions.items():
27
+ t, crit, ins = q["type"], q.get("criteria"), q.get("instructions", "")
28
+ if t == "choice":
29
+ if isinstance(crit, dict):
30
+ labs = list(crit)
31
+ opts = [f"{l}: {crit[l]}" if crit[l] else l for l in labs]
32
+ else:
33
+ labs = [str(x) for x in (crit or [])]
34
+ opts = labs
35
+ out[name] = {"type": "choice", "options": opts, "prompt": ins}
36
+ elif t == "noul":
37
+ c = crit or {}
38
+ labs = ["no", "yes"]
39
+ out[name] = {"type": "choice", "prompt": ins,
40
+ "options": [f"no: {c['false']}" if c.get("false") else "no",
41
+ f"yes: {c['true']}" if c.get("true") else "yes"]}
42
+ elif t == "score":
43
+ levels = [str(x) for x in (crit or [])]
44
+ labs = [str(i) for i in range(len(levels))]
45
+ out[name] = {"type": "score", "levels": levels, "prompt": ins}
46
+ else:
47
+ raise ValueError(f"unknown question type {t!r}")
48
+ labels[name] = (t, labs)
49
+ return out, labels
50
+
51
+
52
+ def main():
53
+ ap = argparse.ArgumentParser()
54
+ ap.add_argument("--backend", choices=["mlx", "cuda"], required=True)
55
+ ap.add_argument("--model", required=True)
56
+ ap.add_argument("--port", type=int, default=8000)
57
+ ap.add_argument("--host", default="127.0.0.1")
58
+ ap.add_argument("--layout", choices=["state_first", "schema_first"], default="state_first")
59
+ ap.add_argument("--token-budget", type=int, default=10 ** 6, help="cap on tokens per call (default: no truncation)")
60
+ a = ap.parse_args()
61
+ if a.backend == "mlx":
62
+ from decider import Decider
63
+ d = Decider(a.model, token_budget=a.token_budget)
64
+ else:
65
+ from decider_cuda import Decider
66
+ d = Decider(a.model, token_budget=a.token_budget)
67
+ model_name = "neojev-qwen3.8-27b-l56"
68
+
69
+ class H(BaseHTTPRequestHandler):
70
+ def _send(self, code, obj):
71
+ b = json.dumps(obj).encode()
72
+ self.send_response(code)
73
+ self.send_header("Content-Type", "application/json")
74
+ self.send_header("Content-Length", str(len(b)))
75
+ self.end_headers()
76
+ self.wfile.write(b)
77
+
78
+ def log_message(self, *args):
79
+ pass
80
+
81
+ def do_GET(self):
82
+ self._send(200, {"ok": True, "model": model_name}) if self.path == "/health" else self._send(404, {"error": "not found"})
83
+
84
+ def do_POST(self):
85
+ if self.path != "/v1/systemone":
86
+ return self._send(404, {"error": "not found"})
87
+ try:
88
+ req = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0))))
89
+ state = req["state"] if isinstance(req["state"], str) else json.dumps(req["state"], ensure_ascii=False, indent=1)
90
+ qs, labels = translate(req["questions"])
91
+ res = d.ask(state, qs, cache_schema=(a.layout == "schema_first"))
92
+ except Exception as e: # malformed request
93
+ return self._send(400, {"error": f"{type(e).__name__}: {e}"})
94
+ answers = {}
95
+ for name, (t, labs) in labels.items():
96
+ p = res[name]["probs_list"]
97
+ probs = dict(zip(labs, p))
98
+ j = max(range(len(p)), key=lambda k: p[k])
99
+ ans = {"type": t, "probabilities": probs, "confidence": p[j]}
100
+ if t == "choice":
101
+ ans["choice"] = labs[j]
102
+ elif t == "noul":
103
+ ans["noul"] = probs["yes"]
104
+ else:
105
+ ans["score"] = sum(k * v for k, v in enumerate(p))
106
+ answers[name] = ans
107
+ self._send(200, {"model": model_name, "answers": answers, "usage": {"input_tokens": res["_tokens"]}})
108
+
109
+ print(f"NeoJev ({a.backend}, {a.layout}) on http://{a.host}:{a.port}/v1/systemone", flush=True)
110
+ # single-threaded on purpose: one GPU serves one request at a time, and MLX streams are bound to the main thread
111
+ HTTPServer((a.host, a.port), H).serve_forever()
112
+
113
+
114
+ if __name__ == "__main__":
115
+ main()