muthuishere commited on
Commit
0098fc0
·
verified ·
1 Parent(s): 883bd72

Card: v0.5.1 server latency (ORT 1.29); model unchanged

Browse files
Files changed (1) hide show
  1. README.md +7 -3
README.md CHANGED
@@ -40,7 +40,8 @@ curl -s http://127.0.0.1:21118/v1/systemone -d '{
40
 
41
  Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
42
  expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
43
- Prefer a plain binary? macOS, Linux and Windows builds are on the release page.
 
44
 
45
  ## Or in Python
46
 
@@ -80,10 +81,13 @@ The model folder on the Go server, on CPU. The gate sets share **no state** with
80
  | Rule-checking basics (300) | 94.7% | 94.3% | 3.7% |
81
 
82
  **Known miss:** confidently wrong is 3.0–3.7% on the basics gates, against our 2% target. It was accepted for this
83
- release; the fix is planned for v0.5.1. The rest of the gate (held-out test sets, logs gate, 13 fundamentals) is
84
  still being scored and will be added here.
85
 
86
- **Benchmarks:** coming in a follow-up.
 
 
 
87
 
88
  ## What v0.5.0 was trained on
89
 
 
40
 
41
  Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
42
  expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
43
+ Prefer a plain binary? macOS (Apple Silicon), Linux and Windows builds of the v0.5.1 server (ONNX Runtime 1.29)
44
+ are on the [release page](https://github.com/muthuishere/openjevx/releases/latest).
45
 
46
  ## Or in Python
47
 
 
81
  | Rule-checking basics (300) | 94.7% | 94.3% | 3.7% |
82
 
83
  **Known miss:** confidently wrong is 3.0–3.7% on the basics gates, against our 2% target. It was accepted for this
84
+ release; the fix is planned for the next model release. The rest of the gate (held-out test sets, logs gate, 13 fundamentals) is
85
  still being scored and will be added here.
86
 
87
+ **Latency** (v0.5.1 server, ONNX Runtime 1.29, CPU only, Apple M5 Pro, p50 end to end): 22 ms for a short
88
+ request (36 tokens), 85 ms for a typical one (164 tokens), 540 ms near the 1024-token limit. Cold start is about
89
+ 1 s once per process. ONNX Runtime 1.29 or newer is needed for this speed: it keeps the 8-bit weights 8-bit
90
+ (`MatMulNBits`); in Python, `pip install -U onnxruntime`.
91
 
92
  ## What v0.5.0 was trained on
93