Card: v0.5.1 server latency (ORT 1.29); model unchanged
Browse files
README.md
CHANGED
|
@@ -40,7 +40,8 @@ curl -s http://127.0.0.1:21118/v1/systemone -d '{
|
|
| 40 |
|
| 41 |
Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
|
| 42 |
expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
|
| 43 |
-
Prefer a plain binary? macOS, Linux and Windows builds
|
|
|
|
| 44 |
|
| 45 |
## Or in Python
|
| 46 |
|
|
@@ -80,10 +81,13 @@ The model folder on the Go server, on CPU. The gate sets share **no state** with
|
|
| 80 |
| Rule-checking basics (300) | 94.7% | 94.3% | 3.7% |
|
| 81 |
|
| 82 |
**Known miss:** confidently wrong is 3.0–3.7% on the basics gates, against our 2% target. It was accepted for this
|
| 83 |
-
release; the fix is planned for
|
| 84 |
still being scored and will be added here.
|
| 85 |
|
| 86 |
-
**
|
|
|
|
|
|
|
|
|
|
| 87 |
|
| 88 |
## What v0.5.0 was trained on
|
| 89 |
|
|
|
|
| 40 |
|
| 41 |
Each answer has `probabilities` and a `confidence`; `noul` gives P(true), `choice` the picked key, `score` the
|
| 42 |
expected level. The container also serves a live dashboard on `/` and runnable examples on `/recipes`.
|
| 43 |
+
Prefer a plain binary? macOS (Apple Silicon), Linux and Windows builds of the v0.5.1 server (ONNX Runtime 1.29)
|
| 44 |
+
are on the [release page](https://github.com/muthuishere/openjevx/releases/latest).
|
| 45 |
|
| 46 |
## Or in Python
|
| 47 |
|
|
|
|
| 81 |
| Rule-checking basics (300) | 94.7% | 94.3% | 3.7% |
|
| 82 |
|
| 83 |
**Known miss:** confidently wrong is 3.0–3.7% on the basics gates, against our 2% target. It was accepted for this
|
| 84 |
+
release; the fix is planned for the next model release. The rest of the gate (held-out test sets, logs gate, 13 fundamentals) is
|
| 85 |
still being scored and will be added here.
|
| 86 |
|
| 87 |
+
**Latency** (v0.5.1 server, ONNX Runtime 1.29, CPU only, Apple M5 Pro, p50 end to end): 22 ms for a short
|
| 88 |
+
request (36 tokens), 85 ms for a typical one (164 tokens), 540 ms near the 1024-token limit. Cold start is about
|
| 89 |
+
1 s once per process. ONNX Runtime 1.29 or newer is needed for this speed: it keeps the 8-bit weights 8-bit
|
| 90 |
+
(`MatMulNBits`); in Python, `pip install -U onnxruntime`.
|
| 91 |
|
| 92 |
## What v0.5.0 was trained on
|
| 93 |
|