diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..b9ec107afc8401a7cdbc9f36de05937b039706c9 100644 --- a/.gitattributes +++ b/.gitattributes @@ -33,3 +33,20 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +demos/out/browser_hf/hf.gif filter=lfs diff=lfs merge=lfs -text +demos/out/browser_hn2/000000.jpg filter=lfs diff=lfs merge=lfs -text +demos/out/browser_hn2/hn2.gif filter=lfs diff=lfs merge=lfs -text +demos/out/browser_wikipedia/000000.jpg filter=lfs diff=lfs merge=lfs -text +demos/out/browser_wikipedia/013484.jpg filter=lfs diff=lfs merge=lfs -text +demos/out/browser_wikipedia/wikipedia.gif filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/google-flights-colocated.gif filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/google-flights-colocated.mp4 filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/inbox-classification.gif filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/inbox-classification.mp4 filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/loop-qa-grid.gif filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/loop-qa-grid.mp4 filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/shopping-colocated.gif filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/shopping-colocated.mp4 filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/workspace-assembly.gif filter=lfs diff=lfs merge=lfs -text +demos/out/recorded_gpu/workspace-assembly.mp4 filter=lfs diff=lfs merge=lfs -text +demos/out/snake/snake.gif filter=lfs diff=lfs merge=lfs -text diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000000000000000000000000000000000000..aaec8bb50b2759674b6c69e780dd04637cd25273 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,13 @@ +name: ci +on: [push, pull_request] +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: { python-version: "3.12" } + - run: pip install -e ".[dev]" + - run: ruff check openjev_server tests + - run: ruff format --check openjev_server tests + - run: pytest -q tests # the compat tests skip without a local tokenizer; the unit and server tests always run diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..d7ee2502f2d29c86cc31c667ce27d427685cea1d --- /dev/null +++ b/.gitignore @@ -0,0 +1,10 @@ +.venv/ +__pycache__/ +*.pyc +*.egg-info/ +build/ +dist/ +.pytest_cache/ +.ruff_cache/ +demos/out/**/frames*/ +demos/out/**/png/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000000000000000000000000000000000000..bdeaae39d5ccce0e7b194542a82937baa69d3bc8 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,15 @@ +# Changelog + +## 0.2.0 (2026-09-22) +- Modules by job: `profile.py` (calibration constants), `prompt.py` (state text and prompt layouts), `readout.py` (scores to answers), `backends/base.py` (the backend contract), `backends/make_backend` (the only place that knows backend names). Answers unchanged: the compat tests against the released helper still hold. +- Server: `/v1` routes on one router with bearer auth as a router dependency; operations endpoints on their own router. +- vLLM backend: `VllmOptions` replaces ten constructor arguments; one response parser for the chat and raw-completion paths; tests against a fake OpenAI-compatible server. +- `/v1/version.readout_sha256` now hashes `prompt.py` + `readout.py` (the answer-determining code). +- Tests fetch the public OpenJev tokenizer and the released helper from the Hub when no local path is set, so CI needs nothing local. +- Landing page: links open in the top window (the Space iframe could not frame huggingface.co). + +## 0.1.0 (2026-09-22) +- First release: the OpenJev decision API as an installable server. vLLM and MLX backends, profiles, calibration fitter, bench, metrics, bearer auth. +- Answers identical to the helper shipped with the OpenJev model (`helper/shim.py`) under the `openjev` profile. +- Any-model support: label tokenization detection, exact-readout detection, raw prompting without a chat template, assistant prefix for reasoning models, startup probe that refuses to serve noise. +- MLX: the shared part of a request is read once and reused across its questions. diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000000000000000000000000000000000000..2bdba5ecc4e4a286083ba1e569e14e912f936879 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,9 @@ +FROM python:3.12-slim +WORKDIR /app +COPY pyproject.toml README.md LICENSE NOTICE ./ +COPY openjev_server ./openjev_server +COPY profiles ./profiles +RUN pip install --no-cache-dir . +EXPOSE 3000 +ENV OPENJEV_HOST=0.0.0.0 OPENJEV_PORT=3000 OPENJEV_BACKEND=vllm OPENJEV_VLLM_URL=http://vllm:8000/v1 OPENJEV_MODEL=openjev/openjev-FP8 OPENJEV_PROFILE=openjev +CMD ["openjev", "serve"] diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..d645695673349e3947e8e5ae42332d0ac3164cd7 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000000000000000000000000000000000000..ead6a114663a6f78290c06399f576b7940dba013 --- /dev/null +++ b/NOTICE @@ -0,0 +1,6 @@ +openjev-server +Copyright 2026 the OpenJev contributors + +Licensed under the Apache License, Version 2.0. The readout prompt layouts, calibration and confidence formulas +descend from the helper published with the OpenJev model (huggingface.co/openjev/openjev, helper/shim.py, Apache 2.0). +The confidence formulas follow TypeSafe's published adapter (system-one-adapter-python, confidence_metrics.py). diff --git a/README.md b/README.md index 74d9ad27153caea93d44f490065fd4ae28643def..967c3230449e1d78adac4f99857981c2d44fe74c 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,242 @@ --- -title: Openjev Server -emoji: 👁 +title: openjev-server +emoji: ⚖️ colorFrom: green -colorTo: red +colorTo: gray sdk: static -pinned: false +pinned: true +license: apache-2.0 +short_description: A decision API over any open model, one pass per question --- -Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference +# openjev-server + +One-pass decisions from any open language model, served as an API. + +[![ci](https://img.shields.io/github/actions/workflow/status/abhishekgahlot2/openjev-server/ci.yml?label=CI)](https://github.com/abhishekgahlot2/openjev-server/actions/workflows/ci.yml) +[![weights](https://img.shields.io/badge/weights-16bit%20%7C%20FP8%20%7C%20MLX-blue)](https://huggingface.co/openjev) +[![space](https://img.shields.io/badge/Hugging%20Face-Space-yellow)](https://huggingface.co/spaces/openjev/openjev-server) +[![licence](https://img.shields.io/badge/licence-Apache%202.0-green)](LICENSE) + +Send a situation and your questions. Get back one choice, a yes/no probability or a score per question, with a probability for every option. The answer is read off the model's first output position: one forward pass per question, nothing generated, no chain of thought to wait for. + +It is the server behind [OpenJev](https://huggingface.co/openjev), an open 27B decision model for browser and desktop agents. Point it at OpenJev, or at your own model: anything served by vLLM, or an MLX model on a Mac. + +## What you get + +- **Three question types in one request.** `choice`, `noul` (yes/no) and `score`. Questions share the state but each is answered on its own. +- **Probabilities, not just labels.** Every option comes back with its probability and a confidence, so your code can branch on how sure the model is. +- **Any model.** A startup probe finds the label tokens, checks for exact score readout and vision, and refuses to serve a model that would return noise. +- **Honest probabilities for your model.** `openjev calibrate` fits the two calibration constants on your own development rows. +- **Fast where it matters.** Exact candidate scores by token id on vLLM. On a Mac the shared part of a request is read once and reused for every question. +- **Drop-in for clients of the hosted API.** Request and response shapes follow Jev's `/v1/systemone`, so a client written for it can point at your server. +- **Built to run.** Health and readiness, Prometheus metrics, bearer auth, a bench command, Docker Compose, CI. + +## Why OpenJev + +Most small "decision models" are fast because they are narrow. **The point is that it has to be fast and it has to be a general classifier at the same time**: any situation, any set of labels, given at request time, one pass. + +**Same 10,000 questions, same options, same order for every model** (34 public datasets, gold answers from the datasets): + +| model | right of 10,000 | accuracy | +|---|---|---| +| Jev, hosted | 8,540 | **85.4%** | +| **OpenJev**, open weights, one GPU or a Mac | **8,403** | **84.0%** | +| Qwen3.8-27B base, untuned | 8,036 | 80.4% | +| a fine-tuned 9B decision model | 7,574 | 75.7% | + +**137 questions apart out of 10,000** from the hosted model, and **829 ahead of the 9B** that is sold as the fast alternative. On 2,000 desktop screenshot next-action steps OpenJev picks the right element **88.0%** of the time. A text decision takes about **125 ms** on one H100 and about **150 ms** on a Mac with the 4-bit build. + +The comparison is not a controlled one: the hosted model's training data is unknown. The full method, the caveats and every number are on the [model card](https://huggingface.co/openjev/openjev#results). + +## Demos + +Real runs, every decision logged. The scripts ran on a MacBook Pro with the 4-bit build; five recordings were made on an H100 with the FP8 build. Scripts, outputs and provenance are in [`demos/`](demos/README.md). + +| demo | what happens | result | +|---|---|---| +| [Chess](demos/README.md#chess) | legal moves are the options, one call per ply | checkmated a random mover in 16 moves, 1.2 s a move | +| [Poker](demos/README.md#poker) | heads-up no-limit, duplicate deals against an equity bot | 118 chips up on a random bot over 50 hands; loses slowly to an exact-equity bot; its strength rating tracks true equity (Spearman 0.67 to 0.75) | +| [Returns policy](demos/README.md#returns-policy-enterprise) | nine rules, 200 generated cases, exact ground truth | **93.5%** right, **88.0%** when it must count calendar days; majority baseline 43.5% | +| [Agent guardrail](demos/README.md#agent-guardrail-enterprise) | seven tool rules, 200 generated calls, exact ground truth | **200 of 200** right; 0 denied calls executed | +| [Row filters](demos/README.md#natural-language-row-filters-enterprise) | eight English filters per ticket row in one call | **92.1%** over 480 answers, 5 of 8 filters perfect | +| [Prompt injection](demos/README.md#prompt-injection-public-dataset) | 662 public messages, one yes/no each | **95.0%** accuracy, ROC-AUC 0.995, 97.7% recall at 2% false alarms | +| [Intent classification](demos/README.md#intent-classification-public-dataset) | Banking77, all 77 intents as options, two passes | **78.0%** zero shot, top-3 92.3%, 300 messages | +| [Generative UI](demos/README.md#generative-ui) | one sentence, ten layout decisions in one call, shadcn-style render | six distinct, sensible dashboards | +| [Browser agent](demos/README.md#browser-agent) | jev-ultrafast on the local server, page text and controls only | on the GPU build: Google Flights and a shopping task completed, recorded; on the laptop: Hacker News and Wikipedia done, two heavy pages not | +| [Inbox triage](demos/README.md#recorded-on-the-gpu-build) | 24 messages get a team, an urgency score and a review flag | 72 typed fields in 6.41 s on one H100, recorded | +| [Snake](demos/README.md#snake) | live play, one call per move | survived all three 200-move games, scores 14, 2, 17; a random player scores 0.3 | + +

browser agent on Google Flights chess

+

inbox triage dashboard laid out by the model

+ +## Models + +| build | runs on | link | +|---|---|---| +| OpenJev 16 bit | two 80 GB GPUs, or one with offload | [openjev/openjev](https://huggingface.co/openjev/openjev) | +| OpenJev FP8 | one H100 or any 80 GB GPU | [openjev/openjev-FP8](https://huggingface.co/openjev/openjev-FP8) | +| OpenJev MLX 8 bit | Apple silicon, 64 GB | [openjev/openjev-MLX](https://huggingface.co/openjev/openjev-MLX) | +| OpenJev MLX 4 bit | Apple silicon, 32 GB | [openjev/openjev-MLX-4bit](https://huggingface.co/openjev/openjev-MLX-4bit) | + +Accuracy, calibration and latency numbers are on the model cards and the [org page](https://huggingface.co/openjev). + +## Install + +```bash +pip install "openjev-server @ git+https://github.com/abhishekgahlot2/openjev-server" +``` + +## Quick start + +**One GPU, vLLM** + +```bash +vllm serve openjev/openjev-FP8 --served-model-name qwen --port 8000 --enable-prefix-caching \ + --max-model-len 16384 --gpu-memory-utilization 0.90 --limit-mm-per-prompt '{"image":1}' \ + --trust-remote-code --max-num-seqs 64 --max-logprobs 64 --gdn-prefill-backend triton --quantization fp8 + +openjev serve --backend vllm --model openjev/openjev-FP8 --vllm-url http://localhost:8000/v1 --profile openjev --port 3000 +``` + +**Mac, MLX** + +```bash +pip install "openjev-server[mlx] @ git+https://github.com/abhishekgahlot2/openjev-server" +hf download openjev/openjev-MLX-4bit --local-dir openjev-MLX-4bit +openjev serve --backend mlx --model openjev-MLX-4bit --profile openjev --port 3000 +``` + +The MLX build takes text, JSON and DOM states. The shared part of a request is read once and reused for every question of that request. + +## Ask + +```bash +curl -s http://localhost:3000/v1/systemone -H 'Content-Type: application/json' -d '{ + "state": "Customer message: I was charged twice for my order last week and nobody has replied.", + "questions": { + "route": {"type": "choice", "instructions": "Which team should handle this?", + "criteria": {"billing": null, "shipping": null, "technical": null}}, + "angry": {"type": "noul", "instructions": "Is the customer angry?"}, + "urgency": {"type": "score", "instructions": "How urgent is this?", + "criteria": ["can wait", "this week", "today", "right now"]} + } +}' +``` + +```json +{ + "answers": { + "route": {"type": "choice", "choice": "billing", + "probabilities": {"billing": 0.9996, "shipping": 0.0002, "technical": 0.0002}, "confidence": 0.9995}, + "angry": {"type": "noul", "noul": 0.56}, + "urgency": {"type": "score", "score": 2.04, + "legend": {"0": "can wait", "1": "this week", "2": "today", "3": "right now"}, + "probabilities": {"0": 0.02, "1": 0.13, "2": 0.64, "3": 0.21}, "confidence": 0.71} + }, + "usage": {"input_tokens": 239, "output_tokens": 0} +} +``` + +The message is about a double charge, and the routing probability says so with almost nothing left for the other teams. The urgency answer is split between "today" and "right now": that spread is the signal you get from probabilities that a single label would hide. + +**Python** + +```python +import httpx + +r = httpx.post("http://localhost:3000/v1/systemone", json={ + "state": "Customer message: I was charged twice for my order last week and nobody has replied.", + "questions": { + "route": {"type": "choice", "instructions": "Which team should handle this?", + "criteria": {"billing": None, "shipping": None, "technical": None}}, + "angry": {"type": "noul", "instructions": "Is the customer angry?"}, + }, +}, timeout=60) +answers = r.json()["answers"] +print(answers["route"]["choice"], answers["route"]["probabilities"], answers["angry"]["noul"]) +``` + +| type | you send | you get | +|---|---|---| +| `choice` | `criteria`: map of option to description or `null` | the chosen option, a probability per option, confidence | +| `noul` | optional `criteria.true` / `criteria.false` descriptions | probability that the statement is true | +| `score` | `criteria`: ordered list of levels | expected level, probability per level, confidence | + +- `instructions` may be a string or an object, for example a goal plus rules. +- Up to 52 options answer in one pass. More are handled in two passes. +- A screenshot goes in `state.screenshot` as a data URL, on models with a vision tower. +- Request and response shapes follow the hosted Jev API. A client written for it can be pointed at your own server. + +## Any model + +At start the server probes the model and prints what it found: + +```json +{"model_probe": {"served_model": "qwen", "exact_readout": true, "chat_template": true, + "letter_prefix": "", "vision": true, "letter_mass_at_readout_position": 0.9994}} +``` + +- **Labels.** Options are labelled with letters. The server uses whichever single-token form the tokenizer has, `A` or ` A`. +- **Exact scores.** With vLLM it asks for the scores of exactly the candidate tokens and reads them back by id. Servers without that extension fall back to matching the top-K by text. +- **Templates.** Models without a chat template are prompted raw. Thinking is turned off where the template supports it. For models that always reason first, `--assistant-prefix` puts an empty think block before the readout position. +- **Refuses noise.** If the model puts almost none of its probability on the option letters at the readout position, the server refuses to start and says why. `--force` overrides. +- **Vision.** Detected from the model config. Screenshot requests to a text-only model return a clear error. + +## Calibrate a new model + +Probabilities are only useful if they are honest. Start a new model in the `uncalibrated` profile, then fit its two constants on your own development rows, never on a test set: + +```bash +openjev serve --backend vllm --model your/model --profile uncalibrated --port 3000 +openjev calibrate --dev dev.jsonl --endpoint http://localhost:3000 --out profiles/your-model.json +openjev serve --backend vllm --model your/model --profile profiles/your-model.json +``` + +`dev.jsonl` rows look like `{"state": ..., "questions": {"q": {...}}, "gold": "