lev: step 18750 of preset 4b-instruct

README.md

@@ -26,13 +26,9 @@ lev answers typed questions about a piece of context in a single forward pass. Y
 
 <div align="center" style="line-height: 1;"><img src="https://img.shields.io/badge/license-Apache--2.0-2a78d6?style=flat-square" alt="Apache-2.0" style="display: inline-block; vertical-align: middle; margin: 2px;"> <img src="https://img.shields.io/badge/base-Qwen3.5--4B-2a78d6?style=flat-square" alt="Qwen3.5-4B" style="display: inline-block; vertical-align: middle; margin: 2px;"> <img src="https://img.shields.io/badge/output%20tokens-0-2a78d6?style=flat-square" alt="Zero output tokens" style="display: inline-block; vertical-align: middle; margin: 2px;"> <img src="https://img.shields.io/badge/API-%2Fv1%2Fsystemone-2a78d6?style=flat-square" alt="/v1/systemone compatible" style="display: inline-block; vertical-align: middle; margin: 2px;"> <a href="https://github.com/Abhinavexists/lev"><img src="https://img.shields.io/badge/code-GitHub-14181f?style=flat-square&logo=github" alt="GitHub" style="display: inline-block; vertical-align: middle; margin: 2px;"></a> <a href="https://interfaze.ai/blog/jev-now-open-source-lev"><img src="https://img.shields.io/badge/Blog-lev-yellow" alt="Blog" style="display: inline-block; vertical-align: middle; margin: 2px;"></a> <a href="https://interfaze.ai"><img src="https://img.shields.io/badge/Built_by-Interfaze--ai-4C1" alt="interfaze" style="display: inline-block; vertical-align: middle; margin: 2px;"></a></div>
 
-<p align="center">
-  Read full blog here: <a href="https://interfaze.ai/blog/jev-now-open-source-lev">https://interfaze.ai/blog/jev-now-open-source-lev</a>
-</p>
-
-<h2 align="center">68.9% on all 13 S1Bench subsets. 4B parameters. Zero output tokens.</h2>
+<h2 align="center">74.3% on all 13 S1Bench subsets. 4B parameters. Zero output tokens.</h2>
 
-<p align="center"><strong>Qwen3.5-4B + LoRA on one H100.</strong> On the six subsets the public S1Bench board completed, level with reflex-4b and behind only Jev and three open models of 26B–35B.</p>
+<p align="center"><strong>Qwen3.5-4B + LoRA on one H100.</strong> On the six subsets the public S1Bench board completed, behind only Jev and two open models of 26B and up, ahead of reflex-4b and a 35B model.</p>
 
 <p align="center"><a href="#quickstart"><strong>Quickstart</strong></a> · <a href="#self-hosting-a-jev-compatible-http-server">Self-hosting</a> · <a href="#benchmarks">Benchmarks</a> · <a href="#speed">Speed</a> · <a href="#why-it-works">Why it works</a> · <a href="#training">Training</a> · <a href="#boundaries-worth-understanding">Boundaries</a> · <a href="https://github.com/Abhinavexists/lev">GitHub</a></p>
 
@@ -52,7 +48,7 @@ It is built for the high-volume judgement calls inside a product: routing, moder
 pip install "lev[serve] @ git+https://github.com/Abhinavexists/lev#subdirectory=packages/lev"
 ```
 
-This needs Python 3.12 or newer and, for real-time use, a CUDA GPU. The `serve` extra installs torch, transformers, peft, and the HTTP server. The first load downloads the base model (Qwen/Qwen3.5-4B, about 8 GB) and this adapter (about 200 MB).
+This needs Python 3.12 or newer and, for real-time use, a CUDA GPU. The `serve` extra installs torch, transformers, peft, and the HTTP server. The first load downloads the base model (Qwen/Qwen3.5-4B, 9.3 GB of weights) and this release (about 205 MB: the 170 MB adapter, the 15 MB Mode B head and the tokenizer).
 
 ## Quickstart
 
@@ -92,7 +88,7 @@ print(result.usage.output_tokens)  # 0
 
 These are real outputs from this checkpoint. The answers are objects, and `result.model_dump()` gives the same JSON the HTTP server returns.
 
-All the questions share one forward pass, so asking three questions costs about the same as asking one. `lev.load` reads `lev_release.json` from this repository to find the base model and the prompt format the adapter was trained with. It also applies the shipped calibration and loads the matching head, so there is nothing to configure.
+All the questions share one forward pass, so asking three questions costs about the same as asking one. `lev.load` reads `lev_release.json` from this repository to find the base model, the prompt format the adapter was trained with, and the strength to apply the adapter at. It also applies the shipped calibration and loads the matching head, so there is nothing to configure.
 
 Because the probabilities are calibrated, you can gate on them. For example, act automatically above 0.9 and send anything lower to a person.
 
@@ -129,46 +125,7 @@ response = client.system_one(
 print(response.answers["team"].choice, response.answers["bug"].noul)  # technical 0.92
 ```
 
-`GET /health` reports the loaded checkpoint, whether calibration is active, and the routing settings. The server batches every question in a request into one forward pass, accepts concurrent requests, and returns 422 with the reason for a malformed question.
-
-## Hosted classification
-
-Text classification in Interfaze runs on a similar system to lev: the model reads the answer from the set of labels you define. The difference is that Interfaze is still token based, so it's a hybrid.
-
-| Item | lev | Interfaze |
-| :-- | :-- | :-- |
-| Output | Probabilities, zero output tokens | Tokens, returned as structured output |
-| Questions | Typed only: yes/no, choice, score | Any JSON schema, labels included |
-| In the same request | Classification only | OCR, web search, transcription, extraction, and more |
-| Where it runs | Your GPU | Interfaze API |
-
-```bash
-  pip install interfaze
-```
-
-```python
-  import os
-  from typing import Literal
-  from pydantic import BaseModel, Field
-  from interfaze import Interfaze
-  
-  interfaze = Interfaze(api_key=os.environ["INTERFAZE_API_KEY"])
-  
-  class Ticket(BaseModel):
-      intent: Literal["refund", "cancel", "track", "other"]
-      urgent: bool = Field(..., description="Does this need a human within the hour?")
-  
-  response = interfaze.chat.completions.parse(
-      messages=[{"role": "user", "content": "Hi, I was charged twice for my order #4471 and I want a refund."}],
-      response_format=Ticket,
-  )
-  
-  print(response.choices[0].message.parsed)
-```
-
-Tokens cost a little speed, but they let one request classify a document while also reading, searching, and extracting from it. Define your labels as an enum in the schema, and the label comes back as a typed field.
-
-**Interfaze docs → [interfaze.ai/docs](https://interfaze.ai/docs)**
+`GET /health` reports the loaded checkpoint, whether calibration is active, the routing settings, and requests in flight against `max_pending`. The server batches every question in a request into one forward pass, and concurrent requests into shared forwards, so clients need not batch for throughput. Past `--max-pending` requests in flight (default 64) it answers 503 at once rather than queueing; a request queued past 30 s gets 504, and a malformed question gets 422 with the reason.
 
 ## Benchmarks
 
@@ -178,27 +135,27 @@ Tokens cost a little speed, but they let one request classify a document while a
 
 | subset | task | **lev** | Jev | always the most common label |
 | --- | --- | --: | --: | --: |
-| vitaminc-dev | claim verification | 0.668 | **0.801** | 0.503 |
-| massive-en-US | intent routing, 18 scenarios | 0.857 | **0.874** | 0.163 |
-| massive-de-DE | intent routing, German | 0.823 | **0.871** | 0.163 |
-| boolq | yes/no reading comprehension | 0.827 | **0.893** | 0.580 |
-| squad2 | answerability | 0.813 | **0.836** | 0.502 |
-| paws | adversarial paraphrase | 0.776 | **0.900** | 0.516 |
-| multinli | natural language inference | **0.890** | 0.836 | 0.361 |
-| civil_comments | toxicity | 0.760 | **0.803** | 0.893 |
-| aegis2 | safety moderation | 0.800 | **0.804** | 0.568 |
-| helpsteer2 | helpfulness, 5 levels | **0.386** | 0.341 | 0.422 |
-| summeval-relevance | summary relevance, 5 levels | 0.358 | 0.358 | 0.458 |
-| summeval-consistency | summary faithfulness, 5 levels | 0.271 | **0.812** | 0.840 |
-| pubmedqa | biomedical yes/no/maybe | 0.732 | **0.764** | 0.532 |
-| **macro** | | 0.689 | **0.761** | |
-
-lev and TypeSafe Jev ran through the same harness on all 3,880 items S1Bench scores, pinned by [Nimble](https://github.com/bespokelabsai/nimble)'s manifests. Our Jev run lands within 0.8 points of TypeSafe's published figure on every subset, so the harness is not the gap.
-
-- **Noise:** at these sizes a per-subset difference needs roughly 5–9 points to be real. lev's leads on multinli and helpsteer2 are inside that.
-- **Where Jev is clearly ahead:** the minimal-edit pairs (paws −12.4, vitaminc −13.3) and summeval-consistency (−54.1), where lev rates most fully faithful summaries one level low. Fine-tuning introduced that: the untuned backbone scores 0.826.
+| vitaminc-dev | claim verification | 0.726 | **0.801** | 0.503 |
+| massive-en-US | intent routing, 18 scenarios | 0.869 | **0.874** | 0.163 |
+| massive-de-DE | intent routing, German | 0.829 | **0.871** | 0.163 |
+| boolq | yes/no reading comprehension | 0.853 | **0.893** | 0.580 |
+| squad2 | answerability | 0.819 | **0.836** | 0.502 |
+| paws | adversarial paraphrase | 0.844 | **0.900** | 0.516 |
+| multinli | natural language inference | **0.849** | 0.836 | 0.361 |
+| civil_comments | toxicity | 0.783 | **0.803** | 0.893 |
+| aegis2 | safety moderation | **0.836** | 0.804 | 0.568 |
+| helpsteer2 | helpfulness, 5 levels | **0.361** | 0.341 | 0.422 |
+| summeval-relevance | summary relevance, 5 levels | **0.371** | 0.358 | 0.458 |
+| summeval-consistency | summary faithfulness, 5 levels | 0.806 | **0.812** | 0.840 |
+| pubmedqa | biomedical yes/no/maybe | 0.716 | **0.764** | 0.532 |
+| **macro** | | 0.743 | **0.761** | |
+
+lev and TypeSafe Jev ran through the same harness on all 3,880 items S1Bench scores, pinned by [Nimble](https://github.com/bespokelabsai/nimble)'s manifests. The Jev run lands within 0.8 points of TypeSafe's published figure on every subset, so the harness is not the gap. The adapter scale was chosen by leave-one-family-out over these subsets; scored only on families held out of that choice, it estimates 0.728 ([FINDINGS §19](https://github.com/Abhinavexists/lev/blob/main/docs/FINDINGS.md#19-the-adapter-scale-is-an-out-of-distribution-knob-and-it-is-worth-39-macro-points)).
+
+- **Noise:** at these sizes a per-subset difference needs roughly 5–9 points to be real. lev's leads on aegis2, helpsteer2, multinli and summeval-relevance are all inside that.
+- **Where Jev is clearly ahead:** the minimal-edit pairs (vitaminc −7.5, paws −5.6). The adapter runs at a quarter of its trained strength, which narrowed those gaps from −13.3 and −12.4 and closed summeval-consistency's from −54.1 to −0.6: at full strength it rated most fully faithful summaries one level low.
 - **Constant baselines:** on civil_comments (89% not toxic) and the three 5-level rating subsets, always answering the most common label beats both models.
-- **Calibration:** mean ECE 0.115 for lev, 0.091 for Jev; lev is better calibrated on 5 of 13.
+- **Calibration:** mean ECE 0.097 for lev, 0.091 for Jev; lev is better calibrated on 6 of 13. summeval-consistency is the outlier, under-confident at 0.303.
 
 <p align="center"><img src="assets/leaderboard.png" alt="S1Bench leaderboard over the six subsets every listed model completed" width="100%"></p>
 
@@ -216,7 +173,7 @@ A held-out split of the 29 training sources, with no row shared with training:
 | clinc_oos (151 intents) | 0.968 |
 | FEVER claim verification | 0.872\* |
 
-\* Measured on this checkpoint before the last serving update. That update routes choice sets of more than 68 options to label-token readout and re-selects the temperatures. The banking77 and clinc_oos rows come from after the update, which raised banking77 from 0.818. Smaller option sets are routed the same way as before.
+\* Measured on this checkpoint before the serving update that skips split label codes. That update routes choice sets of more than 68 options to label-token readout and re-selects the temperatures. The banking77 and clinc_oos rows come from after the update, which raised banking77 from 0.818. Smaller option sets are routed the same way as before.
 
 ### Speed
 
@@ -230,7 +187,7 @@ The 69 ms is engine compute for a short request (a three-sentence state), measur
 
 <p align="center"><img src="assets/how-it-works.png" alt="State and questions go through one forward pass, then label-token readout or the candidate-path head, then per-bucket temperatures, then typed answers" width="460"></p>
 
-- **Label-token readout.** Each option gets a short code, and the answer is read from the next-token logits over those codes. Codes that would split into two tokens are skipped, so every option stays one token. This handles yes/no, scores, and choice lists up to several hundred options. Choices are read in two option orders and averaged, which cancels position bias.
+- **Label-token readout.** Each option gets a short code, and the answer is read from the next-token logits over those codes. Codes that would split into two tokens are skipped, so every option stays one token. This handles yes/no, scores, and choice lists up to several hundred options. Choices are read in two option orders and averaged, which cancels position bias. Score levels stay in their trained order unless `lev serve` runs with `--score-order-average`, which is off by default.
 - **Candidate-path head.** Past that point, a small learned head matches the state against each option's text, so the number of options has no fixed ceiling.
 - **Calibration.** Temperatures fitted after training are applied at load time. There is one per question type, readout mode, and choice option-count band. They were selected by how well they carry over to task families left out of the fit, not only by how well they fit held-out rows.
 
@@ -255,6 +212,21 @@ The 69 ms is engine compute for a short request (a three-sentence state), measur
 - **English only.**
 - **Needs a GPU for real-time use.** It runs on CPU, but a 4B backbone there takes seconds per call, not milliseconds.
 
+## Hosted classification
+
+Text classification in Interfaze runs on a similar system to lev: the model reads the answer from the set of labels you define. The difference is that Interfaze is still token based, so it's a hybrid.
+
+| Item | lev | Interfaze |
+| :-- | :-- | :-- |
+| Output | Probabilities, zero output tokens | Tokens, returned as structured output |
+| Questions | Typed only: yes/no, choice, score | Any JSON schema, labels included |
+| In the same request | Classification only | OCR, web search, transcription, extraction, and more |
+| Where it runs | Your GPU | Interfaze API |
+
+Tokens cost a little speed, but they let one request classify a document while also reading, searching, and extracting from it. Define your labels as an enum in the schema, and the label comes back as a typed field.
+
+**Interfaze docs → [interfaze.ai/docs](https://interfaze.ai/docs)**
+
 ## Files
 
 | file | role |

calibration.json

@@ -1,23 +1,23 @@
 {
   "temperatures": {
-    "choice:B:large": 0.6333014152276044,
-    "choice:B": 0.6333014152276044,
-    "noul:A": 2.3332930831706458,
-    "score:A": 2.8033236630327885,
-    "choice:A": 1.7666659539679779,
-    "choice:A:small": 1.789783450971802,
-    "choice:A:mid": 1.6085685001661532,
-    "choice:A:large": 1.6641112617330986
+    "score:A": 1.8687166685806083,
+    "choice:A:small": 1.466031501359245,
+    "choice:A": 1.4606453602789027,
+    "noul:A": 1.282162150839547,
+    "choice:A:mid": 1.3228486325144098,
+    "choice:A:large": 1.4983293579095338,
+    "choice:B:large": 0.5794711375492776,
+    "choice:B": 0.5794711375492776
   },
   "fitted_on": "calibration (transfer-selected)",
   "n_samples": {
-    "choice:B:large": 10227,
-    "choice:B": 10227,
-    "noul:A": 12073,
     "score:A": 12258,
-    "choice:A": 14133,
     "choice:A:small": 11480,
+    "choice:A": 14133,
+    "noul:A": 12073,
     "choice:A:mid": 1138,
-    "choice:A:large": 1515
+    "choice:A:large": 1515,
+    "choice:B:large": 10227,
+    "choice:B": 10227
   }
 }

lev_release.json

@@ -18,5 +18,6 @@
     "calibration.json"
   ],
   "metrics": {},
-  "created": "2026-09-24T06:29:31+00:00"
+  "created": "2026-09-24T06:29:31+00:00",
+  "adapter_scale": 0.25
 }