| ---
|
| language: en
|
| license: mit
|
| library_name: pytorch
|
| base_model: BAAI/bge-small-en-v1.5
|
| pipeline_tag: text-classification
|
| datasets:
|
| - WillHeld/top_v2
|
| - mteb/amazon_massive_intent
|
| - clinc/clinc_oos
|
| - mteb/banking77
|
| - google/boolq
|
| - nyu-mll/multi_nli
|
| tags:
|
| - jevlet
|
| - decision-model
|
| - calibration
|
| - system-one
|
| - intent-classification
|
| - slot-filling
|
| - desktop-assistant
|
| - windows
|
| - bge-small
|
| - typed-questions
|
| ---
|
|
|
| # Jevlet v6
|
|
|
| Jevlet is a small **System-One decision model**: it reads a state (a typed command plus what is on
|
| screen) and a set of typed questions, and returns **calibrated probabilities over the options it
|
| is given**. It never generates text. It is a research replication of the idea behind TypeSafe's Jev,
|
| built to drive an always-on Windows command palette where every action is chosen, not written.
|
|
|
| - **Code, data pipeline, and training notebook:** [https://github.com/NAME0x0/Jevlet](https://github.com/NAME0x0/Jevlet) (commit `9d62c46989e9`)
|
| - **Base encoder:** [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5), 33,508,992 parameters in total
|
| - **Skills:** 50 (`shared/skills.json`), each with typed argument questions answered from live options
|
|
|
| ## How it decides
|
|
|
| One packed forward pass answers several questions about the same state at once:
|
|
|
| | Question | Kind | Options come from |
|
| |---|---|---|
|
| | Which action does the command ask for? | Choice | the 50 skill names |
|
| | Is it risky to run without asking? | Noul (yes/no) | fixed |
|
| | Which app / window / file / reminder …? | Choice | the live environment (installed apps, open windows, stores) |
|
| | Which part of the command is the text to use? | Choice | spans copied verbatim from the command (`shared/text_rules.json`) |
|
|
|
| Branches restart their positions after the shared state (`block_bidir` topology), so questions do not
|
| see each other. A pointer head scores each option at its boundary token. Per-kind temperatures
|
| (`temperatures` in the checkpoint) calibrate choices and yes/no answers separately. The app runs an
|
| action on one keypress only when P(safe) ≥ 0.9 and every answer's confidence ≥ 0.75; otherwise it asks.
|
|
|
| ## Results
|
|
|
| > **Reporting corrections (2026-09-26).** This card was corrected after release, during the v7 audit
|
| > (`research/v7-audit.md` in the GitHub repository). The model weights are unchanged; the `v6` tag
|
| > keeps the original card. What changed:
|
| > - grounding is reported per question: the original single figure (78.6%) averaged the control
|
| > question with a near-trivial risk question;
|
| > - training time and loss-log coverage are clarified, with measured and estimated portions
|
| > distinguished;
|
| > - the checkpoint table is labelled as a training progression, not a scaling curve.
|
|
|
| ### Held-out human commands (never used for training or for tuning the span rules)
|
|
|
| Test splits of TOPv2, MASSIVE, and CLINC150 mapped to Jevlet's skills
|
| (23,279 commands).
|
|
|
| | Question type | Questions | Accuracy | ECE | ECE (calibrated) |
|
| |---|---|---|---|---|
|
| | Skill | 23,279 | 98.0% | 0.010 | 0.001 |
|
| | Risk (Noul) | 23,279 | 99.8% | 0.030 | 0.032 |
|
| | Arguments | 14,475 | 97.1% | 0.012 | 0.004 |
|
|
|
| The risk question is trained on soft targets (a risk value per command, for example 0.04 for a
|
| benign command). Hard-label ECE scores its confidence against right/wrong, so it penalises a model
|
| that reproduces its targets; that is why it rises slightly after calibration. Scored against the
|
| soft targets, risk calibration error is 0.0027 before and 0.0013 after temperature scaling, and the
|
| mean distance between predicted and target distributions is 0.0064 → 0.0060 (these 23,279
|
| commands, `jevlet.metrics.target_calibration_error`).
|
|
|
| ### Mixture dev split (all sources)
|
|
|
| The calibration temperatures were fitted on the command and real-command dev splits, which are
|
| part of this set, so calibrated ECE here is in-sample. The held-out human commands above are not.
|
|
|
| | Question type | Questions | Accuracy | ECE | ECE (calibrated) |
|
| |---|---|---|---|---|
|
| | All questions | 113,616 | 97.7% | 0.012 | n/a |
|
| | Commands: skill | 32,682 | 99.1% | 0.005 | 0.001 |
|
| | Commands: arguments | 29,141 | 99.2% | 0.003 | 0.001 |
|
| | Grounding | see the per-question breakdown below | | | |
|
|
|
| ### Grounding: which on-screen control (reported per question)
|
|
|
| Grounding examples come from two apps held out from the grounding training split (Microsoft Teams
|
| and Spotify). Each example asks two questions, reported separately below:
|
|
|
| - **which control** to use, where the answer is "None of these controls" when the task belongs to
|
| another app (a *foreign* task);
|
| - **the risk yes/no**.
|
|
|
| The control question is further split by whether the target control is on screen. Accuracy is the
|
| argmax against the gold option; ECE (calibrated) uses the checkpoint's per-kind temperature.
|
|
|
| **Evaluation A**: the 3,000 grounding examples in the mixture dev split (the table above).
|
|
|
| | Question | Slice | Questions | Accuracy | ECE | ECE (calibrated) |
|
| |---|---|---|---|---|---|
|
| | Which control | all | 3,000 | 57.3% | 0.352 | 0.321 |
|
| | Which control | target on screen | 2,556 | 50.1% | 0.412 | 0.376 |
|
| | Which control | foreign task (answer: none) | 444 | 98.6% | 0.010 | 0.009 |
|
| | Risk yes/no | all | 3,000 | 100.0% | 0.001 | 0.001 |
|
|
|
| **Evaluation B**: the full grounding dev set that Evaluation A samples (30,000 examples; the same
|
| file, sha256 `95acbcaa923c268d27707b8ca8c324a4c0063ad72cb22a272afbdcac5ceb7aa5`).
|
|
|
| | Question | Slice | Questions | Accuracy | ECE | ECE (calibrated) |
|
| |---|---|---|---|---|---|
|
| | Which control | all | 30,000 | 56.3% | 0.362 | 0.332 |
|
| | Which control | target on screen | 25,418 | 48.7% | 0.426 | 0.390 |
|
| | Which control | foreign task (answer: none) | 4,582 | 98.5% | 0.011 | 0.009 |
|
| | Risk yes/no | all | 30,000 | 99.96% | <0.001 | <0.001 |
|
|
|
| **How to read this:**
|
| - The overall control accuracy includes foreign tasks, where answering "None of these" is correct
|
| and easy (98.5%).
|
| - When the target control *is* on screen, accuracy is 48.7% (Evaluation B), and the model answers
|
| "None of these controls" on 48.6% of those examples (12,341 of 25,418).
|
| - Every one of those answers is wrong, because the target is present. So **94.6% of its errors on
|
| target-on-screen examples are false abstentions** (12,341 of 13,042 errors), not wrong controls.
|
| Evaluation A gives 95.6% (1,220 of 1,276).
|
| - On these held-out apps the model abstains instead of choosing, and does so with high
|
| confidence.
|
| - This is v6's weakest capability. Grounding decisions should be confirmed by the user.
|
|
|
| Reproduce with `python -m scripts.eval_grounding jevlet-v6.pt grounding_v6/dev.jsonl` from the
|
| GitHub repository (add `--only-ids-in mixture_v6/dev.jsonl` for Evaluation A). The outputs are in
|
| `corrections/grounding_eval_A_3k.json` and `corrections/grounding_eval_B_30k.json`.
|
|
|
| ### Training progression on the held-out human commands (one run; not a scaling curve)
|
|
|
| 
|
|
|
| These are checkpoints of **one** cosine-scheduled run, so examples seen and learning-rate annealing
|
| are confounded: the final point also benefits from the learning rate decaying to 5%. They show how
|
| this run progressed; they do not show how accuracy scales with data. ECE is uncalibrated.
|
|
|
| | Step | Examples seen | LR factor | Skill accuracy | Skill ECE | Argument accuracy | Risk accuracy |
|
| |---|---|---|---|---|---|---|
|
| | 2,500 | 160,000 | 0.98 | 94.9% | 0.034 | 92.3% | 99.1% |
|
| | 6,250 | 400,000 | 0.88 | 96.8% | 0.019 | 94.9% | 99.5% |
|
| | 12,500 | 800,000 | 0.54 | 97.2% | 0.015 | 96.0% | 99.8% |
|
| | 25,000 (final, exported fp16) | 1,600,000 | 0.05 | 98.0% | 0.010 | 97.1% | 99.8% |
|
|
|
| Calibration temperatures: `{'choice': 1.4180543422698975, 'default': 1.1358089447021484, 'noul': 1.0306620597839355}`.
|
| The hand-written laptop benchmark (`jevlet/assistant/benchmark_v2.py`, 98 commands, 6 dangerous)
|
| runs against the live Windows environment and is reported in the GitHub repository.
|
|
|
| ## Training data
|
|
|
| 1,350,835 training rows. Nothing personal: the Colab build excludes the author's installed-app
|
| list and UI captures. Every row is dropped if it comes within 0.8 token Jaccard (digits collapsed) of a
|
| held-out benchmark command.
|
|
|
| | Source in the mixture | Train rows |
|
| |---|---|
|
| | commands | 600,000 |
|
| | daily | 240,000 |
|
| | grounding | 150,000 |
|
| | intents | 19,647 |
|
| | public | 96,222 |
|
| | real | 182,674 |
|
| | synthetic | 60,000 |
|
| | teacher | 2,292 |
|
|
|
| Public data used:
|
|
|
| | Dataset | Use | License (as reported by the source) |
|
| |---|---|---|
|
| | [WillHeld/top_v2](https://huggingface.co/datasets/WillHeld/top_v2) | TOPv2 (Facebook): human assistant commands, mapped to skills | CC BY-SA 4.0 |
|
| | [mteb/amazon_massive_intent](https://huggingface.co/datasets/mteb/amazon_massive_intent) | MASSIVE (Amazon), English: intents and mapped commands | CC BY 4.0 |
|
| | [clinc/clinc_oos](https://huggingface.co/datasets/clinc/clinc_oos) | CLINC150: intents and mapped commands | CC BY 3.0 |
|
| | [mteb/banking77](https://huggingface.co/datasets/mteb/banking77) | BANKING77: fine-grained intent choice | CC BY 4.0 |
|
| | [google/boolq](https://huggingface.co/datasets/google/boolq) | BoolQ: yes/no reading questions (Noul) | CC BY-SA 3.0 |
|
| | [nyu-mll/multi_nli](https://huggingface.co/datasets/nyu-mll/multi_nli) | MultiNLI: entailment as typed questions | mixed; see the dataset card |
|
|
|
| Synthetic sources are generated by the repository: composed commands with surface variation (typos,
|
| text-speak, casing, courtesy words), grounding (which on-screen control), daily decisions, and
|
| System-One families (contradiction, missing information, calibration, rules).
|
|
|
| ## Training procedure
|
|
|
| | Setting | Value |
|
| |---|---|
|
| | Steps | 25,000 at effective batch 64 |
|
| | Learning rate | head 0.002, encoder 0.0001, cosine, warmup 500 |
|
| | Loss | ce_brier (cross-entropy plus Brier, a proper scoring rule) |
|
| | Precision | bf16 mixed precision |
|
| | Hardware | NVIDIA A100-SXM4-80GB (Google Colab) |
|
| | Training time | ≈5 h total A100 training (estimated): 3.33 h measured for steps 8,550–25,000 after a resume; first-session duration not recorded. |
|
| | Throughput | 25,960 tokens/s (measured over steps 8,550–25,000) |
|
| | Peak VRAM | 10.2 GB |
|
|
|
| 
|
|
|
| Logged steps 8,550–25,000. The first training session's progress log was not preserved during the
|
| original resume; missing values have not been reconstructed. The resume-logging issue has since been
|
| fixed.
|
|
|
| `progress.jsonl` contains the loss curve for steps 8,550–25,000; the first session's log was lost on
|
| resume. `run_config.json` holds the exact configuration and every session's GPU. `results.json` holds
|
| the original evaluation. The corrections above come from `scripts/eval_grounding.py`
|
| (`corrections/*.json`) and `scripts/plot_training.py` (`figures/training_figures.json`).
|
|
|
| ## Files
|
|
|
| | File | What |
|
| |---|---|
|
| | `jevlet-v6.pt` | fp16 checkpoint with per-kind temperatures (loaded by `jevlet.system_one.SystemOne`) |
|
| | `model.safetensors`, `config.json` | the same weights without pickle, plus model config and temperatures |
|
| | `shared/skills.json`, `shared/text_rules.json` | the skill catalogue and span rules the model was trained against |
|
| | `results.json`, `progress.jsonl`, `run_config.json` | evaluation, training curve (steps 8,550–25,000), configuration |
|
| | `figures/`, `corrections/` | card figures with their data, and the per-question grounding evaluation (added 2026-09-26) |
|
|
|
| ## Use
|
|
|
| ```python
|
| from huggingface_hub import hf_hub_download
|
| from jevlet.benchmarks import RISK_QUESTION, daily_state
|
| from jevlet.system_one import ChoiceQuestion, NoulQuestion, SystemOne
|
|
|
| engine = SystemOne(hf_hub_download("NAME0x0/Jevlet", "jevlet-v6.pt"))
|
| answers = engine.evaluate(
|
| daily_state("remind me to call the bank at 5", "OUTLOOK: Inbox - Outlook"),
|
| {
|
| "skill": ChoiceQuestion("Which action does the command ask for?", ["Set a reminder", "Set an alarm clock", "Search the web"]),
|
| "risk": NoulQuestion(RISK_QUESTION),
|
| },
|
| )
|
| print(answers["skill"].selected, answers["skill"].confidence, answers["risk"].probability_true)
|
| ```
|
|
|
| The planner in `jevlet/assistant/planner.py` builds the full question set from the live environment.
|
|
|
| ## Limitations
|
|
|
| - English only; tuned for a Windows laptop's apps, windows, and settings pages.
|
| - It chooses among the options it is given. If the right app, file, or span is not offered it cannot
|
| answer correctly, and should pick "Not applicable" or ask.
|
| - Free-text arguments are spans of the command. On held-out human commands, span extraction offers the
|
| right span for about 85% of reminders, directions, music, and weather requests.
|
| - Arithmetic, dates, and times are parsed by code, not predicted.
|
| - The risk question is a safety net, not a guarantee: destructive actions also have fixed risk floors.
|
|
|
