|
Download README.md from bpHigh/glance-0.1: direct link, hf CLI and curl.
- Browser
- Download file 14.1 kB
-
https://huggingface.co/bpHigh/glance-0.1/resolve/main/README.md
- Command line
-
hf download hf://bpHigh/glance-0.1/README.md
-
curl -L -o README.md https://huggingface.co/bpHigh/glance-0.1/resolve/main/README.md
14.1 kB
| license: cc-by-nc-4.0 | |
| base_model: Qwen/Qwen3.5-4B | |
| base_model_relation: finetune | |
| language: | |
| - en | |
| tags: | |
| - decision-model | |
| - classification | |
| - routing | |
| - llm-as-judge | |
| - calibration | |
| - vision-language | |
| # Glance 0.1 | |
| Glance 0.1 is a 4B-class decision model built on Qwen3.5-4B (4.54B parameters: the base without | |
| its multi-token-prediction head). You give it a state (text, and optionally images) and one or | |
| more typed questions. It returns a calibrated probability distribution for each question from a | |
| single forward pass, with no generated text. That takes 53β58 ms per decision (p50) on one H100, | |
| and under half a second for short prompts on 8 CPU cores. | |
| It is built for the decisions an application or agent makes many times a call: routing, triage, | |
| judging, moderation, extraction checks, tool and action selection, policy and workflow gates. | |
| **Status: research preview.** The weights are for non-commercial research use (CC BY-NC 4.0). The | |
| Glance runtime that runs them is in `runtime/` (MIT). | |
| ## What it answers | |
| | Question type | You give | It returns | | |
| |---|---|---| | |
| | `choice` | 2β255 labelled options, optionally with a definition for each | a probability per option | | |
| | `noul` (yes/no) | a statement, optionally with what true and false mean | the probability it is true | | |
| | `score` | an ordered list of levels | a probability per level and the expected score | | |
| Several questions about the same state are answered in one pass. Example request and answer: | |
| ```json | |
| { | |
| "state": {"ticket": "My card was charged twice for the same order, please refund one."}, | |
| "questions": { | |
| "team": {"type": "choice", "instructions": "Which team should handle this ticket?", | |
| "criteria": {"billing": "payments, refunds, invoices", "tech": "bugs and outages", "sales": "new purchases"}}, | |
| "urgent": {"type": "noul", "instructions": "The customer needs a reply today."}, | |
| "frustration": {"type": "score", "instructions": "How frustrated is the customer?", "criteria": ["0", "1", "2", "3", "4"]} | |
| } | |
| } | |
| ``` | |
| ```json | |
| { | |
| "answers": { | |
| "team": {"type": "choice", "choice": "billing", "probabilities": {"billing": 0.964, "tech": 0.021, "sales": 0.015}, "confidence": 0.946}, | |
| "urgent": {"type": "noul", "noul": 0.731}, | |
| "frustration": {"type": "score", "score": 2.0, "probabilities": {"0": 0.007, "1": 0.301, "2": 0.438, "3": 0.195, "4": 0.059}, "confidence": 0.477} | |
| } | |
| } | |
| ``` | |
| (This is Glance 0.1's output for the request above, recorded with the runtime on CPU and rounded. | |
| On a GPU the numbers differ slightly in floating point.) | |
| ## How it works | |
| - **Base:** Qwen3.5-4B, vision encoder included. | |
| - **Attention:** two-way (bidirectional) attention in the full-attention layers, so the model reads | |
| the state and all questions before answering. The linear-attention layers are unchanged. | |
| - **Readout:** every option is tagged with a two-letter answer code that is a single token. The | |
| answer distribution is the LM-head logits over those codes at each question's answer slot. Codes | |
| and the yes/no order are drawn per decision, so the model cannot learn position shortcuts. | |
| - **Calibration:** per-bucket temperatures (question type Γ option count Γ modality), fitted on | |
| half of the held-out questions and tested on the other half. They are stored in | |
| `decision_config.json` and applied at inference. | |
| ## Files | |
| - `lm/`: the fine-tuned Qwen3.5-4B weights (bf16, 9.1 GB in safetensors shards with an index), | |
| tokenizer, chat template and image processor. | |
| - `decision_config.json`: the answer codes and their token ids, attention mode, readout and | |
| calibration temperatures. | |
| - `results/`: the evaluation results below in full (scores and per-task tables; no benchmark items). | |
| - `runtime/`: the Glance runtime (`glance_runtime`, MIT). It holds the prompt layout, two-way | |
| attention, answer-code readout and calibration, with a Python API and an HTTP server. | |
| Loading `lm/` with plain `transformers` gives the language model, but not the decision model. Use | |
| the runtime. | |
| ## Quick start | |
| ```bash | |
| pip install -U huggingface_hub | |
| hf download bpHigh/glance-0.1 --include "runtime/*" --local-dir glance-0.1 | |
| ``` | |
| On a GPU: | |
| ```bash | |
| pip install "./glance-0.1/runtime[gpu]" | |
| ``` | |
| On a CPU-only machine, install the CPU build of PyTorch first. The default Linux build pulls | |
| several GB of CUDA libraries. | |
| ```bash | |
| pip install torch --index-url https://download.pytorch.org/whl/cpu | |
| pip install ./glance-0.1/runtime | |
| ``` | |
| The runtime needs Python 3.10 or later, torch, transformers 5.20.1 or later, and pillow. It was | |
| tested with torch 2.13 and transformers 5.20.1 on H100, and with torch 2.14.1 (CPU) and | |
| transformers 5.20.2 on CPU. | |
| ```python | |
| from glance_runtime import Glance | |
| glance = Glance.load("bpHigh/glance-0.1", device="cuda") # downloads the weights; or device="cpu" | |
| answers = glance( | |
| state={"ticket": "My card was charged twice for the same order, please refund one."}, | |
| questions={ | |
| "team": {"type": "choice", "instructions": "Which team should handle this ticket?", | |
| "criteria": {"billing": "payments, refunds, invoices", "tech": "bugs and outages", | |
| "sales": "new purchases"}}, | |
| "urgent": {"type": "noul", "instructions": "The customer needs a reply today."}, | |
| }, | |
| ) | |
| answers["team"]["probabilities"] # {"billing": ..., "tech": ..., "sales": ...} | |
| answers["urgent"]["noul"] # P(yes) | |
| # images: paths, PIL images or data: URLs (downscaled to 640 px on the long side, as in training) | |
| glance(state="Photo from the returns desk.", images=["photo.jpg"], | |
| questions={"damaged": {"type": "noul", "instructions": "The item in the photo is damaged."}}) | |
| ``` | |
| Or serve `POST /v1/systemone` over HTTP: | |
| ```bash | |
| python -m glance_runtime serve --model bpHigh/glance-0.1 --device cuda --port 8741 | |
| curl -s localhost:8741/v1/systemone -H 'Content-Type: application/json' -d @request.json | |
| ``` | |
| Each question gets its own sequence over the shared state, and the sequences run as one batched | |
| forward pass, so the other questions' text never enters a question's input. The answer codes are | |
| drawn from a hash of the whole request, so adding or changing a question can shift the other | |
| answers slightly. Answers are deterministic for a given request. | |
| ## Training | |
| Four stages: | |
| | Stage | Data | Method | | |
| |---|---|---| | |
| | 1 | about 1.55M decisions | full fine-tune, causal attention | | |
| | 2 | about 556k harder and more diverse decisions | LoRA (r 64), two-way attention | | |
| | 3 | about 300k decisions with images, plus text replay | LoRA, two-way attention | | |
| | 4 | about 151k routing, judging and agent decisions, plus replay | LoRA, two-way attention | | |
| Each LoRA stage was merged before the next. That is about 1.9B training tokens in all. | |
| **Text sources:** | |
| - the tasksource collection of public classification and NLI sets; | |
| - typed decisions; | |
| - Eikos decisions; | |
| - Open-Jev workflows and games; | |
| - the official train splits of the Decision Index benchmarks; | |
| - procedurally generated rule-book, numeric and game decisions; | |
| - routing, judging and agent-outcome questions written over unlabelled conversations (WildChat, | |
| OpenAssistant, UltraChat) and agent traces. | |
| **Image sources:** The Cauldron and FineVision subsets, Amazon Berkeley Objects, recognition | |
| datasets, visual inspection sets and rendered game screens. | |
| **Targets:** | |
| - gold labels, mixed with the distributions of a larger teacher decision model; | |
| - for the unlabelled routing, judging and agent questions, the teacher's distributions alone. | |
| **Contamination:** every training source was checked against the evaluation sets below with an | |
| 8-gram index, and a held-out manifest kept the held-out splits out. | |
| ## Evaluation | |
| All numbers are our own runs. None is a leaderboard submission yet. | |
| ### DecisionBench (official rules) | |
| A correct answer is one whose first arg-max is the gold candidate. An invalid answer counts as a | |
| miss, and there is no chance correction. Other models' scores are from the official | |
| [DecisionBench 1.0 leaderboard](https://huggingface.co/datasets/Hanno-Labs/decision-bench-results) | |
| (`leaderboard.json`, updated 9 Oct 2026; its newest submission is from 27 Sep). Glance 0.1's is | |
| our own run with the official scoring rules, not a board submission. | |
| | Model | Headline (22,700 non-reasoning rows) | Reasoning track (1,200 rows) | | |
| |---|---:|---:| | |
| | Bosun v3.1 1.7B | 87.3 | 39.7 | | |
| | Bosun v3.1 0.6B | 83.2 | 43.1 | | |
| | imajev-4b (revision 712891d) | 79.9 | 68.0 | | |
| | imajev-4b (revision c9e5f13) | 79.6 | 80.6 | | |
| | **Glance 0.1** | **78.8** | **68.6** | | |
| | Winnow-12B | 77.0 | 70.2 | | |
| | Juspay xor | 76.6 | 66.1 | | |
| | GLM-5.3 Flash | 73.7 | 87.1 | | |
| | Jev 1.13 (snapshot of 21 Sep) | 71.9 | 74.5 | | |
| - **Bosun is in-domain:** its authors score held-out rows of the task families it was trained on. | |
| - **Jev 1.14** (released 14 Oct) is not on the board yet. | |
| - **GLM-5.3 Flash's** reasoning score covers 1,197 of the 1,200 rows. | |
| Glance 0.1 by task family: | |
| - **Strong families:** content moderation 100%, function and agent-skill routing 98.7%, safety | |
| gating 94%, bounded extraction 93%, routing triage 92%. | |
| - **Weak families:** document classification 38%, record classification 47%, target selection | |
| 49%, rubric scoring 49%, review scoring 50%, retrieval ranking 54%. | |
| Full tables: `results/decision-bench.json`, `results/decision-bench-by-task.md`. | |
| ### JevBench, public items v1.2 (local run with a port of the v1.3/v1.4 scorer) | |
| | Tier | Accuracy | | |
| |---|---:| | |
| | Easy (48 items) | 100% | | |
| | Standard (72 items) | 100% | | |
| | Hard (111 items) | 61.3% | | |
| - **Scores:** Intelligence 75.7, Calibration 67.6; hard-tier ECE 0.194. | |
| - **Hard tier is the weak spot:** temporal and numeric reasoning 26.7%, trade-offs 33%, long | |
| policies 52.6%. | |
| - **Strong hard categories:** adversarial, routing and trap items are all at 100%. | |
| Full tables: `results/jevbench-public.json`, `results/jevbench-public-by-category.md`. | |
| ### Decision Index test splits, scored locally (unofficial) | |
| This covers the full test splits of 15 of the 38 Decision Index benchmarks: 124,370 questions, | |
| scored with our harness and metrics. They are not the suite's selected rows, benchmark-specific | |
| metrics or area weights, so they are **not comparable with the Decision Index board**. The model | |
| was trained on these benchmarks' official train splits, so the scores are in-domain. | |
| **Overall:** 83.0% accuracy, chance-corrected 74.3. | |
| | Range | Benchmarks | | |
| |---|---| | |
| | Strong | ARC-Easy 97.6%, ARC-Challenge 93.4%, HellaSwag 91.6%, BANKING77 90.8%, SGD 90.1%, ContractNLI 88.4%, CLINC150 86.6% | | |
| | Weak | GSM8K as choices 58.9β66.6%, Humicroedit 61.1%, CLadder 64.6%, ANLI 67.9%, VAST 70.9% | | |
| Full table: `results/decision-index-local.md`. | |
| ### Held-out splits of the training sources (in distribution) | |
| Uncalibrated. cc_mean is chance-corrected accuracy: 100 is perfect, 0 is chance. | |
| | Held-out set | Questions | Accuracy | cc_mean | ECE | | |
| |---|---:|---:|---:|---:| | |
| | typed-decisions test | 2,000 | 67.6% | 61.2 | 0.024 | | |
| | Eikos held-out | 1,191 | 91.2% | 84.8 | 0.041 | | |
| | Open-Jev test | 10,356 | 89.0% | 78.8 | 0.041 | | |
| | Open-Jev out of distribution | 15,701 | 85.3% | 70.5 | 0.057 | | |
| **Calibration** (5,220 test questions): NLL 0.337 β 0.309, ECE 0.040 β 0.020. Full tables: | |
| `results/heldout.md`, `results/calibration.json`. | |
| ### Latency and hardware | |
| The 231 JevBench public items, one decision at a time (batch 1), bf16, timed end to end: | |
| | State length (tokens) | Items | 1Γ H100, p50 | CPU (8 cores), p50 | | |
| |---|---:|---:|---:| | |
| | under 250 | 152 | 57 ms | 0.42 s | | |
| | 250β500 | 26 | 59 ms | 1.6 s | | |
| | 500β1,000 | 12 | 62 ms | 3.2 s | | |
| | 1,000β2,000 | 5 | 65 ms | 5.0 s | | |
| | 2,000β3,700 | 36 | 83 ms | 12.3 s | | |
| | **All items: p50 / p95** | **231** | **58 / 92 ms** | **0.47 / 13.3 s** | | |
| On the 124,370 Decision Index questions, which are mostly short, the H100 p50 was 53 ms and the | |
| p95 57 ms. | |
| - **H100:** with the `flash-linear-attention` kernels (`pip install "./runtime[gpu]"`). Without | |
| them the linear-attention layers fall back to slower reference PyTorch code. | |
| - **CPU:** 8 reserved cores of an AMD EPYC 9R45 (AVX-512 BF16), PyTorch 2.13, 8 threads, bf16. | |
| The fast linear-attention kernels are GPU-only, so CPU latency grows with prompt length. | |
| Short routing-style prompts take under half a second. Peak RAM with the model loaded was | |
| 9.1 GB. | |
| - **Numerics:** CPU and GPU answers differ slightly in floating point. On these items the harness | |
| accuracy was 81.8% on CPU and 81.4% on H100. | |
| - **Other GPUs** (consumer or data-centre cards other than H100) are not measured yet. The bf16 | |
| weights alone take 9.1 GB of memory. | |
| ## Limitations | |
| - **One pass, no reasoning.** Multi-step arithmetic, temporal, causal and trade-off questions are | |
| the weakest areas (JevBench hard, CLadder, GSM8K). A reasoning version is planned (below). | |
| - **Overconfident when hard.** It is overconfident on hard out-of-distribution items (JevBench | |
| hard ECE 0.194), even after calibration. | |
| - **Weak DecisionBench families:** document and record classification, ranking, and rubric or | |
| review scoring. | |
| - **Mostly English.** Indic languages are not targeted yet. | |
| - **Needs the Glance runtime** (`runtime/`): plain `transformers` loading gives only the language model. | |
| - **Not for consequential automated decisions** about people without human review. | |
| ## Coming next (TBD) | |
| - **Glance Indic:** an Indic-friendly decision model. | |
| - **Glance Reasoning:** a reasoning decision model that thinks only when its one-pass read is | |
| unsure. | |
| - **RL-pilled Glance:** a decision model post-trained with reinforcement learning. | |
| - **Smaller and larger versions** of Glance. | |
| ## Licence | |
| The weights are released under CC BY-NC 4.0, for non-commercial research use. The training data | |
| includes sources under non-commercial and research-only terms. | |
| The runtime code in `runtime/` is released under the MIT licence (`runtime/LICENSE`). | |
| The base model is [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B), released by the Qwen | |
| team under Apache-2.0. Its licence and notices continue to apply to the base weights. | |