| --- |
| license: apache-2.0 |
| language: |
| - pl |
| - en |
| base_model: speakleash/Bielik-1.5B-v3.0-Instruct |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - typed-decisions |
| - decision-making |
| - classification |
| - calibration |
| - polish |
| --- |
| |
| # basal-2.0-mini |
|
|
| [](https://github.com/rkinas/basal) |
| [](https://basal.si5.pl/) |
| [](https://huggingface.co/collections/Remek/basal-20-6acd1712a8e443c86cd576fc) |
|
|
| **basal-2.0-mini** is part of basal-2.0, a family of typed-decision models for Polish and English; it is the fastest model of the family. Built on Bielik-1.5B-v3.0-Instruct. Trained further from earlier basal checkpoints, with basal-2.0-max as the teacher. |
|
|
| **What it is.** The model reads a *state* (a message, a document, retrieved passages, an agent trace, a web page as |
| JSON) and answers a typed question about it (`choice`, yes/no `noul`, `score`, `multi`, `act`, `tool`) by returning a |
| **calibrated probability for each allowed answer**, in one forward pass, without generating text. The answer never falls |
| outside the options you give, and the probability says how sure the model is. Because the classes are described in the |
| request, one model serves many tasks without retraining: routing, rule and policy checks, questions about long |
| documents, RAG decisions, urgency and risk scores, judging answers, agent and tool decisions, and triage with a |
| confidence threshold. |
|
|
| | model | params | use | |
| |---|---|---| |
| | [basal-2.0-mini](https://huggingface.co/Remek/basal-2.0-mini) | 1.5B | fastest | |
| | [basal-2.0 (4.5B)](https://huggingface.co/Remek/basal-2.0-4.5B) | 4.5B | main model; experimental thinking mode | |
| | [basal-2.0-max](https://huggingface.co/Remek/basal-2.0-max) | 11B | most accurate | |
|
|
| Each model also comes as `-FP8` (NVIDIA Hopper and Blackwell), `-GGUF` (Q8_0 and Q5_K_M; Ollama, llama.cpp) and `-MLX-8bit` (Apple Silicon). |
| |
| ## Intended use |
| |
| - **Good for:** routing messages and documents to one of the options you describe; yes/no checks of rules and policies; |
| urgency and risk scores; questions about long documents and retrieved passages (is the answer there, is it relevant); |
| judging answers; choosing an agent's next action or tool (a `tool` question over OpenAI or MCP tool definitions picks the tool, not its arguments); triage that hands uncertain cases to a person (an `act` |
| question with a defer action and `max_error`). |
| - **Languages:** Polish and English (the state and the question may be in either). |
| - **Not for:** writing text or chatting (the model only picks among the options you give); facts that are not in the |
| state; decisions with serious consequences for people without human review. |
|
|
| ## Quick start |
|
|
| **1. Install** the basal engine v2.0.0 into a fresh [uv](https://docs.astral.sh/uv/) environment (NVIDIA GPU; the cu128 torch |
| needs a driver for CUDA 12.8 or newer; DGX Spark and B300 use the cu130 build, see [docs/HARDWARE.md](https://github.com/rkinas/basal/blob/v2.0.0/docs/HARDWARE.md)): |
|
|
| ```bash |
| uv venv --python 3.12 --allow-existing ~/basal-env && source ~/basal-env/bin/activate |
| uv pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu128 |
| uv pip install "basal[fp8] @ https://github.com/rkinas/basal/archive/refs/tags/v2.0.0.tar.gz" |
| ``` |
|
|
| **2. Start the server** and wait for `basal: model ... ready`: |
|
|
| ```bash |
| basal-serve --model Remek/basal-2.0-mini --mode fast --port 8000 |
| ``` |
|
|
| **3. Ask questions** with curl. Several questions about one state go into one request and share the state: |
|
|
| ```bash |
| curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{ |
| "state": "Klient: od wczoraj nie mog臋 zalogowa膰 si臋 do bankowo艣ci internetowej, system pokazuje b艂膮d has艂a.", |
| "questions": { |
| "dept": {"type": "choice", "instructions": "Do kt贸rego dzia艂u skierowa膰 zg艂oszenie?", |
| "criteria": {"cards": "Reklamacje kart", "online": "Wsparcie bankowo艣ci elektronicznej", "loans": "Kredyty"}}, |
| "urgent": {"type": "noul", "instructions": "Czy klient nie mo偶e korzysta膰 z us艂ugi?"}}}' |
| ``` |
|
|
| Each answer has a probability for every option, the chosen key and the confidence: |
| `"dept": {"type": "choice", "choice": "<key>", "probabilities": {"cards": p, "online": p, "loans": p}, "confidence": p}`. |
|
|
| To send uncertain decisions to a person, ask the same question as an `act` question with a defer action and |
| `"max_error": 0.05` (`max_error` works on `act` and `tool` questions only): |
|
|
| ```bash |
| curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{ |
| "state": "Klient: od wczoraj nie mog臋 zalogowa膰 si臋 do bankowo艣ci internetowej, system pokazuje b艂膮d has艂a.", |
| "questions": { |
| "dept": {"type": "act", "instructions": "Do kt贸rego dzia艂u skierowa膰 zg艂oszenie?", |
| "criteria": {"cards": "Reklamacje kart", "online": "Wsparcie bankowo艣ci elektronicznej", "loans": "Kredyty"}, |
| "option_keys": "hide", "costs": {"wrong": 19, "defer": 1}, "max_error": 0.05}}}' |
| ``` |
|
|
| The answer is the action with the lowest expected cost (an option key, or `defer`). Below the confidence threshold |
| certified for 5% error it returns `defer`, names the `refused` action and says `"deferred": true`. `"option_keys": |
| "hide"` shows the model the option descriptions only, as the calibration was fitted (with the keys shown, answers say |
| `"calibration": "unvalidated"`); keep the cost of a wrong answer moderate, because a large one defers by expected cost |
| alone, before `max_error` is applied. |
|
|
| **Or from Python** (the package includes a small client): |
|
|
| ```python |
| from basal.client import Basal |
| b = Basal("http://127.0.0.1:8000") |
| state = "Klient: od wczoraj nie mog臋 zalogowa膰 si臋 do bankowo艣ci internetowej, system pokazuje b艂膮d has艂a." |
| a = b.choice(state, "Do kt贸rego dzia艂u skierowa膰 zg艂oszenie?", |
| {"cards": "Reklamacje kart", "online": "Wsparcie bankowo艣ci elektronicznej", "loans": "Kredyty"}) |
| print(a["choice"], a["confidence"]) |
| print(b.yes_no(state, "Czy klient nie mo偶e korzysta膰 z us艂ugi?")["noul"]) # P(yes) |
| ``` |
|
|
| Many items from a file: `basal-run --input items.jsonl --output answers.jsonl`. API and features: [README](https://github.com/rkinas/basal#api). |
|
|
| The same basal engine v2.0.0 also serves the experimental basal-omni-1.0 models (images and audio): [README](https://github.com/rkinas/basal#images-and-audio-basal-omni-experimental). |
|
|
| ## Results against basal-1.5 |
|
|
| Measured once on the frozen release weights, bf16 (`basal-serve --mode fast`) on one H100, and published as measured, including where basal-2.0 is not better. |
|
|
| | benchmark | basal-1.5 | basal-2.0 | 2.0 - 1.5 | note | |
| |---|---|---|---|---| |
| | Werdykt v1, macro [95% CI] | 0.607 [0.594, 0.619] | 0.618 [0.605, 0.631] | +1.2 pt | hidden PL + EN benchmark, 5,000 decisions | |
| | Werdykt v1, PL / EN | 0.619 / 0.594 | 0.628 / 0.609 | +0.8 / +1.5 pt | | |
| | Werdykt calibration error (ECE) | 0.086 | 0.035 | -0.051 | lower is better | |
| | Werdykt automated at 1% target error | 11.9% at 1.66% observed error | 15.2% at 1.82% observed error | | share automated, error observed | |
| | Werdykt automated at 5% target error | 21.9% at 4.33% observed error | 25.4% at 5.12% observed error | | share automated, error observed | |
| | Werdykt board rank | 82 | 80 | | of 94 systems on the public board | |
| | Werdykt speed pass, decisions/s at 16 clients | 20.5 (c1 p50 14 ms) | 106.4 (c1 p50 16 ms) | x5.2 | both in basal-serve's default mode; bf16, one H100 80GB | |
| | Decision Index (JDI 0.2.1) | 16.41 | 26.69 | +10.28 | | |
| | JDI requests left unanswered | 17.2% (25,782 of 150,317) | 0.02% (26 of 150,317) | | basal-1.5 answers questions with at most 10 options | |
| | LEK, Polish medical licensing exam, PL / EN (% correct) | 47.7 / 48.4 | 47.7 / 49.2 | +0.0 / +0.8 pt | 5 options | |
| | PL-Holdout (4,000 items) | 0.8370 | 0.8293 | -0.0077 [-0.0158, +0.0000] | paired | |
| | Polish decisions (7,081 items) | 0.8805 | 0.8665 | -0.0140 [-0.0201, -0.0079] | paired | |
| | Polish general (3,079 items; dev label) | 0.6710 | 0.6931 | +0.0221 [+0.0097, +0.0341] | paired | |
| | English corpus test (1,479 items; dev label) | 0.5943 | 0.6051 | +0.0108 [-0.0074, +0.0291] | paired | |
| | English panel (10 tasks; mean accuracy) | 0.5818 | 0.5606 | -0.0212 (items -0.0146 [-0.0288, -0.0003]) | mean of tasks; items paired | |
| | PL-Real-300, doc-clean subset (158 items; primary) | 0.9177 | 0.9177 | +0.0000 [-0.0380, +0.0380] | paired | |
| | PL-Real-300, all 300 items | 0.9367 | 0.9167 | -0.0200 [-0.0467, +0.0067] | paired | |
|
|
| basal-1.5 is basal-1.5-mini, the previous release. **Werdykt** rows are the public board's rows (basal-1.5 served in basal-serve's default mode, basal-2.0 by basal-serve v2.0.0 in its default mode), each release with its own calibration file, accuracy with `--deterministic`; *automated at X% target error* = a confidence threshold chosen on one half of the items for that error, applied to the other half. The speed pass is the board's (bf16, 16 clients, one H100 80GB; separate pods). The rank is on the public Werdykt board ([basal.si5.pl/benchmark](https://basal.si5.pl/benchmark/)), 94 systems, one row per system. **All other rows:** both models served by engine v2.0.0 (commit adc4c71) on the same items; differences are paired, with 95% bootstrap intervals. Polish general and the English corpus test were used to choose between training runs, so their numbers are development numbers. LEK questions: NASK-PIB Reassess, 2008-2026 sessions (CC BY-NC-SA 4.0), shown for the basal models only, with no comparison to other systems. |
|
|
| ## What is new since basal-1.5 |
|
|
| - **Prompt format 2.** Up to 255 options per question (basal-1.5: 10), read in one forward pass with one option order |
| by default (basal-1.5 read two orders). |
| - **Training.** Trained further from earlier basal checkpoints. The mini is a full fine-tune distilled from basal-2.0-max. |
| - **Calibration.** Temperatures per question type and option-count bucket, fitted on question families the model was |
| not trained on. |
| - **Engine v2.0.0.** Compact batching (+19% decisions/s at 16 clients, +17% at 32, +8% at one client) and a prompt-template cache (+20% at 8 clients) (one H100, bf16, each change left out in turn). They need FlashAttention 3 (Hopper): the Werdykt rows on this card ran with FlashAttention 3; the release check, the port checks, the FP8 speed and the calibration ran with FlashAttention 2, without them. Engine v2.0.0 schedules long requests (over 2,048 tokens) behind short ones: in a mixed load at 32 clients short requests answer in 0.81 s instead of 1.38 s (p50), and long requests take about 7% longer (p50 1.96 s instead of 1.83 s); `--scheduler off` serves in arrival order. |
|
|
| ## Ports |
|
|
| Every port is checked against this bf16 model on the development set dev v2: **agreement** is the share of decisions where the port picks the same option. Bars for this tier: agreement 2-10 >= 0.9161, agreement at reference margin >= 0.2 >= 0.9858, accuracy within 1 point, ECE at most +0.01. |
|
|
| | package | agreement with bf16 (options 2-10) | agreement at margin >= 0.2 | accuracy diff | ECE diff | size | speed | gate | |
| |---|---|---|---|---|---|---|---| |
| | [FP8](https://huggingface.co/Remek/basal-2.0-mini-FP8) | 0.9333 | 0.9937 | -0.0077 | -0.0032 | 1.8 GB | 103.5 dec/s c16, c1 p50 14 ms (H100) | PASS | |
| | [GGUF Q8_0](https://huggingface.co/Remek/basal-2.0-mini-GGUF) | 0.9769 | 1.0000 | -0.0004 | -0.0019 | 1.6 GB | not measured in this release | PASS | |
| | [GGUF Q5_K_M](https://huggingface.co/Remek/basal-2.0-mini-GGUF) | 0.9339 | 0.9931 | -0.0006 | -0.0016 | 1.1 GB | not measured in this release | PASS | |
| | [MLX 8-bit](https://huggingface.co/Remek/basal-2.0-mini-MLX-8bit) | 0.9792 | 1.0000 | -0.0025 | +0.0024 | 1.7 GB | 93 ms c1 p50, 223 ms for 5 questions (Mac) | PASS | |
|
|
| ## Calibration |
|
|
| For every question basal returns a probability for each answer; the confidence is the probability of the answer it picks. |
| The probabilities are calibrated (a temperature per question type and option-count bucket) on question families that |
| were never used for training, so they hold on new tasks but are cautious. Held-out numbers below are leave-one-family-out |
| on the development and test splits. |
|
|
| | basal-2.0-mini, held-out | | |
| |---|---| |
| | calibration error (ECE) | 0.017 | |
| | worst question family (ECE) | 0.099 (agent) | |
| | automated at the 5% error threshold | 10.5% at 2.94% observed error | |
| | 1% error threshold, shipped file | defers every decision (no 1% threshold certified on the full fit) | |
|
|
| On Werdykt (hidden, measured once, not used to choose the model) the calibration error is 0.035 (basal-1.5: 0.086). The calibration is a file, separate from the weights (`CALIBRATION.json`, one |
| option order; `CALIBRATION.o2.json`, two orders), so you can swap it or refit it on your own data with `basal-calibrate`. |
| Fitted with engine v2.0.0 (commit adc4c71). |
|
|
| ## Speed |
|
|
| One decision = one question, one option order. Hardware is named on every number. |
|
|
| - Werdykt speed pass (bf16, one H100 80GB): basal-2.0 106.4 decisions/s at 16 clients, c1 p50 16 ms (p95 94 ms) in basal-serve v2.0.0's default mode; |
| basal-1.5 20.5 decisions/s at 16 clients, c1 p50 14 ms (p95 346 ms) in basal-serve's default mode, as on the public board. |
| Separate pods of the same GPU type. |
| - FP8 package: 103.5 dec/s c16, c1 p50 14 ms (H100); one NVIDIA H100 80GB (cluster node), one option order. |
| - Measured setup: the Werdykt rows (accuracy and the speed pass) ran on the Werdykt pods with FlashAttention 3 installed, which the engine needs for compact rows and the template prefix (both engine defaults); the release check, the port checks, the FP8 speed and the calibration fits ran with FlashAttention 2, so compact rows and the template prefix were off there. FP8 keeps FlashAttention 2 unless `--attn fa3`. |
| - Accuracy rows use `--deterministic`; default minus `--deterministic` accuracy on dev v2: bf16 +0.0000, FP8 +0.0016 (run 1; 32 requests in flight); `--deterministic` itself: bf16 200/200 identical at 16 clients, FP8 200/200 identical at 16 clients. |
|
|
| **basal-1.5's other configuration.** basal-1.5 also shipped FP8 checkpoints served by vLLM. On one RunPod H100 (median of |
| 3 fresh servers; synthetic ~1.6k-token states and a 400-request mix, a different workload from the Werdykt speed pass) |
| the two basal-1.5 configurations measured: |
|
|
| | basal-1.5 configuration | c1 p50 | decisions/s, 16 / 32 clients | mixed traffic, sequential / 32 clients (requests/s) | 16k-token state x 5 questions | |
| |---|---|---|---|---| |
| | bf16 on v1.5.0 | 30.8 ms | 49.9 / 49.0 | 4.9 / 5.4 | 8,411 ms (beyond its 8,192 context; v1.5.0 does not refuse) | |
| | FP8 package on v1.5.0 vLLM | 55.3 ms | 51.3 / 86.7 | 14.8 / 81.2 (~290 refused) | refused (8,192 context) | |
|
|
| basal-2.0 was not measured on that machine, so the speed-ups in the results table hold only against the basal-1.5 rows of the |
| Werdykt speed pass. For mixed traffic with long, repeated documents, measure both on your own load before you switch. |
|
|
| ## Evaluation notes |
|
|
| - **Werdykt (hidden benchmark).** Every training file after basal-1.5 was checked against the Werdykt guard list (exact |
| state, rare 13-word shingles, source documents) and builds fail on any hit. The mini's training rows pass the same guard. |
| - **PL-Real-300.** Training rows of the basal checkpoints between basal-1.5 and basal-2.0 came from 27 of the PL-Real |
| source judgments (their guard excluded PL-Real *states*, not the source *documents*); the 2.0 training files add 0. |
| The primary PL-Real figure is therefore the **doc-clean subset** (158 items on the other judgments), for both models; |
| the full 300 is shown next to it. |
| - **Selection sets.** Polish general and the English corpus test were used to choose between training runs, so their |
| numbers are development numbers. PL-Holdout, PL-Real, LEK, JDI and Werdykt were not used to select basal-2.0. |
|
|
| ## Files |
|
|
| - `model.safetensors`, tokenizer, `chat_template.jinja`: the model |
| - `CALIBRATION.json` (one option order) and `CALIBRATION.o2.json` (two orders): per-type temperatures and confidence thresholds for 1% / 5% error |
| - `basal.json`: prompt format 2 and readout protocol |
|
|
| ## How it works |
|
|
| The model receives a fixed chat prompt with the state, the question and coded options; the decision is the softmax over the next-token logits of the option codes only (one forward pass, fp32 code logits, no text generation), then the per-type calibrated temperature from `CALIBRATION.json`. Confidence thresholds were chosen on calibration data, before testing, for 1% or 5% error among accepted decisions. |
|
|
| ## Training |
|
|
| Fine-tuned from Bielik-1.5B-v3.0-Instruct for typed decisions in Polish and English, trained further from earlier basal checkpoints. The mini is a full fine-tune distilled from basal-2.0-max. |
|
|
| ## Limitations |
|
|
| - Validate on your own documents before relying on it: hidden and held-out sets do not cover every domain. |
| - Polish world knowledge of a small model is limited: put the relevant facts in the state. Legal rules change. |
| - Exact arithmetic and thresholds are a weak spot of one-pass decisions: compute them in code (or `"facts": "auto"`). |
| - The calibration is cautious on unseen tasks: no 1% threshold is certified in the shipped files, so `"max_error": 0.01` |
| on an `act` or `tool` question defers every decision; use 0.05, or certify thresholds on your own data with `basal-calibrate`. |
| - `act` and `multi` over many labels are experimental. Up to 255 options per question (Ollama: 20); the context is |
| 32,768 tokens (mini: 8,192). The 2.0 models ship no evidence head. |
| - Decisions with serious consequences for people should be reviewed by a person. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @software{kinas2026basal20, |
| title = {basal-2.0: typed-decision models and inference engine for Polish and English}, |
| author = {Kinas, Remigiusz}, |
| year = {2026}, |
| version = {2.0.0}, |
| url = {https://github.com/rkinas/basal} |
| } |
| ``` |
|
|
| ## License and attribution |
|
|
| Apache-2.0. Fine-tuned from [speakleash/Bielik-1.5B-v3.0-Instruct](https://huggingface.co/speakleash/Bielik-1.5B-v3.0-Instruct) (Apache-2.0). Bielik is a project of SpeakLeash and ACK Cyfronet AGH. |
|
|
