docs: 1.0.1, results on the published jerzyk-bench with basal and confidence intervals

README.md

@@ -36,7 +36,7 @@ model-index:
       split: test
     metrics:
     - type: accuracy
-      value: 85.8
+      value: 85.5
       name: Mean accuracy over domains, real documents
 ---
 
@@ -78,40 +78,39 @@ The model itself reads Polish questions and answer options (it was trained on th
 
 | Question | **Jerzyk 1.0** (322M) | basal-2.0-mini (1.5B) | Qwen3 8B | Bielik 11B v3 |
 |---|---|---|---|---|
-| Administrative: urgency | 77.5 | 61.7 | 42.5 | 54.2 |
-| Administrative: document_type | 90.0 | 92.5 | 90.8 | 95.0 |
-| Administrative: deadline | 82.5 | 79.2 | 84.2 | 83.3 |
-| Customer e-mail: department | 93.3 | 88.3 | 89.2 | 88.3 |
-| Customer e-mail: urgency | 79.2 | 74.2 | 70.8 | 80.8 |
+| Administrative: urgency | 77.5 | 61.7 | 41.7 | 54.2 |
+| Administrative: document_type | 90.0 | 93.3 | 90.8 | 95.0 |
+| Administrative: deadline | 82.5 | 79.2 | 83.3 | 83.3 |
+| Customer e-mail: department | 93.3 | 88.3 | 89.2 | 89.2 |
+| Customer e-mail: urgency | 79.2 | 75.0 | 70.0 | 80.0 |
 | Customer e-mail: cancellation | 99.2 | 100.0 | 97.5 | 96.7 |
-| Customer e-mail: upset | 91.7 | 90.8 | 77.5 | 85.0 |
-| Accounting: cost_category | 97.5 | 95.0 | 85.0 | 75.8 |
-| Accounting: urgency | 64.2 | 35.8 | 48.3 | 63.3 |
+| Customer e-mail: upset | 91.7 | 90.8 | 77.5 | 85.8 |
+| Accounting: cost_category | 97.5 | 95.0 | 85.8 | 75.8 |
+| Accounting: urgency | 64.2 | 35.8 | 45.8 | 62.5 |
 | Accounting: document_type | 94.2 | 91.7 | 95.8 | 95.8 |
-| Accounting: deadline | 85.8 | 95.8 | 94.2 | 77.5 |
-| Court: urgency | 80.0 | 42.5 | 38.3 | 54.2 |
+| Accounting: deadline | 85.8 | 95.8 | 93.3 | 76.7 |
+| Court: urgency | 80.8 | 42.5 | 40.0 | 54.2 |
 | Court: document_type | 84.2 | 96.7 | 94.2 | 92.5 |
-| Court: deadline | 93.3 | 85.0 | 83.3 | 82.5 |
-| **Mean over domains** | **86.4** | **80.1** | **77.3** | **79.9** |
+| Court: deadline | 93.3 | 84.2 | 82.5 | 82.5 |
+| **Mean over domains** | **86.4** | **80.2** | **77.0** | **79.9** |
 
 **Real documents (89 court rulings + 56 administrative notices)** — accuracy (%), 95% confidence interval in brackets
 
 | Question | **Jerzyk 1.0** (322M) | basal-2.0-mini (1.5B) | Qwen3 8B | Bielik 11B v3 |
 |---|---|---|---|---|
-| Admin. notices (real): urgency | 58.9 (46–71) | 37.5 (26–51) | 48.2 (36–61) | 32.1 (21–45) |
-| Admin. notices (real): document_type | 85.7 (74–93) | 76.8 (64–86) | 83.9 (72–91) | 85.7 (74–93) |
-| Admin. notices (real): deadline | 73.2 (60–83) | 76.8 (64–86) | 62.5 (49–74) | 82.1 (70–90) |
-| Court rulings (real): document_type | 98.9 (94–100) | 93.3 (86–97) | 100.0 (96–100) | 100.0 (96–100) |
-| **Mean over domains** | **85.8** | **78.5** | **82.4** | **83.3** |
+| Admin. notices (real): urgency | 58.9 (46–71) | 32.1 (21–45) | 50.0 (37–63) | 30.4 (20–43) |
+| Admin. notices (real): document_type | 85.7 (74–93) | 78.6 (66–87) | 83.9 (72–91) | 85.7 (74–93) |
+| Admin. notices (real): deadline | 71.4 (58–82) | 75.0 (62–84) | 64.3 (51–76) | 78.6 (66–87) |
+| Court rulings (real): document_type | 98.9 (94–100) | 94.4 (88–98) | 100.0 (96–100) | 100.0 (96–100) |
+| **Mean over domains** | **85.5** | **78.1** | **83.0** | **82.5** |
 
 | | **Jerzyk 1.0** (322M) | basal-2.0-mini (1.5B) | Qwen3 8B | Bielik 11B v3 |
 |---|---|---|---|---|
-| ECE, real documents (lower = better) | 0.104 | 0.127 | 0.197 | 0.142 |
-| Median latency per document (all models on the same computer) | 0.05 s | 0.28 s | 1.75 s | 3.01 s |
+| ECE, real documents (lower = better) | 0.106 | 0.112 | 0.191 | 0.154 |
+| Median latency per document (NVIDIA GeForce RTX 5080 Laptop GPU) | 0.02 s | 1.13 s (Apple M1, MLX) | 0.24 s | 0.41 s |
 
 Models compared: [Jerzyk 1.0](https://huggingface.co/trenify/jerzyk-1.0) (calibrated); [basal-2.0-mini](https://huggingface.co/Remek/basal-2.0-mini-MLX-8bit) (MLX 8-bit, native `/v1/systemone`); [Qwen3 8B](https://huggingface.co/Qwen/Qwen3-8B-GGUF) and [Bielik 11B v3.0 Instruct](https://huggingface.co/speakleash/Bielik-11B-v3.0-Instruct-GGUF) (GGUF Q4_K_M in llama.cpp, zero-shot, answer chosen from option log-probabilities). *Mean over domains* averages the questions of each domain first, then the domains, so every domain counts equally.
 
-> **Note.** These results were measured on the benchmark text before anonymisation of the real documents (names of judges, officials and private persons replaced with placeholders such as `[SĘDZIA]` and `[OSOBA]`). On the published, anonymised text Jerzyk 1.0 scores 85.5% instead of 85.8% on real documents. The other models will be re-measured on the published text in a later update.
 
 Calibration: one temperature per question type, fitted on the dev split (`calibration.json`: choice 2.6, yes/no 1.75, score 3.5).
 
@@ -140,7 +139,7 @@ Weights are in `safetensors` only; there are no pickle files.
 ## Limitations
 
 - Polish only. Input is capped at 1024 tokens (about 3,000 characters).
-- Deadlines in long real administrative notices are detected in about 73% of cases (Bielik 11B: 82%). Use the rule-based `jerzyk.deadlines` alongside the model and send low-confidence cases to a person.
+- Deadlines in long real administrative notices are detected in about 71% of cases (Bielik 11B: 79%). Use the rule-based `jerzyk.deadlines` alongside the model and send low-confidence cases to a person.
 - Urgency on real administrative notices is the weakest result: 58.9% (95% CI 46–71) on 56 documents.
 - How to read the evaluation: jerzyk-bench was built by the authors of Jerzyk. The generated court, administrative and accounting test documents were written by Bielik 11B v3, which is also one of the compared models. The labels of the 56 real notices were made by Claude and checked by two blind re-labellings, not by a human annotator.
 - Urgency follows the conventions in jerzyk-bench `SPEC.md` (7/14-day thresholds), which may differ from your organisation's.