Model card: Ollama build (iapp/OpenThai-SystemOne-Ollama) and Ollama 0.35 evaluation

README.md

@@ -11,6 +11,7 @@ tags:
   - system-one
   - decision-model
   - computer-use
+  - ollama
   - openthaigpt
   - iapp
 ---
@@ -80,6 +81,43 @@ from transformers import AutoModel, AutoTokenizer
 model = AutoModel.from_pretrained("iapp/OpenThai-SystemOne", trust_remote_code=True)
 ```
 
+### Ollama (≥ 0.35)
+
+Ollama 0.35 serves System One models locally at `POST /v1/systemone`, with the same Jev-style contract. OpenThai-SystemOne
+has an **Ollama build**: [`iapp/OpenThai-SystemOne-Ollama`](https://huggingface.co/iapp/OpenThai-SystemOne-Ollama).
+No API key; nothing leaves your machine.
+
+```bash
+ollama pull hf.co/iapp/OpenThai-SystemOne-Ollama:Q8_0
+curl http://localhost:11434/v1/systemone -d '{
+  "model": "hf.co/iapp/OpenThai-SystemOne-Ollama:Q8_0",
+  "state": {"ticket": "ลูกค้าแจ้งว่าโดนหักเงินซ้ำสองครั้ง ขอเงินคืนด่วน โทรมาสามรอบแล้ว"},
+  "questions": {
+    "department": {"type": "choice", "instructions": "ทีมใดควรรับผิดชอบ",
+                   "criteria": {"billing": "การเงิน/ค่าบริการ", "technical": "ระบบใช้งานไม่ได้", "sales": null}},
+    "frustration": {"type": "score", "instructions": "ลูกค้าหงุดหงิดแค่ไหน", "criteria": ["ใจเย็น", "หงุดหงิดแต่สุภาพ", "โกรธมาก"]},
+    "refund_requested": {"type": "noul", "instructions": "ลูกค้าขอเงินคืนอย่างชัดเจนหรือไม่"}
+  }
+}'
+```
+
+Answer (Q8_0, probabilities rounded):
+
+```json
+{"model": "hf.co/iapp/OpenThai-SystemOne-Ollama:Q8_0",
+ "answers": {
+   "department": {"type": "choice", "choice": "billing", "probabilities": {"billing": 0.9515, "technical": 0.0334, "sales": 0.0151}, "confidence": 0.7959},
+   "frustration": {"type": "score", "score": 1.8771, "legend": {"0": "ใจเย็น", "1": "หงุดหงิดแต่สุภาพ", "2": "โกรธมาก"},
+                   "probabilities": {"0": 0.0216, "1": 0.0797, "2": 0.8987}, "confidence": 0.6537},
+   "refund_requested": {"type": "noul", "noul": 0.9805}},
+ "usage": {"input_tokens": 741, "output_tokens": 4}}
+```
+
+Ollama does not run this repo's 256-slot head: it asks each question as a chat prompt and reads the probabilities of the
+answer letters A–Z. The Ollama build is therefore **v0.3 fine-tuned for Ollama's prompt** (same training data; accuracy
+through Ollama equals this repo's API, see [Evaluation](#through-ollama-up-to-26-options-per-question)). Through Ollama a
+question takes 2–26 options (this repo's API: 255), and there is no order-invariant mode and no abstain answer.
+
 ## Demos
 
 ### Thai showcase (real v0.3 outputs via the API)
@@ -202,6 +240,29 @@ summary *relevance* scoring, which is the one subset where our score head is bad
 Batch-1 latency, one question with 255 options, H100 shared with a training job: **44 ms** (public bench run),
 48 ms (held-out run). A 3-question Thai ticket (166 tokens): ~40 ms on H100, 154 ms on a MacBook M3 Max (MPS).
 
+### Through Ollama (up to 26 options per question)
+
+All models below were run by us through Ollama 0.35's `/v1/systemone` on the same records: the first 800 of each set,
+keeping only records whose questions have ≤ 26 options, since Ollama rejects more. That drops banking77 and the
+60-way MASSIVE-th intent questions. "This repo's API" is the v0.3 weights with the slot head on the same records.
+
+| model | size | public 13 (macro) | Thai 8 sets / 12 rows (macro) | latency, 1 / 3 questions |
+|---|---|---|---|---|
+| OpenThai-SystemOne v0.3, this repo's API | 0.8B | 74.3 | 82.2 | – |
+| **OpenThai-SystemOne v0.3, Ollama build Q8_0** | 0.8B | **74.3** | **82.5** | 23 / 136 ms |
+| Ollama build Q4_K_M | 0.8B | 74.0 | 81.6 | 23 / 127 ms |
+| Tev1 0.8B (Together AI, `tev1:0.8b`) | 0.8B | 64.3 | 62.7 | 22 / 102 ms |
+| Tev1 4B (`tev1:4b`) | 4B | 74.9 | 75.4 | 66 / 406 ms |
+| Nimble 9B (Bespoke Labs, `nimble`) | 9B | 74.8 | 78.3 | 69 / 392 ms |
+
+Latency: median end to end through Ollama, one model loaded, idle H100, Thai requests. Ollama runs one prompt per question.
+The Ollama build ties this repo's API. It leads the other 0.8B model by 10 points (public) and 20 points (Thai), and on Thai
+it also leads the 4B and 9B models (82.5 vs 75.4 / 78.3). On English the 4B and 9B are within a point of it. They are
+ahead on reading tasks (BoolQ, PubMedQA, VitaminC, SummEval-relevance) and on the Thai contrastive pairs. The Ollama build
+is ahead on Thai topic, rating and NLI (Prachathai 97.8 vs 61.5, Wongnai 64.0 vs 52.1, XNLI-th 79.2 vs 76.0 for Nimble), on
+PAWS and SQuAD2. Per-set tables:
+[iapp/OpenThai-SystemOne-Ollama](https://huggingface.co/iapp/OpenThai-SystemOne-Ollama).
+
 ### Calibration (Stage 3)
 
 v0.3 learned temperatures: choice 1.055, noul 1.047, score 1.008 (the before/after table below was measured on v0.1;
@@ -243,17 +304,27 @@ Thai sets and on NLI/topic tasks, and route low-confidence English yes/no decisi
   scoring (42) — the two 5-level rating tasks; fine-grained 77-way English intents (banking77 45 single-order); Thai social
   sentiment (wisesight 52); PubMedQA 3-way (64).
 * English calibration is weaker than Thai (median ECE 0.15 vs ≤ 0.05): the training mix is Thai-heavy by design.
+* Ollama build: ≤ 26 options per question (Ollama's limit), no order-invariant mode, no abstain; use this repo's API for more.
 
 ## Versions
 
 | version | date | change | public macro | Wisesight |
 |---|---|---|---|---|
+| v0.3 Ollama build | 2026-09-30 | v0.3 fine-tuned for Ollama 0.35's `/v1/systemone` prompt; GGUF Q8_0 / Q4_K_M / BF16 in [iapp/OpenThai-SystemOne-Ollama](https://huggingface.co/iapp/OpenThai-SystemOne-Ollama) | 74.3 (≤ 26 options) | 49.9 (first 800) |
 | v0.3 | 2026-09-22 | +5,000 SFT steps from v0.2 with 177k real train-split records + 78k targeted synthetic records for the weak spots (grounded QA, summary rating, fine-grained intents, safety, paraphrase), re-calibrated | **74.3** | 51.6 |
 | v0.2 | 2026-09-21 | +3,000 SFT steps from v0.1 with a 22k-record synthetic Thai social-sentiment set (4/3/5-class, yes/no, score schemes), re-calibrated | **63.2** | 51.5 |
 | v0.1 | 2026-09-20 | initial release: Thai CPT 4.47B tokens, 12k-step SFT, calibration | 61.9 | 38.7 |
 
 ## Changelog
 
+**v0.3 Ollama build — 2026-09-30**
+- New repo [iapp/OpenThai-SystemOne-Ollama](https://huggingface.co/iapp/OpenThai-SystemOne-Ollama) for Ollama ≥ 0.35's System One
+  API: `ollama pull hf.co/iapp/OpenThai-SystemOne-Ollama:Q8_0`, then `POST /v1/systemone`. The weights in this repo are unchanged.
+- How: Ollama renders each question as a Qwen3.5 chat prompt and scores the answer letters, so v0.3 was fine-tuned for
+  3,000 steps (192k questions, same data mix as v0.3) on prompts built by a byte-exact port of Ollama's code, tokenized by
+  llama.cpp as Ollama does, with cross-entropy over the candidate letters and one fitted temperature (0.83) folded into the weights.
+- Through Ollama: public 13 macro 74.3, Thai 82.5 (this repo's API on the same records: 74.3 / 82.2); Q4_K_M 74.0 / 81.6.
+
 **v0.3 — 2026-09-22**
 - Continued fine-tuning for 5,000 steps from v0.2 with the weak-spot data: real train splits (SQuAD2, BoolQ, PubMedQA-artificial, PAWS, Aegis2, ToxicChat, XQuAD-th, MASSIVE-de; 177k records) and five targeted synthetic sets generated with Qwen3.6-35B-A3B and blind-checked (grounded yes/no QA with near-miss unanswerables 30k, summary rating on the SummEval rubrics 20k, 40–120-way near-duplicate intent taxonomies 12k, safety judgments 8k, adversarial paraphrases 8k); sentiment set weight lowered from 4× to 2×; re-calibrated.
 - Public 13-subset macro 63.2 → **74.3** (Nimble-9B 74.8): SQuAD2 50.2 → 89.3, PAWS 68.0 → 94.0, Aegis2 61.6 → 83.2, MASSIVE-de 67.4 → 88.3, BoolQ 64.7 → 79.7, MASSIVE-en 79.1 → 88.3, PubMedQA 56.4 → 64.0, SummEval-relevance 14.2 → 21.7, VitaminC 68.6 → 72.5, MultiNLI 87.3 → 89.0. ECE improved on 11 of 13 subsets.