Knack v3 as an Ollama decision model (--ask, /v1/systemone)

README.md

@@ -73,43 +73,90 @@ ollama pull devshub/knack:q4_k_m       # 516 MB, near-identical accuracy
 ollama pull hf.co/devshub-ai/knack:Q8_0   # same model, straight from this repo
 ```
 
-Knack is a classifier, not a chat model. A request is a **state** (one `field: value` line
-per field) followed by one **question id**. The answer is the next token: a letter `A`, `B`,
-`C`... that picks the question's option in the order listed below.
+Then use it like any Ollama decision model:
 
-```bash
-curl -s localhost:11434/api/generate -d '{
-  "model": "devshub/knack", "raw": true, "stream": false,
-  "logprobs": true, "top_logprobs": 20, "options": {"temperature": 0, "num_predict": 1},
-  "prompt": "<|im_start|>user\ncommand: tail -f app.log\ndescription: <|im_end|>\n<|im_start|>assistant\nlong_running:"
+```
+echo "npm run dev" | ollama run devshub/knack --ask "Will it keep running?"
+```
+
+```
+yes  97%
+```
+
+Ask by question id or by its plain phrasing (both listed below). Knack reads the text as a
+shell command for the command questions and as a user message for the turn questions:
+
+```
+echo "git push --force" | ollama run devshub/knack --ask risk --level safe --level "needs care" --level destructive
+echo "fix the failing login test" | ollama run devshub/knack --ask "Is this a new task?"
+echo "fix the failing login test" | ollama run devshub/knack --ask task.intent \
+  --choice bug_fix --choice feature --choice refactor --choice api_schema --choice config_deploy \
+  --choice test --choice documentation --choice investigation --choice unknown
+```
+
+Knack only knows its 29 questions. Anything else ("Is a refund being requested?") is
+rejected with the list of what it can answer, rather than guessed. A question asked with
+the wrong options gets back the exact flags to use.
+
+### cURL
+
+```
+curl http://localhost:11434/v1/systemone \
+  -H "Content-Type: application/json" \
+  -d '{
+  "model": "devshub/knack",
+  "state": "npm run dev",
+  "questions": {
+    "long_running": {"type": "noul", "instructions": "Will it keep running?"},
+    "serves": {"type": "noul", "instructions": "Does it start a server?"}
+  }
 }'
 ```
 
-**Read the answer from `logprobs`, not from `response`.** A question with N options uses
-only the first N letters. Take the probabilities of those letters, renormalise them, and
-pick the highest (for `long_running` above: `A` = yes, `B` = no). The generated letter on
-its own can be an unused letter. `knack.py` does this for you.
+```
+{"model":"devshub/knack","answers":{"long_running":{"type":"noul","noul":0.9708},"serves":{"type":"noul","noul":0.9701}}}
+```
+
+The state can also be an object with Knack's own fields, for example
+`{"command": "npm run dev", "description": "start the app"}`.
+
+### Python
+
+```
+pip install ollama
+```
+
+```python
+import ollama
+
+response = ollama.systemone(
+    model="devshub/knack",
+    state="npm run dev",
+    questions={"long_running": {"type": "noul", "instructions": "Will it keep running?"}},
+)
+print(response.answers["long_running"].noul)  # 0.9708
+```
 
 ## What it decides
 
 **Every shell command** (state fields: `command`, `description`):
 
-| id | options (A, B, C...) |
-|---|---|
-| `risk` | 0 safe, 1 needs care, 2 destructive |
-| `long_running` | yes, no |
-| `activity` | testing, building, installing, git, inspecting, running |
-| `verifies` | yes, no |
-| `mutates_repo` | yes, no |
-| `deploys` | yes, no |
-| `runtime_check` | yes, no |
-| `serves` | yes, no |
+| id | options | plain question |
+|---|---|---|
+| `risk` | 0 safe, 1 needs care, 2 destructive | How risky is this command? |
+| `long_running` | yes, no | Will it keep running? |
+| `activity` | testing, building, installing, git, inspecting, running | What kind of command is this? |
+| `verifies` | yes, no | Does it verify the work? |
+| `mutates_repo` | yes, no | Does it change the repo? |
+| `deploys` | yes, no | Does it deploy? |
+| `runtime_check` | yes, no | Does it need a running app? |
+| `serves` | yes, no | Does it start a server? |
 
 **Every user turn** (state fields, in this order: `prompt`, `current_mode`,
 `recent_activity`, `conversation_size`, `open_task`, `turns_on_open_task`,
 `interrupted_work`, `browser_context`; see `turn_state()` in `knack.py` for defaults):
 
-| id | options (A, B, C...) |
+| id | options |
 |---|---|
 | `turn.chat_or_task` | chat, task |
 | `turn.actionable` / `turn.question_shaped` / `turn.new_task` / `turn.resume` / `turn.mission` / `turn.plan_authoring` / `turn.demo` / `turn.skip_orientation` | yes, no |
@@ -123,8 +170,8 @@ its own can be an unused letter. `knack.py` does this for you.
 | `route.risk` | low, medium, high |
 | `browser` / `messaging` / `delivery.feature_task` / `delivery.slash_task` | yes, no |
 
-Knack only knows these 29 questions. Inside CodeSync, anything else goes to CodeSync's cloud
-judgment service, so a new kind of question never gets a guess.
+Knack only knows these 29 questions; Ollama rejects any other one instead of guessing.
+Inside CodeSync, anything else goes to CodeSync's cloud judgment service.
 
 ## Results
 
@@ -135,10 +182,9 @@ Measured on CodeSync's own judgments (version v3, November 2026):
 | Hand-labelled set | **97.9%** | 97.9% | 97.4% | 233 decisions labelled by hand |
 | Held-out set | **95.5%** | 95.7% | 95.7% | 3,267 decisions from real sessions, never trained on |
 | Rare-case set | **93.7%** | 93.6% | 93.6% | 4,120 decisions on the hardest, rarest cases |
-| Latency | **13 ms** | ~80 ms | ~80 ms | median per command (8 questions) on a laptop GPU |
+| Latency | **13 ms** | 17 ms | ~17 ms | median per command (8 questions) on a laptop GPU |
 
-The Ollama files run one request per question; CodeSync's built-in runtime answers all
-questions of a turn in a single forward pass, which is why it is faster.
+Latency is the median for all 8 command questions in one `/v1/systemone` request.
 
 ## Use it inside CodeSync
 
@@ -156,13 +202,12 @@ picked up automatically. To turn it off: `CODESYNC_KNACK_DISABLED=1`.
 
 | File | Purpose |
 |---|---|
-| `knack-v3-Q8_0.gguf`, `knack-v3-Q4_K_M.gguf` | Ollama / llama.cpp edition |
+| `knack-v3-Q8_0.gguf`, `knack-v3-Q4_K_M.gguf` | Ollama / llama.cpp decision model (`--ask`, `/v1/systemone`) |
 | `knack.py` | standalone runner (CLI and local server), standard library only |
-| `template`, `params` | Ollama prompt template and parameters (raw prompt, 1 token, temperature 0) |
 | `manifest.json`, `knack-<version>.tar.gz` | the PyTorch model folder CodeSync installs |
 
-The GGUF edition is the same weights with an untied output head: the rows of the answer
-letters carry a shared offset, so the letters always lead the vocabulary while their relative
-probabilities, and so every answer, stay the same.
+The GGUF edition is the same weights packaged as an Ollama decision model: a `systemone`
+prompt template rebuilds Knack's training prompt from the request, and the answer is read
+from the option letters exactly as CodeSync's own runtime reads it.
 
 Built by [DevsHub](https://app.devshub.ai) for [CodeSync](https://app.devshub.ai).

knack-v3-Q4_K_M.gguf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c55e4831fc36f7adf5bd90573bb3c2c81c0409451ddebd2e13bdce9f49ea1095
-size 515961664
+oid sha256:efa47599cece417c1e66daa86b876e7665fa6d2e28a3e5c79a2425ae64dd5e23
+size 515970400

knack-v3-Q8_0.gguf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2eed77e11930251a46eed2bf1cf6401e8fe499c2a953bca5e8fdaa9bc4eeeb6b
-size 816653120
+oid sha256:18aaebf8083a4b123495e373131bf3fe858599c95421b412bdcc1db38d16f753
+size 816661856

knack.py

@@ -15,11 +15,9 @@ Usage:
     python knack.py --turn "add dark mode" --mode plan --json
     python knack.py serve --port 33727              # CodeSync-compatible POST /v1/systemone
 
-How Knack answers: the prompt is a state (``field: value`` lines) followed by one question
-id, and the answer is the next token, a letter ``A``, ``B``, ``C``... picking the question's
-option in order. Only the first N letters belong to a question with N options, so the
-answer is the most likely of THOSE letters, renormalised (this script does that from
-Ollama's logprobs). Do not read the raw generated letter on its own.
+Knack is an Ollama decision model: this script sends all questions of a command or turn in
+one ``POST /v1/systemone`` request (needs an Ollama with decision-model support). Without the
+script: ``echo "npm run dev" | ollama run devshub/knack --ask long_running``.
 
 Env: KNACK_OLLAMA_HOST (default http://127.0.0.1:11434),
      KNACK_MODEL (default devshub/knack; devshub/knack:q4_k_m is smaller, 516 MB;
@@ -30,20 +28,15 @@ from __future__ import annotations
 
 import argparse
 import json
-import math
 import os
 import sys
 import urllib.error
 import urllib.request
-from concurrent.futures import ThreadPoolExecutor
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 from typing import Any
 
 HOST = os.environ.get("KNACK_OLLAMA_HOST", "http://127.0.0.1:11434").rstrip("/")
 MODEL = os.environ.get("KNACK_MODEL", "devshub/knack")
-PREFIX = "<|im_start|>user\n"
-SUFFIX = "<|im_end|>\n<|im_start|>assistant\n"
-LETTERS = [chr(65 + i) for i in range(26)]
 _READY: set[str] = set()
 
 # question id -> (type, option keys in letter order, what it decides)
@@ -110,19 +103,25 @@ def _post(path: str, body: dict[str, Any], timeout: float = 60.0) -> dict[str, A
         return json.loads(r.read())
 
 
-def ask_one(state: dict[str, Any], qid: str, model: str = MODEL) -> dict[str, Any]:
-    """{answer, confidence, probabilities} for one question."""
+def _spec(qid: str) -> dict[str, Any]:
+    """System One question for /v1/systemone (the id is what Knack matches on)."""
     kind, keys, _ = QUESTIONS[qid]
-    out = _post("/api/generate", {
-        "model": model, "prompt": PREFIX + render_state(state) + SUFFIX + qid + ":",
-        "raw": True, "stream": False, "logprobs": True, "top_logprobs": 20,
-        "options": {"temperature": 0, "num_predict": 1}})
-    letters = LETTERS[: len(keys)]
-    logp = {e["token"].strip(): e["logprob"]
-            for e in (out.get("logprobs") or [{}])[0].get("top_logprobs", [])}
-    raw = {k: math.exp(logp[c]) if c in logp else 0.0 for k, c in zip(keys, letters)}
-    total = sum(raw.values()) or 1.0
-    probs = {k: v / total for k, v in raw.items()}
+    if kind == "noul":
+        return {"type": "noul", "instructions": qid}
+    if kind == "score":
+        return {"type": "score", "instructions": qid, "criteria": keys}
+    return {"type": "choice", "instructions": qid, "criteria": {k: None for k in keys}}
+
+
+def _answer(qid: str, a: dict[str, Any]) -> dict[str, Any]:
+    """{answer, confidence, probabilities, type} from one /v1/systemone answer."""
+    kind, keys, _ = QUESTIONS[qid]
+    if kind == "noul":
+        yes = float(a["noul"])
+        probs = {"yes": yes, "no": 1.0 - yes}
+    else:
+        raw = a.get("probabilities") or {}
+        probs = {k: float(raw.get(k, raw.get(str(i), 0.0))) for i, k in enumerate(keys)}
     best = max(probs, key=probs.__getitem__)
     return {"answer": best, "confidence": probs[best], "probabilities": probs, "type": kind}
 
@@ -147,8 +146,15 @@ def ask(state: dict[str, Any], qids: list[str], model: str = MODEL) -> dict[str,
     if unknown:
         raise KeyError(f"Knack was not trained on: {unknown}")
     ensure_model(model)
-    with ThreadPoolExecutor(max_workers=8) as pool:
-        return dict(zip(qids, pool.map(lambda q: ask_one(state, q, model), qids)))
+    try:
+        out = _post("/v1/systemone", {"model": model, "state": state,
+                                      "questions": {q: _spec(q) for q in qids}})
+    except urllib.error.HTTPError as exc:
+        detail = exc.read().decode(errors="replace")[:300]
+        raise RuntimeError(f"Ollama /v1/systemone: HTTP {exc.code} {detail} "
+                           "(update Ollama and re-pull the model if this is an old build)") from None
+    answers = out.get("answers") or {}
+    return {q: _answer(q, answers[q]) for q in qids if q in answers}
 
 
 def to_systemone(qid: str, a: dict[str, Any]) -> dict[str, Any]:
@@ -213,8 +219,11 @@ def main(argv: list[str] | None = None) -> int:
     qids = a.question or (TURN if a.turn else BASH)
     try:
         res = ask(state, qids, a.model)
+    except RuntimeError as exc:
+        print(exc, file=sys.stderr)
+        return 1
     except OSError as exc:
-        print(f"cannot reach Ollama at {HOST} ({exc}). Run: ollama pull {a.model}", file=sys.stderr)
+        print(f"cannot reach Ollama at {HOST} ({exc}). Is Ollama running?", file=sys.stderr)
         return 1
     if a.json:
         print(json.dumps(res, indent=1))