Docs: specialized decision model, ask by id, precise meanings, known weak spots

README.md

@@ -12,12 +12,12 @@ inference: false
 
 # CodeSync Knack
 
-**The free, private judgment model behind CodeSync.**
+**A specialized decision model: the free, private judgment model behind CodeSync.**
 
 Every time you send CodeSync a message or it runs a shell command, it makes dozens of quick
 calls: Is this command risky? Will it run forever? Are you asking to continue earlier work?
 Is this a question or a task? Which mode fits? Knack makes those calls on your own machine,
-in about 10 milliseconds.
+in about 10 milliseconds. It answers only those decisions, asked by question id.
 
 - **Free.** Ships with every CodeSync install. No API key, no account, no usage limits.
 - **Private.** Runs locally. Your prompts and commands never leave your machine to be judged.
@@ -27,45 +27,7 @@ in about 10 milliseconds.
   rare cases that cost the most when missed: destructive deletes, commands that never exit,
   resuming interrupted work, refactor / test / documentation requests.
 
-## Try it in one line
-
-You need [Ollama](https://ollama.com/download) running. Nothing else to install: the script
-uses only the Python standard library and pulls the model on first run.
-
-```bash
-curl -fsSL https://huggingface.co/devshub-ai/knack/resolve/main/knack.py | python3 - "npm run dev"
-```
-
-```
-risk           1             0.95   0 safe, 1 needs care, 2 destructive / dangerous
-long_running   yes           0.97   keeps running (server, watch, poll, tail -f)
-activity       running       0.92   what kind of command it is
-verifies       no            0.97   checks that work is correct (tests, lint, build)
-mutates_repo   no            0.97   changes files or git state of the repo
-deploys        no            0.94   deploys or publishes
-runtime_check  no            0.95   needs a running app to check
-serves         yes           0.97   starts a server
-```
-
-Knack only reads the command you pass and judges it. Nothing is executed, so it is safe
-to ask about any command, including destructive ones.
-
-Judge a user message instead of a command:
-
-```bash
-curl -fsSL https://huggingface.co/devshub-ai/knack/resolve/main/knack.py | python3 - --turn "fix the failing login test"
-```
-
-Or save the script and use it as a tool or a local server:
-
-```bash
-curl -fsSLO https://huggingface.co/devshub-ai/knack/resolve/main/knack.py
-python3 knack.py "docker compose up" --json          # machine-readable
-python3 knack.py --turn "add dark mode" -q route.mode -q task.intent
-python3 knack.py serve --port 33727                    # POST /v1/systemone (CodeSync format)
-```
-
-## Use it with Ollama directly
+## Get it
 
 ```bash
 ollama pull devshub/knack              # Q8_0, 817 MB  (https://ollama.com/devshub/knack)
@@ -73,30 +35,54 @@ ollama pull devshub/knack:q4_k_m       # 516 MB, near-identical accuracy
 ollama pull hf.co/devshub-ai/knack:Q8_0   # same model, straight from this repo
 ```
 
-Then use it like any Ollama decision model:
+## A specialized decision model
+
+General decision models, like clef, read any question you write in plain language. Knack is
+a different kind: a **specialized decision model**. It was trained on one product's fixed
+decisions, the 29 that CodeSync makes on every shell command and every user message, and it
+answers only those. It does not read question text: you ask by **question id**. In exchange
+it is small, private, and answers all 8 command questions in about 17 ms.
+
+| | General decision model (clef) | Specialized decision model (Knack) |
+| --- | --- | --- |
+| Questions | anything you write | 29 fixed ones, asked by id |
+| Input | any text or image | a shell command, or a message to a coding assistant |
+| Size | 27B | 0.75B |
+| A question it was not trained on | answers it | rejects it |
+
+## Try it in one line
 
 ```
-echo "npm run dev" | ollama run devshub/knack --ask "Will it keep running?"
+echo "npm run dev" | ollama run devshub/knack --ask serves
 ```
 
 ```
 yes  97%
 ```
 
-Ask by question id or by its plain phrasing (both listed below). Knack reads the text as a
-shell command for the command questions and as a user message for the turn questions:
+`serves` asks whether the command keeps running until someone stops it (a dev server, a
+watcher, `tail -f`). Each id has one precise meaning, listed in the tables below. Read them
+before relying on an answer: `long_running`, for example, means "takes minutes", so a build
+counts.
+
+Knack only reads the text and judges it. Nothing is executed, so it is safe to ask about any
+command, including destructive ones.
+
+More examples:
 
 ```
 echo "git push --force" | ollama run devshub/knack --ask risk --level safe --level "needs care" --level destructive
-echo "fix the failing login test" | ollama run devshub/knack --ask "Is this a new task?"
+echo "fix the failing login test" | ollama run devshub/knack --ask turn.new_task
 echo "fix the failing login test" | ollama run devshub/knack --ask task.intent \
   --choice bug_fix --choice feature --choice refactor --choice api_schema --choice config_deploy \
   --choice test --choice documentation --choice investigation --choice unknown
 ```
 
-Knack only knows its 29 questions. Anything else ("Is a refund being requested?") is
-rejected with the list of what it can answer, rather than guessed. A question asked with
-the wrong options gets back the exact flags to use.
+Ask with an id from the tables. Any other question is rejected rather than guessed. A choice
+or score question needs its options in the order shown; if they do not match, the error
+shows the exact flags to use.
+
+## API
 
 ### cURL
 
@@ -107,18 +93,18 @@ curl http://localhost:11434/v1/systemone \
   "model": "devshub/knack",
   "state": "npm run dev",
   "questions": {
-    "long_running": {"type": "noul", "instructions": "Will it keep running?"},
-    "serves": {"type": "noul", "instructions": "Does it start a server?"}
+    "serves": {"type": "noul", "instructions": "serves"},
+    "long_running": {"type": "noul", "instructions": "long_running"}
   }
 }'
 ```
 
 ```
-{"model":"devshub/knack","answers":{"long_running":{"type":"noul","noul":0.9708},"serves":{"type":"noul","noul":0.9701}}}
+{"model":"devshub/knack","answers":{"serves":{"type":"noul","noul":0.9701},"long_running":{"type":"noul","noul":0.9708}}}
 ```
 
-The state can also be an object with Knack's own fields, for example
-`{"command": "npm run dev", "description": "start the app"}`.
+The question id goes in `instructions` (or as the key). The state can also be an object
+with Knack's own fields, for example `{"command": "npm run dev", "description": "start the app"}`.
 
 ### Python
 
@@ -132,46 +118,102 @@ import ollama
 response = ollama.systemone(
     model="devshub/knack",
     state="npm run dev",
-    questions={"long_running": {"type": "noul", "instructions": "Will it keep running?"}},
+    questions={"serves": {"type": "noul", "instructions": "serves"}},
 )
-print(response.answers["long_running"].noul)  # 0.9708
+print(response.answers["serves"].noul)  # 0.9701
 ```
 
-## What it decides
+## Or use the standalone script
+
+`knack.py` asks every question of a command or message at once and prints a table. It uses
+only the Python standard library and pulls the model on first run.
+
+```
+curl -fsSL https://huggingface.co/devshub-ai/knack/resolve/main/knack.py | python3 - "npm run dev"
+```
 
-**Every shell command** (state fields: `command`, `description`):
+```
+risk           1             0.95   0 read-only, 1 recoverable change, 2 irreversible / remote
+long_running   yes           0.97   takes minutes, blocks, polls, watches or serves
+activity       running       0.92   what kind of work it does
+verifies       no            0.97   a test, type or lint check whose pass proves the change works
+mutates_repo   no            0.97   changes git state (commit, push, merge, rebase, tag)
+deploys        no            0.94   publishes, releases or changes a remote environment
+runtime_check  no            0.95   a health probe of a running service that fails if unhealthy
+serves         yes           0.97   keeps running until stopped (server, watcher, tail -f)
+```
 
-| id | options | plain question |
-|---|---|---|
-| `risk` | 0 safe, 1 needs care, 2 destructive | How risky is this command? |
-| `long_running` | yes, no | Will it keep running? |
-| `activity` | testing, building, installing, git, inspecting, running | What kind of command is this? |
-| `verifies` | yes, no | Does it verify the work? |
-| `mutates_repo` | yes, no | Does it change the repo? |
-| `deploys` | yes, no | Does it deploy? |
-| `runtime_check` | yes, no | Does it need a running app? |
-| `serves` | yes, no | Does it start a server? |
+```
+curl -fsSL https://huggingface.co/devshub-ai/knack/resolve/main/knack.py | python3 - --turn "fix the failing login test"
+```
 
-**Every user turn** (state fields, in this order: `prompt`, `current_mode`,
-`recent_activity`, `conversation_size`, `open_task`, `turns_on_open_task`,
-`interrupted_work`, `browser_context`; see `turn_state()` in `knack.py` for defaults):
+## What it decides
 
-| id | options |
-|---|---|
-| `turn.chat_or_task` | chat, task |
-| `turn.actionable` / `turn.question_shaped` / `turn.new_task` / `turn.resume` / `turn.mission` / `turn.plan_authoring` / `turn.demo` / `turn.skip_orientation` | yes, no |
-| `turn.change_intent` | create, adjust, enhance, fix, refactor, none |
-| `turn.screencast` | start, stop, none |
-| `task.intent` | bug_fix, feature, refactor, api_schema, config_deploy, test, documentation, investigation, unknown |
-| `task.execution_strategy` | direct, plan, parallel |
-| `route.mode` | plan, build, ask, debug |
-| `route.tier` | light, medium, heavy |
-| `route.thinking` | off, low, medium, high, xhigh, max |
-| `route.risk` | low, medium, high |
-| `browser` / `messaging` / `delivery.feature_task` / `delivery.slash_task` | yes, no |
-
-Knack only knows these 29 questions; Ollama rejects any other one instead of guessing.
-Inside CodeSync, anything else goes to CodeSync's cloud judgment service.
+**Every shell command.** Input: the command (state fields `command`, `description`).
+
+| id | yes / the answer means | options |
+| --- | --- | --- |
+| `risk` | the worst it can do: 0 read-only or trivially reversible, 1 changes recoverable local state, 2 irreversible loss or reaches beyond the checkout | 0, 1, 2 |
+| `long_running` | takes minutes, blocks, polls, watches or serves, rather than finishing in seconds. A build or test suite counts | yes, no |
+| `serves` | keeps running until it is stopped: server, daemon, watcher, tunnel, `tail -f`. A build that finishes on its own does not | yes, no |
+| `activity` | what kind of work it does | testing, building, installing, git, inspecting, running |
+| `verifies` | a test run, type or lint check whose passing exit is evidence the change works. Builds and plain runs do not count | yes, no |
+| `mutates_repo` | changes version-control state: commit, push, merge, rebase, tag, amend. Editing or deleting files does not count | yes, no |
+| `deploys` | publishes, releases, or changes a remote or runtime environment (deploy, rollout, push an image, bring a service up) | yes, no |
+| `runtime_check` | a health or smoke probe of a running service that fails on an unhealthy response (`curl --fail`) | yes, no |
+
+**Every user turn.** Input: the message a user sends to a coding assistant (state fields, in
+this order: `prompt`, `current_mode`, `recent_activity`, `conversation_size`, `open_task`,
+`turns_on_open_task`, `interrupted_work`, `browser_context`; plain text fills the rest with
+defaults).
+
+| id | the answer means | options |
+| --- | --- | --- |
+| `turn.chat_or_task` | conversation, or a request for work | chat, task |
+| `turn.actionable` | asks for something to be done | yes, no |
+| `turn.question_shaped` | phrased as a question | yes, no |
+| `turn.change_intent` | kind of code change asked for | create, adjust, enhance, fix, refactor, none |
+| `turn.new_task` | starts new work rather than continuing | yes, no |
+| `turn.resume` | asks to resume interrupted work | yes, no |
+| `turn.mission` | a long, multi-step job | yes, no |
+| `turn.plan_authoring` | asks for a plan to be written | yes, no |
+| `turn.screencast` | start or stop a screen recording | start, stop, none |
+| `turn.demo` | asks for a demo | yes, no |
+| `turn.skip_orientation` | can be done without exploring the codebase | yes, no |
+| `task.intent` | type of task | bug_fix, feature, refactor, api_schema, config_deploy, test, documentation, investigation, unknown |
+| `task.execution_strategy` | how to execute it | direct, plan, parallel |
+| `route.mode` | best assistant mode | plan, build, ask, debug |
+| `route.tier` | size of model needed | light, medium, heavy |
+| `route.thinking` | reasoning effort needed | off, low, medium, high, xhigh, max |
+| `route.risk` | risk of the requested work | low, medium, high |
+| `browser` | needs a web browser | yes, no |
+| `messaging` | needs to send an email or chat message | yes, no |
+| `delivery.feature_task` | a feature delivery task | yes, no |
+| `delivery.slash_task` | a slash-command style task | yes, no |
+
+## What it does not do
+
+- **Other questions.** Anything outside these 29, such as "is this log an error?" or "is a
+  refund requested?", is rejected. Use a general decision model like clef for those.
+- **Other input.** Logs, emails, tickets, documents or images give meaningless answers,
+  even to these 29 questions.
+- **Conversation history.** Through Ollama it sees a single message. Inside CodeSync it also
+  gets the session context, which makes the turn questions more reliable.
+- **Text generation.** It returns one pick with a confidence, never sentences.
+
+## Known weak spots
+
+Measured with the commands above, so you know where to double-check:
+
+| command | question | Knack says | should be |
+| --- | --- | --- | --- |
+| `tail -f app.log` | `serves` | no, 75% | yes |
+| `npm run build` | `serves` | yes, 79% | no |
+| `docker compose up` | `long_running` | 51% yes | yes |
+| `npm test` | `long_running` | 45% yes | unclear, depends on the suite |
+
+These come from the model, not the Ollama packaging (the original PyTorch model gives the
+same numbers), and are the next training targets.
 
 ## Results
 

knack.py

@@ -17,7 +17,7 @@ Usage:
 
 Knack is an Ollama decision model: this script sends all questions of a command or turn in
 one ``POST /v1/systemone`` request (needs an Ollama with decision-model support). Without the
-script: ``echo "npm run dev" | ollama run devshub/knack --ask long_running``.
+script: ``echo "npm run dev" | ollama run devshub/knack --ask serves``.
 
 Env: KNACK_OLLAMA_HOST (default http://127.0.0.1:11434),
      KNACK_MODEL (default devshub/knack; devshub/knack:q4_k_m is smaller, 516 MB;
@@ -42,15 +42,15 @@ _READY: set[str] = set()
 # question id -> (type, option keys in letter order, what it decides)
 QUESTIONS: dict[str, tuple[str, list[str], str]] = {
     # every shell command (state: command, description)
-    "risk": ("score", ["0", "1", "2"], "0 safe, 1 needs care, 2 destructive / dangerous"),
-    "long_running": ("noul", ["yes", "no"], "keeps running (server, watch, poll, tail -f)"),
+    "risk": ("score", ["0", "1", "2"], "0 read-only, 1 recoverable change, 2 irreversible / remote"),
+    "long_running": ("noul", ["yes", "no"], "takes minutes, blocks, polls, watches or serves"),
     "activity": ("choice", ["testing", "building", "installing", "git", "inspecting", "running"],
-                 "what kind of command it is"),
-    "verifies": ("noul", ["yes", "no"], "checks that work is correct (tests, lint, build)"),
-    "mutates_repo": ("noul", ["yes", "no"], "changes files or git state of the repo"),
-    "deploys": ("noul", ["yes", "no"], "deploys or publishes"),
-    "runtime_check": ("noul", ["yes", "no"], "needs a running app to check"),
-    "serves": ("noul", ["yes", "no"], "starts a server"),
+                 "what kind of work it does"),
+    "verifies": ("noul", ["yes", "no"], "a test, type or lint check whose pass proves the change works"),
+    "mutates_repo": ("noul", ["yes", "no"], "changes git state (commit, push, merge, rebase, tag)"),
+    "deploys": ("noul", ["yes", "no"], "publishes, releases or changes a remote environment"),
+    "runtime_check": ("noul", ["yes", "no"], "a health probe of a running service that fails if unhealthy"),
+    "serves": ("noul", ["yes", "no"], "keeps running until stopped (server, watcher, tail -f)"),
     # every user turn (state: prompt, current_mode, ...)
     "turn.chat_or_task": ("choice", ["chat", "task"], "conversation or work request"),
     "turn.actionable": ("noul", ["yes", "no"], "asks for something to be done"),