| --- |
| tags: |
| - codesync |
| - devshub |
| - decision-model |
| - developer-tools |
| - gguf |
| - ollama |
| pipeline_tag: text-classification |
| inference: false |
| --- |
| |
| # CodeSync Knack |
|
|
| **A specialized decision model: the free, private judgment model behind CodeSync.** |
|
|
| Every time you send CodeSync a message or it runs a shell command, it makes dozens of quick |
| calls: Is this command risky? Will it run forever? Are you asking to continue earlier work? |
| Is this a question or a task? Which mode fits? Knack makes those calls on your own machine, |
| in about 10 milliseconds. It answers only those decisions, asked by question id. |
|
|
| - **Free.** Ships with every CodeSync install. No API key, no account, no usage limits. |
| - **Private.** Runs locally. Your prompts and commands never leave your machine to be judged. |
| - **Fast.** About 10 ms per decision on a GPU, about 250 ms on a plain CPU. |
| - **Small.** 0.75B parameters in the Ollama edition: 817 MB (Q8_0) or 516 MB (Q4_K_M). |
| - **Accurate where it matters.** Trained on real CodeSync sessions, with extra focus on the |
| rare cases that cost the most when missed: destructive deletes, commands that never exit, |
| resuming interrupted work, refactor / test / documentation requests. |
| |
| ## Get it |
| |
| ```bash |
| ollama pull devshub/knack # Q8_0, 817 MB (https://ollama.com/devshub/knack) |
| ollama pull devshub/knack:q4_k_m # 516 MB, near-identical accuracy |
| ollama pull hf.co/devshub-ai/knack:Q8_0 # same model, straight from this repo |
| ``` |
| |
| ## A specialized decision model |
| |
| General decision models, like clef, read any question you write in plain language. Knack is |
| a different kind: a **specialized decision model**. It was trained on one product's fixed |
| decisions, the 29 that CodeSync makes on every shell command and every user message, and it |
| answers only those. It does not read question text: you ask by **question id**. In exchange |
| it is small, private, and answers all 8 command questions in about 17 ms. |
| |
| | | General decision model (clef) | Specialized decision model (Knack) | |
| | --- | --- | --- | |
| | Questions | anything you write | 29 fixed ones, asked by id | |
| | Input | any text or image | a shell command, or a message to a coding assistant | |
| | Size | 27B | 0.75B | |
| | A question it was not trained on | answers it | rejects it | |
| |
| ## Ask it a question |
| |
| Every call has the same three parts: **the text to judge**, **the question id**, and |
| **Knack's answer**. `echo` only hands the text to Knack. Nothing in the text is ever run, |
| so you can safely ask about dangerous commands. |
| |
| ``` |
| echo "<text to judge>" | ollama run devshub/knack --ask <question id> |
| ``` |
| |
| For example, "will `npm run dev` keep running until I stop it?": |
| |
| ``` |
| $ echo "npm run dev" | ollama run devshub/knack --ask serves |
| yes 97% |
| ``` |
| |
| ### Questions about a shell command |
| |
| Will it keep running until stopped? |
| |
| ``` |
| $ echo "python -m http.server 8000" | ollama run devshub/knack --ask serves |
| yes 98% |
| $ echo "pytest -q" | ollama run devshub/knack --ask serves |
| no 97% |
| ``` |
| |
| How dangerous is it? |
| |
| ``` |
| $ echo "ls -la" | ollama run devshub/knack --ask risk --level safe --level "needs care" --level destructive |
| safe 97% 0.0 / 2 |
| $ echo "npm install" | ollama run devshub/knack --ask risk --level safe --level "needs care" --level destructive |
| needs care 96% 1.0 / 2 |
| $ echo "git push --force origin main" | ollama run devshub/knack --ask risk --level safe --level "needs care" --level destructive |
| destructive 96% 1.9 / 2 |
| ``` |
| |
| Will it take minutes? |
| |
| ``` |
| $ echo "cargo build --release" | ollama run devshub/knack --ask long_running |
| yes 84% |
| $ echo "git status" | ollama run devshub/knack --ask long_running |
| no 97% |
| ``` |
| |
| What kind of command is it? |
| |
| ``` |
| $ echo "pip install requests" | ollama run devshub/knack --ask activity \ |
| --choice testing --choice building --choice installing --choice git --choice inspecting --choice running |
| installing 94% |
| $ echo "grep -rn TODO src" | ollama run devshub/knack --ask activity \ |
| --choice testing --choice building --choice installing --choice git --choice inspecting --choice running |
| inspecting 95% |
| ``` |
| |
| Does a pass prove the change works? Does it change git? Does it deploy? |
| |
| ``` |
| $ echo "pytest tests/test_login.py" | ollama run devshub/knack --ask verifies |
| yes 97% |
| $ echo "npm run build" | ollama run devshub/knack --ask verifies |
| no 96% |
| $ echo "git commit -am 'fix login'" | ollama run devshub/knack --ask mutates_repo |
| yes 96% |
| $ echo "git diff" | ollama run devshub/knack --ask mutates_repo |
| no 97% |
| $ echo "kubectl apply -f k8s/" | ollama run devshub/knack --ask deploys |
| yes 95% |
| ``` |
| |
| ### Questions about a message to a coding assistant |
| |
| What kind of task is this? |
| |
| ``` |
| $ echo "fix the failing login test" | ollama run devshub/knack --ask task.intent \ |
| --choice bug_fix --choice feature --choice refactor --choice api_schema --choice config_deploy \ |
| --choice test --choice documentation --choice investigation --choice unknown |
| bug_fix 90% |
| ``` |
| |
| With the same `--choice` list: "add dark mode to the settings page" gives `feature 93%`, |
| "write docs for the payments API" gives `documentation 97%`, and "why does the build take |
| so long?" gives `investigation 97%`. |
|
|
| Is it chat or a request for work? |
|
|
| ``` |
| $ echo "hi, how are you?" | ollama run devshub/knack --ask turn.chat_or_task --choice chat --choice task |
| chat 98% |
| $ echo "rename the user module" | ollama run devshub/knack --ask turn.chat_or_task --choice chat --choice task |
| task 97% |
| ``` |
|
|
| Which assistant mode fits? |
|
|
| ``` |
| $ echo "what does this repo do?" | ollama run devshub/knack --ask route.mode --choice plan --choice build --choice ask --choice debug |
| ask 96% |
| $ echo "plan the migration to Postgres" | ollama run devshub/knack --ask route.mode --choice plan --choice build --choice ask --choice debug |
| plan 90% |
| ``` |
|
|
| Start or stop a screen recording? |
|
|
| ``` |
| $ echo "record my screen" | ollama run devshub/knack --ask turn.screencast --choice start --choice stop --choice none |
| start 89% |
| ``` |
|
|
| The answer is the option Knack picks and its confidence. Each id has one precise meaning, |
| listed in the tables below; read them before relying on an answer. `long_running`, for |
| example, means "takes minutes", so a build counts. Questions with options (`--choice`, |
| `--level`) need them in the order shown; if they do not match, the error shows the exact |
| flags to use. Any question that is not one of the 29 ids is rejected rather than guessed. |
|
|
| ## API |
|
|
| ### cURL |
|
|
| ``` |
| curl http://localhost:11434/v1/systemone \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "model": "devshub/knack", |
| "state": "npm run dev", |
| "questions": { |
| "serves": {"type": "noul", "instructions": "serves"}, |
| "long_running": {"type": "noul", "instructions": "long_running"} |
| } |
| }' |
| ``` |
|
|
| ``` |
| {"model":"devshub/knack","answers":{"serves":{"type":"noul","noul":0.9701},"long_running":{"type":"noul","noul":0.9708}}} |
| ``` |
|
|
| The question id goes in `instructions` (or as the key). The state can also be an object |
| with Knack's own fields, for example `{"command": "npm run dev", "description": "start the app"}`. |
|
|
| ### Python |
|
|
| ``` |
| pip install ollama |
| ``` |
|
|
| ```python |
| import ollama |
| |
| response = ollama.systemone( |
| model="devshub/knack", |
| state="npm run dev", |
| questions={"serves": {"type": "noul", "instructions": "serves"}}, |
| ) |
| print(response.answers["serves"].noul) # 0.9701 |
| ``` |
|
|
| ## Or use the standalone script |
|
|
| `knack.py` asks every question of a command or message at once and prints a table. It uses |
| only the Python standard library and pulls the model on first run. |
|
|
| ``` |
| curl -fsSL https://huggingface.co/devshub-ai/knack/resolve/main/knack.py | python3 - "npm run dev" |
| ``` |
|
|
| ``` |
| risk 1 0.95 0 read-only, 1 recoverable change, 2 irreversible / remote |
| long_running yes 0.97 takes minutes, blocks, polls, watches or serves |
| activity running 0.92 what kind of work it does |
| verifies no 0.97 a test, type or lint check whose pass proves the change works |
| mutates_repo no 0.97 changes git state (commit, push, merge, rebase, tag) |
| deploys no 0.94 publishes, releases or changes a remote environment |
| runtime_check no 0.95 a health probe of a running service that fails if unhealthy |
| serves yes 0.97 keeps running until stopped (server, watcher, tail -f) |
| ``` |
|
|
| ``` |
| curl -fsSL https://huggingface.co/devshub-ai/knack/resolve/main/knack.py | python3 - --turn "fix the failing login test" |
| ``` |
|
|
| ## What it decides |
|
|
| **Every shell command.** Input: the command (state fields `command`, `description`). |
|
|
| | id | yes / the answer means | options | |
| | --- | --- | --- | |
| | `risk` | the worst it can do: 0 read-only or trivially reversible, 1 changes recoverable local state, 2 irreversible loss or reaches beyond the checkout | 0, 1, 2 | |
| | `long_running` | takes minutes, blocks, polls, watches or serves, rather than finishing in seconds. A build or test suite counts | yes, no | |
| | `serves` | keeps running until it is stopped: server, daemon, watcher, tunnel, `tail -f`. A build that finishes on its own does not | yes, no | |
| | `activity` | what kind of work it does | testing, building, installing, git, inspecting, running | |
| | `verifies` | a test run, type or lint check whose passing exit is evidence the change works. Builds and plain runs do not count | yes, no | |
| | `mutates_repo` | changes version-control state: commit, push, merge, rebase, tag, amend. Editing or deleting files does not count | yes, no | |
| | `deploys` | publishes, releases, or changes a remote or runtime environment (deploy, rollout, push an image, bring a service up) | yes, no | |
| | `runtime_check` | a health or smoke probe of a running service that fails on an unhealthy response (`curl --fail`) | yes, no | |
|
|
| **Every user turn.** Input: the message a user sends to a coding assistant (state fields, in |
| this order: `prompt`, `current_mode`, `recent_activity`, `conversation_size`, `open_task`, |
| `turns_on_open_task`, `interrupted_work`, `browser_context`; plain text fills the rest with |
| defaults). |
|
|
| | id | the answer means | options | |
| | --- | --- | --- | |
| | `turn.chat_or_task` | conversation, or a request for work | chat, task | |
| | `turn.actionable` | asks for something to be done | yes, no | |
| | `turn.question_shaped` | phrased as a question | yes, no | |
| | `turn.change_intent` | kind of code change asked for | create, adjust, enhance, fix, refactor, none | |
| | `turn.new_task` | starts new work rather than continuing | yes, no | |
| | `turn.resume` | asks to resume interrupted work | yes, no | |
| | `turn.mission` | a long, multi-step job | yes, no | |
| | `turn.plan_authoring` | asks for a plan to be written | yes, no | |
| | `turn.screencast` | start or stop a screen recording | start, stop, none | |
| | `turn.demo` | asks for a demo | yes, no | |
| | `turn.skip_orientation` | can be done without exploring the codebase | yes, no | |
| | `task.intent` | type of task | bug_fix, feature, refactor, api_schema, config_deploy, test, documentation, investigation, unknown | |
| | `task.execution_strategy` | how to execute it | direct, plan, parallel | |
| | `route.mode` | best assistant mode | plan, build, ask, debug | |
| | `route.tier` | size of model needed | light, medium, heavy | |
| | `route.thinking` | reasoning effort needed | off, low, medium, high, xhigh, max | |
| | `route.risk` | risk of the requested work | low, medium, high | |
| | `browser` | needs a web browser | yes, no | |
| | `messaging` | needs to send an email or chat message | yes, no | |
| | `delivery.feature_task` | a feature delivery task | yes, no | |
| | `delivery.slash_task` | a slash-command style task | yes, no | |
|
|
| ## What it does not do |
|
|
| - **Other questions.** Anything outside these 29, such as "is this log an error?" or "is a |
| refund requested?", is rejected. Use a general decision model like clef for those. |
| - **Other input.** Logs, emails, tickets, documents or images give meaningless answers, |
| even to these 29 questions. |
| - **Conversation history.** Through Ollama it sees a single message, so a question like |
| `turn.resume` ("continue where you left off") answers no: there is nothing to resume. Inside CodeSync it also |
| gets the session context, which makes the turn questions more reliable. |
| - **Text generation.** It returns one pick with a confidence, never sentences. |
|
|
| ## Known weak spots |
|
|
| Measured with the commands above, so you know where to double-check: |
|
|
| | command | question | Knack says | should be | |
| | --- | --- | --- | --- | |
| | `tail -f app.log` | `serves` | no, 75% | yes | |
| | `npm run build` | `serves` | yes, 79% | no | |
| | `docker compose up` | `long_running` | 51% yes | yes | |
| | `curl --fail http://localhost:3000/health` | `runtime_check` | no, 92% | yes | |
| | `npm test` | `long_running` | 45% yes | unclear, depends on the suite | |
|
|
| These come from the model's training, not the Ollama packaging: on every command compared, |
| the Ollama build matches the original model to within 0.002. They are the next training targets. |
|
|
| ## Results |
|
|
| Measured on CodeSync's own judgments (version v3, November 2026): |
|
|
| | Benchmark | Knack v3 | Ollama Q8_0 | Ollama Q4_K_M | What it measures | |
| |---|---:|---:|---:|---| |
| | Hand-labelled set | **97.9%** | 97.9% | 97.4% | 233 decisions labelled by hand | |
| | Held-out set | **95.5%** | 95.7% | 95.7% | 3,267 decisions from real sessions, never trained on | |
| | Rare-case set | **93.7%** | 93.6% | 93.6% | 4,120 decisions on the hardest, rarest cases | |
| | Latency | **13 ms** | 17 ms | ~17 ms | median per command (8 questions) on a laptop GPU | |
| |
| Latency is the median for all 8 command questions in one `/v1/systemone` request. |
| |
| ## Use it inside CodeSync |
| |
| Install CodeSync. That's it. |
| |
| ```bash |
| curl -fsSL https://app.devshub.ai/install.sh | bash |
| ``` |
| |
| On first start, CodeSync downloads Knack in the background and keeps working while it |
| does. Once Knack is ready it becomes CodeSync's first judgment engine, and new versions are |
| picked up automatically. To turn it off: `CODESYNC_KNACK_DISABLED=1`. |
| |
| ## Files |
| |
| | File | Purpose | |
| |---|---| |
| | `knack-v3-Q8_0.gguf`, `knack-v3-Q4_K_M.gguf` | Ollama / llama.cpp decision model (`--ask`, `/v1/systemone`) | |
| | `knack.py` | standalone runner (CLI and local server), standard library only | |
| | `manifest.json`, `knack-<version>.tar.gz` | the PyTorch model folder CodeSync installs | |
|
|
| The GGUF edition is the same weights packaged as an Ollama decision model: a `systemone` |
| prompt template rebuilds Knack's training prompt from the request, and the answer is read |
| from the option letters exactly as CodeSync's own runtime reads it. |
|
|
| Built by [DevsHub](https://app.devshub.ai) for [CodeSync](https://app.devshub.ai). |
|
|
