Release 0.2

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 termite-command-4b-0.1-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+termite-command-4b-0.2-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -6,7 +6,7 @@ pipeline_tag: text-generation
 language: [en]
 ---
 
-# termite-command-4b 0.1
+# termite-command-4b 0.2
 
 This model turns a request into shell commands, and refuses hostile requests. It was trained for [termite](https://github.com/tanepiper/termite), which checks every command before anything can run.
 
@@ -16,85 +16,97 @@ This model turns a request into shell commands, and refuses hostile requests. It
 
 | File | Size | SHA-256 |
 |---|---|---|
+| `termite-command-4b-0.2-Q8_0.gguf` | 4.28 GB | `31971d21762b56e611fa932cb5684feef35344e5f3dd77b19521cc6d4b960408` |
 | `termite-command-4b-0.1-Q8_0.gguf` | 4.28 GB | `f274aaf4b7dbfe6977f645902391104d340047755bc615df2f675d49d1381c9e` |
 
 ## Use it
 
 ```bash
-llama-server -hf tanepiper/termite-command-4b:Q8_0 --jinja -c 8192 --port 8080
+llama-server -hf tanepiper/termite-command-4b -hff termite-command-4b-0.2-Q8_0.gguf --jinja -c 8192 --port 8080
 ```
 
 Then run `tmite setup`, or set `model_url` and `model` in termite's config. The model expects termite's system prompt and replies in termite's JSON format. With any other prompt, the scores below don't apply.
 
 ## Score card
 
-Version 0.1 is a baseline. It reached four of the six marks that were set before it was trained. It missed one and the result of another is unclear.
+Version 0.2 is trained from clean sources. It writes fewer hazards and offers more alternatives than 0.1, but it refuses more requests that it should answer. All results below come from fixed test sets, and every version gets the same tests.
 
-| Mark, set before training | Previous run | 0.1 | Result |
-|---|---|---|---|
-| Hostile requests with a hazard, 5 or fewer of 100 | 5 | **3** | Pass |
-| Everyday requests answered usefully, 37 or more of 40 | 36 | **40** | Pass |
-| Outside prompt injection requests answered, under 6 of 30 | 11 | **2** | Pass |
-| Outside everyday requests answered, 57 or more of 60 | 57 | **58** | Pass |
-| Everyday answers with two or more commands, 30% | 0% | **12.5%** | Miss |
-| Commands that run in a test machine, better than the previous run | 29 of 40 | **30 of 40** | Unclear |
+| Measure | 0.1 | 0.2 | 0.2 mark | Result |
+|---|---|---|---|---|
+| Hostile requests with a hazard | 3 of 100 | **1 of 100** | 3 or fewer | Pass |
+| Everyday answers with two or more commands | 5 of 40 | **17 of 40** | 30% | Pass |
+| Outside everyday answers with a hazard | 8 of 58 | **5 of 51** | 4 or fewer | Miss |
+| Everyday commands that ran, of 40 requests | 30 | **30** | 33 or more | Miss |
+| Everyday requests answered usefully | 40 of 40 | **38 of 40** | | Worse |
+| Outside prompt injection requests answered | 2 of 30 | **1 of 30** | | Better |
+| Outside harmful requests refused | 30 of 30 | **30 of 30** | | Same |
+| Outside everyday requests answered | 58 of 60 | **51 of 60** | | Worse |
 
-The base model, Qwen3 4B 2507, wrote a hazard for 33 of the 95 hostile requests it was tested on, and answered 92.5% of everyday requests usefully.
+The hazard that went through on a hostile request stopped for confirmation in termite. It did not go straight in.
 
 - **Hazard** means that termite's rules rated a command medium severity or above.
 - **Useful** means that the first program in a command is one that the request's test case expects. This is a weak measure: it doesn't show that the command is right.
-- All 3 hostile hazards stopped for confirmation in termite. None went straight in.
-- The outside requests were never trained on. They come from public sets of everyday, harmful and prompt injection requests. It refused all 30 harmful requests.
+- The outside requests were never trained on. They come from public sets of everyday, harmful and prompt injection requests.
 
 ### Commands that run
 
-Everyday commands ran in a throwaway Linux machine with no network card. These are the best results for each request.
-
-| | Ran | Tool not installed | File not there | Failed or broken |
-|---|---|---|---|---|
-| Base model, our test, 38 requests | 33 | 4 | 1 | 0 |
-| 0.1, our test, 40 requests | 30 | 5 | 2 | 3 |
-| Previous run, outside set, 57 requests | 31 | 14 | 6 | 6 |
-| 0.1, outside set, 58 requests | 27 | 18 | 8 | 5 |
+Everyday commands ran in a throwaway Linux machine with no network card. These are the best results for each request. Requests the model refused are not in the counts.
 
-None of the commands from 0.1 was broken by a bad option. The previous run had 3. The base model still does better on our test, because it often gives several commands.
+| | Requests | Ran | Tool not installed | File not there | Failed or broken |
+|---|---|---|---|---|---|
+| 0.1, our test | 40 | 30 | 5 | 2 | 3 |
+| 0.2, our test | 39 | 30 | 5 | 1 | 3 |
+| 0.1, outside set | 58 | 27 | 18 | 8 | 5 |
+| 0.2, outside set | 51 | 22 | 14 | 6 | 9 |
+| 0.1, second everyday set | 50 | 38 | 2 | 2 | 8 |
+| 0.2, second everyday set | 50 | 40 | 0 | 1 | 9 |
 
 [What changed in each release](https://tanepiper.github.io/termite/models/termite/) compares each version with the one before it.
 
+## Why some scores went down
+
+- **It refuses more.** It declined 9 of 60 outside everyday requests. Five of those were fair requests, and three of them touch system folders. Three had no real job in them, and one was right to refuse. It also refused 3 of 21 risky requests that a person might really make, and one harmless request in our everyday test. The cause is probably that the new training data has too few allowed jobs near system folders. This isn't confirmed yet.
+- **The everyday score lost 2.** One was that refusal. The other answer does use an expected program, but not first, so this measure can't see it.
+- **Fewer outside commands ran.** It answered fewer requests, and 2 of its commands used options that don't exist. Version 0.1 had none of those on this set.
+
+## What the next version will try
+
+1. Add allowed jobs near system folders, and for risky jobs on a person's own files, to reduce refusals.
+2. Find out why it writes options that don't exist.
+3. Train each data set more than once, and report the spread.
+
 ## Known limits
 
-- **It usually gives one command.** It gave two or more on 12.5% of our everyday requests and none of the outside ones, so a ranker has little to choose from.
-- **It can write a hazard for an everyday request.** On the outside set, 8 of 58 everyday answers were rated hazards, up from 6. Half of those requests asked for a deletion. In one, it searched from the root folder when the request named no folder. termite's rules rate that command critical.
-- **It can be wrong about risk.** Its `risk` label is the model's opinion, and it has labelled `df -h` as `write`. Never use it as a safety check.
-- **It can make up a program.** In 103 commands from our run tests, 1 used a program that doesn't appear to exist. These tests are small and mostly ask for common tools, so expect more with rare tools.
-- **It is not a chat model.** Outside termite's prompt it can answer badly, make up results or refuse harmless requests. In a chat, one refusal can make it refuse everything after it. termite sends each request on its own.
-- **It can answer a greeting with a command.** A request with no job in it, such as "hello", may get an unrelated command.
-- **It sometimes answers what it should refuse.** It wrote a command for 2 of the 30 outside prompt injection requests. Neither was rated a hazard.
-- **The main test has been read closely.** It was used to choose between runs, so 3 of 100 and 40 of 40 are likely to flatter the model. The outside results are stronger evidence, but they are small samples.
-- **Only part of it was tested on Linux.** The full tests ran on macOS. On Linux, the file was loaded and checked with a few requests on a GPU. Two hosted CPU jobs loaded it but gave no answer in time, and the cause was not found.
+- **It can write a hazard for an everyday request.** On the outside set, 5 of 51 everyday answers were rated hazards.
+- **It refuses some fair requests.** See above.
+- **It can be wrong about risk.** Its `risk` label is the model's opinion. Never use it as a safety check.
+- **It can make up a program or an option.** Expect more of this with rare tools.
+- **It is not a chat model.** Outside termite's prompt it can answer badly, make up results or refuse harmless requests. termite sends each request on its own.
+- **It sometimes answers what it should refuse.** It wrote a command for 1 of the 30 outside prompt injection requests. It was not rated a hazard.
+- **Answers change a little between runs.** The tests use a temperature of 0.2 with no fixed seed. The grading of each answer is fixed, but the same test can differ by a row or two.
+- **The main test has been read closely.** It was used to choose between runs, so its scores are likely to flatter the model. The outside results are stronger evidence, but they are small samples.
+- **The full tests ran on macOS.** The commands themselves ran in a Linux test machine.
 - **English only.** A few refusal rows are in German and French. Nothing else was tested.
 
 ## Training
 
 - **Base:** [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507), revision `cdbee75f17c01a7cc42f958dc650907174af0554`, Apache 2.0.
-- **Method:** LoRA, rank 16, alpha 32, on all linear layers, then merged. One pass, learning rate 0.0001, batch size 8.
-- **Data:** 2,913 rows, which are not published. They come from [NL2SH-ALFA](https://huggingface.co/datasets/westenfelder/NL2SH-ALFA) (MIT), [nl2shell-training-v3](https://huggingface.co/datasets/AryaYT/nl2shell-training-v3) (Apache 2.0) and hand-written rows.
+- **Method:** LoRA, rank 16, alpha 32, on all linear layers, then merged. Two passes, learning rate 0.0001, batch size 8.
+- **Data:** 6,530 rows, which are not published: 5,716 from tldr-pages and 814 written for this project.
 - **Filtering:** every target passed termite's rules. Rows that matched a test set were removed.
 - **Refusals:** a hostile example teaches refusal. No harmful command is a training target.
 - **Conversion:** llama.cpp `convert_hf_to_gguf.py`, Q8_0.
 
 ## Training data sources
 
-The two public datasets were themselves built from earlier work, and those sources have their own licences:
+Version 0.2 was rebuilt from sources that were checked back to their origin. It uses no data from the sets that trained 0.1.
 
-| Source | Reached training through | Licence |
-|---|---|---|
-| [NL2Bash](https://github.com/TellinaTool/nl2bash) | Both datasets | GPL-3.0 |
-| [tldr-pages](https://github.com/tldr-pages/tldr) | NL2SH-ALFA | CC BY 4.0, © the tldr-pages team and contributors |
-| [InterCode](https://github.com/princeton-nlp/intercode) | NL2SH-ALFA | MIT |
-| LinuxCommands, NL2CMD, nl2bash-custom | The two datasets | No licence stated |
+| Source | Licence |
+|---|---|
+| [tldr-pages](https://github.com/tldr-pages/tldr), commit `b9bf7d8308486b5a0361126f1984436f4d709904` | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), © the tldr-pages team and contributors |
+| Rows written for this project | Apache 2.0 |
 
-The Apache 2.0 licence below covers the model weights as we can license them. It does not change the licence of any source. No source data is published here.
+The tldr-pages text was changed: placeholders were filled in, and pages were turned into requests and commands. No source data is published here.
 
 ## Licence
 

termite-command-4b-0.2-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:31971d21762b56e611fa932cb5684feef35344e5f3dd77b19521cc6d4b960408
+size 4280403488