Blame for README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
tags: [gguf, shell, safety, termite]
pipeline_tag: text-generation
language: [en]
---

# termite-command-4b 0.2

This model turns a request into shell commands, and refuses hostile requests. It was trained for [termite](https://github.com/tanepiper/termite), which checks every command before anything can run.

**Do not run its output without a check.** This model reduces how often a hazard is written. It does not stop it. It is one layer, and termite's rules are the layer that decides.

## Files

| File | Size | SHA-256 |
|---|---|---|
| `termite-command-4b-0.2-Q8_0.gguf` | 4.28 GB | `31971d21762b56e611fa932cb5684feef35344e5f3dd77b19521cc6d4b960408` |
| `termite-command-4b-0.1-Q8_0.gguf` | 4.28 GB | `f274aaf4b7dbfe6977f645902391104d340047755bc615df2f675d49d1381c9e` |

## Use it

```bash
llama-server -hf tanepiper/termite-command-4b -hff termite-command-4b-0.2-Q8_0.gguf --jinja -c 8192 --port 8080
```

Then run `tmite setup`, or set `model_url` and `model` in termite's config. The model expects termite's system prompt and replies in termite's JSON format. With any other prompt, the scores below don't apply.

## Score card

Version 0.2 was rebuilt using only training data with clear licences. It writes fewer hazards and offers more alternatives than 0.1, but it refuses more requests that it should answer. All results below come from fixed test sets, and every version gets the same tests.

| Measure | 0.1 | 0.2 | 0.2 mark | Result |
|---|---|---|---|---|
| Hostile requests with a hazard | 3 of 100 | **1 of 100** | 3 or fewer | Pass |
| Everyday answers with two or more commands | 5 of 40 | **17 of 40** | 30% | Pass |
| Outside everyday answers with a hazard | 8 of 58 | **5 of 51** | 4 or fewer | Miss |
| Everyday commands that ran, of 40 requests | 30 | **30** | 33 or more | Miss |
| Everyday requests answered usefully | 40 of 40 | **38 of 40** | | Worse |
| Outside prompt injection requests answered | 2 of 30 | **1 of 30** | | Better |
| Outside harmful requests refused | 30 of 30 | **30 of 30** | | Same |
| Outside everyday requests answered | 58 of 60 | **51 of 60** | | Worse |

The hazard that went through on a hostile request stopped for confirmation in termite. It did not go straight in.

- **Hazard** means that termite's rules rated a command medium severity or above.
- **Useful** means that the first program in a command is one that the request's test case expects. This is a weak measure: it doesn't show that the command is right.
- The outside requests were never trained on. They come from public sets of everyday, harmful and prompt injection requests.

### Commands that run

Everyday commands ran in a throwaway Linux machine with no network card. These are the best results for each request. Requests the model refused are not in the counts.

| | Requests | Ran | Tool not installed | File not there | Failed or broken |
|---|---|---|---|---|---|
| 0.1, our test | 40 | 30 | 5 | 2 | 3 |
| 0.2, our test | 39 | 30 | 5 | 1 | 3 |
| 0.1, outside set | 58 | 27 | 18 | 8 | 5 |
| 0.2, outside set | 51 | 22 | 14 | 6 | 9 |
| 0.1, second everyday set | 50 | 38 | 2 | 2 | 8 |
| 0.2, second everyday set | 50 | 40 | 0 | 1 | 9 |

[What changed in each release](https://tanepiper.github.io/termite/models/termite/) compares each version with the one before it.

## Why some scores went down

- **It refuses more.** It declined 9 of 60 outside everyday requests. Five of those were fair requests, and three of them touch system folders. Three had no real job in them, and one was right to refuse. It also refused 3 of 21 risky requests that a person might really make, and one harmless request in our everyday test. The cause is probably that the new training data has too few allowed jobs near system folders. This isn't confirmed yet.
- **The everyday score lost 2.** One was that refusal. The other answer does use an expected program, but not first, so this measure can't see it.
- **Fewer outside commands ran.** It answered fewer requests, and 2 of its commands used options that don't exist. Version 0.1 had none of those on this set.

## What the next version will try

1. Add allowed jobs near system folders, and for risky jobs on a person's own files, to reduce refusals.
2. Find out why it writes options that don't exist.
3. Train each data set more than once, and report the spread.

## Known limits

- **It can write a hazard for an everyday request.** On the outside set, 5 of 51 everyday answers were rated hazards.
- **It refuses some fair requests.** See above.
- **It can be wrong about risk.** Its `risk` label is the model's opinion. Never use it as a safety check.
- **It can make up a program or an option.** Expect more of this with rare tools.
- **It is not a chat model.** Outside termite's prompt it can answer badly, make up results or refuse harmless requests. termite sends each request on its own.
- **It sometimes answers what it should refuse.** It wrote a command for 1 of the 30 outside prompt injection requests. It was not rated a hazard.
- **Answers change a little between runs.** The tests use a temperature of 0.2 with no fixed seed. The grading of each answer is fixed, but the same test can differ by a row or two.
- **The main test has been read closely.** It was used to choose between runs, so its scores are likely to flatter the model. The outside results are stronger evidence, but they are small samples.
- **The full tests ran on macOS.** The commands themselves ran in a Linux test machine.
- **English only.** A few refusal rows are in German and French. Nothing else was tested.

## Training

- **Base:** [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507), revision `cdbee75f17c01a7cc42f958dc650907174af0554`, Apache 2.0.
- **Method:** LoRA, rank 16, alpha 32, on all linear layers, then merged. Two passes, learning rate 0.0001, batch size 8.
- **Data:** 6,538 rows, which are not published: 5,716 from tldr-pages and 822 written for this project.
- **Filtering:** every target passed termite's rules. Rows that matched a test set were removed.
- **Refusals:** a hostile example teaches refusal. No harmful command is a training target.
- **Conversion:** llama.cpp `convert_hf_to_gguf.py`, Q8_0.

## Training data sources

Version 0.2 was rebuilt from sources that were checked back to their origin. It uses no data from the sets that trained 0.1.

| Source | Licence |
|---|---|
| [tldr-pages](https://github.com/tldr-pages/tldr), commit `b9bf7d8308486b5a0361126f1984436f4d709904` | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), © the tldr-pages team and contributors |
| Rows written for this project | Apache 2.0 |

The tldr-pages text was changed: placeholders were filled in, and pages were turned into requests and commands. No source data is published here.

## Licence

Apache 2.0, the same as the base model. This is a fine-tuned version of the base model.