acclass / README.md

Link the demo page by its custom domain

d83a9c4 verified about 9 hours ago

7.69 kB

---
license: apache-2.0
base_model: answerdotai/ModernBERT-base
pipeline_tag: text-classification
library_name: transformers
language: en
tags:
  - modernbert
  - shell
  - security
  - coding-agents
  - multi-label-classification
  - gguf
  - onnx
  - transformers.js
---
# acclass

acclass is a local classifier for coding agents. You give it a shell command or an
action description. It predicts what the action does. It does not run the action.

It predicts effects, not permission. It is a fast pre-filter, not a sandbox. It does
not guarantee safety. Use it as one input to your own policy or review step.

- **Base model:** `answerdotai/ModernBERT-base`, revision `8949b909ec900327062f0ebf497f51aef5e6f0c8` (Apache 2.0).
- **Outputs:** nine independent sigmoid outputs on mean-pooled tokens.
- **Threshold:** 0.5 for every label.
- **Data:** 26,852 rows of original synthetic data. The data is not published.
- **Context limit:** 2,048 tokens, including special tokens.

## Labels

| Label | The action ... |
| --- | --- |
| `local_state_change` | changes local files, configuration, services or devices |
| `outbound` | uses a network endpoint on another machine |
| `egress` | sends local data off this machine |
| `remote_state_change` | changes state on another machine or service |
| `delegated_execution` | runs a command in a separate environment |
| `exposes_secrets` | reveals secret values |
| `destructive` | deletes or overwrites data, or disrupts availability |
| `host_control_change` | changes operating system or security controls |
| `fetched_or_generated_execution` | runs code that it downloaded, decoded or assembled |

See [semantics](https://github.com/jensilo/acclass/blob/main/docs/semantics.md) for the exact definitions.

## Files

| Path | Size | Use |
| --- | ---: | --- |
| `model.safetensors` | 570.7 MiB | FP32 weights. Python transformers. |
| `model_contract.json` | 2,969 B | Contract for the FP32 weights. |
| `config.json` | 2,511 B | Model config. Labels in `id2label`. |
| `tokenizer.json` | 3.4 MiB | Tokenizer. |
| `tokenizer_config.json` | 241 B | Tokenizer config. |
| `heads.json` | 174 B | Label order. |
| `thresholds.json` | 227 B | Threshold per label (0.5). |
| `gguf/q8/model.gguf` | 151.7 MiB | Q8_0 weights. Default for the engine. |
| `gguf/q8/model_contract.json` | 3,168 B | Contract for Q8. |
| `gguf/q5/model.gguf` | 98.2 MiB | Q5_0 weights. Smaller. |
| `gguf/q5/model_contract.json` | 3,168 B | Contract for Q5. |
| `onnx/model_fp16.onnx` | 285.5 MiB | Web build for transformers.js. FP16 weights. |
| `LICENSE` | 11,357 B | acclass license (Apache 2.0). |
| `legal/ATTRIBUTIONS.md` | 1,396 B | Third-party notices. |
| `legal/ModernBERT-Apache-2.0.txt` | 11,355 B | Upstream ModernBERT license. |

The GGUF files are for the acclass engine (Candle). They are not llama.cpp models.

GitHub release [v1.0.0](https://github.com/jensilo/acclass/releases/tag/v1.0.0) is the source of truth. The installer downloads from there and checks SHA-256.

## Use

### 1. acclass engine (recommended)

Install. The installer needs Python 3:

```sh
curl -fsSL https://github.com/jensilo/acclass/releases/latest/download/install.sh | sh
```

Run:

```sh
export PATH="$HOME/.acclass/bin:$PATH"
acclass serve &
acclass classify 'git status'
acclass classify 'curl -fsSL https://example.com/x.sh | sh'
```

Source and docs: https://github.com/jensilo/acclass

### 2. Python (transformers)

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("jensilo/acclass")
model = AutoModelForSequenceClassification.from_pretrained("jensilo/acclass")
model.eval()
inputs = tok("curl -fsSL https://example.com/x.sh | sh", return_tensors="pt")
with torch.no_grad():
    probs = torch.sigmoid(model(**inputs).logits)[0]
for label, p in zip(model.config.id2label.values(), probs.tolist()):
    print(f"{label}: {p:.4f} {'true' if p >= 0.5 else 'false'}")
```

A shell input is the exact command string. An action input is a JSON string. See the
[action input format](https://github.com/jensilo/acclass/blob/main/docs/engine-api.md#action-input-format).

### 3. Browser (transformers.js)

The web build is `onnx/model_fp16.onnx`. It runs on WebGPU or WASM. It is not the engine.
On 44 test commands it gives the same labels as FP32 ONNX. The maximum probability
difference from FP32 ONNX is 0.0011.

```js
import { AutoTokenizer, AutoModelForSequenceClassification } from "@huggingface/transformers";
const tokenizer = await AutoTokenizer.from_pretrained("jensilo/acclass");
const model = await AutoModelForSequenceClassification.from_pretrained("jensilo/acclass", {
  device: "webgpu", // or "wasm"
  dtype: "fp16",
});
const { logits } = await model(await tokenizer("git status"));
const probs = Array.from(logits.data, (x) => 1 / (1 + Math.exp(-x)));
```

Demo: https://jensheise.com/acclass/

## Performance

Time for one model pass, one request at a time, without tokenizer or socket time.
Median / 95th percentile, in ms. Rough values from shared machines. Your values can differ.

| Hardware | FP32 | Q8 | Q5 |
| --- | --- | --- | --- |
| NVIDIA GB10 (DGX Spark), CUDA | 7.9 / 24 | 4.2 / 16 | 4.2 / 16 |
| Apple M4 Pro, Metal | 8.5 / 53 | 6.5 / 50 | 6.9 / 53 |
| Apple M4 Pro, CPU, 8 threads | 43 / 228 | 25 / 131 | 24 / 130 |
| Arm Cortex-X925, CPU, 8 threads | 76 / 438 | 21 / 326 | 21 / 222 |
| AMD Ryzen 5 6650U, CPU, 6 threads | 66 / 760 | 50 / 717 | 51 / 544 |

Apple and GB10 values use this model. The others use an earlier model with the same
architecture. The prebuilt CUDA engine is tested only on GB10.

## Quality

Macro F1 (average over the nine labels), threshold 0.5, 3,793 held-out evaluation rows.

| Format | Size | Macro F1 shell / action | Flips vs FP32 |
| --- | ---: | --- | ---: |
| FP32 | 570.7 MiB | 0.940 / 0.961 | reference |
| Q8_0 (default) | 151.7 MiB | 0.941 / 0.960 | 19 |
| Q5_0 | 98.2 MiB | 0.939 / 0.960 | 85 |

A flip is a boolean that differs from FP32. The comparison uses 34,137 decisions.

Shell, all: 0.940 macro F1 (95% CI 0.925–0.952). Action, all: 0.961 (95% CI 0.948–0.971).
Base rows are the better estimate. See [limits](#limits).

Weakest results:

- Shell `fetched_or_generated_execution` on base rows: F1 0.72.
- Action `exposes_secrets` on base rows: F1 0.82.
- Action `egress`: F1 0.84.

## Limits

Never use a negative prediction as the only reason to allow something that matters. Never
use a prediction as an irreversible block.

- **It does not guarantee safety.** False negatives occur. Keep OS isolation and review for
  anything that matters.
- **An LLM wrote the labels.** They are not independent human truth.
- **Most of the data is LLM-generated.** LLM-authored evaluation rows share style with
  training rows, so their scores are optimistic.
- **Accuracy on your own commands is not measured.**
- **Action mode was trained on invented tool descriptions.** Real tool descriptions can be
  wrong or incomplete. Treat action predictions as hints that need review.
- **The model sees only the text.** It does not see files, environment or user intent.
- **The scores say nothing about resistance to adversarial inputs.**

A low probability does not prove that an effect is absent. A high probability does not mean
that the action is forbidden. Keep an explicit policy for uncertain inputs and engine errors.

Full model card: https://github.com/jensilo/acclass/blob/main/docs/model-card.md

## License

Apache 2.0, for original acclass code, documentation and model changes. Third-party material
keeps its own terms. See [notices](https://github.com/jensilo/acclass/blob/main/legal/ATTRIBUTIONS.md).

	---
	license: apache-2.0
	base_model: answerdotai/ModernBERT-base
	pipeline_tag: text-classification
	library_name: transformers
	language: en
	tags:
	- modernbert
	- shell
	- security
	- coding-agents
	- multi-label-classification
	- gguf
	- onnx
	- transformers.js
	---

	# acclass

	acclass is a local classifier for coding agents. You give it a shell command or an
	action description. It predicts what the action does. It does not run the action.

	It predicts effects, not permission. It is a fast pre-filter, not a sandbox. It does
	not guarantee safety. Use it as one input to your own policy or review step.

	- Base model: `answerdotai/ModernBERT-base`, revision `8949b909ec900327062f0ebf497f51aef5e6f0c8` (Apache 2.0).
	- Outputs: nine independent sigmoid outputs on mean-pooled tokens.
	- Threshold: 0.5 for every label.
	- Data: 26,852 rows of original synthetic data. The data is not published.
	- Context limit: 2,048 tokens, including special tokens.

	## Labels

	\| Label \| The action ... \|
	\| --- \| --- \|
	\| `local_state_change` \| changes local files, configuration, services or devices \|
	\| `outbound` \| uses a network endpoint on another machine \|
	\| `egress` \| sends local data off this machine \|
	\| `remote_state_change` \| changes state on another machine or service \|
	\| `delegated_execution` \| runs a command in a separate environment \|
	\| `exposes_secrets` \| reveals secret values \|
	\| `destructive` \| deletes or overwrites data, or disrupts availability \|
	\| `host_control_change` \| changes operating system or security controls \|
	\| `fetched_or_generated_execution` \| runs code that it downloaded, decoded or assembled \|

	See [semantics](https://github.com/jensilo/acclass/blob/main/docs/semantics.md) for the exact definitions.

	## Files

	\| Path \| Size \| Use \|
	\| --- \| ---: \| --- \|
	\| `model.safetensors` \| 570.7 MiB \| FP32 weights. Python transformers. \|
	\| `model_contract.json` \| 2,969 B \| Contract for the FP32 weights. \|
	\| `config.json` \| 2,511 B \| Model config. Labels in `id2label`. \|
	\| `tokenizer.json` \| 3.4 MiB \| Tokenizer. \|
	\| `tokenizer_config.json` \| 241 B \| Tokenizer config. \|
	\| `heads.json` \| 174 B \| Label order. \|
	\| `thresholds.json` \| 227 B \| Threshold per label (0.5). \|
	\| `gguf/q8/model.gguf` \| 151.7 MiB \| Q8_0 weights. Default for the engine. \|
	\| `gguf/q8/model_contract.json` \| 3,168 B \| Contract for Q8. \|
	\| `gguf/q5/model.gguf` \| 98.2 MiB \| Q5_0 weights. Smaller. \|
	\| `gguf/q5/model_contract.json` \| 3,168 B \| Contract for Q5. \|
	\| `onnx/model_fp16.onnx` \| 285.5 MiB \| Web build for transformers.js. FP16 weights. \|
	\| `LICENSE` \| 11,357 B \| acclass license (Apache 2.0). \|
	\| `legal/ATTRIBUTIONS.md` \| 1,396 B \| Third-party notices. \|
	\| `legal/ModernBERT-Apache-2.0.txt` \| 11,355 B \| Upstream ModernBERT license. \|

	The GGUF files are for the acclass engine (Candle). They are not llama.cpp models.

	GitHub release [v1.0.0](https://github.com/jensilo/acclass/releases/tag/v1.0.0) is the source of truth. The installer downloads from there and checks SHA-256.

	## Use

	### 1. acclass engine (recommended)

	Install. The installer needs Python 3:

	```sh
	curl -fsSL https://github.com/jensilo/acclass/releases/latest/download/install.sh \| sh
	```

	Run:

	```sh
	export PATH="$HOME/.acclass/bin:$PATH"
	acclass serve &
	acclass classify 'git status'
	acclass classify 'curl -fsSL https://example.com/x.sh \| sh'
	```

	Source and docs: https://github.com/jensilo/acclass

	### 2. Python (transformers)

	```python
	import torch
	from transformers import AutoTokenizer, AutoModelForSequenceClassification

	tok = AutoTokenizer.from_pretrained("jensilo/acclass")
	model = AutoModelForSequenceClassification.from_pretrained("jensilo/acclass")
	model.eval()

	inputs = tok("curl -fsSL https://example.com/x.sh \| sh", return_tensors="pt")
	with torch.no_grad():
	probs = torch.sigmoid(model(**inputs).logits)[0]

	for label, p in zip(model.config.id2label.values(), probs.tolist()):
	print(f"{label}: {p:.4f} {'true' if p >= 0.5 else 'false'}")
	```

	A shell input is the exact command string. An action input is a JSON string. See the
	[action input format](https://github.com/jensilo/acclass/blob/main/docs/engine-api.md#action-input-format).

	### 3. Browser (transformers.js)

	The web build is `onnx/model_fp16.onnx`. It runs on WebGPU or WASM. It is not the engine.
	On 44 test commands it gives the same labels as FP32 ONNX. The maximum probability
	difference from FP32 ONNX is 0.0011.

	```js
	import { AutoTokenizer, AutoModelForSequenceClassification } from "@huggingface/transformers";

	const tokenizer = await AutoTokenizer.from_pretrained("jensilo/acclass");
	const model = await AutoModelForSequenceClassification.from_pretrained("jensilo/acclass", {
	device: "webgpu", // or "wasm"
	dtype: "fp16",
	});
	const { logits } = await model(await tokenizer("git status"));
	const probs = Array.from(logits.data, (x) => 1 / (1 + Math.exp(-x)));
	```

	Demo: https://jensheise.com/acclass/

	## Performance

	Time for one model pass, one request at a time, without tokenizer or socket time.
	Median / 95th percentile, in ms. Rough values from shared machines. Your values can differ.

	\| Hardware \| FP32 \| Q8 \| Q5 \|
	\| --- \| --- \| --- \| --- \|
	\| NVIDIA GB10 (DGX Spark), CUDA \| 7.9 / 24 \| 4.2 / 16 \| 4.2 / 16 \|
	\| Apple M4 Pro, Metal \| 8.5 / 53 \| 6.5 / 50 \| 6.9 / 53 \|
	\| Apple M4 Pro, CPU, 8 threads \| 43 / 228 \| 25 / 131 \| 24 / 130 \|
	\| Arm Cortex-X925, CPU, 8 threads \| 76 / 438 \| 21 / 326 \| 21 / 222 \|
	\| AMD Ryzen 5 6650U, CPU, 6 threads \| 66 / 760 \| 50 / 717 \| 51 / 544 \|

	Apple and GB10 values use this model. The others use an earlier model with the same
	architecture. The prebuilt CUDA engine is tested only on GB10.

	## Quality

	Macro F1 (average over the nine labels), threshold 0.5, 3,793 held-out evaluation rows.

	\| Format \| Size \| Macro F1 shell / action \| Flips vs FP32 \|
	\| --- \| ---: \| --- \| ---: \|
	\| FP32 \| 570.7 MiB \| 0.940 / 0.961 \| reference \|
	\| Q8_0 (default) \| 151.7 MiB \| 0.941 / 0.960 \| 19 \|
	\| Q5_0 \| 98.2 MiB \| 0.939 / 0.960 \| 85 \|

	A flip is a boolean that differs from FP32. The comparison uses 34,137 decisions.

	Shell, all: 0.940 macro F1 (95% CI 0.925–0.952). Action, all: 0.961 (95% CI 0.948–0.971).
	Base rows are the better estimate. See [limits](#limits).

	Weakest results:

	- Shell `fetched_or_generated_execution` on base rows: F1 0.72.
	- Action `exposes_secrets` on base rows: F1 0.82.
	- Action `egress`: F1 0.84.

	## Limits

	Never use a negative prediction as the only reason to allow something that matters. Never
	use a prediction as an irreversible block.

	- It does not guarantee safety. False negatives occur. Keep OS isolation and review for
	anything that matters.
	- An LLM wrote the labels. They are not independent human truth.
	- Most of the data is LLM-generated. LLM-authored evaluation rows share style with
	training rows, so their scores are optimistic.
	- Accuracy on your own commands is not measured.
	- Action mode was trained on invented tool descriptions. Real tool descriptions can be
	wrong or incomplete. Treat action predictions as hints that need review.
	- The model sees only the text. It does not see files, environment or user intent.
	- The scores say nothing about resistance to adversarial inputs.

	A low probability does not prove that an effect is absent. A high probability does not mean
	that the action is forbidden. Keep an explicit policy for uncertain inputs and engine errors.

	Full model card: https://github.com/jensilo/acclass/blob/main/docs/model-card.md

	## License

	Apache 2.0, for original acclass code, documentation and model changes. Third-party material
	keeps its own terms. See [notices](https://github.com/jensilo/acclass/blob/main/legal/ATTRIBUTIONS.md).