ProCreations/auto-0.4b-gguf

auto-0.4b — GGUF

llama.cpp builds of ProCreations/auto-0.4b, the 0.4B encoder that gates an AI agent's tool calls.

file	quant
auto-0.4b-Q4_K_M.gguf	260 MiB
auto-0.4b-Q5_K_M.gguf	289 MiB
auto-0.4b-Q6_K.gguf	329 MiB
auto-0.4b-Q8_0.gguf	403 MiB
auto-0.4b-f16.gguf	757 MiB

⚠️ Read this before using these files for classification

llama.cpp converts this model correctly — it maps ModernBertForSequenceClassification onto the modern-bert architecture, all 174 tensors including the classifier head (cls.weight, cls.norm.weight, cls.output.weight/bias) are present, and the approve/deny labels are carried in the GGUF metadata.

But as of the llama.cpp build used here, the classifier head does not produce usable scores. --pooling rank returns ~0.000 for both classes regardless of input, which matches what you would see if the pooled vector never reached the head (the output equals the classifier bias). Verified against PyTorch on identical input, where the same weights separate the classes correctly.

What does work is the encoder:

# correct 1024-d embeddings, verified non-degenerate
llama-embedding -m auto-0.4b-Q8_0.gguf --pooling cls -p "<the serialized input>"

So today these files are useful as a ModernBERT encoder in the llama.cpp ecosystem, and as a ready-to-go artifact if/when llama.cpp wires up multi-class heads. For actual approve/deny decisions right now, use the PyTorch or ONNX builds — the ONNX int8 file is 379 MB and matches PyTorch to 2.4e-06.

If you want to classify from the GGUF encoder, take the CLS embedding above and apply the head yourself; the head weights are tiny and published in the main repo.

Q8_0 is the sensible default for a 0.4B safety classifier — quantisation error here means a dangerous call slipping through, so do not drop to Q4_K_M without measuring.