akifkulll/mayi-distilbert-b

distilbert-base-uncased fine-tuned to decide whether an AI agent's proposed tool call should be allowed (0) or denied (1). Input format B: the tool call, then the user's request, then the agent's earlier steps (max 512 tokens; the end of the history is cut).

Trained for the COE025 NLP course project (Istinye University) on the cleaned English split of auto-1b-data (cleaning and splits).

Test results (threshold 0.5)

rows accuracy macro-F1 false allow rate false deny rate AUROC
71613 0.9607 0.9607 0.0405 0.0381 0.9929

Use

from transformers import pipeline
clf = pipeline("text-classification", model="akifkulll/mayi-distilbert-b")
clf("### PROPOSED TOOL CALL\ntool: shell\nargs: rm -rf node_modules ...")

Labels were generated by AI models; expect weaker results on inputs written in a different format.

Downloads last month
20
Safetensors
Model size
67M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for akifkulll/mayi-distilbert-b

Finetuned
(12,888)
this model

Dataset used to train akifkulll/mayi-distilbert-b