Tynapse/drift-sentry-4b-v1

DriftSentry-4B-v1

DriftSentry-4B-v1 is a 4B behavioral-safety judge for AI-agent responses. It classifies one primary failure tier from six risk categories—goal, reasoning, environment, integration, memory, and reward—or returns normal for a passing response.

This repository contains both the merged model at the repository root and the original LoRA adapter under adapter/.

• Benchmark: Tynapse/drift-sentry-bench-50k-v1
• Reproduction code: Tynapse/drift-sentry

Model identity

• Base model: Qwen/Qwen3.5-4B
• Base revision: 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a
• Fine-tuning: off-policy sequence-level distillation with supervised fine-tuning
• LoRA: rank 32, alpha 64, dropout 0.05
• Adapter SHA-256: 6fc9c40fa24fa03d648b4a017d4ce8b5fe948e632633dac5528d746616c44306
• Merged model.safetensors SHA-256: a2bc5669196283217881e52bbdb28423aac7506797b9bc1f417bf9798d9bac6d

The base revision is published under Apache-2.0. This derivative model is released under the same license.

Output schema

The judge6.label.v1 value below is the immutable v1 wire-format identifier retained for compatibility and reproduction. It does not denote the public model name.

The model emits a JSON object with these fields:

{
  "schema_version": "judge6.label.v1",
  "verdict": "PASS | BLOCK | ESCALATE",
  "tier": "goal | reasoning | environment | integration | memory | reward | normal",
  "secondary_tiers": [],
  "severity": "none | low | medium | high",
  "confidence": 0.0,
  "rationale": "..."
}

The complete taxonomy is preserved in taxonomy.yaml.

Evaluation

The final model was evaluated on the 50,000-row source benchmark with temperature 0, JSON response format, and a 512-token output cap.

Metric	Measured value
Accuracy	0.87346
6-tier Macro-F1	0.8600247542914515
7-way Macro-F1	0.8733699133625255
Memory recall	0.9385412291754165
Reward recall	0.9134817303653927
Combined high-risk recall	0.9260114797704045
Parse-failure rate	0.00122
Request failures	0

The path- and endpoint-free metrics and provenance records are under evaluation/. Before publication, 324 credential-shaped synthetic strings in 123 prompts were replaced with deterministic nonfunctional placeholders. IDs and gold labels did not change, but those 123 public prompts were not re-inferred. The reported metrics are therefore exact for the pre-publication 50,000-row source, while 49,877 public rows remain message-identical. Both input hashes and this scope limitation are recorded in the evidence.

The public evidence records the served name drift-sentry-4b-v1. Internal endpoint URLs, host identifiers, and absolute filesystem paths are retained only in the private evidence archive.

Dataset-quality audit

On 149,515 semantic-corpus rows that retained both an assigned target tier and the teacher's unmodified original tier, target-versus-teacher Cohen's kappa was 0.9781300729494184 (146,769 agreements, 2,746 disagreements). The 459 rows without a valid raw teacher tier were excluded rather than imputed. The exact method and confusion matrix are in evaluation/label_consistency_audit.json. This is not a human inter-annotator-agreement claim.

Operational latency

On one NVIDIA H200 with vLLM, batch 1, concurrency 1, a 2,048-token rendered prompt, and max_tokens=512, 1,000 measured requests succeeded with no failures. Natural completion length was 166 tokens and p95 latency was 664.2498452 ms. This is a capped-output production measurement, not a forced 512-token stress test.

Serving with vLLM

vllm serve Tynapse/drift-sentry-4b-v1 \
  --served-model-name drift-sentry-4b-v1 \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gdn-prefill-backend triton

Send the policy or taxonomy instructions as a system message, followed by the agent context and candidate response as user content. Use temperature 0 and a JSON response format where supported.

Using the LoRA adapter

The unmerged adapter is stored under adapter/. Its adapter_config.json pins the same base revision shown above. The merged root model and adapter represent the same selected checkpoint.

Training data and method

The student was trained on synthetic behavioral-safety conversations produced by the single teacher Qwen/Qwen3.6-27B at revision 6a9e13bd6fc8f0983b9b99948120bc37f49c13e9 and labeled using the DriftSentry taxonomy. Training, validation, and semantic-test archetype families were separated. The public 50,000-row evaluation benchmark was not used as model training data.

This release does not claim multi-teacher cross-tokenizer or on-policy logit distillation.

Limitations and responsible use

• Labels are synthetic teacher labels, not a fully human-adjudicated gold standard.
• The benchmark is intended for controlled behavioral-safety evaluation; it is not evidence of universal robustness across agents, domains, languages, or tools.
• The model can make false-positive and false-negative decisions. Do not use it as the sole enforcement mechanism for high-impact decisions.
• Safety examples may contain harmful, deceptive, privacy-sensitive, or policy-violating scenarios. Treat inputs and outputs as sensitive research content.
• A production integration should validate JSON parsing, retain audit logs, and define a safe fallback for malformed or uncertain outputs.

Acknowledgement

This work was supported by the NIPA Advanced GPU Utilization Support Program, project no. 04-26-03-0029.

Files

• model.safetensors: merged model
• adapter/: original LoRA adapter and configuration
• taxonomy.yaml: six-tier taxonomy and output schema
• evaluation/: exact metrics, provenance, and H200 operational-latency evidence
• release_manifest.json: model-file hashes and base revision