Link explainer Space

087c1da verified about 7 hours ago

5.57 kB

---
license: apache-2.0
language:
- en
base_model: answerdotai/ModernBERT-base
datasets:
- danielrosehill/Agent-Task-Break-Detector
pipeline_tag: text-classification
library_name: transformers
tags:
- agents
- context-management
- topic-shift
- cross-encoder
- onnx
---
# Agent Task Break Detector (ModernBERT-base)

A small cross-encoder that answers one question on every user turn of an AI agent session:

> **Does this new message need the conversation so far, or is it a new task that a fresh
> session could handle just as well?**

It outputs `p(new_task)`. A harness hook can use it to start a fresh session (`/clear`, `/new`)
automatically when the user drops an unrelated task into a long session. It is meant to run
**locally on every turn**, so it is a 150M-parameter encoder rather than an API call.

**Try it:** [explainer and live demo Space](https://huggingface.co/spaces/danielrosehill/Agent-Task-Break-Detector) (int8 ONNX on CPU).

**Status: v0.1, 2026-10-24. Experimental.** Trained only on synthetic data. It has not yet been
evaluated on real, hand-labelled sessions.

## Files

| Path | What |
|---|---|
| `model.safetensors`, `config.json`, `tokenizer*` | Fine-tuned weights (fp32), for `transformers` |
| `new_task_split.json` | Recommended int8 threshold and input settings (read by the New-Task-Split plugin) |
| `onnx/model_int8.onnx` + tokenizer | Dynamic int8 quantisation (onnxruntime `quantize_dynamic`), 151 MB, for CPU inference |

## Input format

The model was trained on pairs `(message, context)`:

- **First segment:** the new user message.
- **Second segment:** the last 4 exchanges, **newest first**, each written as
  `AGENT: <agent turn>\nUSER: <user turn>`, joined by newlines.
- `truncation="only_second"`, `max_length=512`. Label 1 = `new_task`, 0 = `continue`.

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
repo = "danielrosehill/Agent-Task-Break-Detector-ModernBERT"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()
turns = [{"user": "the payments migration fails on the currency column", "agent": "Added a default of 'USD' in migration 0042 and re-ran it; it passes."}]
ctx = "\n".join(f"AGENT: {t['agent']}\nUSER: {t['user']}" for t in turns[-4:][::-1])
enc = tok("now backfill the currency column for old rows", ctx,
          truncation="only_second", max_length=512, return_tensors="pt")
with torch.no_grad():
    p_new_task = model(**enc).logits.softmax(-1)[0, 1].item()
```

ONNX (inputs `input_ids`, `attention_mask`; output `logits`):

```python
import onnxruntime as ort, numpy as np
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download
repo = "danielrosehill/Agent-Task-Break-Detector-ModernBERT"
tok = AutoTokenizer.from_pretrained(repo, subfolder="onnx")
sess = ort.InferenceSession(hf_hub_download(repo, "onnx/model_int8.onnx"))
enc = tok(message, ctx, truncation="only_second", max_length=512, return_tensors="np")
logits = sess.run(None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0]
p_new_task = float(np.exp(logits[0, 1]) / np.exp(logits[0]).sum())
```

## Training

- Base: `answerdotai/ModernBERT-base`.
- Data: [danielrosehill/Agent-Task-Break-Detector](https://huggingface.co/datasets/danielrosehill/Agent-Task-Break-Detector)
  v0.2, rows with `agreed == true` (5,104 train rows, including 1,025 matched pairs).
- 3 epochs, lr 5e-5, batch 16, bf16, warmup 10%, best epoch by validation AUC-PR. One NVIDIA L4
  on Modal, 6.9 minutes. Run `atbd-r3-modernbert-full3ep`.
- Code: [danielrosehill/Agent-Context-Harness](https://github.com/danielrosehill/Agent-Context-Harness),
  `model/train.py`.

## Results (synthetic test split, fp32)

n = 608 agreed rows, 115 matched pairs. A matched pair is the same message once as a new task and
once, in a different context, as a continuation, so only the context can separate them.

| | Accuracy | AUC-PR | `new_task` recall @ precision 0.95 | Pair accuracy |
|---|---|---|---|---|
| **This model** | 0.951 | 0.986 | 0.902 | 0.791 |
| TF-IDF, message only | 0.793 | 0.758 | 0.037 | — |
| TF-IDF, message + context | 0.760 | 0.781 | 0.140 | — |

The weakest cases are a message that looks self-contained but continues the work
(`counterfactual_continue`, 0.896) and a different task in the same workspace (`same_workspace`,
0.867).

**int8 ONNX** (validation n = 697, test n = 608; tokenised with `tokenizers`, identical ids to training):

| | Accuracy | AUC-PR | `new_task` recall @ precision 0.95 |
|---|---|---|---|
| validation | 0.937 | 0.988 | 0.925 |
| test | 0.929 | 0.982 | 0.888 |

int8 probabilities sit lower than fp32 (mean shift 0.14), so **use threshold 0.76 for int8**, not the
fp32 value. At the validation threshold of 0.751, test precision is 0.941 and recall 0.893. This is recorded in
`new_task_split.json`. Session load takes about 1.2 s on CPU; per-call latency has not been measured yet.

## Intended use and limitations

- The operating point should be **high precision on `new_task`**: a wrong split throws away the
  agent's working context, while a missed split only costs extra context. Pick a threshold for
  precision ≥ 0.95 on your own data.
- Synthetic training data from one generator (Claude Haiku 5.5), verified by Claude Sonnet 5.5.
  Real sessions are longer and messier. English only.
- The training data is about 50/50; in real use most turns are continuations.
## Licence
Apache-2.0 (as the base model). Training data CC BY 4.0.

	---
	license: apache-2.0
	language:
	- en
	base_model: answerdotai/ModernBERT-base
	datasets:
	- danielrosehill/Agent-Task-Break-Detector
	pipeline_tag: text-classification
	library_name: transformers
	tags:
	- agents
	- context-management
	- topic-shift
	- cross-encoder
	- onnx
	---

	# Agent Task Break Detector (ModernBERT-base)

	A small cross-encoder that answers one question on every user turn of an AI agent session:

	> **Does this new message need the conversation so far, or is it a new task that a fresh
	> session could handle just as well?**

	It outputs `p(new_task)`. A harness hook can use it to start a fresh session (`/clear`, `/new`)
	automatically when the user drops an unrelated task into a long session. It is meant to run
	locally on every turn, so it is a 150M-parameter encoder rather than an API call.

	Try it: [explainer and live demo Space](https://huggingface.co/spaces/danielrosehill/Agent-Task-Break-Detector) (int8 ONNX on CPU).

	Status: v0.1, 2026-10-24. Experimental. Trained only on synthetic data. It has not yet been
	evaluated on real, hand-labelled sessions.

	## Files

	\| Path \| What \|
	\|---\|---\|
	\| `model.safetensors`, `config.json`, `tokenizer*` \| Fine-tuned weights (fp32), for `transformers` \|
	\| `new_task_split.json` \| Recommended int8 threshold and input settings (read by the New-Task-Split plugin) \|
	\| `onnx/model_int8.onnx` + tokenizer \| Dynamic int8 quantisation (onnxruntime `quantize_dynamic`), 151 MB, for CPU inference \|

	## Input format

	The model was trained on pairs `(message, context)`:

	- First segment: the new user message.
	- Second segment: the last 4 exchanges, newest first, each written as
	`AGENT: <agent turn>\nUSER: <user turn>`, joined by newlines.
	- `truncation="only_second"`, `max_length=512`. Label 1 = `new_task`, 0 = `continue`.

	```python
	from transformers import AutoTokenizer, AutoModelForSequenceClassification
	import torch

	repo = "danielrosehill/Agent-Task-Break-Detector-ModernBERT"
	tok = AutoTokenizer.from_pretrained(repo)
	model = AutoModelForSequenceClassification.from_pretrained(repo).eval()

	turns = [{"user": "the payments migration fails on the currency column", "agent": "Added a default of 'USD' in migration 0042 and re-ran it; it passes."}]
	ctx = "\n".join(f"AGENT: {t['agent']}\nUSER: {t['user']}" for t in turns[-4:][::-1])
	enc = tok("now backfill the currency column for old rows", ctx,
	truncation="only_second", max_length=512, return_tensors="pt")
	with torch.no_grad():
	p_new_task = model(**enc).logits.softmax(-1)[0, 1].item()
	```

	ONNX (inputs `input_ids`, `attention_mask`; output `logits`):

	```python
	import onnxruntime as ort, numpy as np
	from transformers import AutoTokenizer
	from huggingface_hub import hf_hub_download

	repo = "danielrosehill/Agent-Task-Break-Detector-ModernBERT"
	tok = AutoTokenizer.from_pretrained(repo, subfolder="onnx")
	sess = ort.InferenceSession(hf_hub_download(repo, "onnx/model_int8.onnx"))
	enc = tok(message, ctx, truncation="only_second", max_length=512, return_tensors="np")
	logits = sess.run(None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0]
	p_new_task = float(np.exp(logits[0, 1]) / np.exp(logits[0]).sum())
	```

	## Training

	- Base: `answerdotai/ModernBERT-base`.
	- Data: [danielrosehill/Agent-Task-Break-Detector](https://huggingface.co/datasets/danielrosehill/Agent-Task-Break-Detector)
	v0.2, rows with `agreed == true` (5,104 train rows, including 1,025 matched pairs).
	- 3 epochs, lr 5e-5, batch 16, bf16, warmup 10%, best epoch by validation AUC-PR. One NVIDIA L4
	on Modal, 6.9 minutes. Run `atbd-r3-modernbert-full3ep`.
	- Code: [danielrosehill/Agent-Context-Harness](https://github.com/danielrosehill/Agent-Context-Harness),
	`model/train.py`.

	## Results (synthetic test split, fp32)

	n = 608 agreed rows, 115 matched pairs. A matched pair is the same message once as a new task and
	once, in a different context, as a continuation, so only the context can separate them.

	\| \| Accuracy \| AUC-PR \| `new_task` recall @ precision 0.95 \| Pair accuracy \|
	\|---\|---\|---\|---\|---\|
	\| This model \| 0.951 \| 0.986 \| 0.902 \| 0.791 \|
	\| TF-IDF, message only \| 0.793 \| 0.758 \| 0.037 \| — \|
	\| TF-IDF, message + context \| 0.760 \| 0.781 \| 0.140 \| — \|

	The weakest cases are a message that looks self-contained but continues the work
	(`counterfactual_continue`, 0.896) and a different task in the same workspace (`same_workspace`,
	0.867).

	int8 ONNX (validation n = 697, test n = 608; tokenised with `tokenizers`, identical ids to training):

	\| \| Accuracy \| AUC-PR \| `new_task` recall @ precision 0.95 \|
	\|---\|---\|---\|---\|
	\| validation \| 0.937 \| 0.988 \| 0.925 \|
	\| test \| 0.929 \| 0.982 \| 0.888 \|

	int8 probabilities sit lower than fp32 (mean shift 0.14), so use threshold 0.76 for int8, not the
	fp32 value. At the validation threshold of 0.751, test precision is 0.941 and recall 0.893. This is recorded in
	`new_task_split.json`. Session load takes about 1.2 s on CPU; per-call latency has not been measured yet.

	## Intended use and limitations

	- The operating point should be high precision on `new_task`: a wrong split throws away the
	agent's working context, while a missed split only costs extra context. Pick a threshold for
	precision ≥ 0.95 on your own data.
	- Synthetic training data from one generator (Claude Haiku 5.5), verified by Claude Sonnet 5.5.
	Real sessions are longer and messier. English only.
	- The training data is about 50/50; in real use most turns are continuations.

	## Licence

	Apache-2.0 (as the base model). Training data CC BY 4.0.