README.md

---
language:
- en
license: mit
library_name: sentence-transformers
pipeline_tag: text-classification
base_model: BAAI/bge-base-en-v1.5
tags:
- intent-classification
- multi-label-classification
- voice-notes
- asr
- routing
- mcp
- agents
---

# Voice-Note-Intent-Classifier

> **v0.1.0 (2026-10-15): placeholder release.** A first, deliberately simple model
> that works end to end and sets a baseline. It is not tuned, and its accuracy on real
> notes is modest (micro F1 0.56). Expect it to be replaced; the
> `taxonomy_version` in every prediction lets a router detect that.

A small multi-label classifier that reads a voice-note **transcript** and predicts
its **intent** (shopping list, to-do, calendar event, email draft, AI prompt and so
on), so that an agent system can route it to the right integration.

## Use case

```
voice note ─► ASR ─► transcript ─► [ this classifier ] ─► router
                                                            │  one dispatch per predicted label
              ┌──────────────┬──────────────┬───────────────┼──────────────┐
              ▼              ▼              ▼               ▼              ▼
         shopping-list   to-do         calendar         email          AI prompt  …
           subagent     subagent       subagent        subagent        subagent
```

Each subagent holds an MCP connection to its service and extracts the details it
needs. The classifier only decides where a note goes.

## How to use

```bash
pip install sentence-transformers numpy huggingface_hub
hf download danielrosehill/Voice-Note-Intent-Classifier --local-dir vnic
python vnic/predict.py "um need to grab oat milk eggs and uh dish soap"
```

```json
{"model_version": "0.1.0", "taxonomy_version": "0.1.0",
 "scores": {"shopping_list": 0.88, "todo": 0.54, "...": "..."},
 "labels": ["shopping_list", "todo"], "abstain": false}
```

`abstain: true` means no label cleared its threshold. The router should then fall
back to an LLM or ask the user.

## What it is

- **Encoder:** frozen [`BAAI/bge-base-en-v1.5`](https://huggingface.co/BAAI/bge-base-en-v1.5)
  sentence embeddings (normalised, 512 tokens).
- **Head:** one logistic regression per label, stored as plain coefficients in
  `head.json`. Per-label thresholds were calibrated on real notes.
- **Files:** `head.json` (head, thresholds, config), `predict.py` (inference),
  `labels.json` (taxonomy v0.1.0 with definitions), `eval_results.json`.

## Labels (taxonomy v0.1.0)

| Label | Meaning |
|---|---|
| `shopping_list` | Items to buy |
| `todo` | Tasks to do, with no fixed time slot |
| `note_to_self` | Information to remember; nothing to do |
| `calendar_event` | Something at a specific date/time that occupies a slot |
| `email_draft` | Text to send as an email, or an instruction to write one |
| `idea` | A single discrete idea, stated briefly |
| `brainstorm` | Extended exploratory thinking across options |
| `content_draft` | Material for publication: blog post, article, script, outline |
| `ai_prompt` | Instructions addressed to an AI model or agent |

## Training data

- **Synthetic (1492 transcripts).** LLM-generated voice notes
  styled as raw ASR output: single intents, compound pairs, hard cases on confusable
  labels, and notes that fit no label. Generated by one model family (Gemini) and
  kept only where two other families (Claude, GPT), labelling blind, both agreed with
  the requested labels. This dataset is private for now.
- **Real (226 notes).** The author's own voice notes, screened for
  privacy with personal details replaced by fake names, labelled where two LLM
  families agreed. The notes the author has approved for release are public at
  [`danielrosehill/Voice-Notes-Screened`](https://huggingface.co/datasets/danielrosehill/Voice-Notes-Screened).

## Evaluation

5-fold cross-validation on the 226 real notes: each fold's head was
trained on synthetic data plus the other four folds and scored on notes it never saw.

| Metric | Value |
|---|---|
| Micro precision / recall / F1 | 0.50 / 0.64 / 0.56 |
| Macro F1 (labels present) | 0.48 |
| Exact match | 0.39 |
| Notes with no intent, correctly abstained | 10/26 |

| Label | F1 |
|---|---|
| `shopping_list` | 0.78 |
| `todo` | 0.41 |
| `note_to_self` | 0.23 |
| `email_draft` | 0.50 |
| `idea` | 0.36 |
| `brainstorm` | 0.45 |
| `content_draft` | 0.35 |
| `ai_prompt` | 0.76 |

`calendar_event` had no examples among the real notes, so it is unscored on real data.

**Caveat:** the real labels are agreement between two LLMs, not hand labels. The two
LLMs gave identical labels on only 63% of the real notes, so these numbers are a
rough guide, not a benchmark.

## Fit and limitations

- **Built around one person's notes.** The real data comes from a single speaker
  whose notes are dominated by long spoken prompts to AI coding agents. Shopping
  lists and AI prompts work best. Other intents, and other people's dictation
  habits, are weaker.
- **Long agent-style prompts are sometimes labelled `todo`**, because the
  instructions describe tasks.
- **English only.** Hebrew and English-Hebrew code-switched notes are out of scope.
- **Adapting it:** retrain the head on your own transcripts. That takes seconds on
  a CPU because the encoder is frozen. Or swap in the planned fine-tuned encoder
  when it ships.

## Roadmap

| # | Phase | Status |
|---|---|---|
| 0 | Taxonomy definition | draft v0.1.0 |
| 1 | Baselines on real notes | v0.1.0 (this release) |
| 2 | Synthetic data v2 (length-matched to real notes) | generated, not yet trained on |
| 3 | Fine-tuned encoder (ModernBERT-base) | planned |
| 4 | Hand-labelled real evaluation set | planned |
| 5 | ONNX export | planned |
| 6 | Integration into a voice-to-action router | planned |