mailroom-triage v1

A 12-head English billing-email triage artifact: one shared MiniLM encoder, one linear decision head per billing question, per-head temperature calibration, and an acceptance threshold.

This is a custom PyTorch artifact, not a standard transformers model. AutoModel.from_pretrained() will not load it directly. Use the companion code:

• Code: https://github.com/deepakchoudhary-dc/Mailroom-Triage
• Full model card: https://github.com/deepakchoudhary-dc/Mailroom-Triage/blob/main/reports/model_card.md

Scope

Intended for English business-email triage: routing, tagging, prioritisation, and review queues over billing-related mail. Input is subject, from, date and body.

Not a general email assistant, phishing detector, security classifier, or marketing/promotion classifier. Do not use it for automatic payment, credit, employment, insurance, or legal decisions.

Files

encoder/                 fine-tuned MiniLM encoder (safetensors)
tokenizer/               tokenizer files
heads.pt                 state dict: one linear head per registry key
registry.json            questions, kinds, options, default-head flags
temperatures.json        per-head calibration temperatures
thresholds.json          per-head operating thresholds
refit.json               provenance for the rare category#7 refit

The training config.json is deliberately not included: it contained local paths and is not read by the loader.

Quick start

git clone https://github.com/deepakchoudhary-dc/Mailroom-Triage.git
cd Mailroom-Triage
python -m venv .venv
# Windows: .venv\Scripts\activate
# macOS/Linux: source .venv/bin/activate
pip install -e .
python -c "from huggingface_hub import snapshot_download; snapshot_download('allen65/Mailroom-Triage', local_dir='artifacts/model')"
python -m mailroom_triage.serve

Open http://127.0.0.1:8000/docs. The server binds IPv4 only by default.

curl -X POST http://127.0.0.1:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"subject":"Invoice INV-1","from":"billing@luma.example","body":"Amount now owed: EUR 42.00","heads":["kind","states_amount_owed"]}'

heads is optional; omit it for all 12. POST /predict/batch accepts up to 256 emails. POST /predict returns the selected label, calibrated probabilities, and accepted against the fitted threshold.

Evaluation

Split	Rows	Micro accuracy
test (iid synthetic)	8,323	1.0000
OOD	22,960	0.9978
English OOD	7,760	0.9997

Formatting stress on 3,000 validation rows: clean/HTML/quoted/signature/forwarded 100%, line reordering 99.97%, typos 99.97%, case noise 99.83%. A deliberate truncation control drops to 81.20%, showing the cost of missing evidence.

The 100% test figure is an iid synthetic result, not a real-world accuracy claim. OOD was used to diagnose and fix a line-order failure, so the final OOD number is optimistic for arbitrary distribution shift. The non-tuning evidence for order robustness is the validation stress split (99.97%).

Known limitations

• Synthetic, explicit-evidence data. Natural mail with implicit or conflicting evidence is harder.
• 320-token limit. Long bodies are trimmed; tail evidence may be invisible. Chunk long mail before production.
• English focus. Other languages were present in training but are not a supported SLA.
• Billing vocabulary can override intent. A bank rewards promotion was read as kind=invoice with is_promotional=no; a security alert with an account number was also flipped to invoice. Do not use this artifact for phishing, security, or promotion decisions.
• Missing from is treated as empty. Send a real sender value.
• Threshold is not an uncertainty filter on clean synthetic data; all 12 thresholds fit to 0.50 and calibration accuracy is 100%.
• No auth, rate limit, TLS, or drift monitoring in the sample service.

License

• Model weights: Apache-2.0, as a derivative of sentence-transformers/all-MiniLM-L6-v2.
• Companion code: MIT.
• Dataset: original generated rows are CC0-1.0; see ZefanCai/Open-Jev, config mailroom-control-v1.

See LICENSE and NOTICE in this repository.