Polydec-base / README.md
Abayona-PS's picture
Upload folder using huggingface_hub
42efc42 verified
|
Raw History Blame Contribute Delete
5.22 kB
---
license: gpl-3.0
language:
- en
base_model: answerdotai/ModernBERT-large
pipeline_tag: text-classification
tags:
- decision-model
- classification
- calibration
- modernbert
- polydec
datasets:
- fancyzhx/ag_news
- google/boolq
- mteb/banking77
- clinc/clinc_oos
- nyu-mll/multi_nli
- tals/vitaminc
- google/civil_comments
- jackhhao/jailbreak-classification
- nvidia/HelpSteer2
- DeepMostInnovations/saas-sales-conversations
---
# polydec-base
**polydec-base is the foundation model of ThePolysight.**
It is a **decision model**, not a text generator. You give it a piece of text (a message, a document, a JSON record or
a conversation) and one or more **typed questions**, and it answers all of them in a single call, with calibrated
probabilities and a recommendation on whether to trust the answer.
## What it does
| Question type | Answers | Example |
|---|---|---|
| `choice` | one option from a list, with a probability for each | intent, department, topic |
| `score` | a level on an ordinal scale (expected value and most likely level) | urgency 0–2, quality 0–4 |
| `noul` | a calibrated probability that a statement is true | spam, toxicity, "will this sale close?" |
For every answer it also returns a decision: **`act`** (the answer is reliable enough to automate) or **`escalate`**
(a person should review it). The questions are plain text, so the same model answers questions it has never seen,
without retraining.
## Simple and portable
- **One small folder:** weights (`model.safetensors`, 1.6 GB), configuration, encoder configuration and tokenizer.
Nothing else is downloaded at load time, so it works offline.
- **One call, all answers:** the questions are scored in a single forward pass, with no token-by-token generation.
About 30 ms per call on a laptop GPU (RTX 4060).
- **Runs anywhere PyTorch runs:** it picks CUDA, Apple MPS or CPU automatically.
- **Same interface for every product model:** fine-tuned models load and answer exactly like this one.
## Architecture
| | |
|---|---|
| Encoder | `answerdotai/ModernBERT-large` (394.8 M parameters) |
| Heads | 2-layer question head, option scorer and act/escalate head (26.5 M parameters) |
| Total | **421.3 M parameters** |
| Input | up to 512 tokens (text + question + options) |
| Calibration | temperature scaling per question type and number of options |
Each option is preceded by a `[MASK]` token; the model scores the hidden state of each `[MASK]` and turns the scores
into calibrated probabilities.
## Training
- **Data:** ~224,000 typed questions built from 10 public datasets: AG News, BoolQ, Banking77, CLINC150, MultiNLI,
VitaminC, Civil Comments, Jailbreak Classification, HelpSteer2 and SaaS Sales Conversations. Instructions are
paraphrased and long label lists are shortened with distractors, so the model learns to follow the question rather
than a fixed label set.
- **Objective:** strictly proper scoring rules (logarithmic, spherical and, for ordinal scales, RPS), so the predicted
probabilities match observed frequencies. The act/escalate head is trained with a reward of +1 for a correct
automated answer, −3 for a wrong one and −0.5 for escalating, so it automates only when it is likely to be right.
- **Run:** 2 epochs, 7.9 hours on a single 8 GB GPU.
## Results
Measured on 11,569 validation questions and on 2,116 questions from three **zero-shot** tasks never seen in training
(emotion, SST-5 sentiment, prompt injections).
| | Accuracy | NLL | ECE (calibration error) |
|---|---|---|---|
| Validation | **81.2%** | 0.457 | **0.015** |
| Zero-shot | 48.8% | 1.298 | 0.099 |
| Decision quality (validation) | |
|---|---|
| Accuracy when it decides to `act` | **89.5%** |
| Share of answers escalated to a person | 25% |
| Accuracy of the 50% most confident answers | **96.1%** |
| Validation task | Accuracy |
|---|---|
| Banking77 (intent, 20 options) | 99.1% |
| Jailbreak detection | 99.2% |
| CLINC150 (intent) | 97.5% |
| AG News (topic) | 93.2% |
| Civil Comments (toxicity) | 89.7% |
| MultiNLI (inference) | 88.9% |
| VitaminC (fact checking) | 86.1% |
| BoolQ (reading comprehension) | 85.7% |
| SaaS sales (conversation outcome) | 62.9% |
| HelpSteer2 (response quality, 0–4) | 56.8% |
Its strength is **calibration**: when it says 90%, it is right about 90% of the time, and the act/escalate decision
uses that to automate the easy cases and route the hard ones to a person.
## Limitations
- **English only.** Accuracy in other languages has not been measured and is expected to be lower.
- **General-purpose.** For a specific domain, fine-tune it; that is how our product models are built.
- **Unseen questions are less reliable** than trained ones. Check them on a few known examples and rely on the
`decision` field.
- **Subjective scales** (response quality, conversation outcomes) remain hard: 57–63% accuracy.
## License
polydec-base is licensed under the **GNU General Public License v3.0** (see `LICENSE` and `NOTICE`). It is a
derivative of ModernBERT-large, licensed under the Apache License 2.0 (see `LICENSE-APACHE-2.0`). The training
datasets keep their own licenses.
Contact: adrian.b@thepolysight.com