|
Download README.md from the-polysight/Polydec-base: direct link, hf CLI and curl.
- Browser
- Download file 5.22 kB
-
https://huggingface.co/the-polysight/Polydec-base/resolve/main/README.md
- Command line
-
hf download hf://the-polysight/Polydec-base/README.md
-
curl -L -o README.md https://huggingface.co/the-polysight/Polydec-base/resolve/main/README.md
5.22 kB
| license: gpl-3.0 | |
| language: | |
| - en | |
| base_model: answerdotai/ModernBERT-large | |
| pipeline_tag: text-classification | |
| tags: | |
| - decision-model | |
| - classification | |
| - calibration | |
| - modernbert | |
| - polydec | |
| datasets: | |
| - fancyzhx/ag_news | |
| - google/boolq | |
| - mteb/banking77 | |
| - clinc/clinc_oos | |
| - nyu-mll/multi_nli | |
| - tals/vitaminc | |
| - google/civil_comments | |
| - jackhhao/jailbreak-classification | |
| - nvidia/HelpSteer2 | |
| - DeepMostInnovations/saas-sales-conversations | |
| # polydec-base | |
| **polydec-base is the foundation model of ThePolysight.** | |
| It is a **decision model**, not a text generator. You give it a piece of text (a message, a document, a JSON record or | |
| a conversation) and one or more **typed questions**, and it answers all of them in a single call, with calibrated | |
| probabilities and a recommendation on whether to trust the answer. | |
| ## What it does | |
| | Question type | Answers | Example | | |
| |---|---|---| | |
| | `choice` | one option from a list, with a probability for each | intent, department, topic | | |
| | `score` | a level on an ordinal scale (expected value and most likely level) | urgency 0–2, quality 0–4 | | |
| | `noul` | a calibrated probability that a statement is true | spam, toxicity, "will this sale close?" | | |
| For every answer it also returns a decision: **`act`** (the answer is reliable enough to automate) or **`escalate`** | |
| (a person should review it). The questions are plain text, so the same model answers questions it has never seen, | |
| without retraining. | |
| ## Simple and portable | |
| - **One small folder:** weights (`model.safetensors`, 1.6 GB), configuration, encoder configuration and tokenizer. | |
| Nothing else is downloaded at load time, so it works offline. | |
| - **One call, all answers:** the questions are scored in a single forward pass, with no token-by-token generation. | |
| About 30 ms per call on a laptop GPU (RTX 4060). | |
| - **Runs anywhere PyTorch runs:** it picks CUDA, Apple MPS or CPU automatically. | |
| - **Same interface for every product model:** fine-tuned models load and answer exactly like this one. | |
| ## Architecture | |
| | | | | |
| |---|---| | |
| | Encoder | `answerdotai/ModernBERT-large` (394.8 M parameters) | | |
| | Heads | 2-layer question head, option scorer and act/escalate head (26.5 M parameters) | | |
| | Total | **421.3 M parameters** | | |
| | Input | up to 512 tokens (text + question + options) | | |
| | Calibration | temperature scaling per question type and number of options | | |
| Each option is preceded by a `[MASK]` token; the model scores the hidden state of each `[MASK]` and turns the scores | |
| into calibrated probabilities. | |
| ## Training | |
| - **Data:** ~224,000 typed questions built from 10 public datasets: AG News, BoolQ, Banking77, CLINC150, MultiNLI, | |
| VitaminC, Civil Comments, Jailbreak Classification, HelpSteer2 and SaaS Sales Conversations. Instructions are | |
| paraphrased and long label lists are shortened with distractors, so the model learns to follow the question rather | |
| than a fixed label set. | |
| - **Objective:** strictly proper scoring rules (logarithmic, spherical and, for ordinal scales, RPS), so the predicted | |
| probabilities match observed frequencies. The act/escalate head is trained with a reward of +1 for a correct | |
| automated answer, −3 for a wrong one and −0.5 for escalating, so it automates only when it is likely to be right. | |
| - **Run:** 2 epochs, 7.9 hours on a single 8 GB GPU. | |
| ## Results | |
| Measured on 11,569 validation questions and on 2,116 questions from three **zero-shot** tasks never seen in training | |
| (emotion, SST-5 sentiment, prompt injections). | |
| | | Accuracy | NLL | ECE (calibration error) | | |
| |---|---|---|---| | |
| | Validation | **81.2%** | 0.457 | **0.015** | | |
| | Zero-shot | 48.8% | 1.298 | 0.099 | | |
| | Decision quality (validation) | | | |
| |---|---| | |
| | Accuracy when it decides to `act` | **89.5%** | | |
| | Share of answers escalated to a person | 25% | | |
| | Accuracy of the 50% most confident answers | **96.1%** | | |
| | Validation task | Accuracy | | |
| |---|---| | |
| | Banking77 (intent, 20 options) | 99.1% | | |
| | Jailbreak detection | 99.2% | | |
| | CLINC150 (intent) | 97.5% | | |
| | AG News (topic) | 93.2% | | |
| | Civil Comments (toxicity) | 89.7% | | |
| | MultiNLI (inference) | 88.9% | | |
| | VitaminC (fact checking) | 86.1% | | |
| | BoolQ (reading comprehension) | 85.7% | | |
| | SaaS sales (conversation outcome) | 62.9% | | |
| | HelpSteer2 (response quality, 0–4) | 56.8% | | |
| Its strength is **calibration**: when it says 90%, it is right about 90% of the time, and the act/escalate decision | |
| uses that to automate the easy cases and route the hard ones to a person. | |
| ## Limitations | |
| - **English only.** Accuracy in other languages has not been measured and is expected to be lower. | |
| - **General-purpose.** For a specific domain, fine-tune it; that is how our product models are built. | |
| - **Unseen questions are less reliable** than trained ones. Check them on a few known examples and rely on the | |
| `decision` field. | |
| - **Subjective scales** (response quality, conversation outcomes) remain hard: 57–63% accuracy. | |
| ## License | |
| polydec-base is licensed under the **GNU General Public License v3.0** (see `LICENSE` and `NOTICE`). It is a | |
| derivative of ModernBERT-large, licensed under the Apache License 2.0 (see `LICENSE-APACHE-2.0`). The training | |
| datasets keep their own licenses. | |
| Contact: adrian.b@thepolysight.com | |