Text Classification
Transformers
Safetensors
English
qwen3
text-generation
qwen
decision-model
finance
wealth-management
sec-filings
news
central-banks
calibration
lora
jev
text-embeddings-inference
Instructions to use ternary/Arbiter-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ternary/Arbiter-8B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="ternary/Arbiter-8B")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("ternary/Arbiter-8B") model = AutoModelForCausalLM.from_pretrained("ternary/Arbiter-8B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Model card: examples for implied sentiment and event classification; cleanup
Browse files
README.md
CHANGED
|
@@ -98,13 +98,47 @@ answer = arbiter.decide(state=passage, question=OUTLOOK, focus="Constellium")
|
|
| 98 |
print(answer["choice"]) # e.g. "improved"
|
| 99 |
print(answer["probabilities"]) # {"improved": ..., "worsened": ..., "unchanged": ..., "unclear": ...}
|
| 100 |
|
| 101 |
-
answer = arbiter.decide(state="
|
| 102 |
question=FOMC_ORIGINAL, focus="Federal Reserve (FOMC)")
|
| 103 |
print(answer["choice"]) # e.g. "hawkish"
|
| 104 |
```
|
| 105 |
|
| 106 |
`decide()` returns a typed answer: `{"type": "choice", "choice": ..., "probabilities": {...}}` for choice questions, `{"type": "noul", "noul": P(true), ...}` for yes/no questions, and `{"type": "score", "score": level, "probabilities": {...}}` for ordered scales. If a state is too long, it is shortened from the end (never the options), and the answer reports `state_truncated: true`. `arbiter.fallback_counts` counts every fallback taken.
|
| 107 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
## Requirements
|
| 109 |
|
| 110 |
- **Python** 3.10 or later, with `torch`, `transformers`, `accelerate`, `safetensors` and `huggingface_hub`.
|
|
@@ -114,25 +148,6 @@ print(answer["choice"]) # e.g. "hawkish"
|
|
| 114 |
|
| 115 |
Load the model through `arbiter.py` as shown above. The backbone also loads with plain `AutoModelForCausalLM.from_pretrained`, but typed decisions need the decision head, which `arbiter.py` applies. The Hub's automatic code snippet labels this model `text-classification`; it doesn't apply.
|
| 116 |
|
| 117 |
-
## Questions it was trained on
|
| 118 |
-
|
| 119 |
-
Import the exact definitions from `questions.py`; Arbiter is most reliable with them.
|
| 120 |
-
|
| 121 |
-
| Constant | Type | Options |
|
| 122 |
-
|---|---|---|
|
| 123 |
-
| `EVENT_TYPE` | choice | corporate_action, financial_results, company_guidance, product_operations, analyst_action, analyst_opinion, market_move, legal_safety, industry_macro, insufficient_evidence |
|
| 124 |
-
| `OUTLOOK` | choice | improved, worsened, unchanged, unclear |
|
| 125 |
-
| `CERTAINTY` | score | 0 rumoured … 3 realised |
|
| 126 |
-
| `MAIN_SUBJECT` | yes/no | is the company the main subject of the event? |
|
| 127 |
-
| `ROLE` | choice | acquirer, target, regulator, mentioned_only |
|
| 128 |
-
| `FILING_RISK` | choice | ten risk categories for an Item 1A passage |
|
| 129 |
-
| `FILING_LEGAL` | score | 0 possible … 3 resolved |
|
| 130 |
-
| `GOING_CONCERN`, `CUSTOMER_CONCENTRATION`, `CYBER_INCIDENT` | yes/no | 10-K flags |
|
| 131 |
-
| `EDT_ORIGINAL`, `EDT_TIGHT` (new) | choice | acquisition, clinical_trial, regular_dividend, dividend_cut, dividend_increase, guidance_change, new_contract, reverse_stock_split, special_dividend, stock_repurchase, stock_split, none |
|
| 132 |
-
| `FOMC_ORIGINAL`, `FOMC_TIGHT` (new) | choice | hawkish, neutral, dovish |
|
| 133 |
-
| `SENTIMENT_ORIGINAL`, `SENTIMENT_IMPLIED`, `SENTIMENT_STATED` (new) | choice | positive, neutral, negative |
|
| 134 |
-
|
| 135 |
-
The `_TIGHT`, `_IMPLIED` and `_STATED` variants spell out a labelling convention in the definitions, for example that a partnership is not a new contract. v0.5 was trained on both wordings of each question so that it follows whichever definition it is given; tighter definitions usually score higher (SEntFiN, F16 GGUF: 81.9% tightened vs 76.8% original).
|
| 136 |
|
| 137 |
You can pass your own question in the same shape (`{"type", "instructions", "criteria"}`, at most 16 options). The base model's general decision ability carries over, but quality on questions outside this list has not been measured.
|
| 138 |
|
|
@@ -151,52 +166,7 @@ You can pass your own question in the same shape (`{"type", "instructions", "cri
|
|
| 151 |
|
| 152 |
95% confidence half-widths: SEntFiN ±0.7, EDT ±1.1, FOMC ±4.2 points. Original definitions, options in the published order. EDT and FOMC are scored on their held-out test splits. SEntFiN was never trained on. Paired McNemar tests against base Qwen3-8B on the same items: FOMC +11.3 points (z 4.9), EDT +4.4 (z 9.1), SEntFiN +1.9 (z 5.9), all p < 0.001.
|
| 153 |
|
| 154 |
-
**Calibration.** Expected calibration error of the full model: SEntFiN 0.078, EDT 0.012, FOMC 0.067. When Arbiter v0.5 is more than 99% confident it is right 91.5–92.6% of the time; base Qwen3-8B gives such answers on 85–98% of items and is right on 59.9% (FOMC) to 90.2% (EDT) of them.
|
| 155 |
-
|
| 156 |
-
**Option order.** Share of answers that change when the options are shuffled: FOMC 4.8%, SEntFiN 6.3%, EDT 1.0%.
|
| 157 |
-
|
| 158 |
-
### Human-labelled decisions
|
| 159 |
-
|
| 160 |
-
| | Dev, 151 (news + 10-K) | Test, 56 (10-K) |
|
| 161 |
-
|---|---|---|
|
| 162 |
-
| **Arbiter v0.5** | 84.1% | 82.1% |
|
| 163 |
-
| Arbiter v0.4 | 85.4% | 83.9% |
|
| 164 |
-
| Jev 1.14 | 84.8% | 82.1% |
|
| 165 |
-
| Qwen3-8B zero-shot | 74.2% | 76.8% |
|
| 166 |
-
|
| 167 |
-
Companies held out from training, labelled by a single human adjudicator. These sets are small: a difference of one or two points is within noise. Calibration temperatures were fitted on the dev set.
|
| 168 |
-
|
| 169 |
-
## Training
|
| 170 |
-
|
| 171 |
-
- **Data:** 140,705 training decisions.
|
| 172 |
-
- 100,000 news, 11,752 SEC 10-K, 8,000 news-outlook and 10,300 headline-sentiment decisions labelled by a local Qwen3.8-27B teacher (one-pass letter readout, soft probability targets);
|
| 173 |
-
- 12,869 corporate-event and policy-stance decisions with published labels, under original and tightened definitions, plus same-article contrast pairs and hard `none` examples;
|
| 174 |
-
- 106 human-reviewed 10-K decisions.
|
| 175 |
-
- SEntFiN was never trained on, and no benchmark test item was trained on. No output of Jev, OpenAI or any Claude model was used for training. The training data is not released: the news article text is licensed and not redistributable.
|
| 176 |
-
- **Recipe:** 4-bit QLoRA (NF4, double quantization), rank 32, alpha 64, on all attention and MLP projections, trained jointly with the decision head. 2,400 steps × 16 decisions, sampled per question family. Options are shuffled on every presentation. Cross-entropy against soft targets, with label smoothing 0.05 on hard labels. Details in `metadata.json`.
|
| 177 |
-
|
| 178 |
-
## Limitations
|
| 179 |
-
|
| 180 |
-
- English, mostly US companies. Not investment advice; use it as decision support with human review.
|
| 181 |
-
- **EDT clinical trials:** recall fell to 54.8% in v0.5. Dividend cuts are found only 47.6% of the time under the original definition (95.2% with `EDT_TIGHT`).
|
| 182 |
-
- **SEntFiN neutral:** Arbiter under-calls neutral headlines (neutral recall 52.0%), so it raises more positive/negative signals than the labels contain.
|
| 183 |
-
- **FOMC dovish:** dovish recall is 41.5%; Arbiter leans hawkish or neutral.
|
| 184 |
-
- **Certainty** (how settled an event is) remains the weakest of the original news questions.
|
| 185 |
-
- Labels follow this project's definitions and conventions. A desk with other conventions should re-evaluate on its own labelled examples before relying on the probabilities.
|
| 186 |
-
- The model sees only the supplied text; it has no knowledge of later events.
|
| 187 |
-
|
| 188 |
-
## Files
|
| 189 |
-
|
| 190 |
-
| File | Purpose |
|
| 191 |
-
|---|---|
|
| 192 |
-
| `model-*.safetensors`, `config.json`, tokenizer files, `chat_template.jinja` | full model: Qwen3-8B with the LoRA merged in, fp16 |
|
| 193 |
-
| `adapter/` | the unmerged LoRA for Qwen/Qwen3-8B (4-bit use), with its `calibration.json` |
|
| 194 |
-
| `decision_head.safetensors`, `head_config.json` | decision head weights and the letter token ids it expects |
|
| 195 |
-
| `calibration.json` | per-type temperatures, fitted on the dev split |
|
| 196 |
-
| `arbiter.py`, `model.py` | loading and inference code |
|
| 197 |
-
| `questions.py` | the exact question definitions used in training |
|
| 198 |
-
| `metadata.json` | training settings and counts |
|
| 199 |
|
| 200 |
## License
|
| 201 |
|
| 202 |
-
Apache-2.0
|
|
|
|
| 98 |
print(answer["choice"]) # e.g. "improved"
|
| 99 |
print(answer["probabilities"]) # {"improved": ..., "worsened": ..., "unchanged": ..., "unclear": ...}
|
| 100 |
|
| 101 |
+
answer = arbiter.decide(state="Inflation remains elevated.", # FOMC statement wording, 2023-24
|
| 102 |
question=FOMC_ORIGINAL, focus="Federal Reserve (FOMC)")
|
| 103 |
print(answer["choice"]) # e.g. "hawkish"
|
| 104 |
```
|
| 105 |
|
| 106 |
`decide()` returns a typed answer: `{"type": "choice", "choice": ..., "probabilities": {...}}` for choice questions, `{"type": "noul", "noul": P(true), ...}` for yes/no questions, and `{"type": "score", "score": level, "probabilities": {...}}` for ordered scales. If a state is too long, it is shortened from the end (never the options), and the answer reports `state_truncated: true`. `arbiter.fallback_counts` counts every fallback taken.
|
| 107 |
|
| 108 |
+
### Example: implied effect vs stated sentiment
|
| 109 |
+
|
| 110 |
+
The same headline, asked two ways. Under the "implied" definition Arbiter judges what the reported facts mean for the company; under the "stated" definition it only counts a direction the headline itself expresses.
|
| 111 |
+
|
| 112 |
+
```python
|
| 113 |
+
from questions import SENTIMENT_IMPLIED, SENTIMENT_STATED
|
| 114 |
+
|
| 115 |
+
headline = "Apple declares quarterly dividend of $0.25 per share" # Apple, 2 May 2024
|
| 116 |
+
arbiter.decide(state=headline, question=SENTIMENT_IMPLIED, focus="Apple")
|
| 117 |
+
# choice: positive {"positive": 0.897, "neutral": 0.076, "negative": 0.028}
|
| 118 |
+
arbiter.decide(state=headline, question=SENTIMENT_STATED, focus="Apple")
|
| 119 |
+
# choice: neutral {"positive": 0.010, "neutral": 0.985, "negative": 0.005}
|
| 120 |
+
```
|
| 121 |
+
|
| 122 |
+
### Example: corporate event classification
|
| 123 |
+
|
| 124 |
+
`EDT_TIGHT` classifies the event a company announces into twelve types. Its definitions separate genuine contracts from look-alikes such as partnerships.
|
| 125 |
+
|
| 126 |
+
```python
|
| 127 |
+
from questions import EDT_TIGHT
|
| 128 |
+
|
| 129 |
+
contract = ("Palantir Technologies has been awarded a five-year, $480 million contract by the U.S. Army "
|
| 130 |
+
"to expand its Maven Smart System prototype.") # Palantir, 29 May 2024
|
| 131 |
+
arbiter.decide(state=contract, question=EDT_TIGHT)
|
| 132 |
+
# choice: new_contract P = 0.977
|
| 133 |
+
|
| 134 |
+
partnership = ("Apple today announced a partnership with OpenAI to integrate ChatGPT into iOS 18, "
|
| 135 |
+
"iPadOS 18 and macOS Sequoia.") # Apple WWDC, 10 June 2024
|
| 136 |
+
arbiter.decide(state=partnership, question=EDT_TIGHT)
|
| 137 |
+
# choice: none P = 0.975 (a partnership is not a new contract)
|
| 138 |
+
```
|
| 139 |
+
|
| 140 |
+
Outputs above are from the full model in this repository.
|
| 141 |
+
|
| 142 |
## Requirements
|
| 143 |
|
| 144 |
- **Python** 3.10 or later, with `torch`, `transformers`, `accelerate`, `safetensors` and `huggingface_hub`.
|
|
|
|
| 148 |
|
| 149 |
Load the model through `arbiter.py` as shown above. The backbone also loads with plain `AutoModelForCausalLM.from_pretrained`, but typed decisions need the decision head, which `arbiter.py` applies. The Hub's automatic code snippet labels this model `text-classification`; it doesn't apply.
|
| 150 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
|
| 152 |
You can pass your own question in the same shape (`{"type", "instructions", "criteria"}`, at most 16 options). The base model's general decision ability carries over, but quality on questions outside this list has not been measured.
|
| 153 |
|
|
|
|
| 166 |
|
| 167 |
95% confidence half-widths: SEntFiN ±0.7, EDT ±1.1, FOMC ±4.2 points. Original definitions, options in the published order. EDT and FOMC are scored on their held-out test splits. SEntFiN was never trained on. Paired McNemar tests against base Qwen3-8B on the same items: FOMC +11.3 points (z 4.9), EDT +4.4 (z 9.1), SEntFiN +1.9 (z 5.9), all p < 0.001.
|
| 168 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
|
| 170 |
## License
|
| 171 |
|
| 172 |
+
Apache-2.0
|