| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-8B |
| base_model_relation: finetune |
| library_name: transformers |
| language: |
| - en |
| pipeline_tag: text-classification |
| tags: |
| - qwen3 |
| - qwen |
| - decision-model |
| - finance |
| - wealth-management |
| - sec-filings |
| - news |
| - central-banks |
| - calibration |
| - lora |
| - jev |
| --- |
| |
| # Arbiter-8B (v0.5) |
|
|
| Arbiter-8B is a Jev-inspired decision model built on Alibaba's Qwen3-8B, for wealth and asset management. Arbiter is aimed to be run on-prem at low cost. It reads financial text (news articles, headlines, SEC Form 10-K passages, central-bank statements) and answers a typed question about it, such as the type of corporate event, the implied outlook for a company, the sentiment toward it, or the policy stance of a sentence. It returns a **calibrated probability for every option** in a single forward pass, with no generated text. |
|
|
| This repository holds **Arbiter v0.5**. It replaces v0.4, which is still available in the repository history. |
|
|
| ## Results at a glance |
|
|
| Same items, same question and same option definitions for every system. EDT and FOMC are scored on their held-out test splits; SEntFiN was never trained on. |
|
|
| | Benchmark | Arbiter v0.5 | Qwen3-8B (base) | Jev 1.14 | OpenAI Decisions API | |
| |---|---|---|---|---| |
| | **EDT**, corporate event detection, 1,933 articles, accuracy | **94.2%** | 89.3% | 88.3% | 90.8% | |
| | **FOMC**, policy stance, 496 sentences, accuracy | **67.5%** | 55.0% | 66.7% | 67.3% | |
| | **SEntFiN**, entity sentiment, 1,000-headline sample, accuracy | 75.7% | 75.4% | **81.6%** | 78.0% | |
| | SEntFiN, signals caught (positive/negative labels called correctly) | **92.0%** | 82.8% | 91.0% | 91.7% | |
| | SEntFiN, signal precision (positive/negative calls that are right) | 72.2% | 76.3% | **78.8%** | 74.8% | |
|
|
| - **EDT:** Arbiter leads all three systems; every gap is significant (paired McNemar, z ≥ 7.5). |
| - **FOMC:** Arbiter, Jev and OpenAI are level (z ≤ 0.4); all three are far ahead of the base model (z 4.8). |
| - **SEntFiN:** Arbiter catches the most positive and negative signals but is less precise than Jev and OpenAI; it labels too few headlines neutral. |
|
|
|
|
| ## What changed from v0.4 |
|
|
| v0.5 was trained to follow the option definitions it is given, to separate genuine corporate events from look-alikes, and to read central-bank language. |
|
|
| | | v0.4 | v0.5 | |
| |---|---|---| |
| | EDT accuracy | 83.5% | **94.2%** | |
| | FOMC accuracy | 60.1% | **67.5%** | |
| | SEntFiN accuracy (full benchmark, 14,329 headlines, zero-shot) | 76.0% | **76.4%** | |
| | EDT `new_contract` precision | 38.3% | **86.7%** | |
| | EDT `none` recall | 81.2% | **95.2%** | |
| | Answers that change when options are shuffled, FOMC | 15.5% | **4.8%** | |
| | Calibration error (ECE), EDT / FOMC | 0.099 / 0.127 | **0.012 / 0.067** | |
| | Human-labelled dev set (151, news + 10-K) | **85.4%** | 84.1% | |
| | Human-labelled test set (56, 10-K) | **83.9%** | 82.1% | |
|
|
| The two human-labelled sets moved by one or two items, which is within noise: the new data neither helped nor hurt the original news and 10-K tasks. One clear regression: **EDT clinical-trial recall fell from 87.1% to 54.8%** (precision held at 79.1%); missed trials mostly go to `none`. |
|
|
| ## Model details |
|
|
| - **Model type:** decision model (single-pass scoring of a fixed set of options), built on the Qwen3 causal language model (`Qwen3ForCausalLM`, 8B parameters) |
| - **Base model:** [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B), fine-tuned with LoRA (rank 32) and a decision head |
| - **Language:** English |
| - **Domain:** financial news, headlines, SEC Form 10-K filings and central-bank communication, for wealth and asset management |
| - **Precision:** fp16 (the fine-tuning is merged into the weights) |
| - **Method:** follows "Releasing the Inner Jev" (Geuter et al., 2026) |
|
|
| ### How it works |
|
|
| You give Arbiter a text, a question and a set of options, each with a short definition. It scores every option in one forward pass and returns the probabilities, calibrated per question type with the temperatures in `calibration.json`. |
|
|
|
|
| For llama.cpp, Ollama and LM Studio, use [ternary/Arbiter-8B-GGUF](https://huggingface.co/ternary/Arbiter-8B-GGUF). |
|
|
| ## Quick start |
|
|
| ```bash |
| pip install torch transformers accelerate safetensors huggingface_hub # + peft bitsandbytes for the 4-bit adapter |
| hf download ternary/Arbiter-8B --local-dir Arbiter-8B |
| cd Arbiter-8B |
| ``` |
|
|
| ```python |
| from arbiter import Arbiter |
| from questions import OUTLOOK, EDT_TIGHT, FOMC_ORIGINAL |
| |
| arbiter = Arbiter.from_pretrained(".") # full model (fp16) |
| # arbiter = Arbiter.from_pretrained(".", adapter=True) # LoRA on Qwen/Qwen3-8B in 4-bit, as trained |
| |
| passage = ("Revenue increased 15% to $8,449 million, reflecting higher shipments and higher " |
| "revenue per ton. Cost of sales increased 14%.") |
| answer = arbiter.decide(state=passage, question=OUTLOOK, focus="Constellium") |
| print(answer["choice"]) # e.g. "improved" |
| print(answer["probabilities"]) # {"improved": ..., "worsened": ..., "unchanged": ..., "unclear": ...} |
| |
| answer = arbiter.decide(state="Inflation remains elevated.", # FOMC statement wording, 2023-24 |
| question=FOMC_ORIGINAL, focus="Federal Reserve (FOMC)") |
| print(answer["choice"]) # e.g. "hawkish" |
| ``` |
|
|
| `decide()` returns a typed answer: `{"type": "choice", "choice": ..., "probabilities": {...}}` for choice questions, `{"type": "noul", "noul": P(true), ...}` for yes/no questions, and `{"type": "score", "score": level, "probabilities": {...}}` for ordered scales. If a state is too long, it is shortened from the end (never the options), and the answer reports `state_truncated: true`. `arbiter.fallback_counts` counts every fallback taken. |
|
|
| ### Example: implied effect vs stated sentiment |
|
|
| The same headline, asked two ways. Under the "implied" definition Arbiter judges what the reported facts mean for the company; under the "stated" definition it only counts a direction the headline itself expresses. |
|
|
| ```python |
| from questions import SENTIMENT_IMPLIED, SENTIMENT_STATED |
| |
| headline = "Apple declares quarterly dividend of $0.25 per share" # Apple, 2 May 2024 |
| arbiter.decide(state=headline, question=SENTIMENT_IMPLIED, focus="Apple") |
| # choice: positive {"positive": 0.897, "neutral": 0.076, "negative": 0.028} |
| arbiter.decide(state=headline, question=SENTIMENT_STATED, focus="Apple") |
| # choice: neutral {"positive": 0.010, "neutral": 0.985, "negative": 0.005} |
| ``` |
|
|
| ### Example: corporate event classification |
|
|
| `EDT_TIGHT` classifies the event a company announces into twelve types. Its definitions separate genuine contracts from look-alikes such as partnerships. |
|
|
| ```python |
| from questions import EDT_TIGHT |
| |
| contract = ("Palantir Technologies has been awarded a five-year, $480 million contract by the U.S. Army " |
| "to expand its Maven Smart System prototype.") # Palantir, 29 May 2024 |
| arbiter.decide(state=contract, question=EDT_TIGHT) |
| # choice: new_contract P = 0.977 |
| |
| partnership = ("Apple today announced a partnership with OpenAI to integrate ChatGPT into iOS 18, " |
| "iPadOS 18 and macOS Sequoia.") # Apple WWDC, 10 June 2024 |
| arbiter.decide(state=partnership, question=EDT_TIGHT) |
| # choice: none P = 0.975 (a partnership is not a new contract) |
| ``` |
|
|
| Outputs above are from the full model in this repository. |
|
|
| ## Requirements |
|
|
| - **Python** 3.10 or later, with `torch`, `transformers`, `accelerate`, `safetensors` and `huggingface_hub`. |
| - **Full model (default):** about 17 GB of GPU memory in fp16 or bf16, for example one 24 GB GPU. `device_map="auto"` also spreads the model over several smaller GPUs. If part of it is offloaded to CPU memory, it still runs, only more slowly. |
| - **4-bit adapter** (`adapter=True`): about 6.5 GB of GPU memory, so an 8 GB GPU is enough. Also needs `peft` and `bitsandbytes` (CUDA). |
| - **Compute:** one forward pass per decision, no text generation. |
|
|
| Load the model through `arbiter.py` as shown above. The backbone also loads with plain `AutoModelForCausalLM.from_pretrained`, but typed decisions need the decision head, which `arbiter.py` applies. The Hub's automatic code snippet labels this model `text-classification`; it doesn't apply. |
|
|
|
|
| You can pass your own question in the same shape (`{"type", "instructions", "criteria"}`, at most 16 options). The base model's general decision ability carries over, but quality on questions outside this list has not been measured. |
|
|
| ## Evaluation |
|
|
| ### Public benchmarks |
|
|
| | | SEntFiN (14,329) | EDT (1,933) | FOMC (496) | |
| |---|---|---|---| |
| | **Arbiter v0.5, full model (this repository)** | 76.4% | **94.2%** | **67.5%** | |
| | Arbiter v0.5, F16 GGUF | 76.8% | 93.7% | 66.3% | |
| | Arbiter v0.5, Q4_K_M GGUF (imatrix) | **77.4%** | 93.6% | 67.3% | |
| | Arbiter v0.4, full model | 76.0% | 83.5% | 60.1% | |
| | Qwen3-8B, F16 GGUF | 75.0% | 89.3% | 55.0% | |
| | Majority class | 38.1% | 77.2% | 49.8% | |
|
|
| 95% confidence half-widths: SEntFiN ±0.7, EDT ±1.1, FOMC ±4.2 points. Original definitions, options in the published order. EDT and FOMC are scored on their held-out test splits. SEntFiN was never trained on. Paired McNemar tests against base Qwen3-8B on the same items: FOMC +11.3 points (z 4.9), EDT +4.4 (z 9.1), SEntFiN +1.9 (z 5.9), all p < 0.001. |
|
|
|
|
| ## License |
|
|
| Apache-2.0 |
