jsemrau commited on
Commit
e94b090
·
verified ·
1 Parent(s): baa22fe

Model card: examples for implied sentiment and event classification; cleanup

Browse files
Files changed (1) hide show
  1. README.md +36 -66
README.md CHANGED
@@ -98,13 +98,47 @@ answer = arbiter.decide(state=passage, question=OUTLOOK, focus="Constellium")
98
  print(answer["choice"]) # e.g. "improved"
99
  print(answer["probabilities"]) # {"improved": ..., "worsened": ..., "unchanged": ..., "unclear": ...}
100
 
101
- answer = arbiter.decide(state="The Committee judges that inflation remains elevated.",
102
  question=FOMC_ORIGINAL, focus="Federal Reserve (FOMC)")
103
  print(answer["choice"]) # e.g. "hawkish"
104
  ```
105
 
106
  `decide()` returns a typed answer: `{"type": "choice", "choice": ..., "probabilities": {...}}` for choice questions, `{"type": "noul", "noul": P(true), ...}` for yes/no questions, and `{"type": "score", "score": level, "probabilities": {...}}` for ordered scales. If a state is too long, it is shortened from the end (never the options), and the answer reports `state_truncated: true`. `arbiter.fallback_counts` counts every fallback taken.
107
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
108
  ## Requirements
109
 
110
  - **Python** 3.10 or later, with `torch`, `transformers`, `accelerate`, `safetensors` and `huggingface_hub`.
@@ -114,25 +148,6 @@ print(answer["choice"]) # e.g. "hawkish"
114
 
115
  Load the model through `arbiter.py` as shown above. The backbone also loads with plain `AutoModelForCausalLM.from_pretrained`, but typed decisions need the decision head, which `arbiter.py` applies. The Hub's automatic code snippet labels this model `text-classification`; it doesn't apply.
116
 
117
- ## Questions it was trained on
118
-
119
- Import the exact definitions from `questions.py`; Arbiter is most reliable with them.
120
-
121
- | Constant | Type | Options |
122
- |---|---|---|
123
- | `EVENT_TYPE` | choice | corporate_action, financial_results, company_guidance, product_operations, analyst_action, analyst_opinion, market_move, legal_safety, industry_macro, insufficient_evidence |
124
- | `OUTLOOK` | choice | improved, worsened, unchanged, unclear |
125
- | `CERTAINTY` | score | 0 rumoured … 3 realised |
126
- | `MAIN_SUBJECT` | yes/no | is the company the main subject of the event? |
127
- | `ROLE` | choice | acquirer, target, regulator, mentioned_only |
128
- | `FILING_RISK` | choice | ten risk categories for an Item 1A passage |
129
- | `FILING_LEGAL` | score | 0 possible … 3 resolved |
130
- | `GOING_CONCERN`, `CUSTOMER_CONCENTRATION`, `CYBER_INCIDENT` | yes/no | 10-K flags |
131
- | `EDT_ORIGINAL`, `EDT_TIGHT` (new) | choice | acquisition, clinical_trial, regular_dividend, dividend_cut, dividend_increase, guidance_change, new_contract, reverse_stock_split, special_dividend, stock_repurchase, stock_split, none |
132
- | `FOMC_ORIGINAL`, `FOMC_TIGHT` (new) | choice | hawkish, neutral, dovish |
133
- | `SENTIMENT_ORIGINAL`, `SENTIMENT_IMPLIED`, `SENTIMENT_STATED` (new) | choice | positive, neutral, negative |
134
-
135
- The `_TIGHT`, `_IMPLIED` and `_STATED` variants spell out a labelling convention in the definitions, for example that a partnership is not a new contract. v0.5 was trained on both wordings of each question so that it follows whichever definition it is given; tighter definitions usually score higher (SEntFiN, F16 GGUF: 81.9% tightened vs 76.8% original).
136
 
137
  You can pass your own question in the same shape (`{"type", "instructions", "criteria"}`, at most 16 options). The base model's general decision ability carries over, but quality on questions outside this list has not been measured.
138
 
@@ -151,52 +166,7 @@ You can pass your own question in the same shape (`{"type", "instructions", "cri
151
 
152
  95% confidence half-widths: SEntFiN ±0.7, EDT ±1.1, FOMC ±4.2 points. Original definitions, options in the published order. EDT and FOMC are scored on their held-out test splits. SEntFiN was never trained on. Paired McNemar tests against base Qwen3-8B on the same items: FOMC +11.3 points (z 4.9), EDT +4.4 (z 9.1), SEntFiN +1.9 (z 5.9), all p < 0.001.
153
 
154
- **Calibration.** Expected calibration error of the full model: SEntFiN 0.078, EDT 0.012, FOMC 0.067. When Arbiter v0.5 is more than 99% confident it is right 91.5–92.6% of the time; base Qwen3-8B gives such answers on 85–98% of items and is right on 59.9% (FOMC) to 90.2% (EDT) of them.
155
-
156
- **Option order.** Share of answers that change when the options are shuffled: FOMC 4.8%, SEntFiN 6.3%, EDT 1.0%.
157
-
158
- ### Human-labelled decisions
159
-
160
- | | Dev, 151 (news + 10-K) | Test, 56 (10-K) |
161
- |---|---|---|
162
- | **Arbiter v0.5** | 84.1% | 82.1% |
163
- | Arbiter v0.4 | 85.4% | 83.9% |
164
- | Jev 1.14 | 84.8% | 82.1% |
165
- | Qwen3-8B zero-shot | 74.2% | 76.8% |
166
-
167
- Companies held out from training, labelled by a single human adjudicator. These sets are small: a difference of one or two points is within noise. Calibration temperatures were fitted on the dev set.
168
-
169
- ## Training
170
-
171
- - **Data:** 140,705 training decisions.
172
- - 100,000 news, 11,752 SEC 10-K, 8,000 news-outlook and 10,300 headline-sentiment decisions labelled by a local Qwen3.8-27B teacher (one-pass letter readout, soft probability targets);
173
- - 12,869 corporate-event and policy-stance decisions with published labels, under original and tightened definitions, plus same-article contrast pairs and hard `none` examples;
174
- - 106 human-reviewed 10-K decisions.
175
- - SEntFiN was never trained on, and no benchmark test item was trained on. No output of Jev, OpenAI or any Claude model was used for training. The training data is not released: the news article text is licensed and not redistributable.
176
- - **Recipe:** 4-bit QLoRA (NF4, double quantization), rank 32, alpha 64, on all attention and MLP projections, trained jointly with the decision head. 2,400 steps × 16 decisions, sampled per question family. Options are shuffled on every presentation. Cross-entropy against soft targets, with label smoothing 0.05 on hard labels. Details in `metadata.json`.
177
-
178
- ## Limitations
179
-
180
- - English, mostly US companies. Not investment advice; use it as decision support with human review.
181
- - **EDT clinical trials:** recall fell to 54.8% in v0.5. Dividend cuts are found only 47.6% of the time under the original definition (95.2% with `EDT_TIGHT`).
182
- - **SEntFiN neutral:** Arbiter under-calls neutral headlines (neutral recall 52.0%), so it raises more positive/negative signals than the labels contain.
183
- - **FOMC dovish:** dovish recall is 41.5%; Arbiter leans hawkish or neutral.
184
- - **Certainty** (how settled an event is) remains the weakest of the original news questions.
185
- - Labels follow this project's definitions and conventions. A desk with other conventions should re-evaluate on its own labelled examples before relying on the probabilities.
186
- - The model sees only the supplied text; it has no knowledge of later events.
187
-
188
- ## Files
189
-
190
- | File | Purpose |
191
- |---|---|
192
- | `model-*.safetensors`, `config.json`, tokenizer files, `chat_template.jinja` | full model: Qwen3-8B with the LoRA merged in, fp16 |
193
- | `adapter/` | the unmerged LoRA for Qwen/Qwen3-8B (4-bit use), with its `calibration.json` |
194
- | `decision_head.safetensors`, `head_config.json` | decision head weights and the letter token ids it expects |
195
- | `calibration.json` | per-type temperatures, fitted on the dev split |
196
- | `arbiter.py`, `model.py` | loading and inference code |
197
- | `questions.py` | the exact question definitions used in training |
198
- | `metadata.json` | training settings and counts |
199
 
200
  ## License
201
 
202
- Apache-2.0, like the base model Qwen3-8B.
 
98
  print(answer["choice"]) # e.g. "improved"
99
  print(answer["probabilities"]) # {"improved": ..., "worsened": ..., "unchanged": ..., "unclear": ...}
100
 
101
+ answer = arbiter.decide(state="Inflation remains elevated.", # FOMC statement wording, 2023-24
102
  question=FOMC_ORIGINAL, focus="Federal Reserve (FOMC)")
103
  print(answer["choice"]) # e.g. "hawkish"
104
  ```
105
 
106
  `decide()` returns a typed answer: `{"type": "choice", "choice": ..., "probabilities": {...}}` for choice questions, `{"type": "noul", "noul": P(true), ...}` for yes/no questions, and `{"type": "score", "score": level, "probabilities": {...}}` for ordered scales. If a state is too long, it is shortened from the end (never the options), and the answer reports `state_truncated: true`. `arbiter.fallback_counts` counts every fallback taken.
107
 
108
+ ### Example: implied effect vs stated sentiment
109
+
110
+ The same headline, asked two ways. Under the "implied" definition Arbiter judges what the reported facts mean for the company; under the "stated" definition it only counts a direction the headline itself expresses.
111
+
112
+ ```python
113
+ from questions import SENTIMENT_IMPLIED, SENTIMENT_STATED
114
+
115
+ headline = "Apple declares quarterly dividend of $0.25 per share" # Apple, 2 May 2024
116
+ arbiter.decide(state=headline, question=SENTIMENT_IMPLIED, focus="Apple")
117
+ # choice: positive {"positive": 0.897, "neutral": 0.076, "negative": 0.028}
118
+ arbiter.decide(state=headline, question=SENTIMENT_STATED, focus="Apple")
119
+ # choice: neutral {"positive": 0.010, "neutral": 0.985, "negative": 0.005}
120
+ ```
121
+
122
+ ### Example: corporate event classification
123
+
124
+ `EDT_TIGHT` classifies the event a company announces into twelve types. Its definitions separate genuine contracts from look-alikes such as partnerships.
125
+
126
+ ```python
127
+ from questions import EDT_TIGHT
128
+
129
+ contract = ("Palantir Technologies has been awarded a five-year, $480 million contract by the U.S. Army "
130
+ "to expand its Maven Smart System prototype.") # Palantir, 29 May 2024
131
+ arbiter.decide(state=contract, question=EDT_TIGHT)
132
+ # choice: new_contract P = 0.977
133
+
134
+ partnership = ("Apple today announced a partnership with OpenAI to integrate ChatGPT into iOS 18, "
135
+ "iPadOS 18 and macOS Sequoia.") # Apple WWDC, 10 June 2024
136
+ arbiter.decide(state=partnership, question=EDT_TIGHT)
137
+ # choice: none P = 0.975 (a partnership is not a new contract)
138
+ ```
139
+
140
+ Outputs above are from the full model in this repository.
141
+
142
  ## Requirements
143
 
144
  - **Python** 3.10 or later, with `torch`, `transformers`, `accelerate`, `safetensors` and `huggingface_hub`.
 
148
 
149
  Load the model through `arbiter.py` as shown above. The backbone also loads with plain `AutoModelForCausalLM.from_pretrained`, but typed decisions need the decision head, which `arbiter.py` applies. The Hub's automatic code snippet labels this model `text-classification`; it doesn't apply.
150
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
151
 
152
  You can pass your own question in the same shape (`{"type", "instructions", "criteria"}`, at most 16 options). The base model's general decision ability carries over, but quality on questions outside this list has not been measured.
153
 
 
166
 
167
  95% confidence half-widths: SEntFiN ±0.7, EDT ±1.1, FOMC ±4.2 points. Original definitions, options in the published order. EDT and FOMC are scored on their held-out test splits. SEntFiN was never trained on. Paired McNemar tests against base Qwen3-8B on the same items: FOMC +11.3 points (z 4.9), EDT +4.4 (z 9.1), SEntFiN +1.9 (z 5.9), all p < 0.001.
168
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
169
 
170
  ## License
171
 
172
+ Apache-2.0