Add DeBERTa-v3-large financial sentiment model, card, training and scoring scripts

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nagizaaz Shaik
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,334 @@
+---
+language: en
+license: mit
+library_name: transformers
+pipeline_tag: text-classification
+base_model: microsoft/deberta-v3-large
+tags:
+  - financial-sentiment
+  - finance
+  - sentiment-analysis
+  - text-classification
+  - deberta-v3
+datasets:
+  - takala/financial_phrasebank
+  - zeroshot/twitter-financial-news-sentiment
+  - Jean-Baptiste/financial_news_sentiment
+metrics:
+  - accuracy
+  - f1
+model-index:
+  - name: finbert-deberta-large-sentiment
+    results:
+      - task:
+          type: text-classification
+          name: Financial Sentiment Classification
+        dataset:
+          name: Financial PhraseBank (Sentences_AllAgree)
+          type: takala/financial_phrasebank
+          config: sentences_allagree
+        metrics:
+          - type: accuracy
+            value: 0.9941
+            name: Test Accuracy
+          - type: f1
+            value: 0.9928
+            name: Test Macro-F1
+      - task:
+          type: text-classification
+          name: Financial Sentiment Classification
+        dataset:
+          name: Twitter Financial News Sentiment
+          type: zeroshot/twitter-financial-news-sentiment
+          split: validation
+        metrics:
+          - type: accuracy
+            value: 0.9016
+            name: Validation Accuracy
+          - type: f1
+            value: 0.8793
+            name: Validation Macro-F1
+      - task:
+          type: text-classification
+          name: Financial Sentiment Classification
+        dataset:
+          name: Jean-Baptiste financial news sentiment (headlines)
+          type: Jean-Baptiste/financial_news_sentiment
+          split: test
+        metrics:
+          - type: accuracy
+            value: 0.7828
+            name: Test Accuracy
+          - type: f1
+            value: 0.6137
+            name: Test Macro-F1
+      - task:
+          type: text-classification
+          name: Financial Sentiment Classification
+        dataset:
+          name: SEntFiN headlines (unanimous, test split)
+          type: tasksource/sentfin
+          split: test
+        metrics:
+          - type: accuracy
+            value: 0.7974
+            name: Test Accuracy
+          - type: f1
+            value: 0.8030
+            name: Test Macro-F1
+widget:
+  - text: "The company reported record quarterly profit, beating analyst expectations."
+  - text: "Net sales decreased by 12% and the firm warned of further losses ahead."
+  - text: "The board will meet next Tuesday to review the quarterly schedule."
+---
+
+# finbert-deberta-large-sentiment
+
+**A larger, more accurate sibling of [finbert-minilm-sentiment](https://huggingface.co/9mark9/finbert-minilm-sentiment): 90.2% on finance tweets (+3.7 points), 99.4% on PhraseBank test, at 13x the parameters and about 19x the CPU latency.**
+
+`finbert-deberta-large-sentiment` classifies short financial text into **negative / neutral / positive**.
+It is `microsoft/deberta-v3-large` (435M parameters) fine-tuned with the same data mix as the compact
+model (v3): Financial PhraseBank, finance tweets and press-release headlines. Use it when accuracy
+matters more than speed. For CPU serving at scale the 33.4M-parameter compact model is still the
+better trade (see [Size and speed](#size-and-speed)).
+
+## Results (held-out sets)
+
+None of these sets was used for training or for choosing the model. Differences are against v3
+(`9mark9/finbert-minilm-sentiment`), with paired bootstrap 95% intervals (2,000 resamples of the same
+rows for both models). All numbers are for this repository's weights loaded in fp32.
+
+| Set | n | v3 accuracy | This model | Change (points) [95% CI] | v3 macro-F1 | This model | Change [95% CI] |
+|-----|---|-------------|------------|--------------------------|-------------|------------|-----------------|
+| PhraseBank `AllAgree` test (15%) | 340 | 97.94 | **99.41** | +1.47 [+0.00, +3.24] | 96.43 | **99.28** | +2.85 [+0.28, +5.97] |
+| Finance tweets, validation file | 2,388 | 86.43 | **90.16** | +3.73 [+2.47, +5.07] | 83.00 | **87.93** | +4.94 [+3.36, +6.57] |
+| Press-release headlines (JB), test | 267 | 75.28 | **78.28** | +3.00 [-2.25, +8.24] | 59.04 | **61.37** | +2.33 [-9.45, +14.79] |
+| SEntFiN headlines, test (one label per headline) | 933 | 75.99 | **79.74** | +3.75 [+1.29, +6.22] | 76.27 | **80.30** | +4.03 [+1.53, +6.51] |
+
+Per-class precision / recall (negative, neutral, positive):
+
+| Set | v3 precision | v3 recall | This model precision | This model recall |
+|-----|--------------|-----------|----------------------|-------------------|
+| PhraseBank | 0.953 / 1.000 / 0.944 | 0.911 / 0.990 / 0.988 | 0.978 / 1.000 / 0.989 | 1.000 / 0.990 / 1.000 |
+| Tweets | 0.725 / 0.946 / 0.752 | 0.850 / 0.870 / 0.857 | 0.831 / 0.947 / 0.819 | 0.879 / 0.908 / 0.897 |
+| JB headlines | 0.667 / 0.764 / 0.733 | 0.167 / 0.853 / 0.667 | 0.667 / 0.794 / 0.766 | 0.167 / 0.865 / 0.727 |
+| SEntFiN | 0.850 / 0.676 / 0.811 | 0.722 / 0.825 / 0.723 | 0.891 / 0.696 / 0.866 | 0.792 / 0.836 / 0.762 |
+
+Datasets and splits: PhraseBank `Sentences_AllAgree`, stratified 70/15/15 with
+`train_test_split(random_state=42)` (test carved first). Tweets: `zeroshot/twitter-financial-news-sentiment`
+`sent_valid.csv` (labels 0 bearish, 1 bullish, 2 neutral). JB: `Jean-Baptiste/financial_news_sentiment`
+test split, headline text (0 negative, 1 neutral, 2 positive). SEntFiN: `tasksource/sentfin` test split
+(revision d6294b7), headlines where every entity has the same label. All revisions are pinned in
+`training/common.py`.
+
+## Release rule
+
+Fixed before training started: release a new repository only if tweet accuracy beats v3 by at least
+2.0 points with a paired 95% interval entirely above zero, and PhraseBank accuracy is not more than
+1.0 point worse. Result: +3.73 [+2.47, +5.07] on tweets and +1.47 on PhraseBank, so it passes.
+
+### How this model was chosen
+
+Every candidate was ranked by the mean macro-F1 over three validation slices carved from the
+training files (PhraseBank validation split, 10% of the tweet training file, 10% of the headline
+training file). The held-out sets above were scored for every candidate but not used to choose.
+
+One change was made during the search, and it is stated here. Once this model was trained, a
+probability-average ensemble containing it ranked slightly higher on validation (by 0.0005). A
+tie-break rule was added at that point, after this model's own held-out scores had been seen, but
+before roberta-large had finished and before any ensemble containing this model had been scored on
+the held-out sets. The rule: an ensemble wins only if it leads the best single model on validation by at
+least 0.005, because it costs at least 1.5x the CPU time for a gap inside validation noise. The final
+top ensemble (this model + roberta-large + v3, 0.8772) led this model (0.8751) by 0.0021, so the single
+model was kept. For transparency, that ensemble scored 99.12 / 90.12 / 79.03 / 80.92 on the four sets
+(tweets +3.69 [+2.51, +4.94]), about the same as this model.
+
+### All candidates
+
+Accuracy on the held-out sets; validation = mean macro-F1 over the three validation slices.
+`ens_*` = average of class probabilities. Runs marked (Mac) were trained on an Apple GPU in fp32,
+the others on a ZeroGPU Space with bf16 autocast.
+
+| Candidate | Params | Validation | PhraseBank | Tweets | JB | SEntFiN | Tweets vs v3 [95% CI] |
+|-----------|--------|------------|------------|--------|----|---------|-----------------------|
+| v3 (finbert-minilm-sentiment) | 33.4M | 0.8239 | 97.94 | 86.43 | 75.28 | 75.99 | - |
+| deberta-v3-base, seed 42 | 184.4M | 0.8650 | 99.12 | 88.61 | 80.15 | 79.96 | +2.18 [+0.96, +3.48] |
+| deberta-v3-base, seed 1 | 184.4M | 0.8486 | 99.41 | 87.48 | 77.90 | 79.31 | +1.05 [-0.21, +2.35] |
+| deberta-v3-base, seed 2 | 184.4M | 0.8500 | 98.82 | 86.64 | 74.91 | 80.39 | +0.21 [-1.09, +1.55] |
+| deberta-v3-base, weight average of the 3 seeds | 184.4M | 0.8375 | 98.53 | 83.88 | 78.65 | 80.60 | -2.55 [-3.94, -1.17] |
+| roberta-base, seed 42 (Mac) | 124.6M | 0.8463 | 99.41 | 88.32 | 68.91 | 80.17 | +1.88 [+0.63, +3.22] |
+| roberta-base, seed 1 | 124.6M | 0.8421 | 99.41 | 88.86 | 72.28 | 77.17 | +2.43 [+1.13, +3.73] |
+| distilroberta-base, seed 42 (Mac) | 82.1M | 0.8389 | 97.94 | 87.56 | 77.53 | 75.99 | +1.13 [-0.21, +2.43] |
+| ProsusAI/finbert, seed 42 (Mac) * | 109.5M | 0.8283 | 99.71 | 87.23 | 72.66 | 75.88 | +0.80 [-0.50, +2.05] |
+| roberta-large, seed 42 | 355.4M | 0.8617 | 98.82 | 86.77 | 78.28 | 80.71 | +0.34 [-1.01, +1.84] |
+| **deberta-v3-large, seed 42 (this model)** | 435.1M | 0.8751 | 99.41 | 90.16 | 78.28 | 79.74 | +3.73 [+2.47, +5.07] |
+| ens: deberta-v3-base s42 + s1 + v3 | 402.2M | 0.8700 | 99.41 | 88.74 | 77.53 | 80.39 | +2.30 [+1.17, +3.52] |
+| ens: deberta-v3-large + roberta-large + v3 | 823.8M | 0.8772 | 99.12 | 90.12 | 79.03 | 80.92 | +3.69 [+2.51, +4.94] |
+| ens: deberta-v3-large + deberta-v3-base s2 + v3 | 652.9M | 0.8756 | 99.12 | 90.62 | 79.40 | 80.28 | +4.19 [+2.97, +5.40] |
+
+\* ProsusAI/finbert is CC-BY-NC-SA-4.0 and was itself fine-tuned on Financial PhraseBank, so its
+PhraseBank score is not a clean held-out number; it was a reference only.
+
+Seeds matter: three deberta-v3-base seeds span 86.6 to 88.6 on tweets. deberta-v3-large was trained
+with one seed only, so part of its margin may be seed luck.
+
+## Size and speed
+
+CPU latency per sentence, batch size 1, 4 threads, median (90th percentile) over 300 tweets after 20
+warm-up calls, fp32, measured on a shared cloud CPU (absolute numbers vary by machine; ratios are
+the useful part):
+
+| Model | Parameters | Weights (fp32) | CPU latency | Tweets accuracy |
+|-------|------------|----------------|-------------|-----------------|
+| finbert-minilm-sentiment (v3) | 33.4M | 0.13 GB | 13.5 ms (18.5) | 86.43 |
+| deberta-v3-base fine-tune (seed 42) | 184.4M | 0.74 GB | 83.6 ms (104.9) | 88.61 |
+| roberta-large fine-tune | 355.4M | 1.4 GB | 127.3 ms (155.6) | 86.77 |
+| **this model** | 435.1M | 1.7 GB | 259.3 ms (301.2) | 90.16 |
+
+## How far can tweet accuracy go? (label-noise estimate)
+
+This model gets 235 of the 2,388 validation tweets wrong. I read 100 of those errors, drawn at
+random (seed 0), and judged each one against the text:
+
+| Judgement | Count |
+|-----------|-------|
+| Gold label looks wrong; the model's label fits the text better | 32 |
+| Genuinely ambiguous (either label defensible) | 29 |
+| Clear model error (gold label is right) | 39 |
+
+Examples of the first kind: "Margins suffer at Ituran Location in Q3" (gold neutral, model negative),
+"iAnthus Defaults On Debt" (gold neutral), "Howard Bancorp Shareholders Have Enjoyed A 54% Share
+Price Gain" (gold neutral, model positive), "U.S. oil firms vented or flared record amount of gas"
+(gold bullish).
+
+If a model fixed every clear error and nothing else, it would reach about **94.0%** (95% interval
+93.1% to 95.0%, from the Wilson interval on 39/100). Going past that means agreeing with labels that look
+wrong or with coin-flip judgements, which is fitting the annotators, not the sentiment. So 95% or more on
+this tweet set is not a realistic honest target. This is one reviewer's judgement, made while seeing
+the model's prediction, so treat it as an estimate.
+
+## Training data
+
+Held out and never used for training or model selection: the PhraseBank 15% test split, the full
+tweet validation file, the headline test split and all of SEntFiN. Before training, any row whose text
+matches a held-out or validation row was dropped. The match key is lower-cased text with URLs and
+punctuation removed. 111 rows were dropped.
+
+Note on v3: v3 deduplicated on lower-cased, whitespace-normalised text only. With the stricter key, 80
+rows of the tweet training file and 8 rows of the headline training file match held-out text (for
+example the same headline with a different link). v3 was trained with those rows; this model was not.
+
+| Source | Used for training | Used for validation |
+|--------|-------------------|---------------------|
+| PhraseBank `Sentences_AllAgree` | 70% train split (1,584), repeated 3x per epoch | 15% validation split (340) |
+| PhraseBank `Sentences_66Agree` | only sentences not in the all-agree file at all (1,951); majority label | - |
+| Tweets, `sent_train.csv` | 90% stratified (8,588) | 10% stratified (955) |
+| Headlines (JB), train split | 90% stratified (1,360), repeated 2x per epoch | 10% stratified (152) |
+
+17,900 training rows per epoch after deduplication. Dataset licences apply to the data.
+
+## Training details
+
+| Setting | Value |
+|---------|-------|
+| Base encoder | `microsoft/deberta-v3-large` (MIT) |
+| Parameters | 435,064,835 |
+| Objective | class-weighted cross-entropy (inverse class frequency) |
+| Optimizer | AdamW, lr 1e-5, weight decay 0.01, gradient clip 1.0 |
+| Schedule | linear, 10% warmup; batch 16; max 96 tokens; 3 epochs (3,357 steps) |
+| Precision | fp32 weights, bf16 autocast for the forward pass |
+| Checkpoint | best mean validation macro-F1, evaluated twice per epoch (step 3,354) |
+| Hardware | Hugging Face ZeroGPU (half of an RTX Pro 6000), 481 GPU-seconds |
+
+The learning rate (1e-5 for the large encoders, 2e-5 for base, 3e-5 for distilroberta) was fixed
+before training, not tuned.
+
+## Reproduce
+
+`training/` holds the code that produced and scored this model: `app.py` (a Gradio app for a
+ZeroGPU Space: `train_chunk`, `evaluate`, `latency`, `soup`), `common.py` (data loading at pinned
+revisions, dedupe key, splits), `requirements.txt`, `drive.py` (client that calls the Space) and
+`analyze.py` (accuracy, macro-F1, per-class precision/recall, paired bootstrap against v3).
+
+1. Create a Gradio Space with ZeroGPU hardware, add `app.py`, `common.py` and `requirements.txt`, and
+   set a write token as the secret `HF_TOKEN` (used to push checkpoints to private repos).
+2. Point `SPACE` in `drive.py` at it, then run:
+
+```bash
+mkdir -p probs
+python drive.py train debL_s42 '{"base":"microsoft/deberta-v3-large","lr":1e-5,"pb_extra":"66Agree","jb_rep":2,"seed":42,"gpu_seconds":600,"margin":80}' 2
+python drive.py eval debL_s42_fp32 <your-user>/tmp-train-debL_s42 best:fp32 probs/debL_s42_fp32.npz
+python drive.py eval v3 9mark9/finbert-minilm-sentiment - probs/v3.npz
+python analyze.py results.json v3 debL_s42_fp32
+```
+
+GPU training is not bit-for-bit deterministic, so a rerun lands close to, not exactly on, these numbers.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+
+model_id = "9mark9/finbert-deberta-large-sentiment"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForSequenceClassification.from_pretrained(model_id)
+model.eval()
+
+texts = [
+    "The company reported record quarterly profit, beating analyst expectations.",
+    "Net sales decreased by 12% and the firm warned of further losses ahead.",
+    "The board will meet next Tuesday to review the quarterly schedule.",
+]
+
+enc = tokenizer(texts, padding=True, truncation=True, max_length=96, return_tensors="pt")
+with torch.no_grad():
+    probs = model(**enc).logits.softmax(-1)
+
+for text, p in zip(texts, probs):
+    label = model.config.id2label[int(p.argmax())]
+    print(f"[{label:8s} {p.max():.2f}] {text}")
+```
+
+Tested with `torch` 2.8, `transformers` 4.x and `sentencepiece` installed. The three example
+sentences come out positive, negative and neutral.
+
+## Intended use
+
+- Sentiment scoring of **short financial text**: news sentences, headlines, finance tweets, analyst notes.
+- Research and offline or GPU batch scoring where accuracy matters more than latency.
+
+## Limitations
+
+- English only. Max sequence length 96 tokens; longer text is truncated.
+- **Weak on negative press-release headlines**: it finds 2 of the 12 negatives in the JB test split,
+  the same as v3. On the larger SEntFiN set (259 negatives) negative recall is 0.792.
+- On finance tweets about 17% of the tweets it calls negative are labelled neutral or bullish.
+- Tweet labels have a noise ceiling (see above); about 94% is a realistic upper bound on that set.
+- PhraseBank `AllAgree` is the cleanest, easiest part of that benchmark; expect lower accuracy on
+  ambiguous sentences.
+- Tweet labels describe bullish / bearish market direction, which is not the same as the tone of a
+  sentence. The model mixes both conventions.
+- About 19x slower than the compact model on CPU.
+- Not financial advice. Outputs are sentiment labels, not buy/sell signals.
+
+## Citation
+
+Dataset:
+
+```bibtex
+@article{malo2014good,
+  title={Good debt or bad debt: Detecting semantic orientations in economic texts},
+  author={Malo, Pekka and Sinha, Ankur and Korhonen, Pekka and Wallenius, Jyrki and Takala, Pyry},
+  journal={Journal of the Association for Information Science and Technology},
+  volume={65}, number={4}, pages={782--796}, year={2014}
+}
+```
+
+## Contact
+
+Maintainer: nagizaazs@gmail.com
+
+## License
+
+MIT

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "[MASK]": 128000
+}

config.json

@@ -0,0 +1,45 @@
+{
+  "architectures": [
+    "DebertaV2ForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "dtype": "float32",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "negative",
+    "1": "neutral",
+    "2": "positive"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "label2id": {
+    "negative": 0,
+    "neutral": 1,
+    "positive": 2
+  },
+  "layer_norm_eps": 1e-07,
+  "legacy": true,
+  "max_position_embeddings": 512,
+  "max_relative_positions": -1,
+  "model_type": "deberta-v2",
+  "norm_rel_ebd": "layer_norm",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "pad_token_id": 0,
+  "pooler_dropout": 0,
+  "pooler_hidden_act": "gelu",
+  "pooler_hidden_size": 1024,
+  "pos_att_type": [
+    "p2c",
+    "c2p"
+  ],
+  "position_biased_input": false,
+  "position_buckets": 256,
+  "relative_attention": true,
+  "share_att_key": true,
+  "transformers_version": "4.57.6",
+  "type_vocab_size": 0,
+  "vocab_size": 128100
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dba5705c2a3e00e979426826596f6febd25c5c24d2e2b2b83250c860a238d7b1
+size 1740308548

results.json

@@ -0,0 +1,1663 @@
+{
+ "candidates": {
+  "v3": {
+   "spec": "v3",
+   "params": 33361155,
+   "val_mean_mf1": 0.8239,
+   "val_tw_acc": 0.8524,
+   "phrasebank_test": {
+    "acc": 97.94,
+    "mf1": 96.43,
+    "d_acc": 0.0,
+    "d_acc_ci": [
+     0.0,
+     0.0
+    ],
+    "d_mf1": 0.0,
+    "d_mf1_ci": [
+     0.0,
+     0.0
+    ],
+    "P": [
+     0.953,
+     1.0,
+     0.944
+    ],
+    "R": [
+     0.911,
+     0.99,
+     0.988
+    ]
+   },
+   "twitter_valid": {
+    "acc": 86.43,
+    "mf1": 83.0,
+    "d_acc": 0.0,
+    "d_acc_ci": [
+     0.0,
+     0.0
+    ],
+    "d_mf1": 0.0,
+    "d_mf1_ci": [
+     0.0,
+     0.0
+    ],
+    "P": [
+     0.725,
+     0.946,
+     0.752
+    ],
+    "R": [
+     0.85,
+     0.87,
+     0.857
+    ]
+   },
+   "jb_test": {
+    "acc": 75.28,
+    "mf1": 59.04,
+    "d_acc": 0.0,
+    "d_acc_ci": [
+     0.0,
+     0.0
+    ],
+    "d_mf1": 0.0,
+    "d_mf1_ci": [
+     0.0,
+     0.0
+    ],
+    "P": [
+     0.667,
+     0.764,
+     0.733
+    ],
+    "R": [
+     0.167,
+     0.853,
+     0.667
+    ]
+   },
+   "sentfin_test": {
+    "acc": 75.99,
+    "mf1": 76.27,
+    "d_acc": 0.0,
+    "d_acc_ci": [
+     0.0,
+     0.0
+    ],
+    "d_mf1": 0.0,
+    "d_mf1_ci": [
+     0.0,
+     0.0
+    ],
+    "P": [
+     0.85,
+     0.676,
+     0.811
+    ],
+    "R": [
+     0.722,
+     0.825,
+     0.723
+    ]
+   }
+  },
+  "deb_s42": {
+   "spec": "deb_s42",
+   "params": 184424451,
+   "val_mean_mf1": 0.865,
+   "val_tw_acc": 0.8639,
+   "phrasebank_test": {
+    "acc": 99.12,
+    "mf1": 99.01,
+    "d_acc": 1.18,
+    "d_acc_ci": [
+     -0.29,
+     2.94
+    ],
+    "d_mf1": 2.58,
+    "d_mf1_ci": [
+     0.09,
+     5.62
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.977
+    ],
+    "R": [
+     1.0,
+     0.986,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 88.61,
+    "mf1": 86.34,
+    "d_acc": 2.18,
+    "d_acc_ci": [
+     0.96,
+     3.48
+    ],
+    "d_mf1": 3.34,
+    "d_mf1_ci": [
+     1.77,
+     5.02
+    ],
+    "P": [
+     0.815,
+     0.948,
+     0.772
+    ],
+    "R": [
+     0.865,
+     0.883,
+     0.914
+    ]
+   },
+   "jb_test": {
+    "acc": 80.15,
+    "mf1": 62.1,
+    "d_acc": 4.87,
+    "d_acc_ci": [
+     0.37,
+     9.36
+    ],
+    "d_mf1": 3.07,
+    "d_mf1_ci": [
+     -7.61,
+     14.44
+    ],
+    "P": [
+     0.4,
+     0.831,
+     0.775
+    ],
+    "R": [
+     0.167,
+     0.853,
+     0.798
+    ]
+   },
+   "sentfin_test": {
+    "acc": 79.96,
+    "mf1": 80.25,
+    "d_acc": 3.97,
+    "d_acc_ci": [
+     1.29,
+     6.32
+    ],
+    "d_mf1": 3.98,
+    "d_mf1_ci": [
+     1.37,
+     6.27
+    ],
+    "P": [
+     0.873,
+     0.726,
+     0.834
+    ],
+    "R": [
+     0.772,
+     0.804,
+     0.816
+    ]
+   }
+  },
+  "deb_s1": {
+   "spec": "deb_s1",
+   "params": 184424451,
+   "val_mean_mf1": 0.8486,
+   "val_tw_acc": 0.8377,
+   "phrasebank_test": {
+    "acc": 99.41,
+    "mf1": 99.28,
+    "d_acc": 1.47,
+    "d_acc_ci": [
+     -0.29,
+     3.24
+    ],
+    "d_mf1": 2.85,
+    "d_mf1_ci": [
+     0.24,
+     5.99
+    ],
+    "P": [
+     0.978,
+     0.995,
+     1.0
+    ],
+    "R": [
+     1.0,
+     0.995,
+     0.988
+    ]
+   },
+   "twitter_valid": {
+    "acc": 87.48,
+    "mf1": 84.93,
+    "d_acc": 1.05,
+    "d_acc_ci": [
+     -0.21,
+     2.35
+    ],
+    "d_mf1": 1.94,
+    "d_mf1_ci": [
+     0.4,
+     3.55
+    ],
+    "P": [
+     0.7,
+     0.951,
+     0.818
+    ],
+    "R": [
+     0.916,
+     0.863,
+     0.882
+    ]
+   },
+   "jb_test": {
+    "acc": 77.9,
+    "mf1": 65.34,
+    "d_acc": 2.62,
+    "d_acc_ci": [
+     -1.87,
+     7.12
+    ],
+    "d_mf1": 6.3,
+    "d_mf1_ci": [
+     -0.4,
+     14.95
+    ],
+    "P": [
+     1.0,
+     0.809,
+     0.725
+    ],
+    "R": [
+     0.25,
+     0.84,
+     0.747
+    ]
+   },
+   "sentfin_test": {
+    "acc": 79.31,
+    "mf1": 79.65,
+    "d_acc": 3.32,
+    "d_acc_ci": [
+     0.96,
+     5.79
+    ],
+    "d_mf1": 3.37,
+    "d_mf1_ci": [
+     1.02,
+     5.8
+    ],
+    "P": [
+     0.793,
+     0.722,
+     0.885
+    ],
+    "R": [
+     0.846,
+     0.781,
+     0.765
+    ]
+   }
+  },
+  "deb_s2": {
+   "spec": "deb_s2",
+   "params": 184424451,
+   "val_mean_mf1": 0.85,
+   "val_tw_acc": 0.8366,
+   "phrasebank_test": {
+    "acc": 98.82,
+    "mf1": 98.74,
+    "d_acc": 0.88,
+    "d_acc_ci": [
+     -0.59,
+     2.65
+    ],
+    "d_mf1": 2.31,
+    "d_mf1_ci": [
+     -0.18,
+     5.37
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.966
+    ],
+    "R": [
+     1.0,
+     0.981,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 86.64,
+    "mf1": 84.27,
+    "d_acc": 0.21,
+    "d_acc_ci": [
+     -1.09,
+     1.55
+    ],
+    "d_mf1": 1.27,
+    "d_mf1_ci": [
+     -0.3,
+     2.94
+    ],
+    "P": [
+     0.728,
+     0.955,
+     0.755
+    ],
+    "R": [
+     0.911,
+     0.846,
+     0.901
+    ]
+   },
+   "jb_test": {
+    "acc": 74.91,
+    "mf1": 58.68,
+    "d_acc": -0.37,
+    "d_acc_ci": [
+     -5.24,
+     4.12
+    ],
+    "d_mf1": -0.36,
+    "d_mf1_ci": [
+     -4.67,
+     3.33
+    ],
+    "P": [
+     0.4,
+     0.824,
+     0.675
+    ],
+    "R": [
+     0.167,
+     0.75,
+     0.818
+    ]
+   },
+   "sentfin_test": {
+    "acc": 80.39,
+    "mf1": 80.55,
+    "d_acc": 4.39,
+    "d_acc_ci": [
+     1.82,
+     6.97
+    ],
+    "d_mf1": 4.28,
+    "d_mf1_ci": [
+     1.79,
+     6.85
+    ],
+    "P": [
+     0.835,
+     0.753,
+     0.833
+    ],
+    "R": [
+     0.803,
+     0.766,
+     0.843
+    ]
+   }
+  },
+  "deb_soup3": {
+   "spec": "deb_soup3",
+   "params": 184424451,
+   "val_mean_mf1": 0.8375,
+   "val_tw_acc": 0.8031,
+   "phrasebank_test": {
+    "acc": 98.53,
+    "mf1": 98.3,
+    "d_acc": 0.59,
+    "d_acc_ci": [
+     -1.18,
+     2.35
+    ],
+    "d_mf1": 1.87,
+    "d_mf1_ci": [
+     -0.82,
+     5.04
+    ],
+    "P": [
+     0.957,
+     1.0,
+     0.966
+    ],
+    "R": [
+     1.0,
+     0.976,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 83.88,
+    "mf1": 81.55,
+    "d_acc": -2.55,
+    "d_acc_ci": [
+     -3.94,
+     -1.17
+    ],
+    "d_mf1": -1.45,
+    "d_mf1_ci": [
+     -3.02,
+     0.2
+    ],
+    "P": [
+     0.632,
+     0.966,
+     0.742
+    ],
+    "R": [
+     0.939,
+     0.793,
+     0.916
+    ]
+   },
+   "jb_test": {
+    "acc": 78.65,
+    "mf1": 64.37,
+    "d_acc": 3.37,
+    "d_acc_ci": [
+     -0.76,
+     7.49
+    ],
+    "d_mf1": 5.33,
+    "d_mf1_ci": [
+     -1.01,
+     13.48
+    ],
+    "P": [
+     0.5,
+     0.854,
+     0.718
+    ],
+    "R": [
+     0.25,
+     0.788,
+     0.848
+    ]
+   },
+   "sentfin_test": {
+    "acc": 80.6,
+    "mf1": 80.68,
+    "d_acc": 4.61,
+    "d_acc_ci": [
+     1.93,
+     7.4
+    ],
+    "d_mf1": 4.41,
+    "d_mf1_ci": [
+     1.72,
+     7.12
+    ],
+    "P": [
+     0.792,
+     0.793,
+     0.828
+    ],
+    "R": [
+     0.884,
+     0.696,
+     0.858
+    ]
+   }
+  },
+  "rob_s42": {
+   "spec": "rob_s42",
+   "params": 124647939,
+   "val_mean_mf1": 0.8463,
+   "val_tw_acc": 0.8618,
+   "phrasebank_test": {
+    "acc": 99.41,
+    "mf1": 99.28,
+    "d_acc": 1.47,
+    "d_acc_ci": [
+     -0.29,
+     3.24
+    ],
+    "d_mf1": 2.85,
+    "d_mf1_ci": [
+     0.19,
+     6.08
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.989
+    ],
+    "R": [
+     1.0,
+     0.99,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 88.32,
+    "mf1": 85.91,
+    "d_acc": 1.88,
+    "d_acc_ci": [
+     0.63,
+     3.22
+    ],
+    "d_mf1": 2.92,
+    "d_mf1_ci": [
+     1.36,
+     4.55
+    ],
+    "P": [
+     0.774,
+     0.959,
+     0.769
+    ],
+    "R": [
+     0.896,
+     0.872,
+     0.909
+    ]
+   },
+   "jb_test": {
+    "acc": 68.91,
+    "mf1": 54.97,
+    "d_acc": -6.37,
+    "d_acc_ci": [
+     -11.61,
+     -1.12
+    ],
+    "d_mf1": -4.07,
+    "d_mf1_ci": [
+     -15.56,
+     7.66
+    ],
+    "P": [
+     0.5,
+     0.789,
+     0.592
+    ],
+    "R": [
+     0.167,
+     0.673,
+     0.778
+    ]
+   },
+   "sentfin_test": {
+    "acc": 80.17,
+    "mf1": 80.51,
+    "d_acc": 4.18,
+    "d_acc_ci": [
+     1.61,
+     6.65
+    ],
+    "d_mf1": 4.24,
+    "d_mf1_ci": [
+     1.7,
+     6.62
+    ],
+    "P": [
+     0.879,
+     0.737,
+     0.818
+    ],
+    "R": [
+     0.788,
+     0.787,
+     0.828
+    ]
+   }
+  },
+  "rob_s1": {
+   "spec": "rob_s1",
+   "params": 124647939,
+   "val_mean_mf1": 0.8421,
+   "val_tw_acc": 0.8743,
+   "phrasebank_test": {
+    "acc": 99.41,
+    "mf1": 98.99,
+    "d_acc": 1.47,
+    "d_acc_ci": [
+     -0.29,
+     3.24
+    ],
+    "d_mf1": 2.56,
+    "d_mf1_ci": [
+     -0.32,
+     5.81
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.989
+    ],
+    "R": [
+     0.978,
+     0.995,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 88.86,
+    "mf1": 85.98,
+    "d_acc": 2.43,
+    "d_acc_ci": [
+     1.13,
+     3.73
+    ],
+    "d_mf1": 2.98,
+    "d_mf1_ci": [
+     1.35,
+     4.6
+    ],
+    "P": [
+     0.825,
+     0.925,
+     0.82
+    ],
+    "R": [
+     0.83,
+     0.915,
+     0.844
+    ]
+   },
+   "jb_test": {
+    "acc": 72.28,
+    "mf1": 59.7,
+    "d_acc": -3.0,
+    "d_acc_ci": [
+     -7.87,
+     1.87
+    ],
+    "d_mf1": 0.66,
+    "d_mf1_ci": [
+     -12.13,
+     13.68
+    ],
+    "P": [
+     0.5,
+     0.781,
+     0.655
+    ],
+    "R": [
+     0.25,
+     0.756,
+     0.727
+    ]
+   },
+   "sentfin_test": {
+    "acc": 77.17,
+    "mf1": 77.6,
+    "d_acc": 1.18,
+    "d_acc_ci": [
+     -1.29,
+     3.54
+    ],
+    "d_mf1": 1.33,
+    "d_mf1_ci": [
+     -1.05,
+     3.65
+    ],
+    "P": [
+     0.889,
+     0.658,
+     0.862
+    ],
+    "R": [
+     0.714,
+     0.851,
+     0.735
+    ]
+   }
+  },
+  "drob_s42": {
+   "spec": "drob_s42",
+   "params": 82120707,
+   "val_mean_mf1": 0.8389,
+   "val_tw_acc": 0.8586,
+   "phrasebank_test": {
+    "acc": 97.94,
+    "mf1": 97.49,
+    "d_acc": 0.0,
+    "d_acc_ci": [
+     -1.76,
+     1.76
+    ],
+    "d_mf1": 1.06,
+    "d_mf1_ci": [
+     -1.8,
+     4.39
+    ],
+    "P": [
+     0.938,
+     1.0,
+     0.955
+    ],
+    "R": [
+     1.0,
+     0.971,
+     0.988
+    ]
+   },
+   "twitter_valid": {
+    "acc": 87.56,
+    "mf1": 84.57,
+    "d_acc": 1.13,
+    "d_acc_ci": [
+     -0.21,
+     2.43
+    ],
+    "d_mf1": 1.57,
+    "d_mf1_ci": [
+     -0.05,
+     3.27
+    ],
+    "P": [
+     0.781,
+     0.932,
+     0.78
+    ],
+    "R": [
+     0.844,
+     0.89,
+     0.853
+    ]
+   },
+   "jb_test": {
+    "acc": 77.53,
+    "mf1": 62.67,
+    "d_acc": 2.25,
+    "d_acc_ci": [
+     -1.87,
+     6.37
+    ],
+    "d_mf1": 3.63,
+    "d_mf1_ci": [
+     -3.03,
+     12.43
+    ],
+    "P": [
+     0.429,
+     0.784,
+     0.787
+    ],
+    "R": [
+     0.25,
+     0.859,
+     0.707
+    ]
+   },
+   "sentfin_test": {
+    "acc": 75.99,
+    "mf1": 76.39,
+    "d_acc": 0.0,
+    "d_acc_ci": [
+     -2.36,
+     2.14
+    ],
+    "d_mf1": 0.12,
+    "d_mf1_ci": [
+     -2.2,
+     2.21
+    ],
+    "P": [
+     0.851,
+     0.659,
+     0.842
+    ],
+    "R": [
+     0.726,
+     0.825,
+     0.72
+    ]
+   }
+  },
+  "fin_s42": {
+   "spec": "fin_s42",
+   "params": 109484547,
+   "val_mean_mf1": 0.8283,
+   "val_tw_acc": 0.8545,
+   "phrasebank_test": {
+    "acc": 99.71,
+    "mf1": 99.55,
+    "d_acc": 1.76,
+    "d_acc_ci": [
+     0.59,
+     3.24
+    ],
+    "d_mf1": 3.12,
+    "d_mf1_ci": [
+     0.99,
+     5.96
+    ],
+    "P": [
+     0.978,
+     1.0,
+     1.0
+    ],
+    "R": [
+     1.0,
+     0.995,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 87.23,
+    "mf1": 83.85,
+    "d_acc": 0.8,
+    "d_acc_ci": [
+     -0.5,
+     2.05
+    ],
+    "d_mf1": 0.86,
+    "d_mf1_ci": [
+     -0.75,
+     2.43
+    ],
+    "P": [
+     0.751,
+     0.927,
+     0.801
+    ],
+    "R": [
+     0.85,
+     0.895,
+     0.815
+    ]
+   },
+   "jb_test": {
+    "acc": 72.66,
+    "mf1": 57.64,
+    "d_acc": -2.62,
+    "d_acc_ci": [
+     -7.12,
+     1.87
+    ],
+    "d_mf1": -1.39,
+    "d_mf1_ci": [
+     -8.38,
+     7.06
+    ],
+    "P": [
+     0.273,
+     0.756,
+     0.727
+    ],
+    "R": [
+     0.25,
+     0.814,
+     0.646
+    ]
+   },
+   "sentfin_test": {
+    "acc": 75.88,
+    "mf1": 76.2,
+    "d_acc": -0.11,
+    "d_acc_ci": [
+     -2.36,
+     2.14
+    ],
+    "d_mf1": -0.07,
+    "d_mf1_ci": [
+     -2.34,
+     2.16
+    ],
+    "P": [
+     0.87,
+     0.641,
+     0.886
+    ],
+    "R": [
+     0.699,
+     0.883,
+     0.678
+    ]
+   }
+  },
+  "ens_deb42_deb1_v3": {
+   "spec": "deb_s42+deb_s1+v3",
+   "params": 402210057,
+   "val_mean_mf1": 0.87,
+   "val_tw_acc": 0.8639,
+   "phrasebank_test": {
+    "acc": 99.41,
+    "mf1": 99.28,
+    "d_acc": 1.47,
+    "d_acc_ci": [
+     0.0,
+     2.94
+    ],
+    "d_mf1": 2.85,
+    "d_mf1_ci": [
+     0.28,
+     5.79
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.989
+    ],
+    "R": [
+     1.0,
+     0.99,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 88.74,
+    "mf1": 86.38,
+    "d_acc": 2.3,
+    "d_acc_ci": [
+     1.17,
+     3.52
+    ],
+    "d_mf1": 3.38,
+    "d_mf1_ci": [
+     1.99,
+     4.87
+    ],
+    "P": [
+     0.753,
+     0.954,
+     0.812
+    ],
+    "R": [
+     0.914,
+     0.88,
+     0.893
+    ]
+   },
+   "jb_test": {
+    "acc": 77.53,
+    "mf1": 60.59,
+    "d_acc": 2.25,
+    "d_acc_ci": [
+     -1.5,
+     5.99
+    ],
+    "d_mf1": 1.56,
+    "d_mf1_ci": [
+     -1.58,
+     4.64
+    ],
+    "P": [
+     0.5,
+     0.811,
+     0.731
+    ],
+    "R": [
+     0.167,
+     0.827,
+     0.768
+    ]
+   },
+   "sentfin_test": {
+    "acc": 80.39,
+    "mf1": 80.64,
+    "d_acc": 4.39,
+    "d_acc_ci": [
+     2.14,
+     6.54
+    ],
+    "d_mf1": 4.37,
+    "d_mf1_ci": [
+     2.16,
+     6.55
+    ],
+    "P": [
+     0.824,
+     0.733,
+     0.874
+    ],
+    "R": [
+     0.815,
+     0.804,
+     0.795
+    ]
+   }
+  },
+  "ens_deb42_deb1": {
+   "spec": "deb_s42+deb_s1",
+   "params": 368848902,
+   "val_mean_mf1": 0.868,
+   "val_tw_acc": 0.8597,
+   "phrasebank_test": {
+    "acc": 99.41,
+    "mf1": 99.28,
+    "d_acc": 1.47,
+    "d_acc_ci": [
+     0.0,
+     2.94
+    ],
+    "d_mf1": 2.85,
+    "d_mf1_ci": [
+     0.28,
+     5.79
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.989
+    ],
+    "R": [
+     1.0,
+     0.99,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 88.44,
+    "mf1": 86.06,
+    "d_acc": 2.01,
+    "d_acc_ci": [
+     0.79,
+     3.23
+    ],
+    "d_mf1": 3.06,
+    "d_mf1_ci": [
+     1.6,
+     4.62
+    ],
+    "P": [
+     0.74,
+     0.954,
+     0.812
+    ],
+    "R": [
+     0.916,
+     0.875,
+     0.893
+    ]
+   },
+   "jb_test": {
+    "acc": 77.15,
+    "mf1": 56.62,
+    "d_acc": 1.87,
+    "d_acc_ci": [
+     -2.25,
+     5.99
+    ],
+    "d_mf1": -2.42,
+    "d_mf1_ci": [
+     -11.75,
+     4.21
+    ],
+    "P": [
+     0.333,
+     0.806,
+     0.731
+    ],
+    "R": [
+     0.083,
+     0.827,
+     0.768
+    ]
+   },
+   "sentfin_test": {
+    "acc": 79.96,
+    "mf1": 80.24,
+    "d_acc": 3.97,
+    "d_acc_ci": [
+     1.71,
+     6.11
+    ],
+    "d_mf1": 3.97,
+    "d_mf1_ci": [
+     1.64,
+     6.15
+    ],
+    "P": [
+     0.819,
+     0.729,
+     0.87
+    ],
+    "R": [
+     0.819,
+     0.795,
+     0.789
+    ]
+   }
+  },
+  "ens_deb3_v3": {
+   "spec": "deb_s42+deb_s2+deb_s1+v3",
+   "params": 586634508,
+   "val_mean_mf1": 0.8664,
+   "val_tw_acc": 0.8597,
+   "phrasebank_test": {
+    "acc": 99.12,
+    "mf1": 99.01,
+    "d_acc": 1.18,
+    "d_acc_ci": [
+     -0.29,
+     2.94
+    ],
+    "d_mf1": 2.58,
+    "d_mf1_ci": [
+     0.09,
+     5.62
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.977
+    ],
+    "R": [
+     1.0,
+     0.986,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 88.69,
+    "mf1": 86.41,
+    "d_acc": 2.26,
+    "d_acc_ci": [
+     1.09,
+     3.52
+    ],
+    "d_mf1": 3.41,
+    "d_mf1_ci": [
+     2.0,
+     4.91
+    ],
+    "P": [
+     0.751,
+     0.957,
+     0.806
+    ],
+    "R": [
+     0.914,
+     0.875,
+     0.907
+    ]
+   },
+   "jb_test": {
+    "acc": 77.15,
+    "mf1": 60.02,
+    "d_acc": 1.87,
+    "d_acc_ci": [
+     -1.87,
+     5.62
+    ],
+    "d_mf1": 0.98,
+    "d_mf1_ci": [
+     -2.78,
+     4.26
+    ],
+    "P": [
+     0.4,
+     0.814,
+     0.726
+    ],
+    "R": [
+     0.167,
+     0.814,
+     0.778
+    ]
+   },
+   "sentfin_test": {
+    "acc": 79.74,
+    "mf1": 80.03,
+    "d_acc": 3.75,
+    "d_acc_ci": [
+     1.5,
+     5.89
+    ],
+    "d_mf1": 3.76,
+    "d_mf1_ci": [
+     1.52,
+     5.91
+    ],
+    "P": [
+     0.824,
+     0.726,
+     0.864
+    ],
+    "R": [
+     0.811,
+     0.798,
+     0.786
+    ]
+   }
+  },
+  "debL_s42": {
+   "spec": "debL_s42_fp32",
+   "params": 435064835,
+   "val_mean_mf1": 0.8751,
+   "val_tw_acc": 0.8901,
+   "phrasebank_test": {
+    "acc": 99.41,
+    "mf1": 99.28,
+    "d_acc": 1.47,
+    "d_acc_ci": [
+     0.0,
+     3.24
+    ],
+    "d_mf1": 2.85,
+    "d_mf1_ci": [
+     0.28,
+     5.97
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.989
+    ],
+    "R": [
+     1.0,
+     0.99,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 90.16,
+    "mf1": 87.93,
+    "d_acc": 3.73,
+    "d_acc_ci": [
+     2.47,
+     5.07
+    ],
+    "d_mf1": 4.94,
+    "d_mf1_ci": [
+     3.36,
+     6.57
+    ],
+    "P": [
+     0.831,
+     0.947,
+     0.819
+    ],
+    "R": [
+     0.879,
+     0.908,
+     0.897
+    ]
+   },
+   "jb_test": {
+    "acc": 78.28,
+    "mf1": 61.37,
+    "d_acc": 3.0,
+    "d_acc_ci": [
+     -2.25,
+     8.24
+    ],
+    "d_mf1": 2.33,
+    "d_mf1_ci": [
+     -9.45,
+     14.79
+    ],
+    "P": [
+     0.667,
+     0.794,
+     0.766
+    ],
+    "R": [
+     0.167,
+     0.865,
+     0.727
+    ]
+   },
+   "sentfin_test": {
+    "acc": 79.74,
+    "mf1": 80.3,
+    "d_acc": 3.75,
+    "d_acc_ci": [
+     1.29,
+     6.22
+    ],
+    "d_mf1": 4.03,
+    "d_mf1_ci": [
+     1.53,
+     6.51
+    ],
+    "P": [
+     0.891,
+     0.696,
+     0.866
+    ],
+    "R": [
+     0.792,
+     0.836,
+     0.762
+    ]
+   },
+   "note": "this model, fp32 inference"
+  },
+  "robL_s42": {
+   "spec": "robL_s42",
+   "params": 355362819,
+   "val_mean_mf1": 0.8617,
+   "val_tw_acc": 0.8429,
+   "phrasebank_test": {
+    "acc": 98.82,
+    "mf1": 98.57,
+    "d_acc": 0.88,
+    "d_acc_ci": [
+     -0.88,
+     2.65
+    ],
+    "d_mf1": 2.14,
+    "d_mf1_ci": [
+     -0.57,
+     5.26
+    ],
+    "P": [
+     0.957,
+     1.0,
+     0.977
+    ],
+    "R": [
+     1.0,
+     0.981,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 86.77,
+    "mf1": 84.81,
+    "d_acc": 0.34,
+    "d_acc_ci": [
+     -1.01,
+     1.84
+    ],
+    "d_mf1": 1.82,
+    "d_mf1_ci": [
+     0.21,
+     3.61
+    ],
+    "P": [
+     0.711,
+     0.98,
+     0.744
+    ],
+    "R": [
+     0.951,
+     0.823,
+     0.954
+    ]
+   },
+   "jb_test": {
+    "acc": 78.28,
+    "mf1": 67.45,
+    "d_acc": 3.0,
+    "d_acc_ci": [
+     -1.5,
+     7.5
+    ],
+    "d_mf1": 8.41,
+    "d_mf1_ci": [
+     -1.63,
+     19.11
+    ],
+    "P": [
+     0.455,
+     0.838,
+     0.741
+    ],
+    "R": [
+     0.417,
+     0.795,
+     0.808
+    ]
+   },
+   "sentfin_test": {
+    "acc": 80.71,
+    "mf1": 80.9,
+    "d_acc": 4.72,
+    "d_acc_ci": [
+     1.82,
+     7.5
+    ],
+    "d_mf1": 4.62,
+    "d_mf1_ci": [
+     1.79,
+     7.37
+    ],
+    "P": [
+     0.821,
+     0.783,
+     0.818
+    ],
+    "R": [
+     0.865,
+     0.719,
+     0.852
+    ]
+   }
+  },
+  "ens_debL_robL_v3": {
+   "spec": "debL_s42+robL_s42+v3",
+   "params": 823788809,
+   "val_mean_mf1": 0.8772,
+   "val_tw_acc": 0.888,
+   "phrasebank_test": {
+    "acc": 99.12,
+    "mf1": 99.01,
+    "d_acc": 1.18,
+    "d_acc_ci": [
+     -0.29,
+     2.94
+    ],
+    "d_mf1": 2.58,
+    "d_mf1_ci": [
+     0.09,
+     5.62
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.977
+    ],
+    "R": [
+     1.0,
+     0.986,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 90.12,
+    "mf1": 88.19,
+    "d_acc": 3.69,
+    "d_acc_ci": [
+     2.51,
+     4.94
+    ],
+    "d_mf1": 5.19,
+    "d_mf1_ci": [
+     3.76,
+     6.79
+    ],
+    "P": [
+     0.808,
+     0.965,
+     0.802
+    ],
+    "R": [
+     0.919,
+     0.889,
+     0.928
+    ]
+   },
+   "jb_test": {
+    "acc": 79.03,
+    "mf1": 61.01,
+    "d_acc": 3.75,
+    "d_acc_ci": [
+     -0.75,
+     8.24
+    ],
+    "d_mf1": 1.97,
+    "d_mf1_ci": [
+     -8.9,
+     13.14
+    ],
+    "P": [
+     0.333,
+     0.815,
+     0.778
+    ],
+    "R": [
+     0.167,
+     0.846,
+     0.778
+    ]
+   },
+   "sentfin_test": {
+    "acc": 80.92,
+    "mf1": 81.33,
+    "d_acc": 4.93,
+    "d_acc_ci": [
+     2.68,
+     7.29
+    ],
+    "d_mf1": 5.06,
+    "d_mf1_ci": [
+     2.79,
+     7.33
+    ],
+    "P": [
+     0.862,
+     0.729,
+     0.864
+    ],
+    "R": [
+     0.822,
+     0.804,
+     0.804
+    ]
+   }
+  },
+  "ens_debL_deb2_v3": {
+   "spec": "debL_s42+deb_s2+v3",
+   "params": 652850441,
+   "val_mean_mf1": 0.8756,
+   "val_tw_acc": 0.8817,
+   "phrasebank_test": {
+    "acc": 99.12,
+    "mf1": 99.01,
+    "d_acc": 1.18,
+    "d_acc_ci": [
+     -0.29,
+     2.94
+    ],
+    "d_mf1": 2.58,
+    "d_mf1_ci": [
+     0.09,
+     5.62
+    ],
+    "P": [
+     0.978,
+     1.0,
+     0.977
+    ],
+    "R": [
+     1.0,
+     0.986,
+     1.0
+    ]
+   },
+   "twitter_valid": {
+    "acc": 90.62,
+    "mf1": 88.6,
+    "d_acc": 4.19,
+    "d_acc_ci": [
+     2.97,
+     5.4
+    ],
+    "d_mf1": 5.61,
+    "d_mf1_ci": [
+     4.14,
+     7.13
+    ],
+    "P": [
+     0.835,
+     0.956,
+     0.817
+    ],
+    "R": [
+     0.902,
+     0.906,
+     0.909
+    ]
+   },
+   "jb_test": {
+    "acc": 79.4,
+    "mf1": 61.84,
+    "d_acc": 4.12,
+    "d_acc_ci": [
+     -0.37,
+     8.61
+    ],
+    "d_mf1": 2.8,
+    "d_mf1_ci": [
+     -8.47,
+     14.5
+    ],
+    "P": [
+     0.5,
+     0.817,
+     0.768
+    ],
+    "R": [
+     0.167,
+     0.859,
+     0.768
+    ]
+   },
+   "sentfin_test": {
+    "acc": 80.28,
+    "mf1": 80.68,
+    "d_acc": 4.29,
+    "d_acc_ci": [
+     2.04,
+     6.43
+    ],
+    "d_mf1": 4.41,
+    "d_mf1_ci": [
+     2.23,
+     6.54
+    ],
+    "P": [
+     0.872,
+     0.713,
+     0.867
+    ],
+    "R": [
+     0.792,
+     0.827,
+     0.786
+    ]
+   }
+  }
+ },
+ "cpu_latency_ms_batch1_4threads": {
+  "v3": [
+   13.5,
+   18.5
+  ],
+  "deberta-v3-base_s42": [
+   83.6,
+   104.9
+  ],
+  "roberta-large_s42": [
+   127.3,
+   155.6
+  ],
+  "this_model": [
+   259.3,
+   301.2
+  ]
+ },
+ "tweet_error_audit": {
+  "errors": 235,
+  "sampled": 100,
+  "seed": 0,
+  "gold_wrong": 32,
+  "ambiguous": 29,
+  "model_wrong": 39
+ }
+}

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "[CLS]",
+  "cls_token": "[CLS]",
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c679fbf93643d19aab7ee10c0b99e460bdbc02fedf34b92b05af343b4af586fd
+size 2464616

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128000": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "unk_token": "[UNK]",
+  "vocab_type": "spm"
+}

training/analyze.py

@@ -0,0 +1,58 @@
+"""Usage: analyze.py out.json name[=a+b+...] ... - scores models (probs/<name>.npz) or probability-averaged
+ensembles on the held-out sets, with paired bootstrap 95% intervals of the difference vs v3."""
+import sys, json
+import numpy as np
+from sklearn.metrics import f1_score, precision_recall_fscore_support
+
+HELD = ["phrasebank_test", "twitter_valid", "jb_test", "sentfin_test"]
+VAL = ["val_pb", "val_tw", "val_jb"]
+LAB = ["negative", "neutral", "positive"]
+cache = {}
+
+
+def load(name):
+    if name not in cache:
+        cache[name] = dict(np.load(f"probs/{name}.npz"))
+    return cache[name]
+
+
+def probs(spec, k):
+    return np.mean([load(m)[k] for m in spec.split("+")], axis=0)
+
+
+def mf1(g, p):
+    return f1_score(g, p, labels=[0, 1, 2], average="macro", zero_division=0)
+
+
+def paired(g, p, q, n=2000, seed=0):
+    """95% percentile intervals of (candidate - v3) accuracy and macro-F1, same resampled rows for both."""
+    rng = np.random.default_rng(seed)
+    da, df = [], []
+    for _ in range(n):
+        i = rng.integers(0, len(g), len(g))
+        da.append((p[i] == g[i]).mean() - (q[i] == g[i]).mean())
+        df.append(mf1(g[i], p[i]) - mf1(g[i], q[i]))
+    c = lambda a: [round(float(np.percentile(a, 2.5)) * 100, 2), round(float(np.percentile(a, 97.5)) * 100, 2)]
+    return c(da), c(df)
+
+
+out = {}
+path, specs = sys.argv[1], sys.argv[2:]
+for s in specs:
+    name, spec = s.split("=") if "=" in s else (s, s)
+    r = {"spec": spec, "params": int(sum(load(m)["params"] for m in spec.split("+")))}
+    r["val_mean_mf1"] = round(float(np.mean([mf1(load(spec.split("+")[0])[k + "_gold"], probs(spec, k).argmax(1)) for k in VAL])), 4)
+    r["val_tw_acc"] = round(float((probs(spec, "val_tw").argmax(1) == load(spec.split("+")[0])["val_tw_gold"]).mean()), 4)
+    for k in HELD:
+        g = load("v3")[k + "_gold"]
+        p, q = probs(spec, k).argmax(1), load("v3")[k].argmax(1)
+        P, R, _, _ = precision_recall_fscore_support(g, p, labels=[0, 1, 2], zero_division=0)
+        ca, cf = paired(g, p, q)
+        r[k] = {"acc": round(float((p == g).mean()) * 100, 2), "mf1": round(mf1(g, p) * 100, 2),
+                "d_acc": round(float((p == g).mean() - (q == g).mean()) * 100, 2), "d_acc_ci": ca,
+                "d_mf1": round((mf1(g, p) - mf1(g, q)) * 100, 2), "d_mf1_ci": cf,
+                "P": [round(float(x), 3) for x in P], "R": [round(float(x), 3) for x in R]}
+    out[name] = r
+    print(f"{name:28s} {r['params']/1e6:6.1f}M val {r['val_mean_mf1']:.4f} valtw {r['val_tw_acc']:.4f} | " + " | ".join(
+        f"{k[:5]} {r[k]['acc']:.2f} ({r[k]['d_acc']:+.2f} [{r[k]['d_acc_ci'][0]:+.2f},{r[k]['d_acc_ci'][1]:+.2f}]) F1 {r[k]['mf1']:.2f}" for k in HELD), flush=True)
+json.dump(out, open(path, "w"), indent=1)

training/app.py

@@ -0,0 +1,205 @@
+"""Resumable fine-tuning and held-out scoring for financial-sentiment encoders (private working Space).
+train_chunk trains for at most one GPU call and pushes resumable state to a private model repo; evaluate
+returns class probabilities on the four held-out sets and the three training-derived validation slices."""
+import os, json, time
+import numpy as np, torch, gradio as gr, spaces
+from huggingface_hub import HfApi, snapshot_download
+from transformers import AutoModelForSequenceClassification, AutoTokenizer, get_linear_schedule_with_warmup
+from common import LABELS, L2I, dnorm, carve, phrasebank_split, phrasebank_extra, nosible, twitter, jb, heldout_sets, report
+
+TOKEN = os.environ.get("HF_TOKEN")
+api = HfApi(token=TOKEN)
+
+# Data is built once at start-up (CPU). Held-out and validation text never enters training.
+HELD = heldout_sets()
+(PBX, PBY), (PBVX, PBVY), _ = phrasebank_split()
+TWX, TWVX, TWY, TWVY = carve(*twitter("train"))
+JBX, JBVX, JBY, JBVY = carve(*jb("train"))
+VAL = {"pb": (PBVX, PBVY), "tw": (TWVX, TWVY), "jb": (JBVX, JBVY)}
+BLOCK = {dnorm(t) for X, _ in list(HELD.values()) + list(VAL.values()) for t in X}
+# Fetch base weights at start-up so downloads are not billed as GPU time.
+for _b in os.environ.get("PRELOAD", "microsoft/deberta-v3-base,FacebookAI/roberta-base").split(","):
+    snapshot_download(_b, ignore_patterns=["*.h5", "*.msgpack", "*.ot", "onnx/*", "tf_*", "flax_*", "rust_*"])
+
+
+def build_train(c):
+    exX, exy = phrasebank_extra(c["pb_extra"]) if c.get("pb_extra") else ([], [])
+    nsX, nsy = nosible(c["nosible"]) if c.get("nosible") else ([], [])
+    raw = (list(zip(PBX, PBY)) * c.get("pb_rep", 3) + list(zip(TWX, TWY)) + list(zip(JBX, JBY)) * c.get("jb_rep", 1)
+           + list(zip(exX, exy)) * c.get("extra_rep", 1) + list(zip(nsX, nsy)))
+    return [(t, l) for t, l in raw if dnorm(t) not in BLOCK], len(raw)
+
+
+@torch.no_grad()
+def probs(model, tok, texts, bs=128, amp=True):
+    model.eval()
+    order = [{v.lower(): int(k) for k, v in model.config.id2label.items()}[l] for l in LABELS]
+    out = []
+    for i in range(0, len(texts), bs):
+        enc = tok(texts[i:i + bs], padding=True, truncation=True, max_length=96, return_tensors="pt").to("cuda")
+        with torch.autocast("cuda", dtype=torch.bfloat16, enabled=amp):
+            out.append(model(**enc).logits.float().softmax(-1)[:, order].cpu().numpy())
+    return np.concatenate(out)
+
+
+def _train_dur(config_json, resume_repo):
+    return int(json.loads(config_json).get("gpu_seconds", 120))
+
+
+@spaces.GPU(duration=_train_dur)
+def _train_gpu(config_json, resume_repo):
+    t0 = time.time()
+    c = json.loads(config_json)
+    budget = c.get("gpu_seconds", 120) - c.get("margin", 45)
+    work = f"/tmp/runs/{c['run']}"
+    seed, bs, epochs = c.get("seed", 42), c.get("bs", 16), c.get("epochs", 3)
+    torch.manual_seed(seed)
+    train, n_raw = build_train(c)
+    tok = AutoTokenizer.from_pretrained(c["base"])
+    model = AutoModelForSequenceClassification.from_pretrained(
+        c["base"], num_labels=3, id2label=dict(enumerate(LABELS)), label2id=L2I, torch_dtype=torch.float32,
+        ignore_mismatched_sizes=True).cuda()
+    counts = np.bincount([L2I[l] for _, l in train], minlength=3)
+    w = len(train) / (3 * counts)
+    loss_fn = torch.nn.CrossEntropyLoss(weight=torch.tensor(w, dtype=torch.float32, device="cuda"), label_smoothing=c.get("ls", 0.0))
+    opt = torch.optim.AdamW(model.parameters(), lr=c["lr"], weight_decay=0.01)
+    steps_ep = (len(train) + bs - 1) // bs
+    total = steps_ep * epochs
+    sched = get_linear_schedule_with_warmup(opt, int(c.get("warmup", 0.1) * total), total)
+    eval_every = steps_ep // c.get("evals_per_epoch", 2)
+    st = {"step": 0, "best": -1.0, "bad": 0, "hist": [], "done": False, "train_rows": len(train), "dropped": n_raw - len(train)}
+    if os.path.exists(f"{work}/state.json") and os.path.exists(f"{work}/state.pt"):
+        st = json.load(open(f"{work}/state.json"))
+        if not st["done"]:
+            s = torch.load(f"{work}/state.pt", map_location="cuda", weights_only=False)
+            model.load_state_dict(s["model"]); opt.load_state_dict(s["opt"]); sched.load_state_dict(s["sched"])
+            torch.set_rng_state(s["rng"])
+    start = st["step"]
+    while not st["done"] and st["step"] < total:
+        ep, i = divmod(st["step"], steps_ep)
+        idx = np.random.RandomState(seed * 1000 + ep).permutation(len(train))[i * bs:(i + 1) * bs]
+        batch = [train[j] for j in idx]
+        model.train()
+        enc = tok([t for t, _ in batch], padding=True, truncation=True, max_length=96, return_tensors="pt").to("cuda")
+        with torch.autocast("cuda", dtype=torch.bfloat16):
+            logits = model(**enc).logits
+        loss = loss_fn(logits.float(), torch.tensor([L2I[l] for _, l in batch], device="cuda"))
+        if not torch.isfinite(loss):
+            raise gr.Error(f"non-finite loss at step {st['step']}")
+        loss.backward(); torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+        opt.step(); sched.step(); opt.zero_grad(); st["step"] += 1
+        if st["step"] % eval_every == 0 or st["step"] == total:
+            r = {k: report(y, [LABELS[j] for j in probs(model, tok, X).argmax(1)]) for k, (X, y) in VAL.items()}
+            s = float(np.mean([v["macro_f1"] for v in r.values()]))
+            st["hist"].append({"step": st["step"], "val_mean_f1": round(s, 4), "f1": {k: round(v["macro_f1"], 4) for k, v in r.items()},
+                               "acc": {k: round(v["accuracy"], 4) for k, v in r.items()}, "loss": round(loss.item(), 4)})
+            if s > st["best"]:
+                st["best"], st["bad"] = s, 0
+                model.save_pretrained(f"{work}/best"); tok.save_pretrained(f"{work}/best")
+            else:
+                st["bad"] += 1
+                st["done"] = st["bad"] >= c.get("patience", 2)
+        if time.time() - t0 > budget:
+            break
+    st["done"] = st["done"] or st["step"] >= total
+    st["config"] = c
+    if not st["done"]:
+        torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "sched": sched.state_dict(), "rng": torch.get_rng_state()}, f"{work}/state.pt")
+    json.dump(st, open(f"{work}/state.json", "w"), indent=1)
+    return json.dumps({"run": c["run"], "steps_this_call": st["step"] - start, "step": st["step"], "total": total, "done": st["done"],
+                       "best": st["best"], "last_val": st["hist"][-1] if st["hist"] else None, "gpu_wall_s": round(time.time() - t0, 1)})
+
+
+def train_chunk(config_json, resume_repo):
+    c = json.loads(config_json)
+    work = f"/tmp/runs/{c['run']}"
+    os.makedirs(work, exist_ok=True)
+    api.create_repo(resume_repo, private=True, exist_ok=True)
+    if "state.json" in api.list_repo_files(resume_repo) and not os.path.exists(f"{work}/state.json"):
+        snapshot_download(resume_repo, local_dir=work, token=TOKEN)
+    out = json.loads(_train_gpu(config_json, resume_repo))
+    ignore = ["state.pt"] if out["done"] else []
+    api.upload_folder(folder_path=work, repo_id=resume_repo, commit_message=f"step {out['step']}", ignore_patterns=ignore)
+    return json.dumps(out)
+
+
+@spaces.GPU(duration=90)
+def _eval_gpu(path, amp=True):
+    t0 = time.time()
+    tok = AutoTokenizer.from_pretrained(path)
+    model = AutoModelForSequenceClassification.from_pretrained(path, torch_dtype=torch.float32).cuda()
+    sets = dict(HELD) | {"val_pb": VAL["pb"], "val_tw": VAL["tw"], "val_jb": VAL["jb"]}
+    out = {"params": sum(p.numel() for p in model.parameters())}
+    for k, (X, y) in sets.items():
+        out[k] = np.round(probs(model, tok, X, amp=amp), 5).tolist()
+        out[k + "_gold"] = [LABELS.index(l) for l in y]
+    out["gpu_wall_s"] = round(time.time() - t0, 1)
+    return json.dumps(out)
+
+
+def evaluate(model_repo, subfolder):
+    """subfolder may end in ":fp32" to score without bf16 autocast (what a default fp32 load gives)."""
+    subfolder, _, prec = subfolder.partition(":")
+    path = snapshot_download(model_repo, token=TOKEN, allow_patterns=[f"{subfolder}/*"] if subfolder else None,
+                             ignore_patterns=["state.pt", "*.bin"] if not subfolder else None)
+    return _eval_gpu(os.path.join(path, subfolder) if subfolder else path, amp=prec != "fp32")
+
+
+def soup(repos, out_repo):
+    """CPU only: uniform average of the best checkpoints of same-architecture runs, pushed to out_repo/best."""
+    from safetensors.torch import load_file, save_file
+    import shutil
+    paths = [os.path.join(snapshot_download(r, token=TOKEN, allow_patterns=["best/*"]), "best") for r in repos.split(",")]
+    sds = [load_file(f"{p}/model.safetensors") for p in paths]
+    avg = {k: (sum(sd[k].float() for sd in sds) / len(sds)).to(sds[0][k].dtype) for k in sds[0]}
+    work = f"/tmp/soup/{out_repo.split('/')[-1]}/best"
+    shutil.copytree(paths[0], work, dirs_exist_ok=True)
+    save_file({k: v.contiguous() for k, v in avg.items()}, f"{work}/model.safetensors", metadata={"format": "pt"})
+    api.create_repo(out_repo, private=True, exist_ok=True)
+    api.upload_folder(folder_path=os.path.dirname(work), repo_id=out_repo, commit_message=f"average of {repos}")
+    return json.dumps({"averaged": len(sds), "out": out_repo})
+
+
+def latency(model_repo, subfolder):
+    """CPU only: batch-1 latency over 300 tweet-validation texts after 20 warm-up calls, 4 threads; also
+    returns the labels the loaded model gives three fixed sentences (a load check)."""
+    torch.set_num_threads(4)
+    kw = {"subfolder": subfolder} if subfolder else {}
+    tok = AutoTokenizer.from_pretrained(model_repo, token=TOKEN, **kw)
+    model = AutoModelForSequenceClassification.from_pretrained(model_repo, token=TOKEN, **kw).eval()
+    X = HELD["twitter_valid"][0][:320]
+    ts = []
+    with torch.no_grad():
+        for i, t in enumerate(X):
+            t0 = time.perf_counter()
+            model(**tok(t, truncation=True, max_length=96, return_tensors="pt"))
+            if i >= 20:
+                ts.append((time.perf_counter() - t0) * 1000)
+        demo_texts = ["The company reported record quarterly profit, beating analyst expectations.",
+                      "Net sales decreased by 12% and the firm warned of further losses ahead.",
+                      "The board will meet next Tuesday to review the quarterly schedule."]
+        enc = tok(demo_texts, padding=True, truncation=True, max_length=96, return_tensors="pt")
+        labels = [model.config.id2label[int(i)] for i in model(**enc).logits.argmax(-1)]
+    return json.dumps({"params": sum(p.numel() for p in model.parameters()), "threads": torch.get_num_threads(), "cpu_count": os.cpu_count(),
+                       "median_ms": round(float(np.median(ts)), 1), "p90_ms": round(float(np.percentile(ts, 90)), 1), "demo_labels": labels})
+
+
+def _probe_dur(seconds):
+    return int(seconds)
+
+
+@spaces.GPU(duration=_probe_dur)
+def probe(seconds):
+    return json.dumps({"device": torch.cuda.get_device_name(0), "torch": torch.__version__, "requested": int(seconds)})
+
+
+with gr.Blocks() as demo:
+    gr.Markdown("Private working Space: resumable fine-tuning and held-out scoring.")
+    cfg, repo, sub, sec, out = gr.Textbox(label="config"), gr.Textbox(label="repo"), gr.Textbox(label="subfolder"), gr.Number(label="seconds"), gr.Textbox(label="out")
+    gr.Button("train").click(train_chunk, [cfg, repo], out, api_name="train_chunk")
+    gr.Button("eval").click(evaluate, [repo, sub], out, api_name="evaluate")
+    gr.Button("probe").click(probe, [sec], out, api_name="probe")
+    gr.Button("soup").click(soup, [repo, sub], out, api_name="soup")
+    gr.Button("latency").click(latency, [repo, sub], out, api_name="latency")
+
+demo.queue(default_concurrency_limit=1).launch()

training/common.py

@@ -0,0 +1,154 @@
+"""Shared data loading and evaluation for old-vs-new comparison (same code for both models)."""
+import csv, zipfile, json, sys, re
+import numpy as np, torch
+import pyarrow.parquet as pq
+from huggingface_hub import hf_hub_download
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import accuracy_score, f1_score, precision_recall_fscore_support
+
+LABELS = ["negative", "neutral", "positive"]
+L2I = {l: i for i, l in enumerate(LABELS)}
+TW_REV = "ccbe24de388e287beb92dd393a335c376b350ac3"
+JB_REV = "547fa98342d2c80734a1c841dbf3de2cfcaaab95"
+TW_MAP = {"0": "negative", "1": "positive", "2": "neutral"}  # Bearish, Bullish, Neutral
+JB_MAP = {0: "negative", 1: "neutral", 2: "positive"}  # ClassLabel names in dataset_infos.json
+
+
+PB_REV = "8d3fe0c36d5feec6b3cc5e455b0fcb4820fb9964"
+SF_REV = "d6294b768091ae7198051294c7fdeee2fbfbd224"
+NS_REV = "9bd8206a729ccd64fa259f0368f0b4025bb472af"
+
+
+def norm(t):
+    return " ".join(t.lower().split())
+
+
+def dnorm(t):
+    """Dedupe key: lower-case, URLs removed, punctuation removed, whitespace collapsed."""
+    t = re.sub(r"(https?://\S+|www\.\S+)", " ", t.lower())
+    return " ".join(re.sub(r"[^\w\s]|_", " ", t).split())
+
+
+def carve(X, y):
+    """90/10 stratified train/validation slice of a training file (random_state=42)."""
+    return train_test_split(X, y, test_size=0.10, stratify=y, random_state=42)
+
+
+def phrasebank(agree="AllAgree"):  # AllAgree, 75Agree, 66Agree or 50Agree
+    z = zipfile.ZipFile(hf_hub_download("takala/financial_phrasebank", "data/FinancialPhraseBank-v1.0.zip", repo_type="dataset", revision=PB_REV))
+    name = [x for x in z.namelist() if x.endswith(f"Sentences_{agree}.txt") and not x.startswith("__MACOSX")][0]
+    rows = [l.rsplit("@", 1) for l in z.read(name).decode("latin-1").splitlines() if "@" in l]
+    return [t.strip() for t, _ in rows], [s.strip() for _, s in rows]
+
+
+def phrasebank_split():
+    """Stratified 70/15/15, random_state=42: test first, then validation from the rest."""
+    X, y = phrasebank()
+    Xtv, Xte, ytv, yte = train_test_split(X, y, test_size=0.15, stratify=y, random_state=42)
+    Xtr, Xva, ytr, yva = train_test_split(Xtv, ytv, test_size=0.15 / 0.85, stratify=ytv, random_state=42)
+    return (Xtr, ytr), (Xva, yva), (Xte, yte)
+
+
+def twitter(split):  # "train" or "valid"
+    p = hf_hub_download("zeroshot/twitter-financial-news-sentiment", f"sent_{split}.csv", repo_type="dataset", revision=TW_REV)
+    rows = list(csv.DictReader(open(p, newline="", encoding="utf-8")))
+    return [r["text"] for r in rows], [TW_MAP[r["label"]] for r in rows]
+
+
+def jb(split):  # "train" or "test"; text = headline (title)
+    f = {"train": "data/train-00000-of-00001-f6c9e78174b9882e.parquet", "test": "data/test-00000-of-00001-f8c02546a74f96bb.parquet"}[split]
+    rows = pq.read_table(hf_hub_download("Jean-Baptiste/financial_news_sentiment", f, repo_type="dataset", revision=JB_REV)).to_pylist()
+    return [r["title"] for r in rows], [JB_MAP[int(r["labels"])] for r in rows]
+
+
+def phrasebank_extra(agree):
+    """Sentences in a lower-agreement PhraseBank file that are not in AllAgree at all (so none of the
+    AllAgree test or validation sentences can leak in); label = the file's majority label."""
+    allagree = {dnorm(t) for t in phrasebank()[0]}
+    X, y = phrasebank(agree)
+    keep = [(t, l) for t, l in zip(X, y) if dnorm(t) not in allagree]
+    return [t for t, _ in keep], [l for _, l in keep]
+
+
+def sentfin(split="test"):
+    """SEntFiN headlines (tasksource/sentfin repackaging, 80/10/10 split grouped by headline). The source
+    labels each entity in a headline; a headline is kept only when every entity has the same label."""
+    rows = pq.read_table(hf_hub_download("tasksource/sentfin", f"data/{split}-00000-of-00001.parquet", repo_type="dataset", revision=SF_REV)).to_pylist()
+    by = {}
+    for r in rows:
+        hid = json.loads(r["metadata"])["headline_id"]
+        by.setdefault(hid, (r["sentence1"], set()))[1].add(r["labels"])
+    keep = [(t, next(iter(ls))) for t, ls in by.values() if len(ls) == 1]
+    return [t for t, _ in keep], [l for _, l in keep]
+
+
+def nosible(n, max_words=60, seed=42):
+    """n random short snippets (<= max_words words) from NOSIBLE/financial-sentiment; its labels come from an
+    automated multi-annotator pipeline (see that dataset card), not from human annotators."""
+    import random
+    t = pq.read_table(hf_hub_download("NOSIBLE/financial-sentiment", "data.parquet", repo_type="dataset", revision=NS_REV), columns=["text", "label"]).to_pylist()
+    t = [r for r in t if len(r["text"].split()) <= max_words and r["label"] in L2I]
+    t = random.Random(seed).sample(t, min(n, len(t)))
+    return [r["text"] for r in t], [r["label"] for r in t]
+
+
+def device():
+    import os
+    return torch.device("mps" if torch.backends.mps.is_available() and not os.environ.get("FORCE_CPU") else "cpu")
+
+
+@torch.no_grad()
+def predict(model, tok, texts, max_len=96, bs=64, dev=None):
+    dev = dev or next(model.parameters()).device
+    model.eval()
+    id2l = {int(k): v.lower() for k, v in model.config.id2label.items()}
+    out = []
+    for i in range(0, len(texts), bs):
+        # fixed padding buckets: each new batch shape makes the MPS allocator keep another set of buffers
+        enc = tok(texts[i:i + bs], padding=True, truncation=True, max_length=max_len, pad_to_multiple_of=32, return_tensors="pt").to(dev)
+        out += [id2l[int(j)] for j in model(**enc).logits.argmax(-1).cpu()]
+    if dev.type == "mps":
+        torch.mps.empty_cache()
+    return out
+
+
+def report(gold, pred):
+    p, r, f, s = precision_recall_fscore_support(gold, pred, labels=LABELS, zero_division=0)
+    return {"n": len(gold), "accuracy": accuracy_score(gold, pred), "macro_f1": f1_score(gold, pred, labels=LABELS, average="macro"),
+            "per_class": {l: {"precision": float(p[i]), "recall": float(r[i]), "f1": float(f[i]), "support": int(s[i])} for i, l in enumerate(LABELS)}}
+
+
+def heldout_sets():
+    _, _, (Xte, yte) = phrasebank_split()
+    return {"phrasebank_test": (Xte, yte), "twitter_valid": twitter("valid"), "jb_test": jb("test"), "sentfin_test": sentfin("test")}
+
+
+def bootstrap(gold, pred, n=2000, seed=0):
+    """95% percentile intervals for macro-F1 and negative recall, resampling rows with replacement."""
+    from sklearn.metrics import recall_score
+    rng = np.random.default_rng(seed)
+    g, p = np.array(gold), np.array(pred)
+    mf, nr = [], []
+    for _ in range(n):
+        i = rng.integers(0, len(g), len(g))
+        mf.append(f1_score(g[i], p[i], labels=LABELS, average="macro", zero_division=0))
+        nr.append(recall_score(g[i], p[i], labels=["negative"], average="macro", zero_division=0))
+    q = lambda a: [float(np.percentile(a, 2.5)), float(np.percentile(a, 97.5))]
+    return {"macro_f1_ci": q(mf), "neg_recall_ci": q(nr)}
+
+
+def evaluate(model, tok):
+    out = {}
+    for k, (X, y) in heldout_sets().items():
+        pred = predict(model, tok, X)
+        out[k] = {**report(y, pred), **bootstrap(y, pred), "pred": pred}
+    return out
+
+
+def fmt(res):
+    lines = []
+    for k, v in res.items():
+        pc = "  ".join(f"{l[:3]} P{c['precision']:.3f} R{c['recall']:.3f} (n={c['support']})" for l, c in v["per_class"].items())
+        ci = f" [{v['macro_f1_ci'][0]:.3f},{v['macro_f1_ci'][1]:.3f}]" if "macro_f1_ci" in v else ""
+        lines.append(f"{k:16s} n={v['n']:5d} acc={v['accuracy']:.4f} mF1={v['macro_f1']:.4f}{ci} negF1={v['per_class']['negative']['f1']:.3f} | {pc}")
+    return "\n".join(lines)

training/drive.py

@@ -0,0 +1,77 @@
+"""Light client for the private training Space (no local model work).
+Usage: drive.py stage | probe <s> | train <run> <config-json> [max_calls] | eval <run> <repo> <subfolder|-> <out.npz>
+Each GPU call's measured seconds are appended to GPU_LOG.md; train stops before the daily cap."""
+import sys, json, time, re
+from pathlib import Path
+import numpy as np
+from huggingface_hub import HfApi
+
+W = Path(__file__).parent
+SPACE = "9mark9/trainer-tmp"
+CAP_S = 35 * 60
+TOK = Path.home().joinpath(".cache/huggingface/token").read_text().strip()
+LOG = W / "GPU_LOG.md"
+
+
+def used_today():
+    """GPU seconds logged since the first logged call of the current 24 h window."""
+    if not LOG.exists():
+        return 0.0
+    rows = [l.split("|") for l in LOG.read_text().splitlines() if re.match(r"\| \d{4}-", l)]
+    rows = [(time.mktime(time.strptime(r[1].strip(), "%Y-%m-%d %H:%M:%S")), float(r[4])) for r in rows]
+    start = None
+    for t, _ in rows:
+        if start is None or t - start >= 86400:
+            start = t
+    # 10% + 5 s per call on top of the measured time, to cover allocation overhead the function cannot see
+    return sum(s * 1.1 + 5 for t, s in rows if t >= start) if start else 0.0
+
+
+def log(kind, name, gpu_s, wall_s, note=""):
+    if not LOG.exists():
+        LOG.write_text("# ZeroGPU usage (9mark9/trainer-tmp)\n\nSeconds are measured inside the GPU function (load + work).\n"
+                       "Quota window: 24 h from the first call.\n\n| time | call | name | gpu_s | client_wall_s | note |\n|---|---|---|---|---|---|\n")
+    with LOG.open("a") as f:
+        f.write(f"| {time.strftime('%Y-%m-%d %H:%M:%S')} | {kind} | {name} | {gpu_s:.1f} | {wall_s:.1f} | {note} |\n")
+
+
+def client():
+    from gradio_client import Client
+    return Client(SPACE, token=TOK, verbose=False)
+
+
+cmd = sys.argv[1]
+if cmd == "stage":
+    i = HfApi(token=TOK).space_info(SPACE)
+    print(i.runtime.stage, i.runtime.hardware, f"used today {used_today()/60:.1f} min")
+elif cmd == "probe":
+    t0 = time.time()
+    r = client().predict(float(sys.argv[2]), api_name="/probe")
+    log("probe", sys.argv[2], time.time() - t0, time.time() - t0, "upper bound (client wall)")
+    print(r)
+elif cmd == "train":
+    run, cfg, max_calls = sys.argv[2], json.loads(sys.argv[3]), int(sys.argv[4]) if len(sys.argv) > 4 else 6
+    cfg["run"] = run
+    c = client()
+    for _ in range(max_calls):
+        if used_today() + cfg.get("gpu_seconds", 120) > CAP_S:
+            print(f"STOP cap: used {used_today()/60:.1f} min"); break
+        t0 = time.time()
+        r = json.loads(c.predict(json.dumps(cfg), f"9mark9/tmp-train-{run}", api_name="/train_chunk"))
+        log("train", run, r["gpu_wall_s"], time.time() - t0, f"step {r['step']}/{r['total']} done={r['done']} best={r['best']:.4f}")
+        print(json.dumps(r), flush=True)
+        if r["done"]:
+            break
+elif cmd == "latency":
+    print(client().predict(sys.argv[2], "" if sys.argv[3] == "-" else sys.argv[3], api_name="/latency"))
+elif cmd == "soup":
+    print(client().predict(sys.argv[2], sys.argv[3], api_name="/soup"))
+elif cmd == "eval":
+    run, repo, sub, out = sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5]
+    if used_today() + 90 > CAP_S:
+        sys.exit(f"STOP cap: used {used_today()/60:.1f} min")
+    t0 = time.time()
+    r = json.loads(client().predict(repo, "" if sub == "-" else sub, api_name="/evaluate"))
+    log("eval", run, r.pop("gpu_wall_s"), time.time() - t0)
+    np.savez(W / out, params=np.array(r.pop("params")), **{k: np.array(v) for k, v in r.items()})
+    print("saved", out, {k: round(float((np.array(r[k]).argmax(1) == np.array(r[k + "_gold"])).mean()), 4) for k in r if not k.endswith("_gold")})

training/requirements.txt

@@ -0,0 +1,9 @@
+torch==2.8.0
+transformers>=4.51,<5
+sentencepiece
+protobuf
+scikit-learn
+pyarrow
+numpy
+spaces
+gradio==5.50.0