Note that the checkpoint was retrained and its metrics may differ from the paper

3e15858 verified about 1 month ago

15.1 kB

---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
base_model: MAPAi/InflaBERT
base_model_relation: finetune
tags:
- economics
- finance
- inflation
- inflation-expectations
- reddit
- roberta
- distilroberta
metrics:
- accuracy
- f1
- roc_auc
widget:
- text: Fed officials warn prices will keep climbing as CPI hits 40-year high
  example_title: Up
- text: Inflation expectations drop to lowest level since 2021, NY Fed survey shows
  example_title: Down
- text: What is the difference between CPI and PCE?
  example_title: Neutral
- text: Rent and grocery bills are getting out of hand this year
  example_title: Up (colloquial)
- text: Gas prices fall for the sixth straight week as demand cools
  example_title: Down (prices)
model-index:
- name: Reddit-pulse InflaBERT
  results:
  - task:
      type: text-classification
      name: Directional inflation-expectation classification (up / neutral / down)
    dataset:
      name: Reddit inflation gold set (Del Monaco, Longo, Marcucci & Tafani, 2026), held-out test split of seed 2786505123
      type: reddit-inflation-gold-set
      split: test
    metrics:
    - type: accuracy
      value: 0.7410
      name: Accuracy
    - type: f1
      value: 0.7387
      name: F1 (weighted)
    - type: f1
      value: 0.7216
      name: F1 (macro)
    - type: roc_auc
      value: 0.8716
      name: ROC-AUC (macro, one-vs-rest)
---
# Reddit-pulse InflaBERT

> **Retrained checkpoint.** This model was trained again after the paper, on the
> same gold dataset and with the same seed protocol. The metrics on this card come
> from that retraining and may differ from the ones reported in the paper.

**Reddit-pulse InflaBERT** is a three-way *directional* inflation-expectation
classifier for short English texts about the economy, fine-tuned on Reddit
submission titles. Given a title (or any sentence-length text), it predicts
whether the text conveys that inflation / prices are going **up**, going
**down**, or carries **no directional signal** (**neutral**).

It is *not* a sentiment model: "inflation falls sharply" is *good* news but
labelled **down**; "rents are out of control" is *bad* news but labelled
**up**. The direction is about the price level, not the mood.

The checkpoint is the model behind the Reddit inflation signal in

> Del Monaco, A., Longo, L., Marcucci, J. & Tafani, I. (2026).
> *Reddit's 'pulse' on US inflation: forecasting with large language models.*
> Journal of Applied Econometrics, forthcoming.
> Working-paper version: Banca d'Italia, Questioni di Economia e Finanza
> (Occasional Papers) No. 1028, June 2026, doi:10.32057/0.QEF.2026.1028.

<!-- TODO: add the JAE DOI / article URL once assigned. -->

The fine-tuning and full-corpus inference code lives at
[andrea-dm/reddit-pulse](https://github.com/andrea-dm/reddit-pulse).

## Model lineage

| Stage | Model | Notes |
|---|---|---|
| Pre-training | [`distilroberta-base`](https://huggingface.co/distilroberta-base) | 6-layer distilled RoBERTa, 82.1 M parameters |
| Domain adaptation | [`mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis`](https://huggingface.co/mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis) | Financial-news sentiment |
| Task adaptation | [`MAPAi/InflaBERT`](https://huggingface.co/MAPAi/InflaBERT) | Inflation-news sentiment (negative / neutral / positive) |
| **This checkpoint** | **`andreadm/reddit-pulse-bert`** | Full fine-tuning on Reddit titles, labels re-mapped to **down / neutral / up** |

Architecture: `RobertaForSequenceClassification`, 6 hidden layers, hidden
size 768, 12 attention heads, 50 265-token byte-level BPE vocabulary, 512-token
context. Every parameter was updated (no adapters, no quantization); the
classification head was re-initialised for the three directional classes.

## Labels

| id | label | meaning | encoding used in the paper's corpus files |
|---|---|---|---|
| 0 | `down` | inflation / prices heading lower, disinflation, falling expectations | `-1` |
| 1 | `neutral` | no directional signal (questions, definitions, unrelated, mixed) | `0` |
| 2 | `up` | inflation / prices heading higher, rising expectations | `+1` |

## How to use

```python
from transformers import pipeline
clf = pipeline("text-classification", model="andreadm/reddit-pulse-bert")
clf("Fed officials warn prices will keep climbing as CPI hits 40-year high")
# [{'label': 'up', 'score': 0.999}]
```

Batched, with the paper's `-1 / 0 / +1` encoding:

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
name = "andreadm/reddit-pulse-bert"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()
encoding = {"down": -1, "neutral": 0, "up": 1}
texts = [
    "Inflation expectations drop to lowest level since 2021, NY Fed survey shows",
    "What is the difference between CPI and PCE?",
]
with torch.no_grad():
    batch = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    ids = model(**batch).logits.argmax(dim=-1).tolist()
print([encoding[model.config.id2label[i]] for i in ids])   # [-1, 0]
```

Tested with `transformers` 5.16 and PyTorch 2.13; the checkpoint uses the
standard `safetensors` + fast-tokenizer layout and loads on any recent
4.x release as well. Inference is cheap: the full paper corpus
(≈ 243 k texts) labels in a few minutes on one A100 at batch size 64.

## Intended use and limitations

**Intended use.** Labelling large volumes of short, informal, English,
economy-related texts (Reddit titles and comments, headlines, social media
posts) with a directional inflation signal that is then *aggregated* over
time — the paper's use case. The model is a building block for a
high-frequency indicator, not a stand-alone oracle.

**Limitations.**

- **Single predictions are noisy.** Held-out accuracy is ≈ 74 %; the value
  of the model comes from averaging thousands of predictions per period,
  where idiosyncratic errors wash out. Do not rely on any single label.
- **Domain and register.** Trained on r/economy, r/Economics and
  r/wallstreetbets titles about *US* inflation from 2008 to 2022. Other
  countries, other registers (formal reports, central-bank prose) and
  post-2022 vocabulary are out of distribution.
- **Class imbalance.** `down` is the minority class (16 % of the gold set)
  and the hardest one (test F1 0.67 vs 0.79 for `neutral`). The loss was
  class-balanced during training, but `down` recall remains lower.
- **Short texts.** Fine-tuned on titles (median 11 words, max 52). Long
  comments are truncated at 512 tokens and were not seen during training.
- **Direction, not stance or sentiment.** The model does not say whether
  the author *wants* inflation to move, nor whether the news is good or
  bad; only which way prices are said to be going.
- **No factuality.** A confidently worded false claim is labelled by its
  direction, not its truth.

## Training data

The gold set is a hand-labelled sample of **1 383 Reddit submission titles**
from r/economy, r/Economics and r/wallstreetbets, dated February 2008 to
December 2022. Labels were produced by a human-in-the-loop protocol
(manual annotation assisted by zero-shot LLaMA-70B, a fine-tuned LLaMA-8B
classifier and ChatGPT-assisted adjudication of disagreements) described in
the paper. The gold set itself is not redistributed here.

| label | titles | share |
|---|---|---|
| neutral | 623 | 45.0 % |
| up | 537 | 38.8 % |
| down | 223 | 16.1 % |

Titles are lower-cased as found on Reddit; median length 11 words
(IQR 8–16, max 52).

## Training procedure

### Seed protocol and model selection

The paper's protocol asks how sensitive the classifier is to *which* titles
it is trained on, so it separates two sources of randomness:

- **19 split seeds** (`training.seeds`) each draw a different stratified
  **71 / 19 / 10 %** train / validation / test partition
  (981 / 263 / 139 titles) of the same gold set.
- **One fixed seed (42)** governs everything else: classification-head
  initialisation, batch shuffling and dropout. Every split therefore trains
  *the same model the same way* on different data.

Each split is fine-tuned independently; the checkpoint with the **median**
test weighted-F1 across the 19 runs (the upper median, rank 10 of 19) is
kept and the others discarded. The result is a *typical* run, not the best
one — the reported metrics are an honest estimate of what a rerun yields,
not a lucky draw. This checkpoint is split seed **2786505123**.

### Hyperparameters

| | |
|---|---|
| Objective | Cross-entropy with `balanced` class weights (`sklearn.utils.class_weight`) |
| Optimizer | AdamW (fused), β = (0.9, 0.999), ε = 1e-8, weight decay 0.01 |
| Learning rate | 5e-5, linear decay, no warm-up |
| Batch size | 64 (train and eval), dynamic padding to multiples of 8 |
| Epochs | up to 15, early stopping with patience 5 on validation weighted-F1 |
| Checkpoint | best epoch by validation weighted-F1 (`load_best_model_at_end`) |
| Precision | fp16 mixed precision, fp32 master weights |
| Max sequence length | 512 tokens (truncation only; titles are far shorter) |
| Gradient clipping | 1.0 |
| Hardware | 1 × NVIDIA A100 80 GB; ≈ 2 min per seed, 38 min for all 19 |

The exact `TrainingArguments` are in [`training_args.json`](training_args.json)
and the governing configuration extract in
[`training_config.yml`](training_config.yml).

## Evaluation

### Selected checkpoint (split seed 2786505123)

| split | n | accuracy | F1 weighted | F1 macro | precision macro | recall macro | ROC-AUC |
|---|---|---|---|---|---|---|---|
| validation | 263 | 0.757 | 0.756 | 0.739 | 0.742 | 0.738 | 0.864 |
| **test** | **139** | **0.741** | **0.739** | **0.722** | **0.725** | **0.724** | **0.872** |

Per class, on the test split:

| label | precision | recall | F1 | support |
|---|---|---|---|---|
| down | 0.652 | 0.682 | 0.667 | 22 |
| neutral | 0.746 | 0.841 | 0.791 | 63 |
| up | 0.778 | 0.648 | 0.707 | 54 |

Confusion matrix (rows = true, columns = predicted):

| | down | neutral | up |
|---|---|---|---|
| **down** | 15 | 3 | 4 |
| **neutral** | 4 | 53 | 6 |
| **up** | 4 | 15 | 35 |

Most errors are `up` titles absorbed by `neutral` (15 of 54): hedged or
question-shaped titles whose direction a human infers from context.
Confusions between `up` and `down` — the ones that would bias an aggregate
signal — are rare (8 of 139).

### Split sensitivity across the 19 seeds

Test metrics of every run, sorted by weighted F1. The selected checkpoint
is the median.

| split seed | val F1w | test acc | test F1w | test F1m | test AUC | |
|---|---|---|---|---|---|---|
| 709964709 | 0.792 | 0.676 | 0.676 | 0.640 | 0.813 |
| 477284336 | 0.726 | 0.691 | 0.689 | 0.688 | 0.907 |
| 107935903 | 0.793 | 0.698 | 0.702 | 0.680 | 0.878 |
| 1245093080 | 0.740 | 0.705 | 0.707 | 0.693 | 0.866 |
| 4280088979 | 0.803 | 0.712 | 0.714 | 0.704 | 0.891 |
| 2928142788 | 0.744 | 0.712 | 0.716 | 0.694 | 0.863 |
| 3144693271 | 0.810 | 0.719 | 0.719 | 0.700 | 0.862 |
| 239080115 | 0.759 | 0.734 | 0.732 | 0.716 | 0.876 |
| 1361883482 | 0.786 | 0.734 | 0.733 | 0.709 | 0.901 |
| 2786505123 | 0.756 | 0.741 | 0.739 | 0.722 | 0.872 | **selected**
| 228277762 | 0.707 | 0.748 | 0.749 | 0.714 | 0.895 |
| 3266123502 | 0.794 | 0.748 | 0.752 | 0.731 | 0.916 |
| 1329496050 | 0.792 | 0.748 | 0.752 | 0.740 | 0.923 |
| 4054871397 | 0.768 | 0.755 | 0.755 | 0.736 | 0.874 |
| 1389303030 | 0.750 | 0.755 | 0.757 | 0.745 | 0.898 |
| 3154447144 | 0.783 | 0.784 | 0.785 | 0.767 | 0.886 |
| 2565555162 | 0.775 | 0.791 | 0.794 | 0.766 | 0.921 |
| 2078237541 | 0.783 | 0.799 | 0.797 | 0.779 | 0.911 |
| 4203596092 | 0.765 | 0.813 | 0.813 | 0.804 | 0.923 |

| test metric | mean | std | min | median | max |
|---|---|---|---|---|---|
| accuracy | 0.740 | 0.038 | 0.676 | 0.741 | 0.813 |
| F1 weighted | 0.741 | 0.038 | 0.676 | 0.739 | 0.813 |
| F1 macro | 0.722 | 0.039 | 0.640 | 0.716 | 0.804 |
| ROC-AUC | 0.888 | 0.028 | 0.813 | 0.891 | 0.923 |

The spread is what a 139-title test set implies (one title ≈ 0.7 accuracy
points) and is the reason the paper reports the median run rather than a
single split. Full per-seed numbers: [`evaluation/`](evaluation/).

## Corpus labelling in the paper

Applied to the paper's filtered corpus, this checkpoint labels **33 460
submissions** and **209 995 comments** (r/economy, r/Economics,
r/wallstreetbets, 2008–2022):

| | up | neutral | down |
|---|---|---|---|
| submissions | 43.6 % | 42.0 % | 14.4 % |
| comments | 40.3 % | 51.8 % | 8.0 % |

## Files

| file | content |
|---|---|
| `model.safetensors`, `config.json` | weights (fp32, 328 MB) and architecture / label map |
| `tokenizer.json`, `tokenizer_config.json` | byte-level BPE tokenizer inherited from RoBERTa |
| `training_args.json` | the `transformers.TrainingArguments` of the selected run |
| `training_config.yml` | dataset split, hyperparameters, seed list and label map from the project config |
| `evaluation/seeds_test_metrics.csv` | test metrics of all 19 split seeds |
| `evaluation/seeds_validation_metrics.csv` | validation metrics of all 19 split seeds |
| `evaluation/selected_seed_report.json` | per-class report and confusion matrices of this checkpoint |
| `LICENSE.md` | MIT license, scope statement and upstream Apache-2.0 / MIT notices |

## Reproducing

```bash
git clone https://github.com/andrea-dm/reddit-pulse && cd reddit-pulse
uv venv && uv pip install -e .
reddit run --model inflabert --gpu 0     # 19 seeds, median selection, corpus labelling
```

The gold set (`data/labelled.xlsx`) and the subreddit corpus are not part of
the repository; see the paper for the data-construction stages.

## Citation

```bibtex
@article{delmonaco2026reddit,
  title   = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author  = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  journal = {Journal of Applied Econometrics},
  year    = {2026},
  note    = {forthcoming},
}
@techreport{delmonaco2026reddit_qef,
  title       = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author      = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  institution = {Banca d'Italia},
  series      = {Questioni di Economia e Finanza (Occasional Papers)},
  number      = {1028},
  year        = {2026},
  month       = jun,
  doi         = {10.32057/0.QEF.2026.1028},
}
```

## License

Apache License, Version 2.0 (see [`LICENSE.md`](LICENSE.md)). The direct
parent `MAPAi/InflaBERT` is MIT-licensed; the deeper lineage
(`distilroberta-base`, `mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis`)
is Apache-2.0, which this checkpoint now matches. The attribution notices
these upstream licenses require are included in `LICENSE.md`.

The views expressed in the paper are those of the authors and do not
necessarily reflect those of the Bank of Italy, the Eurosystem, or the
European Commission.

	---
	language:
	- en
	license: apache-2.0
	library_name: transformers
	pipeline_tag: text-classification
	base_model: MAPAi/InflaBERT
	base_model_relation: finetune
	tags:
	- economics
	- finance
	- inflation
	- inflation-expectations
	- reddit
	- roberta
	- distilroberta
	metrics:
	- accuracy
	- f1
	- roc_auc
	widget:
	- text: Fed officials warn prices will keep climbing as CPI hits 40-year high
	example_title: Up
	- text: Inflation expectations drop to lowest level since 2021, NY Fed survey shows
	example_title: Down
	- text: What is the difference between CPI and PCE?
	example_title: Neutral
	- text: Rent and grocery bills are getting out of hand this year
	example_title: Up (colloquial)
	- text: Gas prices fall for the sixth straight week as demand cools
	example_title: Down (prices)
	model-index:
	- name: Reddit-pulse InflaBERT
	results:
	- task:
	type: text-classification
	name: Directional inflation-expectation classification (up / neutral / down)
	dataset:
	name: Reddit inflation gold set (Del Monaco, Longo, Marcucci & Tafani, 2026), held-out test split of seed 2786505123
	type: reddit-inflation-gold-set
	split: test
	metrics:
	- type: accuracy
	value: 0.7410
	name: Accuracy
	- type: f1
	value: 0.7387
	name: F1 (weighted)
	- type: f1
	value: 0.7216
	name: F1 (macro)
	- type: roc_auc
	value: 0.8716
	name: ROC-AUC (macro, one-vs-rest)
	---

	# Reddit-pulse InflaBERT

	> Retrained checkpoint. This model was trained again after the paper, on the
	> same gold dataset and with the same seed protocol. The metrics on this card come
	> from that retraining and may differ from the ones reported in the paper.

	Reddit-pulse InflaBERT is a three-way directional inflation-expectation
	classifier for short English texts about the economy, fine-tuned on Reddit
	submission titles. Given a title (or any sentence-length text), it predicts
	whether the text conveys that inflation / prices are going up, going
	down, or carries no directional signal (neutral).

	It is not a sentiment model: "inflation falls sharply" is good news but
	labelled down; "rents are out of control" is bad news but labelled
	up. The direction is about the price level, not the mood.

	The checkpoint is the model behind the Reddit inflation signal in

	> Del Monaco, A., Longo, L., Marcucci, J. & Tafani, I. (2026).
	> Reddit's 'pulse' on US inflation: forecasting with large language models.
	> Journal of Applied Econometrics, forthcoming.
	> Working-paper version: Banca d'Italia, Questioni di Economia e Finanza
	> (Occasional Papers) No. 1028, June 2026, doi:10.32057/0.QEF.2026.1028.

	<!-- TODO: add the JAE DOI / article URL once assigned. -->

	The fine-tuning and full-corpus inference code lives at
	[andrea-dm/reddit-pulse](https://github.com/andrea-dm/reddit-pulse).

	## Model lineage

	\| Stage \| Model \| Notes \|
	\|---\|---\|---\|
	\| Pre-training \| [`distilroberta-base`](https://huggingface.co/distilroberta-base) \| 6-layer distilled RoBERTa, 82.1 M parameters \|
	\| Domain adaptation \| [`mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis`](https://huggingface.co/mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis) \| Financial-news sentiment \|
	\| Task adaptation \| [`MAPAi/InflaBERT`](https://huggingface.co/MAPAi/InflaBERT) \| Inflation-news sentiment (negative / neutral / positive) \|
	\| This checkpoint \| `andreadm/reddit-pulse-bert` \| Full fine-tuning on Reddit titles, labels re-mapped to down / neutral / up \|

	Architecture: `RobertaForSequenceClassification`, 6 hidden layers, hidden
	size 768, 12 attention heads, 50 265-token byte-level BPE vocabulary, 512-token
	context. Every parameter was updated (no adapters, no quantization); the
	classification head was re-initialised for the three directional classes.

	## Labels

	\| id \| label \| meaning \| encoding used in the paper's corpus files \|
	\|---\|---\|---\|---\|
	\| 0 \| `down` \| inflation / prices heading lower, disinflation, falling expectations \| `-1` \|
	\| 1 \| `neutral` \| no directional signal (questions, definitions, unrelated, mixed) \| `0` \|
	\| 2 \| `up` \| inflation / prices heading higher, rising expectations \| `+1` \|

	## How to use

	```python
	from transformers import pipeline

	clf = pipeline("text-classification", model="andreadm/reddit-pulse-bert")
	clf("Fed officials warn prices will keep climbing as CPI hits 40-year high")
	# [{'label': 'up', 'score': 0.999}]
	```

	Batched, with the paper's `-1 / 0 / +1` encoding:

	```python
	import torch
	from transformers import AutoModelForSequenceClassification, AutoTokenizer

	name = "andreadm/reddit-pulse-bert"
	tokenizer = AutoTokenizer.from_pretrained(name)
	model = AutoModelForSequenceClassification.from_pretrained(name).eval()
	encoding = {"down": -1, "neutral": 0, "up": 1}

	texts = [
	"Inflation expectations drop to lowest level since 2021, NY Fed survey shows",
	"What is the difference between CPI and PCE?",
	]
	with torch.no_grad():
	batch = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
	ids = model(**batch).logits.argmax(dim=-1).tolist()

	print([encoding[model.config.id2label[i]] for i in ids]) # [-1, 0]
	```

	Tested with `transformers` 5.16 and PyTorch 2.13; the checkpoint uses the
	standard `safetensors` + fast-tokenizer layout and loads on any recent
	4.x release as well. Inference is cheap: the full paper corpus
	(≈ 243 k texts) labels in a few minutes on one A100 at batch size 64.

	## Intended use and limitations

	Intended use. Labelling large volumes of short, informal, English,
	economy-related texts (Reddit titles and comments, headlines, social media
	posts) with a directional inflation signal that is then aggregated over
	time — the paper's use case. The model is a building block for a
	high-frequency indicator, not a stand-alone oracle.

	Limitations.

	- Single predictions are noisy. Held-out accuracy is ≈ 74 %; the value
	of the model comes from averaging thousands of predictions per period,
	where idiosyncratic errors wash out. Do not rely on any single label.
	- Domain and register. Trained on r/economy, r/Economics and
	r/wallstreetbets titles about US inflation from 2008 to 2022. Other
	countries, other registers (formal reports, central-bank prose) and
	post-2022 vocabulary are out of distribution.
	- Class imbalance. `down` is the minority class (16 % of the gold set)
	and the hardest one (test F1 0.67 vs 0.79 for `neutral`). The loss was
	class-balanced during training, but `down` recall remains lower.
	- Short texts. Fine-tuned on titles (median 11 words, max 52). Long
	comments are truncated at 512 tokens and were not seen during training.
	- Direction, not stance or sentiment. The model does not say whether
	the author wants inflation to move, nor whether the news is good or
	bad; only which way prices are said to be going.
	- No factuality. A confidently worded false claim is labelled by its
	direction, not its truth.

	## Training data

	The gold set is a hand-labelled sample of 1 383 Reddit submission titles
	from r/economy, r/Economics and r/wallstreetbets, dated February 2008 to
	December 2022. Labels were produced by a human-in-the-loop protocol
	(manual annotation assisted by zero-shot LLaMA-70B, a fine-tuned LLaMA-8B
	classifier and ChatGPT-assisted adjudication of disagreements) described in
	the paper. The gold set itself is not redistributed here.

	\| label \| titles \| share \|
	\|---\|---\|---\|
	\| neutral \| 623 \| 45.0 % \|
	\| up \| 537 \| 38.8 % \|
	\| down \| 223 \| 16.1 % \|

	Titles are lower-cased as found on Reddit; median length 11 words
	(IQR 8–16, max 52).

	## Training procedure

	### Seed protocol and model selection

	The paper's protocol asks how sensitive the classifier is to which titles
	it is trained on, so it separates two sources of randomness:

	- 19 split seeds (`training.seeds`) each draw a different stratified
	71 / 19 / 10 % train / validation / test partition
	(981 / 263 / 139 titles) of the same gold set.
	- One fixed seed (42) governs everything else: classification-head
	initialisation, batch shuffling and dropout. Every split therefore trains
	the same model the same way on different data.

	Each split is fine-tuned independently; the checkpoint with the median
	test weighted-F1 across the 19 runs (the upper median, rank 10 of 19) is
	kept and the others discarded. The result is a typical run, not the best
	one — the reported metrics are an honest estimate of what a rerun yields,
	not a lucky draw. This checkpoint is split seed 2786505123.

	### Hyperparameters

	\| \| \|
	\|---\|---\|
	\| Objective \| Cross-entropy with `balanced` class weights (`sklearn.utils.class_weight`) \|
	\| Optimizer \| AdamW (fused), β = (0.9, 0.999), ε = 1e-8, weight decay 0.01 \|
	\| Learning rate \| 5e-5, linear decay, no warm-up \|
	\| Batch size \| 64 (train and eval), dynamic padding to multiples of 8 \|
	\| Epochs \| up to 15, early stopping with patience 5 on validation weighted-F1 \|
	\| Checkpoint \| best epoch by validation weighted-F1 (`load_best_model_at_end`) \|
	\| Precision \| fp16 mixed precision, fp32 master weights \|
	\| Max sequence length \| 512 tokens (truncation only; titles are far shorter) \|
	\| Gradient clipping \| 1.0 \|
	\| Hardware \| 1 × NVIDIA A100 80 GB; ≈ 2 min per seed, 38 min for all 19 \|

	The exact `TrainingArguments` are in [`training_args.json`](training_args.json)
	and the governing configuration extract in
	[`training_config.yml`](training_config.yml).

	## Evaluation

	### Selected checkpoint (split seed 2786505123)

	\| split \| n \| accuracy \| F1 weighted \| F1 macro \| precision macro \| recall macro \| ROC-AUC \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| validation \| 263 \| 0.757 \| 0.756 \| 0.739 \| 0.742 \| 0.738 \| 0.864 \|
	\| test \| 139 \| 0.741 \| 0.739 \| 0.722 \| 0.725 \| 0.724 \| 0.872 \|

	Per class, on the test split:

	\| label \| precision \| recall \| F1 \| support \|
	\|---\|---\|---\|---\|---\|
	\| down \| 0.652 \| 0.682 \| 0.667 \| 22 \|
	\| neutral \| 0.746 \| 0.841 \| 0.791 \| 63 \|
	\| up \| 0.778 \| 0.648 \| 0.707 \| 54 \|

	Confusion matrix (rows = true, columns = predicted):

	\| \| down \| neutral \| up \|
	\|---\|---\|---\|---\|
	\| down \| 15 \| 3 \| 4 \|
	\| neutral \| 4 \| 53 \| 6 \|
	\| up \| 4 \| 15 \| 35 \|

	Most errors are `up` titles absorbed by `neutral` (15 of 54): hedged or
	question-shaped titles whose direction a human infers from context.
	Confusions between `up` and `down` — the ones that would bias an aggregate
	signal — are rare (8 of 139).

	### Split sensitivity across the 19 seeds

	Test metrics of every run, sorted by weighted F1. The selected checkpoint
	is the median.

	\| split seed \| val F1w \| test acc \| test F1w \| test F1m \| test AUC \| \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| 709964709 \| 0.792 \| 0.676 \| 0.676 \| 0.640 \| 0.813 \|
	\| 477284336 \| 0.726 \| 0.691 \| 0.689 \| 0.688 \| 0.907 \|
	\| 107935903 \| 0.793 \| 0.698 \| 0.702 \| 0.680 \| 0.878 \|
	\| 1245093080 \| 0.740 \| 0.705 \| 0.707 \| 0.693 \| 0.866 \|
	\| 4280088979 \| 0.803 \| 0.712 \| 0.714 \| 0.704 \| 0.891 \|
	\| 2928142788 \| 0.744 \| 0.712 \| 0.716 \| 0.694 \| 0.863 \|
	\| 3144693271 \| 0.810 \| 0.719 \| 0.719 \| 0.700 \| 0.862 \|
	\| 239080115 \| 0.759 \| 0.734 \| 0.732 \| 0.716 \| 0.876 \|
	\| 1361883482 \| 0.786 \| 0.734 \| 0.733 \| 0.709 \| 0.901 \|
	\| 2786505123 \| 0.756 \| 0.741 \| 0.739 \| 0.722 \| 0.872 \| selected
	\| 228277762 \| 0.707 \| 0.748 \| 0.749 \| 0.714 \| 0.895 \|
	\| 3266123502 \| 0.794 \| 0.748 \| 0.752 \| 0.731 \| 0.916 \|
	\| 1329496050 \| 0.792 \| 0.748 \| 0.752 \| 0.740 \| 0.923 \|
	\| 4054871397 \| 0.768 \| 0.755 \| 0.755 \| 0.736 \| 0.874 \|
	\| 1389303030 \| 0.750 \| 0.755 \| 0.757 \| 0.745 \| 0.898 \|
	\| 3154447144 \| 0.783 \| 0.784 \| 0.785 \| 0.767 \| 0.886 \|
	\| 2565555162 \| 0.775 \| 0.791 \| 0.794 \| 0.766 \| 0.921 \|
	\| 2078237541 \| 0.783 \| 0.799 \| 0.797 \| 0.779 \| 0.911 \|
	\| 4203596092 \| 0.765 \| 0.813 \| 0.813 \| 0.804 \| 0.923 \|

	\| test metric \| mean \| std \| min \| median \| max \|
	\|---\|---\|---\|---\|---\|---\|
	\| accuracy \| 0.740 \| 0.038 \| 0.676 \| 0.741 \| 0.813 \|
	\| F1 weighted \| 0.741 \| 0.038 \| 0.676 \| 0.739 \| 0.813 \|
	\| F1 macro \| 0.722 \| 0.039 \| 0.640 \| 0.716 \| 0.804 \|
	\| ROC-AUC \| 0.888 \| 0.028 \| 0.813 \| 0.891 \| 0.923 \|

	The spread is what a 139-title test set implies (one title ≈ 0.7 accuracy
	points) and is the reason the paper reports the median run rather than a
	single split. Full per-seed numbers: [`evaluation/`](evaluation/).

	## Corpus labelling in the paper

	Applied to the paper's filtered corpus, this checkpoint labels **33 460
	submissions and 209 995 comments** (r/economy, r/Economics,
	r/wallstreetbets, 2008–2022):

	\| \| up \| neutral \| down \|
	\|---\|---\|---\|---\|
	\| submissions \| 43.6 % \| 42.0 % \| 14.4 % \|
	\| comments \| 40.3 % \| 51.8 % \| 8.0 % \|

	## Files

	\| file \| content \|
	\|---\|---\|
	\| `model.safetensors`, `config.json` \| weights (fp32, 328 MB) and architecture / label map \|
	\| `tokenizer.json`, `tokenizer_config.json` \| byte-level BPE tokenizer inherited from RoBERTa \|
	\| `training_args.json` \| the `transformers.TrainingArguments` of the selected run \|
	\| `training_config.yml` \| dataset split, hyperparameters, seed list and label map from the project config \|
	\| `evaluation/seeds_test_metrics.csv` \| test metrics of all 19 split seeds \|
	\| `evaluation/seeds_validation_metrics.csv` \| validation metrics of all 19 split seeds \|
	\| `evaluation/selected_seed_report.json` \| per-class report and confusion matrices of this checkpoint \|
	\| `LICENSE.md` \| MIT license, scope statement and upstream Apache-2.0 / MIT notices \|

	## Reproducing

	```bash
	git clone https://github.com/andrea-dm/reddit-pulse && cd reddit-pulse
	uv venv && uv pip install -e .
	reddit run --model inflabert --gpu 0 # 19 seeds, median selection, corpus labelling
	```

	The gold set (`data/labelled.xlsx`) and the subreddit corpus are not part of
	the repository; see the paper for the data-construction stages.

	## Citation

	```bibtex
	@article{delmonaco2026reddit,
	title = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
	author = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
	journal = {Journal of Applied Econometrics},
	year = {2026},
	note = {forthcoming},
	}

	@techreport{delmonaco2026reddit_qef,
	title = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
	author = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
	institution = {Banca d'Italia},
	series = {Questioni di Economia e Finanza (Occasional Papers)},
	number = {1028},
	year = {2026},
	month = jun,
	doi = {10.32057/0.QEF.2026.1028},
	}
	```

	## License

	Apache License, Version 2.0 (see [`LICENSE.md`](LICENSE.md)). The direct
	parent `MAPAi/InflaBERT` is MIT-licensed; the deeper lineage
	(`distilroberta-base`, `mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis`)
	is Apache-2.0, which this checkpoint now matches. The attribution notices
	these upstream licenses require are included in `LICENSE.md`.

	The views expressed in the paper are those of the authors and do not
	necessarily reflect those of the Bank of Italy, the Eurosystem, or the
	European Commission.