---
language:
- en
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
base_model: MAPAi/InflaBERT
base_model_relation: finetune
tags:
- economics
- finance
- inflation
- inflation-expectations
- reddit
- roberta
- distilroberta
metrics:
- accuracy
- f1
- roc_auc
widget:
- text: Fed officials warn prices will keep climbing as CPI hits 40-year high
  example_title: Up
- text: Inflation expectations drop to lowest level since 2021, NY Fed survey shows
  example_title: Down
- text: What is the difference between CPI and PCE?
  example_title: Neutral
- text: Rent and grocery bills are getting out of hand this year
  example_title: Up (colloquial)
- text: Gas prices fall for the sixth straight week as demand cools
  example_title: Down (prices)
model-index:
- name: Reddit-pulse InflaBERT
  results:
  - task:
      type: text-classification
      name: Directional inflation-expectation classification (up / neutral / down)
    dataset:
      name: Reddit inflation gold set (Del Monaco, Longo, Marcucci & Tafani, 2026), held-out test split of seed 2786505123
      type: reddit-inflation-gold-set
      split: test
    metrics:
    - type: accuracy
      value: 0.7410
      name: Accuracy
    - type: f1
      value: 0.7387
      name: F1 (weighted)
    - type: f1
      value: 0.7216
      name: F1 (macro)
    - type: roc_auc
      value: 0.8716
      name: ROC-AUC (macro, one-vs-rest)
---

# Reddit-pulse InflaBERT

> **Retrained checkpoint.** This model was trained again after the paper, on the
> same gold dataset and with the same seed protocol. The metrics on this card come
> from that retraining and may differ from the ones reported in the paper.

**Reddit-pulse InflaBERT** is a three-way *directional* inflation-expectation
classifier for short English texts about the economy, fine-tuned on Reddit
submission titles. Given a title (or any sentence-length text), it predicts
whether the text conveys that inflation / prices are going **up**, going
**down**, or carries **no directional signal** (**neutral**).

It is *not* a sentiment model: "inflation falls sharply" is *good* news but
labelled **down**; "rents are out of control" is *bad* news but labelled
**up**. The direction is about the price level, not the mood.

The checkpoint is the model behind the Reddit inflation signal in

> Del Monaco, A., Longo, L., Marcucci, J. & Tafani, I. (2026).
> *Reddit's 'pulse' on US inflation: forecasting with large language models.*
> Journal of Applied Econometrics, forthcoming.
> Working-paper version: Banca d'Italia, Questioni di Economia e Finanza
> (Occasional Papers) No. 1028, June 2026, doi:10.32057/0.QEF.2026.1028.

<!-- TODO: add the JAE DOI / article URL once assigned. -->

The fine-tuning and full-corpus inference code lives at
[andrea-dm/reddit-pulse](https://github.com/andrea-dm/reddit-pulse).

## Model lineage

| Stage | Model | Notes |
|---|---|---|
| Pre-training | [`distilroberta-base`](https://huggingface.co/distilroberta-base) | 6-layer distilled RoBERTa, 82.1 M parameters |
| Domain adaptation | [`mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis`](https://huggingface.co/mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis) | Financial-news sentiment |
| Task adaptation | [`MAPAi/InflaBERT`](https://huggingface.co/MAPAi/InflaBERT) | Inflation-news sentiment (negative / neutral / positive) |
| **This checkpoint** | **`andreadm/reddit-pulse-bert`** | Full fine-tuning on Reddit titles, labels re-mapped to **down / neutral / up** |

Architecture: `RobertaForSequenceClassification`, 6 hidden layers, hidden
size 768, 12 attention heads, 50 265-token byte-level BPE vocabulary, 512-token
context. Every parameter was updated (no adapters, no quantization); the
classification head was re-initialised for the three directional classes.

## Labels

| id | label | meaning | encoding used in the paper's corpus files |
|---|---|---|---|
| 0 | `down` | inflation / prices heading lower, disinflation, falling expectations | `-1` |
| 1 | `neutral` | no directional signal (questions, definitions, unrelated, mixed) | `0` |
| 2 | `up` | inflation / prices heading higher, rising expectations | `+1` |

## How to use

```python
from transformers import pipeline

clf = pipeline("text-classification", model="andreadm/reddit-pulse-bert")
clf("Fed officials warn prices will keep climbing as CPI hits 40-year high")
# [{'label': 'up', 'score': 0.999}]
```

Batched, with the paper's `-1 / 0 / +1` encoding:

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

name = "andreadm/reddit-pulse-bert"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()
encoding = {"down": -1, "neutral": 0, "up": 1}

texts = [
    "Inflation expectations drop to lowest level since 2021, NY Fed survey shows",
    "What is the difference between CPI and PCE?",
]
with torch.no_grad():
    batch = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    ids = model(**batch).logits.argmax(dim=-1).tolist()

print([encoding[model.config.id2label[i]] for i in ids])   # [-1, 0]
```

Tested with `transformers` 5.16 and PyTorch 2.13; the checkpoint uses the
standard `safetensors` + fast-tokenizer layout and loads on any recent
4.x release as well. Inference is cheap: the full paper corpus
(≈ 243 k texts) labels in a few minutes on one A100 at batch size 64.

## Intended use and limitations

**Intended use.** Labelling large volumes of short, informal, English,
economy-related texts (Reddit titles and comments, headlines, social media
posts) with a directional inflation signal that is then *aggregated* over
time — the paper's use case. The model is a building block for a
high-frequency indicator, not a stand-alone oracle.

**Limitations.**

- **Single predictions are noisy.** Held-out accuracy is ≈ 74 %; the value
  of the model comes from averaging thousands of predictions per period,
  where idiosyncratic errors wash out. Do not rely on any single label.
- **Domain and register.** Trained on r/economy, r/Economics and
  r/wallstreetbets titles about *US* inflation from 2008 to 2022. Other
  countries, other registers (formal reports, central-bank prose) and
  post-2022 vocabulary are out of distribution.
- **Class imbalance.** `down` is the minority class (16 % of the gold set)
  and the hardest one (test F1 0.67 vs 0.79 for `neutral`). The loss was
  class-balanced during training, but `down` recall remains lower.
- **Short texts.** Fine-tuned on titles (median 11 words, max 52). Long
  comments are truncated at 512 tokens and were not seen during training.
- **Direction, not stance or sentiment.** The model does not say whether
  the author *wants* inflation to move, nor whether the news is good or
  bad; only which way prices are said to be going.
- **No factuality.** A confidently worded false claim is labelled by its
  direction, not its truth.

## Training data

The gold set is a hand-labelled sample of **1 383 Reddit submission titles**
from r/economy, r/Economics and r/wallstreetbets, dated February 2008 to
December 2022. Labels were produced by a human-in-the-loop protocol
(manual annotation assisted by zero-shot LLaMA-70B, a fine-tuned LLaMA-8B
classifier and ChatGPT-assisted adjudication of disagreements) described in
the paper. The gold set itself is not redistributed here.

| label | titles | share |
|---|---|---|
| neutral | 623 | 45.0 % |
| up | 537 | 38.8 % |
| down | 223 | 16.1 % |

Titles are lower-cased as found on Reddit; median length 11 words
(IQR 8–16, max 52).

## Training procedure

### Seed protocol and model selection

The paper's protocol asks how sensitive the classifier is to *which* titles
it is trained on, so it separates two sources of randomness:

- **19 split seeds** (`training.seeds`) each draw a different stratified
  **71 / 19 / 10 %** train / validation / test partition
  (981 / 263 / 139 titles) of the same gold set.
- **One fixed seed (42)** governs everything else: classification-head
  initialisation, batch shuffling and dropout. Every split therefore trains
  *the same model the same way* on different data.

Each split is fine-tuned independently; the checkpoint with the **median**
test weighted-F1 across the 19 runs (the upper median, rank 10 of 19) is
kept and the others discarded. The result is a *typical* run, not the best
one — the reported metrics are an honest estimate of what a rerun yields,
not a lucky draw. This checkpoint is split seed **2786505123**.

### Hyperparameters

| | |
|---|---|
| Objective | Cross-entropy with `balanced` class weights (`sklearn.utils.class_weight`) |
| Optimizer | AdamW (fused), β = (0.9, 0.999), ε = 1e-8, weight decay 0.01 |
| Learning rate | 5e-5, linear decay, no warm-up |
| Batch size | 64 (train and eval), dynamic padding to multiples of 8 |
| Epochs | up to 15, early stopping with patience 5 on validation weighted-F1 |
| Checkpoint | best epoch by validation weighted-F1 (`load_best_model_at_end`) |
| Precision | fp16 mixed precision, fp32 master weights |
| Max sequence length | 512 tokens (truncation only; titles are far shorter) |
| Gradient clipping | 1.0 |
| Hardware | 1 × NVIDIA A100 80 GB; ≈ 2 min per seed, 38 min for all 19 |

The exact `TrainingArguments` are in [`training_args.json`](training_args.json)
and the governing configuration extract in
[`training_config.yml`](training_config.yml).

## Evaluation

### Selected checkpoint (split seed 2786505123)

| split | n | accuracy | F1 weighted | F1 macro | precision macro | recall macro | ROC-AUC |
|---|---|---|---|---|---|---|---|
| validation | 263 | 0.757 | 0.756 | 0.739 | 0.742 | 0.738 | 0.864 |
| **test** | **139** | **0.741** | **0.739** | **0.722** | **0.725** | **0.724** | **0.872** |

Per class, on the test split:

| label | precision | recall | F1 | support |
|---|---|---|---|---|
| down | 0.652 | 0.682 | 0.667 | 22 |
| neutral | 0.746 | 0.841 | 0.791 | 63 |
| up | 0.778 | 0.648 | 0.707 | 54 |

Confusion matrix (rows = true, columns = predicted):

| | down | neutral | up |
|---|---|---|---|
| **down** | 15 | 3 | 4 |
| **neutral** | 4 | 53 | 6 |
| **up** | 4 | 15 | 35 |

Most errors are `up` titles absorbed by `neutral` (15 of 54): hedged or
question-shaped titles whose direction a human infers from context.
Confusions between `up` and `down` — the ones that would bias an aggregate
signal — are rare (8 of 139).

### Split sensitivity across the 19 seeds

Test metrics of every run, sorted by weighted F1. The selected checkpoint
is the median.

| split seed | val F1w | test acc | test F1w | test F1m | test AUC | |
|---|---|---|---|---|---|---|
| 709964709 | 0.792 | 0.676 | 0.676 | 0.640 | 0.813 |
| 477284336 | 0.726 | 0.691 | 0.689 | 0.688 | 0.907 |
| 107935903 | 0.793 | 0.698 | 0.702 | 0.680 | 0.878 |
| 1245093080 | 0.740 | 0.705 | 0.707 | 0.693 | 0.866 |
| 4280088979 | 0.803 | 0.712 | 0.714 | 0.704 | 0.891 |
| 2928142788 | 0.744 | 0.712 | 0.716 | 0.694 | 0.863 |
| 3144693271 | 0.810 | 0.719 | 0.719 | 0.700 | 0.862 |
| 239080115 | 0.759 | 0.734 | 0.732 | 0.716 | 0.876 |
| 1361883482 | 0.786 | 0.734 | 0.733 | 0.709 | 0.901 |
| 2786505123 | 0.756 | 0.741 | 0.739 | 0.722 | 0.872 | **selected**
| 228277762 | 0.707 | 0.748 | 0.749 | 0.714 | 0.895 |
| 3266123502 | 0.794 | 0.748 | 0.752 | 0.731 | 0.916 |
| 1329496050 | 0.792 | 0.748 | 0.752 | 0.740 | 0.923 |
| 4054871397 | 0.768 | 0.755 | 0.755 | 0.736 | 0.874 |
| 1389303030 | 0.750 | 0.755 | 0.757 | 0.745 | 0.898 |
| 3154447144 | 0.783 | 0.784 | 0.785 | 0.767 | 0.886 |
| 2565555162 | 0.775 | 0.791 | 0.794 | 0.766 | 0.921 |
| 2078237541 | 0.783 | 0.799 | 0.797 | 0.779 | 0.911 |
| 4203596092 | 0.765 | 0.813 | 0.813 | 0.804 | 0.923 |

| test metric | mean | std | min | median | max |
|---|---|---|---|---|---|
| accuracy | 0.740 | 0.038 | 0.676 | 0.741 | 0.813 |
| F1 weighted | 0.741 | 0.038 | 0.676 | 0.739 | 0.813 |
| F1 macro | 0.722 | 0.039 | 0.640 | 0.716 | 0.804 |
| ROC-AUC | 0.888 | 0.028 | 0.813 | 0.891 | 0.923 |

The spread is what a 139-title test set implies (one title ≈ 0.7 accuracy
points) and is the reason the paper reports the median run rather than a
single split. Full per-seed numbers: [`evaluation/`](evaluation/).

## Corpus labelling in the paper

Applied to the paper's filtered corpus, this checkpoint labels **33 460
submissions** and **209 995 comments** (r/economy, r/Economics,
r/wallstreetbets, 2008–2022):

| | up | neutral | down |
|---|---|---|---|
| submissions | 43.6 % | 42.0 % | 14.4 % |
| comments | 40.3 % | 51.8 % | 8.0 % |

## Files

| file | content |
|---|---|
| `model.safetensors`, `config.json` | weights (fp32, 328 MB) and architecture / label map |
| `tokenizer.json`, `tokenizer_config.json` | byte-level BPE tokenizer inherited from RoBERTa |
| `training_args.json` | the `transformers.TrainingArguments` of the selected run |
| `training_config.yml` | dataset split, hyperparameters, seed list and label map from the project config |
| `evaluation/seeds_test_metrics.csv` | test metrics of all 19 split seeds |
| `evaluation/seeds_validation_metrics.csv` | validation metrics of all 19 split seeds |
| `evaluation/selected_seed_report.json` | per-class report and confusion matrices of this checkpoint |
| `LICENSE.md` | MIT license, scope statement and upstream Apache-2.0 / MIT notices |

## Reproducing

```bash
git clone https://github.com/andrea-dm/reddit-pulse && cd reddit-pulse
uv venv && uv pip install -e .
reddit run --model inflabert --gpu 0     # 19 seeds, median selection, corpus labelling
```

The gold set (`data/labelled.xlsx`) and the subreddit corpus are not part of
the repository; see the paper for the data-construction stages.

## Citation

```bibtex
@article{delmonaco2026reddit,
  title   = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author  = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  journal = {Journal of Applied Econometrics},
  year    = {2026},
  note    = {forthcoming},
}

@techreport{delmonaco2026reddit_qef,
  title       = {Reddit's `pulse' on {US} inflation: forecasting with large language models},
  author      = {Del Monaco, Andrea and Longo, Luigi and Marcucci, Juri and Tafani, Irene},
  institution = {Banca d'Italia},
  series      = {Questioni di Economia e Finanza (Occasional Papers)},
  number      = {1028},
  year        = {2026},
  month       = jun,
  doi         = {10.32057/0.QEF.2026.1028},
}
```

## License

Apache License, Version 2.0 (see [`LICENSE.md`](LICENSE.md)). The direct
parent `MAPAi/InflaBERT` is MIT-licensed; the deeper lineage
(`distilroberta-base`, `mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis`)
is Apache-2.0, which this checkpoint now matches. The attribution notices
these upstream licenses require are included in `LICENSE.md`.

The views expressed in the paper are those of the authors and do not
necessarily reflect those of the Bank of Italy, the Eurosystem, or the
European Commission.