policy-vs-other-v2

A fine-tuned version of xlm-roberta-base that classifies whether a social media post by a political party is about a policy issue or not.

It was developed for PartySOME, a comparative dataset of European parties' social media activity, as the first step of a two-step issue classification. Many party posts have no policy content (campaign events, rallies, greetings, personal content), but issue classifiers such as the Comparative Agendas Project (CAP) models assign a topic to every text. Filtering out non-policy posts first, then classifying the remaining ones with a CAP model, reduces this noise.

Labels

  • policy issue: the post addresses a policy issue
  • other: it does not

Training data

5,895 manually annotated posts from parties' Facebook, Instagram and Twitter accounts in 18 European countries (Austria, Belgium, Czechia, Denmark, Finland, France, Germany, Hungary, Ireland, Italy, the Netherlands, Norway, Poland, Portugal, Spain, Sweden, Switzerland and the United Kingdom), in their original languages. The annotation criteria are described in the supplementary material of the PartySOME paper (Section B.1).

The posts were split into training (4,422), validation (589) and test (884) sets, stratified by country.

How to use

from transformers import pipeline

classifier = pipeline("text-classification", model="mljn/policy-vs-other-v2", truncation=True)
classifier("The new government policy on taxation was announced today.")

In PartySOME, posts predicted as policy issue are then classified into CAP topics with poltextlab/xlm-roberta-large-pooled-cap-v4.

Evaluation

Results on the held-out test set (884 posts), with policy issue as the positive class:

Metric Value
Accuracy 0.893
Precision 0.905
Recall 0.914
F1 0.910

By platform:

Platform Posts Accuracy F1
Facebook 304 0.891 0.919
Instagram 282 0.908 0.927
Twitter 298 0.879 0.875

By country, F1 ranges from 0.77 (Italy) to 0.98 (Denmark), on about 49 test posts per country.

Limitations

  • Trained on posts by European political parties. Performance on other kinds of political text, or on other regions, has not been tested.
  • Country-level estimates rest on about 49 test posts each and should be read as indicative.
  • Whether a post "is about a policy issue" involves judgement calls, for example for posts mixing campaign content with policy statements.

Training hyperparameters

  • learning_rate: 2e-05
  • train_batch_size: 8
  • gradient_accumulation_steps: 2
  • eval_batch_size: 16
  • weight_decay: 0.01
  • optimizer: AdamW
  • num_epochs: 4 (best checkpoint selected on validation F1)
  • seed: 1234

Citation

If you use this model, please cite:

Jan, Malo, and Luis Sattelmayer. "Issue Competition on Social Media: A New Comparative Dataset on Parties' Issue Agendas." SocArXiv. https://osf.io/preprints/socarxiv/x47qr_v1

Downloads last month
86
Safetensors
Model size
0.3B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mljn/policy-vs-other-v2

Finetuned
(4,321)
this model
Quantizations
1 model