RoBERTa-base for News Domain Named Entity Recognition using Transformer Architecture

A FacebookAI/roberta-base model fine-tuned for Named Entity Recognition (NER) on political, business, defence, crime, and international news text, using a custom 17-tag entity schema.

This model was trained as part of a backbone comparison study (BERT vs. RoBERTa vs. ELECTRA vs. DeBERTa-v3) for an automated political news monitoring system. RoBERTa achieved the best overall micro F1-score (0.8788) among all four backbones evaluated.

Model Details

Base model FacebookAI/roberta-base
Task Token Classification (Named Entity Recognition)
Language English
Labels 17 entity types (35 BIO tags + O)
Training framework πŸ€— Transformers Trainer (custom WeightedTrainer)
Max sequence length 256 tokens
License MIT

Entity Schema

The model predicts the following 17 entity types in BIO format (B-TAG / I-TAG / O):

Tag Description Tag Description
PER Person ORG Organization
LOC Location FAC Facility
DATE Date TIME Time
EVT Event PROD Product
MONEY Monetary value PCT Percentage
LAW Law / legal reference NORP Nationality, religious, or political group
LANG Language WOA Work of Art
QTY Quantity ORD Ordinal Number
CARD Cardinal Number

Training Data

The model was trained on a manually annotated and audited dataset of political/business/defence/crime/international news sentences, originally collected from 40+ sources (Reuters, Al Jazeera, NPR, DW, BBC, Indian Express, and others) and annotated in Label Studio.

Key dataset characteristics:

  • BIO-tagged spans across all 17 entity categories
  • Cleaned via a multi-stage audit pipeline (overlap resolution, whitespace trimming, deduplication, majority-vote label correction)
  • Split 80% train / 10% validation / 10% test (held-out test set never used during training, tuning, or checkpoint selection)

Evaluation Results

Evaluated on a held-out test set never seen during training, validation, or hyperparameter tuning.

Overall Test Set Performance

Metric Score
Precision 0.8641
Recall 0.8941
Micro F1-score 0.8788
Macro F1-score 0.8134
Weighted F1-score 0.8799

The model achieved a micro F1-score of 87.88%, slightly exceeding the best validation F1-score of 87.39% β€” indicating strong generalization with no signs of overfitting.

Per-Entity F1-Scores

High-Frequency F1 Medium-Frequency F1 Low-Frequency F1
PER 0.9501 ORG 0.8404 LAW 0.7417
LOC 0.9425 PROD 0.8094 EVT 0.7407
NORP 0.9273 TIME 0.7831 FAC 0.6316
PCT 0.9055 LANG 0.7719 WOA 0.5732
MONEY 0.8869 QTY 0.7005
ORD 0.8816
CARD 0.8791
DATE 0.8627

Validation vs. Test Generalization

Metric Validation Test
Precision 0.8585 0.8641
Recall 0.8898 0.8941
F1-score 0.8739 0.8788

The test F1-score exceeds the validation F1-score by ~0.49 points, confirming the model generalizes well to unseen data.

Interpretation

Performance is strongest on high-frequency, well-bounded categories (PER, LOC, NORP, PCT, MONEY), all scoring near or above 0.90 F1. The weighted loss strategy (3Γ— weight on rare tags) helped lift historically difficult classes like LAW, EVT, and FAC.

WOA (Work of Art) remains the most challenging category. This is attributed less to raw data sparsity and more to annotation noise and semantic boundary overlap with adjacent categories (e.g. titles, named works, and policy/product references being inconsistently distinguished) β€” a pattern that has held consistently across backbone comparisons on this dataset.

Intended Use

This model is intended for extracting structured entities from English-language political, business, defence, crime, and international news etc. text β€” e.g. for automated news monitoring, media analysis, or knowledge-base population pipelines.

It is not intended for:

  • Non-English text
  • Domains substantially different from news (e.g. legal contracts, medical records, social media slang)
  • High-stakes automated decision-making without human review

How to Use

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="Souvikbasur/roberta-base-news-ner",
    aggregation_strategy="simple",
)

result = ner("Narendra Modi met officials in New Delhi on Monday regarding the new defence policy.")
for entity in result:
    print(f"{entity['entity_group']:6s} | {entity['word']:20s} | score={entity['score']:.3f}")

Or load the model and tokenizer directly:

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("Souvikbasur/roberta-base-news-ner")
model = AutoModelForTokenClassification.from_pretrained("Souvikbasur/roberta-base-news-ner")

text = "The United Nations Security Council discussed the proposed law on climate change in Geneva."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

predictions = torch.argmax(outputs.logits, dim=-1)
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

for token, pred_id in zip(tokens, predictions[0]):
    label = model.config.id2label[pred_id.item()]
    if label != "O" and token not in tokenizer.all_special_tokens:
        print(f"{token:15s} -> {label}")

Limitations

  • WOA (Work of Art) and FAC (Facility) categories have the lowest F1-scores (0.57 and 0.63 respectively) due to annotation boundary ambiguity in the training data, not just low support.
  • The model was trained and evaluated primarily on political/news-domain text; performance on other domains is not guaranteed.
  • As with any NER model, performance may degrade on entity names, events, or terminology not well represented in the training distribution.

Citation

If you use this model, please reference the base model and this fine-tuned checkpoint:

Base model: FacebookAI/roberta-base (Liu et al., 2019 β€” RoBERTa: A Robustly Optimized BERT Pretraining Approach)
Fine-tuned checkpoint: Souvikbasur/roberta-base-news-ner
Downloads last month
8
Safetensors
Model size
0.1B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Souvikbasur/roberta-base-news-ner

Finetuned
(2,421)
this model