BERT-base-cased fine-tuned for Named Entity Recognition (CoNLL-2003)

This model is bert-base-cased fine-tuned on CoNLL-2003 to detect four entity types in English text: PER (person), ORG (organization), LOC (location) and MISC (miscellaneous). Labels follow the IOB2 scheme.

It was trained as part of an NER project that compares sequence-labeling architectures built from scratch (LSTM, BiLSTM, BiLSTM+CRF) against fine-tuned transformers.

Results (CoNLL-2003 test set, seqeval)

Entity Precision Recall F1 Support
LOC 0.9287 0.9365 0.9325 1668
MISC 0.7889 0.8305 0.8092 702
ORG 0.8960 0.9133 0.9046 1661
PER 0.9588 0.9635 0.9611 1617
micro avg 0.9096 0.9242 0.9168 5648

Validation F1 (best epoch): 0.9528.

Comparison with the other models in the project

Model Precision Recall F1
LSTM (GloVe + char-CNN) 0.7922 0.8430 0.8168
BiLSTM (GloVe + char-CNN) 0.8830 0.8962 0.8896
BiLSTM + CRF (GloVe + char-CNN) 0.9000 0.8952 0.8976
DistilBERT-base-cased 0.8956 0.9008 0.8982
BERT-base-cased (this model) 0.9096 0.9242 0.9168

The CRF mainly improves structural validity: on the test set it produced 3 invalid I- transitions versus 75 for the BiLSTM without CRF, and fewer entity-boundary errors (89 vs. 136).

Training details

  • Base model: bert-base-cased
  • Data: CoNLL-2003 (14,041 train / 3,250 validation / 3,453 test sentences)
  • Label alignment: the label is placed on the first sub-word of each word; remaining sub-words are ignored (-100) in the loss
  • Epochs: 5, learning rate 3e-5, batch size 16, weight decay 0.01, fp16, seed 42
  • Best checkpoint selected by validation F1 (epoch 5)
  • Framework: Hugging Face Transformers Trainer

Usage

import re
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

name = "mai-data-ai/ner-bert-base-cased-conll2003"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForTokenClassification.from_pretrained(name).eval()

def predict_entities(text):
    words = re.findall(r"\w+(?:[-'’]\w+)*|[^\w\s]", text)
    enc = tok(words, is_split_into_words=True, return_tensors="pt")
    with torch.no_grad():
        pred = model(**enc).logits[0].argmax(-1).tolist()
    seen, entities, cur = set(), [], None
    for i, wid in enumerate(enc.word_ids(0)):
        if wid is None or wid in seen:
            continue                      # use the first sub-word of each word only
        seen.add(wid)
        label = model.config.id2label[pred[i]]
        if label == "O":
            cur = None
        elif label.startswith("B-") or cur is None or cur["type"] != label[2:]:
            cur = {"type": label[2:], "text": words[wid]}
            entities.append(cur)
        else:
            cur["text"] += " " + words[wid]
    return [(e["text"], e["type"]) for e in entities]

print(predict_entities("Satya Nadella announced that Microsoft will invest in Egypt."))

The model was trained with one label per word (on the first sub-word), so decode at word level as above. Generic pipeline("token-classification", aggregation_strategy="simple") can split a name into two entities when a middle sub-word is predicted as B-.

Limitations

  • Case-sensitive. In a quick manual test, the fully lowercase sentence washington signed the deal with amazon. produced no detected entities. The model relies on capitalization, so it is not suited to informal or lowercased text.
  • News-domain data. CoNLL-2003 is English Reuters news from 1996-1997, so performance is likely lower on other domains and on names or organizations that are rare or newer than the data.
  • MISC is the weakest class (F1 0.81) because it is a heterogeneous category.
  • English only. Arabic or mixed-language text is not supported.
  • The scores come from a single training run with one seed, so differences of a few tenths of a point between models should not be treated as significant.
  • CoNLL-2003 is built on Reuters news text; check the dataset's terms of use before any commercial use.
Downloads last month
49
Safetensors
Model size
0.1B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for mai-data-ai/ner-bert-base-cased-conll2003

Finetuned
(2,956)
this model

Dataset used to train mai-data-ai/ner-bert-base-cased-conll2003

Space using mai-data-ai/ner-bert-base-cased-conll2003 1

Evaluation results