license: mit
datasets:
- eriktks/conll2003
language:
- en
metrics:
- accuracy
base_model:
- distilbert/distilbert-base-uncased
pipeline_tag: token-classification
library_name: transformers
tags:
- ner
- named-entity-recognition
Results
This repository contains the training results of a fine-tuned DistilBERT (distilbert-base-uncased) model on the CoNLL-2003 dataset for Named Entity Recognition (NER).
The model achieves the following results on the validation set:
- Loss: 0.2416
- Accuracy: 0.9433 (94.33%)
The trained model is available here.
Model Description
The model performs token classification to identify named entities in English text, including persons (PER), organizations (ORG), locations (LOC), and miscellaneous entities (MISC).
Intended Uses & Limitations
The model is intended for English Named Entity Recognition. Its performance may be limited when applied to specialized terminology or text outside the training domain.
Training and Evaluation Data
The model was trained and evaluated using the CoNLL-2003 dataset, which provides separate training, validation, and test splits.
The text was tokenized using the DistilBERT tokenizer with a maximum sequence length of 128 tokens. Word-level entity labels were aligned with the corresponding tokens.
Training Procedure
The pretrained DistilBERT encoder was frozen, and the token classification head was trained using the Hugging Face Trainer API. The best model checkpoint was selected based on validation accuracy.
Training Hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 16
- eval_batch_size: 16
- optimizer: AdamW
- weight_decay: 0.01
- lr_scheduler_type: linear
- num_epochs: 3
- max_sequence_length: 128
- eval_strategy: epoch
- save_strategy: epoch
- logging_steps: 50
Training Results
| Training Loss | Epoch | Step | Validation Loss | Accuracy |
|---|---|---|---|---|
| 0.278032 | 1.0 | 878 | 0.275000 | 0.935337 |
| 0.236482 | 2.0 | 1756 | 0.248705 | 0.941511 |
| 0.237519 | 3.0 | 2634 | 0.241559 | 0.943264 |
Training loss values represent the most recent logged losses at each epoch.
Final Evaluation Results
- Validation Loss: 0.241559
- Validation Accuracy: 94.33%
- Overall Training Loss: 0.267937
- Total Training Steps: 2,634
- Total Epochs: 3
Frameworks
- PyTorch
- Hugging Face Transformers
- Hugging Face Datasets
- Accelerate
- Evaluate