Results

This repository contains the training results of a fine-tuned DistilBERT (distilbert-base-uncased) model on the CoNLL-2003 dataset for Named Entity Recognition (NER).

The model achieves the following results on the validation set:

• Loss: 0.2416
• Accuracy: 0.9433 (94.33%)

The trained model is available here.

Model Description

The model performs token classification to identify named entities in English text, including persons (PER), organizations (ORG), locations (LOC), and miscellaneous entities (MISC).

Intended Uses & Limitations

The model is intended for English Named Entity Recognition. Its performance may be limited when applied to specialized terminology or text outside the training domain.

Training and Evaluation Data

The model was trained and evaluated using the CoNLL-2003 dataset, which provides separate training, validation, and test splits.

The text was tokenized using the DistilBERT tokenizer with a maximum sequence length of 128 tokens. Word-level entity labels were aligned with the corresponding tokens.

Training Procedure

The pretrained DistilBERT encoder was frozen, and the token classification head was trained using the Hugging Face Trainer API. The best model checkpoint was selected based on validation accuracy.

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• optimizer: AdamW
• weight_decay: 0.01
• lr_scheduler_type: linear
• num_epochs: 3
• max_sequence_length: 128
• eval_strategy: epoch
• save_strategy: epoch
• logging_steps: 50

Training Results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.278032	1.0	878	0.275000	0.935337
0.236482	2.0	1756	0.248705	0.941511
0.237519	3.0	2634	0.241559	0.943264

Training loss values represent the most recent logged losses at each epoch.

Final Evaluation Results

• Validation Loss: 0.241559
• Validation Accuracy: 94.33%
• Overall Training Loss: 0.267937
• Total Training Steps: 2,634
• Total Epochs: 3

Frameworks

• PyTorch
• Hugging Face Transformers
• Hugging Face Datasets
• Accelerate
• Evaluate