| --- |
| license: mit |
| datasets: |
| - eriktks/conll2003 |
| language: |
| - en |
| metrics: |
| - accuracy |
| base_model: |
| - distilbert/distilbert-base-uncased |
| pipeline_tag: token-classification |
| library_name: transformers |
| tags: |
| - ner |
| - named-entity-recognition |
| --- |
| |
| # Results |
|
|
| This repository contains the training results of a fine-tuned [**DistilBERT (distilbert-base-uncased)**](https://huggingface.co/distilbert/distilbert-base-uncased) model on the **CoNLL-2003** dataset for Named Entity Recognition (NER). |
|
|
| The model achieves the following results on the validation set: |
|
|
| - **Loss:** 0.2416 |
| - **Accuracy:** 0.9433 (94.33%) |
|
|
| The trained model is available [here](https://huggingface.co/mohammad748/distilbert-conll2003-ner). |
|
|
| ## Model Description |
|
|
| The model performs token classification to identify named entities in English text, including persons (PER), organizations (ORG), locations (LOC), and miscellaneous entities (MISC). |
|
|
| ## Intended Uses & Limitations |
|
|
| The model is intended for English Named Entity Recognition. Its performance may be limited when applied to specialized terminology or text outside the training domain. |
|
|
| ## Training and Evaluation Data |
|
|
| The model was trained and evaluated using the [CoNLL-2003](https://huggingface.co/datasets/eriktks/conll2003) dataset, which provides separate training, validation, and test splits. |
|
|
| The text was tokenized using the DistilBERT tokenizer with a maximum sequence length of 128 tokens. Word-level entity labels were aligned with the corresponding tokens. |
|
|
| ## Training Procedure |
|
|
| The pretrained DistilBERT encoder was frozen, and the token classification head was trained using the Hugging Face Trainer API. The best model checkpoint was selected based on validation accuracy. |
|
|
| ### Training Hyperparameters |
|
|
| The following hyperparameters were used during training: |
|
|
| - learning_rate: 5e-05 |
| - train_batch_size: 16 |
| - eval_batch_size: 16 |
| - optimizer: AdamW |
| - weight_decay: 0.01 |
| - lr_scheduler_type: linear |
| - num_epochs: 3 |
| - max_sequence_length: 128 |
| - eval_strategy: epoch |
| - save_strategy: epoch |
| - logging_steps: 50 |
|
|
| ### Training Results |
|
|
| | Training Loss | Epoch | Step | Validation Loss | Accuracy | |
| |---|---|---|---|---| |
| | 0.278032 | 1.0 | 878 | 0.275000 | 0.935337 | |
| | 0.236482 | 2.0 | 1756 | 0.248705 | 0.941511 | |
| | 0.237519 | 3.0 | 2634 | 0.241559 | 0.943264 | |
|
|
| *Training loss values represent the most recent logged losses at each epoch.* |
|
|
| ### Final Evaluation Results |
|
|
| - **Validation Loss:** 0.241559 |
| - **Validation Accuracy:** 94.33% |
| - **Overall Training Loss:** 0.267937 |
| - **Total Training Steps:** 2,634 |
| - **Total Epochs:** 3 |
|
|
| ### Frameworks |
|
|
| - PyTorch |
| - Hugging Face Transformers |
| - Hugging Face Datasets |
| - Accelerate |
| - Evaluate |
