| --- |
| language: |
| - english |
| thumbnail: |
| tags: |
| - token classification |
| license: agpl-3.0 |
| datasets: |
| - EMBO/sd-nlp |
| metrics: |
| - |
| --- |
| |
| # sd-ner |
|
|
| ## Model description |
|
|
| This model is a [RoBERTa base model](https://huggingface.co/roberta-base) that was further trained using a masked language modeling task on a compendium of English scientific textual examples from the life sciences using the [BioLang dataset](https://huggingface.co/datasets/EMBO/biolang). It was then fine-tuned for token classification on the SourceData [sd-nlp](https://huggingface.co/datasets/EMBO/sd-nlp) dataset with the `NER` configuration to perform Named Entity Recognition of bioentities. |
|
|
|
|
| ## Intended uses & limitations |
|
|
| #### How to use |
|
|
| The intended use of this model is for Named Entity Recognition of biological entities used in SourceData annotations (https://sourcedata.embo.org), including small molecules, gene products (genes and proteins), subcellular components, cell line and cell types, organ and tissues, species as well as experimental methods. |
|
|
| To have a quick check of the model: |
|
|
| ```python |
| from transformers import pipeline, RobertaTokenizerFast, RobertaForTokenClassification |
| example = """<s> F. Western blot of input and eluates of Upf1 domains purification in a Nmd4-HA strain. The band with the # might corresponds to a dimer of Upf1-CH, bands marked with a star correspond to residual signal with the anti-HA antibodies (Nmd4). Fragments in the eluate have a smaller size because the protein A part of the tag was removed by digestion with the TEV protease. G6PDH served as a loading control in the input samples </s>""" |
| tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base', max_len=512) |
| model = RobertaForTokenClassification.from_pretrained('EMBO/sd-ner') |
| ner = pipeline('ner', model, tokenizer=tokenizer) |
| res = ner(example) |
| for r in res: |
| print(r['word'], r['entity']) |
| ``` |
|
|
| #### Limitations and bias |
|
|
| The model must be used with the `roberta-base` tokenizer. |
|
|
| ## Training data |
|
|
| The model was trained for token classification using the [EMBO/sd-nlp dataset](https://huggingface.co/datasets/EMBO/sd-nlp) dataset which includes manually annotated examples. |
|
|
| ## Training procedure |
|
|
| The training was run on an NVIDIA DGX Station with 4XTesla V100 GPUs. |
|
|
| Training code is available at https://github.com/source-data/soda-roberta |
|
|
| - Model fine-tuned: EMBO/bio-lm |
| - Tokenizer vocab size: 50265 |
| - Training data: EMBO/sd-nlp |
| - Dataset configuration: NER |
| - Training with 48771 examples. |
| - Evaluating on 13801 examples. |
| - Training on 15 features: O, I-SMALL_MOLECULE, B-SMALL_MOLECULE, I-GENEPROD, B-GENEPROD, I-SUBCELLULAR, B-SUBCELLULAR, I-CELL, B-CELL, I-TISSUE, B-TISSUE, I-ORGANISM, B-ORGANISM, I-EXP_ASSAY, B-EXP_ASSAY |
| - Epochs: 0.6 |
| - `per_device_train_batch_size`: 16 |
| - `per_device_eval_batch_size`: 16 |
| - `learning_rate`: 0.0001 |
| - `weight_decay`: 0.0 |
| - `adam_beta1`: 0.9 |
| - `adam_beta2`: 0.999 |
| - `adam_epsilon`: 1e-08 |
| - `max_grad_norm`: 1.0 |
|
|
| ## Eval results |
|
|
| Testing on 7178 examples of test set with `sklearn.metrics`: |
|
|
| ``` |
| precision recall f1-score support |
| |
| CELL 0.69 0.81 0.74 5245 |
| EXP_ASSAY 0.56 0.57 0.56 10067 |
| GENEPROD 0.77 0.89 0.82 23587 |
| ORGANISM 0.72 0.82 0.77 3623 |
| SMALL_MOLECULE 0.70 0.80 0.75 6187 |
| SUBCELLULAR 0.65 0.72 0.69 3700 |
| TISSUE 0.62 0.73 0.67 3207 |
| |
| micro avg 0.70 0.79 0.74 55616 |
| macro avg 0.67 0.77 0.72 55616 |
| weighted avg 0.70 0.79 0.74 55616 |
| |
| {'test_loss': 0.1830928772687912, 'test_accuracy_score': 0.9334821000160841, 'test_precision': 0.6987463009514112, 'test_recall': 0.789682825086306, 'test_f1': 0.7414366506288511, 'test_runtime': 61.0547, 'test_samples_per_second': 117.567, 'test_steps_per_second': 1.851} |
| ``` |
|
|
