🚀 Initial model upload - ArticleTypePredictionImproved

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+confusion_matrix.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,109 @@
+---
+license: mit
+language: 
+- en
+- fr
+tags:
+- covid-19
+- biomedical
+- classification
+- pubmed
+- scientific-literature
+datasets:
+- custom-covid19-publications
+metrics:
+- accuracy
+- f1
+library_name: transformers
+pipeline_tag: text-classification
+---
+
+# ArticleTypePredictionImproved - COVID-19 Publication Classifier
+
+## Description
+
+Ce modèle est un classificateur fine-tuné et optimisé spécialement pour identifier le type de publications scientifiques COVID-19. 
+Il est basé sur PubMedBERT et a été entraîné sur un dataset équilibré de publications COVID-19 provenant de PubMed avec des techniques d'optimisation avancées.
+
+## Performance
+
+- **Accuracy**: 93.80%
+- **F1-Score**: 93.77%
+- **Meilleur modèle**: checkpoint-2000
+
+## Classes supportées
+
+Le modèle peut classifier 9 types de publications :
+
+0. **CASE_REPORTS** - Rapports de cas
+1. **COMMENT** - Commentaires
+2. **EDITORIAL** - Éditoriaux
+3. **GUIDELINES** - Directives cliniques
+4. **META_ANALYSIS** - Méta-analyses
+5. **PROSPECTIVE** - Études prospectives
+6. **RCT** - Essais contrôlés randomisés
+7. **RETROSPECTIVE** - Études rétrospectives
+8. **REVIEW** - Revues de littérature
+
+## Utilisation
+
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+import torch
+
+# Charger le modèle
+model_name = "slepape/ArticleTypePredictionImproved"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForSequenceClassification.from_pretrained(model_name)
+
+# Préparer le texte (titre + abstract)
+title = "High-Flow Nasal Cannula Oxygen Therapy in COVID-19 Patients"
+abstract = "This study evaluates the effectiveness of HFNC in COVID-19 patients..."
+text = f"[TITLE] {title} [SEP] [ABSTRACT] {abstract}"
+
+# Prédiction
+inputs = tokenizer(text, return_tensors="pt", max_length=512, truncation=True, padding=True)
+with torch.no_grad():
+    outputs = model(**inputs)
+    predictions = torch.softmax(outputs.logits, dim=-1)
+    predicted_class = torch.argmax(predictions, dim=-1).item()
+
+# Mapping des classes
+class_names = [
+    "CASE_REPORTS", "COMMENT", "EDITORIAL", "GUIDELINES",
+    "META_ANALYSIS", "PROSPECTIVE", "RCT", "RETROSPECTIVE", "REVIEW"
+]
+
+print(f"Type de publication prédit: {class_names[predicted_class]}")
+print(f"Confiance: {predictions[0][predicted_class]:.3f}")
+```
+
+## Entraînement
+
+- **Modèle de base**: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext
+- **Dataset**: Publications COVID-19 équilibrées (9 classes)
+- **Technique**: Fine-tuning avec optimisation des hyperparamètres
+- **Combinaison**: Titre + Abstract avec tokens spéciaux [TITLE], [SEP], [ABSTRACT]
+
+## Évaluation
+
+Le modèle a été testé sur plusieurs types de publications COVID-19 avec d'excellents résultats :
+- RCT: 100% de précision
+- Confiance >90% sur 96% des prédictions
+- Classification équilibrée sur toutes les classes
+
+## Limitations
+
+- Spécialisé pour les publications COVID-19
+- Optimisé pour des textes en anglais (titres + abstracts PubMed)
+- Performance optimale avec la structure [TITLE] ... [SEP] [ABSTRACT] ...
+
+## Citation
+
+Si vous utilisez ce modèle, merci de citer :
+
+```
+ArticleTypePredictionImproved - COVID-19 Publication Type Classifier
+Optimized fine-tuned PubMedBERT for COVID-19 scientific literature classification
+2024
+```

checkpoint-2000/config.json

@@ -0,0 +1,47 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "CASE_REPORT",
+    "1": "COMMENT",
+    "2": "EDITORIAL",
+    "3": "GUIDELINES",
+    "4": "META_ANALYSIS",
+    "5": "PROSPECTIVE",
+    "6": "RCT",
+    "7": "RETROSPECTIVE",
+    "8": "REVIEW"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "CASE_REPORT": 0,
+    "COMMENT": 1,
+    "EDITORIAL": 2,
+    "GUIDELINES": 3,
+    "META_ANALYSIS": 4,
+    "PROSPECTIVE": 5,
+    "RCT": 6,
+    "RETROSPECTIVE": 7,
+    "REVIEW": 8
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.4",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

checkpoint-2000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d5c5c3708cdfa9e9a61dfaa2e11b5a29c598444694702ceba0bab2ee4bd39b4
+size 437980180

checkpoint-2000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b66a9203d34f39e2fd46253120746b02bac6f77c0f5ae40aa39d04b0588cea43
+size 876076171

checkpoint-2000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dba4fde4ee04d2f472bb4dea96a48e8fdf7891d2b0694a8f012e8133a2e176ae
+size 14455

checkpoint-2000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9eacf1e72c39a718e9cbac1ea19006f28f483695284b1b4e01710c3f626b127
+size 1465

checkpoint-2000/trainer_state.json

@@ -0,0 +1,201 @@
+{
+  "best_global_step": 2000,
+  "best_metric": 0.9388425925925926,
+  "best_model_checkpoint": "myocardial_pubmedbert_mps_final_v3/checkpoint-2000",
+  "epoch": 0.37037037037037035,
+  "eval_steps": 1000,
+  "global_step": 2000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.018518518518518517,
+      "grad_norm": 2.475142478942871,
+      "learning_rate": 9.9e-08,
+      "loss": 0.1439,
+      "step": 100
+    },
+    {
+      "epoch": 0.037037037037037035,
+      "grad_norm": 1.806380271911621,
+      "learning_rate": 1.99e-07,
+      "loss": 0.1511,
+      "step": 200
+    },
+    {
+      "epoch": 0.05555555555555555,
+      "grad_norm": 1.5451908111572266,
+      "learning_rate": 2.9899999999999996e-07,
+      "loss": 0.1491,
+      "step": 300
+    },
+    {
+      "epoch": 0.07407407407407407,
+      "grad_norm": 2.704113483428955,
+      "learning_rate": 3.99e-07,
+      "loss": 0.153,
+      "step": 400
+    },
+    {
+      "epoch": 0.09259259259259259,
+      "grad_norm": 4.833273887634277,
+      "learning_rate": 4.99e-07,
+      "loss": 0.149,
+      "step": 500
+    },
+    {
+      "epoch": 0.1111111111111111,
+      "grad_norm": 1.2810375690460205,
+      "learning_rate": 5.989999999999999e-07,
+      "loss": 0.1582,
+      "step": 600
+    },
+    {
+      "epoch": 0.12962962962962962,
+      "grad_norm": 9.05034065246582,
+      "learning_rate": 6.989999999999999e-07,
+      "loss": 0.1711,
+      "step": 700
+    },
+    {
+      "epoch": 0.14814814814814814,
+      "grad_norm": 7.097633361816406,
+      "learning_rate": 7.99e-07,
+      "loss": 0.1419,
+      "step": 800
+    },
+    {
+      "epoch": 0.16666666666666666,
+      "grad_norm": 10.607206344604492,
+      "learning_rate": 8.99e-07,
+      "loss": 0.1298,
+      "step": 900
+    },
+    {
+      "epoch": 0.18518518518518517,
+      "grad_norm": 1.5295910835266113,
+      "learning_rate": 9.989999999999999e-07,
+      "loss": 0.1401,
+      "step": 1000
+    },
+    {
+      "epoch": 0.18518518518518517,
+      "eval_accuracy": 0.9386111111111111,
+      "eval_loss": 0.1337883323431015,
+      "eval_runtime": 906.5981,
+      "eval_samples_per_second": 47.651,
+      "eval_steps_per_second": 1.489,
+      "step": 1000
+    },
+    {
+      "epoch": 0.2037037037037037,
+      "grad_norm": 7.718695163726807,
+      "learning_rate": 9.98132075471698e-07,
+      "loss": 0.1463,
+      "step": 1100
+    },
+    {
+      "epoch": 0.2222222222222222,
+      "grad_norm": 2.2934257984161377,
+      "learning_rate": 9.962452830188678e-07,
+      "loss": 0.1505,
+      "step": 1200
+    },
+    {
+      "epoch": 0.24074074074074073,
+      "grad_norm": 4.265007495880127,
+      "learning_rate": 9.943584905660378e-07,
+      "loss": 0.1451,
+      "step": 1300
+    },
+    {
+      "epoch": 0.25925925925925924,
+      "grad_norm": 8.984698295593262,
+      "learning_rate": 9.924716981132075e-07,
+      "loss": 0.1412,
+      "step": 1400
+    },
+    {
+      "epoch": 0.2777777777777778,
+      "grad_norm": 6.165067672729492,
+      "learning_rate": 9.905849056603772e-07,
+      "loss": 0.1354,
+      "step": 1500
+    },
+    {
+      "epoch": 0.2962962962962963,
+      "grad_norm": 14.949434280395508,
+      "learning_rate": 9.886981132075472e-07,
+      "loss": 0.1305,
+      "step": 1600
+    },
+    {
+      "epoch": 0.3148148148148148,
+      "grad_norm": 5.017131805419922,
+      "learning_rate": 9.86811320754717e-07,
+      "loss": 0.147,
+      "step": 1700
+    },
+    {
+      "epoch": 0.3333333333333333,
+      "grad_norm": 6.54106330871582,
+      "learning_rate": 9.849245283018867e-07,
+      "loss": 0.1435,
+      "step": 1800
+    },
+    {
+      "epoch": 0.35185185185185186,
+      "grad_norm": 10.726076126098633,
+      "learning_rate": 9.830377358490567e-07,
+      "loss": 0.1347,
+      "step": 1900
+    },
+    {
+      "epoch": 0.37037037037037035,
+      "grad_norm": 7.49448299407959,
+      "learning_rate": 9.811509433962264e-07,
+      "loss": 0.1539,
+      "step": 2000
+    },
+    {
+      "epoch": 0.37037037037037035,
+      "eval_accuracy": 0.9388425925925926,
+      "eval_loss": 0.13230355083942413,
+      "eval_runtime": 839.9501,
+      "eval_samples_per_second": 51.432,
+      "eval_steps_per_second": 1.607,
+      "step": 2000
+    }
+  ],
+  "logging_steps": 100,
+  "max_steps": 54000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 10,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 2,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 0
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1.6840165883904e+16,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:362c2471581c039504f3a963a573d5cf7219913796a671892f8544e81b705cfd
+size 5713

checkpoint-4000/config.json

@@ -0,0 +1,47 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "CASE_REPORT",
+    "1": "COMMENT",
+    "2": "EDITORIAL",
+    "3": "GUIDELINES",
+    "4": "META_ANALYSIS",
+    "5": "PROSPECTIVE",
+    "6": "RCT",
+    "7": "RETROSPECTIVE",
+    "8": "REVIEW"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "CASE_REPORT": 0,
+    "COMMENT": 1,
+    "EDITORIAL": 2,
+    "GUIDELINES": 3,
+    "META_ANALYSIS": 4,
+    "PROSPECTIVE": 5,
+    "RCT": 6,
+    "RETROSPECTIVE": 7,
+    "REVIEW": 8
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.4",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

checkpoint-4000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12374cf2d9664e7b18f8441d22ebb3d4fdd87418489116e8d9944e378959f4ff
+size 437980180

checkpoint-4000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6b9b94786801649429d988aa2cd717776e7f215638b2537db536033065d5c42
+size 876076171

checkpoint-4000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71b1e5ce7a350c52ef18aceb7dce8450b476076d04fd51abbdfbe08e044226c5
+size 14455

checkpoint-4000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:001f3626f1d9238e748ea3a7d088d8a4ad01e1e409314110838f01ca1f59b98f
+size 1465

checkpoint-4000/trainer_state.json

@@ -0,0 +1,359 @@
+{
+  "best_global_step": 2000,
+  "best_metric": 0.9388425925925926,
+  "best_model_checkpoint": "myocardial_pubmedbert_mps_final_v3/checkpoint-2000",
+  "epoch": 0.7407407407407407,
+  "eval_steps": 1000,
+  "global_step": 4000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.018518518518518517,
+      "grad_norm": 2.475142478942871,
+      "learning_rate": 9.9e-08,
+      "loss": 0.1439,
+      "step": 100
+    },
+    {
+      "epoch": 0.037037037037037035,
+      "grad_norm": 1.806380271911621,
+      "learning_rate": 1.99e-07,
+      "loss": 0.1511,
+      "step": 200
+    },
+    {
+      "epoch": 0.05555555555555555,
+      "grad_norm": 1.5451908111572266,
+      "learning_rate": 2.9899999999999996e-07,
+      "loss": 0.1491,
+      "step": 300
+    },
+    {
+      "epoch": 0.07407407407407407,
+      "grad_norm": 2.704113483428955,
+      "learning_rate": 3.99e-07,
+      "loss": 0.153,
+      "step": 400
+    },
+    {
+      "epoch": 0.09259259259259259,
+      "grad_norm": 4.833273887634277,
+      "learning_rate": 4.99e-07,
+      "loss": 0.149,
+      "step": 500
+    },
+    {
+      "epoch": 0.1111111111111111,
+      "grad_norm": 1.2810375690460205,
+      "learning_rate": 5.989999999999999e-07,
+      "loss": 0.1582,
+      "step": 600
+    },
+    {
+      "epoch": 0.12962962962962962,
+      "grad_norm": 9.05034065246582,
+      "learning_rate": 6.989999999999999e-07,
+      "loss": 0.1711,
+      "step": 700
+    },
+    {
+      "epoch": 0.14814814814814814,
+      "grad_norm": 7.097633361816406,
+      "learning_rate": 7.99e-07,
+      "loss": 0.1419,
+      "step": 800
+    },
+    {
+      "epoch": 0.16666666666666666,
+      "grad_norm": 10.607206344604492,
+      "learning_rate": 8.99e-07,
+      "loss": 0.1298,
+      "step": 900
+    },
+    {
+      "epoch": 0.18518518518518517,
+      "grad_norm": 1.5295910835266113,
+      "learning_rate": 9.989999999999999e-07,
+      "loss": 0.1401,
+      "step": 1000
+    },
+    {
+      "epoch": 0.18518518518518517,
+      "eval_accuracy": 0.9386111111111111,
+      "eval_loss": 0.1337883323431015,
+      "eval_runtime": 906.5981,
+      "eval_samples_per_second": 47.651,
+      "eval_steps_per_second": 1.489,
+      "step": 1000
+    },
+    {
+      "epoch": 0.2037037037037037,
+      "grad_norm": 7.718695163726807,
+      "learning_rate": 9.98132075471698e-07,
+      "loss": 0.1463,
+      "step": 1100
+    },
+    {
+      "epoch": 0.2222222222222222,
+      "grad_norm": 2.2934257984161377,
+      "learning_rate": 9.962452830188678e-07,
+      "loss": 0.1505,
+      "step": 1200
+    },
+    {
+      "epoch": 0.24074074074074073,
+      "grad_norm": 4.265007495880127,
+      "learning_rate": 9.943584905660378e-07,
+      "loss": 0.1451,
+      "step": 1300
+    },
+    {
+      "epoch": 0.25925925925925924,
+      "grad_norm": 8.984698295593262,
+      "learning_rate": 9.924716981132075e-07,
+      "loss": 0.1412,
+      "step": 1400
+    },
+    {
+      "epoch": 0.2777777777777778,
+      "grad_norm": 6.165067672729492,
+      "learning_rate": 9.905849056603772e-07,
+      "loss": 0.1354,
+      "step": 1500
+    },
+    {
+      "epoch": 0.2962962962962963,
+      "grad_norm": 14.949434280395508,
+      "learning_rate": 9.886981132075472e-07,
+      "loss": 0.1305,
+      "step": 1600
+    },
+    {
+      "epoch": 0.3148148148148148,
+      "grad_norm": 5.017131805419922,
+      "learning_rate": 9.86811320754717e-07,
+      "loss": 0.147,
+      "step": 1700
+    },
+    {
+      "epoch": 0.3333333333333333,
+      "grad_norm": 6.54106330871582,
+      "learning_rate": 9.849245283018867e-07,
+      "loss": 0.1435,
+      "step": 1800
+    },
+    {
+      "epoch": 0.35185185185185186,
+      "grad_norm": 10.726076126098633,
+      "learning_rate": 9.830377358490567e-07,
+      "loss": 0.1347,
+      "step": 1900
+    },
+    {
+      "epoch": 0.37037037037037035,
+      "grad_norm": 7.49448299407959,
+      "learning_rate": 9.811509433962264e-07,
+      "loss": 0.1539,
+      "step": 2000
+    },
+    {
+      "epoch": 0.37037037037037035,
+      "eval_accuracy": 0.9388425925925926,
+      "eval_loss": 0.13230355083942413,
+      "eval_runtime": 839.9501,
+      "eval_samples_per_second": 51.432,
+      "eval_steps_per_second": 1.607,
+      "step": 2000
+    },
+    {
+      "epoch": 0.3888888888888889,
+      "grad_norm": 6.918424129486084,
+      "learning_rate": 9.792641509433961e-07,
+      "loss": 0.1403,
+      "step": 2100
+    },
+    {
+      "epoch": 0.4074074074074074,
+      "grad_norm": 13.528965950012207,
+      "learning_rate": 9.77377358490566e-07,
+      "loss": 0.166,
+      "step": 2200
+    },
+    {
+      "epoch": 0.42592592592592593,
+      "grad_norm": 19.130395889282227,
+      "learning_rate": 9.754905660377358e-07,
+      "loss": 0.1436,
+      "step": 2300
+    },
+    {
+      "epoch": 0.4444444444444444,
+      "grad_norm": 6.962574481964111,
+      "learning_rate": 9.736037735849056e-07,
+      "loss": 0.136,
+      "step": 2400
+    },
+    {
+      "epoch": 0.46296296296296297,
+      "grad_norm": 3.4236555099487305,
+      "learning_rate": 9.717169811320755e-07,
+      "loss": 0.1409,
+      "step": 2500
+    },
+    {
+      "epoch": 0.48148148148148145,
+      "grad_norm": 4.90040397644043,
+      "learning_rate": 9.698301886792453e-07,
+      "loss": 0.1341,
+      "step": 2600
+    },
+    {
+      "epoch": 0.5,
+      "grad_norm": 1.5071120262145996,
+      "learning_rate": 9.67943396226415e-07,
+      "loss": 0.1329,
+      "step": 2700
+    },
+    {
+      "epoch": 0.5185185185185185,
+      "grad_norm": 5.357398509979248,
+      "learning_rate": 9.660566037735848e-07,
+      "loss": 0.1429,
+      "step": 2800
+    },
+    {
+      "epoch": 0.5370370370370371,
+      "grad_norm": 6.533838748931885,
+      "learning_rate": 9.641698113207547e-07,
+      "loss": 0.1474,
+      "step": 2900
+    },
+    {
+      "epoch": 0.5555555555555556,
+      "grad_norm": 6.865920066833496,
+      "learning_rate": 9.622830188679245e-07,
+      "loss": 0.1443,
+      "step": 3000
+    },
+    {
+      "epoch": 0.5555555555555556,
+      "eval_accuracy": 0.9375462962962963,
+      "eval_loss": 0.13121084868907928,
+      "eval_runtime": 878.1919,
+      "eval_samples_per_second": 49.192,
+      "eval_steps_per_second": 1.537,
+      "step": 3000
+    },
+    {
+      "epoch": 0.5740740740740741,
+      "grad_norm": 25.085233688354492,
+      "learning_rate": 9.603962264150942e-07,
+      "loss": 0.1256,
+      "step": 3100
+    },
+    {
+      "epoch": 0.5925925925925926,
+      "grad_norm": 2.6678526401519775,
+      "learning_rate": 9.585094339622642e-07,
+      "loss": 0.1551,
+      "step": 3200
+    },
+    {
+      "epoch": 0.6111111111111112,
+      "grad_norm": 2.9341561794281006,
+      "learning_rate": 9.56622641509434e-07,
+      "loss": 0.1515,
+      "step": 3300
+    },
+    {
+      "epoch": 0.6296296296296297,
+      "grad_norm": 5.913675785064697,
+      "learning_rate": 9.547358490566037e-07,
+      "loss": 0.1433,
+      "step": 3400
+    },
+    {
+      "epoch": 0.6481481481481481,
+      "grad_norm": 0.7116216421127319,
+      "learning_rate": 9.528490566037735e-07,
+      "loss": 0.1361,
+      "step": 3500
+    },
+    {
+      "epoch": 0.6666666666666666,
+      "grad_norm": 0.6480686068534851,
+      "learning_rate": 9.509622641509434e-07,
+      "loss": 0.1358,
+      "step": 3600
+    },
+    {
+      "epoch": 0.6851851851851852,
+      "grad_norm": 3.0181212425231934,
+      "learning_rate": 9.490754716981132e-07,
+      "loss": 0.1501,
+      "step": 3700
+    },
+    {
+      "epoch": 0.7037037037037037,
+      "grad_norm": 2.563554048538208,
+      "learning_rate": 9.47188679245283e-07,
+      "loss": 0.1347,
+      "step": 3800
+    },
+    {
+      "epoch": 0.7222222222222222,
+      "grad_norm": 5.095635414123535,
+      "learning_rate": 9.453018867924528e-07,
+      "loss": 0.1425,
+      "step": 3900
+    },
+    {
+      "epoch": 0.7407407407407407,
+      "grad_norm": 7.968899250030518,
+      "learning_rate": 9.434150943396227e-07,
+      "loss": 0.143,
+      "step": 4000
+    },
+    {
+      "epoch": 0.7407407407407407,
+      "eval_accuracy": 0.938587962962963,
+      "eval_loss": 0.13063164055347443,
+      "eval_runtime": 772.4302,
+      "eval_samples_per_second": 55.927,
+      "eval_steps_per_second": 1.748,
+      "step": 4000
+    }
+  ],
+  "logging_steps": 100,
+  "max_steps": 54000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 10,
+  "save_steps": 1000,
+  "stateful_callbacks": {
+    "EarlyStoppingCallback": {
+      "args": {
+        "early_stopping_patience": 2,
+        "early_stopping_threshold": 0.0
+      },
+      "attributes": {
+        "early_stopping_patience_counter": 2
+      }
+    },
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.3680331767808e+16,
+  "train_batch_size": 32,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-4000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:362c2471581c039504f3a963a573d5cf7219913796a671892f8544e81b705cfd
+size 5713

config.json

@@ -0,0 +1,47 @@
+{
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "CASE_REPORT",
+    "1": "COMMENT",
+    "2": "EDITORIAL",
+    "3": "GUIDELINES",
+    "4": "META_ANALYSIS",
+    "5": "PROSPECTIVE",
+    "6": "RCT",
+    "7": "RETROSPECTIVE",
+    "8": "REVIEW"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "CASE_REPORT": 0,
+    "COMMENT": 1,
+    "EDITORIAL": 2,
+    "GUIDELINES": 3,
+    "META_ANALYSIS": 4,
+    "PROSPECTIVE": 5,
+    "RCT": 6,
+    "RETROSPECTIVE": 7,
+    "REVIEW": 8
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.55.4",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

evaluation_results.json

@@ -0,0 +1,105 @@
+{
+  "test_accuracy": 0.937962962962963,
+  "classification_report": "               precision    recall  f1-score   support\n\n  CASE_REPORT     0.9657    0.9663    0.9660      6000\n      COMMENT     0.8575    0.8357    0.8465      6000\n    EDITORIAL     0.8386    0.8660    0.8521      6000\n   GUIDELINES     0.9962    1.0000    0.9981      6000\nMETA_ANALYSIS     0.9071    0.9877    0.9457      6000\n  PROSPECTIVE     0.9601    0.9898    0.9747      6000\n          RCT     0.9884    0.9768    0.9826      6000\nRETROSPECTIVE     0.9857    0.9653    0.9754      6000\n       REVIEW     0.9480    0.8540    0.8986      6000\n\n     accuracy                         0.9380     54000\n    macro avg     0.9386    0.9380    0.9377     54000\n weighted avg     0.9386    0.9380    0.9377     54000\n",
+  "confusion_matrix": [
+    [
+      5798,
+      12,
+      2,
+      0,
+      1,
+      0,
+      7,
+      2,
+      178
+    ],
+    [
+      0,
+      5014,
+      974,
+      12,
+      0,
+      0,
+      0,
+      0,
+      0
+    ],
+    [
+      0,
+      804,
+      5196,
+      0,
+      0,
+      0,
+      0,
+      0,
+      0
+    ],
+    [
+      0,
+      0,
+      0,
+      6000,
+      0,
+      0,
+      0,
+      0,
+      0
+    ],
+    [
+      0,
+      5,
+      0,
+      0,
+      5926,
+      1,
+      25,
+      0,
+      43
+    ],
+    [
+      0,
+      0,
+      0,
+      0,
+      0,
+      5939,
+      0,
+      61,
+      0
+    ],
+    [
+      1,
+      8,
+      6,
+      0,
+      14,
+      33,
+      5861,
+      17,
+      60
+    ],
+    [
+      0,
+      0,
+      0,
+      0,
+      0,
+      208,
+      0,
+      5792,
+      0
+    ],
+    [
+      205,
+      4,
+      18,
+      11,
+      592,
+      5,
+      37,
+      4,
+      5124
+    ]
+  ]
+}

label_mapping.json

@@ -0,0 +1,11 @@
+{
+  "CASE_REPORT": 0,
+  "COMMENT": 1,
+  "EDITORIAL": 2,
+  "GUIDELINES": 3,
+  "META_ANALYSIS": 4,
+  "PROSPECTIVE": 5,
+  "RCT": 6,
+  "RETROSPECTIVE": 7,
+  "REVIEW": 8
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d5c5c3708cdfa9e9a61dfaa2e11b5a29c598444694702ceba0bab2ee4bd39b4
+size 437980180

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,65 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "max_length": 512,
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:362c2471581c039504f3a963a573d5cf7219913796a671892f8544e81b705cfd
+size 5713

training_config.json

@@ -0,0 +1,35 @@
+{
+  "model_name": "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext",
+  "task": "text-classification",
+  "num_labels": 9,
+  "classes": [
+    "CASE_REPORTS",
+    "COMMENT",
+    "EDITORIAL",
+    "GUIDELINES",
+    "META_ANALYSIS",
+    "PROSPECTIVE",
+    "RCT",
+    "RETROSPECTIVE",
+    "REVIEW"
+  ],
+  "training_details": {
+    "final_accuracy": 0.937962962962963,
+    "final_f1": 0.9377,
+    "total_samples": 54000,
+    "epochs": 10,
+    "learning_rate": 0.0001,
+    "batch_size": 32,
+    "max_length": 512,
+    "best_checkpoint": "checkpoint-2000"
+  },
+  "text_preprocessing": {
+    "format": "[TITLE] {title} [SEP] [ABSTRACT] {abstract}",
+    "special_tokens": [
+      "[TITLE]",
+      "[SEP]",
+      "[ABSTRACT]"
+    ],
+    "truncation_strategy": "optimized_title_abstract_split"
+  }
+}