Add HF card metadata; fix duplicate metrics row; polish

README.md

@@ -1,22 +1,47 @@
+---
+language: en
+pipeline_tag: fill-mask
+tags:
+  - materials-science
+  - science
+  - bert
+  - fill-mask
+  - scientific-text
+base_model: allenai/scibert_scivocab_cased
+# TODO: set an explicit `license:` (the repo currently declares none).
+---
+
 # Material SciBERT (TPU): Improving language understanding in materials science
 
 **Work in progress**
 
-## Introduction 
-SciBERT-based model pre-trained with materials science scientific fulltext
+## Introduction
+SciBERT-based model pre-trained with materials-science scientific full text.
 
 ## Authors
 Luca Foppiano
 Pedro Ortiz Suarez
 
 ## TLDR
-- Collected full-text from ~700000 articles provided by the National Institute for Materials Science (NIMS) TDM platform (https://dice.nims.go.jp/services/TDM-PF/en/), dataset called ScienceCorpus (SciCorpus)
-- We added to the SciBERT vocabulary (32k tokens), 100 domain-specific unknown words extracted from SciCorpus with a keywords modeler (KeyBERT)
-- Starting conditions: original SciBERT weights 
-- Pre-train the model MatTpuSciBERT from on the Google Cloud with the TPU (Tensor Processing Unit) as follow:
-  - 800000 steps with batch_size: 256, max_seq_length:512 
-  - 100000 steps with batch_size: 2048, max_seq_length:128
-- Fine-tuning and testing on NER on superconductors (https://github.com/lfoppiano/grobid-superconductors) and physical quantities (https://github.com/kermitt2/grobid-quantities)
+- Collected full text from ~700,000 articles provided by the National Institute for Materials Science (NIMS) TDM platform (https://dice.nims.go.jp/services/TDM-PF/en/), a dataset called ScienceCorpus (SciCorpus)
+- We added to the SciBERT vocabulary (32k tokens) 100 domain-specific unknown words extracted from SciCorpus with a keyword modeller (KeyBERT)
+- Starting conditions: original SciBERT weights
+- Pre-trained the MatTPUSciBERT model on Google Cloud with a TPU (Tensor Processing Unit) as follows:
+  - 800,000 steps with batch_size 256, max_seq_length 512
+  - 100,000 steps with batch_size 2048, max_seq_length 128
+- Fine-tuned and tested on NER for superconductors (https://github.com/lfoppiano/grobid-superconductors) and physical quantities (https://github.com/kermitt2/grobid-quantities)
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("lfoppiano/MatTPUSciBERT")
+model = AutoModelForMaskedLM.from_pretrained("lfoppiano/MatTPUSciBERT")
+```
+
+The repository ships both PyTorch (`pytorch_model.bin`, `model.safetensors`) and the original
+TensorFlow checkpoint (`bert_model.ckpt.*`).
 
 ## Related work
 
@@ -26,14 +51,14 @@ Pedro Ortiz Suarez
 
 ### Relevant models
 - SciBERT: BERT, from scratch, scientific articles (biology + CS) https://github.com/allenai/scibert
-- MatSciBERT (Gupta): RoBERTa, from scratch, SciBERT vocab and weights, ~150 K paper limited to 4 MS families http://github.com/m3rg-iitd/matscibert
+- MatSciBERT (Gupta): RoBERTa, from scratch, SciBERT vocab and weights, ~150 K papers limited to 4 MS families http://github.com/m3rg-iitd/matscibert
 - MaterialBERT: Not yet published
-- MatBERT (CEDER): BERT, from scratch, 2M documents on materials science  (~60M paragraphs) https://github.com/lbnlp/MatBERT
+- MatBERT (CEDER): BERT, from scratch, 2M documents on materials science (~60M paragraphs) https://github.com/lbnlp/MatBERT
 - BatteryBERT (Cole): BERT, mixed from scratch and with predefined weights https://github.com/ShuHuang/batterybert/
 
-### Results 
+### Results
 
-Results obtained via 10-fold cross-validation, using DeLFT (https://github.com/kermitt2/delft)
+Results obtained via 10-fold cross-validation, using DeLFT (https://github.com/kermitt2/delft).
 
 #### NER Superconductors
 
@@ -54,13 +79,11 @@ Results obtained via 10-fold cross-validation, using DeLFT (https://github.com/k
 | MatTPUSciBERT        | 88.62%    | 86.33%  | 87.46%   |
 | MatBERT (Ceder)      | 85.08%    | 89.93%  | 87.44%   |
 | BatteryScibert-cased | 85.02%    | 89.30%  | 87.11%   |
-| BatteryScibert-cased | 81.09%    | 84.14%  | 82.59%   |
-   
 
 ## References
 
-This work was supported by Google, through the researchers program https://cloud.google.com/edu/researchers 
+This work was supported by Google, through the researchers program https://cloud.google.com/edu/researchers
 
 ## Acknowledgements
 
-TBA
+TBA