Duplicate from ncbi/MedCPT-Query-Encoder

Co-authored-by: Qiao Jin <qiaojin@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,20 @@
+                          PUBLIC DOMAIN NOTICE
+              National Center for Biotechnology Information
+
+This software/database is a "United States Government Work" under the terms of
+the United States Copyright Act.  It was written as part of the author's
+official duties as a United States Government employee and thus cannot be
+copyrighted.  This software/database is freely available to the public for use.
+The National Library of Medicine and the U.S. Government have not placed any
+restriction on its use or reproduction.
+
+Although all reasonable efforts have been taken to ensure the accuracy and
+reliability of the software and data, the NLM and the U.S. Government do not and
+cannot warrant the performance or results that may be obtained by using this
+software or data. The NLM and the U.S. Government disclaim all warranties,
+express or implied, including warranties of performance, merchantability or
+fitness for any particular purpose.
+
+Please cite the author in any work or product based on this material:
+Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, John Wilbur, Zhiyong Lu.
+MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval

README.md

@@ -0,0 +1,89 @@
+---
+license: other
+license_name: public-domain
+license_link: LICENSE
+---
+# MedCPT Introduction
+
+**MedCPT generates embeddings of biomedical texts that can be used for semantic search (dense retrieval)**. The model contains two encoders:
+- [MedCPT Query Encoder](https://huggingface.co/ncbi/MedCPT-Query-Encoder): compute the embeddings of short texts (e.g., questions, search queries, sentences).
+- [MedCPT Article Encoder](https://huggingface.co/ncbi/MedCPT-Article-Encoder): compute the embeddings of articles (e.g., PubMed titles & abstracts).
+
+**This repo contains the MedCPT Query Encoder.**
+
+**MedCPT has been pre-trained by an unprecedented scale of 255M query-article pairs from PubMed search logs**, and has been shown to achieve state-of-the-art performance on several zero-shot biomedical IR datasets. In general, there are three use cases:
+1. Query-to-article search with both encoders.
+2. Query representation for clustering or query-to-query search with the [query encoder](https://huggingface.co/ncbi/MedCPT-Query-Encoder).
+3. Article representation for clustering or article-to-article search with the [article encoder](https://huggingface.co/ncbi/MedCPT-Article-Encoder).
+
+For more details, please check out our [paper](https://arxiv.org/abs/2307.00589) (Bioinformatics, 2023). Please note that the released version is slightly different from the version reported in the paper.
+
+# Case 1. Using the MedCPT Query Encoder
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModel
+
+model = AutoModel.from_pretrained("ncbi/MedCPT-Query-Encoder")
+tokenizer = AutoTokenizer.from_pretrained("ncbi/MedCPT-Query-Encoder")
+
+queries = [
+	"diabetes treatment", 
+	"How to treat diabetes?", 
+	"A 45-year-old man presents with increased thirst and frequent urination over the past 3 months.",
+]
+
+with torch.no_grad():
+	# tokenize the queries
+	encoded = tokenizer(
+		queries, 
+		truncation=True, 
+		padding=True, 
+		return_tensors='pt', 
+		max_length=64,
+	)
+	
+	# encode the queries (use the [CLS] last hidden states as the representations)
+	embeds = model(**encoded).last_hidden_state[:, 0, :]
+
+	print(embeds)
+	print(embeds.size())
+```
+The output will be:
+```bash
+tensor([[ 0.0413,  0.0084, -0.0491,  ..., -0.4963, -0.3830, -0.3593],
+        [ 0.0801,  0.1193, -0.0905,  ..., -0.5380, -0.5059, -0.2944],
+        [-0.3412,  0.1521, -0.0946,  ...,  0.0952,  0.1660, -0.0902]])
+torch.Size([3, 768])
+```
+These embeddings are also in the same space as those generated by the MedCPT article encoder. 
+
+# Case 2. Semantically searching PubMed with your query
+
+We have provided the embeddings of all PubMed articles generated by the MedCPT article encoder at https://ftp.ncbi.nlm.nih.gov/pub/lu/MedCPT/pubmed_embeddings/.
+You can simply download these embeddings to search PubMed with your query.
+
+# Acknowledgments
+
+This work was supported by the Intramural Research Programs of the National Institutes of Health, National Library of Medicine.
+
+# Disclaimer
+
+This tool shows the results of research conducted in the Computational Biology Branch, NCBI/NLM. The information produced on this website is not intended for direct diagnostic use or medical decision-making without review and oversight by a clinical professional. Individuals should not change their health behavior solely on the basis of information produced on this website. NIH does not independently verify the validity or utility of the information produced by this tool. If you have questions about the information produced on this website, please see a health care professional. More information about NCBI's disclaimer policy is available.
+
+# Citation
+
+If you find this repo helpful, please cite MedCPT by:
+
+```bibtext
+@article{jin2023medcpt,
+  title={MedCPT: Contrastive Pre-trained Transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval},
+  author={Jin, Qiao and Kim, Won and Chen, Qingyu and Comeau, Donald C and Yeganova, Lana and Wilbur, W John and Lu, Zhiyong},
+  journal={Bioinformatics},
+  volume={39},
+  number={11},
+  pages={btad651},
+  year={2023},
+  publisher={Oxford University Press}
+}
+```

added_tokens.json

@@ -0,0 +1,7 @@
+{
+  "[CLS]": 2,
+  "[MASK]": 4,
+  "[PAD]": 0,
+  "[SEP]": 3,
+  "[UNK]": 1
+}

config.json

@@ -0,0 +1,25 @@
+{
+  "_name_or_path": "./",
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.34.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 30522
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19d78c0d5eaee2f81e6c47c5425bbadcc0c6af016cbb5da4a000d64e59d6e342
+size 437951328

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:debe354238bfd65f4f059d37e9c49a41c0908a2584d7a9328fc3155865a0fe9b
+size 437993382

special_tokens_map.json

@@ -0,0 +1,7 @@
+{
+  "cls_token": "[CLS]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,65 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "max_length": 512,
+  "model_max_length": 1000000000000000019884624838656,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "[PAD]",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}