NetworkIsLife commited on
Commit
f213e70
·
verified ·
1 Parent(s): 392fe04

Add model.safetensors converted from pytorch_adapter.bin

Browse files
Files changed (5) hide show
  1. .gitattributes +0 -1
  2. README.md +202 -0
  3. adapter_config.json +40 -0
  4. model.safetensors +3 -0
  5. pytorch_adapter.bin +3 -0
.gitattributes CHANGED
@@ -25,7 +25,6 @@
25
  *.safetensors filter=lfs diff=lfs merge=lfs -text
26
  saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
  *.tar.* filter=lfs diff=lfs merge=lfs -text
28
- *.tar filter=lfs diff=lfs merge=lfs -text
29
  *.tflite filter=lfs diff=lfs merge=lfs -text
30
  *.tgz filter=lfs diff=lfs merge=lfs -text
31
  *.wasm filter=lfs diff=lfs merge=lfs -text
 
25
  *.safetensors filter=lfs diff=lfs merge=lfs -text
26
  saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
  *.tar.* filter=lfs diff=lfs merge=lfs -text
 
28
  *.tflite filter=lfs diff=lfs merge=lfs -text
29
  *.tgz filter=lfs diff=lfs merge=lfs -text
30
  *.wasm filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - adapter-transformers
4
+ - bert
5
+ datasets:
6
+ - allenai/scirepeval
7
+ ---
8
+
9
+ ## SPECTER2
10
+
11
+ <!-- Provide a quick summary of what the model is/does. -->
12
+
13
+ SPECTER2 is a family of models that succeeds [SPECTER](https://huggingface.co/allenai/specter) and is capable of generating task specific embeddings for scientific tasks when paired with [adapters](https://huggingface.co/models?search=allenai/specter-2_).
14
+ Given the combination of title and abstract of a scientific paper or a short texual query, the model can be used to generate effective embeddings to be used in downstream applications.
15
+
16
+ **Note:For general embedding purposes, please use [allenai/specter2](https://huggingface.co/allenai/specter2).**
17
+
18
+ **To get the best performance on a downstream task type please load the associated adapter () with the base model as in the example below.**
19
+
20
+ **Dec 2023 Update:**
21
+
22
+ Model usage updated to be compatible with latest versions of transformers and adapters (newly released update to adapter-transformers) libraries.
23
+
24
+ **Aug 2023 Update:**
25
+ 1. **The SPECTER2 Base and proximity adapter models have been renamed in Hugging Face based upon usage patterns as follows:**
26
+
27
+ |Old Name|New Name|
28
+ |--|--|
29
+ |allenai/specter2|[allenai/specter2_base](https://huggingface.co/allenai/specter2_base)|
30
+ |allenai/specter2_proximity|[allenai/specter2](https://huggingface.co/allenai/specter2)|
31
+
32
+ 2. **We have a parallel version (termed [aug2023refresh](https://huggingface.co/allenai/specter2_aug2023refresh)) where the base transformer encoder version is pre-trained on a collection of newer papers (published after 2018).
33
+ However, for benchmarking purposes, please continue using the current version.**
34
+
35
+ # Adapter `allenai/specter2` for allenai/specter2_base
36
+
37
+ An [adapter](https://adapterhub.ml) for the [allenai/specter2_base](https://huggingface.co/allenai/specter2_base) model that was trained on the [allenai/scirepeval](https://huggingface.co/datasets/allenai/scirepeval/) dataset.
38
+
39
+ This adapter was created for usage with the **[adapters](https://github.com/adapter-hub/adapters)** library.
40
+
41
+
42
+ ## Adapter Usage
43
+
44
+ First, install `adapters`:
45
+
46
+ ```
47
+ pip install -U adapters
48
+ ```
49
+ _Note: adapters is built as an add on to transformers and acts as a drop-in replacement with adapter support. [More](https://docs.adapterhub.ml/installation.html)_
50
+
51
+ Now, the adapter can be loaded and activated like this:
52
+
53
+ ```python
54
+ from adapters import AutoAdapterModel
55
+
56
+ model = AutoAdapterModel.from_pretrained("allenai/specter2_base")
57
+ adapter_name = model.load_adapter("allenai/specter2", source="hf", set_active=True)
58
+ ```
59
+
60
+
61
+ # Model Details
62
+
63
+ ## Model Description
64
+
65
+ SPECTER2 has been trained on over 6M triplets of scientific paper citations, which are available [here](https://huggingface.co/datasets/allenai/scirepeval/viewer/cite_prediction_new/evaluation).
66
+ Post that it is trained with additionally attached task format specific adapter modules on all the [SciRepEval](https://huggingface.co/datasets/allenai/scirepeval) training tasks.
67
+
68
+ Task Formats trained on:
69
+ - Classification
70
+ - Regression
71
+ - Proximity (Retrieval)
72
+ - Adhoc Search
73
+
74
+ This is a retrieval specific adapter. For tasks where given a paper query, other relevant papers have to be retrieved from a corpus, use this adapter to generate the embeddings.
75
+
76
+
77
+ It builds on the work done in [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://api.semanticscholar.org/CorpusID:254018137) and we evaluate the trained model on this benchmark as well.
78
+
79
+
80
+
81
+ - **Developed by:** Amanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey, Sergey Feldman
82
+ - **Shared by :** Allen AI
83
+ - **Model type:** bert-base-uncased + adapters
84
+ - **License:** Apache 2.0
85
+ - **Finetuned from model:** [allenai/scibert](https://huggingface.co/allenai/scibert_scivocab_uncased).
86
+
87
+ ## Model Sources
88
+
89
+ <!-- Provide the basic links for the model. -->
90
+
91
+ - **Repository:** [https://github.com/allenai/SPECTER2](https://github.com/allenai/SPECTER2)
92
+ - **Paper:** [https://api.semanticscholar.org/CorpusID:254018137](https://api.semanticscholar.org/CorpusID:254018137)
93
+ - **Demo:** [Usage](https://github.com/allenai/SPECTER2_0/blob/main/README.md)
94
+
95
+ # Uses
96
+
97
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
98
+
99
+ ## Direct Use
100
+
101
+ |Model|Name and HF link|Description|
102
+ |--|--|--|
103
+ |Proximity*|[allenai/specter2](https://huggingface.co/allenai/specter2)|Encode papers as queries and candidates eg. Link Prediction, Nearest Neighbor Search|
104
+ |Adhoc Query|[allenai/specter2_adhoc_query](https://huggingface.co/allenai/specter2_adhoc_query)|Encode short raw text queries for search tasks. (Candidate papers can be encoded with the proximity adapter)|
105
+ |Classification|[allenai/specter2_classification](https://huggingface.co/allenai/specter2_classification)|Encode papers to feed into linear classifiers as features|
106
+ |Regression|[allenai/specter2_regression](https://huggingface.co/allenai/specter2_regression)|Encode papers to feed into linear regressors as features|
107
+
108
+ *Proximity model should suffice for downstream task types not mentioned above
109
+
110
+
111
+ ```python
112
+ from transformers import AutoTokenizer
113
+ from adapters import AutoAdapterModel
114
+
115
+ # load model and tokenizer
116
+ tokenizer = AutoTokenizer.from_pretrained('allenai/specter2_base')
117
+
118
+ #load base model
119
+ model = AutoAdapterModel.from_pretrained('allenai/specter2_base')
120
+
121
+ #load the adapter(s) as per the required task, provide an identifier for the adapter in load_as argument and activate it
122
+ model.load_adapter("allenai/specter2", source="hf", load_as="specter2", set_active=True)
123
+
124
+ papers = [{'title': 'BERT', 'abstract': 'We introduce a new language representation model called BERT'},
125
+ {'title': 'Attention is all you need', 'abstract': ' The dominant sequence transduction models are based on complex recurrent or convolutional neural networks'}]
126
+
127
+ # concatenate title and abstract
128
+ text_batch = [d['title'] + tokenizer.sep_token + (d.get('abstract') or '') for d in papers]
129
+ # preprocess the input
130
+ inputs = self.tokenizer(text_batch, padding=True, truncation=True,
131
+ return_tensors="pt", return_token_type_ids=False, max_length=512)
132
+ output = model(**inputs)
133
+ # take the first token in the batch as the embedding
134
+ embeddings = output.last_hidden_state[:, 0, :]
135
+ ```
136
+
137
+ ## Downstream Use
138
+
139
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
140
+
141
+ For evaluation and downstream usage, please refer to [https://github.com/allenai/scirepeval/blob/main/evaluation/INFERENCE.md](https://github.com/allenai/scirepeval/blob/main/evaluation/INFERENCE.md).
142
+
143
+ # Training Details
144
+
145
+ ## Training Data
146
+
147
+ <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
148
+
149
+ The base model is trained on citation links between papers and the adapters are trained on 8 large scale tasks across the four formats.
150
+ All the data is a part of SciRepEval benchmark and is available [here](https://huggingface.co/datasets/allenai/scirepeval).
151
+
152
+ The citation link are triplets in the form
153
+
154
+ ```json
155
+ {"query": {"title": ..., "abstract": ...}, "pos": {"title": ..., "abstract": ...}, "neg": {"title": ..., "abstract": ...}}
156
+ ```
157
+
158
+ consisting of a query paper, a positive citation and a negative which can be from the same/different field of study as the query or citation of a citation.
159
+
160
+ ## Training Procedure
161
+
162
+ Please refer to the [SPECTER paper](https://api.semanticscholar.org/CorpusID:215768677).
163
+
164
+
165
+ ### Training Hyperparameters
166
+
167
+
168
+ The model is trained in two stages using [SciRepEval](https://github.com/allenai/scirepeval/blob/main/training/TRAINING.md):
169
+ - Base Model: First a base model is trained on the above citation triplets.
170
+ ``` batch size = 1024, max input length = 512, learning rate = 2e-5, epochs = 2 warmup steps = 10% fp16```
171
+ - Adapters: Thereafter, task format specific adapters are trained on the SciRepEval training tasks, where 600K triplets are sampled from above and added to the training data as well.
172
+ ``` batch size = 256, max input length = 512, learning rate = 1e-4, epochs = 6 warmup = 1000 steps fp16```
173
+
174
+
175
+ # Evaluation
176
+
177
+ We evaluate the model on [SciRepEval](https://github.com/allenai/scirepeval), a large scale eval benchmark for scientific embedding tasks which which has [SciDocs] as a subset.
178
+ We also evaluate and establish a new SoTA on [MDCR](https://github.com/zoranmedic/mdcr), a large scale citation recommendation benchmark.
179
+
180
+ |Model|SciRepEval In-Train|SciRepEval Out-of-Train|SciRepEval Avg|MDCR(MAP, Recall@5)|
181
+ |--|--|--|--|--|
182
+ |[BM-25](https://api.semanticscholar.org/CorpusID:252199740)|n/a|n/a|n/a|(33.7, 28.5)|
183
+ |[SPECTER](https://huggingface.co/allenai/specter)|54.7|72.0|67.5|(30.6, 25.5)|
184
+ |[SciNCL](https://huggingface.co/malteos/scincl)|55.6|73.4|68.8|(32.6, 27.3)|
185
+ |[SciRepEval-Adapters](https://huggingface.co/models?search=scirepeval)|61.9|73.8|70.7|(35.3, 29.6)|
186
+ |[SPECTER2 Base](allenai/specter2_base)|56.3|73.6|69.1|(38.0, 32.4)|
187
+ |[SPECTER2-Adapters](https://huggingface.co/models?search=allenai/specter-2)|**62.3**|**74.1**|**71.1**|**(38.4, 33.0)**|
188
+
189
+ Please cite the following works if you end up using SPECTER 2.0:
190
+
191
+ [SciRepEval paper](https://api.semanticscholar.org/CorpusID:254018137)
192
+ ```bibtex
193
+ @inproceedings{Singh2022SciRepEvalAM,
194
+ title={SciRepEval: A Multi-Format Benchmark for Scientific Document Representations},
195
+ author={Amanpreet Singh and Mike D'Arcy and Arman Cohan and Doug Downey and Sergey Feldman},
196
+ booktitle={Conference on Empirical Methods in Natural Language Processing},
197
+ year={2022},
198
+ url={https://api.semanticscholar.org/CorpusID:254018137}
199
+ }
200
+ ```
201
+
202
+
adapter_config.json ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "config": {
3
+ "adapter_residual_before_ln": false,
4
+ "cross_adapter": false,
5
+ "factorized_phm_W": true,
6
+ "factorized_phm_rule": false,
7
+ "hypercomplex_nonlinearity": "glorot-uniform",
8
+ "init_weights": "bert",
9
+ "inv_adapter": null,
10
+ "inv_adapter_reduction_factor": null,
11
+ "is_parallel": false,
12
+ "learn_phm": true,
13
+ "leave_out": [],
14
+ "ln_after": false,
15
+ "ln_before": false,
16
+ "mh_adapter": false,
17
+ "non_linearity": "relu",
18
+ "original_ln_after": true,
19
+ "original_ln_before": true,
20
+ "output_adapter": true,
21
+ "phm_bias": true,
22
+ "phm_c_init": "normal",
23
+ "phm_dim": 4,
24
+ "phm_init_range": 0.0001,
25
+ "phm_layer": false,
26
+ "phm_rank": 1,
27
+ "reduction_factor": 16,
28
+ "residual_before_ln": true,
29
+ "scaling": 1.0,
30
+ "shared_W_phm": false,
31
+ "shared_phm_rule": true,
32
+ "use_gating": false
33
+ },
34
+ "hidden_size": 768,
35
+ "model_class": "BertAdapterModel",
36
+ "model_name": "allenai/specter_plus_plus",
37
+ "model_type": "bert",
38
+ "name": "[PRX]",
39
+ "version": "3.1.0"
40
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f5b8fbdb600bd16e22b6150787cf381456ae3a018c8093accc4d52e6475d6ac2
3
+ size 3584248
pytorch_adapter.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b6105714f78b294daa7e2d181134df12f0acb7a4ec7c1e3c13e512c908db94a
3
+ size 3593365