swetlanas commited on
Commit
45599fb
·
verified ·
1 Parent(s): d202e1a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +32 -54
README.md CHANGED
@@ -28,7 +28,7 @@ It is tailored for multi-label [PhySH](https://physh.org/) tag recommendation fo
28
  - **Developed by:** swetlanas
29
  - **Model type:** Transformer-based sequence classification with LoRA adapter (PEFT)
30
  - **Language(s) (NLP):** English
31
- - **License:** [More Information Needed]
32
  - **Finetuned from model :** [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased)
33
 
34
  ### Model Sources [optional]
@@ -37,7 +37,7 @@ It is tailored for multi-label [PhySH](https://physh.org/) tag recommendation fo
37
 
38
  - **Repository:** [PhySH-Tank](https://github.com/swetlanas/PhySH-Tank)
39
  - **Base model :** [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased)
40
- - **Demo [optional]:** [More Information Needed]
41
 
42
  ## Uses
43
 
@@ -68,7 +68,7 @@ This model is designed to recommend Physics subject heading (PhySH) tags for APS
68
 
69
  Use the code snippet below to run inference on a title and abstract pair:
70
 
71
- ```
72
  import torch
73
  from transformers import AutoTokenizer, AutoModelForSequenceClassification
74
  from peft import PeftModel
@@ -164,90 +164,68 @@ The model was fine-tuned using Hugging Face's `Trainer` and `peft` library with
164
 
165
  <!-- This should link to a Dataset Card if possible. -->
166
 
167
- [More Information Needed]
 
168
 
169
  #### Factors
170
 
171
  <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
172
 
173
- [More Information Needed]
 
174
 
175
  #### Metrics
176
 
177
  <!-- These are the evaluation metrics being used, ideally with a description of why. -->
178
 
179
- [More Information Needed]
 
 
180
 
181
  ### Results
182
 
183
- [More Information Needed]
 
 
 
 
184
 
185
  #### Summary
186
 
 
187
 
188
 
189
- ## Model Examination [optional]
190
-
191
- <!-- Relevant interpretability work for the model goes here -->
192
-
193
- [More Information Needed]
194
-
195
- ## Environmental Impact
196
-
197
- <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
198
-
199
- Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
200
-
201
- - **Hardware Type:** RTX 3060 12GB graphics card
202
- - **Hours used:** [More Information Needed]
203
- - **Cloud Provider:** [More Information Needed]
204
- - **Compute Region:** [More Information Needed]
205
- - **Carbon Emitted:** PyTorch, Hugging Face Transformers, PEFT
206
-
207
  ## Technical Specifications [optional]
208
 
209
  ### Model Architecture and Objective
210
 
211
- [More Information Needed]
 
 
212
 
213
  ### Compute Infrastructure
214
 
215
- [More Information Needed]
216
-
217
  #### Hardware
218
-
219
- [More Information Needed]
220
 
221
  #### Software
 
222
 
223
- [More Information Needed]
224
-
225
- ## Citation [optional]
226
 
227
  <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
228
 
229
  **BibTeX:**
230
 
231
- [More Information Needed]
232
-
233
- **APA:**
234
-
235
- [More Information Needed]
236
-
237
- ## Glossary [optional]
238
-
239
- <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
240
-
241
- [More Information Needed]
242
-
243
- ## More Information [optional]
244
-
245
- [More Information Needed]
246
-
247
- ## Model Card Authors [optional]
248
-
249
- [More Information Needed]
250
 
251
- ## Model Card Contact
 
252
 
253
- [More Information Needed]
 
28
  - **Developed by:** swetlanas
29
  - **Model type:** Transformer-based sequence classification with LoRA adapter (PEFT)
30
  - **Language(s) (NLP):** English
31
+ - **License:** apache-2.0
32
  - **Finetuned from model :** [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased)
33
 
34
  ### Model Sources [optional]
 
37
 
38
  - **Repository:** [PhySH-Tank](https://github.com/swetlanas/PhySH-Tank)
39
  - **Base model :** [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased)
40
+ - **Demo :** [Coming soon]
41
 
42
  ## Uses
43
 
 
68
 
69
  Use the code snippet below to run inference on a title and abstract pair:
70
 
71
+ ```python
72
  import torch
73
  from transformers import AutoTokenizer, AutoModelForSequenceClassification
74
  from peft import PeftModel
 
164
 
165
  <!-- This should link to a Dataset Card if possible. -->
166
 
167
+ Held-out test split from the ~50,000 APS Physical Review B abstracts (2016–2026), using Hugging Face Dataset `dataset.train_test_split(test_size=0.2, seed=0)` for consistency across all model tracks.
168
+
169
 
170
  #### Factors
171
 
172
  <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
173
 
174
+ Evaluation was run across the full label space (~3,000 PhySH tags) with no per-subfield breakdown; performance is expected to be stronger on condensed matter / materials science / solid-state topics, which dominate the training distribution.
175
+
176
 
177
  #### Metrics
178
 
179
  <!-- These are the evaluation metrics being used, ideally with a description of why. -->
180
 
181
+ Precision@5 (P@5) was used as the primary metric and the early-stopping / best-checkpoint criterion, since the task is to recommend the top 5 most relevant tags per paper.
182
+ Recall@5, NDCG@5, and Hit@5 were also tracked via a custom `compute_metrics` function in the HuggingFace `Trainer`.
183
+
184
 
185
  ### Results
186
 
187
+ | Model | Precision@5 |
188
+ |---|---|
189
+ | TF-IDF + MultinomialNB (baseline) | 0.332 |
190
+ | napkinXC (PLT) + frozen PhysBERT embeddings | 0.371 |
191
+ | **LoRA fine-tuned PhysBERT (this model)** | **0.408** |
192
 
193
  #### Summary
194
 
195
+ LoRA fine-tuning of PhysBERT outperforms both a classical TF-IDF baseline and a tree-based extreme multi-label classifier (napkinXC) using frozen PhysBERT embeddings, while updating only ~2.6M parameters (~2.27% of the base model's total).
196
 
197
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
198
  ## Technical Specifications [optional]
199
 
200
  ### Model Architecture and Objective
201
 
202
+ **Base architecture**: PhysBERT [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased), a BERT-style transformer encoder pretrained on physics (arXiv) text, with a sequence classification head for multi-label PhySH tag prediction.
203
+ A LoRA (Low-Rank Adaptation) adapter (r=8, α=16, dropout=0.1) was applied to the `query` and `value` attention projection matrices, with the classification head fully fine-tuned. This updates ~2.6M parameters (~2.27% of the base model's total).
204
+ The training objective is multi-label binary classification: each of the ~3,000 PhySH tags is treated as an independent binary target, optimized with `BCEWithLogitsLoss` (Binary Cross-Entropy with logits) over sigmoid outputs, rather than a single softmax over mutually exclusive classes.
205
 
206
  ### Compute Infrastructure
207
 
 
 
208
  #### Hardware
209
+ RTX 3060 12GB GPU
 
210
 
211
  #### Software
212
+ PyTorch, Hugging Face Transformers, PEFT
213
 
214
+ ## Citation
 
 
215
 
216
  <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
217
 
218
  **BibTeX:**
219
 
220
+ ```bibtex
221
+ @misc{physh-tank,
222
+ author = {Swetlana S},
223
+ title = {PhySH-Tank: PhySH Tag Recommendation for PRB Abstracts},
224
+ year = {2026},
225
+ howpublished = {\url{https://github.com/swetlanas/PhySH-Tank}}
226
+ }
227
+ ```
 
 
 
 
 
 
 
 
 
 
 
228
 
229
+ ## Model Card Authors
230
+ [swetlanas](https://huggingface.co/swetlanas)
231