Instructions to use swetlanas/physbert-tag-recommender with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use swetlanas/physbert-tag-recommender with PEFT:
from peft import PeftModel from transformers import AutoModelForSequenceClassification base_model = AutoModelForSequenceClassification.from_pretrained("thellert/physbert_uncased") model = PeftModel.from_pretrained(base_model, "swetlanas/physbert-tag-recommender") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -28,7 +28,7 @@ It is tailored for multi-label [PhySH](https://physh.org/) tag recommendation fo
|
|
| 28 |
- **Developed by:** swetlanas
|
| 29 |
- **Model type:** Transformer-based sequence classification with LoRA adapter (PEFT)
|
| 30 |
- **Language(s) (NLP):** English
|
| 31 |
-
- **License:**
|
| 32 |
- **Finetuned from model :** [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased)
|
| 33 |
|
| 34 |
### Model Sources [optional]
|
|
@@ -37,7 +37,7 @@ It is tailored for multi-label [PhySH](https://physh.org/) tag recommendation fo
|
|
| 37 |
|
| 38 |
- **Repository:** [PhySH-Tank](https://github.com/swetlanas/PhySH-Tank)
|
| 39 |
- **Base model :** [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased)
|
| 40 |
-
- **Demo
|
| 41 |
|
| 42 |
## Uses
|
| 43 |
|
|
@@ -68,7 +68,7 @@ This model is designed to recommend Physics subject heading (PhySH) tags for APS
|
|
| 68 |
|
| 69 |
Use the code snippet below to run inference on a title and abstract pair:
|
| 70 |
|
| 71 |
-
```
|
| 72 |
import torch
|
| 73 |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 74 |
from peft import PeftModel
|
|
@@ -164,90 +164,68 @@ The model was fine-tuned using Hugging Face's `Trainer` and `peft` library with
|
|
| 164 |
|
| 165 |
<!-- This should link to a Dataset Card if possible. -->
|
| 166 |
|
| 167 |
-
|
|
|
|
| 168 |
|
| 169 |
#### Factors
|
| 170 |
|
| 171 |
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
|
| 172 |
|
| 173 |
-
|
|
|
|
| 174 |
|
| 175 |
#### Metrics
|
| 176 |
|
| 177 |
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
|
| 178 |
|
| 179 |
-
|
|
|
|
|
|
|
| 180 |
|
| 181 |
### Results
|
| 182 |
|
| 183 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 184 |
|
| 185 |
#### Summary
|
| 186 |
|
|
|
|
| 187 |
|
| 188 |
|
| 189 |
-
## Model Examination [optional]
|
| 190 |
-
|
| 191 |
-
<!-- Relevant interpretability work for the model goes here -->
|
| 192 |
-
|
| 193 |
-
[More Information Needed]
|
| 194 |
-
|
| 195 |
-
## Environmental Impact
|
| 196 |
-
|
| 197 |
-
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
|
| 198 |
-
|
| 199 |
-
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
|
| 200 |
-
|
| 201 |
-
- **Hardware Type:** RTX 3060 12GB graphics card
|
| 202 |
-
- **Hours used:** [More Information Needed]
|
| 203 |
-
- **Cloud Provider:** [More Information Needed]
|
| 204 |
-
- **Compute Region:** [More Information Needed]
|
| 205 |
-
- **Carbon Emitted:** PyTorch, Hugging Face Transformers, PEFT
|
| 206 |
-
|
| 207 |
## Technical Specifications [optional]
|
| 208 |
|
| 209 |
### Model Architecture and Objective
|
| 210 |
|
| 211 |
-
|
|
|
|
|
|
|
| 212 |
|
| 213 |
### Compute Infrastructure
|
| 214 |
|
| 215 |
-
[More Information Needed]
|
| 216 |
-
|
| 217 |
#### Hardware
|
| 218 |
-
|
| 219 |
-
[More Information Needed]
|
| 220 |
|
| 221 |
#### Software
|
|
|
|
| 222 |
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
## Citation [optional]
|
| 226 |
|
| 227 |
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
| 228 |
|
| 229 |
**BibTeX:**
|
| 230 |
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
|
| 239 |
-
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
|
| 240 |
-
|
| 241 |
-
[More Information Needed]
|
| 242 |
-
|
| 243 |
-
## More Information [optional]
|
| 244 |
-
|
| 245 |
-
[More Information Needed]
|
| 246 |
-
|
| 247 |
-
## Model Card Authors [optional]
|
| 248 |
-
|
| 249 |
-
[More Information Needed]
|
| 250 |
|
| 251 |
-
## Model Card
|
|
|
|
| 252 |
|
| 253 |
-
[More Information Needed]
|
|
|
|
| 28 |
- **Developed by:** swetlanas
|
| 29 |
- **Model type:** Transformer-based sequence classification with LoRA adapter (PEFT)
|
| 30 |
- **Language(s) (NLP):** English
|
| 31 |
+
- **License:** apache-2.0
|
| 32 |
- **Finetuned from model :** [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased)
|
| 33 |
|
| 34 |
### Model Sources [optional]
|
|
|
|
| 37 |
|
| 38 |
- **Repository:** [PhySH-Tank](https://github.com/swetlanas/PhySH-Tank)
|
| 39 |
- **Base model :** [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased)
|
| 40 |
+
- **Demo :** [Coming soon]
|
| 41 |
|
| 42 |
## Uses
|
| 43 |
|
|
|
|
| 68 |
|
| 69 |
Use the code snippet below to run inference on a title and abstract pair:
|
| 70 |
|
| 71 |
+
```python
|
| 72 |
import torch
|
| 73 |
from transformers import AutoTokenizer, AutoModelForSequenceClassification
|
| 74 |
from peft import PeftModel
|
|
|
|
| 164 |
|
| 165 |
<!-- This should link to a Dataset Card if possible. -->
|
| 166 |
|
| 167 |
+
Held-out test split from the ~50,000 APS Physical Review B abstracts (2016–2026), using Hugging Face Dataset `dataset.train_test_split(test_size=0.2, seed=0)` for consistency across all model tracks.
|
| 168 |
+
|
| 169 |
|
| 170 |
#### Factors
|
| 171 |
|
| 172 |
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
|
| 173 |
|
| 174 |
+
Evaluation was run across the full label space (~3,000 PhySH tags) with no per-subfield breakdown; performance is expected to be stronger on condensed matter / materials science / solid-state topics, which dominate the training distribution.
|
| 175 |
+
|
| 176 |
|
| 177 |
#### Metrics
|
| 178 |
|
| 179 |
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
|
| 180 |
|
| 181 |
+
Precision@5 (P@5) was used as the primary metric and the early-stopping / best-checkpoint criterion, since the task is to recommend the top 5 most relevant tags per paper.
|
| 182 |
+
Recall@5, NDCG@5, and Hit@5 were also tracked via a custom `compute_metrics` function in the HuggingFace `Trainer`.
|
| 183 |
+
|
| 184 |
|
| 185 |
### Results
|
| 186 |
|
| 187 |
+
| Model | Precision@5 |
|
| 188 |
+
|---|---|
|
| 189 |
+
| TF-IDF + MultinomialNB (baseline) | 0.332 |
|
| 190 |
+
| napkinXC (PLT) + frozen PhysBERT embeddings | 0.371 |
|
| 191 |
+
| **LoRA fine-tuned PhysBERT (this model)** | **0.408** |
|
| 192 |
|
| 193 |
#### Summary
|
| 194 |
|
| 195 |
+
LoRA fine-tuning of PhysBERT outperforms both a classical TF-IDF baseline and a tree-based extreme multi-label classifier (napkinXC) using frozen PhysBERT embeddings, while updating only ~2.6M parameters (~2.27% of the base model's total).
|
| 196 |
|
| 197 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 198 |
## Technical Specifications [optional]
|
| 199 |
|
| 200 |
### Model Architecture and Objective
|
| 201 |
|
| 202 |
+
**Base architecture**: PhysBERT [thellert/physbert_uncased](https://huggingface.co/thellert/physbert_uncased), a BERT-style transformer encoder pretrained on physics (arXiv) text, with a sequence classification head for multi-label PhySH tag prediction.
|
| 203 |
+
A LoRA (Low-Rank Adaptation) adapter (r=8, α=16, dropout=0.1) was applied to the `query` and `value` attention projection matrices, with the classification head fully fine-tuned. This updates ~2.6M parameters (~2.27% of the base model's total).
|
| 204 |
+
The training objective is multi-label binary classification: each of the ~3,000 PhySH tags is treated as an independent binary target, optimized with `BCEWithLogitsLoss` (Binary Cross-Entropy with logits) over sigmoid outputs, rather than a single softmax over mutually exclusive classes.
|
| 205 |
|
| 206 |
### Compute Infrastructure
|
| 207 |
|
|
|
|
|
|
|
| 208 |
#### Hardware
|
| 209 |
+
RTX 3060 12GB GPU
|
|
|
|
| 210 |
|
| 211 |
#### Software
|
| 212 |
+
PyTorch, Hugging Face Transformers, PEFT
|
| 213 |
|
| 214 |
+
## Citation
|
|
|
|
|
|
|
| 215 |
|
| 216 |
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
| 217 |
|
| 218 |
**BibTeX:**
|
| 219 |
|
| 220 |
+
```bibtex
|
| 221 |
+
@misc{physh-tank,
|
| 222 |
+
author = {Swetlana S},
|
| 223 |
+
title = {PhySH-Tank: PhySH Tag Recommendation for PRB Abstracts},
|
| 224 |
+
year = {2026},
|
| 225 |
+
howpublished = {\url{https://github.com/swetlanas/PhySH-Tank}}
|
| 226 |
+
}
|
| 227 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 228 |
|
| 229 |
+
## Model Card Authors
|
| 230 |
+
[swetlanas](https://huggingface.co/swetlanas)
|
| 231 |
|
|
|