Hugging Face's logo

SIRIS-Lab
/
Toscana-TechDomain-classifier

Text Classification
Transformers
Safetensors
bert
text-embeddings-inference
Model card Files
xet
 Community
KseniiaY commited on
Commit
70bd4f0
·
verified ·
1 Parent(s): 0c22615

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +130 -0
README.md CHANGED
@@ -22,3 +22,133 @@ This model is a fine-tuned variant of **SPECTER2** (`allenai/specter2_base`) ada
22
 
23
  The model takes as input the **title and abstract** of a scientific project or publication and predicts **one or more domains**.
24
  Since scientific outputs may legitimately span multiple domains, the model is trained using **sigmoid activation** with **binary cross-entropy loss**, allowing independent assignment of multiple labels.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
22
 
23
  The model takes as input the **title and abstract** of a scientific project or publication and predicts **one or more domains**.
24
  Since scientific outputs may legitimately span multiple domains, the model is trained using **sigmoid activation** with **binary cross-entropy loss**, allowing independent assignment of multiple labels.
25
+
26
+ ### Key characteristics
27
+
28
+ - **Base model:** allenai/specter2_base
29
+ - **Task:** multilabel document classification
30
+ - **Labels:** 28 S3 technical domains
31
+ - **Activation:** sigmoid (independent scores per label)
32
+ - **Loss:** BCEWithLogitsLoss
33
+ - **Output:** list of predicted panels with associated probabilities
34
+ - **Decision threshold:** 0.5 (tunable)
35
+
36
+ This model enables automatic tagging aligned with the S3 technical domains structure.
37
+
38
+ ---
39
+
40
+ ## Intended uses & limitations
41
+
42
+ ### Intended uses
43
+
44
+ This model is designed for:
45
+
46
+ - Automatic assignment of S3 technical domains
47
+
48
+ The model supports:
49
+
50
+ - title only
51
+ - abstract only
52
+ - **title + abstract (recommended)**
53
+
54
+ ### Limitations
55
+
56
+ - Labels are derived from automatically annotated data
57
+ - Class imbalance may affect recall for underrepresented panels
58
+ - The model does not encode explicit hierarchical relationships between panels
59
+
60
+ ---
61
+
62
+ ## How to use
63
+ ## Training and evaluation data
64
+
65
+ ### Training data
66
+
67
+ - Scientific documents with S3 technical domains annotations
68
+ - Inputs:
69
+ - title
70
+ - abstract
71
+ - Task type: **multilabel classification**
72
+
73
+ ### Dataset characteristics
74
+
75
+ | Property | Value |
76
+ |--------|------|
77
+ | Documents | ~16k |
78
+ | Labels | 28 panels |
79
+ | Input fields | Title, Abstract |
80
+ | Task type | Multilabel |
81
+ | License | Dataset-dependent |
82
+
83
+ ---
84
+ ## Training procedure
85
+
86
+ ### Preprocessing
87
+
88
+ - Input text constructed as:
89
+
90
+ `title + ". " + abstract`
91
+
92
+ - Tokenization using the SPECTER2 tokenizer
93
+ - Maximum sequence length: **512 tokens**
94
+
95
+ ### Model
96
+
97
+ - Base model: `allenai/specter2_base`
98
+ - Classification head: linear → sigmoid
99
+ - Loss function: BCEWithLogitsLoss
100
+ - Predictions: independent probability per label
101
+
102
+ ### Training hyperparameters
103
+
104
+ | Hyperparameter | Value |
105
+ |--------------|------|
106
+ | Learning rate | 5e-5 |
107
+ | Train batch size | 4 |
108
+ | Eval batch size | 8 |
109
+ | Epochs | 5 |
110
+ | Weight decay | 0.01 |
111
+ | Optimizer | AdamW |
112
+ | Metric for best model | Micro F1 |
113
+
114
+ ---
115
+ ## Training results
116
+
117
+ ## Evaluation results (multilabel test set)
118
+ | Panel | Precision | Recall | F1-score | Support |
119
+ |------|-----------|--------|----------|---------|
120
+ | 2D materials | 0.90 | 0.65 | 0.75 | 71 |
121
+ | 3D printing materials | 0.89 | 0.50 | 0.64 | 48 |
122
+ | Additive manufacturing | 0.69 | 0.75 | 0.72 | 79 |
123
+ | Artificial Intelligence and machine learning | 0.55 | 0.90 | 0.69 | 176 |
124
+ | Big-data and analytics | 0.66 | 0.67 | 0.66 | 218 |
125
+ | Biomaterials, biosensors, bioactuators and bioactivators | 0.76 | 0.78 | 0.77 | 117 |
126
+ | Biomedical, pharmaceutical, pharmacological technologies | 0.90 | 0.78 | 0.84 | 278 |
127
+ | Biorobotics | 0.92 | 0.75 | 0.82 | 59 |
128
+ | Biotechnology for health | 0.55 | 0.93 | 0.69 | 142 |
129
+ | Business models and digital creativity | 0.60 | 0.74 | 0.66 | 323 |
130
+ | Cyber connectivity and 5G | 0.77 | 0.79 | 0.78 | 119 |
131
+ | Cyber-security, blockchain | 0.88 | 0.92 | 0.90 | 120 |
132
+ | Development of energy-saving solutions | 0.63 | 0.76 | 0.69 | 143 |
133
+ | Development of production and organisational processes based on intelligent sensors/devices | 0.68 | 0.49 | 0.57 | 126 |
134
+ | Environmental resilience and pollution | 0.71 | 0.76 | 0.74 | 167 |
135
+ | Exploitation of renewable energy sources | 0.83 | 0.80 | 0.81 | 113 |
136
+ | Genomics | 0.90 | 0.86 | 0.88 | 42 |
137
+ | Green propulsion technologies | 0.95 | 0.52 | 0.67 | 69 |
138
+ | High-performance and sustainable composite materials | 0.77 | 0.64 | 0.70 | 181 |
139
+ | Industrial biotechnology | 0.52 | 0.88 | 0.65 | 58 |
140
+ | Internet of the things and services | 0.68 | 0.78 | 0.73 | 228 |
141
+ | Nanomaterials and nanotechnology | 0.82 | 0.89 | 0.85 | 266 |
142
+ | Neuroscience | 0.72 | 0.93 | 0.81 | 67 |
143
+ | Nutraceutics | 0.36 | 0.94 | 0.52 | 17 |
144
+ | Photonic, micro and nanoelectronic solutions | 0.79 | 0.88 | 0.84 | 259 |
145
+ | Robotic, mechatronic and automation solutions | 0.42 | 0.94 | 0.58 | 144 |
146
+ | Technologies for decarbonisation | 0.74 | 0.86 | 0.80 | 225 |
147
+ | Technologies for the circular economy and bioeconomy | 0.87 | 0.76 | 0.81 | 123 |
148
+
149
+ ## Framework versions
150
+
151
+ - **Transformers:** 4.57.x
152
+ - **PyTorch:** 2.8.0
153
+ - **Datasets:** 3.x
154
+ - **Tokenizers:** 0.22.x