File size: 5,501 Bytes
0c22615 70bd4f0 e49e4d9 70bd4f0 def108d 70bd4f0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 | ---
license: apache-2.0
base_model:
- allenai/specter2_base
pipeline_tag: text-classification
library_name: transformers
---
# Technical Domain Classifier
This model is a fine-tuned version of **allenai/specter2_base** for multilabel scientific domain classification aligned with **ERC panel taxonomy**.
It achieves the following results on the held-out test set:
- **Best validation loss:** 0.0372
- **Micro F1:** 0.74
---
## Model description
This model is a fine-tuned variant of **SPECTER2** (`allenai/specter2_base`) adapted for **multilabel classification of scientific documents** into S3 technical domains.
The model takes as input the **title and abstract** of a scientific project or publication and predicts **one or more domains**.
Since scientific outputs may legitimately span multiple domains, the model is trained using **sigmoid activation** with **binary cross-entropy loss**, allowing independent assignment of multiple labels.
### Key characteristics
- **Base model:** allenai/specter2_base
- **Task:** multilabel document classification
- **Labels:** 28 S3 technical domains
- **Activation:** sigmoid (independent scores per label)
- **Loss:** BCEWithLogitsLoss
- **Output:** list of predicted panels with associated probabilities
- **Decision threshold:** 0.5 (tunable)
This model enables automatic tagging aligned with the S3 technical domains structure.
---
## Intended uses & limitations
### Intended uses
This model is designed for:
- Automatic assignment of S3 technical domains
The model supports:
- title only
- abstract only
- **title + abstract (recommended)**
### Limitations
- Labels are derived from automatically annotated data
- Class imbalance may affect recall for underrepresented panels
- The model does not encode explicit hierarchical relationships between panels
---
## How to use
## Training and evaluation data
### Training data
- Scientific documents with S3 technical domains annotations
- Inputs:
- title
- abstract
- Task type: **multilabel classification**
### Dataset characteristics
| Property | Value |
|--------|------|
| Documents | ~16k |
| Labels | 28 panels |
| Input fields | Title, Abstract |
| Task type | Multilabel |
| License | Dataset-dependent |
---
## Training procedure
### Preprocessing
- Input text constructed as:
`title + ". " + abstract`
- Tokenization using the SPECTER2 tokenizer
- Maximum sequence length: **512 tokens**
### Model
- Base model: `allenai/specter2_base`
- Classification head: linear → sigmoid
- Loss function: BCEWithLogitsLoss
- Predictions: independent probability per label
### Training hyperparameters
| Hyperparameter | Value |
|--------------|------|
| Learning rate | 5e-5 |
| Train batch size | 4 |
| Eval batch size | 8 |
| Epochs | 5 |
| Weight decay | 0.01 |
| Optimizer | AdamW |
| Metric for best model | Micro F1 |
---
## Training results
| Epoch | Training Loss | Duration seconds |
|------|---------------|-----------------|
| 1 | 0.1618 | 234.58 |
| 2 | 0.0857 | 234.35 |
| 3 | 0.0575 | 234.92 |
| 4 | 0.0365 | 235.46 |
| 5 | 0.0217 | 236.22 |
## Evaluation results (multilabel test set)
| Panel | Precision | Recall | F1-score | Support |
|------|-----------|--------|----------|---------|
| 2D materials | 0.90 | 0.65 | 0.75 | 71 |
| 3D printing materials | 0.89 | 0.50 | 0.64 | 48 |
| Additive manufacturing | 0.69 | 0.75 | 0.72 | 79 |
| Artificial Intelligence and machine learning | 0.55 | 0.90 | 0.69 | 176 |
| Big-data and analytics | 0.66 | 0.67 | 0.66 | 218 |
| Biomaterials, biosensors, bioactuators and bioactivators | 0.76 | 0.78 | 0.77 | 117 |
| Biomedical, pharmaceutical, pharmacological technologies | 0.90 | 0.78 | 0.84 | 278 |
| Biorobotics | 0.92 | 0.75 | 0.82 | 59 |
| Biotechnology for health | 0.55 | 0.93 | 0.69 | 142 |
| Business models and digital creativity | 0.60 | 0.74 | 0.66 | 323 |
| Cyber connectivity and 5G | 0.77 | 0.79 | 0.78 | 119 |
| Cyber-security, blockchain | 0.88 | 0.92 | 0.90 | 120 |
| Development of energy-saving solutions | 0.63 | 0.76 | 0.69 | 143 |
| Development of production and organisational processes based on intelligent sensors/devices | 0.68 | 0.49 | 0.57 | 126 |
| Environmental resilience and pollution | 0.71 | 0.76 | 0.74 | 167 |
| Exploitation of renewable energy sources | 0.83 | 0.80 | 0.81 | 113 |
| Genomics | 0.90 | 0.86 | 0.88 | 42 |
| Green propulsion technologies | 0.95 | 0.52 | 0.67 | 69 |
| High-performance and sustainable composite materials | 0.77 | 0.64 | 0.70 | 181 |
| Industrial biotechnology | 0.52 | 0.88 | 0.65 | 58 |
| Internet of the things and services | 0.68 | 0.78 | 0.73 | 228 |
| Nanomaterials and nanotechnology | 0.82 | 0.89 | 0.85 | 266 |
| Neuroscience | 0.72 | 0.93 | 0.81 | 67 |
| Nutraceutics | 0.36 | 0.94 | 0.52 | 17 |
| Photonic, micro and nanoelectronic solutions | 0.79 | 0.88 | 0.84 | 259 |
| Robotic, mechatronic and automation solutions | 0.42 | 0.94 | 0.58 | 144 |
| Technologies for decarbonisation | 0.74 | 0.86 | 0.80 | 225 |
| Technologies for the circular economy and bioeconomy | 0.87 | 0.76 | 0.81 | 123 |
**Overall performance**
| | Precision | Recall | F1-score | Support |
|------|-----------|--------|----------|---------|
| **Micro avg** | **0.70** | **0.79** | **0.74** | **2077** |
| **Macro avg** | **0.73** | **0.78** | **0.73** | **2077** |
---
## Framework versions
- **Transformers:** 4.57.x
- **PyTorch:** 2.8.0
- **Datasets:** 3.x
- **Tokenizers:** 0.22.x |