---
license: apache-2.0
base_model:
- allenai/specter2_base
pipeline_tag: text-classification
library_name: transformers
---

# Technical Domain Classifier

This model is a fine-tuned version of **allenai/specter2_base** for multilabel scientific domain classification aligned with **ERC panel taxonomy**.  
It achieves the following results on the held-out test set:

- **Best validation loss:** 0.0372  
- **Micro F1:** 0.74
  
---

## Model description

This model is a fine-tuned variant of **SPECTER2** (`allenai/specter2_base`) adapted for **multilabel classification of scientific documents** into S3 technical domains.

The model takes as input the **title and abstract** of a scientific project or publication and predicts **one or more domains**.  
Since scientific outputs may legitimately span multiple domains, the model is trained using **sigmoid activation** with **binary cross-entropy loss**, allowing independent assignment of multiple labels.

### Key characteristics

- **Base model:** allenai/specter2_base  
- **Task:** multilabel document classification  
- **Labels:** 28 S3 technical domains  
- **Activation:** sigmoid (independent scores per label)  
- **Loss:** BCEWithLogitsLoss  
- **Output:** list of predicted panels with associated probabilities  
- **Decision threshold:** 0.5 (tunable)

This model enables automatic tagging aligned with the S3 technical domains structure.

---

## Intended uses & limitations

### Intended uses

This model is designed for:

- Automatic assignment of S3 technical domains

The model supports:

- title only  
- abstract only  
- **title + abstract (recommended)**

### Limitations

- Labels are derived from automatically annotated data
- Class imbalance may affect recall for underrepresented panels  
- The model does not encode explicit hierarchical relationships between panels
   
---

## How to use
## Training and evaluation data

### Training data

- Scientific documents with S3 technical domains annotations  
- Inputs:
  - title  
  - abstract  
- Task type: **multilabel classification**

### Dataset characteristics

| Property | Value |
|--------|------|
| Documents | ~16k |
| Labels | 28 panels |
| Input fields | Title, Abstract |
| Task type | Multilabel |
| License | Dataset-dependent |

---
## Training procedure

### Preprocessing

- Input text constructed as:

  `title + ". " + abstract`

- Tokenization using the SPECTER2 tokenizer  
- Maximum sequence length: **512 tokens**

### Model

- Base model: `allenai/specter2_base`  
- Classification head: linear → sigmoid  
- Loss function: BCEWithLogitsLoss  
- Predictions: independent probability per label  

### Training hyperparameters

| Hyperparameter | Value |
|--------------|------|
| Learning rate | 5e-5 |
| Train batch size | 4 |
| Eval batch size | 8 |
| Epochs | 5 |
| Weight decay | 0.01 |
| Optimizer | AdamW |
| Metric for best model | Micro F1 |

---
## Training results

| Epoch | Training Loss | Duration seconds |
|------|---------------|-----------------|
| 1 | 0.1618 | 234.58 | 
| 2 | 0.0857 | 234.35 | 
| 3 | 0.0575 | 234.92 | 
| 4 | 0.0365 | 235.46 | 
| 5 | 0.0217 | 236.22 | 


## Evaluation results (multilabel test set)
| Panel | Precision | Recall | F1-score | Support |
|------|-----------|--------|----------|---------|
| 2D materials | 0.90 | 0.65 | 0.75 | 71 |
| 3D printing materials | 0.89 | 0.50 | 0.64 | 48 |
| Additive manufacturing | 0.69 | 0.75 | 0.72 | 79 |
| Artificial Intelligence and machine learning | 0.55 | 0.90 | 0.69 | 176 |
| Big-data and analytics | 0.66 | 0.67 | 0.66 | 218 |
| Biomaterials, biosensors, bioactuators and bioactivators | 0.76 | 0.78 | 0.77 | 117 |
| Biomedical, pharmaceutical, pharmacological technologies | 0.90 | 0.78 | 0.84 | 278 |
| Biorobotics | 0.92 | 0.75 | 0.82 | 59 |
| Biotechnology for health | 0.55 | 0.93 | 0.69 | 142 |
| Business models and digital creativity | 0.60 | 0.74 | 0.66 | 323 |
| Cyber connectivity and 5G | 0.77 | 0.79 | 0.78 | 119 |
| Cyber-security, blockchain | 0.88 | 0.92 | 0.90 | 120 |
| Development of energy-saving solutions | 0.63 | 0.76 | 0.69 | 143 |
| Development of production and organisational processes based on intelligent sensors/devices | 0.68 | 0.49 | 0.57 | 126 |
| Environmental resilience and pollution | 0.71 | 0.76 | 0.74 | 167 |
| Exploitation of renewable energy sources | 0.83 | 0.80 | 0.81 | 113 |
| Genomics | 0.90 | 0.86 | 0.88 | 42 |
| Green propulsion technologies | 0.95 | 0.52 | 0.67 | 69 |
| High-performance and sustainable composite materials | 0.77 | 0.64 | 0.70 | 181 |
| Industrial biotechnology | 0.52 | 0.88 | 0.65 | 58 |
| Internet of the things and services | 0.68 | 0.78 | 0.73 | 228 |
| Nanomaterials and nanotechnology | 0.82 | 0.89 | 0.85 | 266 |
| Neuroscience | 0.72 | 0.93 | 0.81 | 67 |
| Nutraceutics | 0.36 | 0.94 | 0.52 | 17 |
| Photonic, micro and nanoelectronic solutions | 0.79 | 0.88 | 0.84 | 259 |
| Robotic, mechatronic and automation solutions | 0.42 | 0.94 | 0.58 | 144 |
| Technologies for decarbonisation | 0.74 | 0.86 | 0.80 | 225 |
| Technologies for the circular economy and bioeconomy | 0.87 | 0.76 | 0.81 | 123 |

**Overall performance**

|  | Precision | Recall | F1-score | Support |
|------|-----------|--------|----------|---------|
| **Micro avg** | **0.70** | **0.79** | **0.74** | **2077** |
| **Macro avg** | **0.73** | **0.78** | **0.73** | **2077** |
  
---

## Framework versions

- **Transformers:** 4.57.x  
- **PyTorch:** 2.8.0  
- **Datasets:** 3.x  
- **Tokenizers:** 0.22.x