| --- |
| license: apache-2.0 |
| base_model: |
| - allenai/specter2_base |
| pipeline_tag: text-classification |
| library_name: transformers |
| --- |
| |
| # Technical Domain Classifier |
|
|
| This model is a fine-tuned version of **allenai/specter2_base** for multilabel scientific domain classification aligned with **ERC panel taxonomy**. |
| It achieves the following results on the held-out test set: |
| |
| - **Best validation loss:** 0.0372 |
| - **Micro F1:** 0.74 |
| |
| --- |
| |
| ## Model description |
| |
| This model is a fine-tuned variant of **SPECTER2** (`allenai/specter2_base`) adapted for **multilabel classification of scientific documents** into S3 technical domains. |
|
|
| The model takes as input the **title and abstract** of a scientific project or publication and predicts **one or more domains**. |
| Since scientific outputs may legitimately span multiple domains, the model is trained using **sigmoid activation** with **binary cross-entropy loss**, allowing independent assignment of multiple labels. |
|
|
| ### Key characteristics |
|
|
| - **Base model:** allenai/specter2_base |
| - **Task:** multilabel document classification |
| - **Labels:** 28 S3 technical domains |
| - **Activation:** sigmoid (independent scores per label) |
| - **Loss:** BCEWithLogitsLoss |
| - **Output:** list of predicted panels with associated probabilities |
| - **Decision threshold:** 0.5 (tunable) |
| |
| This model enables automatic tagging aligned with the S3 technical domains structure. |
| |
| --- |
| |
| ## Intended uses & limitations |
| |
| ### Intended uses |
| |
| This model is designed for: |
| |
| - Automatic assignment of S3 technical domains |
| |
| The model supports: |
| |
| - title only |
| - abstract only |
| - **title + abstract (recommended)** |
| |
| ### Limitations |
| |
| - Labels are derived from automatically annotated data |
| - Class imbalance may affect recall for underrepresented panels |
| - The model does not encode explicit hierarchical relationships between panels |
| |
| --- |
| |
| ## How to use |
| ## Training and evaluation data |
| |
| ### Training data |
| |
| - Scientific documents with S3 technical domains annotations |
| - Inputs: |
| - title |
| - abstract |
| - Task type: **multilabel classification** |
| |
| ### Dataset characteristics |
| |
| | Property | Value | |
| |--------|------| |
| | Documents | ~16k | |
| | Labels | 28 panels | |
| | Input fields | Title, Abstract | |
| | Task type | Multilabel | |
| | License | Dataset-dependent | |
| |
| --- |
| ## Training procedure |
| |
| ### Preprocessing |
| |
| - Input text constructed as: |
| |
| `title + ". " + abstract` |
| |
| - Tokenization using the SPECTER2 tokenizer |
| - Maximum sequence length: **512 tokens** |
| |
| ### Model |
| |
| - Base model: `allenai/specter2_base` |
| - Classification head: linear → sigmoid |
| - Loss function: BCEWithLogitsLoss |
| - Predictions: independent probability per label |
|
|
| ### Training hyperparameters |
|
|
| | Hyperparameter | Value | |
| |--------------|------| |
| | Learning rate | 5e-5 | |
| | Train batch size | 4 | |
| | Eval batch size | 8 | |
| | Epochs | 5 | |
| | Weight decay | 0.01 | |
| | Optimizer | AdamW | |
| | Metric for best model | Micro F1 | |
|
|
| --- |
| ## Training results |
|
|
| | Epoch | Training Loss | Duration seconds | |
| |------|---------------|-----------------| |
| | 1 | 0.1618 | 234.58 | |
| | 2 | 0.0857 | 234.35 | |
| | 3 | 0.0575 | 234.92 | |
| | 4 | 0.0365 | 235.46 | |
| | 5 | 0.0217 | 236.22 | |
|
|
|
|
| ## Evaluation results (multilabel test set) |
| | Panel | Precision | Recall | F1-score | Support | |
| |------|-----------|--------|----------|---------| |
| | 2D materials | 0.90 | 0.65 | 0.75 | 71 | |
| | 3D printing materials | 0.89 | 0.50 | 0.64 | 48 | |
| | Additive manufacturing | 0.69 | 0.75 | 0.72 | 79 | |
| | Artificial Intelligence and machine learning | 0.55 | 0.90 | 0.69 | 176 | |
| | Big-data and analytics | 0.66 | 0.67 | 0.66 | 218 | |
| | Biomaterials, biosensors, bioactuators and bioactivators | 0.76 | 0.78 | 0.77 | 117 | |
| | Biomedical, pharmaceutical, pharmacological technologies | 0.90 | 0.78 | 0.84 | 278 | |
| | Biorobotics | 0.92 | 0.75 | 0.82 | 59 | |
| | Biotechnology for health | 0.55 | 0.93 | 0.69 | 142 | |
| | Business models and digital creativity | 0.60 | 0.74 | 0.66 | 323 | |
| | Cyber connectivity and 5G | 0.77 | 0.79 | 0.78 | 119 | |
| | Cyber-security, blockchain | 0.88 | 0.92 | 0.90 | 120 | |
| | Development of energy-saving solutions | 0.63 | 0.76 | 0.69 | 143 | |
| | Development of production and organisational processes based on intelligent sensors/devices | 0.68 | 0.49 | 0.57 | 126 | |
| | Environmental resilience and pollution | 0.71 | 0.76 | 0.74 | 167 | |
| | Exploitation of renewable energy sources | 0.83 | 0.80 | 0.81 | 113 | |
| | Genomics | 0.90 | 0.86 | 0.88 | 42 | |
| | Green propulsion technologies | 0.95 | 0.52 | 0.67 | 69 | |
| | High-performance and sustainable composite materials | 0.77 | 0.64 | 0.70 | 181 | |
| | Industrial biotechnology | 0.52 | 0.88 | 0.65 | 58 | |
| | Internet of the things and services | 0.68 | 0.78 | 0.73 | 228 | |
| | Nanomaterials and nanotechnology | 0.82 | 0.89 | 0.85 | 266 | |
| | Neuroscience | 0.72 | 0.93 | 0.81 | 67 | |
| | Nutraceutics | 0.36 | 0.94 | 0.52 | 17 | |
| | Photonic, micro and nanoelectronic solutions | 0.79 | 0.88 | 0.84 | 259 | |
| | Robotic, mechatronic and automation solutions | 0.42 | 0.94 | 0.58 | 144 | |
| | Technologies for decarbonisation | 0.74 | 0.86 | 0.80 | 225 | |
| | Technologies for the circular economy and bioeconomy | 0.87 | 0.76 | 0.81 | 123 | |
|
|
| **Overall performance** |
|
|
| | | Precision | Recall | F1-score | Support | |
| |------|-----------|--------|----------|---------| |
| | **Micro avg** | **0.70** | **0.79** | **0.74** | **2077** | |
| | **Macro avg** | **0.73** | **0.78** | **0.73** | **2077** | |
| |
| --- |
|
|
| ## Framework versions |
|
|
| - **Transformers:** 4.57.x |
| - **PyTorch:** 2.8.0 |
| - **Datasets:** 3.x |
| - **Tokenizers:** 0.22.x |
