| --- |
| tags: |
| - bert |
| - scientific-abstract |
| - multi-label-classification |
| - natural-language-processing |
| datasets: |
| - custom-scientific-abstracts |
| license: apache-2.0 |
| --- |
| |
| # SciAbstract-MultiLabel-BERT-Base |
|
|
| ## 📝 Overview |
|
|
| **SciAbstract-MultiLabel-BERT-Base** is a specialized multi-label text classification model fine-tuned on scientific paper abstracts. It simultaneously classifies an abstract into its **Primary Topic** and determines the underlying **Sentiment/Impact** of the research findings (e.g., highly positive breakthrough, negative result/concern). |
|
|
| The model is based on the robust `bert-base-uncased` architecture and is ideal for automating the categorization and high-level assessment of large volumes of academic literature. |
|
|
| ## 🧠 Model Architecture |
|
|
| The model uses the `BertForSequenceClassification` head, configured for a multi-label setup. |
|
|
| * **Base Model:** `bert-base-uncased` |
| * **Input:** Scientific abstract text. |
| * **Output:** A 17-dimensional vector of logits, where each dimension corresponds to one of the 17 potential labels (12 Topics + 5 Sentiments). The model uses a sigmoid activation function for the final layer to handle the multi-label nature, allowing it to predict multiple positive labels (e.g., one Topic and one Sentiment) for a single input. |
| * **Loss Function:** Binary Cross-Entropy with Logits (BCEWithLogitsLoss). |
| * **Labels:** |
| * **Topics:** Materials Science, Neuroscience, Computer Science, Ecology, Astrophysics, Medicine, etc. |
| * **Sentiments:** Highly Positive, Positive, Moderately Negative, Negative, Highly Negative. |
|
|
| ## 🚀 Intended Use |
|
|
| * **Automated Document Triage:** Rapidly categorize new research papers for subject-matter experts. |
| * **Literature Review:** Filter and prioritize papers based on topic and the detected impact (sentiment) of the findings. |
| * **Trend Analysis:** Track the volume of positive vs. negative research outcomes within specific scientific fields over time. |
|
|
| ## ⚠️ Limitations |
|
|
| * **Multi-Label Complexity:** The model may struggle with abstracts that span highly ambiguous or highly interdisciplinary topics not well-represented in the training data. |
| * **Sentiment Scope:** The sentiment classification is specifically tailored to the tone of scientific findings (e.g., success of an experiment, critical failure, potential concern) and may not generalize well to general public sentiment. |
| * **Maximum Length:** Input text is truncated to 512 tokens (the BERT standard). Extremely long abstracts may lose critical information. |
|
|
| ## 💻 Example Code |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| import torch |
| |
| # Load model and tokenizer |
| model_name = "Your-HF-Username/SciAbstract-MultiLabel-BERT-Base" |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForSequenceClassification.from_pretrained(model_name) |
| |
| # Sample abstract |
| abstract = "Development of a quantum entanglement system achieving coherence for over 10 seconds at room temperature, a significant breakthrough for quantum computing." |
| |
| # Tokenize input |
| inputs = tokenizer(abstract, return_tensors="pt", truncation=True, padding=True) |
| |
| # Make prediction |
| with torch.no_grad(): |
| logits = model(**inputs).logits |
| |
| # Apply sigmoid to get probabilities for each label |
| probabilities = torch.sigmoid(logits).squeeze() |
| |
| # Get the label IDs and names |
| id2label = model.config.id2label |
| predicted_labels = [id2label[i] for i, prob in enumerate(probabilities) if prob > 0.5] # Threshold at 0.5 |
| |
| print(f"Abstract: {abstract}") |
| print("-" * 30) |
| print(f"Predicted Labels: {predicted_labels}") |
| # Expected Output Example: ['Topic: Physics', 'Sentiment: Highly Positive'] |
