docs: publish verified NeuralOps model card for roberta-base-adasplash

b53b5bb verified 11 days ago

6.02 kB

---
language:
- en
license: mit
library_name: transformers
tags:
- neuralops
- adasplash
- entmax
- sparse-attention
- subquadratic-attention
- roberta
- sst2
pipeline_tag: text-classification
datasets:
- glue
- sst2
metrics:
- accuracy
- f1
base_model: roberta-base
---
# RoBERTa Base with AdaSplash Attention via NeuralOps

## Model Details
- **Model Identifier**: `Hooshaai/roberta-base-adasplash`
- **Base Architecture**: RoBERTa Base fine-tuned on GLUE SST-2 (125M parameters, 12 layers, 12 heads, hidden size 768)
- **Attention Mechanism**: `adasplash_attention` (Adaptive Splash Attention with Tsallis $\alpha=1.5$ Entmax Sparse Simplex Projection)
- **Attention Sparsity**: 40% exact mathematical zero weights ($A_{ij} \equiv 0$) across all 12 encoder blocks
- **Parameters**: 125.8M (+0.94% parameter count via `AdaSplashLinear` residual LoRA adapters)
- **Framework**: `neuralops` v1.4.3
- **Organization**: [Hooshaai](https://huggingface.co/Hooshaai)
## Architecture Overview
RoBERTa Base relies on deep bidirectional contextual representations across 12 transformer encoder blocks. Because RoBERTa features byte-pair encoding with larger sequence vocabularies, dense Softmax attention tends to over-allocate probability mass to punctuation, subword tokens, and padding symbols.
**AdaSplash Attention** incorporates **Tsallis 1.5-entmax** into RoBERTa's multi-head attention blocks:
1. **40% Exact Attention Sparsity**: Replaces dense softmax with Tsallis $\alpha=1.5$ entmax, projecting attention distributions onto the simplex boundary and ensuring that 40% of attention matrix entries are strictly zero.
2. **LoRA-Augmented Residual Projections**: Projections ($W_Q, W_K, W_V, W_O$) are implemented via `AdaSplashLinear`, freezing pre-trained weights while learning additive rank-16 adapter pathways.
3. **Full Encoder Context Retention**: Rather than applying local causal splash windowing, the bidirectional encoder retains full receptive field reach while filtering irrelevant cross-attention via mathematical thresholding.
```
Input Activation X [B, L, 768]
       │
       ├── AdaSplashLinear Projections: Q, K, V ∈ ℝ^{B × 12 × L × 64}
       ├── Normalized Scaled Dot-Products: S_{i, j} = (Q_i K_j^T) / 8
       ├── Exact Tsallis 1.5-Entmax Simplex Projection:
       │     Solves: ∑ max(0, (S_{i, j} - τ)/2)^2 = 1 via bisection search
       │     Produces A_{i, j} with 40% exact zeros
       ├── Sparse Context Aggregation: O_i = ∑_{j: A_{i, j} > 0} A_{i, j} V_j
       └── Output Synthesis: Y = AdaSplashLinear_O(O)  [B, L, 768]
```
## Mathematical Formulation & Derivation

### 1. Entmax Projection on RoBERTa Hidden States
Given logit vector $s \in \mathbb{R}^L$ for each query position:
\[
p^*(s) = \arg\max_{p \in \Delta^{L-1}} \left\{ p^\top s + \frac{4}{3} \sum_{i=1}^L (p_i - p_i^{1.5}) \right\}
\]
The analytical solution is:
\[
p_i^*(s) = \left[ \frac{s_i - \tau}{2} \right]_+^2
\]
where $\tau$ is uniquely determined by monotonic root-finding on the simplex constraint.

### 2. Gradient Flow and Non-Vanishing Derivatives
Unlike hard step thresholding, the gradient of 1.5-entmax with respect to logits is continuous and Lipschitz bounded:
\[
\frac{\partial p_i^*}{\partial s_j} = \left[ \frac{s_i - \tau}{2} \right]_+ \left( \delta_{ij} - \frac{\left[ \frac{s_j - \tau}{2} \right]_+}{\sum_{k \in \operatorname{supp}(p^*)} \left[ \frac{s_k - \tau}{2} \right]_+} \right)
\]
guaranteeing stable gradient backpropagation into the pre-trained encoder weights and LoRA adapters.
## Empirical Benchmark Table
Evaluated on the GLUE SST-2 validation split on NVIDIA GeForce RTX 4070 (SM 8.9) with sequence length $L = 128$ and batch size $B = 32$:
| Metric | Dense RoBERTa SST-2 Baseline | Zero-Shot AdaSplash | Post-Distillation (v1.4.3) | Benefit vs. Baseline |
| :--- | :---: | :---: | :---: | :---: |
| **Accuracy (SST-2)** | 94.38% | 75.46% | **82.10%** | Distilled recovery |
| **F1 Score** | 0.9445 | 0.7277 | **0.8240** | High classification fidelity |
| **Attention Sparsity (% Zeros)**| 0.0% (Softmax) | **40.0%** | **40.0%** | **Strict Simplex Sparsity** |
| **Active Parameters** | 124.6M | 125.8M | **125.8M** | +0.94% LoRA adapters |
| **Triton Kernel Latency** | 4.10 ms | 1.54 ms | **1.54 ms** | **2.66x Kernel Speedup** |
| **Inference Step Time** | 25.36 ms | 16.68 ms | **16.68 ms** | **1.52x Speedup** |
| **Peak VRAM Footprint** | 1,063 MB | 712 MB | **712 MB** | **-33.0% Reduction** |
## Quickstart Code
```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from neuralops import replace_attention
# 1. Load tokenizer and model from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("roberta-base")
model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-adasplash")
# 2. Or surgically inject AdaSplash into any RoBERTa in 1 line
base_model = AutoModelForSequenceClassification.from_pretrained("roberta-base")
adasplash_model = replace_attention(base_model, "adasplash_attention")
# 3. Predict sentiment
inputs = tokenizer("The sparse entmax projection preserves salient features with extreme precision!", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
    prediction = logits.argmax(-1).item()
print(f"Predicted class: {prediction} ({'Positive' if prediction == 1 else 'Negative'})")
```
## Hardware & Software Requirements
- **PyTorch**: 2.11+
- **CUDA**: 12.8+
- **Triton**: 3.6+
- **Accelerators**: NVIDIA SM 8.0+ (Ampere, Ada Lovelace, Hopper, Blackwell)
- **Universal Fallback**: Automatic Level 3 pure PyTorch vectorization available on CPU and Apple Silicon (MPS).
## Citation
```bibtex
@software{neuralops2026,
  author = {Majlesi, Taha and contributors},
  title = {NeuralOps: Unified Platform for Subquadratic Attention and Deep Model Compression},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/tahamajs/NeuralOps}},
  version = {1.4.3}
}
```

	---
	language:
	- en
	license: mit
	library_name: transformers
	tags:
	- neuralops
	- adasplash
	- entmax
	- sparse-attention
	- subquadratic-attention
	- roberta
	- sst2
	pipeline_tag: text-classification
	datasets:
	- glue
	- sst2
	metrics:
	- accuracy
	- f1
	base_model: roberta-base
	---

	# RoBERTa Base with AdaSplash Attention via NeuralOps

	## Model Details
	- Model Identifier: `Hooshaai/roberta-base-adasplash`
	- Base Architecture: RoBERTa Base fine-tuned on GLUE SST-2 (125M parameters, 12 layers, 12 heads, hidden size 768)
	- Attention Mechanism: `adasplash_attention` (Adaptive Splash Attention with Tsallis $\alpha=1.5$ Entmax Sparse Simplex Projection)
	- Attention Sparsity: 40% exact mathematical zero weights ($A_{ij} \equiv 0$) across all 12 encoder blocks
	- Parameters: 125.8M (+0.94% parameter count via `AdaSplashLinear` residual LoRA adapters)
	- Framework: `neuralops` v1.4.3
	- Organization: [Hooshaai](https://huggingface.co/Hooshaai)

	## Architecture Overview

	RoBERTa Base relies on deep bidirectional contextual representations across 12 transformer encoder blocks. Because RoBERTa features byte-pair encoding with larger sequence vocabularies, dense Softmax attention tends to over-allocate probability mass to punctuation, subword tokens, and padding symbols.

	AdaSplash Attention incorporates Tsallis 1.5-entmax into RoBERTa's multi-head attention blocks:
	1. 40% Exact Attention Sparsity: Replaces dense softmax with Tsallis $\alpha=1.5$ entmax, projecting attention distributions onto the simplex boundary and ensuring that 40% of attention matrix entries are strictly zero.
	2. LoRA-Augmented Residual Projections: Projections ($W_Q, W_K, W_V, W_O$) are implemented via `AdaSplashLinear`, freezing pre-trained weights while learning additive rank-16 adapter pathways.
	3. Full Encoder Context Retention: Rather than applying local causal splash windowing, the bidirectional encoder retains full receptive field reach while filtering irrelevant cross-attention via mathematical thresholding.

	```
	Input Activation X [B, L, 768]
	│
	├── AdaSplashLinear Projections: Q, K, V ∈ ℝ^{B × 12 × L × 64}
	├── Normalized Scaled Dot-Products: S_{i, j} = (Q_i K_j^T) / 8
	├── Exact Tsallis 1.5-Entmax Simplex Projection:
	│ Solves: ∑ max(0, (S_{i, j} - τ)/2)^2 = 1 via bisection search
	│ Produces A_{i, j} with 40% exact zeros
	├── Sparse Context Aggregation: O_i = ∑_{j: A_{i, j} > 0} A_{i, j} V_j
	└── Output Synthesis: Y = AdaSplashLinear_O(O) [B, L, 768]
	```

	## Mathematical Formulation & Derivation

	### 1. Entmax Projection on RoBERTa Hidden States
	Given logit vector $s \in \mathbb{R}^L$ for each query position:
	\[
	p^*(s) = \arg\max_{p \in \Delta^{L-1}} \left\{ p^\top s + \frac{4}{3} \sum_{i=1}^L (p_i - p_i^{1.5}) \right\}
	\]
	The analytical solution is:
	\[
	p_i^*(s) = \left[ \frac{s_i - \tau}{2} \right]_+^2
	\]
	where $\tau$ is uniquely determined by monotonic root-finding on the simplex constraint.

	### 2. Gradient Flow and Non-Vanishing Derivatives
	Unlike hard step thresholding, the gradient of 1.5-entmax with respect to logits is continuous and Lipschitz bounded:
	\[
	\frac{\partial p_i^}{\partial s_j} = \left[ \frac{s_i - \tau}{2} \right]_+ \left( \delta_{ij} - \frac{\left[ \frac{s_j - \tau}{2} \right]_+}{\sum_{k \in \operatorname{supp}(p^)} \left[ \frac{s_k - \tau}{2} \right]_+} \right)
	\]
	guaranteeing stable gradient backpropagation into the pre-trained encoder weights and LoRA adapters.

	## Empirical Benchmark Table

	Evaluated on the GLUE SST-2 validation split on NVIDIA GeForce RTX 4070 (SM 8.9) with sequence length $L = 128$ and batch size $B = 32$:

	\| Metric \| Dense RoBERTa SST-2 Baseline \| Zero-Shot AdaSplash \| Post-Distillation (v1.4.3) \| Benefit vs. Baseline \|
	\| :--- \| :---: \| :---: \| :---: \| :---: \|
	\| Accuracy (SST-2) \| 94.38% \| 75.46% \| 82.10% \| Distilled recovery \|
	\| F1 Score \| 0.9445 \| 0.7277 \| 0.8240 \| High classification fidelity \|
	\| Attention Sparsity (% Zeros)\| 0.0% (Softmax) \| 40.0% \| 40.0% \| Strict Simplex Sparsity \|
	\| Active Parameters \| 124.6M \| 125.8M \| 125.8M \| +0.94% LoRA adapters \|
	\| Triton Kernel Latency \| 4.10 ms \| 1.54 ms \| 1.54 ms \| 2.66x Kernel Speedup \|
	\| Inference Step Time \| 25.36 ms \| 16.68 ms \| 16.68 ms \| 1.52x Speedup \|
	\| Peak VRAM Footprint \| 1,063 MB \| 712 MB \| 712 MB \| -33.0% Reduction \|

	## Quickstart Code

	```python
	import torch
	from transformers import AutoModelForSequenceClassification, AutoTokenizer
	from neuralops import replace_attention

	# 1. Load tokenizer and model from Hugging Face Hub
	tokenizer = AutoTokenizer.from_pretrained("roberta-base")
	model = AutoModelForSequenceClassification.from_pretrained("Hooshaai/roberta-base-adasplash")

	# 2. Or surgically inject AdaSplash into any RoBERTa in 1 line
	base_model = AutoModelForSequenceClassification.from_pretrained("roberta-base")
	adasplash_model = replace_attention(base_model, "adasplash_attention")

	# 3. Predict sentiment
	inputs = tokenizer("The sparse entmax projection preserves salient features with extreme precision!", return_tensors="pt")
	with torch.no_grad():
	logits = model(**inputs).logits
	prediction = logits.argmax(-1).item()

	print(f"Predicted class: {prediction} ({'Positive' if prediction == 1 else 'Negative'})")
	```

	## Hardware & Software Requirements
	- PyTorch: 2.11+
	- CUDA: 12.8+
	- Triton: 3.6+
	- Accelerators: NVIDIA SM 8.0+ (Ampere, Ada Lovelace, Hopper, Blackwell)
	- Universal Fallback: Automatic Level 3 pure PyTorch vectorization available on CPU and Apple Silicon (MPS).

	## Citation

	```bibtex
	@software{neuralops2026,
	author = {Majlesi, Taha and contributors},
	title = {NeuralOps: Unified Platform for Subquadratic Attention and Deep Model Compression},
	year = {2026},
	publisher = {GitHub},
	howpublished = {\url{https://github.com/tahamajs/NeuralOps}},
	version = {1.4.3}
	}
	```