File size: 7,629 Bytes

fd9ac7d

---
base_model: answerdotai/ModernBERT-base
library_name: peft
pipeline_tag: text-classification
tags:
  - lora
  - peft
  - modernbert
  - llm-routing
  - cost-quality-pareto
  - mmlu-pro
datasets:
  - localgate/mmlu-pro-open
license: mit
language:
  - en
metrics:
  - kappa_w
  - accuracy
model-index:
  - name: larpbert
    results:
      - task:
          type: text-classification
          name: LLM Routing Difficulty Classification
        dataset:
          name: localgate/mmlu-pro-open
          type: localgate/mmlu-pro-open
        metrics:
          - name: Out-of-Fold Mean Log Loss (NLL)
            type: loss
            value: 0.5887
          - name: Linear Cohen's Kappa (kappa_w)
            type: kappa_w
            value: 0.3154
          - name: Macro F1
            type: f1
            value: 0.6957
          - name: Spearman Rank Correlation
            type: spearman
            value: 0.5001
---

# larpBERT: Learned Adaptive Routing Predictor (ModernBERT-LoRA)

**larpBERT** is a parameter-efficient routing classifier for hybrid local/cloud LLM inference systems. Built by fine-tuning [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) with low-rank adaptation (LoRA, $r=8$), larpBERT predicts query-level solvability for lightweight local models (e.g. Gemma 4 E2B) before dispatching to expensive frontier cloud endpoints.

This repository hosts the **tournament champion** (`beta_binomial`) at the root level, alongside the two runner-up finalist adapters in dedicated subfolders (`binomial/` and `softmax-rps/`).

---

## 1. Model Summary

- **Base Architecture**: [`answerdotai/ModernBERT-base`](https://huggingface.co/answerdotai/ModernBERT-base) (22 layers, 768 hidden dimension, 149M total parameters)
- **Adaptation Method**: PEFT LoRA ($r = 8$, $\alpha = 16$, dropout $= 0.1$, targeting `Wqkv`)
- **Trainable Parameters**: 1,123,590 parameters (~0.75% of base model)
- **Output Space**: 6 ordinal probability bands $\ell \in \{0, 1, 2, 3, 4, 5\}$ corresponding to the probability that a lightweight local model solves the incoming query correctly.
- **Context Length**: 1,024 tokens (native ModernBERT architecture with rotary embeddings, unpadding, and FlashAttention-2 support).

---

## 2. Tournament Finalists & Evaluation

The models were evaluated in a rigorous 108-configuration screening tournament using 5-fold cross-validation on $n = 6{,}705$ stratified development questions from [localgate/mmlu-pro-open](https://huggingface.co/datasets/localgate/mmlu-pro-open). Selection was governed by out-of-fold log loss (a proper scoring rule to guard against overconfident border pushing), with linear Cohen's $\kappa_w$ as the primary ordinal metric.

| Model Variant | Objective Function | Best Epoch | OOF Mean Log Loss $\downarrow$ | Linear $\kappa_w$ $\uparrow$ | Macro F1 $\uparrow$ | Spearman $\rho$ $\uparrow$ | HF Location |
|---|---|:---:|:---:|:---:|:---:|:---:|---|
| **Beta-Binomial (Champion)** | Overdispersed Count Likelihood | 8 | **0.58867** | **0.3154** | **0.6957** | **0.5001** | *Root* (`localgate/larpbert`) |
| **Binomial** | Standard Binomial Likelihood | 5 | 0.59068 | 0.3023 | 0.6908 | 0.4962 | `localgate/larpbert` (`subfolder="binomial"`) |
| **Softmax-RPS** | Ranked Probability Score | 6 | 0.59160 | 0.3102 | 0.6912 | 0.4955 | `localgate/larpbert` (`subfolder="softmax-rps"`) |

*Note: All three finalists substantially outperform dense embedding baselines (BGE-small Centroid $\kappa_w = 0.1918$) and non-parametric Category Mean baselines ($\kappa_w = 0.1979$).*

---

## 3. The 6-Level Probability Binning Scheme

Ground truth labels were constructed by evaluating Gemma 4 E2B across $k = 5$ independent stochastic passes ($T = 1.0, p = 0.95, k = 64$) graded by a 3-judge LLM panel on AWS Bedrock (2-of-3 majority consensus). The empirical success count $c \in \{0, 1, 2, 3, 4, 5\}$ gives continuous rate $\hat{p} = c / 5$.

The unit interval is mapped into 6 equal-width bands:
$$\ell = \min(\lfloor 6\hat{p}\rfloor, 5) \in \{0, 1, 2, 3, 4, 5\}$$

| Level ($\ell$) | Empirical Probability Band | Semantic Meaning | Recommended Action |
|:---:|:---:|---|---|
| **0** | $0.0\% \le \hat{p} < 16.7\%$ | Extremely unlikely local success | **Route to Cloud Frontier** |
| **1** | $16.7\% \le \hat{p} < 33.3\%$ | Very unlikely local success | **Route to Cloud Frontier** |
| **2** | $33.3\% \le \hat{p} < 50.0\%$ | Unlikely local success | **Route to Cloud Frontier** |
| **3** | $50.0\% \le \hat{p} < 66.7\%$ | Likely local success | **Execute Locally** |
| **4** | $66.7\% \le \hat{p} < 83.3\%$ | Very likely local success | **Execute Locally** |
| **5** | $83.3\% \le \hat{p} \le 100.0\%$ | Extremely likely local success | **Execute Locally** |

---

## 4. Quickstart: Usage & Inference

### Installation

```bash
pip install transformers peft torch
```

### Loading the Champion (Root)

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel

repo_id = "localgate/larpbert"
base_model_id = "answerdotai/ModernBERT-base"

# 1. Load Tokenizer & Base ModernBERT
tokenizer = AutoTokenizer.from_pretrained(repo_id)
base_model = AutoModelForSequenceClassification.from_pretrained(
    base_model_id, 
    num_labels=6
)

# 2. Load the Champion Adapter (Beta-Binomial)
model = PeftModel.from_pretrained(base_model, repo_id)
model.eval()

# 3. Classify an incoming prompt
query = "What is the rank of a 3x3 real symmetric matrix with eigenvalues 2, 0, -1?"
inputs = tokenizer(query, return_tensors="pt", max_length=1024, truncation=True)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)
    pred_level = torch.argmax(probs, dim=-1).item()

# Standard routing threshold (Level >= 3 -> Local; Level < 3 -> Cloud)
decision = "LOCAL" if pred_level >= 3 else "CLOUD"
print(f"Predicted Difficulty Band: {pred_level} | Routing Decision: {decision}")
```

### Loading Finalist Runner-Up Variants (`subfolder`)

To load the **Binomial** or **Softmax-RPS** models from the same repository:

```python
# Load Binomial Finalist
binomial_model = PeftModel.from_pretrained(
    base_model, 
    repo_id, 
    subfolder="binomial"
)

# Load Softmax-RPS Finalist
softmax_rps_model = PeftModel.from_pretrained(
    base_model, 
    repo_id, 
    subfolder="softmax-rps"
)
```

---

## 5. Training Details

- **Hardware**: RunPod NVIDIA A100-SXM4 (80GB VRAM)
- **Batch Size**: 32 (effective batch size 32, zero gradient accumulation delay)
- **Optimizer**: AdamW ($\beta_1 = 0.9, \beta_2 = 0.999, \epsilon = 10^{-8}$)
- **Learning Rate**: $5 \times 10^{-5}$ with linear warmup across the first 10% of training steps and cosine decay
- **Weight Decay**: $0.01$
- **Dropout**: $0.1$ on attention and LoRA layers
- **Sequence Length**: 1,024 tokens (zero prompt truncation across all training and evaluation instances)

---

## 6. Datasets & Reproducibility

- **Training Corpus**: [localgate/mmlu-pro-open](https://huggingface.co/datasets/localgate/mmlu-pro-open) ($n = 6{,}705$ development prompts, $5{,}848$ train / $857$ validation).
- **Code & Verification Pipeline**: Full evaluation manifests, cross-validation out-of-fold predictions, noise ceiling models, and paper drafts are maintained in the [LocalGate GitHub repository](https://github.com/abullard1/localgate).

---

## 7. Citation

```bibtex
@article{localgate2026,
  title={Learned Adaptive Routing for Hybrid Local-Cloud LLM Inference},
  author={LocalGate Research Team},
  year={2026},
  journal={arXiv preprint},
  url={https://github.com/abullard1/localgate}
}
```