README.md

---
base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:meta-llama/Llama-3.1-8B-Instruct
- lora
- transformers
license: apache-2.0
language:
- en
---

# SSA-Llama-3.1-8B-Instruct

**Llama 3.1 8B Instruct with content-adaptive sparse attention for efficient long-context inference.**

SSA dynamically selects relevant regions of long context instead of requiring selected attention layers to attend densely across the entire sequence.

The goal is simple:

> **Preserve long-context capability while making attention increasingly efficient as context grows.**

This repository contains the primary **SSA-Llama-3.1-8B-Instruct checkpoint** trained in the SSA research.

## Highlights

* Built on **Meta Llama 3.1 8B Instruct**
* Content-adaptive sparse attention
* Long-context retrieval evaluated from **16K to 128K**
* Sparse-attention operator evaluated up to **1M tokens**
* Strong single-fact retrieval through **64K**
* Increasing computational advantage as context length grows
* Designed for research into practical long-context LLM inference

## Long-Context Retrieval

Using the current SSA sparse-attention backend:

| Context | Single-Fact NLL | Multi-Fact NLL |
| ------: | --------------: | -------------: |
|     16K |         0.00001 |        0.00006 |
|     32K |         0.00005 |        0.00114 |
|     48K |         0.00026 |        0.00237 |
|     64K |         0.00015 |        0.06979 |

**First-answer-token Top-1 remained 100% across all of these diagnostic cases.**

Single-fact retrieval remained extremely strong through 64K. Multi-fact retrieval was also strong through 48K, with increased difficulty appearing at 64K.

These numbers are controlled teacher-forced retrieval diagnostics. Autoregressive Needle-in-a-Haystack results will be added separately after evaluation is complete.

## Why Sparse Attention?

Standard dense causal attention becomes increasingly expensive as context grows.

SSA limits the amount of historical context processed by its sparse attention layers while still selecting information based on content.

Conceptually:

```text
Long Context
     ↓
Find Relevant Historical Regions
     ↓
Local Context + Selected Context
     ↓
Sparse Attention
     ↓
Generation
```

For the currently evaluated configuration, the nominal semantic attention support is approximately **8,196 keys per query**.

### Semantic Attention Arithmetic

The ideal semantic QK/PV arithmetic reduction grows rapidly with context length:

| Context | Dense causal avg. keys/query | SSA fanout cap | Attention arithmetic reduction | QK/PV work saved |
| ------: | ---------------------------: | -------------: | -----------------------------: | ---------------: |
|     32K |                       16,384 |          8,196 |                      **2.00×** |        **50.0%** |
|     64K |                       32,768 |          8,196 |                      **4.00×** |        **75.0%** |
|    128K |                       65,536 |          8,196 |                      **8.00×** |        **87.5%** |
|    256K |                      131,072 |          8,196 |                     **15.99×** |       **93.75%** |
|    512K |                      262,144 |          8,196 |                     **31.98×** |       **96.87%** |
|      1M |                     ~500,000 |          8,196 |                     **61.01×** |       **98.36%** |

These numbers represent **semantic attention arithmetic**, not total model FLOPs or end-to-end speedup.

## Measured Sparse-Attention Scaling

The custom SSA attention operator shows increasing benefit as context grows:

| Context | Relative operator speed |
| ------: | ----------------------: |
|     64K |                   ~1.0× |
|    128K |                   ~1.9× |
|    256K |                   ~3.7× |
|    512K |                   ~7.3× |
|      1M |              **~14.2×** |

The important distinction is that these are **attention-operator measurements**, not end-to-end LLM speedups.

The results show the central systems motivation behind SSA: sparse attention becomes increasingly valuable as context length grows.

## Model Details

* **Model:** `onlyaady/SSA-Llama-3.1-8B-Instruct`
* **Base model:** `meta-llama/Llama-3.1-8B-Instruct`
* **Developer:** Hrudayaditya Jallu
* **Affiliation:** Independent research
* **Checkpoint:** SSA-Llama-3.1-8B-Instruct
* **Format:** PEFT / LoRA
* **Language:** English
* **Status:** Research release

## Using the Model

This repository currently contains the trained PEFT checkpoint.

SSA also relies on a custom sparse-attention runtime. Loading this adapter through a standard Transformers pipeline alone will **not reproduce the sparse-attention behavior or performance measurements above**.

A reproducible inference/runtime package will be released separately.

This checkpoint is currently intended for research in:

* long-context LLM inference
* sparse attention
* content-based routing
* retrieval
* efficient inference
* quality/compute tradeoffs

## Current Limitations

SSA is in active research.

A few important limitations remain:

* only a subset of transformer layers currently use the SSA sparse path
* the complete model is therefore not yet fully subquadratic
* multi-fact retrieval becomes more difficult at larger context lengths, the model was only trained to support 16k.
* 1M-token speed results are operator-level measurements
* additional free-generation evaluations are still in progress

This checkpoint should currently be treated as a **research release rather than a production replacement for standard Llama 3.1 inference**.

## Research

This model is part of my research project.

Manuscript in preparation.

## Citation

```bibtex
@misc{jallu2026ssa,
  title  = {Sub-Quadratic Sparse Attention (SSA) for Long-Context LLMs},
  author = {Jallu, Hrudayaditya},
  year   = {2026},
  note   = {Manuscript in preparation}
}
```

## Contact for queries
Mail - [hjallu@umass.edu](mailto:hjallu@umass.edu)

## Framework

* PyTorch
* Hugging Face Transformers
* PEFT 0.19.1
* Meta Llama 3.1 8B Instruct