SSA-Llama-3.1-8B-Instruct

Llama 3.1 8B Instruct with content-adaptive sparse attention for efficient long-context inference.

SSA dynamically selects relevant regions of long context instead of requiring selected attention layers to attend densely across the entire sequence.

The goal is simple:

Preserve long-context capability while making attention increasingly efficient as context grows.

This repository contains the primary SSA-Llama-3.1-8B-Instruct checkpoint trained in the SSA research.

Highlights

• Built on Meta Llama 3.1 8B Instruct
• Content-adaptive sparse attention
• Long-context retrieval evaluated from 16K to 128K
• Sparse-attention operator evaluated up to 1M tokens
• Strong single-fact retrieval through 64K
• Increasing computational advantage as context length grows
• Designed for research into practical long-context LLM inference

Long-Context Retrieval

Using the current SSA sparse-attention backend:

Context	Single-Fact NLL	Multi-Fact NLL
16K	0.00001	0.00006
32K	0.00005	0.00114
48K	0.00026	0.00237
64K	0.00015	0.06979

First-answer-token Top-1 remained 100% across all of these diagnostic cases.

Single-fact retrieval remained extremely strong through 64K. Multi-fact retrieval was also strong through 48K, with increased difficulty appearing at 64K.

These numbers are controlled teacher-forced retrieval diagnostics. Autoregressive Needle-in-a-Haystack results will be added separately after evaluation is complete.

Why Sparse Attention?

Standard dense causal attention becomes increasingly expensive as context grows.

SSA limits the amount of historical context processed by its sparse attention layers while still selecting information based on content.

Conceptually:

Long Context
     ↓
Find Relevant Historical Regions
     ↓
Local Context + Selected Context
     ↓
Sparse Attention
     ↓
Generation

For the currently evaluated configuration, the nominal semantic attention support is approximately 8,196 keys per query.

Semantic Attention Arithmetic

The ideal semantic QK/PV arithmetic reduction grows rapidly with context length:

Context	Dense causal avg. keys/query	SSA fanout cap	Attention arithmetic reduction	QK/PV work saved
32K	16,384	8,196	2.00×	50.0%
64K	32,768	8,196	4.00×	75.0%
128K	65,536	8,196	8.00×	87.5%
256K	131,072	8,196	15.99×	93.75%
512K	262,144	8,196	31.98×	96.87%
1M	~500,000	8,196	61.01×	98.36%

These numbers represent semantic attention arithmetic, not total model FLOPs or end-to-end speedup.

Measured Sparse-Attention Scaling

The custom SSA attention operator shows increasing benefit as context grows:

Context	Relative operator speed
64K	~1.0×
128K	~1.9×
256K	~3.7×
512K	~7.3×
1M	~14.2×

The important distinction is that these are attention-operator measurements, not end-to-end LLM speedups.

The results show the central systems motivation behind SSA: sparse attention becomes increasingly valuable as context length grows.

Model Details

• Model: onlyaady/SSA-Llama-3.1-8B-Instruct
• Base model: meta-llama/Llama-3.1-8B-Instruct
• Developer: Hrudayaditya Jallu
• Affiliation: Independent research
• Checkpoint: SSA-Llama-3.1-8B-Instruct
• Format: PEFT / LoRA
• Language: English
• Status: Research release

Using the Model

This repository currently contains the trained PEFT checkpoint.

SSA also relies on a custom sparse-attention runtime. Loading this adapter through a standard Transformers pipeline alone will not reproduce the sparse-attention behavior or performance measurements above.

A reproducible inference/runtime package will be released separately.

This checkpoint is currently intended for research in:

• long-context LLM inference
• sparse attention
• content-based routing
• retrieval
• efficient inference
• quality/compute tradeoffs

Current Limitations

SSA is in active research.

A few important limitations remain:

• only a subset of transformer layers currently use the SSA sparse path
• the complete model is therefore not yet fully subquadratic
• multi-fact retrieval becomes more difficult at larger context lengths, the model was only trained to support 16k.
• 1M-token speed results are operator-level measurements
• additional free-generation evaluations are still in progress

This checkpoint should currently be treated as a research release rather than a production replacement for standard Llama 3.1 inference.

Research

This model is part of my research project.

Manuscript in preparation.

Citation

@misc{jallu2026ssa,
  title  = {Sub-Quadratic Sparse Attention (SSA) for Long-Context LLMs},
  author = {Jallu, Hrudayaditya},
  year   = {2026},
  note   = {Manuscript in preparation}
}

Contact for queries

Mail - hjallu@umass.edu

Framework

• PyTorch
• Hugging Face Transformers
• PEFT 0.19.1
• Meta Llama 3.1 8B Instruct