Instructions to use startlux-models/Massive-Activations-HLA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use startlux-models/Massive-Activations-HLA with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="startlux-models/Massive-Activations-HLA")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("startlux-models/Massive-Activations-HLA", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use startlux-models/Massive-Activations-HLA with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "startlux-models/Massive-Activations-HLA"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/Massive-Activations-HLA",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/startlux-models/Massive-Activations-HLA

SGLang

How to use startlux-models/Massive-Activations-HLA with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "startlux-models/Massive-Activations-HLA" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/Massive-Activations-HLA",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "startlux-models/Massive-Activations-HLA" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/Massive-Activations-HLA",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use startlux-models/Massive-Activations-HLA with Docker Model Runner:
```
docker model run hf.co/startlux-models/Massive-Activations-HLA
```

Massive Activations in Hybrid Linear Attention Models

Controlled-pretraining checkpoints accompanying Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus.

Legacy aggregate repository. This repository is retained permanently for compatibility with existing links, citations, caches, and workflows that use repo_id="startlux-models/Massive-Activations-HLA" together with a subfolder. For new downloads, code, and citations, use the Hugging Face Collection and the independent model repositories listed below. New model releases will be published as independent repositories rather than added here.

Resources: Model Collection · Official analysis repository · Quickstart · Installation · Reproduction guide

The Collection is the recommended entry point. Each checkpoint now has an independent public model repository, with its files at the repository root for direct download. No existing files have been removed from this aggregate repository, so legacy links and subfolder-based workflows continue to work.

Model collection

Model repository	Scale	Experiment	Full-attention layers
`gdn-340m-pas-fa-layer04-10b`	340M	PAS placement	4
`gdn-340m-pas-fa-layer12-10b`	340M	PAS placement	12
`gdn-340m-pas-fa-layer20-10b`	340M	PAS placement	20
`gdn-nooutgate-340m-pas-fa-layer12-10b`	340M	PAS gating ablation	12
`gdn-gatedfa-340m-pas-fa-layer12-10b`	340M	PAS gated-FA ablation	12
`gdn-1.3b-pas-fa-layer12-50b`	1.3B	PAS scale study	12
`gdn-340m-isp-hybrid-3to1-10b`	340M	ISP 3:1 hybrid	3, 6, 9, 12, 15, 18, 21, 24
`gdn-nooutgate-340m-isp-hybrid-3to1-10b`	340M	ISP gating ablation	3, 6, 9, 12, 15, 18, 21, 24
`gdn-gatedfa-340m-isp-hybrid-3to1-10b`	340M	ISP gated-FA ablation	3, 6, 9, 12, 15, 18, 21, 24
`gdn-1.3b-isp-hybrid-3to1-50b`	1.3B	ISP scale study	3, 6, 9, 12, 15, 18, 21, 24

Layer numbers in this table are one-based for readability.

Compatibility and reproducibility scope

The baseline and gdn-nooutgate-* checkpoints load with the public, pinned environment documented in the GitHub repository:

conda create -n ma-hla python=3.12 -y
conda activate ma-hla
bash scripts/install_released_gdn_cu126.sh

The two gdn-gatedfa-* checkpoints are weights-only research artifacts. Their full-attention layers use a post-SDPA, head-specific sigmoid output gate inspired by the G1 design in Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (official code). The exact GatedDeltaNet integration is not distributed. These two checkpoints are not part of the public from-scratch quickstart.

The repository loader raises an explicit compatibility message for gated-FA models unless a user deliberately configures a compatible local implementation.

Loading a baseline checkpoint

Register the public FLA architecture before using Transformers directly. The independent repositories do not require a subfolder argument:

import fla.models.gated_deltanet  # registers the custom config/model
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "startlux-models/gdn-340m-pas-fa-layer12-10b"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
)

For analysis, prefer the GitHub registry and scripts because they validate the FLA version and recover full-attention layer metadata consistently.

Reproducibility notes

Public FLA: v0.5.2, commit 9c8e42e762fce087c27b673af4922795d9edb85e.
Exact A800/CUDA 12.6 package versions are recorded in requirements/released-gdn-cu126.txt in the code repository.
The Summer prompt and inline five-domain examples reproduce the public smoke pipeline. Exact input-level regeneration of every paper panel requires the sampled JSONL inputs used for that panel.
Model weights are Apache-2.0; repository analysis code is MIT.

Limitations

These are research checkpoints, not instruction-tuned or safety-tuned models. They have not been validated for production use. The gated-FA variants require an undistributed compatibility implementation as described above.

Downloads last month: -; Downloads are not tracked for this model. How to track

Paper for startlux-models/Massive-Activations-HLA

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free

Paper • 2505.06708 • Published May 10, 2025 • 11