🌟 Maria-1B: Dynamic Query Adaptation (HyperGQE)

Pretrained from Scratch on 20 Billion Tokens for High-Efficiency Edge Inference

---

Key Hardware Takeaway: Under maximum sequence and batch workloads ($b=16, s=1024$), Maria-1B is 13.5% faster than Dense GQA while consuming 827 MB less peak VRAM and delivering +15.6% higher token throughput.

What Problem Are We Solving?
Frontier MoE models save FLOPs in datacenters, but are unusable on edge devices because all inactive experts must stay loaded in limited RAM. Standard dense models, on the other hand, burn 100% compute across every head.
Maria solves this through structured Query Sparsity: it dynamically drops 56.6M unneeded query parameters per token while using a lightweight hypernetwork to inject rich context back into active heads.

---

🔬 Architectural Context & Research Foundation

• 📄 Architecture Paper Preprint: https://doi.org/10.5281/zenodo.22747827
• 👨‍🔬 Author: Yadki Zahid Ahmed
• 💡 Relationship to Paper: The research preprint introduces and proves the HyperGQE architecture (originally validated on a TinyLlama-1.1B testbed). Maria-1B is the first standalone, from-scratch 20-Billion-token pretrained foundation model implementing HyperGQE with a modern 128,256 Llama-3 vocabulary, RoPE, and SwiGLU feed-forward networks.

---

📊 Model Specifications

Parameter	Specification	Visual Indicator
Total Instantiated Parameters	1,073,515,904 (1.074 Billion)	📦 Full Model
Active Parameters per Token	1,016,878,976 (1.017 Billion)	⚡ Sparse Compute
Sparsity per Block	12 out of 16 query expert heads dropped per layer (56.6M inactive params)	✂️ Query Pruned
Pretraining Token Budget	20 Billion Tokens (Pretrained from scratch)	🎯 Clean Slate
Hidden Dimension ($d_{\text{model}}$)	2,048	📐 Width
Number of Layers	18 Transformer Blocks	🧱 Depth
Query Heads ($N_{\text{heads}}$)	16 (4 groups $\times$ 4 expert heads per group)	🔀 Routed
Key-Value Heads ($N_{\text{kv}}$)	4 (Grouped-Query Attention 4:1 compression)	💾 Compressed
Intermediate Size (FFN)	5,632 (SwiGLU gating)	⚡ Non-linear
Vocabulary Size	128,256 (Llama-3 BPE Tokenizer)	📚 Rich Vocab
Context Window ($L_{\text{max}}$)	8,192 tokens with RoPE ($\theta = 500,000$)	📜 Extended
Weight Precision	bfloat16 (~2.1 GB safetensors / ~650 MB quantized Q4_K_M)	🪶 Ultra-Light

---

⚡ Hardware Profiling & Latency Benchmarks

Measured on identical NVIDIA hardware comparing Maria (HyperGQE) against Dense GQA and Static GQE:

⏱️ Latency & Throughput Matrix

Batch Size ($b$)	Sequence Length ($s$)	Dense GQA Latency	Static GQE Latency	Maria (HyperGQE) Latency	Speedup vs. Dense GQA
1	512	103.50 ms	90.47 ms	106.55 ms	Competitive
1	1024	227.76 ms	161.53 ms	204.36 ms	🟢 +10.3% faster
8	512	763.26 ms	591.01 ms	716.07 ms	🟢 +6.2% faster
8	1024	1,750.23 ms	1,161.57 ms	1,421.50 ms	🟢 +18.8% faster
16	512	1,475.45 ms	1,114.95 ms	1,361.72 ms	🟢 +7.7% faster
16	1024 (Peak Load)	3,526.88 ms	2,333.79 ms	3,049.37 ms	🚀 +13.5% FASTER

💾 VRAM & Throughput (Heavy Load: $b=16, s=1024$)

Architecture	Peak VRAM	Token Throughput	Theoretical FLOPs	Edge Viability
Dense GQA Baseline	4,181.55 MB	4,645 tokens/s	$3.60 \times 10^{13}$	Baseline
Static GQE (Sparse)	2,456.19 MB	7,020 tokens/s	$3.31 \times 10^{13}$	Collapsed Quality
Maria (HyperGQE)	3,354.19 MB	5,372 tokens/s	$3.60 \times 10^{13}$	🏆 Optimal Balance
Net Advantage	📉 827.36 MB Less VRAM	⚡ +15.6% Higher Throughput	Identical FLOP Ceiling	Zero Accuracy Drop

---

🎯 Downstream Accuracy & Representation Recovery

Static query routing (GQE) suffers from severe representational collapse because permanently pruned query heads lose attention context. Maria's dynamic hypernetwork completely eliminates this collapse:

1. Fine-Tuned Sentiment Classification (SST-2)

Architecture	Total Params	Active Params	SST-2 Accuracy	Margin vs. Dense GQA
Static GQE	875.39 M	875.39 M	75.92%	🔴 -4.24% (Degradation)
Dense GQA Baseline	946.04 M	946.04 M	80.16%	Baseline
Maria (HyperGQE)	924.13 M	843.35 M	80.85%	🟢 +0.69% HIGHER

Maria not only recovers the 4.24% performance drop caused by static pruning—it surpasses the dense baseline while activating 102M fewer parameters.

2. Zero-Shot STEM Domain Recovery (16 Benchmarks)

Benchmark / Task	Static GQE Accuracy	Maria (HyperGQE) Accuracy	Net Margin ($\Delta$)
MMLU: College Chemistry	19.00%	28.00%	🟢 +9.00%
MMLU: High School Biology	21.00%	25.80%	🟢 +4.80%
MMLU: Virology	24.10%	28.30%	🟢 +4.20%
MMLU: College Biology	22.20%	26.30%	🟢 +4.10%
MMLU: Medical Genetics	23.00%	27.00%	🟢 +4.00%
MMLU: High School Chemistry	19.20%	22.70%	🟢 +3.50%
ARC (Challenge)	21.80%	23.30%	🟢 +1.50%
ARC (Easy)	41.50%	42.90%	🟢 +1.40%
BoolQ	57.00%	58.20%	🟢 +1.20%

---

🚀 Quickstart: Run Maria-1B

Base Model Inference Notice: Maria-1B is a foundation pretraining base model (trained from scratch on 20B tokens of raw web text, books, and code), not an instruction-aligned chat assistant. For clean, fluent text continuation, use greedy decoding (do_sample=False) or controlled sampling (temperature=0.3, top_k=40, repetition_penalty=1.2).

Hardware Precision Guide:

• Google Colab Free Tier (Tesla T4) / Turing / Pascal GPUs: Use torch_dtype=torch.float16 (Tesla T4 lacks native hardware bfloat16).
• NVIDIA Ampere / Hopper / Ada (A100, H100, RTX 3090/4090): You can use either torch_dtype=torch.bfloat16 or torch.float16.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Zahid2005/Maria-1B"

# 1. Load tokenizer & model directly from Hugging Face
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.float16,  # Recommended for Colab Tesla T4 & FP16 GPUs
    device_map="auto"
)
model.eval()

# 2. Format Prompt
prompt = "Artificial intelligence and future edge computing"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# 3. Generate with controlled base-model parameters
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=40,
        do_sample=True,
        temperature=0.3,
        top_k=40,
        repetition_penalty=1.2,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

📱 Edge Computing & Mobile Deployment

Because Maria features dense FFNs and structured query-level sparsity, it avoids the massive memory penalties of MoE architectures:

• 🗜️ 4-Bit Quantization (GGUF / Q4_K_M): Compresses the model weights to ~650 MB.
• 📱 RAM Requirement: Requires only ~800 MB of total RAM, running at 45–60 tokens/sec on mobile chips (Apple A/M series, Qualcomm Snapdragon, MediaTek Dimensity, Raspberry Pi 5) with zero cloud dependencies.

---

📚 Research Background & Citation

The HyperGQE (Dynamic Query Adaptation) sparse attention mechanism was originally introduced and validated on a TinyLlama-1.1B backbone in:

"CAN WE IMPROVE THE PERFORMANCE OF SPARSE ATTENTION IN GROUPED QUERY EXPERTS WITHOUT INCREASING EXPERTS?"
Preprint DOI: 10.5281/zenodo.22747827

Maria-1B represents the first standalone foundation model trained from scratch for 20 Billion tokens to validate HyperGQE in an end-to-end pretraining regime.

If you use Maria-1B or the HyperGQE architecture in your work, please cite the preprint:

@article{yadki2026sparseattention,
  title={CAN WE IMPROVE THE PERFORMANCE OF SPARSE ATTENTION IN GROUPED QUERY EXPERTS WITHOUT INCREASING EXPERTS?},
  author={Yadki, Zahid Ahmed},
  journal={Zenodo},
  year={2026},
  month={September},
  doi={10.5281/zenodo.22747827},
  url={https://doi.org/10.5281/zenodo.22747827}
}