Zahid2005
/

Maria-1B

@@ -1,159 +1,166 @@
----
-language:
-- en
-license: mit
-tags:
-- pytorch
-- causal-lm
-- 1b
-- edge-ai
-- hypergqe
-- dynamic-attention
-- grouped-query-attention
-- text-generation
-pipeline_tag: text-generation
-inference: false
----
-# Maria-1B (20B Tokens Pretrained from Scratch)
-**Maria-1B** is a compact 1.07B parameter causal language model pretrained from scratch on a **20 Billion token budget**. It introduces **HyperGQE** (Dynamic Query Adaptation), a novel sparse attention architecture that breaks the traditional trade-off between KV-cache compression and query representational collapse.
-Unlike static Grouped Query Experts (GQE) which permanently drop unselected query heads, Maria dynamically conditions query projections using a lightweight context hypernetwork. This enables the model to achieve **higher throughput and lower memory usage than Dense GQA** while **surpassing Dense GQA in classification accuracy and STEM reasoning**.
----
-## Model Specifications
-| Parameter | Value |
-| :--- | :--- |
-| **Total Instantiated Parameters** | **1,073,515,904 (1.074 Billion)** |
-| **Active Parameters per Token** | **1,016,878,976 (1.017 Billion)** |
-| **Sparsity per Block** | 12 out of 16 query expert heads dropped per layer (56.6M inactive params) |
-| **Pretraining Token Budget** | **20 Billion Tokens** (Trained from scratch) |
-| **Hidden Dimension ($d_{\text{model}}$)** | 2,048 |
-| **Number of Layers** | 18 Transformer Blocks |
-| **Query Heads ($N_{\text{heads}}$)** | 16 (4 groups $\times$ 4 expert heads per group) |
-| **Key-Value Heads ($N_{\text{kv}}$)** | 4 (Grouped-Query Attention 4:1 compression) |
-| **Intermediate Size (FFN)** | 5,632 (SwiGLU gating) |
-| **Vocabulary Size** | 128,256 (Llama-3 BPE Tokenizer) |
-| **Context Length ($L_{\text{max}}$)** | 8,192 tokens with RoPE ($\theta = 500,000$) |
-| **Weight Precision** | `bfloat16` (~2.1 GB storage / ~650 MB quantized Q4_K_M) |
----
-## Hardware Profiling & Latency Benchmarks
-Measured on identical NVIDIA hardware comparing **Maria (HyperGQE)** against **Dense GQA** and **Static GQE**:
-### Throughput & Latency Comparison
-| Batch Size ($b$) | Sequence Length ($s$) | Dense GQA Latency | Static GQE Latency | **Maria (HyperGQE)** Latency | **Maria vs. Dense GQA Speedup** |
-| :---: | :---: | :---: | :---: | :---: | :---: |
-| 1 | 512 | 103.50 ms | 90.47 ms | **106.55 ms** | Competitive |
-| 1 | 1024 | 227.76 ms | 161.53 ms | **204.36 ms** | **+10.3% faster** |
-| 8 | 512 | 763.26 ms | 591.01 ms | **716.07 ms** | **+6.2% faster** |
-| 8 | 1024 | 1,750.23 ms | 1,161.57 ms | **1,421.50 ms** | **+18.8% faster** |
-| 16 | 512 | 1,475.45 ms | 1,114.95 ms | **1,361.72 ms** | **+7.7% faster** |
-| **16** | **1024 (Heavy Load)** | **3,526.88 ms** | **2,333.79 ms** | **3,049.37 ms** | **+13.5% FASTER** |
-### VRAM & Throughput (Heavy Load: $b=16, s=1024$)
-| Architecture | Peak VRAM | Throughput | Theoretical FLOPs |
-| :--- | :---: | :---: | :---: |
-| **Dense GQA Baseline** | 4,181.55 MB | 4,645 tokens/s | $3.60 \times 10^{13}$ |
-| **Static GQE (Sparse)** | 2,456.19 MB | 7,020 tokens/s | $3.31 \times 10^{13}$ |
-| **Maria (HyperGQE)** | **3,354.19 MB** *(827 MB less)* | **5,372 tokens/s** *(+15.6% higher)* | **$3.60 \times 10^{13}$** |
-> **Key Hardware Takeaway:** Under maximum sequence and batch workloads, Maria is **13.5% faster** than Dense GQA while consuming **827 MB less VRAM**, maintaining an identical FLOP budget.
----
-## Downstream Accuracy Benchmarks
-While static query routing (GQE) suffers from severe information loss due to hard-dropping query heads, Maria's dynamic hypernetwork compensates for unselected heads:
-### 1. Fine-Tuned Sentiment Classification (SST-2)
-| Architecture | Total Parameters | Active Parameters | SST-2 Accuracy | Margin vs. Dense |
-| :--- | :---: | :---: | :---: | :---: |
-| **GQE (Static Sparse)** | 875.39 M | 875.39 M | 75.92% | -4.24% *(Degradation)* |
-| **Dense GQA Baseline** | 946.04 M | 946.04 M | 80.16% | Baseline |
-| **Maria (HyperGQE)** | **924.13 M** | **843.35 M** | **80.85%** | **+0.69% HIGHER** |
-*Maria completely eliminates the 4.24% sparsity collapse of static GQE and pushes past the dense baseline by +0.69%.*
-### 2. Zero-Shot STEM Domain Recovery (16 Benchmarks)
-When compared against static sparse routing, Maria demonstrates pronounced accuracy gains in knowledge-intensive STEM domains:
-| Domain / Task | Static GQE Accuracy | Maria (HyperGQE) Accuracy | Net Margin ($\Delta$) |
-| :--- | :---: | :---: | :---: |
-| **MMLU: College Chemistry** | 19.0% | **28.0%** | **+9.0%** |
-| **MMLU: College Biology** | 22.2% | **26.3%** | **+4.1%** |
-| **MMLU: High School Biology** | 21.0% | **25.8%** | **+4.8%** |
-| **MMLU: High School Chemistry** | 19.2% | **22.7%** | **+3.5%** |
-| **MMLU: Medical Genetics** | 23.0% | **27.0%** | **+4.0%** |
-| **MMLU: Virology** | 24.1% | **28.3%** | **+4.2%** |
-| **ARC (Challenge)** | 21.8% | **23.3%** | **+1.5%** |
-| **ARC (Easy)** | 41.5% | **42.9%** | **+1.4%** |
-| **BoolQ** | 57.0% | **58.2%** | **+1.2%** |
----
-## Quickstart Usage
-You can load and run Maria directly with Hugging Face `transformers` using `trust_remote_code=True`:
-```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
-model_id = "your-username/Maria-1B-20Btokens"
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(
-    model_id,
-    torch_dtype=torch.bfloat16,
-    device_map="auto",
-    trust_remote_code=True
-)
-prompt = "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\nYou are Maria, a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\nWhat is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n"
-inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
-with torch.inference_mode():
-    outputs = model.generate(
-        **inputs,
-        max_new_tokens=60,
-        temperature=0.15,
-        do_sample=True,
-        pad_token_id=tokenizer.eos_token_id
-    )
-print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
-# Output: The capital of France is Paris.
-```
----
-## Quantization & Edge Deployment
-Because Maria has dense FFNs and structured query sparsity, it is an optimal model for on-device and mobile deployment:
-* **4-bit Quantization (GGUF / Q4_K_M)**: Compresses the model to **~650 MB**.
-* **RAM Requirement**: Requires only **~800 MB of RAM**, running at 45–60 tokens/sec on mobile Snapdragon/MediaTek/Apple Silicon chips with zero cloud API costs.
----
-## Citation
-If you use Maria or the HyperGQE architecture in your research, please cite:
-```bibtex
-@article{maria2026hypergqe,
-  title={HyperGQE: Dynamic Query Adaptation for Knowledge Recovery in Compressed Attention Models},
-  author={Yadki Zahid Ahmed and Contributors},
-  year={2026}
-}
 ```

+---
+language:
+- en
+license: mit
+tags:
+- pytorch
+- causal-lm
+- 1b
+- edge-ai
+- hypergqe
+- dynamic-attention
+- grouped-query-attention
+- text-generation
+pipeline_tag: text-generation
+inference: false
+---
+# Maria-1B (20B Tokens Pretrained from Scratch)
+**Maria-1B** is a compact 1.07B parameter causal language model pretrained from scratch on a **20 Billion token budget**. It introduces **HyperGQE** (Dynamic Query Adaptation), a novel sparse attention architecture that breaks the traditional trade-off between KV-cache compression and query representational collapse.
+Unlike static Grouped Query Experts (GQE) which permanently drop unselected query heads, Maria dynamically conditions query projections using a lightweight context hypernetwork. This enables the model to achieve **higher throughput and lower memory usage than Dense GQA** while **surpassing Dense GQA in classification accuracy and STEM reasoning**.
+* **Research Paper Preprint**: [https://doi.org/10.5281/zenodo.22747827](https://doi.org/10.5281/zenodo.22747827)
+* **Author**: Yadki Zahid Ahmed
+---
+## Model Specifications
+| Parameter | Value |
+| :--- | :--- |
+| **Total Instantiated Parameters** | **1,073,515,904 (1.074 Billion)** |
+| **Active Parameters per Token** | **1,016,878,976 (1.017 Billion)** |
+| **Sparsity per Block** | 12 out of 16 query expert heads dropped per layer (56.6M inactive params) |
+| **Pretraining Token Budget** | **20 Billion Tokens** (Trained from scratch) |
+| **Hidden Dimension ($d_{\text{model}}$)** | 2,048 |
+| **Number of Layers** | 18 Transformer Blocks |
+| **Query Heads ($N_{\text{heads}}$)** | 16 (4 groups $\times$ 4 expert heads per group) |
+| **Key-Value Heads ($N_{\text{kv}}$)** | 4 (Grouped-Query Attention 4:1 compression) |
+| **Intermediate Size (FFN)** | 5,632 (SwiGLU gating) |
+| **Vocabulary Size** | 128,256 (Llama-3 BPE Tokenizer) |
+| **Context Length ($L_{\text{max}}$)** | 8,192 tokens with RoPE ($\theta = 500,000$) |
+| **Weight Precision** | `bfloat16` (~2.1 GB storage / ~650 MB quantized Q4_K_M) |
+---
+## Hardware Profiling & Latency Benchmarks
+Measured on identical NVIDIA hardware comparing **Maria (HyperGQE)** against **Dense GQA** and **Static GQE**:
+### Throughput & Latency Comparison
+| Batch Size ($b$) | Sequence Length ($s$) | Dense GQA Latency | Static GQE Latency | **Maria (HyperGQE)** Latency | **Maria vs. Dense GQA Speedup** |
+| :---: | :---: | :---: | :---: | :---: | :---: |
+| 1 | 512 | 103.50 ms | 90.47 ms | **106.55 ms** | Competitive |
+| 1 | 1024 | 227.76 ms | 161.53 ms | **204.36 ms** | **+10.3% faster** |
+| 8 | 512 | 763.26 ms | 591.01 ms | **716.07 ms** | **+6.2% faster** |
+| 8 | 1024 | 1,750.23 ms | 1,161.57 ms | **1,421.50 ms** | **+18.8% faster** |
+| 16 | 512 | 1,475.45 ms | 1,114.95 ms | **1,361.72 ms** | **+7.7% faster** |
+| **16** | **1024 (Heavy Load)** | **3,526.88 ms** | **2,333.79 ms** | **3,049.37 ms** | **+13.5% FASTER** |
+### VRAM & Throughput (Heavy Load: $b=16, s=1024$)
+| Architecture | Peak VRAM | Throughput | Theoretical FLOPs |
+| :--- | :---: | :---: | :---: |
+| **Dense GQA Baseline** | 4,181.55 MB | 4,645 tokens/s | $3.60 \times 10^{13}$ |
+| **Static GQE (Sparse)** | 2,456.19 MB | 7,020 tokens/s | $3.31 \times 10^{13}$ |
+| **Maria (HyperGQE)** | **3,354.19 MB** *(827 MB less)* | **5,372 tokens/s** *(+15.6% higher)* | **$3.60 \times 10^{13}$** |
+> **Key Hardware Takeaway:** Under maximum sequence and batch workloads, Maria is **13.5% faster** than Dense GQA while consuming **827 MB less VRAM**, maintaining an identical FLOP budget.
+---
+## Downstream Accuracy Benchmarks
+While static query routing (GQE) suffers from severe information loss due to hard-dropping query heads, Maria's dynamic hypernetwork compensates for unselected heads:
+### 1. Fine-Tuned Sentiment Classification (SST-2)
+| Architecture | Total Parameters | Active Parameters | SST-2 Accuracy | Margin vs. Dense |
+| :--- | :---: | :---: | :---: | :---: |
+| **GQE (Static Sparse)** | 875.39 M | 875.39 M | 75.92% | -4.24% *(Degradation)* |
+| **Dense GQA Baseline** | 946.04 M | 946.04 M | 80.16% | Baseline |
+| **Maria (HyperGQE)** | **924.13 M** | **843.35 M** | **80.85%** | **+0.69% HIGHER** |
+*Maria completely eliminates the 4.24% sparsity collapse of static GQE and pushes past the dense baseline by +0.69%.*
+### 2. Zero-Shot STEM Domain Recovery (16 Benchmarks)
+When compared against static sparse routing, Maria demonstrates pronounced accuracy gains in knowledge-intensive STEM domains:
+| Domain / Task | Static GQE Accuracy | Maria (HyperGQE) Accuracy | Net Margin ($\Delta$) |
+| :--- | :---: | :---: | :---: |
+| **MMLU: College Chemistry** | 19.0% | **28.0%** | **+9.0%** |
+| **MMLU: College Biology** | 22.2% | **26.3%** | **+4.1%** |
+| **MMLU: High School Biology** | 21.0% | **25.8%** | **+4.8%** |
+| **MMLU: High School Chemistry** | 19.2% | **22.7%** | **+3.5%** |
+| **MMLU: Medical Genetics** | 23.0% | **27.0%** | **+4.0%** |
+| **MMLU: Virology** | 24.1% | **28.3%** | **+4.2%** |
+| **ARC (Challenge)** | 21.8% | **23.3%** | **+1.5%** |
+| **ARC (Easy)** | 41.5% | **42.9%** | **+1.4%** |
+| **BoolQ** | 57.0% | **58.2%** | **+1.2%** |
+---
+## Quickstart Usage
+You can load and run Maria directly with Hugging Face `transformers` using `trust_remote_code=True`:
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model_id = "your-username/Maria-1B-20Btokens"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True
+)
+prompt = "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\nYou are Maria, a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\nWhat is the capital of France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+with torch.inference_mode():
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=60,
+        temperature=0.15,
+        do_sample=True,
+        pad_token_id=tokenizer.eos_token_id
+    )
+print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
+# Output: The capital of France is Paris.
 ```
+---
+## Quantization & Edge Deployment
+Because Maria has dense FFNs and structured query sparsity, it is an optimal model for on-device and mobile deployment:
+* **4-bit Quantization (GGUF / Q4_K_M)**: Compresses the model to **~650 MB**.
+* **RAM Requirement**: Requires only **~800 MB of RAM**, running at 45–60 tokens/sec on mobile Snapdragon/MediaTek/Apple Silicon chips with zero cloud API costs.
+---
+## Citation
+If you use Maria-1B or build upon the architecture in your research, please cite the official research paper preprint:
+`ibtex
+@article{yadki2026sparseattention,
+  title={CAN WE IMPROVE THE PERFORMANCE OF SPARSE ATTENTION IN GROUPED QUERY EXPERTS WITHOUT INCREASING EXPERTS?},
+  author={Yadki, Zahid Ahmed},
+  journal={Zenodo},
+  year={2026},
+  month={September},
+  doi={10.5281/zenodo.22747827},
+  url={https://doi.org/10.5281/zenodo.22747827}
+}
+`