Release llama_2_7b-PATCH-35Sparse (mask only)

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+PATCH-Logo.png filter=lfs diff=lfs merge=lfs -text

NOTICE

@@ -0,0 +1,10 @@
+This repository distributes ONLY a binary pruning mask (which weight
+positions are kept vs. zeroed). It contains no weights from the base model.
+
+The mask is a derivative of the base model listed in the model card and is
+provided under that base model's license. You must independently obtain the base
+model and comply with its license and acceptable-use policy.
+
+Mask-generation code: MIT (https://github.com/Paramathic/patch).
+MaskLLM baselines are our own re-implementation; the MaskLLM method is due to
+Fang et al. (NVIDIA), NeurIPS 2024.

README.md

@@ -0,0 +1,132 @@
+---
+base_model:
+- meta-llama/Llama-2-7b-hf
+library_name: transformers
+license: llama2
+tags:
+- pruning
+- sparsity
+- 2:4-sparsity
+- patch
+- maskllm
+- mask
+pipeline_tag: text-generation
+---
+
+<div align="center">
+<img src="./PATCH-Logo.png" alt="PATCH" width="360">
+</div>
+
+# llama_2_7b-PATCH-35Sparse
+
+> **This checkpoint** (LLaMA-2 7B, PATCH-Tile, 35% sparsity): **50.08%** average zero-shot accuracy, **6.18** WikiText2 perplexity.
+
+[![Paper](https://img.shields.io/badge/arXiv-2509.23410-b31b1b.svg)](https://arxiv.org/abs/2509.23410) [![GitHub](https://img.shields.io/badge/GitHub-Paramathic%2Fpatch-black.svg?logo=github)](https://github.com/Paramathic/patch)
+
+This repository hosts a **mask only** release for the paper
+**[PATCH: Learnable Tile-level Hybrid Sparsity for LLMs](https://arxiv.org/abs/2509.23410)**.
+PATCH (Pruning with a Learnable Tile-level Configuration for Hybrid Sparsity) learns a structured mask on **frozen** pretrained weights, assigning each tile as dense (0% sparsity) or 2:4 sparse (50% sparsity) to hit a flexible global sparsity target while staying hardware-friendly.
+
+Because PATCH/MaskLLM keep the base weights **frozen**, we distribute *only the
+binary keep/prune mask* (bit-packed in `mask.npz`) - **no weight values**. You
+recover the sparse model by downloading the original base model and applying the
+mask.
+
+- Base model: [`meta-llama/Llama-2-7b-hf`](https://huggingface.co/meta-llama/Llama-2-7b-hf)
+- Method: **PATCH-Tile** &nbsp;|&nbsp; Target sparsity: **35%** &nbsp;|&nbsp; Pattern: **Dense / 2:4 tiles**
+- Measured mask sparsity: **35.08%** over 224 Linear layers (6,476,005,376 weights).
+
+<div align="center">
+<img src="./PATCH-Pipeline.svg" alt="PATCH pipeline" width="760">
+</div>
+
+## Results (LLaMA-2 7B)
+
+| Sparsity | Method | Pattern | Avg Acc (% ↑) | WikiText2 PPL (↓) |
+|---|---|---|---|---|
+| 0% | Dense | - | 54.61 | 5.12 |
+| 50% | Magnitude | 2:4 | 43.44 | 54.39 |
+| 50% | Wanda | 2:4 | 44.30 | 11.15 |
+| 50% | SparseGPT | 2:4 | 45.09 | 10.12 |
+| 50% | Thanos | 2:4 | 44.80 | 11.19 |
+| 50% | ProxSparse | 2:4 | 45.92 | 9.18 |
+| 50% | MaskLLM | 2:4 | 48.62 | 6.78 |
+| 45% | PATCH-Tile | Dense/2:4 | 48.99 | 6.55 |
+| 35% | **PATCH-Tile** ⭐ | Dense/2:4 | **50.08** | **6.18** |
+| 25% | PATCH-Tile | Dense/2:4 | 51.58 | 5.86 |
+
+
+Per-task zero-shot accuracy (%) for this checkpoint:
+
+| MMLU | PIQA | ARC-E | ARC-C | WinoG. | OBQA | RACE | HellaS. | **Average** |
+|---|---|---|---|---|---|---|---|---|
+| 29.93 | 76.71 | 70.88 | 36.95 | 65.67 | 28.20 | 39.33 | 52.96 | **50.08** |
+
+All numbers are from the PATCH paper ([arXiv:2509.23410](https://arxiv.org/abs/2509.23410)); accuracy
+is the average over MMLU, PIQA, ARC-Easy, ARC-Challenge, Winogrande, OpenBookQA,
+RACE and HellaSwag, evaluated with the LM-Evaluation-Harness. PPL is WikiText2.
+
+## Training hyper-parameters
+
+| Hyper-parameter | Value |
+|---|---|
+| Fine-tuning dataset | SlimPajama (2B tokens) |
+| Training steps | 2000 |
+| Global batch size | 256 |
+| Sequence length | 4096 |
+| Mask tile size | 128 x 128 (hardware tiles: 128x128 / 128x64 / 64x128 / 64x64) |
+| Logits init. | N(0, 0.014) |
+| Tile-logit prior | SparseGPT (strength 3) |
+| Regularization scope | Global (single target density) |
+| Evaluation | LM-Eval-Harness (8 zero-shot tasks) + WikiText2 PPL @ seqlen 4096 |
+| Hardware | 1 node x 4 GPUs, data parallel (HuggingFace Trainer) |
+| Optimizer | Adam |
+| Learning rate | 1e-4 |
+| Gumbel scaling (kappa) | 100 -> 500 |
+| Gumbel temp (tau) | 2 -> 0.05 |
+| Sparsity reg. (lambda1) | 3 |
+| Weight reg. (lambda2) | 0.1 |
+
+## How to use
+
+```python
+from huggingface_hub import hf_hub_download
+from transformers import AutoModelForCausalLM
+import torch
+from load_patch_mask import apply_patch_mask  # shipped in this repo
+
+npz = hf_hub_download(repo_id="mohammad-mozaffari/llama_2_7b-PATCH-35Sparse", filename="mask.npz")
+model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", torch_dtype=torch.bfloat16)
+apply_patch_mask(model, npz)   # zeroes the pruned weights in place
+```
+
+Or from the command line:
+
+```bash
+python load_patch_mask.py --base_model meta-llama/Llama-2-7b-hf --mask_repo mohammad-mozaffari/llama_2_7b-PATCH-35Sparse
+```
+
+Speedup on real hardware requires a 2:4-aware / hybrid sparse kernel; see the
+[GitHub repository](https://github.com/Paramathic/patch) and [STOICC](https://github.com/Paramathic/stoicc).
+
+## License
+
+The released mask is a derivative of the base model and is distributed under the
+base model's license (**`llama2`**). You must comply with that license and
+obtain access to the base model separately.
+
+> Use governed by the Llama 2 Community License.
+
+The mask-generation code is released under the MIT license (see the
+[PATCH repository](https://github.com/Paramathic/patch)).
+
+## Citation
+
+```bibtex
+@article{hourri2025patch,
+    title  = {PATCH: Learnable Tile-level Hybrid Sparsity for LLMs},
+    author = {Hourri, Younes and Mozaffari, Mohammad and Mehri Dehnavi, Maryam},
+    year   = 2025,
+    journal = {arXiv preprint arXiv:2509.23410}
+}
+```

load_patch_mask.py

@@ -0,0 +1,93 @@
+"""
+Apply a released PATCH / MaskLLM sparsity mask to a base model.
+
+The released repositories contain ONLY the binary keep/prune mask
+(bit-packed in `mask.npz`) - no weight values. To materialize the sparse
+model you download the original base model and zero out the pruned weights:
+
+    python load_patch_mask.py \
+        --base_model meta-llama/Llama-2-7b-hf \
+        --mask_repo mohammad-mozaffari/llama_2_7b-PATCH-45Sparse
+
+or use `apply_patch_mask(...)` from your own code.
+"""
+
+import argparse
+import json
+
+import numpy as np
+
+
+def load_mask_npz(npz_path):
+    """
+    Load `mask.npz` and return {weight_name: bool_ndarray} plus metadata.
+
+    The archive stores, per masked weight:
+      * "<name>"          : bit-packed uint8 array (np.packbits of the flat mask)
+      * "<name>.__shape__": original 2-D shape
+    and a single JSON blob under "__meta__".
+    """
+    data = np.load(npz_path, allow_pickle=False)
+    meta = json.loads(str(data["__meta__"])) if "__meta__" in data else {}
+    masks = {}
+    for key in data.files:
+        if key == "__meta__" or key.endswith(".__shape__"):
+            continue
+        shape = tuple(int(x) for x in data[f"{key}.__shape__"])
+        n = int(np.prod(shape))
+        flat = np.unpackbits(data[key], count=n).astype(bool)
+        masks[key] = flat.reshape(shape)
+    return masks, meta
+
+
+def apply_patch_mask(model, npz_path):
+    """Zero out pruned weights of `model` in place using the mask archive."""
+    import torch
+
+    masks, meta = load_mask_npz(npz_path)
+    sd = dict(model.named_parameters())
+    applied = 0
+    for name, mask in masks.items():
+        if name not in sd:
+            raise KeyError(
+                f"Mask key '{name}' not found in model parameters. "
+                f"Is '{meta.get('base_model', '?')}' the correct base model?"
+            )
+        w = sd[name]
+        m = torch.from_numpy(mask).to(device=w.device)
+        if tuple(m.shape) != tuple(w.shape):
+            raise ValueError(f"Shape mismatch for {name}: mask {tuple(m.shape)} vs weight {tuple(w.shape)}")
+        with torch.no_grad():
+            w.mul_(m.to(w.dtype))
+        applied += 1
+    print(f"Applied mask to {applied} weight tensors "
+          f"(target sparsity {meta.get('sparsity', '?')}, pattern {meta.get('pattern', '?')}).")
+    return model
+
+
+def _main():
+    from huggingface_hub import hf_hub_download
+    from transformers import AutoModelForCausalLM
+
+    p = argparse.ArgumentParser()
+    p.add_argument("--base_model", required=True, help="HF id of the original dense base model")
+    p.add_argument("--mask_repo", required=True, help="HF id of the released mask repo")
+    p.add_argument("--dtype", default="bfloat16")
+    args = p.parse_args()
+
+    import torch
+
+    npz_path = hf_hub_download(repo_id=args.mask_repo, filename="mask.npz")
+    model = AutoModelForCausalLM.from_pretrained(
+        args.base_model, torch_dtype=getattr(torch, args.dtype)
+    )
+    apply_patch_mask(model, npz_path)
+
+    total = sum(p.numel() for p in model.parameters())
+    nz = sum(int((p != 0).sum()) for p in model.parameters())
+    print(f"Overall model density after masking: {nz / total:.4f}")
+    return model
+
+
+if __name__ == "__main__":
+    _main()

mask.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cfb1e7360b3108b60ea8dcc4b6af01872eb973beb3c66c07b152908a94736d6
+size 405350657