|
Download metapruning/README.md from ESPR3SS0/neural-pruning-impl: direct link, hf CLI and curl.
- Browser
- Download file 5.28 kB
-
https://huggingface.co/ESPR3SS0/neural-pruning-impl/resolve/main/metapruning/README.md
- Command line
-
hf download hf://ESPR3SS0/neural-pruning-impl/metapruning/README.md
-
curl -L -o README.md https://huggingface.co/ESPR3SS0/neural-pruning-impl/resolve/main/metapruning/README.md
5.28 kB
MetaPruning: Meta Pruning via Graph Metanetworks
Implementation of "Meta Pruning via Graph Metanetworks: A Universal Meta-Learning Framework for Network Pruning"
Paper: https://arxiv.org/abs/2506.12041
Core Idea
Unlike prior "learning to prune" methods that train per-model, MetaPruning trains once, prunes forever. A GNN metanetwork takes an entire neural network as input (converted to a graph where neurons = nodes, weights = edges), and outputs a transformed version that is easier to prune. After one feedforward pass + standard finetuning, you get SOTA pruning β no per-model special training.
Architecture
Network β Graph Bijection
Network Graph
Node = each output channel/neuron in Conv/Linear layers
Edge = connection between channels (conv, linear, residual skip)
Node feature = [weight_mean, weight_std, BN_weight, BN_bias, BN_run_mean, BN_run_var]
Edge feature = flattened conv kernel (padded to uniform size)
GNN Metanetwork (Appendix B.1)
Node/Edge Encoder (MLP) β hidden_dim
β
N Γ PNAConv Message Passing Layers
β’ Message: m_ij = MLP^1(v_i) β MLP^2(v_j) β e_ij
β’ Message': m'_ji = MLP^1(v_j) β MLP^2(v_i) β (e_ij β EdgeInvertor)
β’ Aggregation: PNA([MEAN, STD, MAX, MIN]) via MLP_Aggr
β’ Update: v_i += aggr(m_ij) + aggr(m'_ji)
β’ Edge: e_ij += MLP^1(v_i) β MLP^2(v_j) β e_ij + ...
β
Node/Edge Decoder (MLP) β original dims
β
Residual: v_out = Ξ±Β·v_pred + v_in, e_out = Ξ²Β·e_pred + e_in
(Ξ± = Ξ² = 0.01, learns deltas only)
Meta-Training Loop
- Select a data model (pre-trained network)
- Convert network β graph
- Feed graph through GNN metanetwork β transformed graph
- Convert transformed graph β new network
- Compute accuracy loss (subset of training data) + sparsity loss (L1 weight penalty)
- Backprop through metanetwork (data model params frozen)
Inference Pipeline
Target Model β Graph β [Metanetwork] β Transformed Graph β New Model
β
Finetune (standard SGD, 100-200 epochs)
β
Prune (DepGraph / magnitude criterion)
β
Pruned Model (ready for inference)
Files
| File | Description |
|---|---|
graph.py |
Network β Graph bijection (resnet_to_graph, graph_to_resnet, create_transformed_model) |
gnn.py |
PNAConv GNN (Metanetwork, PNAConvLayer, EdgeInvertor) |
train_metanetwork.py |
Meta-training loop on CIFAR-10 |
inference.py |
Inference: metanetwork β finetune β prune β evaluate |
Usage
1. Meta-Train the Metanetwork
python -m metapruning.train_metanetwork \
--meta_epochs 100 \
--hidden_dim 32 \
--num_layers 3 \
--alpha 0.01 \
--beta 0.01 \
--lr 1e-3 \
--weight_decay 5e-4 \
--pruner_reg 10.0 \
--num_data_models 1 \
--pretrain_data_models \
--pretrain_epochs 100
This creates checkpoints_metapruning/metanetwork.pt.
2. Prune Any Target Model
python -m metapruning.inference \
--metanetwork_path checkpoints_metapruning/metanetwork.pt \
--target_model resnet56 \
--finetune_epochs 100 \
--prune_sparsity 0.5 \
--lr 0.01
Paper Results
| Task | Base Acc | Pruned Acc (Ξ) | Pruned FLOPs |
|---|---|---|---|
| ResNet56 / CIFAR-10 | 93.51% | 93.64% (+0.13%) | 65.6% |
| VGG19 / CIFAR-100 | 73.65% | 69.75% (β3.90%) | 88.83% |
| ResNet50 / ImageNet | 76.14% | 76.13% (β0.01%) | 57.2% |
Key Properties
- Transferable: Metanetwork trained on ResNet56/CIFAR-10 β prunes ResNet110, VGG, ViT on different datasets
- One-shot pruning: Single metanetwork feedforward + finetuning, no iterative pruning
- Universal: Applies to any CNN or ViT via graph bijection
- Low cost (amortized): Expensive meta-training once, cheap pruning forever
Hyperparameters
| Param | Default | Description |
|---|---|---|
hidden_dim |
32 | GNN hidden dimension |
num_layers |
3 | Message passing layers |
alpha |
0.01 | Node feature residual coefficient |
beta |
0.01 | Edge feature residual coefficient |
meta_epochs |
100 | Meta-training epochs |
lr |
1e-3 | Metanetwork learning rate |
weight_decay |
5e-4 | AdamW weight decay |
pruner_reg |
10.0 | Sparsity loss weight |
Notes
- GraphβModel differentiability: The current
graph_to_resnetuses in-place data modification (module.weight.data += delta). For fully differentiable meta-training, this should construct newnn.Parameterobjects from GNN outputs instead. The current implementation demonstrates the architecture; production use should usetorch.autograd.Functionortorch.nn.Parameterconstruction for end-to-end differentiability. - For proper pruning: Use
torch_pruning(DepGraph) for structural pruning with dependency groups. The inference script includes a simple magnitude-based channel pruner as a placeholder. - Full paper: Appendix B.1 contains the complete GNN architecture equations, and Appendix D contains per-task hyperparameters.