File size: 5,279 Bytes
ca31dfe | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | # MetaPruning: Meta Pruning via Graph Metanetworks
Implementation of **"Meta Pruning via Graph Metanetworks: A Universal Meta-Learning Framework for Network Pruning"**
**Paper:** https://arxiv.org/abs/2506.12041
## Core Idea
Unlike prior "learning to prune" methods that train per-model, MetaPruning **trains once, prunes forever**. A **GNN metanetwork** takes an entire neural network as input (converted to a graph where neurons = nodes, weights = edges), and outputs a transformed version that is easier to prune. After one feedforward pass + standard finetuning, you get SOTA pruning β no per-model special training.
## Architecture
### Network β Graph Bijection
```
Network Graph
Node = each output channel/neuron in Conv/Linear layers
Edge = connection between channels (conv, linear, residual skip)
Node feature = [weight_mean, weight_std, BN_weight, BN_bias, BN_run_mean, BN_run_var]
Edge feature = flattened conv kernel (padded to uniform size)
```
### GNN Metanetwork (Appendix B.1)
```
Node/Edge Encoder (MLP) β hidden_dim
β
N Γ PNAConv Message Passing Layers
β’ Message: m_ij = MLP^1(v_i) β MLP^2(v_j) β e_ij
β’ Message': m'_ji = MLP^1(v_j) β MLP^2(v_i) β (e_ij β EdgeInvertor)
β’ Aggregation: PNA([MEAN, STD, MAX, MIN]) via MLP_Aggr
β’ Update: v_i += aggr(m_ij) + aggr(m'_ji)
β’ Edge: e_ij += MLP^1(v_i) β MLP^2(v_j) β e_ij + ...
β
Node/Edge Decoder (MLP) β original dims
β
Residual: v_out = Ξ±Β·v_pred + v_in, e_out = Ξ²Β·e_pred + e_in
(Ξ± = Ξ² = 0.01, learns deltas only)
```
### Meta-Training Loop
1. Select a **data model** (pre-trained network)
2. Convert network β graph
3. Feed graph through GNN metanetwork β transformed graph
4. Convert transformed graph β new network
5. Compute **accuracy loss** (subset of training data) + **sparsity loss** (L1 weight penalty)
6. Backprop through metanetwork (data model params frozen)
### Inference Pipeline
```
Target Model β Graph β [Metanetwork] β Transformed Graph β New Model
β
Finetune (standard SGD, 100-200 epochs)
β
Prune (DepGraph / magnitude criterion)
β
Pruned Model (ready for inference)
```
## Files
| File | Description |
|------|-------------|
| `graph.py` | Network β Graph bijection (`resnet_to_graph`, `graph_to_resnet`, `create_transformed_model`) |
| `gnn.py` | PNAConv GNN (`Metanetwork`, `PNAConvLayer`, `EdgeInvertor`) |
| `train_metanetwork.py` | Meta-training loop on CIFAR-10 |
| `inference.py` | Inference: metanetwork β finetune β prune β evaluate |
## Usage
### 1. Meta-Train the Metanetwork
```bash
python -m metapruning.train_metanetwork \
--meta_epochs 100 \
--hidden_dim 32 \
--num_layers 3 \
--alpha 0.01 \
--beta 0.01 \
--lr 1e-3 \
--weight_decay 5e-4 \
--pruner_reg 10.0 \
--num_data_models 1 \
--pretrain_data_models \
--pretrain_epochs 100
```
This creates `checkpoints_metapruning/metanetwork.pt`.
### 2. Prune Any Target Model
```bash
python -m metapruning.inference \
--metanetwork_path checkpoints_metapruning/metanetwork.pt \
--target_model resnet56 \
--finetune_epochs 100 \
--prune_sparsity 0.5 \
--lr 0.01
```
## Paper Results
| Task | Base Acc | Pruned Acc (Ξ) | Pruned FLOPs |
|------|----------|----------------|--------------|
| ResNet56 / CIFAR-10 | 93.51% | **93.64%** (+0.13%) | 65.6% |
| VGG19 / CIFAR-100 | 73.65% | **69.75%** (β3.90%) | 88.83% |
| ResNet50 / ImageNet | 76.14% | **76.13%** (β0.01%) | 57.2% |
## Key Properties
- **Transferable**: Metanetwork trained on ResNet56/CIFAR-10 β prunes ResNet110, VGG, ViT on different datasets
- **One-shot pruning**: Single metanetwork feedforward + finetuning, no iterative pruning
- **Universal**: Applies to any CNN or ViT via graph bijection
- **Low cost (amortized)**: Expensive meta-training once, cheap pruning forever
## Hyperparameters
| Param | Default | Description |
|-------|---------|-------------|
| `hidden_dim` | 32 | GNN hidden dimension |
| `num_layers` | 3 | Message passing layers |
| `alpha` | 0.01 | Node feature residual coefficient |
| `beta` | 0.01 | Edge feature residual coefficient |
| `meta_epochs` | 100 | Meta-training epochs |
| `lr` | 1e-3 | Metanetwork learning rate |
| `weight_decay` | 5e-4 | AdamW weight decay |
| `pruner_reg` | 10.0 | Sparsity loss weight |
## Notes
- **GraphβModel differentiability**: The current `graph_to_resnet` uses in-place data modification (`module.weight.data += delta`). For fully differentiable meta-training, this should construct new `nn.Parameter` objects from GNN outputs instead. The current implementation demonstrates the architecture; production use should use `torch.autograd.Function` or `torch.nn.Parameter` construction for end-to-end differentiability.
- **For proper pruning**: Use `torch_pruning` (DepGraph) for structural pruning with dependency groups. The inference script includes a simple magnitude-based channel pruner as a placeholder.
- **Full paper**: Appendix B.1 contains the complete GNN architecture equations, and Appendix D contains per-task hyperparameters.
|