File size: 5,279 Bytes
ca31dfe
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
# MetaPruning: Meta Pruning via Graph Metanetworks

Implementation of **"Meta Pruning via Graph Metanetworks: A Universal Meta-Learning Framework for Network Pruning"**

**Paper:** https://arxiv.org/abs/2506.12041

## Core Idea

Unlike prior "learning to prune" methods that train per-model, MetaPruning **trains once, prunes forever**. A **GNN metanetwork** takes an entire neural network as input (converted to a graph where neurons = nodes, weights = edges), and outputs a transformed version that is easier to prune. After one feedforward pass + standard finetuning, you get SOTA pruning β€” no per-model special training.

## Architecture

### Network ↔ Graph Bijection

```
Network Graph
  Node  = each output channel/neuron in Conv/Linear layers
  Edge  = connection between channels (conv, linear, residual skip)
  Node feature = [weight_mean, weight_std, BN_weight, BN_bias, BN_run_mean, BN_run_var]
  Edge feature = flattened conv kernel (padded to uniform size)
```

### GNN Metanetwork (Appendix B.1)

```
Node/Edge Encoder (MLP) β†’ hidden_dim
        ↓
N Γ— PNAConv Message Passing Layers
  β€’ Message:  m_ij = MLP^1(v_i) βŠ™ MLP^2(v_j) βŠ™ e_ij
  β€’ Message': m'_ji = MLP^1(v_j) βŠ™ MLP^2(v_i) βŠ™ (e_ij βŠ™ EdgeInvertor)
  β€’ Aggregation: PNA([MEAN, STD, MAX, MIN]) via MLP_Aggr
  β€’ Update: v_i += aggr(m_ij) + aggr(m'_ji)
  β€’ Edge: e_ij += MLP^1(v_i) βŠ™ MLP^2(v_j) βŠ™ e_ij + ...
        ↓
Node/Edge Decoder (MLP) β†’ original dims
        ↓
Residual: v_out = Ξ±Β·v_pred + v_in,  e_out = Ξ²Β·e_pred + e_in
          (Ξ± = Ξ² = 0.01, learns deltas only)
```

### Meta-Training Loop

1. Select a **data model** (pre-trained network)
2. Convert network β†’ graph
3. Feed graph through GNN metanetwork β†’ transformed graph
4. Convert transformed graph β†’ new network
5. Compute **accuracy loss** (subset of training data) + **sparsity loss** (L1 weight penalty)
6. Backprop through metanetwork (data model params frozen)

### Inference Pipeline

```
Target Model β†’ Graph β†’ [Metanetwork] β†’ Transformed Graph β†’ New Model
                          ↓
                    Finetune (standard SGD, 100-200 epochs)
                          ↓
                    Prune (DepGraph / magnitude criterion)
                          ↓
                    Pruned Model (ready for inference)
```

## Files

| File | Description |
|------|-------------|
| `graph.py` | Network ↔ Graph bijection (`resnet_to_graph`, `graph_to_resnet`, `create_transformed_model`) |
| `gnn.py` | PNAConv GNN (`Metanetwork`, `PNAConvLayer`, `EdgeInvertor`) |
| `train_metanetwork.py` | Meta-training loop on CIFAR-10 |
| `inference.py` | Inference: metanetwork β†’ finetune β†’ prune β†’ evaluate |

## Usage

### 1. Meta-Train the Metanetwork

```bash
python -m metapruning.train_metanetwork \
    --meta_epochs 100 \
    --hidden_dim 32 \
    --num_layers 3 \
    --alpha 0.01 \
    --beta 0.01 \
    --lr 1e-3 \
    --weight_decay 5e-4 \
    --pruner_reg 10.0 \
    --num_data_models 1 \
    --pretrain_data_models \
    --pretrain_epochs 100
```

This creates `checkpoints_metapruning/metanetwork.pt`.

### 2. Prune Any Target Model

```bash
python -m metapruning.inference \
    --metanetwork_path checkpoints_metapruning/metanetwork.pt \
    --target_model resnet56 \
    --finetune_epochs 100 \
    --prune_sparsity 0.5 \
    --lr 0.01
```

## Paper Results

| Task | Base Acc | Pruned Acc (Ξ”) | Pruned FLOPs |
|------|----------|----------------|--------------|
| ResNet56 / CIFAR-10 | 93.51% | **93.64%** (+0.13%) | 65.6% |
| VGG19 / CIFAR-100 | 73.65% | **69.75%** (βˆ’3.90%) | 88.83% |
| ResNet50 / ImageNet | 76.14% | **76.13%** (βˆ’0.01%) | 57.2% |

## Key Properties

- **Transferable**: Metanetwork trained on ResNet56/CIFAR-10 β†’ prunes ResNet110, VGG, ViT on different datasets
- **One-shot pruning**: Single metanetwork feedforward + finetuning, no iterative pruning
- **Universal**: Applies to any CNN or ViT via graph bijection
- **Low cost (amortized)**: Expensive meta-training once, cheap pruning forever

## Hyperparameters

| Param | Default | Description |
|-------|---------|-------------|
| `hidden_dim` | 32 | GNN hidden dimension |
| `num_layers` | 3 | Message passing layers |
| `alpha` | 0.01 | Node feature residual coefficient |
| `beta` | 0.01 | Edge feature residual coefficient |
| `meta_epochs` | 100 | Meta-training epochs |
| `lr` | 1e-3 | Metanetwork learning rate |
| `weight_decay` | 5e-4 | AdamW weight decay |
| `pruner_reg` | 10.0 | Sparsity loss weight |

## Notes

- **Graph→Model differentiability**: The current `graph_to_resnet` uses in-place data modification (`module.weight.data += delta`). For fully differentiable meta-training, this should construct new `nn.Parameter` objects from GNN outputs instead. The current implementation demonstrates the architecture; production use should use `torch.autograd.Function` or `torch.nn.Parameter` construction for end-to-end differentiability.
- **For proper pruning**: Use `torch_pruning` (DepGraph) for structural pruning with dependency groups. The inference script includes a simple magnitude-based channel pruner as a placeholder.
- **Full paper**: Appendix B.1 contains the complete GNN architecture equations, and Appendix D contains per-task hyperparameters.