|
Download metapruning/README.md from ESPR3SS0/neural-pruning-impl: direct link, hf CLI and curl.
- Browser
- Download file 5.28 kB
-
https://huggingface.co/ESPR3SS0/neural-pruning-impl/resolve/main/metapruning/README.md
- Command line
-
hf download hf://ESPR3SS0/neural-pruning-impl/metapruning/README.md
-
curl -L -o README.md https://huggingface.co/ESPR3SS0/neural-pruning-impl/resolve/main/metapruning/README.md
5.28 kB
| # MetaPruning: Meta Pruning via Graph Metanetworks | |
| Implementation of **"Meta Pruning via Graph Metanetworks: A Universal Meta-Learning Framework for Network Pruning"** | |
| **Paper:** https://arxiv.org/abs/2506.12041 | |
| ## Core Idea | |
| Unlike prior "learning to prune" methods that train per-model, MetaPruning **trains once, prunes forever**. A **GNN metanetwork** takes an entire neural network as input (converted to a graph where neurons = nodes, weights = edges), and outputs a transformed version that is easier to prune. After one feedforward pass + standard finetuning, you get SOTA pruning β no per-model special training. | |
| ## Architecture | |
| ### Network β Graph Bijection | |
| ``` | |
| Network Graph | |
| Node = each output channel/neuron in Conv/Linear layers | |
| Edge = connection between channels (conv, linear, residual skip) | |
| Node feature = [weight_mean, weight_std, BN_weight, BN_bias, BN_run_mean, BN_run_var] | |
| Edge feature = flattened conv kernel (padded to uniform size) | |
| ``` | |
| ### GNN Metanetwork (Appendix B.1) | |
| ``` | |
| Node/Edge Encoder (MLP) β hidden_dim | |
| β | |
| N Γ PNAConv Message Passing Layers | |
| β’ Message: m_ij = MLP^1(v_i) β MLP^2(v_j) β e_ij | |
| β’ Message': m'_ji = MLP^1(v_j) β MLP^2(v_i) β (e_ij β EdgeInvertor) | |
| β’ Aggregation: PNA([MEAN, STD, MAX, MIN]) via MLP_Aggr | |
| β’ Update: v_i += aggr(m_ij) + aggr(m'_ji) | |
| β’ Edge: e_ij += MLP^1(v_i) β MLP^2(v_j) β e_ij + ... | |
| β | |
| Node/Edge Decoder (MLP) β original dims | |
| β | |
| Residual: v_out = Ξ±Β·v_pred + v_in, e_out = Ξ²Β·e_pred + e_in | |
| (Ξ± = Ξ² = 0.01, learns deltas only) | |
| ``` | |
| ### Meta-Training Loop | |
| 1. Select a **data model** (pre-trained network) | |
| 2. Convert network β graph | |
| 3. Feed graph through GNN metanetwork β transformed graph | |
| 4. Convert transformed graph β new network | |
| 5. Compute **accuracy loss** (subset of training data) + **sparsity loss** (L1 weight penalty) | |
| 6. Backprop through metanetwork (data model params frozen) | |
| ### Inference Pipeline | |
| ``` | |
| Target Model β Graph β [Metanetwork] β Transformed Graph β New Model | |
| β | |
| Finetune (standard SGD, 100-200 epochs) | |
| β | |
| Prune (DepGraph / magnitude criterion) | |
| β | |
| Pruned Model (ready for inference) | |
| ``` | |
| ## Files | |
| | File | Description | | |
| |------|-------------| | |
| | `graph.py` | Network β Graph bijection (`resnet_to_graph`, `graph_to_resnet`, `create_transformed_model`) | | |
| | `gnn.py` | PNAConv GNN (`Metanetwork`, `PNAConvLayer`, `EdgeInvertor`) | | |
| | `train_metanetwork.py` | Meta-training loop on CIFAR-10 | | |
| | `inference.py` | Inference: metanetwork β finetune β prune β evaluate | | |
| ## Usage | |
| ### 1. Meta-Train the Metanetwork | |
| ```bash | |
| python -m metapruning.train_metanetwork \ | |
| --meta_epochs 100 \ | |
| --hidden_dim 32 \ | |
| --num_layers 3 \ | |
| --alpha 0.01 \ | |
| --beta 0.01 \ | |
| --lr 1e-3 \ | |
| --weight_decay 5e-4 \ | |
| --pruner_reg 10.0 \ | |
| --num_data_models 1 \ | |
| --pretrain_data_models \ | |
| --pretrain_epochs 100 | |
| ``` | |
| This creates `checkpoints_metapruning/metanetwork.pt`. | |
| ### 2. Prune Any Target Model | |
| ```bash | |
| python -m metapruning.inference \ | |
| --metanetwork_path checkpoints_metapruning/metanetwork.pt \ | |
| --target_model resnet56 \ | |
| --finetune_epochs 100 \ | |
| --prune_sparsity 0.5 \ | |
| --lr 0.01 | |
| ``` | |
| ## Paper Results | |
| | Task | Base Acc | Pruned Acc (Ξ) | Pruned FLOPs | | |
| |------|----------|----------------|--------------| | |
| | ResNet56 / CIFAR-10 | 93.51% | **93.64%** (+0.13%) | 65.6% | | |
| | VGG19 / CIFAR-100 | 73.65% | **69.75%** (β3.90%) | 88.83% | | |
| | ResNet50 / ImageNet | 76.14% | **76.13%** (β0.01%) | 57.2% | | |
| ## Key Properties | |
| - **Transferable**: Metanetwork trained on ResNet56/CIFAR-10 β prunes ResNet110, VGG, ViT on different datasets | |
| - **One-shot pruning**: Single metanetwork feedforward + finetuning, no iterative pruning | |
| - **Universal**: Applies to any CNN or ViT via graph bijection | |
| - **Low cost (amortized)**: Expensive meta-training once, cheap pruning forever | |
| ## Hyperparameters | |
| | Param | Default | Description | | |
| |-------|---------|-------------| | |
| | `hidden_dim` | 32 | GNN hidden dimension | | |
| | `num_layers` | 3 | Message passing layers | | |
| | `alpha` | 0.01 | Node feature residual coefficient | | |
| | `beta` | 0.01 | Edge feature residual coefficient | | |
| | `meta_epochs` | 100 | Meta-training epochs | | |
| | `lr` | 1e-3 | Metanetwork learning rate | | |
| | `weight_decay` | 5e-4 | AdamW weight decay | | |
| | `pruner_reg` | 10.0 | Sparsity loss weight | | |
| ## Notes | |
| - **GraphβModel differentiability**: The current `graph_to_resnet` uses in-place data modification (`module.weight.data += delta`). For fully differentiable meta-training, this should construct new `nn.Parameter` objects from GNN outputs instead. The current implementation demonstrates the architecture; production use should use `torch.autograd.Function` or `torch.nn.Parameter` construction for end-to-end differentiability. | |
| - **For proper pruning**: Use `torch_pruning` (DepGraph) for structural pruning with dependency groups. The inference script includes a simple magnitude-based channel pruner as a placeholder. | |
| - **Full paper**: Appendix B.1 contains the complete GNN architecture equations, and Appendix D contains per-task hyperparameters. | |