| --- |
| license: apache-2.0 |
| library_name: pin |
| tags: |
| - weight-sharing |
| - model-compression |
| - parameter-efficient-fine-tuning |
| - convolutional-network |
| - cifar10 |
| - reproducible-research |
| --- |
| |
| # PIN: folded models, and what you can add to them |
|
|
| **0.9080 on CIFAR-10 from 270,277 stored values**, standing for |
| 2,415,919,104 connections: a fold of 8,939 times. Code for the PIN papers, |
| [archived on Zenodo](https://doi.org/10.5281/zenodo.21980454) with a DOI. |
|
|
| Code for the PIN papers. Every script here is one file that runs on its own, |
| downloads or verifies its own data, checks itself before doing any real |
| work, prints a table, and writes results to disk as it goes. There is no |
| package to install and no configuration to write: paste one into a hosted |
| notebook and it runs. |
|
|
| That shape is deliberate. The work is done from a phone as often as from a |
| desk, so a script that needs four files and a setup step is a script that |
| does not get run. |
|
|
| --- |
|
|
| ## What a fold is, in one paragraph |
|
|
| An ordinary weight matrix stores one number per connection. A folded one |
| stores a much smaller set of VALUES and a PARTITION saying which connections |
| share which value: |
|
|
| W[i, j] = v[ idx(i, j) ] |
| |
| A convolution is one choice of `idx`. So are a patch layer, a butterfly, a |
| block-circulant matrix, and an arbitrary tying. The framework separates the |
| partition from the values, which makes the architecture a data structure |
| rather than a code path, and lets a model be reshaped by swapping the index |
| while keeping what it learned. |
|
|
| --- |
|
|
| ## Start here |
|
|
| | you want to | run | |
| |---|---| |
| | see a folded model reach 0.9080 on CIFAR-10 | `resnet.py` | |
| | see the fast path checked against the general one | `sliceconv.py` | |
| | build the saved bases everything else reads | `testbed_all.py` | |
| | see a member added to a frozen base | `injection.py` | |
| | see a model fail to notice its own ignorance | `confidence_member.py` | |
|
|
| `resnet.py` takes about half an hour on a modest GPU and prints its own |
| timing estimate after ten measured steps, so you can stop it early if that |
| is not what you want. |
|
|
| --- |
|
|
| ## Every figure and table in v5, and what produced it |
|
|
| | section | claim | script | |
| |---|---|---| |
| | §1 | 0.9080 on CIFAR-10 from 270,277 values | `resnet.py` | |
| | §2.1 | pooling, and the depth curve inverting | `pooling.py` | |
| | §2.2 | depth has a low optimum; reach does not predict it | `grow_depth.py`, `depth_law.py` | |
| | §2.3 | the stacks are chaotic and self-regulating | `criticality.py`, `homeostat.py` | |
| | §2.4 | blindness, visible in the gradients | `strain.py` | |
| | §3 | four curricula, four nulls | `grown.py`, `curriculum.py`, `scramble.py` | |
| | §4.1 | injection at exactly zero disturbance | `injection.py` | |
| | §4.2 | head capacity, and containment withdrawn | `head_capacity.py`, `provisioning.py`, `bodies.py` | |
| | §4.3 | a member from five examples, in 0.6 ms | `sample_efficiency.py` | |
| | §4.4 | the weight as a dial, and negative weights | `member_weight.py` | |
| | §4.5 | correction is bounded; extension is not | `gate.py`, `ignorance.py` | |
| | §4.6 | ignorance is invisible at the output | `confidence_member.py` | |
| | §4.7 | qualifying a member: concede or override | `qualify.py` | |
| | §4.9 | borrowing across bases | `borrow.py`, `calibration.py` | |
| | §4.10 | what a buyer is actually paying for | `marketplace.py` | |
| | §5 | the cascade, and confidence signals | `cascade.py`, `routing_signals.py` | |
| | §6 | determinism, and the 0.002 noise floor | `determinism.py` | |
|
|
| Supporting: `testbed_all.py` trains the saved bases; `sliceconv.py` holds the |
| fast convolution and its check; `switch.py` measures what reshaping costs; |
| `make_v5_figures.py` draws the six figures from the measured values. |
|
|
| --- |
|
|
| ## Reproducibility |
|
|
| Folded training was not reproducible until recently and now is. GPU atomic |
| scatter completes in hardware order and floating-point addition is not |
| associative, so the same seed produced models differing by about 0.002 in |
| final accuracy. A fixed-order reduction is exact, and it is cheap here |
| because the partition never changes during training, so the sort is paid |
| once for the whole run. `determinism.py` measures both. |
|
|
| Two protocol notes, learned the hard way and worth inheriting: |
|
|
| - **Compare paired within a seed.** An unpaired spread misled this project |
| three times, once hiding a result at 2.6 deviations. |
| - **Read best epoch, not final, whenever an arm overfits.** A control with |
| more epochs to overfit with once produced an apparent +0.0051 that was |
| really +0.0001. |
|
|
| --- |
|
|
| ## What is not here |
|
|
| The papers withdraw things, and the code keeps the scripts that produced |
| the withdrawn results as well as the corrections. That is deliberate. A |
| project that never retracted anything is one nobody checked. |
|
|
| Superseded versions of a script are not included; where a fault mattered it |
| is described in the file that replaced it, in a comment at the place it |
| happened. Several of those comments are the most useful documentation in |
| the repository: an index sized on uniform test data that asked for 149 GB |
| against a real one, a member applied inside a suppression dead zone, a |
| storage figure that omitted the head it was compared against. |
|
|
| --- |
|
|
| ## Requirements |
|
|
| Python 3, NumPy, and CuPy if a GPU is present. Every script falls back to |
| NumPy on the CPU, more slowly. Datasets are fetched through Keras and |
| cached; `resnet.py` caches to a directory you choose so a restarted session |
| does not download CIFAR again. |
|
|
| Nothing else. There is no framework dependency, no custom kernel, and no |
| compiled extension. A folded convolution is expressed in operations any |
| array library already has, which is part of the claim. |
|
|
| --- |
|
|
| ## Citing |
|
|
| The papers carry the results and the reasoning; this repository carries what |
| produced them. Cite the paper for a finding and this for the method that |
| produced it. |
|
|
| ## Licence |
|
|
| Apache 2.0. It permits commercial use and includes an explicit patent |
| grant, which matters more here than the licence text: the framework |
| describes mechanisms someone may want to build on, and a permissive licence |
| without a patent clause leaves that ambiguous. |
|
|
