Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,69 @@
|
|
| 1 |
---
|
| 2 |
license: cc-by-nc-nd-4.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: cc-by-nc-nd-4.0
|
| 3 |
+
tags:
|
| 4 |
+
- pytorch
|
| 5 |
+
- safetensors
|
| 6 |
+
- weight-reconstruction
|
| 7 |
+
- residual-prediction
|
| 8 |
+
- precision-recovery
|
| 9 |
+
- universal
|
| 10 |
+
- out-of-distribution
|
| 11 |
+
- bf16
|
| 12 |
+
- fp16
|
| 13 |
+
- fp32
|
| 14 |
+
- neural-network-weights
|
| 15 |
+
library_name: pytorch
|
| 16 |
+
pipeline_tag: other
|
| 17 |
---
|
| 18 |
+
|
| 19 |
+
# UniRes
|
| 20 |
+
|
| 21 |
+
**Universal Residual Predictor for Neural Network Weight Reconstruction**
|
| 22 |
+
|
| 23 |
+
UniRes is a universal residual network that reconstructs approximate higher-precision (FP32 / FP64) weights from low-precision (BF16 / FP16) tensors of **arbitrary shape and size**. It operates completely architecture-agnostically and generalises across model families never seen during training (Llama, BERT, T5, Gemma, VAE, CLIP, etc.).
|
| 24 |
+
|
| 25 |
+
## Model Description
|
| 26 |
+
|
| 27 |
+
Given a low-precision tensor \(W_{\text{low}}\) UniRes predicts a residual \(R\) such that
|
| 28 |
+
|
| 29 |
+
\[
|
| 30 |
+
\hat{W} = W_{\text{low}} + R
|
| 31 |
+
\]
|
| 32 |
+
|
| 33 |
+
The final reconstructed weights are emitted in the user-selected precision (FP32 or FP64). The network accepts tensors of any rank and dimension; no architecture-specific adapters are required.
|
| 34 |
+
|
| 35 |
+
## Evaluation (Completely Out-of-Distribution)
|
| 36 |
+
|
| 37 |
+
**Ground-truth vs BF16 cast**
|
| 38 |
+
- Tensors: 194
|
| 39 |
+
- Elements: 126 892 531
|
| 40 |
+
- Total absolute error: \(6.6856248102 \times 10^{2}\)
|
| 41 |
+
- Maximum absolute error: \(3.4952163696 \times 10^{-3}\)
|
| 42 |
+
|
| 43 |
+
**Ground-truth vs UniRes prediction**
|
| 44 |
+
- Total absolute error: \(1.4362314089 \times 10^{3}\)
|
| 45 |
+
- Maximum absolute error: \(3.4807920456 \times 10^{-3}\)
|
| 46 |
+
|
| 47 |
+
**Per-element comparison**
|
| 48 |
+
- UniRes better than BF16: 10 539 994 elements (**8.306 %**)
|
| 49 |
+
- BF16 better than UniRes: 116 350 729 elements (91.692 %)
|
| 50 |
+
- Equal: 1 808 elements (0.001 %)
|
| 51 |
+
|
| 52 |
+
Ratio of total absolute errors: UniRes / BF16 = **2.148×**
|
| 53 |
+
|
| 54 |
+
While UniRes does not improve aggregate absolute error, it produces a strictly lower absolute deviation on more than ten million individual elements.
|
| 55 |
+
|
| 56 |
+
## Interpretation
|
| 57 |
+
|
| 58 |
+
The minority of element-wise wins is consistent with the persistence of correlated Gaussian structure in neural-network weight matrices well beyond initialisation (Hirst & Ramgoolam, 2026). These residual correlations—local spatial structure, higher-order moments and permutation-invariant dependencies—transfer to a limited degree across architectures, allowing a universal residual predictor to capture useful signal on a non-negligible fraction of elements.
|
| 59 |
+
|
| 60 |
+
## Citation
|
| 61 |
+
|
| 62 |
+
If you use UniRes, please cite:
|
| 63 |
+
|
| 64 |
+
```bibtex
|
| 65 |
+
@article{unires2026,
|
| 66 |
+
title={Persistent Correlated Structure in Neural Network Weights Enables Partial Out-of-Distribution Residual Prediction from Low-Precision Representations},
|
| 67 |
+
author={UniRes Authors},
|
| 68 |
+
year={2026}
|
| 69 |
+
}
|