Felldude commited on
Commit
07c762f
·
verified ·
1 Parent(s): 2f5c483

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +66 -0
README.md CHANGED
@@ -1,3 +1,69 @@
1
  ---
2
  license: cc-by-nc-nd-4.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: cc-by-nc-nd-4.0
3
+ tags:
4
+ - pytorch
5
+ - safetensors
6
+ - weight-reconstruction
7
+ - residual-prediction
8
+ - precision-recovery
9
+ - universal
10
+ - out-of-distribution
11
+ - bf16
12
+ - fp16
13
+ - fp32
14
+ - neural-network-weights
15
+ library_name: pytorch
16
+ pipeline_tag: other
17
  ---
18
+
19
+ # UniRes
20
+
21
+ **Universal Residual Predictor for Neural Network Weight Reconstruction**
22
+
23
+ UniRes is a universal residual network that reconstructs approximate higher-precision (FP32 / FP64) weights from low-precision (BF16 / FP16) tensors of **arbitrary shape and size**. It operates completely architecture-agnostically and generalises across model families never seen during training (Llama, BERT, T5, Gemma, VAE, CLIP, etc.).
24
+
25
+ ## Model Description
26
+
27
+ Given a low-precision tensor \(W_{\text{low}}\) UniRes predicts a residual \(R\) such that
28
+
29
+ \[
30
+ \hat{W} = W_{\text{low}} + R
31
+ \]
32
+
33
+ The final reconstructed weights are emitted in the user-selected precision (FP32 or FP64). The network accepts tensors of any rank and dimension; no architecture-specific adapters are required.
34
+
35
+ ## Evaluation (Completely Out-of-Distribution)
36
+
37
+ **Ground-truth vs BF16 cast**
38
+ - Tensors: 194
39
+ - Elements: 126 892 531
40
+ - Total absolute error: \(6.6856248102 \times 10^{2}\)
41
+ - Maximum absolute error: \(3.4952163696 \times 10^{-3}\)
42
+
43
+ **Ground-truth vs UniRes prediction**
44
+ - Total absolute error: \(1.4362314089 \times 10^{3}\)
45
+ - Maximum absolute error: \(3.4807920456 \times 10^{-3}\)
46
+
47
+ **Per-element comparison**
48
+ - UniRes better than BF16: 10 539 994 elements (**8.306 %**)
49
+ - BF16 better than UniRes: 116 350 729 elements (91.692 %)
50
+ - Equal: 1 808 elements (0.001 %)
51
+
52
+ Ratio of total absolute errors: UniRes / BF16 = **2.148×**
53
+
54
+ While UniRes does not improve aggregate absolute error, it produces a strictly lower absolute deviation on more than ten million individual elements.
55
+
56
+ ## Interpretation
57
+
58
+ The minority of element-wise wins is consistent with the persistence of correlated Gaussian structure in neural-network weight matrices well beyond initialisation (Hirst & Ramgoolam, 2026). These residual correlations—local spatial structure, higher-order moments and permutation-invariant dependencies—transfer to a limited degree across architectures, allowing a universal residual predictor to capture useful signal on a non-negligible fraction of elements.
59
+
60
+ ## Citation
61
+
62
+ If you use UniRes, please cite:
63
+
64
+ ```bibtex
65
+ @article{unires2026,
66
+ title={Persistent Correlated Structure in Neural Network Weights Enables Partial Out-of-Distribution Residual Prediction from Low-Precision Representations},
67
+ author={UniRes Authors},
68
+ year={2026}
69
+ }