Correct the loading snippet for PyTorch >= 2.6
Browse files
README.md
CHANGED
|
@@ -45,7 +45,7 @@ attribution built on it inherits that contact.
|
|
| 45 |
| `state_dict` | `OrderedDict` | encoder/decoder weights and `b_pre` |
|
| 46 |
| `width` | `int` | 4096 |
|
| 47 |
| `k` | `int` | 32 |
|
| 48 |
-
| `mu` |
|
| 49 |
| `scale` | `float` | activation scale, divided before encoding |
|
| 50 |
| `thresh` | `float` | BatchTopK threshold, frozen over the whole training matrix |
|
| 51 |
|
|
@@ -56,16 +56,33 @@ value rather than recomputing per batch.
|
|
| 56 |
|
| 57 |
## Loading
|
| 58 |
|
|
|
|
|
|
|
|
|
|
| 59 |
```python
|
| 60 |
import torch
|
| 61 |
|
| 62 |
-
ckpt = torch.load("sae.pt", map_location="cpu")
|
| 63 |
print(ckpt["width"], ckpt["k"], ckpt["thresh"])
|
| 64 |
|
| 65 |
# Activations are normalised with the stored constants before encoding:
|
| 66 |
# x_norm = (x - ckpt["mu"]) / ckpt["scale"]
|
| 67 |
```
|
| 68 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
The model class and the encode path are in `src/04_interpretability/train_sae_hcp.py`
|
| 70 |
in the code repository below.
|
| 71 |
|
|
|
|
| 45 |
| `state_dict` | `OrderedDict` | encoder/decoder weights and `b_pre` |
|
| 46 |
| `width` | `int` | 4096 |
|
| 47 |
| `k` | `int` | 32 |
|
| 48 |
+
| `mu` | `np.ndarray (1, 512)` | activation mean, subtracted before encoding |
|
| 49 |
| `scale` | `float` | activation scale, divided before encoding |
|
| 50 |
| `thresh` | `float` | BatchTopK threshold, frozen over the whole training matrix |
|
| 51 |
|
|
|
|
| 56 |
|
| 57 |
## Loading
|
| 58 |
|
| 59 |
+
`mu` is a NumPy array, so on PyTorch 2.6 and later — where `torch.load` defaults
|
| 60 |
+
to `weights_only=True` — the default call raises `UnpicklingError`. Either:
|
| 61 |
+
|
| 62 |
```python
|
| 63 |
import torch
|
| 64 |
|
| 65 |
+
ckpt = torch.load("sae.pt", map_location="cpu", weights_only=False)
|
| 66 |
print(ckpt["width"], ckpt["k"], ckpt["thresh"])
|
| 67 |
|
| 68 |
# Activations are normalised with the stored constants before encoding:
|
| 69 |
# x_norm = (x - ckpt["mu"]) / ckpt["scale"]
|
| 70 |
```
|
| 71 |
|
| 72 |
+
or, to keep the safe loader, allowlist the NumPy reconstructors:
|
| 73 |
+
|
| 74 |
+
```python
|
| 75 |
+
import numpy as np, torch
|
| 76 |
+
|
| 77 |
+
with torch.serialization.safe_globals(
|
| 78 |
+
[np._core.multiarray._reconstruct, np.ndarray, np.dtype, np.dtypes.Float32DType]
|
| 79 |
+
):
|
| 80 |
+
ckpt = torch.load("sae.pt", map_location="cpu")
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
On PyTorch below 2.6 the plain `torch.load("sae.pt", map_location="cpu")` works
|
| 84 |
+
as written.
|
| 85 |
+
|
| 86 |
The model class and the encode path are in `src/04_interpretability/train_sae_hcp.py`
|
| 87 |
in the code repository below.
|
| 88 |
|