prabad commited on
Commit
005b0cb
·
verified ·
1 Parent(s): fc074fa

Correct the loading snippet for PyTorch >= 2.6

Browse files
Files changed (1) hide show
  1. README.md +19 -2
README.md CHANGED
@@ -45,7 +45,7 @@ attribution built on it inherits that contact.
45
  | `state_dict` | `OrderedDict` | encoder/decoder weights and `b_pre` |
46
  | `width` | `int` | 4096 |
47
  | `k` | `int` | 32 |
48
- | `mu` | tensor `(1, 512)` | activation mean, subtracted before encoding |
49
  | `scale` | `float` | activation scale, divided before encoding |
50
  | `thresh` | `float` | BatchTopK threshold, frozen over the whole training matrix |
51
 
@@ -56,16 +56,33 @@ value rather than recomputing per batch.
56
 
57
  ## Loading
58
 
 
 
 
59
  ```python
60
  import torch
61
 
62
- ckpt = torch.load("sae.pt", map_location="cpu")
63
  print(ckpt["width"], ckpt["k"], ckpt["thresh"])
64
 
65
  # Activations are normalised with the stored constants before encoding:
66
  # x_norm = (x - ckpt["mu"]) / ckpt["scale"]
67
  ```
68
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
  The model class and the encode path are in `src/04_interpretability/train_sae_hcp.py`
70
  in the code repository below.
71
 
 
45
  | `state_dict` | `OrderedDict` | encoder/decoder weights and `b_pre` |
46
  | `width` | `int` | 4096 |
47
  | `k` | `int` | 32 |
48
+ | `mu` | `np.ndarray (1, 512)` | activation mean, subtracted before encoding |
49
  | `scale` | `float` | activation scale, divided before encoding |
50
  | `thresh` | `float` | BatchTopK threshold, frozen over the whole training matrix |
51
 
 
56
 
57
  ## Loading
58
 
59
+ `mu` is a NumPy array, so on PyTorch 2.6 and later — where `torch.load` defaults
60
+ to `weights_only=True` — the default call raises `UnpicklingError`. Either:
61
+
62
  ```python
63
  import torch
64
 
65
+ ckpt = torch.load("sae.pt", map_location="cpu", weights_only=False)
66
  print(ckpt["width"], ckpt["k"], ckpt["thresh"])
67
 
68
  # Activations are normalised with the stored constants before encoding:
69
  # x_norm = (x - ckpt["mu"]) / ckpt["scale"]
70
  ```
71
 
72
+ or, to keep the safe loader, allowlist the NumPy reconstructors:
73
+
74
+ ```python
75
+ import numpy as np, torch
76
+
77
+ with torch.serialization.safe_globals(
78
+ [np._core.multiarray._reconstruct, np.ndarray, np.dtype, np.dtypes.Float32DType]
79
+ ):
80
+ ckpt = torch.load("sae.pt", map_location="cpu")
81
+ ```
82
+
83
+ On PyTorch below 2.6 the plain `torch.load("sae.pt", map_location="cpu")` works
84
+ as written.
85
+
86
  The model class and the encode path are in `src/04_interpretability/train_sae_hcp.py`
87
  in the code repository below.
88