mage0001 commited on
Commit
679eaab
·
verified ·
1 Parent(s): 4e4fa44

Add automatic official-base and S2HAND delta loader

Browse files

User-requested minimal loader with bundled official standalone backbone, exact routing, pinned downloads and validation; no research checkout or training snapshot.

Files changed (6) hide show
  1. README.md +99 -21
  2. SHA256SUMS.json +18 -2
  3. VALIDATION.md +18 -0
  4. requirements.txt +3 -0
  5. s2hand.py +194 -0
  6. sapiens2_backbone.py +908 -0
README.md CHANGED
@@ -2,26 +2,104 @@
2
  license: other
3
  license_name: sapiens2
4
  license_link: LICENSE.md
 
 
 
 
5
  ---
6
 
7
- # S2HAND final model weights
8
-
9
- Private camera-ready backup of S2HAND-MT-5B and S2HAND-Pose at 5B, 1B, 0.8B and 0.4B.
10
- These are replacement-tensor delta checkpoints, not standalone models. Each manifest
11
- records the exact same-size official Sapiens2 pose base filename and SHA256, the
12
- trained Transformer block indices, tensor list and weight checksum. Third-party
13
- base models, datasets and local implementation source are not included.
14
-
15
- S2HAND-MT adapts the complete pose decoder with no learned Transformer updates.
16
- Its final inference head routes only 40 finger channels from the adapted decoder,
17
- protecting the other 268 channels including both wrists. S2HAND-Pose additionally
18
- updates the listed shared Transformer blocks. Applying a delta to a generic model
19
- without the corresponding routing implementation does not reproduce that interface.
20
- The two 5B deltas include the required RoPE buffer override for the official base.
21
- The 0.4B, 0.8B and 1B models use their own same-size official bases.
22
-
23
- Implementation and camera-ready records remain on the source server. This upload
24
- preserves the selected weights; runtime recovery is still to be verified before
25
- local source deletion. The manifests distinguish existing limited 5B replay from
26
- pending fresh small-model replay. Models derive from Sapiens2 by Meta; see the
27
- upstream Sapiens2 License linked above.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: other
3
  license_name: sapiens2
4
  license_link: LICENSE.md
5
+ tags:
6
+ - pose-estimation
7
+ - safetensors
8
+ - delta-checkpoint
9
  ---
10
 
11
+ # S2HAND: automatic base + delta loading
12
+
13
+ Five final model checkpoints and their loading script live in this repository.
14
+ No GitHub checkout or local research environment is required. The loader downloads
15
+ an exact official Sapiens2 pose base, verifies its SHA256, downloads the selected
16
+ S2HAND delta, and applies replacement tensors with the final protected decoder route.
17
+ Downloads use the Hugging Face cache; models sharing a 5B base reuse that same file.
18
+ It does not write another full merged checkpoint.
19
+
20
+ ## Quick start
21
+
22
+ Python 3.12+ and PyTorch 2.7+:
23
+
24
+ ```bash
25
+ pip install torch safetensors huggingface_hub
26
+ # This repository is private: first log in to an account with access.
27
+ hf auth login
28
+ hf download mage0001/S2HAND s2hand.py sapiens2_backbone.py --local-dir S2HAND
29
+ cd S2HAND
30
+ python s2hand.py --model S2HAND-MT-5B --device cuda
31
+ ```
32
+
33
+ The last command downloads, checks and loads the model, then exits if no input is
34
+ provided. To download weights without constructing the model, add `--download-only`.
35
+ The official backbone source is bundled under the included Sapiens2 License.
36
+
37
+ ## Use in an existing pipeline
38
+
39
+ ```python
40
+ import torch
41
+ from s2hand import load_model
42
+
43
+ model = load_model("S2HAND-Pose-0.4B", device="cuda")
44
+ # Your existing person-crop pipeline supplies RGB tensors [B,3,1024,768].
45
+ # Normalize RGB values in [0,1] with ImageNet mean/std:
46
+ # mean = [0.485,0.456,0.406], std = [0.229,0.224,0.225].
47
+ with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
48
+ heatmaps = model(normalized_rgb) # [B,308,256,192]
49
+ ```
50
+
51
+ For a command-line tensor input, save your normalized RGB batch as safetensors
52
+ under key `images`, then run:
53
+
54
+ ```bash
55
+ python s2hand.py --model S2HAND-Pose-0.4B --device cuda --bf16 --input-tensor crops.safetensors --output heatmaps.safetensors
56
+ ```
57
+
58
+ This is the model loader and heatmap forward pass. Detection, person-crop affine
59
+ transforms, UDP coordinate decoding, optional flip testing and TailGuard remain
60
+ steps in the caller's pipeline. Matching paper scores requires the corresponding
61
+ paper preprocessing and evaluation protocol; the loader does not substitute a
62
+ new image-resize or detector protocol.
63
+
64
+ ## Models
65
+
66
+ | Name | Learned Transformer blocks, zero-based | Delta size |
67
+ |---|---|---|
68
+ | S2HAND-MT-5B | None; complete copied pose decoder | 3.31 MiB |
69
+ | S2HAND-Pose-5B | 38,39,40 plus decoder | 1019.04 MiB |
70
+ | S2HAND-Pose-1B | 28,29,30 plus decoder | 410.58 MiB |
71
+ | S2HAND-Pose-0.8B | 17,18,19 plus decoder | 284.78 MiB |
72
+ | S2HAND-Pose-0.4B | 16,17,18 plus decoder | 195.50 MiB |
73
+
74
+ The smaller Pose models use their own same-size official bases. They are not
75
+ compressed versions of the 5B checkpoint. The two 5B variants share one cached
76
+ base, which alone is approximately 19.07 GiB; the delta size is not total model
77
+ memory. Parameters remain FP32, matching the research runtime; `--bf16` enables
78
+ inference autocast. Meta initialization avoids allocating a second random model.
79
+
80
+ ## Exact routing and versions
81
+
82
+ The public decoder is copied before applying the delta. Only the 40 non-wrist
83
+ finger channels use the adapted decoder. The other 268 channels, including wrist
84
+ indices 41 and 62, use the frozen public decoder. Pose variants share their adapted
85
+ backbone across both decoder routes; exact preservation of protected outputs is
86
+ not guaranteed for those variants. MT does not update learned Transformer weights.
87
+ The loader preserves the original BF16 RoPE runtime buffer semantics, including
88
+ the explicit rebased 5B override. A generic base+delta load without this routing
89
+ is insufficient to reproduce the final model interface.
90
+
91
+ Official base revisions and the S2HAND weight revision are pinned in the script.
92
+ Per-model manifests record both SHA256 values and replacement tensor lists. Keep
93
+ the downloaded base and delta in the cache to reuse them on subsequent runs.
94
+ `--cache-dir` controls the cache location. `--base-path`, `--delta-path`,
95
+ `--manifest-path` and `--offline` support existing local files without downloading
96
+ another base. For completely offline use, keep `sapiens2_backbone.py` beside the script.
97
+
98
+ See `VALIDATION.md` for the checks performed before publishing this loader.
99
+ The source research code, camera-ready assets and datasets remain on the original
100
+ server; this repository contains the small loading implementation and final weights.
101
+
102
+ ## Attribution
103
+
104
+ Derived from Sapiens2 by Meta. The bundled official standalone backbone and derived
105
+ model artifacts are distributed under the included Sapiens2 License.
SHA256SUMS.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "README.md": {
3
- "bytes": 1431,
4
- "sha256": "b9cd554e64cf760d0ed944888f3b80b98ace055c810d9509f8b419f5d3b8f04f"
5
  },
6
  "models/S2HAND-MT-5B/manifest.json": {
7
  "bytes": 923,
@@ -46,5 +46,21 @@
46
  "LICENSE.md": {
47
  "bytes": 9931,
48
  "sha256": "1c77a629ee16972ddb9844378038b20321143a6faf0cac26c2c04dc9b61c5288"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
49
  }
50
  }
 
1
  {
2
  "README.md": {
3
+ "bytes": 4622,
4
+ "sha256": "0f079359fcbfce05ad47de4751b9d331a2ed3d222276177bae3c5883b5f9707b"
5
  },
6
  "models/S2HAND-MT-5B/manifest.json": {
7
  "bytes": 923,
 
46
  "LICENSE.md": {
47
  "bytes": 9931,
48
  "sha256": "1c77a629ee16972ddb9844378038b20321143a6faf0cac26c2c04dc9b61c5288"
49
+ },
50
+ "VALIDATION.md": {
51
+ "bytes": 995,
52
+ "sha256": "0f8addc7702e27fb86a34392f6ec91e3a9c97c426ecbafd429afaab6e0ca1131"
53
+ },
54
+ "requirements.txt": {
55
+ "bytes": 51,
56
+ "sha256": "f3b64ae943f09f8a647b45ee446e2b446b59d83464fabf23194907438600e978"
57
+ },
58
+ "s2hand.py": {
59
+ "bytes": 9571,
60
+ "sha256": "a2306dfcda497faa49097de467df9744cceb1b2fd1281e37c6620c64ccb8cb22"
61
+ },
62
+ "sapiens2_backbone.py": {
63
+ "bytes": 31633,
64
+ "sha256": "7046d482da440cbedf56d9c7fd5596fa0fb35747bb7dc9d34d3c5a638d8c51db"
65
  }
66
  }
VALIDATION.md ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Loader validation
2
+
3
+ Before upload, all five models passed:
4
+
5
+ - Exact base/delta SHA256 verification against their manifests.
6
+ - Strict loading of every official base tensor with no missing/meta parameters.
7
+ - Exact replacement tensor values in the original runtime dtypes.
8
+ - Bitwise-equal routed-head outputs against the original research head implementation
9
+ on deterministic feature tensors, including protection of both wrist channels.
10
+ - Correct BF16 RoPE runtime buffer semantics, including the rebased 5B override.
11
+
12
+ Additionally, the 0.4B model's full backbone-and-head forward pass matched the
13
+ original research implementation bitwise on a reduced-resolution 64x48 CPU input.
14
+
15
+ These tests verify loading, architecture and routing. Full-resolution CUDA
16
+ inference and a fresh full-dataset paper-metric replay were not run on this CPU
17
+ host. The script returns heatmaps; the caller supplies the corresponding person
18
+ crop preprocessing and UDP coordinate decoder for paper-protocol predictions.
requirements.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ torch>=2.7
2
+ safetensors>=0.4.3
3
+ huggingface_hub>=1.0
s2hand.py ADDED
@@ -0,0 +1,194 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Download and load official Sapiens2 + S2HAND replacement tensors.
3
+
4
+ Returns 308-channel heatmaps for normalized RGB tensors. This loader preserves
5
+ the final complete-decoder routing; it does not require a research checkout.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import argparse
10
+ import copy
11
+ import hashlib
12
+ import importlib.util
13
+ import json
14
+ from pathlib import Path
15
+
16
+ import torch
17
+ from torch import nn
18
+ from safetensors.torch import load_file
19
+ from huggingface_hub import hf_hub_download
20
+
21
+ REPO = 'mage0001/S2HAND'
22
+ WEIGHT_REVISION = '4e4fa442ec0c3ae4d6b2e05c71eec8229976ed98'
23
+ BASES = {
24
+ '0.4b': '73572e121602e8a6eb40eb841732e88f386d250b',
25
+ '0.8b': '76f06b9854d04bb1e59e0624de39b1e8921b12aa',
26
+ '1b': 'f5fed8b97b99698d5eea1d14ff0855d0b4c3f000',
27
+ '5b': 'ada1f29aa1fd454ca28665c700923a0101b6b24f',
28
+ }
29
+ NAMES = ('S2HAND-MT-5B', 'S2HAND-Pose-5B', 'S2HAND-Pose-1B', 'S2HAND-Pose-0.8B', 'S2HAND-Pose-0.4B')
30
+ # Filled with the exact bundled official standalone source fingerprint.
31
+ BACKBONE_SHA256 = '7046d482da440cbedf56d9c7fd5596fa0fb35747bb7dc9d34d3c5a638d8c51db'
32
+
33
+
34
+ def verify(path, expected):
35
+ h = hashlib.sha256()
36
+ with Path(path).open('rb') as stream:
37
+ for block in iter(lambda: stream.read(8 * 1024 * 1024), b''):
38
+ h.update(block)
39
+ if h.hexdigest() != expected:
40
+ raise ValueError(f'SHA256 mismatch: {path}')
41
+
42
+
43
+ def prepare(model_name='S2HAND-MT-5B', *, cache_dir=None, base_path=None, delta_path=None, manifest_path=None, local_files_only=False):
44
+ """Download pinned weights once, or verify supplied local files offline."""
45
+ if model_name not in NAMES:
46
+ raise ValueError(f'Choose one of {NAMES}')
47
+ scale = model_name.rsplit('-', 1)[1].lower()
48
+ options = dict(cache_dir=cache_dir, local_files_only=local_files_only)
49
+ manifest_path = manifest_path or hf_hub_download(REPO, f'models/{model_name}/manifest.json', revision=WEIGHT_REVISION, **options)
50
+ manifest = json.loads(Path(manifest_path).read_text())
51
+ if manifest['name'] != model_name:
52
+ raise ValueError('Manifest names a different model')
53
+ base_path = base_path or hf_hub_download(f'facebook/sapiens2-pose-{scale}', manifest['official_base']['filename'], revision=BASES[scale], **options)
54
+ delta_path = delta_path or hf_hub_download(REPO, manifest['file'], revision=WEIGHT_REVISION, **options)
55
+ print(f'Verifying {model_name} base and delta SHA256...', flush=True)
56
+ verify(base_path, manifest['official_base']['sha256'])
57
+ verify(delta_path, manifest['sha256'])
58
+ return Path(base_path), Path(delta_path), manifest
59
+
60
+
61
+ def backbone_class(cache_dir=None, local_files_only=False):
62
+ source = Path(__file__).with_name('sapiens2_backbone.py')
63
+ if not source.is_file():
64
+ source = Path(hf_hub_download(REPO, 'sapiens2_backbone.py', cache_dir=cache_dir, local_files_only=local_files_only))
65
+ verify(source, BACKBONE_SHA256)
66
+ spec = importlib.util.spec_from_file_location('_s2hand_official_backbone', source)
67
+ module = importlib.util.module_from_spec(spec)
68
+ spec.loader.exec_module(module)
69
+ return module.Sapiens2
70
+
71
+
72
+ class RoutedPoseHead(nn.Module):
73
+ def __init__(self, channels, deconv_channels=(1024,768), decoder_channels=(512,512,256)):
74
+ super().__init__()
75
+ first, second = deconv_channels
76
+ self.deconv_layers = nn.Sequential(
77
+ nn.ConvTranspose2d(channels, first, 4, stride=2, padding=1, bias=False),
78
+ nn.InstanceNorm2d(first), nn.SiLU(inplace=True),
79
+ nn.ConvTranspose2d(first, second, 4, stride=2, padding=1, bias=False),
80
+ nn.InstanceNorm2d(second), nn.SiLU(inplace=True),
81
+ )
82
+ layers = []
83
+ cin = second
84
+ for cout in decoder_channels:
85
+ layers += [nn.Conv2d(cin, cout, 1), nn.InstanceNorm2d(cout), nn.SiLU(inplace=True)]
86
+ cin = cout
87
+ self.conv_layers = nn.Sequential(*layers)
88
+ self.conv_pose = nn.Conv2d(cin, 308, 1)
89
+ self.public_decoder = None
90
+ self.public_projection = None
91
+ protected = torch.ones(308, dtype=torch.bool, device='cpu')
92
+ protected[21:63] = False
93
+ protected[[41, 62]] = True
94
+ self.register_buffer('protected_mask', protected, persistent=False)
95
+
96
+ def freeze_public_route(self):
97
+ self.public_decoder = copy.deepcopy(self.conv_layers).requires_grad_(False)
98
+ self.public_projection = copy.deepcopy(self.conv_pose).requires_grad_(False)
99
+
100
+ def forward(self, features):
101
+ features = self.deconv_layers(features)
102
+ adapted = self.conv_pose(self.conv_layers(features))
103
+ public = self.public_projection(self.public_decoder(features))
104
+ return torch.where(self.protected_mask[None, :, None, None], public, adapted)
105
+
106
+
107
+ class S2Hand(nn.Module):
108
+ def __init__(self, backbone, head):
109
+ super().__init__()
110
+ self.backbone = backbone
111
+ self.decode_head = head
112
+
113
+ def forward(self, normalized_rgb):
114
+ """B×3×1024×768 RGB, ImageNet normalized -> B×308×256×192 heatmaps."""
115
+ return self.decode_head(self.backbone(normalized_rgb)[0])
116
+
117
+
118
+ def load_model(model_name='S2HAND-MT-5B', *, device='cpu', cache_dir=None,
119
+ base_path=None, delta_path=None, manifest_path=None, local_files_only=False):
120
+ """Load full base, freeze public decoder, then replace trained parameters.
121
+
122
+ Uses meta initialization and mmap-backed safetensors to avoid allocating a
123
+ second randomly initialized 5B model. Keep model parameters in FP32; use
124
+ autocast for optional BF16 inference, as in the research pipeline.
125
+ """
126
+ base, delta, manifest = prepare(model_name, cache_dir=cache_dir, base_path=base_path,
127
+ delta_path=delta_path, manifest_path=manifest_path, local_files_only=local_files_only)
128
+ scale = model_name.rsplit('-', 1)[1].lower()
129
+ cls = backbone_class(cache_dir, local_files_only)
130
+ widths = {'0.4b':1024, '0.8b':1280, '1b':1536, '5b':2432}
131
+ state = load_file(str(base), device='cpu')
132
+ # Decoder widths differ by scale; obtain the exact architecture from base tensors.
133
+ deconv_channels = tuple(state[f'decode_head.deconv_layers.{i}.weight'].shape[1] for i in (0,3))
134
+ decoder_channels = tuple(state[f'decode_head.conv_layers.{i}.weight'].shape[0] for i in (0,3,6))
135
+ with torch.device('meta'):
136
+ backbone = cls(arch=f'sapiens2_{scale}', img_size=(1024, 768), patch_size=16,
137
+ out_type='featmap', pos_embed_rope_dtype='bf16')
138
+ model = S2Hand(backbone, RoutedPoseHead(widths[scale], deconv_channels, decoder_channels))
139
+ model.load_state_dict(state, strict=True, assign=True)
140
+ del state
141
+ # Original runtime uses BF16 RoPE periods; 5B deltas explicitly override it.
142
+ model.backbone.rope_embed.periods = model.backbone.rope_embed.periods.to(torch.bfloat16)
143
+ model.decode_head.freeze_public_route()
144
+ replacements = load_file(str(delta), device='cpu')
145
+ if set(replacements) != set(manifest['tensor_names']):
146
+ raise ValueError('Delta tensor names differ from manifest')
147
+ current = model.state_dict()
148
+ for name, value in replacements.items():
149
+ if name not in current or value.shape != current[name].shape:
150
+ raise ValueError(f'Invalid replacement tensor: {name}')
151
+ # Copy into the runtime tensor dtypes, especially the BF16 RoPE buffer,
152
+ # exactly as the research loader does. assign=True would change its dtype.
153
+ model.load_state_dict(replacements, strict=False)
154
+ del replacements, current
155
+ model.requires_grad_(False).eval()
156
+ model.model_name = model_name
157
+ return model.to(device)
158
+
159
+
160
+ def main():
161
+ parser = argparse.ArgumentParser(description=__doc__)
162
+ parser.add_argument('--model', choices=NAMES, default=NAMES[0])
163
+ parser.add_argument('--cache-dir')
164
+ parser.add_argument('--base-path')
165
+ parser.add_argument('--delta-path')
166
+ parser.add_argument('--manifest-path')
167
+ parser.add_argument('--offline', action='store_true')
168
+ parser.add_argument('--download-only', action='store_true')
169
+ parser.add_argument('--device', default='cpu')
170
+ parser.add_argument('--input-tensor', help='Safetensors file containing ImageNet-normalized RGB tensor named images')
171
+ parser.add_argument('--output', default='s2hand_heatmaps.safetensors')
172
+ parser.add_argument('--bf16', action='store_true', help='Use BF16 autocast during inference; keep FP32 parameters')
173
+ args = parser.parse_args()
174
+ options = dict(cache_dir=args.cache_dir, base_path=args.base_path, delta_path=args.delta_path,
175
+ manifest_path=args.manifest_path, local_files_only=args.offline)
176
+ if args.download_only:
177
+ base, delta, _ = prepare(args.model, **options)
178
+ print(f'Base: {base}\nDelta: {delta}\nReady for load_model().')
179
+ return
180
+ model = load_model(args.model, device=args.device, **options)
181
+ print(f'Loaded {args.model}: full base + delta + protected decoder route.', flush=True)
182
+ if args.input_tensor:
183
+ from safetensors.torch import save_file
184
+ images = load_file(args.input_tensor)['images']
185
+ if images.ndim != 4 or tuple(images.shape[1:]) != (3,1024,768):
186
+ raise ValueError('Expected images shape B×3×1024×768; RGB with ImageNet normalization')
187
+ with torch.inference_mode(), torch.autocast(torch.device(args.device).type, dtype=torch.bfloat16, enabled=args.bf16):
188
+ heatmaps = model(images.to(device=args.device, dtype=torch.float32))
189
+ save_file({'heatmaps':heatmaps.float().cpu().contiguous()},args.output)
190
+ print(f'Saved heatmaps: {args.output}')
191
+
192
+
193
+ if __name__ == '__main__':
194
+ main()
sapiens2_backbone.py ADDED
@@ -0,0 +1,908 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright (c) Meta Platforms, Inc. and affiliates.
2
+ # All rights reserved.
3
+ #
4
+ # This source code is licensed under the license found in the
5
+ # LICENSE file in the root directory of this source tree.
6
+
7
+ import math
8
+ from typing import Any, Dict, List, Literal, Optional, Sequence, Tuple, Union
9
+
10
+ import torch
11
+ import torch.nn as nn
12
+ import torch.nn.functional as F
13
+ from torch import Tensor
14
+ from torch.nn.init import trunc_normal_
15
+ from torch.utils.checkpoint import checkpoint
16
+
17
+
18
+ # ----------------------------------------------------------------------------
19
+ def to_2tuple(x):
20
+ if isinstance(x, (str, bytes)):
21
+ return (x, x)
22
+ if isinstance(x, Sequence):
23
+ x = tuple(x)
24
+ if len(x) == 2:
25
+ return x
26
+ raise ValueError("Expected scalar or length-2 iterable")
27
+ return (x, x)
28
+
29
+
30
+ class RopePositionEmbedding(nn.Module):
31
+ def __init__(
32
+ self,
33
+ embed_dim: int,
34
+ *,
35
+ num_heads: int,
36
+ base: float | None = 100.0,
37
+ min_period: float | None = None,
38
+ max_period: float | None = None,
39
+ normalize_coords: Literal["min", "max", "separate"] = "separate",
40
+ shift_coords: float | None = None,
41
+ jitter_coords: float | None = None,
42
+ rescale_coords: float | None = None,
43
+ dtype: torch.dtype | None = None,
44
+ device: torch.device | None = None,
45
+ ):
46
+ super().__init__()
47
+ assert embed_dim % (4 * num_heads) == 0
48
+ both_periods = min_period is not None and max_period is not None
49
+ if (base is None and not both_periods) or (base is not None and both_periods):
50
+ raise ValueError(
51
+ "Either `base` or `min_period`+`max_period` must be provided."
52
+ )
53
+
54
+ D_head = embed_dim // num_heads
55
+ self.base = base
56
+ self.min_period = min_period
57
+ self.max_period = max_period
58
+ self.D_head = D_head
59
+ self.normalize_coords = normalize_coords
60
+ self.shift_coords = shift_coords
61
+ self.jitter_coords = jitter_coords
62
+ self.rescale_coords = rescale_coords
63
+
64
+ # Needs persistent=True because we do teacher.load_state_dict(student.state_dict()) to initialize the teacher
65
+ self.dtype = dtype or torch.float32 # Don't rely on self.periods.dtype
66
+ self.register_buffer(
67
+ "periods",
68
+ torch.empty(D_head // 4, device=device, dtype=self.dtype),
69
+ persistent=True,
70
+ )
71
+ self._init_weights()
72
+
73
+ def forward(self, *, H: int, W: int) -> tuple[Tensor, Tensor]:
74
+ device = self.periods.device
75
+ dtype = self.dtype
76
+ dd = {"device": device, "dtype": dtype}
77
+ # Prepare coords in range [-1, +1]
78
+ if self.normalize_coords == "max":
79
+ max_HW = max(H, W)
80
+ coords_h = torch.arange(0.5, H, **dd) / max_HW # [H]
81
+ coords_w = torch.arange(0.5, W, **dd) / max_HW # [W]
82
+ elif self.normalize_coords == "min":
83
+ min_HW = min(H, W)
84
+ coords_h = torch.arange(0.5, H, **dd) / min_HW # [H]
85
+ coords_w = torch.arange(0.5, W, **dd) / min_HW # [W]
86
+ elif self.normalize_coords == "separate":
87
+ coords_h = torch.arange(0.5, H, **dd) / H # [H]
88
+ coords_w = torch.arange(0.5, W, **dd) / W # [W]
89
+ else:
90
+ raise ValueError(f"Unknown normalize_coords: {self.normalize_coords}")
91
+ coords = torch.stack(
92
+ torch.meshgrid(coords_h, coords_w, indexing="ij"), dim=-1
93
+ ) # [H, W, 2]
94
+ coords = coords.flatten(0, 1) # [HW, 2]
95
+ coords = 2.0 * coords - 1.0 # Shift range [0, 1] to [-1, +1]
96
+
97
+ # Shift coords by adding a uniform value in [-shift, shift]
98
+ if self.training and self.shift_coords is not None:
99
+ shift_hw = torch.empty(2, **dd).uniform_(
100
+ -self.shift_coords, self.shift_coords
101
+ )
102
+ coords += shift_hw[None, :]
103
+
104
+ # Jitter coords by multiplying the range [-1, 1] by a log-uniform value in [1/jitter, jitter]
105
+ if self.training and self.jitter_coords is not None:
106
+ jitter_max = np.log(self.jitter_coords)
107
+ jitter_min = -jitter_max
108
+ jitter_hw = torch.empty(2, **dd).uniform_(jitter_min, jitter_max).exp()
109
+ coords *= jitter_hw[None, :]
110
+
111
+ # Rescale coords by multiplying the range [-1, 1] by a log-uniform value in [1/rescale, rescale]
112
+ if self.training and self.rescale_coords is not None:
113
+ rescale_max = np.log(self.rescale_coords)
114
+ rescale_min = -rescale_max
115
+ rescale_hw = torch.empty(1, **dd).uniform_(rescale_min, rescale_max).exp()
116
+ coords *= rescale_hw
117
+
118
+ # Prepare angles and sin/cos
119
+ angles = (
120
+ 2 * math.pi * coords[:, :, None] / self.periods[None, None, :]
121
+ ) # [HW, 2, D//4]
122
+ angles = angles.flatten(1, 2) # [HW, D//2]
123
+ angles = angles.tile(2) # [HW, D]
124
+ cos = torch.cos(angles) # [HW, D]
125
+ sin = torch.sin(angles) # [HW, D]
126
+
127
+ return (sin, cos) # 2 * [HW, D]
128
+
129
+ def _init_weights(self):
130
+ device = self.periods.device
131
+ dtype = self.dtype
132
+ if self.base is not None:
133
+ periods = self.base ** (
134
+ 2
135
+ * torch.arange(self.D_head // 4, device=device, dtype=dtype)
136
+ / (self.D_head // 2)
137
+ ) # [D//4]
138
+ else:
139
+ base = self.max_period / self.min_period
140
+ exponents = torch.linspace(
141
+ 0, 1, self.D_head // 4, device=device, dtype=dtype
142
+ ) # [D//4] range [0, 1]
143
+ periods = base**exponents # range [1, max_period / min_period]
144
+ periods = periods / base # range [min_period / max_period, 1]
145
+ periods = periods * self.max_period # range [min_period, max_period]
146
+ self.periods.data = periods
147
+
148
+
149
+ # -------------------------------------------------------------------------------
150
+ class Tokenizer(nn.Module):
151
+ """Stacked window self‑attention that emits one token per window
152
+ by re‑using TransformerEncoderLayer blocks."""
153
+
154
+ def __init__(
155
+ self,
156
+ embed_dims: int,
157
+ window_size: int = 4,
158
+ num_heads: int = 4,
159
+ num_tokenizer_layers: int = 1,
160
+ qkv_bias: bool = True,
161
+ use_qk_norm: bool = False,
162
+ chunk_size: int = 1024, # max windows per chunk
163
+ ):
164
+ super().__init__()
165
+ self.ws = window_size
166
+ self.chunk_size = chunk_size
167
+
168
+ # local absolute positional embeddings for [CLS] + patch tokens
169
+ self.local_pos_embed = nn.Parameter(
170
+ torch.zeros(1, 1 + window_size * window_size, embed_dims)
171
+ )
172
+ trunc_normal_(self.local_pos_embed, std=0.02)
173
+
174
+ # build N identical TransformerEncoderLayer blocks
175
+ self.blocks = nn.ModuleList(
176
+ [
177
+ TransformerEncoderLayer2(
178
+ embed_dims=embed_dims,
179
+ num_heads=num_heads,
180
+ feedforward_channels=embed_dims * 4, # standard FFN size
181
+ qkv_bias=qkv_bias,
182
+ use_qk_norm=use_qk_norm,
183
+ )
184
+ for _ in range(num_tokenizer_layers)
185
+ ]
186
+ )
187
+
188
+ # shared CLS token for pooling
189
+ self.w_cls = nn.Parameter(torch.zeros(1, 1, embed_dims))
190
+ trunc_normal_(self.w_cls, std=0.02)
191
+
192
+ def forward(
193
+ self,
194
+ x: torch.Tensor,
195
+ hw: Tuple[int, int],
196
+ ) -> Tuple[torch.Tensor, Tuple[int, int]]:
197
+ """Args:
198
+ x : B, N, C (N = H*W)
199
+ hw : (H, W) before reduction
200
+ Returns:
201
+ x_ : B, (H/ws)*(W/ws), C
202
+ hw_: (H/ws, W/ws)
203
+ """
204
+ B, N, C = x.shape
205
+ H, W = hw
206
+ ws = self.ws
207
+ assert H % ws == 0 and W % ws == 0, (
208
+ f"Image size {H}×{W} must be divisible by window {ws}."
209
+ )
210
+
211
+ # reshape tokens → non‑overlapping windows
212
+ x = x.view(B, H, W, C)
213
+
214
+ ph, pw = H // ws, W // ws ## ints in eager mode
215
+ ph, pw = int(ph), int(pw) ## ints in scripting mode
216
+ x = x.view(B, ph, ws, pw, ws, C) # B, H/ws, ws, W/ws, ws, C
217
+ x = x.permute(0, 1, 3, 2, 4, 5) # B, H/ws, W/ws, ws, ws, C
218
+ x = x.contiguous().view(B * ph * pw, ws * ws, C) # (B*H/ws*W/ws), ws², C))
219
+
220
+ total_windows = x.size(0)
221
+ chunk_size = int(min(self.chunk_size, total_windows))
222
+ token_out = x.new_empty(total_windows, C)
223
+
224
+ use_ckpt = self.training and torch.is_grad_enabled()
225
+
226
+ def _run_blocks(t: torch.Tensor) -> torch.Tensor:
227
+ for blk in self.blocks:
228
+ t = blk(t)
229
+ return t
230
+
231
+ for i in range(0, total_windows, chunk_size):
232
+ chunk = x[i : i + chunk_size] # (m, ws², C)
233
+ m = chunk.size(0)
234
+ cls = self.w_cls.expand(m, -1, -1) # (m, 1, C)
235
+ chunk = torch.cat([cls, chunk], dim=1) # (m, 1+ws², C)
236
+ chunk = chunk + self.local_pos_embed # add local PE
237
+
238
+ if use_ckpt:
239
+ chunk = checkpoint(_run_blocks, chunk, use_reentrant=False)
240
+ else:
241
+ chunk = _run_blocks(chunk)
242
+
243
+ token_out[i : i + m] = chunk[:, 0] # take CLS out
244
+
245
+ token = token_out.view(B, ph * pw, C) # (B, (H/ws)*(W
246
+ return token, (ph, pw)
247
+
248
+
249
+ # -------------------------------------------------------------------------------
250
+ class GroupedQueryAttention(nn.Module):
251
+ def __init__(
252
+ self,
253
+ embed_dims,
254
+ num_heads,
255
+ num_kv_heads=None,
256
+ input_dims=None,
257
+ attn_drop=0.0,
258
+ proj_drop=0.0,
259
+ qkv_bias=True,
260
+ qk_scale=None,
261
+ proj_bias=True,
262
+ use_qk_norm=True,
263
+ v_shortcut=False,
264
+ layer_scale_init_value=0.0,
265
+ ):
266
+ super().__init__()
267
+ # Core dims
268
+ self.embed_dims = embed_dims
269
+ self.num_heads = num_heads
270
+ self.num_kv_heads = num_kv_heads or num_heads
271
+ assert self.num_heads % self.num_kv_heads == 0, (
272
+ "num_kv_heads must divide num_heads"
273
+ )
274
+ self.head_dim = embed_dims // num_heads
275
+ self.input_dims = input_dims or embed_dims
276
+ # Features
277
+ self.attn_drop = attn_drop
278
+ self.v_shortcut = v_shortcut
279
+ self.use_qk_norm = use_qk_norm
280
+
281
+ # Attention operation selection
282
+ if qk_scale is not None:
283
+ scale = qk_scale
284
+ else:
285
+ scale = self.head_dim**-0.5
286
+
287
+ assert qk_scale is None, "qk_scale is not supported"
288
+ self.attn_op = F.scaled_dot_product_attention
289
+
290
+ # Q/K/V projections
291
+ self.wq = nn.Linear(self.input_dims, embed_dims, bias=qkv_bias)
292
+ self.wk = nn.Linear(
293
+ self.input_dims, self.num_kv_heads * self.head_dim, bias=qkv_bias
294
+ )
295
+ self.wv = nn.Linear(
296
+ self.input_dims, self.num_kv_heads * self.head_dim, bias=qkv_bias
297
+ )
298
+
299
+ if self.use_qk_norm:
300
+ self.q_norm = nn.RMSNorm(self.head_dim, eps=1e-6)
301
+ self.k_norm = nn.RMSNorm(self.head_dim, eps=1e-6)
302
+
303
+ # Output projection + dropout
304
+ self.proj = nn.Linear(embed_dims, embed_dims, bias=proj_bias)
305
+ self.proj_drop = nn.Dropout(proj_drop)
306
+
307
+ # Optional LayerScale
308
+ if layer_scale_init_value > 0:
309
+ self.gamma = LayerScale(embed_dims, scale=layer_scale_init_value)
310
+ else:
311
+ self.gamma = nn.Identity()
312
+
313
+ def apply_rope(
314
+ self, q: Tensor, k: Tensor, rope: Tensor | Tuple[Tensor, Tensor]
315
+ ) -> Tuple[Tensor, Tensor]:
316
+ # All operations will use the dtype of rope, the output is cast back to the dtype of q and k
317
+ q_dtype = q.dtype
318
+ k_dtype = k.dtype
319
+ sin, cos = rope
320
+ rope_dtype = sin.dtype
321
+ q = q.to(dtype=rope_dtype)
322
+ k = k.to(dtype=rope_dtype)
323
+ N = q.shape[-2]
324
+ prefix = N - sin.shape[-2] ## extra tokens
325
+ assert prefix >= 0
326
+ q_prefix = q[:, :, :prefix, :]
327
+ q = self._rope_apply(q[:, :, prefix:, :], sin, cos) # [B, head, hw, D//head]
328
+ q = torch.cat((q_prefix, q), dim=-2) # [B, head, N, D//head]
329
+ k_prefix = k[:, :, :prefix, :]
330
+ k = self._rope_apply(k[:, :, prefix:, :], sin, cos) # [B, head, hw, D//head]
331
+ k = torch.cat((k_prefix, k), dim=-2) # [B, head, N, D//head]
332
+ q = q.to(dtype=q_dtype)
333
+ k = k.to(dtype=k_dtype)
334
+ return q, k
335
+
336
+ def _rope_rotate_half(self, x: Tensor) -> Tensor:
337
+ # x: [ x0 x1 x2 x3 x4 x5]
338
+ # out: [-x3 -x4 -x5 x0 x1 x2]
339
+ x1, x2 = x.chunk(2, dim=-1)
340
+ return torch.cat([-x2, x1], dim=-1)
341
+
342
+ def _rope_apply(self, x: Tensor, sin: Tensor, cos: Tensor) -> Tensor:
343
+ # x: [..., D], eg [x0, x1, x2, x3, x4, x5]
344
+ # sin: [..., D], eg [sin0, sin1, sin2, sin0, sin1, sin2]
345
+ # cos: [..., D], eg [cos0, cos1, cos2, cos0, cos1, cos2]
346
+ return (x * cos) + (self._rope_rotate_half(x) * sin)
347
+
348
+ def forward(self, x, rope=None):
349
+ B, N, _ = x.shape
350
+ # Q: (B, N, num_heads, head_dim)
351
+ q = self.wq(x).view(B, N, self.num_heads, self.head_dim)
352
+ # K/V: (B, N, num_kv_heads, head_dim)
353
+ k = self.wk(x).view(B, N, self.num_kv_heads, self.head_dim)
354
+ v = self.wv(x).view(B, N, self.num_kv_heads, self.head_dim)
355
+
356
+ # (B, heads, N, head_dim)
357
+ q = q.permute(0, 2, 1, 3)
358
+ k = k.permute(0, 2, 1, 3)
359
+ v = v.permute(0, 2, 1, 3)
360
+
361
+ if self.use_qk_norm:
362
+ q = self.q_norm(q)
363
+ k = self.k_norm(k)
364
+
365
+ # Repeat KV heads if group ratio >1
366
+ if self.num_kv_heads != self.num_heads:
367
+ factor = self.num_heads // self.num_kv_heads
368
+ k = k.repeat_interleave(factor, dim=1)
369
+ v = v.repeat_interleave(factor, dim=1)
370
+
371
+ if rope is not None:
372
+ q, k = self.apply_rope(q, k, rope)
373
+
374
+ # Scaled dot-product attention
375
+ attn_out = self.attn_op(
376
+ q, k, v, dropout_p=self.attn_drop if self.training else 0.0
377
+ ) # (B, num_heads, N, head_dim)
378
+
379
+ # Merge heads -> (B, N, embed_dims)
380
+ out = attn_out.permute(0, 2, 1, 3).reshape(B, N, self.embed_dims)
381
+
382
+ # Output projection + drop + layer scale
383
+ out = self.proj(out)
384
+ out = self.gamma(self.proj_drop(out))
385
+
386
+ # Optional V-shortcut (only when MQA)
387
+ if self.v_shortcut and self.num_kv_heads == 1:
388
+ raise NotImplementedError
389
+ return out
390
+
391
+
392
+ # -------------------------------------------------------------------------------
393
+ class TransformerEncoderLayer2(nn.Module):
394
+ def __init__(
395
+ self,
396
+ embed_dims,
397
+ num_heads,
398
+ num_kv_heads=None,
399
+ feedforward_channels=None,
400
+ drop_rate=0.0,
401
+ attn_drop_rate=0.0,
402
+ layer_scale_init_value=0.0,
403
+ use_qk_norm=True,
404
+ qkv_bias=True,
405
+ ):
406
+ super(TransformerEncoderLayer2, self).__init__()
407
+
408
+ self.embed_dims = embed_dims
409
+ self.ln1 = nn.RMSNorm(self.embed_dims, eps=1e-6)
410
+ self.attn = GroupedQueryAttention(
411
+ embed_dims=embed_dims,
412
+ num_heads=num_heads,
413
+ num_kv_heads=num_kv_heads,
414
+ attn_drop=attn_drop_rate,
415
+ proj_drop=drop_rate,
416
+ qkv_bias=qkv_bias,
417
+ layer_scale_init_value=layer_scale_init_value,
418
+ use_qk_norm=use_qk_norm,
419
+ )
420
+
421
+ self.ln2 = nn.RMSNorm(self.embed_dims, eps=1e-6)
422
+ self.ffn = SwiGLUFFN(
423
+ embed_dims=embed_dims,
424
+ feedforward_channels=feedforward_channels,
425
+ )
426
+
427
+ @property
428
+ def norm1(self):
429
+ return self.ln1
430
+
431
+ @property
432
+ def norm2(self):
433
+ return self.ln2
434
+
435
+ def forward(self, x, rope=None):
436
+ x = x + self.attn(self.ln1(x), rope=rope)
437
+ x = self.ffn(self.ln2(x), identity=x)
438
+ return x
439
+
440
+
441
+ ##-----------------------------------
442
+ class Sapiens2(nn.Module):
443
+ arch_zoo = {
444
+ **dict.fromkeys(
445
+ ["sapiens2_0.1b"],
446
+ {
447
+ "embed_dims": 768,
448
+ "num_layers": 12,
449
+ "num_heads": 12,
450
+ "feedforward_channels": 768 * 4,
451
+ "num_tokenizer_layers": 2,
452
+ },
453
+ ),
454
+ **dict.fromkeys(
455
+ ["sapiens2_0.4b"],
456
+ {
457
+ "embed_dims": 1024,
458
+ "num_layers": 24,
459
+ "num_heads": 16,
460
+ "feedforward_channels": 1024 * 4,
461
+ "num_tokenizer_layers": 2,
462
+ },
463
+ ),
464
+ **dict.fromkeys(
465
+ ["sapiens2_0.8b"],
466
+ {
467
+ "embed_dims": 1280,
468
+ "num_layers": 32,
469
+ "num_heads": 16,
470
+ "feedforward_channels": 1280 * 4,
471
+ "num_tokenizer_layers": 3,
472
+ },
473
+ ),
474
+ **dict.fromkeys(
475
+ ["sapiens2_1b"],
476
+ {
477
+ "embed_dims": 1536,
478
+ "num_layers": 40,
479
+ "num_heads": 24,
480
+ "feedforward_channels": 1536 * 4,
481
+ "num_tokenizer_layers": 4,
482
+ },
483
+ ),
484
+ **dict.fromkeys(
485
+ ["sapiens2_5b"],
486
+ {
487
+ "embed_dims": 2432,
488
+ "num_layers": 56,
489
+ "num_heads": 32,
490
+ "feedforward_channels": 2432 * 4,
491
+ "num_tokenizer_layers": 6,
492
+ },
493
+ ),
494
+ }
495
+
496
+ num_extra_tokens = 1 # class token
497
+ OUT_TYPES = {"raw", "cls_token", "featmap"}
498
+
499
+ def __init__(
500
+ self,
501
+ arch="sapiens2_1b",
502
+ img_size=(1024, 768),
503
+ patch_size=16,
504
+ in_channels=3,
505
+ out_indices=-1,
506
+ drop_rate=0.0,
507
+ window_size=4,
508
+ use_tokenizer=False, ## 4k resolution
509
+ use_qk_norm=True,
510
+ qkv_bias=True,
511
+ final_norm=True,
512
+ out_type="raw",
513
+ with_cls_token=True,
514
+ layer_scale_init_value=1e-4, ## non zero init to activate layerscale
515
+ frozen_stages=-1,
516
+ patch_cfg=dict(),
517
+ layer_cfgs=dict(),
518
+ pos_embed_rope_base: float = 100.0,
519
+ pos_embed_rope_min_period: float | None = None,
520
+ pos_embed_rope_max_period: float | None = None,
521
+ pos_embed_rope_normalize_coords: Literal["min", "max", "separate"] = "separate",
522
+ pos_embed_rope_shift_coords: float | None = None,
523
+ pos_embed_rope_jitter_coords: float | None = None,
524
+ pos_embed_rope_rescale_coords: float | None = None,
525
+ pos_embed_rope_dtype: str = "bf16",
526
+ n_storage_tokens: int = 8,
527
+ ):
528
+ super().__init__()
529
+
530
+ arch = arch.lower()
531
+ assert arch in set(self.arch_zoo), (
532
+ f"Arch {arch} is not in default archs {set(self.arch_zoo)}"
533
+ )
534
+ self.arch_settings = self.arch_zoo[arch]
535
+
536
+ self.embed_dims = self.arch_settings["embed_dims"]
537
+ self.num_layers = self.arch_settings["num_layers"]
538
+ self.patch_size = patch_size
539
+
540
+ self.window_size = window_size
541
+ img_size = to_2tuple(img_size)
542
+ encoder_img_size = (
543
+ (img_size[0] // window_size, img_size[1] // window_size)
544
+ if use_tokenizer
545
+ else img_size
546
+ )
547
+ self.img_size = to_2tuple(encoder_img_size)
548
+
549
+ # Set patch embedding
550
+ _patch_cfg = dict(
551
+ in_channels=in_channels,
552
+ input_size=self.img_size,
553
+ embed_dims=self.embed_dims,
554
+ kernel_size=patch_size,
555
+ stride=patch_size,
556
+ bias=True,
557
+ )
558
+ _patch_cfg.update(patch_cfg)
559
+ self.patch_embed = PatchEmbed(**_patch_cfg)
560
+ self.patch_resolution = self.patch_embed.init_out_size
561
+ num_patches = self.patch_resolution[0] * self.patch_resolution[1]
562
+
563
+ self.rope_embed = RopePositionEmbedding(
564
+ embed_dim=self.embed_dims,
565
+ num_heads=self.arch_settings["num_heads"],
566
+ base=pos_embed_rope_base,
567
+ min_period=pos_embed_rope_min_period,
568
+ max_period=pos_embed_rope_max_period,
569
+ normalize_coords=pos_embed_rope_normalize_coords,
570
+ shift_coords=pos_embed_rope_shift_coords,
571
+ jitter_coords=pos_embed_rope_jitter_coords,
572
+ rescale_coords=pos_embed_rope_rescale_coords,
573
+ dtype=torch.bfloat16 if pos_embed_rope_dtype == "bf16" else torch.float32,
574
+ )
575
+
576
+ # Set out type
577
+ if out_type not in self.OUT_TYPES:
578
+ raise ValueError(
579
+ f"Unsupported `out_type` {out_type}, please "
580
+ f"choose from {self.OUT_TYPES}"
581
+ )
582
+ self.out_type = out_type
583
+
584
+ if use_tokenizer == True:
585
+ self.tokenizer = Tokenizer(
586
+ embed_dims=self.embed_dims,
587
+ window_size=self.window_size,
588
+ num_heads=self.arch_settings["num_heads"],
589
+ num_tokenizer_layers=self.arch_settings["num_tokenizer_layers"],
590
+ qkv_bias=True,
591
+ use_qk_norm=False,
592
+ )
593
+ else:
594
+ self.tokenizer = None
595
+
596
+ # Set cls + storage tokens
597
+ self.with_cls_token = with_cls_token
598
+ if with_cls_token:
599
+ self.cls_token = nn.Parameter(torch.zeros(1, 1, self.embed_dims))
600
+ elif out_type != "cls_token":
601
+ self.cls_token = None
602
+ self.num_extra_tokens = 0
603
+ else:
604
+ raise ValueError('with_cls_token must be True when `out_type="cls_token"`.')
605
+
606
+ ## registers
607
+ self.n_storage_tokens = int(n_storage_tokens)
608
+ self.storage_tokens = (
609
+ nn.Parameter(torch.zeros(1, self.n_storage_tokens, self.embed_dims))
610
+ if self.n_storage_tokens > 0
611
+ else None
612
+ )
613
+ # how many non-patch tokens are at the front
614
+ self.num_extra_tokens = (
615
+ 1 if self.cls_token is not None else 0
616
+ ) + self.n_storage_tokens
617
+
618
+ if isinstance(out_indices, int):
619
+ out_indices = [out_indices]
620
+ assert isinstance(out_indices, Sequence), (
621
+ f'"out_indices" must by a sequence or int, get {type(out_indices)} instead.'
622
+ )
623
+ for i, index in enumerate(out_indices):
624
+ if index < 0:
625
+ out_indices[i] = self.num_layers + index
626
+ assert 0 <= out_indices[i] <= self.num_layers, (
627
+ f"Invalid out_indices {index}"
628
+ )
629
+ self.out_indices = out_indices
630
+
631
+ self.blocks = nn.Sequential()
632
+ if isinstance(layer_cfgs, dict):
633
+ layer_cfgs = [layer_cfgs] * self.num_layers
634
+
635
+ mhsa_early, mhsa_late = 8, 8
636
+ for i in range(self.num_layers):
637
+ if i < mhsa_early or i >= self.num_layers - mhsa_late:
638
+ num_kv_heads = None ## use MHSA
639
+ else:
640
+ num_kv_heads = self.arch_settings["num_heads"] // 2 # Use GQA
641
+
642
+ _layer_cfg = dict(
643
+ embed_dims=self.embed_dims,
644
+ num_heads=self.arch_settings["num_heads"],
645
+ num_kv_heads=num_kv_heads,
646
+ feedforward_channels=self.arch_settings["feedforward_channels"],
647
+ use_qk_norm=use_qk_norm,
648
+ layer_scale_init_value=layer_scale_init_value,
649
+ drop_rate=drop_rate,
650
+ qkv_bias=qkv_bias,
651
+ )
652
+ _layer_cfg.update(layer_cfgs[i])
653
+ self.blocks.append(TransformerEncoderLayer2(**_layer_cfg))
654
+
655
+ self.frozen_stages = frozen_stages
656
+
657
+ self.final_norm = final_norm
658
+ if final_norm:
659
+ self.ln1 = nn.RMSNorm(self.embed_dims, eps=1e-6)
660
+
661
+ # freeze stages only when self.frozen_stages > 0
662
+ if self.frozen_stages > 0:
663
+ self._freeze_stages()
664
+
665
+ ## load init weights
666
+ self.init_weights()
667
+
668
+ return
669
+
670
+ def init_weights(self):
671
+ # Initialize class token and storagr token embeddings
672
+ if self.with_cls_token:
673
+ trunc_normal_(self.cls_token, std=0.02)
674
+
675
+ if self.storage_tokens is not None:
676
+ trunc_normal_(self.storage_tokens, std=0.02)
677
+
678
+ # Apply custom initialization to all submodules
679
+ self.apply(self._init_weights)
680
+
681
+ def _init_weights(self, m):
682
+ if isinstance(m, nn.Linear):
683
+ # Use a truncated normal distribution for linear layer weights
684
+ trunc_normal_(m.weight, std=0.02)
685
+ if m.bias is not None:
686
+ nn.init.constant_(m.bias, 0)
687
+
688
+ elif isinstance(m, (nn.LayerNorm, nn.RMSNorm)):
689
+ # Initialize normalization layers to act as an identity function
690
+ if hasattr(m, "bias") and m.bias is not None:
691
+ nn.init.constant_(m.bias, 0)
692
+ if hasattr(m, "weight") and m.weight is not None:
693
+ nn.init.constant_(m.weight, 1.0)
694
+
695
+ elif isinstance(m, nn.Conv2d):
696
+ # Initialize conv layer weights like linear layers
697
+ trunc_normal_(m.weight, std=0.02)
698
+ if m.bias is not None:
699
+ nn.init.constant_(m.bias, 0)
700
+
701
+ def _freeze_stages(self):
702
+ ## freeze tokenizer
703
+ if self.frozen_stages >= 1 and self.tokenizer is not None:
704
+ self.tokenizer.eval()
705
+ for param in self.tokenizer.parameters():
706
+ param.requires_grad = False
707
+
708
+ # freeze patch embedding
709
+ self.patch_embed.eval()
710
+ for param in self.patch_embed.parameters():
711
+ param.requires_grad = False
712
+ # freeze cls_token
713
+ if self.cls_token is not None:
714
+ self.cls_token.requires_grad = False
715
+ if self.storage_tokens is not None:
716
+ self.storage_tokens.requires_grad = False
717
+ # freeze layers
718
+ for i in range(1, self.frozen_stages + 1):
719
+ m = self.blocks[i - 1]
720
+ m.eval()
721
+ for param in m.parameters():
722
+ param.requires_grad = False
723
+
724
+ # freeze the last layer norm
725
+ if self.frozen_stages == len(self.blocks):
726
+ if self.final_norm:
727
+ self.ln1.eval()
728
+ for param in self.ln1.parameters():
729
+ param.requires_grad = False
730
+
731
+ def forward(self, x):
732
+ B = x.shape[0]
733
+
734
+ x, patch_resolution = self.patch_embed(x) # (B, 256*256, C)
735
+ if self.tokenizer is not None:
736
+ x, patch_resolution = self.tokenizer(x, patch_resolution)
737
+
738
+ # prepend [CLS] and storage tokens
739
+ prepend = []
740
+ if self.cls_token is not None:
741
+ prepend.append(self.cls_token.expand(B, -1, -1))
742
+ if self.storage_tokens is not None:
743
+ prepend.append(self.storage_tokens.expand(B, -1, -1))
744
+ if len(prepend) > 0:
745
+ x = torch.cat(prepend + [x], dim=1)
746
+
747
+ rope_sincos = self.rope_embed(H=patch_resolution[0], W=patch_resolution[1])
748
+ outs = []
749
+ for i, layer in enumerate(self.blocks):
750
+ x = layer(x, rope=rope_sincos)
751
+
752
+ if i == len(self.blocks) - 1 and self.final_norm:
753
+ x = self.ln1(x)
754
+
755
+ if i in self.out_indices:
756
+ outs.append(self._format_output(x, patch_resolution))
757
+
758
+ return tuple(outs)
759
+
760
+ def _format_output(self, x, hw):
761
+ if self.out_type == "raw":
762
+ return x
763
+ if self.out_type == "cls_token":
764
+ return x[:, 0]
765
+
766
+ patch_token = x[:, self.num_extra_tokens :]
767
+ if self.out_type == "featmap":
768
+ B = x.size(0)
769
+ # (B, N, C) -> (B, H, W, C) -> (B, C, H, W)
770
+ return patch_token.reshape(B, *hw, -1).permute(0, 3, 1, 2)
771
+
772
+ @property
773
+ def norm1(self):
774
+ return self.ln1
775
+
776
+
777
+ # ----------------------------------------------------------------------------
778
+ class LayerScale(nn.Module):
779
+ def __init__(
780
+ self,
781
+ dim: int,
782
+ inplace: bool = False,
783
+ data_format: str = "channels_last",
784
+ scale: float = 1e-5,
785
+ ):
786
+ super().__init__()
787
+ assert data_format in (
788
+ "channels_last",
789
+ "channels_first",
790
+ ), "'data_format' could only be channels_last or channels_first."
791
+ self.inplace = inplace
792
+ self.data_format = data_format
793
+ self.weight = nn.Parameter(torch.ones(dim) * scale)
794
+
795
+ def forward(self, x) -> torch.Tensor:
796
+ if self.data_format == "channels_first":
797
+ shape = tuple((1, -1, *(1 for _ in range(x.dim() - 2))))
798
+ else:
799
+ shape = tuple((*(1 for _ in range(x.dim() - 1)), -1))
800
+ if self.inplace:
801
+ return x.mul_(self.weight.view(*shape))
802
+ else:
803
+ return x * self.weight.view(*shape)
804
+
805
+
806
+ # ----------------------------------------------------------------------------
807
+ class PatchEmbed(nn.Module):
808
+ def __init__(
809
+ self,
810
+ in_channels=3,
811
+ embed_dims=768,
812
+ kernel_size=16,
813
+ stride=16,
814
+ padding="corner",
815
+ dilation=1,
816
+ bias=True,
817
+ input_size=None,
818
+ ):
819
+ super().__init__()
820
+
821
+ self.embed_dims = embed_dims
822
+ if stride is None:
823
+ stride = kernel_size
824
+
825
+ kernel_size = to_2tuple(kernel_size)
826
+ stride = to_2tuple(stride)
827
+ dilation = to_2tuple(dilation)
828
+ padding = 0
829
+ padding = to_2tuple(padding)
830
+
831
+ self.projection = nn.Conv2d(
832
+ in_channels=in_channels,
833
+ out_channels=embed_dims,
834
+ kernel_size=kernel_size,
835
+ stride=stride,
836
+ padding=padding,
837
+ dilation=dilation,
838
+ bias=bias,
839
+ )
840
+
841
+ if input_size:
842
+ input_size = to_2tuple(input_size)
843
+ self.init_input_size = input_size
844
+ h_out = (
845
+ input_size[0] + 2 * padding[0] - dilation[0] * (kernel_size[0] - 1) - 1
846
+ ) // stride[0] + 1
847
+ w_out = (
848
+ input_size[1] + 2 * padding[1] - dilation[1] * (kernel_size[1] - 1) - 1
849
+ ) // stride[1] + 1
850
+ self.init_out_size = (h_out, w_out)
851
+ else:
852
+ self.init_input_size = None
853
+ self.init_out_size = None
854
+
855
+ def forward(self, x):
856
+ x = self.projection(x)
857
+ out_size = (x.shape[2], x.shape[3])
858
+ x = x.flatten(2).transpose(1, 2)
859
+ return x, out_size
860
+
861
+
862
+ # ----------------------------------------------------------------------------
863
+ class SwiGLUFFN(nn.Module):
864
+ """SwiGLU FFN layer.
865
+ https://github.com/facebookresearch/dinov2/blob/main/dinov2/layers/swiglu_ffn.py
866
+ """ # noqa
867
+
868
+ def __init__(
869
+ self,
870
+ embed_dims: int,
871
+ feedforward_channels: Optional[int] = None,
872
+ out_dims: Optional[int] = None,
873
+ layer_scale_init_value: float = 0.0,
874
+ bias: bool = True,
875
+ add_identity: bool = True,
876
+ ) -> None:
877
+ super().__init__()
878
+ self.embed_dims = embed_dims
879
+ self.out_dims = out_dims or embed_dims
880
+ hidden_dims = feedforward_channels or embed_dims
881
+
882
+ self.w12 = nn.Linear(self.embed_dims, 2 * hidden_dims, bias=bias)
883
+ self.w3 = nn.Linear(hidden_dims, self.out_dims, bias=bias)
884
+
885
+ if layer_scale_init_value > 0:
886
+ self.gamma2 = LayerScale(dim=embed_dims, scale=layer_scale_init_value)
887
+ else:
888
+ self.gamma2 = nn.Identity()
889
+
890
+ self.add_identity = add_identity
891
+
892
+ def forward(
893
+ self, x: torch.Tensor, identity: Optional[torch.Tensor] = None
894
+ ) -> torch.Tensor:
895
+ x12 = self.w12(x)
896
+ x1, x2 = x12.chunk(2, dim=-1)
897
+ hidden = F.silu(x1) * x2
898
+ out = self.w3(hidden)
899
+ out = self.gamma2(out)
900
+
901
+ if self.out_dims != self.embed_dims or not self.add_identity:
902
+ # due to the dimension inconsistence or user setting
903
+ # not to apply residual operation
904
+ return out
905
+
906
+ if identity is None:
907
+ identity = x
908
+ return identity + out