tedzh17 commited on
Commit
6794be9
·
1 Parent(s): 38d1b66

initial commit

Browse files
.gitignore ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ __pycache__/
2
+ *.pyc
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Microsoft
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md ADDED
@@ -0,0 +1,196 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ library_name: stellar
4
+ pipeline_tag: image-feature-extraction
5
+ datasets:
6
+ - imagenet-1k
7
+ tags:
8
+ - vision
9
+ - self-supervised-learning
10
+ - representation-learning
11
+ - sparse-tokens
12
+ - vision-transformer
13
+ - image-feature-extraction
14
+ ---
15
+
16
+ # STELLAR — Sparse Visual Representations via Spatial–Semantic Factorization
17
+
18
+ **STELLAR** learns a **unified sparse visual representation** that supports both
19
+ **reconstruction** and **semantics** using as few as **16 tokens**. By factorizing
20
+ *"what"* (semantics) from *"where"* (spatial layout), each image is encoded as the
21
+ low-rank product of a **localization** matrix and a **semantics** matrix.
22
+
23
+ <p align="center">
24
+ <img src="factorization.svg" alt="Spatial–semantic factorization" width="720">
25
+ </p>
26
+
27
+ - 📄 **Paper:** [arXiv:2602.01905](https://arxiv.org/abs/2602.01905) (ICML 2026)
28
+ - 💻 **Code:** [github.com/microsoft/STELLAR](https://github.com/microsoft/STELLAR)
29
+
30
+ These checkpoints contain the **full set of trained STELLAR modules** (encoder, sparse
31
+ tokens, projections, reconstruction decoder, and clustering heads), so a single file
32
+ supports feature extraction, image reconstruction, and continued pretraining. All
33
+ models are self-supervised on **ImageNet-1K** at 224×224.
34
+
35
+ ## Highlights
36
+
37
+ - **Sparse & unified** — one small set of tokens serves both high-level semantics
38
+ and pixel-level reconstruction.
39
+ - **Factorized latents** — each token captures a concept (*what*) together with a
40
+ spatial map of *where* it appears.
41
+ - **Strong on both axes** — STELLAR-H reaches **2.60 FID** (reconstruction) and
42
+ **79.1%** ImageNet linear-probing accuracy with just **16 tokens**.
43
+
44
+ ## Available models
45
+
46
+ | Model | Backbone | Tokens | Params | Type | File |
47
+ | :--- | :--- | :---: | :---: | :--- | :--- |
48
+ | `stellar-b16` | ViT-B/16 | 16 | 88M | main | [`stellar-b16.safetensors`](stellar-b16.safetensors) |
49
+ | `stellar-l16` | ViT-L/16 | 16 | 307M | main | [`stellar-l16.safetensors`](stellar-l16.safetensors) |
50
+ | `stellar-h16` | ViT-H/14 | 16 | 636M | main | [`stellar-h16.safetensors`](stellar-h16.safetensors) |
51
+ | `stellar-b8` | ViT-B/16 | 8 | 88M | ablation | [`stellar-b8.safetensors`](stellar-b8.safetensors) |
52
+ | `stellar-b24` | ViT-B/16 | 24 | 88M | ablation | [`stellar-b24.safetensors`](stellar-b24.safetensors) |
53
+
54
+ The main models (`b16`, `l16`, `h16`) are recommended for downstream use; the 8- and
55
+ 24-token base models are ablations on the number of sparse tokens.
56
+
57
+ ## Usage
58
+
59
+ Install the STELLAR code and the Hub helpers:
60
+
61
+ ```bash
62
+ pip install huggingface_hub safetensors
63
+ git clone https://github.com/microsoft/STELLAR && cd STELLAR
64
+ pip install -r requirements.txt
65
+ ```
66
+
67
+ ### Quick start
68
+
69
+ From the STELLAR code directory, use the [`load_stellar.py`](load_stellar.py) helper
70
+ (it downloads the weights from the Hub for you):
71
+
72
+ ```python
73
+ import torch
74
+ from load_stellar import load_stellar, list_models
75
+
76
+ print(list_models()) # ['stellar-b16', 'stellar-l16', ...]
77
+ model = load_stellar("stellar-b16") # purpose="encode" (default)
78
+
79
+ # RGB image in [0, 1], resized to 224×224 (ImageNet normalization is applied internally)
80
+ image = torch.rand(1, 3, 224, 224)
81
+ with torch.no_grad():
82
+ out = model.encode(image)
83
+
84
+ out["sparse"] # (1, K, D) sparse concept tokens ("what")
85
+ out["spatial"] # (1, P, K) per-token spatial maps ("where")
86
+ out["dense"] # (1, P, D) dense per-patch features
87
+ out["cls"] # (1, 1, D) global image token
88
+ ```
89
+
90
+ ### Reconstruction & continued pretraining
91
+
92
+ The same checkpoint can be loaded for other purposes via the `purpose` argument. Image
93
+ reconstruction and continued pretraining use the decoder, which predicts
94
+ [MaskGIT-VQGAN](https://huggingface.co/fun-research/TiTok) tokens — pass the tokenizer
95
+ path as `vq_model`:
96
+
97
+ ```python
98
+ # 1. encode -> factorized features (sparse concept tokens + spatial maps)
99
+ model = load_stellar("stellar-b16", purpose="reconstruct", vq_model=VQGAN_PATH)
100
+ features = model.encode(image) # dict: sparse (B,K,D), spatial (B,P,K), ...
101
+
102
+ # 2. decode the factorized features -> VQGAN decoder -> pixels
103
+ out = model.reconstruct(features) # or model.reconstruct(features["sparse"], features["spatial"])
104
+ pixels = out["reconstruction"] # (B, 3, H, W) RGB in [0, 1]
105
+ # 224x224 for /16 models, 256x256 for the /14 H model
106
+ # out["tokens"] : (B, P) predicted VQGAN token ids
107
+ # out["logits"] : (B, P, 1024) raw codebook logits
108
+
109
+ # continued pretraining (all modules, gradients enabled)
110
+ model = load_stellar("stellar-b16", purpose="pretrain", vq_model=VQGAN_PATH)
111
+ losses = model({"image": image, "labels": labels, ...})["predictions"]
112
+ ```
113
+
114
+ `reconstruct` is the **decoder half** of STELLAR: it takes the factorized features and
115
+ runs low-rank dense map → ViT decoder → VQGAN decoder to return RGB pixels. See
116
+ [`examples/reconstruction.ipynb`](examples/reconstruction.ipynb) for an end-to-end demo
117
+ that loads an image and displays the reconstruction.
118
+
119
+
120
+ ### What the model returns
121
+
122
+ | Key | Shape | Description | Typical use |
123
+ | :--- | :--- | :--- | :--- |
124
+ | `sparse` | `(B, K, D)` | sparse concept tokens | classification, retrieval |
125
+ | `spatial` | `(B, P, K)` | spatial map of each token | segmentation, visualization |
126
+ | `dense` | `(B, P, D)` | dense per-patch features | segmentation |
127
+ | `lowrank` | `(B, P, D)` | reassembled dense map | reconstruction |
128
+ | `cls` | `(B, 1, D)` | global representation | classification |
129
+
130
+ `B` = batch, `K` = number of sparse tokens, `P` = number of patches (196 for /16 at
131
+ 224², 256 for /14), `D` = embedding dim (768 / 1024 / 1280 for B / L / H).
132
+
133
+ ### Loading the weights manually
134
+
135
+ ```python
136
+ import json, torch
137
+ from huggingface_hub import hf_hub_download
138
+ from safetensors.torch import load_file
139
+ from src.models.stellar_model import STELLARModel
140
+
141
+ repo = "microsoft/STELLAR"
142
+ cfg = json.load(open(hf_hub_download(repo, "config.json")))["models"]["stellar-b16"]
143
+ state = load_file(hf_hub_download(repo, cfg["weights"]))
144
+
145
+ model = STELLARModel(
146
+ num_sparse_tokens=cfg["num_sparse_tokens"],
147
+ num_decoder_layers=cfg["num_decoder_layers"],
148
+ spatial_temp=cfg["spatial_temp"],
149
+ vit_pretrained=cfg["backbone"],
150
+ do_recon=False, do_clustering=False, vq_model=None,
151
+ )
152
+ model.load_state_dict(state, strict=False) # encoder-only build ignores decoder/heads
153
+ model.eval()
154
+ features = model.encode(torch.rand(1, 3, 224, 224))
155
+ ```
156
+
157
+ > **Tip:** download with `huggingface_hub` (as above) rather than `git clone` so that
158
+ > downloads are registered on the Hub — `git clone` is not counted in download stats.
159
+
160
+ ## Model details
161
+
162
+ - **Architecture:** ViT encoder (MAE-initialized) + learned sparse latent queries with
163
+ spatial–semantic factorization.
164
+ - **Pretraining data:** ImageNet-1K (self-supervised; labels not used).
165
+ - **Input:** RGB images in `[0, 1]`, resized to 224×224 (bicubic). ImageNet mean/std
166
+ normalization is applied **inside** the model — pass raw `[0, 1]` images.
167
+ - **Weights:** the complete set of trained STELLAR modules (encoder, sparse tokens,
168
+ projections, reconstruction decoder, and clustering heads), stored in `safetensors`.
169
+ Only the third-party MaskGIT-VQGAN tokenizer is excluded — it is downloaded separately
170
+ (from [TiTok](https://huggingface.co/fun-research/TiTok)) and passed via `vq_model`.
171
+ - **Framework:** PyTorch.
172
+
173
+ ## Intended uses & limitations
174
+
175
+ - **Intended use:** extracting compact sparse/dense visual features for downstream
176
+ recognition, segmentation, retrieval, reconstruction, and analysis.
177
+ - **Limitations:** pretrained on ImageNet-1K at 224×224, so features reflect that
178
+ distribution; performance on very different domains (e.g. medical, satellite) may
179
+ require fine-tuning. The models are research artifacts and are not safety-tested for
180
+ production decision-making.
181
+
182
+ ## Citation
183
+
184
+ ```bibtex
185
+ @inproceedings{zhao2026stellar,
186
+ title = {Learning Sparse Visual Representations via Spatial-Semantic Factorization},
187
+ author = {Zhao, Theodore Zhengde and Kiblawi, Sid and Yang, Jianwei and Usuyama, Naoto and Tan, Reuben and Codella, Noel C and Naumann, Tristan and Poon, Hoifung and Wei, Mu},
188
+ booktitle = {International Conference on Machine Learning (ICML)},
189
+ year = {2026},
190
+ url = {https://arxiv.org/abs/2602.01905},
191
+ }
192
+ ```
193
+
194
+ ## License
195
+
196
+ Released under the [MIT License](LICENSE).
config.json ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "stellar",
3
+ "library_name": "stellar",
4
+ "code": "https://github.com/microsoft/STELLAR",
5
+ "default": "stellar-b16",
6
+ "models": {
7
+ "stellar-b16": {
8
+ "weights": "stellar-b16.safetensors",
9
+ "category": "main",
10
+ "backbone": "facebook/vit-mae-base",
11
+ "image_size": 224,
12
+ "patch_size": 16,
13
+ "embed_dim": 768,
14
+ "num_sparse_tokens": 16,
15
+ "num_decoder_layers": 6,
16
+ "spatial_temp": 0.06,
17
+ "do_recon": true,
18
+ "do_clustering": true,
19
+ "do_cls": true,
20
+ "num_clusters": 16384,
21
+ "recon_type": "vq"
22
+ },
23
+ "stellar-l16": {
24
+ "weights": "stellar-l16.safetensors",
25
+ "category": "main",
26
+ "backbone": "facebook/vit-mae-large",
27
+ "image_size": 224,
28
+ "patch_size": 16,
29
+ "embed_dim": 1024,
30
+ "num_sparse_tokens": 16,
31
+ "num_decoder_layers": 6,
32
+ "spatial_temp": 0.06,
33
+ "do_recon": true,
34
+ "do_clustering": true,
35
+ "do_cls": true,
36
+ "num_clusters": 16384,
37
+ "recon_type": "vq"
38
+ },
39
+ "stellar-h16": {
40
+ "weights": "stellar-h16.safetensors",
41
+ "category": "main",
42
+ "backbone": "facebook/vit-mae-huge",
43
+ "image_size": 224,
44
+ "patch_size": 14,
45
+ "embed_dim": 1280,
46
+ "num_sparse_tokens": 16,
47
+ "num_decoder_layers": 6,
48
+ "spatial_temp": 0.06,
49
+ "do_recon": true,
50
+ "do_clustering": true,
51
+ "do_cls": true,
52
+ "num_clusters": 16384,
53
+ "recon_type": "vq"
54
+ },
55
+ "stellar-b8": {
56
+ "weights": "stellar-b8.safetensors",
57
+ "category": "ablation",
58
+ "backbone": "facebook/vit-mae-base",
59
+ "image_size": 224,
60
+ "patch_size": 16,
61
+ "embed_dim": 768,
62
+ "num_sparse_tokens": 8,
63
+ "num_decoder_layers": 6,
64
+ "spatial_temp": 0.06,
65
+ "do_recon": true,
66
+ "do_clustering": true,
67
+ "do_cls": true,
68
+ "num_clusters": 16384,
69
+ "recon_type": "vq"
70
+ },
71
+ "stellar-b24": {
72
+ "weights": "stellar-b24.safetensors",
73
+ "category": "ablation",
74
+ "backbone": "facebook/vit-mae-base",
75
+ "image_size": 224,
76
+ "patch_size": 16,
77
+ "embed_dim": 768,
78
+ "num_sparse_tokens": 24,
79
+ "num_decoder_layers": 6,
80
+ "spatial_temp": 0.06,
81
+ "do_recon": true,
82
+ "do_clustering": true,
83
+ "do_cls": true,
84
+ "num_clusters": 16384,
85
+ "recon_type": "vq"
86
+ }
87
+ }
88
+ }
examples/reconstruction.ipynb ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "id": "a0703884",
6
+ "metadata": {},
7
+ "source": [
8
+ "# STELLAR image reconstruction\n",
9
+ "\n",
10
+ "This notebook runs the **full reconstruction pipeline** end to end:\n",
11
+ "\n",
12
+ "```\n",
13
+ "image → encoder → sparse + spatial tokens → low-rank dense map\n",
14
+ " → ViT decoder → VQGAN decoder → RGB pixels\n",
15
+ "```\n",
16
+ "\n",
17
+ "The released STELLAR checkpoints predict [MaskGIT-VQGAN](https://huggingface.co/fun-research/TiTok)\n",
18
+ "tokens, so you also need the VQGAN tokenizer to decode tokens back to pixels.\n",
19
+ "\n",
20
+ "**Requirements:** `torch`, `safetensors`, `huggingface_hub`, `pillow`, `matplotlib`, and the\n",
21
+ "STELLAR model code on your `PYTHONPATH` (so `from src.models.stellar_model import STELLARModel` works)."
22
+ ]
23
+ },
24
+ {
25
+ "cell_type": "code",
26
+ "execution_count": null,
27
+ "id": "3f852740",
28
+ "metadata": {},
29
+ "outputs": [],
30
+ "source": [
31
+ "import os, sys\n",
32
+ "# load_stellar.py and the `src/` model code live in the repo root (one level up).\n",
33
+ "sys.path.insert(0, os.path.abspath(\"..\"))\n",
34
+ "\n",
35
+ "import torch\n",
36
+ "import torch.nn.functional as F\n",
37
+ "from huggingface_hub import hf_hub_download\n",
38
+ "from load_stellar import load_stellar, list_models\n",
39
+ "\n",
40
+ "print(\"Available models:\", list_models())\n",
41
+ "device = \"cuda\" if torch.cuda.is_available() else \"cpu\""
42
+ ]
43
+ },
44
+ {
45
+ "cell_type": "markdown",
46
+ "id": "fa4219aa",
47
+ "metadata": {},
48
+ "source": [
49
+ "## 1. Download the MaskGIT-VQGAN tokenizer\n",
50
+ "\n",
51
+ "STELLAR reuses the MaskGIT-VQGAN tokenizer from TiTok. We only need its decoder to turn\n",
52
+ "predicted tokens back into pixels."
53
+ ]
54
+ },
55
+ {
56
+ "cell_type": "code",
57
+ "execution_count": null,
58
+ "id": "8a6807dc",
59
+ "metadata": {},
60
+ "outputs": [],
61
+ "source": [
62
+ "vq_path = hf_hub_download(\n",
63
+ " repo_id=\"fun-research/TiTok\",\n",
64
+ " filename=\"maskgit-vqgan-imagenet-f16-256.bin\",\n",
65
+ ")\n",
66
+ "print(\"VQGAN tokenizer:\", vq_path)"
67
+ ]
68
+ },
69
+ {
70
+ "cell_type": "markdown",
71
+ "id": "d3ebc639",
72
+ "metadata": {},
73
+ "source": [
74
+ "## 2. Load STELLAR for reconstruction\n",
75
+ "\n",
76
+ "`purpose=\"reconstruct\"` builds the encoder **and** decoder and attaches the VQGAN tokenizer."
77
+ ]
78
+ },
79
+ {
80
+ "cell_type": "code",
81
+ "execution_count": null,
82
+ "id": "8b2e344f",
83
+ "metadata": {},
84
+ "outputs": [],
85
+ "source": [
86
+ "model = load_stellar(\"stellar-b16\", purpose=\"reconstruct\", vq_model=vq_path, device=device)\n",
87
+ "model.eval();"
88
+ ]
89
+ },
90
+ {
91
+ "cell_type": "markdown",
92
+ "id": "0c143298",
93
+ "metadata": {},
94
+ "source": [
95
+ "## 3. Load and preprocess an image\n",
96
+ "\n",
97
+ "Pass **raw** images in `[0, 1]`, resized to `224×224` with bicubic interpolation.\n",
98
+ "ImageNet mean/std normalization is applied **inside** the model — do not normalize yourself."
99
+ ]
100
+ },
101
+ {
102
+ "cell_type": "code",
103
+ "execution_count": null,
104
+ "id": "9deaef69",
105
+ "metadata": {},
106
+ "outputs": [],
107
+ "source": [
108
+ "from PIL import Image\n",
109
+ "import numpy as np\n",
110
+ "\n",
111
+ "# Replace with your own image path.\n",
112
+ "img_path = hf_hub_download(repo_id=\"huggingface/documentation-images\",\n",
113
+ " filename=\"beignets-task-guide.png\", repo_type=\"dataset\")\n",
114
+ "pil = Image.open(img_path).convert(\"RGB\")\n",
115
+ "\n",
116
+ "image = torch.from_numpy(np.asarray(pil)).permute(2, 0, 1).float() / 255.0 # (3, H, W) in [0,1]\n",
117
+ "image = F.interpolate(image[None], size=(224, 224), mode=\"bicubic\", align_corners=False).clamp(0, 1)\n",
118
+ "image = image.to(device)\n",
119
+ "print(\"input:\", tuple(image.shape), float(image.min()), float(image.max()))"
120
+ ]
121
+ },
122
+ {
123
+ "cell_type": "markdown",
124
+ "id": "f845698f",
125
+ "metadata": {},
126
+ "source": [
127
+ "## 4. Encode, then reconstruct\n",
128
+ "\n",
129
+ "First `encode` the image into its factorized features (sparse concept tokens + spatial\n",
130
+ "maps). `reconstruct` is the decoder half of STELLAR: it takes those factorized features\n",
131
+ "\n",
132
+ "and decodes them all the way to pixels. It returns the RGB pixels (`reconstruction`), thepredicted VQGAN token ids (`tokens`), and the raw codebook logits (`logits`)."
133
+ ]
134
+ },
135
+ {
136
+ "cell_type": "code",
137
+ "execution_count": null,
138
+ "id": "6e986c51",
139
+ "metadata": {},
140
+ "outputs": [],
141
+ "source": [
142
+ "with torch.no_grad():\n",
143
+ " features = model.encode(image) # factorized features\n",
144
+ " out = model.reconstruct(features) # or model.reconstruct(features[\"sparse\"], features[\"spatial\"])\n",
145
+ "\n",
146
+ "print(\"factorized features:\")\n",
147
+ "\n",
148
+ "for k in (\"sparse\", \"spatial\"): print(f\" {k:14s}\", tuple(v.shape))\n",
149
+ "\n",
150
+ " print(f\" {k:8s}\", tuple(features[k].shape))for k, v in out.items():\n",
151
+ "print(\"reconstruction outputs:\")"
152
+ ]
153
+ },
154
+ {
155
+ "cell_type": "markdown",
156
+ "id": "e74f4664",
157
+ "metadata": {},
158
+ "source": [
159
+ "## 5. Display original vs. reconstruction"
160
+ ]
161
+ },
162
+ {
163
+ "cell_type": "code",
164
+ "execution_count": null,
165
+ "id": "d552f2c2",
166
+ "metadata": {},
167
+ "outputs": [],
168
+ "source": [
169
+ "import matplotlib.pyplot as plt\n",
170
+ "\n",
171
+ "orig = image[0].permute(1, 2, 0).cpu().numpy()\n",
172
+ "recon = out[\"reconstruction\"][0].permute(1, 2, 0).cpu().numpy()\n",
173
+ "\n",
174
+ "fig, ax = plt.subplots(1, 2, figsize=(8, 4))\n",
175
+ "ax[0].imshow(orig); ax[0].set_title(\"original\"); ax[0].axis(\"off\")\n",
176
+ "ax[1].imshow(recon); ax[1].set_title(\"STELLAR reconstruction\"); ax[1].axis(\"off\")\n",
177
+ "plt.tight_layout(); plt.show()"
178
+ ]
179
+ },
180
+ {
181
+ "cell_type": "markdown",
182
+ "id": "10c7f3d1",
183
+ "metadata": {},
184
+ "source": [
185
+ "## Notes\n",
186
+ "\n",
187
+ "- Output resolution: **224×224** for the `/16` models (`stellar-b16`, `stellar-l16`, and the\n",
188
+ " `b8`/`b24` ablations) and **256×256** for the `/14` `stellar-h16` model.\n",
189
+ "- For larger / higher-fidelity reconstructions, swap in `stellar-l16` or `stellar-h16`.\n",
190
+ "- `reconstruct` takes the factorized features (not the image), so you can edit / analyze the\n",
191
+ " sparse tokens before decoding. It is gradient-transparent — wrap it in `torch.no_grad()`\n",
192
+ " for inference, or keep gradients enabled (e.g. for feature-inversion experiments)."
193
+ ]
194
+ }
195
+ ],
196
+ "metadata": {
197
+ "language_info": {
198
+ "name": "python"
199
+ }
200
+ },
201
+ "nbformat": 4,
202
+ "nbformat_minor": 5
203
+ }
factorization.svg ADDED
load_stellar.py ADDED
@@ -0,0 +1,119 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Convenience loader for the STELLAR models hosted on the Hugging Face Hub.
2
+
3
+ Requires the STELLAR code (https://github.com/microsoft/STELLAR) to be installed
4
+ and importable, plus ``huggingface_hub`` and ``safetensors``::
5
+
6
+ pip install huggingface_hub safetensors
7
+ # and install the STELLAR repo so that `src` is importable
8
+
9
+ The released checkpoints contain every trained STELLAR module, so a single file
10
+ serves three purposes via the ``purpose`` argument:
11
+
12
+ * ``"encode"`` – feature extraction (encoder only). No extra files needed.
13
+ * ``"reconstruct"`` – image reconstruction (encoder + decoder). Needs the
14
+ MaskGIT-VQGAN tokenizer (``vq_model=<path>``).
15
+ * ``"pretrain"`` – continued pretraining (all modules). Needs ``vq_model``.
16
+
17
+ Usage::
18
+
19
+ from load_stellar import load_stellar, list_models
20
+
21
+ print(list_models()) # available checkpoints
22
+ model = load_stellar("stellar-b16") # purpose="encode" (default)
23
+
24
+ import torch
25
+ img = torch.rand(1, 3, 224, 224) # values in [0, 1]
26
+ with torch.no_grad():
27
+ out = model.encode(img)
28
+ out["sparse"] # (1, K, D) sparse concept tokens
29
+ out["dense"] # (1, P, D) dense patch features
30
+ out["spatial"] # (1, P, K) per-token spatial maps
31
+ """
32
+ import json
33
+
34
+ import torch
35
+ from huggingface_hub import hf_hub_download
36
+ from safetensors.torch import load_file
37
+
38
+ from src.models.stellar_model import STELLARModel
39
+
40
+ REPO_ID = "microsoft/STELLAR"
41
+ PURPOSES = ("encode", "reconstruct", "pretrain")
42
+
43
+
44
+ def _load_index(repo_id=REPO_ID):
45
+ # Downloading config.json is what registers a download on the Hub, so always
46
+ # fetch it through hf_hub_download (do not read a local copy).
47
+ return json.load(open(hf_hub_download(repo_id, "config.json")))
48
+
49
+
50
+ def list_models(repo_id=REPO_ID):
51
+ """Return the list of available model names."""
52
+ return list(_load_index(repo_id)["models"].keys())
53
+
54
+
55
+ def load_stellar(model="stellar-b16", purpose="encode", vq_model=None,
56
+ repo_id=REPO_ID, device="cpu"):
57
+ """Download a STELLAR checkpoint from the Hub and build a ready-to-use model.
58
+
59
+ Args:
60
+ model: one of ``list_models()`` (e.g. ``"stellar-b16"``).
61
+ purpose: ``"encode"`` (features), ``"reconstruct"`` (decoder) or
62
+ ``"pretrain"`` (all modules).
63
+ vq_model: path to the MaskGIT-VQGAN tokenizer
64
+ (``maskgit-vqgan-imagenet-f16-256.bin``). Required for
65
+ ``reconstruct``/``pretrain`` when the model decodes into VQ tokens.
66
+ repo_id: Hugging Face repo id (default ``microsoft/STELLAR``).
67
+ device: device to move the model to.
68
+ """
69
+ if purpose not in PURPOSES:
70
+ raise ValueError(f"purpose must be one of {PURPOSES}, got {purpose!r}")
71
+
72
+ index = _load_index(repo_id)
73
+ if model not in index["models"]:
74
+ raise ValueError(f"Unknown model '{model}'. Available: {list(index['models'])}")
75
+ cfg = index["models"][model]
76
+
77
+ weights_path = hf_hub_download(repo_id, cfg["weights"])
78
+ state_dict = load_file(weights_path)
79
+
80
+ needs_vq = cfg.get("recon_type") == "vq"
81
+ if purpose in ("reconstruct", "pretrain") and needs_vq and vq_model is None:
82
+ raise ValueError(
83
+ f"'{model}' reconstructs into VQ tokens; pass vq_model=<path to "
84
+ "maskgit-vqgan-imagenet-f16-256.bin> (download it from "
85
+ "https://huggingface.co/fun-research/TiTok).")
86
+
87
+ common = dict(
88
+ num_sparse_tokens=cfg["num_sparse_tokens"],
89
+ num_decoder_layers=cfg["num_decoder_layers"],
90
+ spatial_temp=cfg["spatial_temp"],
91
+ do_cls=cfg["do_cls"],
92
+ vit_pretrained=cfg["backbone"],
93
+ )
94
+ if purpose == "encode":
95
+ net = STELLARModel(**common, do_recon=False, do_clustering=False, vq_model=None)
96
+ elif purpose == "reconstruct":
97
+ net = STELLARModel(**common, do_recon=True, do_clustering=False, vq_model=vq_model)
98
+ else: # pretrain
99
+ net = STELLARModel(**common, do_recon=True, do_clustering=True, vq_model=vq_model)
100
+
101
+ missing, _ = net.load_state_dict(state_dict, strict=False)
102
+ # `tokenizer.*` come from `vq_model`, never from our weights. Anything else
103
+ # the built model still needs is a genuine error.
104
+ essential_missing = [k for k in missing if not k.startswith("tokenizer.")]
105
+ if essential_missing:
106
+ raise RuntimeError(f"Missing essential weights: {essential_missing[:8]}")
107
+
108
+ net.to(device)
109
+ if purpose == "encode":
110
+ net.eval()
111
+ return net
112
+
113
+
114
+ if __name__ == "__main__":
115
+ print("Available models:", list_models())
116
+ m = load_stellar("stellar-b16")
117
+ with torch.no_grad():
118
+ out = m.encode(torch.rand(1, 3, 224, 224))
119
+ print({k: tuple(v.shape) for k, v in out.items() if torch.is_tensor(v)})
stellar-b16.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:061489f86230a9609c5794018dadad3feaecbff9b31df1e4cab702f9c4414500
3
+ size 483090112
stellar-b24.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6859aead6faa6095d51d0c60821f1f903c4409ab64fd887527a4906e05993295
3
+ size 483114688
stellar-b8.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9b81262070b5b362c93b8ad24020f6c3de51a6fcce82411cf30ad9b82a1daf0
3
+ size 483065536
stellar-h16.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d930de83e0cdbb902a70dbd5a954acf7e69591c5759bca59820677c8ff46ff09
3
+ size 2685054144
stellar-l16.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7428079417b5ec89b609d5ef52c3230ef97423dfa62c73264d05e3d98c9944dd
3
+ size 1363369280