Deagin Claude Opus 5 (1M context) commited on
Commit
e510bac
·
1 Parent(s): 88b423e

Fix 16GB OOM: never build the SigLIP2 text tower on the Space

Browse files

RADIO's SigLIP2Adaptor.__init__ calls AutoModel.from_pretrained on
google/siglip2-giant-opt-patch16-384 — downloading 7.49 GB and materializing
the entire giant model — purely to keep model.text_model, then del's the rest.
On a 16 GB Space that download's page-cache and mmap footprint, stacked on the
~5.5 GB of real allocations, exceeds the container limit before anything can
be freed. The Space was crash-looping: OOM, restart, re-download, OOM.

The previous mitigation (0462568, 8266ce5) could not work. It freed the tower
*after* the peak, and freed 2.8 GB rather than the 7.5 GB the comments
claimed — 7.49 GB is the download size, not the resident text_model.

Our prompt set is 11 hardcoded strings, so the tower is not needed at all:

- export_text_embeddings.py runs once offline and writes text_embeddings.pt
(11 x 1536 fp32, 67 KB), committed via LFS.
- At startup, pop the `siglip2` factory from RADIO's adaptor_registry.
create_adaptor falls back to GenericAdaptor for unregistered names, and
GenericAdaptor builds head_mlp/feat_mlp purely from the RADIO checkpoint's
own state, so AutoModel.from_pretrained never runs.

The vision path is bit-identical: same class, same weights, same code.
verify_stripped_adaptor.py captures a reference from the full path and asserts
max abs diff == 0.0, plus that no adaptor submodule exceeds 1 GB so the tower
cannot silently return.

boot downloads 9.06 GB -> 1.57 GB
adaptor params 3044 MB -> 211 MB

Guards: load_model() refuses to start if the committed prompts drift from the
source list, and zero_shot_segment() rejects prompt overrides, since there is
no text encoder available to embed them.

Also: drop the dead _free_text_encoder, correct the ~7.5 GB comments, fix the
stale "ZeroGPU / loaded to CPU" note in AGENTS.md (changed by fe58fef), and add
the two solar prompts missing from the AGENTS.md prompt list.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitignore CHANGED
@@ -17,4 +17,6 @@ build/
17
  .ipynb_checkpoints/
18
  *.log
19
  .DS_Store
20
- Thumbs.db
 
 
 
17
  .ipynb_checkpoints/
18
  *.log
19
  .DS_Store
20
+ Thumbs.db
21
+ # Local verification artifact (regenerate with export_text_embeddings.py)
22
+ _ref_features.pt
AGENTS.md CHANGED
@@ -43,6 +43,8 @@ Address -> Geocode -> Google Solar API -> [RGB GeoTIFF, Building Mask, DSM]
43
  | `geo_export.py` | Pixel-to-geo conversion, contour finding, Douglas-Peucker, GeoJSON |
44
  | `pipeline.py` | PipelineResult dataclass, `run()` orchestration, overlay visualization |
45
  | `app.py` | Gradio Blocks UI, `@spaces.GPU` decorator, input/output wiring |
 
 
46
 
47
  ### Key Constants
48
  - DSM pixel size: 0.1 m/pixel (Google Solar API)
@@ -57,9 +59,12 @@ Address -> Geocode -> Google Solar API -> [RGB GeoTIFF, Building Mask, DSM]
57
  - `siglip2-g` — Zero-shot text-image alignment (used for RADSeg)
58
 
59
  ### Zero-Shot Text Prompts
60
- Roof classes: "flat roof plane", "pitched roof plane", "hip roof plane", "gable roof plane"
 
61
  Non-roof: "sky", "ground", "tree", "wall", "shadow"
62
 
 
 
63
  ## Common Issues
64
 
65
  ### C-RADIOv4-H resolution error
@@ -81,5 +86,39 @@ If appearance-based splitting is too aggressive, the 15% threshold in
81
 
82
  ## Deployment
83
 
84
- HuggingFace Space with Gradio SDK + ZeroGPU. Model loaded to CPU at startup,
85
- moved to GPU per-request via `@spaces.GPU`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
43
  | `geo_export.py` | Pixel-to-geo conversion, contour finding, Douglas-Peucker, GeoJSON |
44
  | `pipeline.py` | PipelineResult dataclass, `run()` orchestration, overlay visualization |
45
  | `app.py` | Gradio Blocks UI, `@spaces.GPU` decorator, input/output wiring |
46
+ | `export_text_embeddings.py` | One-off: precompute prompt embeddings (heavy, offline) |
47
+ | `verify_stripped_adaptor.py` | Assert the stripped load path is numerically identical |
48
 
49
  ### Key Constants
50
  - DSM pixel size: 0.1 m/pixel (Google Solar API)
 
59
  - `siglip2-g` — Zero-shot text-image alignment (used for RADSeg)
60
 
61
  ### Zero-Shot Text Prompts
62
+ Roof classes: "flat roof plane", "pitched roof plane", "hip roof plane", "gable roof plane",
63
+ "solar panel array on roof", "photovoltaic panels installed on roof"
64
  Non-roof: "sky", "ground", "tree", "wall", "shadow"
65
 
66
+ These 11 strings are frozen into `text_embeddings.pt` — see Deployment.
67
+
68
  ## Common Issues
69
 
70
  ### C-RADIOv4-H resolution error
 
86
 
87
  ## Deployment
88
 
89
+ HuggingFace Space with Gradio SDK, on a dedicated GPU. `app.py` loads straight
90
+ to `cuda` at startup when it is available; the `@spaces.GPU` decorator is kept
91
+ so the app still runs on ZeroGPU, where `torch.cuda.is_available()` is False at
92
+ import time and the model loads to CPU instead.
93
+
94
+ ### The SigLIP2 text tower is never loaded (memory)
95
+
96
+ RADIO's `SigLIP2Adaptor.__init__` calls
97
+ `AutoModel.from_pretrained("google/siglip2-giant-opt-patch16-384")` — pulling a
98
+ 7.49 GB checkpoint and materializing the whole giant model — purely to keep
99
+ `model.text_model`, then `del`s the rest. On a 16 GB Space that download plus
100
+ its page-cache and mmap footprint is enough to exceed the container limit
101
+ before any of it can be freed. (Note the limit is a cgroup limit: page cache
102
+ from downloaded files counts against it, which is why the failure does not
103
+ reproduce from local RSS measurements.)
104
+
105
+ Our prompt set is fixed, so:
106
+
107
+ 1. `export_text_embeddings.py` runs **once** on a machine with ~14 GB headroom,
108
+ encodes `ROOF_PROMPTS + NON_ROOF_PROMPTS`, and writes `text_embeddings.pt`
109
+ (11 x 1536 fp32, ~67 KB). Committed via LFS.
110
+ 2. At startup `radio_backbone._strip_siglip2_text_adaptor()` pops the
111
+ `siglip2` factory out of RADIO's `adaptor_registry`. `create_adaptor` falls
112
+ back to `GenericAdaptor` for unregistered names, and `GenericAdaptor` builds
113
+ `head_mlp`/`feat_mlp` purely from the RADIO checkpoint's own state — so
114
+ `AutoModel.from_pretrained` never runs.
115
+
116
+ The vision path is **bit-identical**: same class, same weights, same code.
117
+ `verify_stripped_adaptor.py` asserts max abs diff == 0.0 against a reference
118
+ captured from the full path. Adaptor params drop 3044 MB -> 211 MB and boot
119
+ downloads drop 9.06 GB -> 1.57 GB.
120
+
121
+ **Changing `ROOF_PROMPTS` or `NON_ROOF_PROMPTS` requires re-running
122
+ `export_text_embeddings.py`.** `load_model()` refuses to start if the committed
123
+ prompts do not match the source, and `zero_shot_segment()` rejects custom
124
+ prompt overrides, since there is no text encoder to embed them with.
README.md CHANGED
@@ -55,3 +55,7 @@ Address -> Geocode -> Google Solar API (RGB + Mask + DSM)
55
  | `geo_export.py` | GeoJSON polygon extraction |
56
  | `pipeline.py` | Orchestration |
57
  | `app.py` | Gradio UI |
 
 
 
 
 
55
  | `geo_export.py` | GeoJSON polygon extraction |
56
  | `pipeline.py` | Orchestration |
57
  | `app.py` | Gradio UI |
58
+
59
+ The SigLIP2 text encoder is **not** loaded at runtime — the fixed prompt set's
60
+ embeddings are precomputed into `text_embeddings.pt` (~67 KB), which keeps the
61
+ Space inside its memory limit. See `AGENTS.md` for details.
export_text_embeddings.py ADDED
@@ -0,0 +1,102 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Offline export of the SigLIP2 text embeddings for RoofSight's fixed prompts.
2
+
3
+ Run this ONCE on a machine with RAM headroom (~14 GB peak). It loads the full
4
+ `siglip2-g` adaptor — which instantiates the entire google/siglip2-giant model
5
+ just to keep its text tower — encodes the fixed prompt set, and writes the
6
+ resulting (num_prompts, D) tensor to disk.
7
+
8
+ The Space then loads with the `siglip2` adaptor factory removed from RADIO's
9
+ registry, so the text tower is never built there. See radio_backbone.load_model.
10
+
11
+ Also captures a reference feature map so the stripped runtime path can be
12
+ verified as numerically identical:
13
+
14
+ python export_text_embeddings.py # export + reference
15
+ python verify_stripped_adaptor.py # compare
16
+ """
17
+
18
+ import os
19
+ import sys
20
+
21
+ import numpy as np
22
+ import torch
23
+ from PIL import Image
24
+
25
+ from radio_backbone import ROOF_PROMPTS, NON_ROOF_PROMPTS, PATCH_SIZE
26
+
27
+ OUT_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "text_embeddings.pt")
28
+ REF_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_ref_features.pt")
29
+ REF_IMAGE = os.path.join(
30
+ os.path.dirname(os.path.abspath(__file__)),
31
+ "_run_output", "1310_susan_circle", "original.png",
32
+ )
33
+
34
+ # CPU + fp32 throughout: the reference must be deterministic so the parity
35
+ # check measures the adaptor swap, not autocast/GPU nondeterminism.
36
+ DEVICE = "cpu"
37
+
38
+
39
+ def main():
40
+ all_labels = ROOF_PROMPTS + NON_ROOF_PROMPTS
41
+ print(f"Loading FULL c-radio_v4-h + siglip2-g adaptor on {DEVICE} (this is the heavy path)...")
42
+
43
+ model = torch.hub.load(
44
+ "NVlabs/RADIO",
45
+ "radio_model",
46
+ version="c-radio_v4-h",
47
+ adaptor_names=["siglip2-g"],
48
+ vitdet_window_size=8,
49
+ progress=True,
50
+ skip_validation=True,
51
+ trust_repo=True,
52
+ )
53
+ model.eval()
54
+ model.to(DEVICE)
55
+
56
+ adaptor = model.adaptors["siglip2-g"]
57
+ assert hasattr(adaptor, "encode_text"), (
58
+ "adaptor has no encode_text — the siglip2 factory was not used. "
59
+ "This script must run WITHOUT the registry patch."
60
+ )
61
+
62
+ print(f"Encoding {len(all_labels)} prompts...")
63
+ text_input = adaptor.tokenizer(all_labels)
64
+ text_input = {k: v.to(DEVICE) for k, v in text_input.items()}
65
+ with torch.no_grad():
66
+ embeddings = adaptor.encode_text(text_input, normalize=True).float().cpu().clone()
67
+
68
+ print(f" embeddings: {tuple(embeddings.shape)} dtype={embeddings.dtype}")
69
+ torch.save({"prompts": all_labels, "embeddings": embeddings}, OUT_PATH)
70
+ print(f" wrote {OUT_PATH} ({os.path.getsize(OUT_PATH) / 1024:.1f} KB)")
71
+
72
+ # --- Reference feature map, for the parity check ---
73
+ if not os.path.exists(REF_IMAGE):
74
+ print(f"\nNo reference image at {REF_IMAGE}; skipping reference capture.")
75
+ return
76
+
77
+ print(f"\nCapturing reference features from {REF_IMAGE}...")
78
+ image = np.array(Image.open(REF_IMAGE).convert("RGB"))
79
+ x = torch.from_numpy(image).permute(2, 0, 1).float().div_(255.0).unsqueeze(0)
80
+ snapped = model.get_nearest_supported_resolution(*x.shape[-2:])
81
+ if snapped != x.shape[-2:]:
82
+ x = torch.nn.functional.interpolate(x, snapped, mode="bilinear", align_corners=False)
83
+
84
+ with torch.no_grad():
85
+ out = model(x.to(DEVICE))
86
+ _, features = out["siglip2-g"]
87
+
88
+ torch.save(
89
+ {
90
+ "features": features.float().cpu().clone(),
91
+ "snapped": tuple(snapped),
92
+ "image_shape": image.shape,
93
+ },
94
+ REF_PATH,
95
+ )
96
+ print(f" features: {tuple(features.shape)}")
97
+ print(f" wrote {REF_PATH}")
98
+ print("\nDone. Now run: python verify_stripped_adaptor.py")
99
+
100
+
101
+ if __name__ == "__main__":
102
+ sys.exit(main())
radio_backbone.py CHANGED
@@ -3,12 +3,21 @@
3
  Distills DINOv3-7B + SAM3 + SigLIP2 into a single 631M-param encoder.
4
  Uses the SigLIP2 adaptor head for zero-shot text-prompted roof segmentation.
5
 
6
- Memory optimization: the SigLIP2 text encoder (~7.5GB) is loaded once to
7
- pre-compute text embeddings for our fixed prompt set, then freed from RAM.
8
- Only the vision backbone + adaptor projection head are kept (~1.6GB).
 
 
 
 
 
 
 
 
9
  """
10
 
11
- import gc
 
12
 
13
  import numpy as np
14
  import torch
@@ -22,6 +31,9 @@ PATCH_SIZE = 16
22
  # Solar-array prompts count as "roof" — the pixels underneath still belong
23
  # to a roof plane; without these prompts, RADIO pushes solar-covered
24
  # pixels toward "wall"/"shadow" and fusion carves holes in the roof mask.
 
 
 
25
  ROOF_PROMPTS = [
26
  "flat roof plane",
27
  "pitched roof plane",
@@ -38,17 +50,88 @@ NON_ROOF_PROMPTS = [
38
  "shadow",
39
  ]
40
 
 
 
 
 
41
  # Module-level caches
42
  _model = None
43
  _device = None
44
- _cached_text_embeddings = None # Pre-computed for ROOF_PROMPTS + NON_ROOF_PROMPTS
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
45
 
46
 
47
  def load_model(device: str = "cuda", vitdet_window_size: int = 8):
48
- """Load C-RADIOv4-H with siglip2-g adaptor.
49
 
50
- Pre-computes text embeddings for fixed prompts, then frees the
51
- SigLIP2 text encoder to reclaim ~7.5GB of RAM.
52
 
53
  Args:
54
  device: 'cuda' or 'cpu'. For ZeroGPU, load to 'cpu' at startup
@@ -66,6 +149,13 @@ def load_model(device: str = "cuda", vitdet_window_size: int = 8):
66
 
67
  print("Loading C-RADIOv4-H (631M params)...")
68
 
 
 
 
 
 
 
 
69
  kwargs = {
70
  "version": "c-radio_v4-h",
71
  "adaptor_names": ["siglip2-g"],
@@ -80,66 +170,12 @@ def load_model(device: str = "cuda", vitdet_window_size: int = 8):
80
  _model.to(device)
81
  _device = device
82
 
83
- # --- Pre-compute text embeddings, then free the text encoder ---
84
- all_labels = ROOF_PROMPTS + NON_ROOF_PROMPTS
85
- print(f"Caching text embeddings for {len(all_labels)} prompts...")
86
-
87
- sig2_adaptor = _model.adaptors["siglip2-g"]
88
- text_input = sig2_adaptor.tokenizer(all_labels).to(device)
89
- with torch.no_grad():
90
- _cached_text_embeddings = sig2_adaptor.encode_text(
91
- text_input, normalize=True
92
- ).cpu().clone()
93
 
94
- # Free the heavy SigLIP2 text encoder (~7.5GB)
95
- _free_text_encoder(sig2_adaptor)
96
-
97
- print(f"C-RADIOv4-H loaded on {device} (text encoder freed, embeddings cached)")
98
  return _model
99
 
100
 
101
- def _free_text_encoder(adaptor):
102
- """Delete the SigLIP2 text encoder from the adaptor to free ~7.5GB RAM.
103
-
104
- Only targets modules > 1GB — the text encoder is ~7.5GB while vision
105
- projection heads (feat_mlp, summary_mlp, etc.) are < 1GB.
106
- """
107
- freed = 0
108
-
109
- # Log all modules so we can see what's there
110
- print(" Adaptor modules:")
111
- for name, module in adaptor.named_children():
112
- param_bytes = sum(
113
- p.numel() * p.element_size() for p in module.parameters()
114
- )
115
- print(f" {name}: {param_bytes / 1e6:.0f} MB")
116
-
117
- # Only delete modules > 1GB (the text encoder is ~7.5GB,
118
- # vision projection heads like feat_mlp are < 1GB)
119
- for name in list(dict(adaptor.named_children()).keys()):
120
- module = getattr(adaptor, name)
121
- param_bytes = sum(
122
- p.numel() * p.element_size() for p in module.parameters()
123
- )
124
- if param_bytes > 1_000_000_000: # > 1GB
125
- size_gb = param_bytes / 1e9
126
- print(f" Freeing adaptor.{name} ({size_gb:.1f} GB)")
127
- try:
128
- delattr(adaptor, name)
129
- freed += param_bytes
130
- except Exception:
131
- pass
132
-
133
- gc.collect()
134
- if torch.cuda.is_available():
135
- torch.cuda.empty_cache()
136
-
137
- if freed > 0:
138
- print(f" Total freed: {freed / 1e9:.1f} GB")
139
- else:
140
- print(" Warning: no module > 1GB found — text encoder may still be in RAM")
141
-
142
-
143
  def get_model():
144
  """Get the cached model, loading if necessary."""
145
  global _model
@@ -226,6 +262,15 @@ def zero_shot_segment(
226
 
227
  all_labels = roof_prompts + non_roof_prompts
228
 
 
 
 
 
 
 
 
 
 
229
  # T4/older GPUs have no native bfloat16 — pick fp16 unless the GPU is Ampere+
230
  if device == "cuda" and torch.cuda.is_available():
231
  major, _ = torch.cuda.get_device_capability()
 
3
  Distills DINOv3-7B + SAM3 + SigLIP2 into a single 631M-param encoder.
4
  Uses the SigLIP2 adaptor head for zero-shot text-prompted roof segmentation.
5
 
6
+ Memory: RADIO's `SigLIP2Adaptor` builds the entire google/siglip2-giant model
7
+ (~7.5 GB, vision tower included) purely to keep its text tower, then discards
8
+ the rest — a peak large enough to OOM a 16 GB Space before anything can be
9
+ freed. Our prompt set is fixed, so its embeddings are precomputed offline
10
+ (see export_text_embeddings.py) and committed as text_embeddings.pt (~67 KB).
11
+
12
+ At runtime we remove the `siglip2` factory from RADIO's adaptor registry.
13
+ AdaptorRegistry.create_adaptor falls back to GenericAdaptor for any
14
+ unregistered name, and GenericAdaptor builds head_mlp/feat_mlp purely from the
15
+ RADIO checkpoint's own state — so the vision path is bit-identical and the text
16
+ tower is never constructed. Verified by verify_stripped_adaptor.py.
17
  """
18
 
19
+ import os
20
+ import sys
21
 
22
  import numpy as np
23
  import torch
 
31
  # Solar-array prompts count as "roof" — the pixels underneath still belong
32
  # to a roof plane; without these prompts, RADIO pushes solar-covered
33
  # pixels toward "wall"/"shadow" and fusion carves holes in the roof mask.
34
+ #
35
+ # NOTE: changing this list invalidates text_embeddings.pt. Re-run
36
+ # export_text_embeddings.py; load_model() refuses to start on a mismatch.
37
  ROOF_PROMPTS = [
38
  "flat roof plane",
39
  "pitched roof plane",
 
50
  "shadow",
51
  ]
52
 
53
+ TEXT_EMBEDDINGS_PATH = os.path.join(
54
+ os.path.dirname(os.path.abspath(__file__)), "text_embeddings.pt"
55
+ )
56
+
57
  # Module-level caches
58
  _model = None
59
  _device = None
60
+ _cached_text_embeddings = None # Loaded from TEXT_EMBEDDINGS_PATH at startup
61
+
62
+
63
+ def _ensure_radio_repo() -> str:
64
+ """Return the cached NVlabs/RADIO repo dir, fetching it if absent.
65
+
66
+ torch.hub.list() downloads and caches the repo without building a model,
67
+ which is what we need: the registry must be patched before radio_model()
68
+ runs, and the `radio` package only becomes importable once it is on disk.
69
+ """
70
+ repo_dir = os.path.join(torch.hub.get_dir(), "NVlabs_RADIO_main")
71
+ if not os.path.isdir(repo_dir):
72
+ torch.hub.list("NVlabs/RADIO", trust_repo=True, skip_validation=True)
73
+ return repo_dir
74
+
75
+
76
+ def _strip_siglip2_text_adaptor():
77
+ """Remove the `siglip2` adaptor factory so GenericAdaptor is used instead.
78
+
79
+ This is what keeps the Space under its memory limit: SigLIP2Adaptor's
80
+ __init__ calls AutoModel.from_pretrained on the giant checkpoint. Dropping
81
+ the factory means that line never runs, so the 7.49 GB download and its
82
+ CPU-RAM spike never happen.
83
+ """
84
+ repo_dir = _ensure_radio_repo()
85
+ if repo_dir not in sys.path:
86
+ sys.path.insert(0, repo_dir)
87
+
88
+ from radio.adaptor_registry import adaptor_registry
89
+
90
+ removed = adaptor_registry._registry.pop("siglip2", None)
91
+ if removed is not None:
92
+ print(" siglip2 text adaptor stripped (text tower will not be built)")
93
+ else:
94
+ print(" warning: no 'siglip2' factory in registry — RADIO's API may have changed")
95
+
96
+
97
+ def _load_text_embeddings() -> torch.Tensor:
98
+ """Load the precomputed prompt embeddings, refusing to run if they drifted."""
99
+ if not os.path.exists(TEXT_EMBEDDINGS_PATH):
100
+ raise FileNotFoundError(
101
+ f"{TEXT_EMBEDDINGS_PATH} is missing. Run export_text_embeddings.py "
102
+ "on a machine with ~14 GB of RAM headroom and commit the result."
103
+ )
104
+
105
+ blob = torch.load(TEXT_EMBEDDINGS_PATH, map_location="cpu")
106
+ expected = ROOF_PROMPTS + NON_ROOF_PROMPTS
107
+ if blob["prompts"] != expected:
108
+ raise ValueError(
109
+ "text_embeddings.pt does not match the current prompt list.\n"
110
+ f" artifact: {blob['prompts']}\n"
111
+ f" expected: {expected}\n"
112
+ "Re-run export_text_embeddings.py."
113
+ )
114
+ return blob["embeddings"]
115
+
116
+
117
+ def _assert_no_text_tower(adaptor):
118
+ """Fail loudly if a text tower leaked into the adaptor despite the strip."""
119
+ print(" Adaptor modules:")
120
+ for name, module in adaptor.named_children():
121
+ mb = sum(p.numel() * p.element_size() for p in module.parameters()) / 1e6
122
+ print(f" {name}: {mb:.0f} MB")
123
+ if mb > 1000:
124
+ raise RuntimeError(
125
+ f"adaptor.{name} is {mb:.0f} MB — the text tower was built anyway. "
126
+ "Startup would exceed the Space memory limit; aborting."
127
+ )
128
 
129
 
130
  def load_model(device: str = "cuda", vitdet_window_size: int = 8):
131
+ """Load C-RADIOv4-H with the siglip2-g vision projection only.
132
 
133
+ The SigLIP2 text tower is never constructed; prompt embeddings come from
134
+ the committed text_embeddings.pt instead.
135
 
136
  Args:
137
  device: 'cuda' or 'cpu'. For ZeroGPU, load to 'cpu' at startup
 
149
 
150
  print("Loading C-RADIOv4-H (631M params)...")
151
 
152
+ # Load the embeddings first: cheap, and it fails fast on prompt drift
153
+ # before we spend a minute pulling 1.57 GB of weights.
154
+ _cached_text_embeddings = _load_text_embeddings()
155
+ print(f" text embeddings: {tuple(_cached_text_embeddings.shape)} (precomputed)")
156
+
157
+ _strip_siglip2_text_adaptor()
158
+
159
  kwargs = {
160
  "version": "c-radio_v4-h",
161
  "adaptor_names": ["siglip2-g"],
 
170
  _model.to(device)
171
  _device = device
172
 
173
+ _assert_no_text_tower(_model.adaptors["siglip2-g"])
 
 
 
 
 
 
 
 
 
174
 
175
+ print(f"C-RADIOv4-H loaded on {device} (text tower never built)")
 
 
 
176
  return _model
177
 
178
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
179
  def get_model():
180
  """Get the cached model, loading if necessary."""
181
  global _model
 
262
 
263
  all_labels = roof_prompts + non_roof_prompts
264
 
265
+ # The text encoder is not loaded, so embeddings cannot be computed on the
266
+ # fly — custom prompts would silently score against the wrong vectors.
267
+ if all_labels != ROOF_PROMPTS + NON_ROOF_PROMPTS:
268
+ raise ValueError(
269
+ "zero_shot_segment received prompts that differ from the ones "
270
+ "baked into text_embeddings.pt. Edit ROOF_PROMPTS/NON_ROOF_PROMPTS "
271
+ "and re-run export_text_embeddings.py to change the prompt set."
272
+ )
273
+
274
  # T4/older GPUs have no native bfloat16 — pick fp16 unless the GPU is Ampere+
275
  if device == "cuda" and torch.cuda.is_available():
276
  major, _ = torch.cuda.get_device_capability()
text_embeddings.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:42518c9307817c52b0b851550a160953edfc56148bb6107b02799b65e1c1db1a
3
+ size 69060
verify_stripped_adaptor.py ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Verify the stripped-adaptor load path is numerically identical to the full one.
2
+
3
+ Loads c-radio_v4-h with the `siglip2` adaptor factory removed from RADIO's
4
+ registry — so GenericAdaptor is built from the checkpoint state instead of
5
+ SigLIP2Adaptor — then compares the resulting feature map against the reference
6
+ captured by export_text_embeddings.py.
7
+
8
+ Expected result: max abs diff == 0.0. Same class, same weights, same code path;
9
+ the only thing removed is the text tower, which never touches the vision path.
10
+ """
11
+
12
+ import os
13
+ import sys
14
+
15
+ import numpy as np
16
+ import torch
17
+ from PIL import Image
18
+
19
+ import radio_backbone
20
+ from radio_backbone import ROOF_PROMPTS, NON_ROOF_PROMPTS
21
+
22
+ HERE = os.path.dirname(os.path.abspath(__file__))
23
+ REF_PATH = os.path.join(HERE, "_ref_features.pt")
24
+ EMB_PATH = os.path.join(HERE, "text_embeddings.pt")
25
+ REF_IMAGE = os.path.join(HERE, "_run_output", "1310_susan_circle", "original.png")
26
+
27
+ DEVICE = "cpu"
28
+
29
+
30
+ def main():
31
+ if not os.path.exists(REF_PATH):
32
+ print(f"Missing {REF_PATH}. Run export_text_embeddings.py first.")
33
+ return 1
34
+
35
+ # --- 1. Embeddings artifact sanity ---
36
+ blob = torch.load(EMB_PATH, map_location="cpu")
37
+ expected = ROOF_PROMPTS + NON_ROOF_PROMPTS
38
+ assert blob["prompts"] == expected, "prompt drift between artifact and source"
39
+ emb = blob["embeddings"]
40
+ norms = emb.norm(dim=-1)
41
+ print(f"text_embeddings.pt: {tuple(emb.shape)}, dtype={emb.dtype}")
42
+ print(f" L2 norms: min={norms.min():.6f} max={norms.max():.6f} (expect ~1.0, normalize=True)")
43
+ assert torch.allclose(norms, torch.ones_like(norms), atol=1e-4), "embeddings are not unit-norm"
44
+
45
+ # --- 2. Load with the text adaptor stripped ---
46
+ print("\nLoading with `siglip2` factory removed from the registry...")
47
+ radio_backbone._strip_siglip2_text_adaptor()
48
+
49
+ model = torch.hub.load(
50
+ "NVlabs/RADIO", "radio_model",
51
+ version="c-radio_v4-h", adaptor_names=["siglip2-g"],
52
+ vitdet_window_size=8, progress=True, skip_validation=True, trust_repo=True,
53
+ )
54
+ model.eval()
55
+ model.to(DEVICE)
56
+
57
+ adaptor = model.adaptors["siglip2-g"]
58
+ print(f" adaptor class: {type(adaptor).__name__}")
59
+ assert type(adaptor).__name__ == "GenericAdaptor", "expected GenericAdaptor fallback"
60
+ assert not hasattr(adaptor, "text_model"), "text_model still present!"
61
+ assert not hasattr(adaptor, "encode_text"), "encode_text still present!"
62
+
63
+ for name, module in adaptor.named_children():
64
+ mb = sum(p.numel() * p.element_size() for p in module.parameters()) / 1e6
65
+ print(f" {name}: {mb:.0f} MB")
66
+ assert mb < 1000, f"{name} is {mb:.0f} MB — a text tower may have leaked in"
67
+
68
+ # --- 3. Feature parity ---
69
+ ref = torch.load(REF_PATH, map_location="cpu")
70
+ image = np.array(Image.open(REF_IMAGE).convert("RGB"))
71
+ x = torch.from_numpy(image).permute(2, 0, 1).float().div_(255.0).unsqueeze(0)
72
+ snapped = model.get_nearest_supported_resolution(*x.shape[-2:])
73
+ if snapped != x.shape[-2:]:
74
+ x = torch.nn.functional.interpolate(x, snapped, mode="bilinear", align_corners=False)
75
+ assert tuple(snapped) == ref["snapped"], f"resolution drift: {snapped} vs {ref['snapped']}"
76
+
77
+ print("\nRunning forward pass...")
78
+ with torch.no_grad():
79
+ out = model(x.to(DEVICE))
80
+ _, features = out["siglip2-g"]
81
+ features = features.float().cpu()
82
+
83
+ ref_feat = ref["features"]
84
+ assert features.shape == ref_feat.shape, f"shape drift: {features.shape} vs {ref_feat.shape}"
85
+ diff = (features - ref_feat).abs()
86
+ print(f" shape: {tuple(features.shape)}")
87
+ print(f" max abs diff: {diff.max().item():.3e}")
88
+ print(f" mean abs diff: {diff.mean().item():.3e}")
89
+
90
+ if diff.max().item() == 0.0:
91
+ print("\nPASS — bit-identical to the full adaptor path.")
92
+ return 0
93
+ if diff.max().item() < 1e-5:
94
+ print("\nPASS — identical within float tolerance.")
95
+ return 0
96
+ print("\nFAIL — features diverged. Do not deploy.")
97
+ return 1
98
+
99
+
100
+ if __name__ == "__main__":
101
+ sys.exit(main())