Spaces:
Sleeping
Fix 16GB OOM: never build the SigLIP2 text tower on the Space
Browse filesRADIO's SigLIP2Adaptor.__init__ calls AutoModel.from_pretrained on
google/siglip2-giant-opt-patch16-384 — downloading 7.49 GB and materializing
the entire giant model — purely to keep model.text_model, then del's the rest.
On a 16 GB Space that download's page-cache and mmap footprint, stacked on the
~5.5 GB of real allocations, exceeds the container limit before anything can
be freed. The Space was crash-looping: OOM, restart, re-download, OOM.
The previous mitigation (0462568, 8266ce5) could not work. It freed the tower
*after* the peak, and freed 2.8 GB rather than the 7.5 GB the comments
claimed — 7.49 GB is the download size, not the resident text_model.
Our prompt set is 11 hardcoded strings, so the tower is not needed at all:
- export_text_embeddings.py runs once offline and writes text_embeddings.pt
(11 x 1536 fp32, 67 KB), committed via LFS.
- At startup, pop the `siglip2` factory from RADIO's adaptor_registry.
create_adaptor falls back to GenericAdaptor for unregistered names, and
GenericAdaptor builds head_mlp/feat_mlp purely from the RADIO checkpoint's
own state, so AutoModel.from_pretrained never runs.
The vision path is bit-identical: same class, same weights, same code.
verify_stripped_adaptor.py captures a reference from the full path and asserts
max abs diff == 0.0, plus that no adaptor submodule exceeds 1 GB so the tower
cannot silently return.
boot downloads 9.06 GB -> 1.57 GB
adaptor params 3044 MB -> 211 MB
Guards: load_model() refuses to start if the committed prompts drift from the
source list, and zero_shot_segment() rejects prompt overrides, since there is
no text encoder available to embed them.
Also: drop the dead _free_text_encoder, correct the ~7.5 GB comments, fix the
stale "ZeroGPU / loaded to CPU" note in AGENTS.md (changed by fe58fef), and add
the two solar prompts missing from the AGENTS.md prompt list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- .gitignore +3 -1
- AGENTS.md +42 -3
- README.md +4 -0
- export_text_embeddings.py +102 -0
- radio_backbone.py +109 -64
- text_embeddings.pt +3 -0
- verify_stripped_adaptor.py +101 -0
|
@@ -17,4 +17,6 @@ build/
|
|
| 17 |
.ipynb_checkpoints/
|
| 18 |
*.log
|
| 19 |
.DS_Store
|
| 20 |
-
Thumbs.db
|
|
|
|
|
|
|
|
|
| 17 |
.ipynb_checkpoints/
|
| 18 |
*.log
|
| 19 |
.DS_Store
|
| 20 |
+
Thumbs.db
|
| 21 |
+
# Local verification artifact (regenerate with export_text_embeddings.py)
|
| 22 |
+
_ref_features.pt
|
|
@@ -43,6 +43,8 @@ Address -> Geocode -> Google Solar API -> [RGB GeoTIFF, Building Mask, DSM]
|
|
| 43 |
| `geo_export.py` | Pixel-to-geo conversion, contour finding, Douglas-Peucker, GeoJSON |
|
| 44 |
| `pipeline.py` | PipelineResult dataclass, `run()` orchestration, overlay visualization |
|
| 45 |
| `app.py` | Gradio Blocks UI, `@spaces.GPU` decorator, input/output wiring |
|
|
|
|
|
|
|
| 46 |
|
| 47 |
### Key Constants
|
| 48 |
- DSM pixel size: 0.1 m/pixel (Google Solar API)
|
|
@@ -57,9 +59,12 @@ Address -> Geocode -> Google Solar API -> [RGB GeoTIFF, Building Mask, DSM]
|
|
| 57 |
- `siglip2-g` — Zero-shot text-image alignment (used for RADSeg)
|
| 58 |
|
| 59 |
### Zero-Shot Text Prompts
|
| 60 |
-
Roof classes: "flat roof plane", "pitched roof plane", "hip roof plane", "gable roof plane"
|
|
|
|
| 61 |
Non-roof: "sky", "ground", "tree", "wall", "shadow"
|
| 62 |
|
|
|
|
|
|
|
| 63 |
## Common Issues
|
| 64 |
|
| 65 |
### C-RADIOv4-H resolution error
|
|
@@ -81,5 +86,39 @@ If appearance-based splitting is too aggressive, the 15% threshold in
|
|
| 81 |
|
| 82 |
## Deployment
|
| 83 |
|
| 84 |
-
HuggingFace Space with Gradio SDK
|
| 85 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
| `geo_export.py` | Pixel-to-geo conversion, contour finding, Douglas-Peucker, GeoJSON |
|
| 44 |
| `pipeline.py` | PipelineResult dataclass, `run()` orchestration, overlay visualization |
|
| 45 |
| `app.py` | Gradio Blocks UI, `@spaces.GPU` decorator, input/output wiring |
|
| 46 |
+
| `export_text_embeddings.py` | One-off: precompute prompt embeddings (heavy, offline) |
|
| 47 |
+
| `verify_stripped_adaptor.py` | Assert the stripped load path is numerically identical |
|
| 48 |
|
| 49 |
### Key Constants
|
| 50 |
- DSM pixel size: 0.1 m/pixel (Google Solar API)
|
|
|
|
| 59 |
- `siglip2-g` — Zero-shot text-image alignment (used for RADSeg)
|
| 60 |
|
| 61 |
### Zero-Shot Text Prompts
|
| 62 |
+
Roof classes: "flat roof plane", "pitched roof plane", "hip roof plane", "gable roof plane",
|
| 63 |
+
"solar panel array on roof", "photovoltaic panels installed on roof"
|
| 64 |
Non-roof: "sky", "ground", "tree", "wall", "shadow"
|
| 65 |
|
| 66 |
+
These 11 strings are frozen into `text_embeddings.pt` — see Deployment.
|
| 67 |
+
|
| 68 |
## Common Issues
|
| 69 |
|
| 70 |
### C-RADIOv4-H resolution error
|
|
|
|
| 86 |
|
| 87 |
## Deployment
|
| 88 |
|
| 89 |
+
HuggingFace Space with Gradio SDK, on a dedicated GPU. `app.py` loads straight
|
| 90 |
+
to `cuda` at startup when it is available; the `@spaces.GPU` decorator is kept
|
| 91 |
+
so the app still runs on ZeroGPU, where `torch.cuda.is_available()` is False at
|
| 92 |
+
import time and the model loads to CPU instead.
|
| 93 |
+
|
| 94 |
+
### The SigLIP2 text tower is never loaded (memory)
|
| 95 |
+
|
| 96 |
+
RADIO's `SigLIP2Adaptor.__init__` calls
|
| 97 |
+
`AutoModel.from_pretrained("google/siglip2-giant-opt-patch16-384")` — pulling a
|
| 98 |
+
7.49 GB checkpoint and materializing the whole giant model — purely to keep
|
| 99 |
+
`model.text_model`, then `del`s the rest. On a 16 GB Space that download plus
|
| 100 |
+
its page-cache and mmap footprint is enough to exceed the container limit
|
| 101 |
+
before any of it can be freed. (Note the limit is a cgroup limit: page cache
|
| 102 |
+
from downloaded files counts against it, which is why the failure does not
|
| 103 |
+
reproduce from local RSS measurements.)
|
| 104 |
+
|
| 105 |
+
Our prompt set is fixed, so:
|
| 106 |
+
|
| 107 |
+
1. `export_text_embeddings.py` runs **once** on a machine with ~14 GB headroom,
|
| 108 |
+
encodes `ROOF_PROMPTS + NON_ROOF_PROMPTS`, and writes `text_embeddings.pt`
|
| 109 |
+
(11 x 1536 fp32, ~67 KB). Committed via LFS.
|
| 110 |
+
2. At startup `radio_backbone._strip_siglip2_text_adaptor()` pops the
|
| 111 |
+
`siglip2` factory out of RADIO's `adaptor_registry`. `create_adaptor` falls
|
| 112 |
+
back to `GenericAdaptor` for unregistered names, and `GenericAdaptor` builds
|
| 113 |
+
`head_mlp`/`feat_mlp` purely from the RADIO checkpoint's own state — so
|
| 114 |
+
`AutoModel.from_pretrained` never runs.
|
| 115 |
+
|
| 116 |
+
The vision path is **bit-identical**: same class, same weights, same code.
|
| 117 |
+
`verify_stripped_adaptor.py` asserts max abs diff == 0.0 against a reference
|
| 118 |
+
captured from the full path. Adaptor params drop 3044 MB -> 211 MB and boot
|
| 119 |
+
downloads drop 9.06 GB -> 1.57 GB.
|
| 120 |
+
|
| 121 |
+
**Changing `ROOF_PROMPTS` or `NON_ROOF_PROMPTS` requires re-running
|
| 122 |
+
`export_text_embeddings.py`.** `load_model()` refuses to start if the committed
|
| 123 |
+
prompts do not match the source, and `zero_shot_segment()` rejects custom
|
| 124 |
+
prompt overrides, since there is no text encoder to embed them with.
|
|
@@ -55,3 +55,7 @@ Address -> Geocode -> Google Solar API (RGB + Mask + DSM)
|
|
| 55 |
| `geo_export.py` | GeoJSON polygon extraction |
|
| 56 |
| `pipeline.py` | Orchestration |
|
| 57 |
| `app.py` | Gradio UI |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
| `geo_export.py` | GeoJSON polygon extraction |
|
| 56 |
| `pipeline.py` | Orchestration |
|
| 57 |
| `app.py` | Gradio UI |
|
| 58 |
+
|
| 59 |
+
The SigLIP2 text encoder is **not** loaded at runtime — the fixed prompt set's
|
| 60 |
+
embeddings are precomputed into `text_embeddings.pt` (~67 KB), which keeps the
|
| 61 |
+
Space inside its memory limit. See `AGENTS.md` for details.
|
|
@@ -0,0 +1,102 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Offline export of the SigLIP2 text embeddings for RoofSight's fixed prompts.
|
| 2 |
+
|
| 3 |
+
Run this ONCE on a machine with RAM headroom (~14 GB peak). It loads the full
|
| 4 |
+
`siglip2-g` adaptor — which instantiates the entire google/siglip2-giant model
|
| 5 |
+
just to keep its text tower — encodes the fixed prompt set, and writes the
|
| 6 |
+
resulting (num_prompts, D) tensor to disk.
|
| 7 |
+
|
| 8 |
+
The Space then loads with the `siglip2` adaptor factory removed from RADIO's
|
| 9 |
+
registry, so the text tower is never built there. See radio_backbone.load_model.
|
| 10 |
+
|
| 11 |
+
Also captures a reference feature map so the stripped runtime path can be
|
| 12 |
+
verified as numerically identical:
|
| 13 |
+
|
| 14 |
+
python export_text_embeddings.py # export + reference
|
| 15 |
+
python verify_stripped_adaptor.py # compare
|
| 16 |
+
"""
|
| 17 |
+
|
| 18 |
+
import os
|
| 19 |
+
import sys
|
| 20 |
+
|
| 21 |
+
import numpy as np
|
| 22 |
+
import torch
|
| 23 |
+
from PIL import Image
|
| 24 |
+
|
| 25 |
+
from radio_backbone import ROOF_PROMPTS, NON_ROOF_PROMPTS, PATCH_SIZE
|
| 26 |
+
|
| 27 |
+
OUT_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "text_embeddings.pt")
|
| 28 |
+
REF_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)), "_ref_features.pt")
|
| 29 |
+
REF_IMAGE = os.path.join(
|
| 30 |
+
os.path.dirname(os.path.abspath(__file__)),
|
| 31 |
+
"_run_output", "1310_susan_circle", "original.png",
|
| 32 |
+
)
|
| 33 |
+
|
| 34 |
+
# CPU + fp32 throughout: the reference must be deterministic so the parity
|
| 35 |
+
# check measures the adaptor swap, not autocast/GPU nondeterminism.
|
| 36 |
+
DEVICE = "cpu"
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def main():
|
| 40 |
+
all_labels = ROOF_PROMPTS + NON_ROOF_PROMPTS
|
| 41 |
+
print(f"Loading FULL c-radio_v4-h + siglip2-g adaptor on {DEVICE} (this is the heavy path)...")
|
| 42 |
+
|
| 43 |
+
model = torch.hub.load(
|
| 44 |
+
"NVlabs/RADIO",
|
| 45 |
+
"radio_model",
|
| 46 |
+
version="c-radio_v4-h",
|
| 47 |
+
adaptor_names=["siglip2-g"],
|
| 48 |
+
vitdet_window_size=8,
|
| 49 |
+
progress=True,
|
| 50 |
+
skip_validation=True,
|
| 51 |
+
trust_repo=True,
|
| 52 |
+
)
|
| 53 |
+
model.eval()
|
| 54 |
+
model.to(DEVICE)
|
| 55 |
+
|
| 56 |
+
adaptor = model.adaptors["siglip2-g"]
|
| 57 |
+
assert hasattr(adaptor, "encode_text"), (
|
| 58 |
+
"adaptor has no encode_text — the siglip2 factory was not used. "
|
| 59 |
+
"This script must run WITHOUT the registry patch."
|
| 60 |
+
)
|
| 61 |
+
|
| 62 |
+
print(f"Encoding {len(all_labels)} prompts...")
|
| 63 |
+
text_input = adaptor.tokenizer(all_labels)
|
| 64 |
+
text_input = {k: v.to(DEVICE) for k, v in text_input.items()}
|
| 65 |
+
with torch.no_grad():
|
| 66 |
+
embeddings = adaptor.encode_text(text_input, normalize=True).float().cpu().clone()
|
| 67 |
+
|
| 68 |
+
print(f" embeddings: {tuple(embeddings.shape)} dtype={embeddings.dtype}")
|
| 69 |
+
torch.save({"prompts": all_labels, "embeddings": embeddings}, OUT_PATH)
|
| 70 |
+
print(f" wrote {OUT_PATH} ({os.path.getsize(OUT_PATH) / 1024:.1f} KB)")
|
| 71 |
+
|
| 72 |
+
# --- Reference feature map, for the parity check ---
|
| 73 |
+
if not os.path.exists(REF_IMAGE):
|
| 74 |
+
print(f"\nNo reference image at {REF_IMAGE}; skipping reference capture.")
|
| 75 |
+
return
|
| 76 |
+
|
| 77 |
+
print(f"\nCapturing reference features from {REF_IMAGE}...")
|
| 78 |
+
image = np.array(Image.open(REF_IMAGE).convert("RGB"))
|
| 79 |
+
x = torch.from_numpy(image).permute(2, 0, 1).float().div_(255.0).unsqueeze(0)
|
| 80 |
+
snapped = model.get_nearest_supported_resolution(*x.shape[-2:])
|
| 81 |
+
if snapped != x.shape[-2:]:
|
| 82 |
+
x = torch.nn.functional.interpolate(x, snapped, mode="bilinear", align_corners=False)
|
| 83 |
+
|
| 84 |
+
with torch.no_grad():
|
| 85 |
+
out = model(x.to(DEVICE))
|
| 86 |
+
_, features = out["siglip2-g"]
|
| 87 |
+
|
| 88 |
+
torch.save(
|
| 89 |
+
{
|
| 90 |
+
"features": features.float().cpu().clone(),
|
| 91 |
+
"snapped": tuple(snapped),
|
| 92 |
+
"image_shape": image.shape,
|
| 93 |
+
},
|
| 94 |
+
REF_PATH,
|
| 95 |
+
)
|
| 96 |
+
print(f" features: {tuple(features.shape)}")
|
| 97 |
+
print(f" wrote {REF_PATH}")
|
| 98 |
+
print("\nDone. Now run: python verify_stripped_adaptor.py")
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
if __name__ == "__main__":
|
| 102 |
+
sys.exit(main())
|
|
@@ -3,12 +3,21 @@
|
|
| 3 |
Distills DINOv3-7B + SAM3 + SigLIP2 into a single 631M-param encoder.
|
| 4 |
Uses the SigLIP2 adaptor head for zero-shot text-prompted roof segmentation.
|
| 5 |
|
| 6 |
-
Memory
|
| 7 |
-
|
| 8 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
"""
|
| 10 |
|
| 11 |
-
import
|
|
|
|
| 12 |
|
| 13 |
import numpy as np
|
| 14 |
import torch
|
|
@@ -22,6 +31,9 @@ PATCH_SIZE = 16
|
|
| 22 |
# Solar-array prompts count as "roof" — the pixels underneath still belong
|
| 23 |
# to a roof plane; without these prompts, RADIO pushes solar-covered
|
| 24 |
# pixels toward "wall"/"shadow" and fusion carves holes in the roof mask.
|
|
|
|
|
|
|
|
|
|
| 25 |
ROOF_PROMPTS = [
|
| 26 |
"flat roof plane",
|
| 27 |
"pitched roof plane",
|
|
@@ -38,17 +50,88 @@ NON_ROOF_PROMPTS = [
|
|
| 38 |
"shadow",
|
| 39 |
]
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
# Module-level caches
|
| 42 |
_model = None
|
| 43 |
_device = None
|
| 44 |
-
_cached_text_embeddings = None #
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
|
| 47 |
def load_model(device: str = "cuda", vitdet_window_size: int = 8):
|
| 48 |
-
"""Load C-RADIOv4-H with siglip2-g
|
| 49 |
|
| 50 |
-
|
| 51 |
-
|
| 52 |
|
| 53 |
Args:
|
| 54 |
device: 'cuda' or 'cpu'. For ZeroGPU, load to 'cpu' at startup
|
|
@@ -66,6 +149,13 @@ def load_model(device: str = "cuda", vitdet_window_size: int = 8):
|
|
| 66 |
|
| 67 |
print("Loading C-RADIOv4-H (631M params)...")
|
| 68 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
kwargs = {
|
| 70 |
"version": "c-radio_v4-h",
|
| 71 |
"adaptor_names": ["siglip2-g"],
|
|
@@ -80,66 +170,12 @@ def load_model(device: str = "cuda", vitdet_window_size: int = 8):
|
|
| 80 |
_model.to(device)
|
| 81 |
_device = device
|
| 82 |
|
| 83 |
-
|
| 84 |
-
all_labels = ROOF_PROMPTS + NON_ROOF_PROMPTS
|
| 85 |
-
print(f"Caching text embeddings for {len(all_labels)} prompts...")
|
| 86 |
-
|
| 87 |
-
sig2_adaptor = _model.adaptors["siglip2-g"]
|
| 88 |
-
text_input = sig2_adaptor.tokenizer(all_labels).to(device)
|
| 89 |
-
with torch.no_grad():
|
| 90 |
-
_cached_text_embeddings = sig2_adaptor.encode_text(
|
| 91 |
-
text_input, normalize=True
|
| 92 |
-
).cpu().clone()
|
| 93 |
|
| 94 |
-
|
| 95 |
-
_free_text_encoder(sig2_adaptor)
|
| 96 |
-
|
| 97 |
-
print(f"C-RADIOv4-H loaded on {device} (text encoder freed, embeddings cached)")
|
| 98 |
return _model
|
| 99 |
|
| 100 |
|
| 101 |
-
def _free_text_encoder(adaptor):
|
| 102 |
-
"""Delete the SigLIP2 text encoder from the adaptor to free ~7.5GB RAM.
|
| 103 |
-
|
| 104 |
-
Only targets modules > 1GB — the text encoder is ~7.5GB while vision
|
| 105 |
-
projection heads (feat_mlp, summary_mlp, etc.) are < 1GB.
|
| 106 |
-
"""
|
| 107 |
-
freed = 0
|
| 108 |
-
|
| 109 |
-
# Log all modules so we can see what's there
|
| 110 |
-
print(" Adaptor modules:")
|
| 111 |
-
for name, module in adaptor.named_children():
|
| 112 |
-
param_bytes = sum(
|
| 113 |
-
p.numel() * p.element_size() for p in module.parameters()
|
| 114 |
-
)
|
| 115 |
-
print(f" {name}: {param_bytes / 1e6:.0f} MB")
|
| 116 |
-
|
| 117 |
-
# Only delete modules > 1GB (the text encoder is ~7.5GB,
|
| 118 |
-
# vision projection heads like feat_mlp are < 1GB)
|
| 119 |
-
for name in list(dict(adaptor.named_children()).keys()):
|
| 120 |
-
module = getattr(adaptor, name)
|
| 121 |
-
param_bytes = sum(
|
| 122 |
-
p.numel() * p.element_size() for p in module.parameters()
|
| 123 |
-
)
|
| 124 |
-
if param_bytes > 1_000_000_000: # > 1GB
|
| 125 |
-
size_gb = param_bytes / 1e9
|
| 126 |
-
print(f" Freeing adaptor.{name} ({size_gb:.1f} GB)")
|
| 127 |
-
try:
|
| 128 |
-
delattr(adaptor, name)
|
| 129 |
-
freed += param_bytes
|
| 130 |
-
except Exception:
|
| 131 |
-
pass
|
| 132 |
-
|
| 133 |
-
gc.collect()
|
| 134 |
-
if torch.cuda.is_available():
|
| 135 |
-
torch.cuda.empty_cache()
|
| 136 |
-
|
| 137 |
-
if freed > 0:
|
| 138 |
-
print(f" Total freed: {freed / 1e9:.1f} GB")
|
| 139 |
-
else:
|
| 140 |
-
print(" Warning: no module > 1GB found — text encoder may still be in RAM")
|
| 141 |
-
|
| 142 |
-
|
| 143 |
def get_model():
|
| 144 |
"""Get the cached model, loading if necessary."""
|
| 145 |
global _model
|
|
@@ -226,6 +262,15 @@ def zero_shot_segment(
|
|
| 226 |
|
| 227 |
all_labels = roof_prompts + non_roof_prompts
|
| 228 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 229 |
# T4/older GPUs have no native bfloat16 — pick fp16 unless the GPU is Ampere+
|
| 230 |
if device == "cuda" and torch.cuda.is_available():
|
| 231 |
major, _ = torch.cuda.get_device_capability()
|
|
|
|
| 3 |
Distills DINOv3-7B + SAM3 + SigLIP2 into a single 631M-param encoder.
|
| 4 |
Uses the SigLIP2 adaptor head for zero-shot text-prompted roof segmentation.
|
| 5 |
|
| 6 |
+
Memory: RADIO's `SigLIP2Adaptor` builds the entire google/siglip2-giant model
|
| 7 |
+
(~7.5 GB, vision tower included) purely to keep its text tower, then discards
|
| 8 |
+
the rest — a peak large enough to OOM a 16 GB Space before anything can be
|
| 9 |
+
freed. Our prompt set is fixed, so its embeddings are precomputed offline
|
| 10 |
+
(see export_text_embeddings.py) and committed as text_embeddings.pt (~67 KB).
|
| 11 |
+
|
| 12 |
+
At runtime we remove the `siglip2` factory from RADIO's adaptor registry.
|
| 13 |
+
AdaptorRegistry.create_adaptor falls back to GenericAdaptor for any
|
| 14 |
+
unregistered name, and GenericAdaptor builds head_mlp/feat_mlp purely from the
|
| 15 |
+
RADIO checkpoint's own state — so the vision path is bit-identical and the text
|
| 16 |
+
tower is never constructed. Verified by verify_stripped_adaptor.py.
|
| 17 |
"""
|
| 18 |
|
| 19 |
+
import os
|
| 20 |
+
import sys
|
| 21 |
|
| 22 |
import numpy as np
|
| 23 |
import torch
|
|
|
|
| 31 |
# Solar-array prompts count as "roof" — the pixels underneath still belong
|
| 32 |
# to a roof plane; without these prompts, RADIO pushes solar-covered
|
| 33 |
# pixels toward "wall"/"shadow" and fusion carves holes in the roof mask.
|
| 34 |
+
#
|
| 35 |
+
# NOTE: changing this list invalidates text_embeddings.pt. Re-run
|
| 36 |
+
# export_text_embeddings.py; load_model() refuses to start on a mismatch.
|
| 37 |
ROOF_PROMPTS = [
|
| 38 |
"flat roof plane",
|
| 39 |
"pitched roof plane",
|
|
|
|
| 50 |
"shadow",
|
| 51 |
]
|
| 52 |
|
| 53 |
+
TEXT_EMBEDDINGS_PATH = os.path.join(
|
| 54 |
+
os.path.dirname(os.path.abspath(__file__)), "text_embeddings.pt"
|
| 55 |
+
)
|
| 56 |
+
|
| 57 |
# Module-level caches
|
| 58 |
_model = None
|
| 59 |
_device = None
|
| 60 |
+
_cached_text_embeddings = None # Loaded from TEXT_EMBEDDINGS_PATH at startup
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def _ensure_radio_repo() -> str:
|
| 64 |
+
"""Return the cached NVlabs/RADIO repo dir, fetching it if absent.
|
| 65 |
+
|
| 66 |
+
torch.hub.list() downloads and caches the repo without building a model,
|
| 67 |
+
which is what we need: the registry must be patched before radio_model()
|
| 68 |
+
runs, and the `radio` package only becomes importable once it is on disk.
|
| 69 |
+
"""
|
| 70 |
+
repo_dir = os.path.join(torch.hub.get_dir(), "NVlabs_RADIO_main")
|
| 71 |
+
if not os.path.isdir(repo_dir):
|
| 72 |
+
torch.hub.list("NVlabs/RADIO", trust_repo=True, skip_validation=True)
|
| 73 |
+
return repo_dir
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def _strip_siglip2_text_adaptor():
|
| 77 |
+
"""Remove the `siglip2` adaptor factory so GenericAdaptor is used instead.
|
| 78 |
+
|
| 79 |
+
This is what keeps the Space under its memory limit: SigLIP2Adaptor's
|
| 80 |
+
__init__ calls AutoModel.from_pretrained on the giant checkpoint. Dropping
|
| 81 |
+
the factory means that line never runs, so the 7.49 GB download and its
|
| 82 |
+
CPU-RAM spike never happen.
|
| 83 |
+
"""
|
| 84 |
+
repo_dir = _ensure_radio_repo()
|
| 85 |
+
if repo_dir not in sys.path:
|
| 86 |
+
sys.path.insert(0, repo_dir)
|
| 87 |
+
|
| 88 |
+
from radio.adaptor_registry import adaptor_registry
|
| 89 |
+
|
| 90 |
+
removed = adaptor_registry._registry.pop("siglip2", None)
|
| 91 |
+
if removed is not None:
|
| 92 |
+
print(" siglip2 text adaptor stripped (text tower will not be built)")
|
| 93 |
+
else:
|
| 94 |
+
print(" warning: no 'siglip2' factory in registry — RADIO's API may have changed")
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def _load_text_embeddings() -> torch.Tensor:
|
| 98 |
+
"""Load the precomputed prompt embeddings, refusing to run if they drifted."""
|
| 99 |
+
if not os.path.exists(TEXT_EMBEDDINGS_PATH):
|
| 100 |
+
raise FileNotFoundError(
|
| 101 |
+
f"{TEXT_EMBEDDINGS_PATH} is missing. Run export_text_embeddings.py "
|
| 102 |
+
"on a machine with ~14 GB of RAM headroom and commit the result."
|
| 103 |
+
)
|
| 104 |
+
|
| 105 |
+
blob = torch.load(TEXT_EMBEDDINGS_PATH, map_location="cpu")
|
| 106 |
+
expected = ROOF_PROMPTS + NON_ROOF_PROMPTS
|
| 107 |
+
if blob["prompts"] != expected:
|
| 108 |
+
raise ValueError(
|
| 109 |
+
"text_embeddings.pt does not match the current prompt list.\n"
|
| 110 |
+
f" artifact: {blob['prompts']}\n"
|
| 111 |
+
f" expected: {expected}\n"
|
| 112 |
+
"Re-run export_text_embeddings.py."
|
| 113 |
+
)
|
| 114 |
+
return blob["embeddings"]
|
| 115 |
+
|
| 116 |
+
|
| 117 |
+
def _assert_no_text_tower(adaptor):
|
| 118 |
+
"""Fail loudly if a text tower leaked into the adaptor despite the strip."""
|
| 119 |
+
print(" Adaptor modules:")
|
| 120 |
+
for name, module in adaptor.named_children():
|
| 121 |
+
mb = sum(p.numel() * p.element_size() for p in module.parameters()) / 1e6
|
| 122 |
+
print(f" {name}: {mb:.0f} MB")
|
| 123 |
+
if mb > 1000:
|
| 124 |
+
raise RuntimeError(
|
| 125 |
+
f"adaptor.{name} is {mb:.0f} MB — the text tower was built anyway. "
|
| 126 |
+
"Startup would exceed the Space memory limit; aborting."
|
| 127 |
+
)
|
| 128 |
|
| 129 |
|
| 130 |
def load_model(device: str = "cuda", vitdet_window_size: int = 8):
|
| 131 |
+
"""Load C-RADIOv4-H with the siglip2-g vision projection only.
|
| 132 |
|
| 133 |
+
The SigLIP2 text tower is never constructed; prompt embeddings come from
|
| 134 |
+
the committed text_embeddings.pt instead.
|
| 135 |
|
| 136 |
Args:
|
| 137 |
device: 'cuda' or 'cpu'. For ZeroGPU, load to 'cpu' at startup
|
|
|
|
| 149 |
|
| 150 |
print("Loading C-RADIOv4-H (631M params)...")
|
| 151 |
|
| 152 |
+
# Load the embeddings first: cheap, and it fails fast on prompt drift
|
| 153 |
+
# before we spend a minute pulling 1.57 GB of weights.
|
| 154 |
+
_cached_text_embeddings = _load_text_embeddings()
|
| 155 |
+
print(f" text embeddings: {tuple(_cached_text_embeddings.shape)} (precomputed)")
|
| 156 |
+
|
| 157 |
+
_strip_siglip2_text_adaptor()
|
| 158 |
+
|
| 159 |
kwargs = {
|
| 160 |
"version": "c-radio_v4-h",
|
| 161 |
"adaptor_names": ["siglip2-g"],
|
|
|
|
| 170 |
_model.to(device)
|
| 171 |
_device = device
|
| 172 |
|
| 173 |
+
_assert_no_text_tower(_model.adaptors["siglip2-g"])
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 174 |
|
| 175 |
+
print(f"C-RADIOv4-H loaded on {device} (text tower never built)")
|
|
|
|
|
|
|
|
|
|
| 176 |
return _model
|
| 177 |
|
| 178 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
def get_model():
|
| 180 |
"""Get the cached model, loading if necessary."""
|
| 181 |
global _model
|
|
|
|
| 262 |
|
| 263 |
all_labels = roof_prompts + non_roof_prompts
|
| 264 |
|
| 265 |
+
# The text encoder is not loaded, so embeddings cannot be computed on the
|
| 266 |
+
# fly — custom prompts would silently score against the wrong vectors.
|
| 267 |
+
if all_labels != ROOF_PROMPTS + NON_ROOF_PROMPTS:
|
| 268 |
+
raise ValueError(
|
| 269 |
+
"zero_shot_segment received prompts that differ from the ones "
|
| 270 |
+
"baked into text_embeddings.pt. Edit ROOF_PROMPTS/NON_ROOF_PROMPTS "
|
| 271 |
+
"and re-run export_text_embeddings.py to change the prompt set."
|
| 272 |
+
)
|
| 273 |
+
|
| 274 |
# T4/older GPUs have no native bfloat16 — pick fp16 unless the GPU is Ampere+
|
| 275 |
if device == "cuda" and torch.cuda.is_available():
|
| 276 |
major, _ = torch.cuda.get_device_capability()
|
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:42518c9307817c52b0b851550a160953edfc56148bb6107b02799b65e1c1db1a
|
| 3 |
+
size 69060
|
|
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Verify the stripped-adaptor load path is numerically identical to the full one.
|
| 2 |
+
|
| 3 |
+
Loads c-radio_v4-h with the `siglip2` adaptor factory removed from RADIO's
|
| 4 |
+
registry — so GenericAdaptor is built from the checkpoint state instead of
|
| 5 |
+
SigLIP2Adaptor — then compares the resulting feature map against the reference
|
| 6 |
+
captured by export_text_embeddings.py.
|
| 7 |
+
|
| 8 |
+
Expected result: max abs diff == 0.0. Same class, same weights, same code path;
|
| 9 |
+
the only thing removed is the text tower, which never touches the vision path.
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
import os
|
| 13 |
+
import sys
|
| 14 |
+
|
| 15 |
+
import numpy as np
|
| 16 |
+
import torch
|
| 17 |
+
from PIL import Image
|
| 18 |
+
|
| 19 |
+
import radio_backbone
|
| 20 |
+
from radio_backbone import ROOF_PROMPTS, NON_ROOF_PROMPTS
|
| 21 |
+
|
| 22 |
+
HERE = os.path.dirname(os.path.abspath(__file__))
|
| 23 |
+
REF_PATH = os.path.join(HERE, "_ref_features.pt")
|
| 24 |
+
EMB_PATH = os.path.join(HERE, "text_embeddings.pt")
|
| 25 |
+
REF_IMAGE = os.path.join(HERE, "_run_output", "1310_susan_circle", "original.png")
|
| 26 |
+
|
| 27 |
+
DEVICE = "cpu"
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def main():
|
| 31 |
+
if not os.path.exists(REF_PATH):
|
| 32 |
+
print(f"Missing {REF_PATH}. Run export_text_embeddings.py first.")
|
| 33 |
+
return 1
|
| 34 |
+
|
| 35 |
+
# --- 1. Embeddings artifact sanity ---
|
| 36 |
+
blob = torch.load(EMB_PATH, map_location="cpu")
|
| 37 |
+
expected = ROOF_PROMPTS + NON_ROOF_PROMPTS
|
| 38 |
+
assert blob["prompts"] == expected, "prompt drift between artifact and source"
|
| 39 |
+
emb = blob["embeddings"]
|
| 40 |
+
norms = emb.norm(dim=-1)
|
| 41 |
+
print(f"text_embeddings.pt: {tuple(emb.shape)}, dtype={emb.dtype}")
|
| 42 |
+
print(f" L2 norms: min={norms.min():.6f} max={norms.max():.6f} (expect ~1.0, normalize=True)")
|
| 43 |
+
assert torch.allclose(norms, torch.ones_like(norms), atol=1e-4), "embeddings are not unit-norm"
|
| 44 |
+
|
| 45 |
+
# --- 2. Load with the text adaptor stripped ---
|
| 46 |
+
print("\nLoading with `siglip2` factory removed from the registry...")
|
| 47 |
+
radio_backbone._strip_siglip2_text_adaptor()
|
| 48 |
+
|
| 49 |
+
model = torch.hub.load(
|
| 50 |
+
"NVlabs/RADIO", "radio_model",
|
| 51 |
+
version="c-radio_v4-h", adaptor_names=["siglip2-g"],
|
| 52 |
+
vitdet_window_size=8, progress=True, skip_validation=True, trust_repo=True,
|
| 53 |
+
)
|
| 54 |
+
model.eval()
|
| 55 |
+
model.to(DEVICE)
|
| 56 |
+
|
| 57 |
+
adaptor = model.adaptors["siglip2-g"]
|
| 58 |
+
print(f" adaptor class: {type(adaptor).__name__}")
|
| 59 |
+
assert type(adaptor).__name__ == "GenericAdaptor", "expected GenericAdaptor fallback"
|
| 60 |
+
assert not hasattr(adaptor, "text_model"), "text_model still present!"
|
| 61 |
+
assert not hasattr(adaptor, "encode_text"), "encode_text still present!"
|
| 62 |
+
|
| 63 |
+
for name, module in adaptor.named_children():
|
| 64 |
+
mb = sum(p.numel() * p.element_size() for p in module.parameters()) / 1e6
|
| 65 |
+
print(f" {name}: {mb:.0f} MB")
|
| 66 |
+
assert mb < 1000, f"{name} is {mb:.0f} MB — a text tower may have leaked in"
|
| 67 |
+
|
| 68 |
+
# --- 3. Feature parity ---
|
| 69 |
+
ref = torch.load(REF_PATH, map_location="cpu")
|
| 70 |
+
image = np.array(Image.open(REF_IMAGE).convert("RGB"))
|
| 71 |
+
x = torch.from_numpy(image).permute(2, 0, 1).float().div_(255.0).unsqueeze(0)
|
| 72 |
+
snapped = model.get_nearest_supported_resolution(*x.shape[-2:])
|
| 73 |
+
if snapped != x.shape[-2:]:
|
| 74 |
+
x = torch.nn.functional.interpolate(x, snapped, mode="bilinear", align_corners=False)
|
| 75 |
+
assert tuple(snapped) == ref["snapped"], f"resolution drift: {snapped} vs {ref['snapped']}"
|
| 76 |
+
|
| 77 |
+
print("\nRunning forward pass...")
|
| 78 |
+
with torch.no_grad():
|
| 79 |
+
out = model(x.to(DEVICE))
|
| 80 |
+
_, features = out["siglip2-g"]
|
| 81 |
+
features = features.float().cpu()
|
| 82 |
+
|
| 83 |
+
ref_feat = ref["features"]
|
| 84 |
+
assert features.shape == ref_feat.shape, f"shape drift: {features.shape} vs {ref_feat.shape}"
|
| 85 |
+
diff = (features - ref_feat).abs()
|
| 86 |
+
print(f" shape: {tuple(features.shape)}")
|
| 87 |
+
print(f" max abs diff: {diff.max().item():.3e}")
|
| 88 |
+
print(f" mean abs diff: {diff.mean().item():.3e}")
|
| 89 |
+
|
| 90 |
+
if diff.max().item() == 0.0:
|
| 91 |
+
print("\nPASS — bit-identical to the full adaptor path.")
|
| 92 |
+
return 0
|
| 93 |
+
if diff.max().item() < 1e-5:
|
| 94 |
+
print("\nPASS — identical within float tolerance.")
|
| 95 |
+
return 0
|
| 96 |
+
print("\nFAIL — features diverged. Do not deploy.")
|
| 97 |
+
return 1
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
if __name__ == "__main__":
|
| 101 |
+
sys.exit(main())
|