BiliSakura/SkySensePlusPlus-transformers

SkySense++ Transformers

HuggingFace-compatible checkpoints for SkySense++ zero-shot MSL backbones, converted from the official release weights.

Checkpoints

Directory	Modality	Architecture	Source
skysensepp-swinv2-msl-hr	High-res optical	SwinV2 Huge + MSL	skysensepp_release_hr.pth
skysensepp-vit-msl-s2	Sentinel-2	ViT-Large + MSL	skysensepp_release_s2.pth
skysensepp-vit-msl-s1	Sentinel-1	ViT-Large + MSL	skysensepp_release_s1.pth
skysensepp-fusion-neck	Multi-modal fusion (optional)	TransformerEncoder	fusion.* from skysensepp_release.ckpt
skysensepp-fewshot-release	Full 1-shot segmentation	HR + S2 + S1 + fusion + VAE + UPerHead	skysensepp_release.ckpt

Each subdirectory is a self-contained HuggingFace model repo with remote code (trust_remote_code=True).

The fusion neck is an optional component — backbone checkpoints do not include or require it by default.

The few-shot release bundles all submodules into one end-to-end model (~6.8 GB).

Usage

from transformers import pipeline
import torch

MODEL = "/path/to/SkySensePlusPlus-transformers/skysensepp-swinv2-msl-hr"

pipe = pipeline(
    task="image-feature-extraction",
    model=MODEL,
    trust_remote_code=True,
    device="cpu",
)

hr_img = torch.randn(1, 3, 512, 512)
annotation = torch.zeros(1, 512, 512, dtype=torch.long)  # semantic class indices

features = pipe(hr_img, annotation=annotation)
print(features["last_hidden_state"].shape)  # (1, 2816, 16, 16)

Sentinel-2 / Sentinel-1 backbones use the same pipeline pattern:

s2_pipe = pipeline(
    task="image-feature-extraction",
    model="/path/to/skysensepp-vit-msl-s2",
    trust_remote_code=True,
    device="cpu",
)

s2_img = torch.randn(1, 10, 16, 16)
s2_anno = torch.zeros(1, 16, 16, dtype=torch.long)
features = s2_pipe(s2_img, annotation=s2_anno)
print(features["last_hidden_state"].shape)

SkySense++ MSL models require both imagery and a semantic annotation map. Use class index 0 for background/unlabeled regions during zero-shot feature extraction.

Optional fusion neck

fusion_pipe = pipeline(
    task="skysensepp-fusion",
    model="/path/to/skysensepp-fusion-neck",
    trust_remote_code=True,
    device="cpu",
)

# Concatenated HR + S2 + S1 stage-3 tokens per spatial location
hidden_states = torch.randn(256, 3, 2816)
fused = fusion_pipe(hidden_states)

print(fused["pooler_output"].shape)  # (256, 1024)

Few-shot / 1-shot segmentation

The full release model expects vertically stacked prompt+query inputs (prompt on top, query on bottom):

from transformers import pipeline
import torch

MODEL = "/path/to/SkySensePlusPlus-transformers/skysensepp-fewshot-release"

pipe = pipeline(
    task="skysensepp-fewshot",
    model=MODEL,
    trust_remote_code=True,
    device=0,  # GPU recommended (~24 GB); CPU OOMs at 1024×512 HR
)

# Stacked HR (3, 1024, 512), S2/S1 with seq=2, RGB targets (ImageNet-normalized)
hr = torch.randn(1, 3, 1024, 512)
s2 = torch.randn(1, 10, 2, 32, 32)
s1 = torch.randn(1, 2, 2, 32, 32)
targets = torch.randn(1, 3, 1024, 512)  # use real RGB annotation maps in practice
anno_mask = torch.zeros(1, 8, 4, dtype=torch.long)
anno_mask[:, 4:, :] = 1  # mask query (bottom) half

result = pipe(hr, s2_img=s2, s1_img=s1, targets=targets, anno_mask=anno_mask)
print(result["logits"].shape)  # (1, 65, 512, 512) — query region only

Conversion

Source project: /home/czy/local/projects/SkySensePlusPlus-transformers

conda activate rsgen
python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysensepp_release_hr.pth \
  --modality hr \
  --output-dir /path/to/skysensepp-swinv2-msl-hr \
  --clean-output

# Full few-shot release (~6.8 GB)
python scripts/convert_checkpoint_to_hf.py \
  --input-path /path/to/skysensepp_release.ckpt \
  --modality fewshot \
  --output-dir /path/to/skysensepp-fewshot-release \
  --clean-output

Notes

• HR conversion skips Swin relative-position buffers (relative_position_index, relative_coords_table). These are deterministically recomputed at init from window geometry — not randomly initialized. Learned CPB weights (cpb_mlp, logit_scale) are loaded.
• The few-shot model uses the same 62 skipped HR buffers; all 1522 learned tensors load with 0 unexpected keys.