File size: 6,780 Bytes
429ec1d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | ---
library_name: "fastforks"
pipeline_tag: "mask-generation"
license: "apache-2.0"
tags:
- "promptable_segmentation"
- "segment-anything"
- "promptable-segmentation"
- "cuda"
- "pytorch"
- "fastforks"
- "fastforks-card"
base_model:
- "CIDAS/clipseg-rd64-refined"
---
# CLIPSeg (text-prompted segmentation, CLIP B/16) (fastforks)
CLIPSeg (Lüddecke and Ecker 2022): a frozen CLIP ViT-B/16 with a small transformer decoder that segments whatever a free-text prompt names; the text embedding conditions the decoder by FiLM. Semantic: one mask per prompt (all pixels of the concept), at 352 x 352, resized to the image. Pass `text=` at call time. The portable `module()` is the network on the normalised 352 x 352 image plus a projected text embedding.
This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/huggingface/transformers> at `c4c4605f9618`.
Author weight repos: <https://huggingface.co/CIDAS/clipseg-rd64-refined>.
Task: promptable_segmentation. Tier: experimental. Outputs: `boxes`, `scores`, `masks`. Backends: `portable`, `kernels`.
Input: one image plus a point, box or text prompt. Output: masks with quality scores.
## Variants
Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/promptable_segmentation/models/clipseg/card.toml).
| variant | weights (author link) | SHA-256 | weights licence | commercial use |
|---|---|---|---|---|
| `clipseg@1` | [clipseg-rd64-refined.config.json](https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/999e0328d9e10b484360c477313983f9afdd7050/config.json)<br>[clipseg-rd64-refined.safetensors](https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/999e0328d9e10b484360c477313983f9afdd7050/model.safetensors)<br>[clip-bpe.vocab.json](https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/999e0328d9e10b484360c477313983f9afdd7050/vocab.json)<br>[clip-bpe.merges.txt](https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/999e0328d9e10b484360c477313983f9afdd7050/merges.txt) | `c023375966d3`<br>`d00ca85d6b85`<br>`e089ad92ba36`<br>`9fd691f7c803` | Apache-2.0 | yes |
## Licence
Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.
Training data: PhraseCut (referring-phrase masks on Visual Genome images) (The decoder is trained on PhraseCut (extended with visual prompts in the paper; the refined rd64 checkpoint, reduce_dim 64); the CLIP towers are OpenAI's frozen ViT-B/16, trained on web image-text pairs that were not released (CLIPSeg paper, arXiv:2112.10003).).
Licence evidence: <https://huggingface.co/CIDAS/clipseg-rd64-refined/blob/999e0328d9e10b484360c477313983f9afdd7050/README.md>
## Input size
any H x W >= 32, padded to a multiple of 32
Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `clipseg` 3000x4000 (0.7 GiB peak allocated, 0.9 reserved)
## Speed and memory
### Promptable segmentation, synthetic, 1024x1024
| model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `clipseg` | – | 15.3 | 4.45 | 2.75 | 5.6x | 1.6x | 346 / 404 | 328 / 356 | 335 / 362 |
### Startup and first call
| model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
| --- | --- | --- | --- | --- | --- | --- |
| `clipseg` | 3.05 | 3.90 | 3.64 | 0.02 | 0.26 | 0.26 |
### torch.compile and TensorRT
| model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| `clipseg` | 1024x1024 | 4.45 ms | 3.19 ms | planned for 0.2.1 | 2.75 ms | 9.92 s | – | 328 / 356 | 348 / 374 | 335 / 362 | – |
Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
## Accuracy
### Parity with the original code
Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0005 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.1 and p99 <= 0.5 (absolute error of the output `boxes`).
## Limitations
- Requires the optional transformers library (the network is its `CLIPSegForImageSegmentation`).
- The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (CIDAS/clipseg-rd64-refined); the authors' timojl/clipseg code was not run.
- Text prompts only (no box exemplars or visual prompts); one instance per prompt: its mask is every pixel above probability 0.5, its score the highest pixel probability, its box the mask's bounding box.
- The image is resized to 352 x 352 without keeping the aspect ratio, as the original processor; masks are coarse (22 x 22 patch tokens).
## Usage
```python
from fastforks import promptable_segmentation
model = promptable_segmentation.load("clipseg") # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg", input_points=[[x, y]], input_labels=[1])
result.masks # [M, H, W] bool
# SAM 3 (text prompts): model("photo.jpg", text="coin") -> boxes, scores, masks
result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
```
Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `clipseg`.
## Kernels backend
`promptable_segmentation.load("clipseg", mode="kernels")` runs our CUDA kernels (`fvm_conv`). It needs a prebuilt kernel for your GPU architecture (downloaded from <https://huggingface.co/fastforks/kernels> on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.
<!-- staged from fastforks 1cbf1671 (2026-11-15) -->
|