Flor commited on
Commit
429ec1d
·
verified ·
1 Parent(s): 4009d09

card: update from fastforks 1cbf1671

Browse files
Files changed (1) hide show
  1. README.md +104 -0
README.md ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: "fastforks"
3
+ pipeline_tag: "mask-generation"
4
+ license: "apache-2.0"
5
+ tags:
6
+ - "promptable_segmentation"
7
+ - "segment-anything"
8
+ - "promptable-segmentation"
9
+ - "cuda"
10
+ - "pytorch"
11
+ - "fastforks"
12
+ - "fastforks-card"
13
+ base_model:
14
+ - "CIDAS/clipseg-rd64-refined"
15
+ ---
16
+ # CLIPSeg (text-prompted segmentation, CLIP B/16) (fastforks)
17
+
18
+ CLIPSeg (Lüddecke and Ecker 2022): a frozen CLIP ViT-B/16 with a small transformer decoder that segments whatever a free-text prompt names; the text embedding conditions the decoder by FiLM. Semantic: one mask per prompt (all pixels of the concept), at 352 x 352, resized to the image. Pass `text=` at call time. The portable `module()` is the network on the normalised 352 x 352 image plus a projected text embedding.
19
+
20
+ This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: <https://github.com/kleinicke/fastforks>. Kernels for the `kernels` backend: <https://huggingface.co/fastforks/kernels>. Original code: <https://github.com/huggingface/transformers> at `c4c4605f9618`.
21
+
22
+ Author weight repos: <https://huggingface.co/CIDAS/clipseg-rd64-refined>.
23
+
24
+ Task: promptable_segmentation. Tier: experimental. Outputs: `boxes`, `scores`, `masks`. Backends: `portable`, `kernels`.
25
+
26
+ Input: one image plus a point, box or text prompt. Output: masks with quality scores.
27
+
28
+ ## Variants
29
+
30
+ Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/promptable_segmentation/models/clipseg/card.toml).
31
+
32
+ | variant | weights (author link) | SHA-256 | weights licence | commercial use |
33
+ |---|---|---|---|---|
34
+ | `clipseg@1` | [clipseg-rd64-refined.config.json](https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/999e0328d9e10b484360c477313983f9afdd7050/config.json)<br>[clipseg-rd64-refined.safetensors](https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/999e0328d9e10b484360c477313983f9afdd7050/model.safetensors)<br>[clip-bpe.vocab.json](https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/999e0328d9e10b484360c477313983f9afdd7050/vocab.json)<br>[clip-bpe.merges.txt](https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/999e0328d9e10b484360c477313983f9afdd7050/merges.txt) | `c023375966d3`<br>`d00ca85d6b85`<br>`e089ad92ba36`<br>`9fd691f7c803` | Apache-2.0 | yes |
35
+
36
+ ## Licence
37
+
38
+ Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.
39
+
40
+ Training data: PhraseCut (referring-phrase masks on Visual Genome images) (The decoder is trained on PhraseCut (extended with visual prompts in the paper; the refined rd64 checkpoint, reduce_dim 64); the CLIP towers are OpenAI's frozen ViT-B/16, trained on web image-text pairs that were not released (CLIPSeg paper, arXiv:2112.10003).).
41
+
42
+ Licence evidence: <https://huggingface.co/CIDAS/clipseg-rd64-refined/blob/999e0328d9e10b484360c477313983f9afdd7050/README.md>
43
+
44
+ ## Input size
45
+
46
+ any H x W >= 32, padded to a multiple of 32
47
+ Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `clipseg` 3000x4000 (0.7 GiB peak allocated, 0.9 reserved)
48
+
49
+ ## Speed and memory
50
+
51
+ ### Promptable segmentation, synthetic, 1024x1024
52
+
53
+ | model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
54
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
55
+ | `clipseg` | – | 15.3 | 4.45 | 2.75 | 5.6x | 1.6x | 346 / 404 | 328 / 356 | 335 / 362 |
56
+
57
+ ### Startup and first call
58
+
59
+ | model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
60
+ | --- | --- | --- | --- | --- | --- | --- |
61
+ | `clipseg` | 3.05 | 3.90 | 3.64 | 0.02 | 0.26 | 0.26 |
62
+
63
+ ### torch.compile and TensorRT
64
+
65
+ | model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
66
+ | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
67
+ | `clipseg` | 1024x1024 | 4.45 ms | 3.19 ms | planned for 0.2.1 | 2.75 ms | 9.92 s | – | 328 / 356 | 348 / 374 | 335 / 362 | – |
68
+
69
+ Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
70
+
71
+ Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
72
+
73
+ ## Accuracy
74
+
75
+ ### Parity with the original code
76
+
77
+ Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.0005 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.1 and p99 <= 0.5 (absolute error of the output `boxes`).
78
+
79
+ ## Limitations
80
+
81
+ - Requires the optional transformers library (the network is its `CLIPSegForImageSegmentation`).
82
+ - The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (CIDAS/clipseg-rd64-refined); the authors' timojl/clipseg code was not run.
83
+ - Text prompts only (no box exemplars or visual prompts); one instance per prompt: its mask is every pixel above probability 0.5, its score the highest pixel probability, its box the mask's bounding box.
84
+ - The image is resized to 352 x 352 without keeping the aspect ratio, as the original processor; masks are coarse (22 x 22 patch tokens).
85
+
86
+ ## Usage
87
+
88
+ ```python
89
+ from fastforks import promptable_segmentation
90
+
91
+ model = promptable_segmentation.load("clipseg") # mode="auto": the preferred backend from recorded measurements
92
+ result = model("photo.jpg", input_points=[[x, y]], input_labels=[1])
93
+ result.masks # [M, H, W] bool
94
+ # SAM 3 (text prompts): model("photo.jpg", text="coin") -> boxes, scores, masks
95
+ result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
96
+ ```
97
+
98
+ Install with `pip install fastforks` (see <https://github.com/kleinicke/fastforks>). Other variants: `clipseg`.
99
+
100
+ ## Kernels backend
101
+
102
+ `promptable_segmentation.load("clipseg", mode="kernels")` runs our CUDA kernels (`fvm_conv`). It needs a prebuilt kernel for your GPU architecture (downloaded from <https://huggingface.co/fastforks/kernels> on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.
103
+
104
+ <!-- staged from fastforks 1cbf1671 (2026-11-15) -->