ifmain commited on
Commit
3d17f6b
·
verified ·
1 Parent(s): 223cf5d

Use internal CAT benchmark and update published reference figures

Browse files
README.md CHANGED
@@ -1,201 +1,160 @@
1
- ---
2
- license: apache-2.0
3
- pipeline_tag: image-classification
4
- metrics:
5
- - accuracy
6
- - precision
7
- - recall
8
- - roc_auc
9
- base_model: black-forest-labs/FLUX.2-klein-base-4B
10
- datasets:
11
- - ozdentarikcan/DiffVaxDataset
12
- extra_gated_prompt: >-
13
- By clicking "Agree", you agree to process only data you own or are licensed
14
- or otherwise authorized to use, and to respect applicable copyright,
15
- privacy, consent and contractual obligations.
16
- extra_gated_button_content: Agree
17
- ---
18
-
19
- # DataPoisoning Detector v1 Medium
20
-
21
- <!-- architecture-figures -->
22
- ## Architecture at a glance
23
-
24
- ![Four-level VAE feature fusion](assets/architecture.png)
25
-
26
- ![Image-level inference and soft-target distillation](assets/inference-distillation.png)
27
-
28
- Original diagrams for this release. Editable SVG versions: [architecture](assets/architecture.svg) · [inference and distillation](assets/inference-distillation.svg).
29
- <!-- /architecture-figures -->
30
-
31
-
32
- Weights and completed test results are included in this release bundle.
33
-
34
- [Code and training](https://github.com/ifmain/datapoisoning-detector-v1) · [Evaluation reports](https://github.com/ifmain/datapoisoning-detector-v1/tree/main/reports/three_tap_run)
35
-
36
- [Model repository](https://huggingface.co/ifmain/datapoisoning-detector-v1-medium)
37
-
38
- The detector compares encoder blocks 1, 2 and 3 with the final posterior-mean representation of a frozen FLUX.2 VAE. Each stream is projected to an 8 × 8 token grid. The three early-to-final differences are fused with all four feature streams before transformer classification. Inputs are native-resolution 128 × 128 RGB crops with a central 96 × 96 active region. The VAE decoder and generative FLUX transformer are not used.
39
-
40
- This run excludes CelebA-HQ from training, validation, test and patch caches. It contains 21,464 training, 1,257 validation and 1,446 test records. Large starts with newly initialized trainable parameters and the licensed frozen VAE encoder. Students learn from this new Large through temperature-2 Bernoulli KL soft-target distillation: 50% distillation and 50% supervised classification plus paired ranking. Checkpoint selection uses validation recall at a target false-positive rate of 1%, with ROC-AUC as the tie-breaker. A decline receives two additional recovery epochs unless a sustained overfitting signal is observed.
41
-
42
- Image-level inference averages five crop logits. Scores are not calibrated probabilities. The revised test set is a filtered subset of the previous experiment, not a fresh blind holdout. These results must not be presented as the old experiment's measurements. Local Glaze performance is reported separately below because that subset previously exposed a failure.
43
-
44
- | Variant | Total parameters | Trainable parameters | Status |
45
- |---|---:|---:|---:|
46
- | Large | 296,264,065 | 261,838,209 | Evaluated |
47
- | Medium | 182,722,433 | 148,296,577 | Evaluated |
48
- | Small | 82,635,137 | 48,209,281 | Evaluated |
49
- | Tiny | 43,280,257 | 8,854,401 | Evaluated |
50
-
51
- ## Original test: model family comparison
52
-
53
- ### Detection recall
54
-
55
- Percentage of protected images correctly detected. Higher is better.
56
-
57
- | Protection | Large | Medium | Small | Tiny |
58
- |---|---:|---:|---:|---:|
59
- | Nightshade | 97.92% (47/48) | 95.83% (46/48) | **100.00% (48/48)** | 95.83% (46/48) |
60
- | Glaze | **95.45% (63/66)** | 93.94% (62/66) | **95.45% (63/66)** | **95.45% (63/66)** |
61
- | Mist | **100.00% (22/22)** | **100.00% (22/22)** | **100.00% (22/22)** | **100.00% (22/22)** |
62
- | MetaCloak | **100.00% (4/4)** | **100.00% (4/4)** | **100.00% (4/4)** | **100.00% (4/4)** |
63
-
64
- ### Clean-image specificity
65
-
66
- Percentage of clean images correctly accepted as clean. Higher is better. Each model uses its own validation-selected threshold on the same clean-control pool.
67
-
68
- | Evaluation pool | Large | Medium | Small | Tiny |
69
- |---|---:|---:|---:|---:|
70
- | Shared clean controls | 99.1129% (1229/1240) | **99.2742% (1231/1240)** | 99.0323% (1228/1240) | 99.1129% (1229/1240) |
71
-
72
- ### Distillation quality
73
-
74
- All students use the newly trained three-tap Large teacher. No previous detector checkpoint is used. Teacher agreement measures matching decisions, not correctness.
75
-
76
- | Metric | Large | Medium | Small | Tiny |
77
- |---|---:|---:|---:|---:|
78
- | Accuracy | 98.8935% | 98.7552% | 98.8935% | 98.8243% |
79
- | Recall | 97.5728% | 95.6311% | 98.0583% | 97.0874% |
80
- | Precision | 94.8113% | 95.6311% | 94.3925% | 94.7867% |
81
- | False positive rate | 0.8871% | 0.7258% | 0.9677% | 0.8871% |
82
- | ROC-AUC | 0.998555 | 0.997851 | 0.999010 | 0.996336 |
83
- | Teacher decision agreement | 100% (self) | 99.4467% | 99.5851% | 99.5159% |
84
- | Image-logit MAE (lower is better) | 0 (self) | 0.658140 | 0.638014 | 0.683860 |
85
- | Recall change vs. teacher (pp) | 0 (self) | -1.941748 | 0.485437 | -0.485437 |
86
-
87
- ## Original test: published comparison
88
-
89
- ![Medium vs. LightShed](assets/published-comparison.png)
90
-
91
- This comparison uses the newly trained **Medium** model.
92
-
93
- ### Detection recall
94
-
95
- Percentage of protected images correctly detected. Higher is better.
96
-
97
- | Protection | Our recall (detected/positive) | LightShed Table 2 recall |
98
- |---|---:|---:|
99
- | Nightshade (binary comparator) | 95.83% (46/48) | **96.55%** |
100
- | Glaze | 93.94% (62/66) | **97.26%** |
101
- | Mist | **100.00% (22/22)** | 99.84% |
102
- | MetaCloak | **100.00% (4/4)** | 91.77% |
103
-
104
- ### Clean-image specificity
105
-
106
- Percentage of clean images correctly accepted as clean. Higher is better. Medium uses one shared clean-control pool.
107
-
108
- | Comparison condition | Our clean specificity | LightShed Table 2 specificity |
109
- |---|---:|---:|
110
- | Nightshade (binary comparator) | **99.27%** | 92.86% |
111
- | Glaze | **99.27%** | 97.70% |
112
- | Mist | 99.27% | **100.00%** |
113
- | MetaCloak | **99.27%** | 84.32% |
114
-
115
- Published independent evaluations; bold marks the higher value in each row. LightShed reports method-specific operating points; ours uses a fixed validation-selected threshold. Its NightShade LPIPS 0.07 condition separately reports 99.98% recall / 100% specificity. [Source: LightShed, Table 2](https://www.usenix.org/conference/usenixsecurity25/presentation/foerster). No LightShed code, weights or gated dataset was used.
116
-
117
- ## Additional internal benchmark: CAT CelebA-HQ
118
-
119
- All 2,000 eligible images in the archived subset: 200 per protection method and 200 clean controls. Noisy baseline excluded. **Original five-crop inference and unchanged release thresholds**, not experimental sliding-window calibration.
120
-
121
- ![CAT benchmark: Medium](assets/cat-celebahq-medium.png)
122
-
123
- ### Detection recall
124
-
125
- Percentage of protected images correctly detected. Higher is better.
126
-
127
- | Protection | Large | Medium | Small | Tiny |
128
- |---|---:|---:|---:|---:|
129
- | advdm- | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
130
- | advdm+ | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
131
- | anti-dreambooth | 3.00% (6/200) | **38.00% (76/200)** | 23.00% (46/200) | 1.50% (3/200) |
132
- | glaze2 | 99.00% (198/200) | 99.00% (198/200) | **99.50% (199/200)** | **99.50% (199/200)** |
133
- | metacloak | 0.50% (1/200) | **29.00% (58/200)** | 12.50% (25/200) | 2.50% (5/200) |
134
- | mist | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
135
- | sds- | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
136
- | sds+ | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
137
- | sdsT5 | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
138
-
139
- ### Clean-image specificity
140
-
141
- Percentage of clean images correctly accepted as clean. Higher is better. All models use the same 200 clean controls.
142
-
143
- | Evaluation pool | Large | Medium | Small | Tiny |
144
- |---|---:|---:|---:|---:|
145
- | Shared clean controls | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
146
-
147
- ### Overall results
148
-
149
- | Metric | Large | Medium | Small | Tiny |
150
- |---|---:|---:|---:|---:|
151
- | Accuracy | 80.2500% | 86.6000% | 83.5000% | 80.3500% |
152
- | Protected-image recall | 78.0556% | 85.1111% | 81.6667% | 78.1667% |
153
- | Clean-image specificity | 100.0000% | 100.0000% | 100.0000% | 100.0000% |
154
- | False positive rate | 0.00% | 0.00% | 0.00% | 0.00% |
155
- | True positives | 1405 | 1532 | 1470 | 1407 |
156
- | False negatives | 395 | 268 | 330 | 393 |
157
- | False positives | 0 | 0 | 0 | 0 |
158
- | True negatives | 200 | 200 | 200 | 200 |
159
- | Decision threshold (logit) | -1.5164062976837156 | -1.1406249999999998 | -0.7234374880790709 | -1.8132812976837156 |
160
-
161
- Bold marks the highest per-method recall or specificity, including ties. No competitor numbers are mixed into this benchmark.
162
-
163
-
164
- **Evaluation-only source:** [CAT](https://github.com/senp98/CAT), CelebA-HQ subset. This subset was excluded from training and validation of the currently released checkpoints and was not used to fit their thresholds. The project author confirms having reviewed the source terms and permission for this evaluation use. Images are not redistributed. Protection methods share underlying originals.
165
-
166
- [Detailed protocol and results](https://github.com/ifmain/datapoisoning-detector-v1/blob/main/reports/cat_celebahq_benchmark/BENCHMARK.md).
167
-
168
- ## Local Glaze check
169
-
170
- | Variant | Detected / positive |
171
- |---|---:|
172
- | Large | 2/2 |
173
- | Medium | 1/2 |
174
- | Small | 2/2 |
175
- | Tiny | 2/2 |
176
-
177
- ## License and data provenance
178
-
179
- The author licenses the code and the rights they hold in the new model weights under Apache-2.0. The frozen BFL VAE component is Apache-2.0. This grant does not establish third-party rights in training images. Retained sources include VGGFace2 and WikiArt through CAT, robust-style-mimicry, XAI, Helen, Open Images, DiffVax and user-supplied data; their commercial permissions have not all been verified. See [the provenance report](https://github.com/ifmain/datapoisoning-detector-v1/blob/main/reports/three_tap_run/provenance.md).
180
-
181
- ## Citation
182
-
183
- If you find our work helpful, feel free to give us a cite.
184
- ```bibtex
185
- @misc{ifmain2026datapoisoningdetector,
186
- title = {DataPoisoning Detector v1: Multi-Level FLUX VAE Features for Protective-Perturbation Detection},
187
- url = {https://github.com/ifmain/datapoisoning-detector-v1},
188
- author = {{ifmain}},
189
- month = {October},
190
- year = {2026}
191
- }
192
- ```
193
-
194
- Selected protection and dataset references: [REFERENCES.bib](https://github.com/ifmain/datapoisoning-detector-v1/blob/main/REFERENCES.bib).
195
-
196
- ## Component layout
197
-
198
- `detector/` contains the trained projection, fusion and classification head.
199
- `vae/` contains the frozen FLUX VAE encoder and its configuration; no decoder is included.
200
- Load the repository root or its `detector/` directory with `dpdetector.load_detector`.
201
- The two component state dictionaries reconstruct the original checkpoint exactly.
 
1
+ ---
2
+ license: apache-2.0
3
+ pipeline_tag: image-classification
4
+ metrics:
5
+ - accuracy
6
+ - precision
7
+ - recall
8
+ - roc_auc
9
+ base_model: black-forest-labs/FLUX.2-klein-base-4B
10
+ datasets:
11
+ - ozdentarikcan/DiffVaxDataset
12
+ extra_gated_prompt: >-
13
+ By clicking "Agree", you agree to process only data you own or are licensed
14
+ or otherwise authorized to use, and to respect applicable copyright,
15
+ privacy, consent and contractual obligations.
16
+ extra_gated_button_content: Agree
17
+ ---
18
+
19
+ # DataPoisoning Detector v1 Medium
20
+
21
+ <!-- architecture-figures -->
22
+ ## Architecture at a glance
23
+
24
+ ![Four-level VAE feature fusion](assets/architecture.png)
25
+
26
+ ![Image-level inference and soft-target distillation](assets/inference-distillation.png)
27
+
28
+ Original diagrams for this release. Editable SVG versions: [architecture](assets/architecture.svg) · [inference and distillation](assets/inference-distillation.svg).
29
+ <!-- /architecture-figures -->
30
+
31
+
32
+ Weights and completed test results are included in this release bundle.
33
+
34
+ [Code and training](https://github.com/ifmain/datapoisoning-detector-v1) · [Evaluation reports](https://github.com/ifmain/datapoisoning-detector-v1/tree/main/reports/three_tap_run)
35
+
36
+ [Model repository](https://huggingface.co/ifmain/datapoisoning-detector-v1-medium)
37
+
38
+ The detector compares encoder blocks 1, 2 and 3 with the final posterior-mean representation of a frozen FLUX.2 VAE. Each stream is projected to an 8 × 8 token grid. The three early-to-final differences are fused with all four feature streams before transformer classification. Inputs are native-resolution 128 × 128 RGB crops with a central 96 × 96 active region. The VAE decoder and generative FLUX transformer are not used.
39
+
40
+ The training run excludes CelebA-HQ from training, validation and patch caches; it is used only for the separate internal evaluation below. It contains 21,464 training, 1,257 validation and 1,446 test records. Large starts with newly initialized trainable parameters and the licensed frozen VAE encoder. Students learn from this new Large through temperature-2 Bernoulli KL soft-target distillation: 50% distillation and 50% supervised classification plus paired ranking. Checkpoint selection uses validation recall at a target false-positive rate of 1%, with ROC-AUC as the tie-breaker. A decline receives two additional recovery epochs unless a sustained overfitting signal is observed.
41
+
42
+ Image-level inference averages five crop logits. Scores are not calibrated probabilities. All benchmark tables below use the internal CAT evaluation and unchanged release thresholds.
43
+
44
+ | Variant | Total parameters | Trainable parameters | Status |
45
+ |---|---:|---:|---:|
46
+ | [Large](https://huggingface.co/ifmain/datapoisoning-detector-v1-large) | 296,264,065 | 261,838,209 | Evaluated |
47
+ | [Medium](https://huggingface.co/ifmain/datapoisoning-detector-v1-medium) | 182,722,433 | 148,296,577 | Evaluated |
48
+ | [Small](https://huggingface.co/ifmain/datapoisoning-detector-v1-small) | 82,635,137 | 48,209,281 | Evaluated |
49
+ | [Tiny](https://huggingface.co/ifmain/datapoisoning-detector-v1-tiny) | 43,280,257 | 8,854,401 | Evaluated |
50
+
51
+ ## Internal benchmark: CAT CelebA-HQ
52
+
53
+ All 2,000 eligible images in the archived subset: 200 per protection method and 200 clean controls. Noisy baseline excluded. **Original five-crop inference and unchanged release thresholds**, not experimental sliding-window calibration.
54
+
55
+ ![CAT benchmark: Medium](assets/cat-celebahq-medium.png)
56
+
57
+ ### Detection recall
58
+
59
+ Percentage of protected images correctly detected. Higher is better.
60
+
61
+ | Protection | [Large](https://huggingface.co/ifmain/datapoisoning-detector-v1-large) | [Medium](https://huggingface.co/ifmain/datapoisoning-detector-v1-medium) | [Small](https://huggingface.co/ifmain/datapoisoning-detector-v1-small) | [Tiny](https://huggingface.co/ifmain/datapoisoning-detector-v1-tiny) |
62
+ |---|---:|---:|---:|---:|
63
+ | advdm- | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
64
+ | advdm+ | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
65
+ | anti-dreambooth | 3.00% (6/200) | **38.00% (76/200)** | 23.00% (46/200) | 1.50% (3/200) |
66
+ | glaze2 | 99.00% (198/200) | 99.00% (198/200) | **99.50% (199/200)** | **99.50% (199/200)** |
67
+ | metacloak | 0.50% (1/200) | **29.00% (58/200)** | 12.50% (25/200) | 2.50% (5/200) |
68
+ | mist | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
69
+ | sds- | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
70
+ | sds+ | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
71
+ | sdsT5 | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
72
+
73
+ ### Clean-image specificity
74
+
75
+ Percentage of clean images correctly accepted as clean. Higher is better. All models use the same 200 clean controls.
76
+
77
+ | Evaluation pool | [Large](https://huggingface.co/ifmain/datapoisoning-detector-v1-large) | [Medium](https://huggingface.co/ifmain/datapoisoning-detector-v1-medium) | [Small](https://huggingface.co/ifmain/datapoisoning-detector-v1-small) | [Tiny](https://huggingface.co/ifmain/datapoisoning-detector-v1-tiny) |
78
+ |---|---:|---:|---:|---:|
79
+ | Shared clean controls | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** | **100.00% (200/200)** |
80
+
81
+ ### Overall results
82
+
83
+ | Metric | [Large](https://huggingface.co/ifmain/datapoisoning-detector-v1-large) | [Medium](https://huggingface.co/ifmain/datapoisoning-detector-v1-medium) | [Small](https://huggingface.co/ifmain/datapoisoning-detector-v1-small) | [Tiny](https://huggingface.co/ifmain/datapoisoning-detector-v1-tiny) |
84
+ |---|---:|---:|---:|---:|
85
+ | Accuracy | 80.2500% | 86.6000% | 83.5000% | 80.3500% |
86
+ | Protected-image recall | 78.0556% | 85.1111% | 81.6667% | 78.1667% |
87
+ | Clean-image specificity | 100.0000% | 100.0000% | 100.0000% | 100.0000% |
88
+ | False positive rate | 0.00% | 0.00% | 0.00% | 0.00% |
89
+ | True positives | 1405 | 1532 | 1470 | 1407 |
90
+ | False negatives | 395 | 268 | 330 | 393 |
91
+ | False positives | 0 | 0 | 0 | 0 |
92
+ | True negatives | 200 | 200 | 200 | 200 |
93
+ | Decision threshold (logit) | -1.5164062976837156 | -1.1406249999999998 | -0.7234374880790709 | -1.8132812976837156 |
94
+
95
+ Bold marks the highest per-method recall or specificity, including ties. Published reference results are listed separately below.
96
+
97
+
98
+ **Evaluation-only source:** [CAT](https://github.com/senp98/CAT), CelebA-HQ subset. This subset was excluded from training and validation of the currently released checkpoints and was not used to fit their thresholds. The project author confirms having reviewed the source terms and permission for this evaluation use. Images are not redistributed. Protection methods share underlying originals.
99
+
100
+ [Detailed protocol and results](https://github.com/ifmain/datapoisoning-detector-v1/blob/main/reports/cat_celebahq_benchmark/BENCHMARK.md).
101
+
102
+ ## Published reference alongside the internal benchmark
103
+
104
+ ![Internal benchmark and published reference](assets/published-comparison.png)
105
+
106
+ ### Detection recall
107
+
108
+ | Protection | Our Medium (%) | LightShed Table 2 (%) |
109
+ |---|---:|---:|
110
+ | AdvDM (+) | 100.00% | Not reported |
111
+ | AdvDM (-) | 100.00% | Not reported |
112
+ | Anti-DreamBooth | 38.00% | Not reported |
113
+ | Glaze 2 / published Glaze | **99.00%** | 97.26% |
114
+ | MetaCloak | 29.00% | **91.77%** |
115
+ | Mist | **100.00%** | 99.84% |
116
+ | SDS (+) | 100.00% | Not reported |
117
+ | SDS (-) | 100.00% | Not reported |
118
+ | SDS T5 | 100.00% | Not reported |
119
+
120
+ ### Clean-image specificity
121
+
122
+ | Protection | Our Medium (%) | LightShed Table 2 (%) |
123
+ |---|---:|---:|
124
+ | AdvDM (+) | 100.00% | Not reported |
125
+ | AdvDM (-) | 100.00% | Not reported |
126
+ | Anti-DreamBooth | 100.00% | Not reported |
127
+ | Glaze 2 / published Glaze | **100.00%** | 97.70% |
128
+ | MetaCloak | **100.00%** | 84.32% |
129
+ | Mist | **100.00%** | **100.00%** |
130
+ | SDS (+) | 100.00% | Not reported |
131
+ | SDS (-) | 100.00% | Not reported |
132
+ | SDS T5 | 100.00% | Not reported |
133
+
134
+ Our numbers come exclusively from the internal CAT benchmark above. Clean specificity uses the same 200 controls for every row. LightShed numbers are published results on its own evaluation, not measurements on this internal set. Glaze 2 is shown alongside published Glaze; versions and settings are not asserted identical. **Bold** marks the larger reported value, including ties, not a controlled head-to-head win. Not reported means absent from Table 2. [LightShed source, Table 2](https://www.usenix.org/system/files/usenixsecurity25-foerster.pdf).
135
+
136
+ ## License and data provenance
137
+
138
+ The author licenses the code and the rights they hold in the new model weights under Apache-2.0. The frozen BFL VAE component is Apache-2.0. This grant does not establish third-party rights in training images. Retained sources include VGGFace2 and WikiArt through CAT, robust-style-mimicry, XAI, Helen, Open Images, DiffVax and user-supplied data; their commercial permissions have not all been verified. See [the provenance report](https://github.com/ifmain/datapoisoning-detector-v1/blob/main/reports/three_tap_run/provenance.md).
139
+
140
+ ## Citation
141
+
142
+ If you find our work helpful, feel free to give us a cite.
143
+ ```bibtex
144
+ @misc{ifmain2026datapoisoningdetector,
145
+ title = {DataPoisoning Detector v1: Multi-Level FLUX VAE Features for Protective-Perturbation Detection},
146
+ url = {https://github.com/ifmain/datapoisoning-detector-v1},
147
+ author = {{ifmain}},
148
+ month = {October},
149
+ year = {2026}
150
+ }
151
+ ```
152
+
153
+ Selected protection and dataset references: [REFERENCES.bib](https://github.com/ifmain/datapoisoning-detector-v1/blob/main/REFERENCES.bib).
154
+
155
+ ## Component layout
156
+
157
+ `detector/` contains the trained projection, fusion and classification head.
158
+ `vae/` contains the frozen FLUX VAE encoder and its configuration; no decoder is included.
159
+ Load the repository root or its `detector/` directory with `dpdetector.load_detector`.
160
+ The two component state dictionaries reconstruct the original checkpoint exactly.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
assets/published-comparison.json CHANGED
@@ -1,54 +1,40 @@
1
  {
2
- "model": "medium",
 
3
  "source": "https://www.usenix.org/system/files/usenixsecurity25-foerster.pdf",
4
- "rows": [
5
- {
6
- "metric": "Detection recall",
7
- "protection": "nightshade",
8
- "ours": 95.83333333333334,
9
- "lightshed": 96.55
10
- },
11
- {
12
- "metric": "Detection recall",
13
- "protection": "glaze",
14
- "ours": 93.93939393939394,
15
- "lightshed": 97.26
16
- },
17
- {
18
- "metric": "Detection recall",
19
- "protection": "mist",
20
- "ours": 100.0,
21
- "lightshed": 99.84
22
- },
23
- {
24
- "metric": "Detection recall",
25
- "protection": "metacloak",
26
- "ours": 100.0,
27
- "lightshed": 91.77
28
- },
29
- {
30
- "metric": "Clean-image specificity",
31
- "protection": "nightshade",
32
- "ours": 99.2741935483871,
33
- "lightshed": 92.86
34
- },
35
- {
36
- "metric": "Clean-image specificity",
37
- "protection": "glaze",
38
- "ours": 99.2741935483871,
39
- "lightshed": 97.7
40
- },
41
- {
42
- "metric": "Clean-image specificity",
43
- "protection": "mist",
44
- "ours": 99.2741935483871,
45
- "lightshed": 100
46
- },
47
- {
48
- "metric": "Clean-image specificity",
49
- "protection": "metacloak",
50
- "ours": 99.2741935483871,
51
- "lightshed": 84.32
52
  }
53
- ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
54
  }
 
1
  {
2
+ "variant": "medium",
3
+ "benchmark": "CAT CelebA-HQ internal 2000",
4
  "source": "https://www.usenix.org/system/files/usenixsecurity25-foerster.pdf",
5
+ "ours": {
6
+ "threshold": -1.1406249999999998,
7
+ "tp": 1532,
8
+ "fn": 268,
9
+ "fp": 0,
10
+ "tn": 200,
11
+ "recall": 0.8511111111111112,
12
+ "specificity": 1.0,
13
+ "accuracy": 0.866,
14
+ "per_method": {
15
+ "advdm+": 200,
16
+ "advdm-": 200,
17
+ "anti-dreambooth": 76,
18
+ "glaze2": 198,
19
+ "metacloak": 58,
20
+ "mist": 200,
21
+ "sds+": 200,
22
+ "sds-": 200,
23
+ "sdsT5": 200
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
  }
25
+ },
26
+ "published_reference": {
27
+ "glaze2": [
28
+ 97.26,
29
+ 97.7
30
+ ],
31
+ "metacloak": [
32
+ 91.77,
33
+ 84.32
34
+ ],
35
+ "mist": [
36
+ 99.84,
37
+ 100.0
38
+ ]
39
+ }
40
  }
assets/published-comparison.png CHANGED

Git LFS Details

  • SHA256: 971fbbe87806c75f33bc372235d2b0f26b7d59f29e847000da3e218e4546a7a8
  • Pointer size: 131 Bytes
  • Size of remote file: 258 kB

Git LFS Details

  • SHA256: 2c7db2241207525dfcc10e9931f5fb8df58f913b3f9b2fe15bbef6da662805dd
  • Pointer size: 131 Bytes
  • Size of remote file: 147 kB
assets/published-comparison.svg CHANGED