TobiasLogic commited on
Commit
6cbc917
·
verified ·
1 Parent(s): 2b338e1

v3 whole-frame retrain: README.md (FID 383.91, CLIP 20.73)

Browse files
Files changed (1) hide show
  1. README.md +38 -32
README.md CHANGED
@@ -22,7 +22,7 @@ model-index:
22
  metrics:
23
  - name: fid
24
  type: fid
25
- value: 392.10
26
  - task:
27
  type: text-to-image
28
  dataset:
@@ -31,7 +31,7 @@ model-index:
31
  metrics:
32
  - name: clip
33
  type: clip
34
- value: 19.88
35
  ---
36
 
37
  # PixelModel v3 🖼️
@@ -47,7 +47,8 @@ The difference this time is the network. v2 was v1 with wider layers and nothing
47
  else (its own card says "that's the only change"). v3 keeps the PNG idea and
48
  rebuilds the architecture: learned word embeddings instead of a fixed hash, sine
49
  activations instead of tanh, and FiLM conditioning instead of concatenation. It
50
- is 919,427 parameters, and it beats v1's FID.
 
51
 
52
  ## What changed from v1
53
 
@@ -69,35 +70,40 @@ v1's real limits were three specific things, and v3 replaces each one.
69
 
70
  ## Results
71
 
72
- The headline number: on the same eval pipeline, v3 beats v1 on FID and lands a
73
- hair short on CLIP.
74
 
75
- ![v1 vs v3 benchmark](pixelmodel-v3-benchmark.png)
76
 
77
- | Metric | v0 | v1 | v2 | v3 |
78
- |---|---|---|---|---|
79
- | Parameters | 202,752 | 23,747 | 200,000 | **919,427** |
80
- | FID, published | 566.84 | 439.46 | not published | see below |
81
- | CLIP, published | 18.60 | 20.02 | not published | see below |
82
- | FID, this pipeline (lower better) | | 420.75 | | **392.10** |
83
- | CLIP, this pipeline (higher better) | | 20.10 | | 19.88 |
84
- | `model.png` | 64x3200 | 160x149 | scaled | 959x959 (1.69 MB) |
85
- | Native resolution | 32 | 64 | 64 | 128 |
86
 
87
- Read this carefully, because the fair comparison is not v3 against v1's published
88
- number. It is v3 against v1 re run through the exact same code.
89
 
90
  - Re scoring v1 here gave FID **420.75**, not the published 439.46, and CLIP
91
  **20.10**, not 20.02. That roughly 19 point FID gap is library and subset drift,
92
- not a quality change. Comparing v3's 392.10 to the published 439.46 would have
93
- overstated the win by about 40 percent, which is why the regression is not
94
- optional.
95
- - On the same pipeline, v3 vs v1 is FID **392.10 vs 420.75** (v3 wins by 28.6,
96
- about 7 percent) and CLIP **19.88 vs 20.10** (v3 is 0.22 behind, essentially
97
- tied). So the honest claim is: v3 clearly improves FID and matches CLIP, at
98
- about 39 times v1's parameter count but less than half of v2's.
99
- - v2 has no published FID or CLIP (its card lists both as "NOT DONE") and its
100
- weights were not re scored here, so it contributes no number.
 
 
 
 
 
 
101
 
102
  Everything labelled "this pipeline" is n=5000, MS COCO val2014 with a 256 center
103
  crop, scored with `torchmetrics` (`FrechetInceptionDistance` and `CLIPScore` with
@@ -108,16 +114,16 @@ eval images are hash checked to be disjoint from training.
108
 
109
  ![training loss](pixelmodel-v3-loss.png)
110
 
111
- 80 epochs on a single RTX 3090, about 40 minutes, batch 32, 128x128 crops, cosine
112
- learning rate from 2e-4, mixed precision. Training loss goes from 0.0741 to
113
- 0.0641 and is still drifting down at the end.
114
 
115
  ### On scaling up
116
 
117
  I also trained a larger variant (about 3.4M parameters, wider and deeper, at
118
  256x256) to see whether the extra capacity helps. It does not. At the base
119
  learning rate it would not descend at all, and even after dropping the learning
120
- rate its best loss (0.070) never reached base's 0.064 before it destabilised. A
121
  deeper SIREN with a large FiLM hypernetwork is simply harder to optimise here, and
122
  the extra parameters buy nothing for this task. So the released model is the
123
  919K base. That is the honest outcome, not a hidden one.
@@ -164,7 +170,7 @@ mapped linearly from the range [-2, 2]:
164
  - G channel is the low byte
165
  - B channel is reserved
166
 
167
- At 919,427 weights the PNG is a roughly square 959x959 image of 1.69 MB that loads
168
  in about 30 ms. Round trip quantisation error is about 3e-5 per weight, and no
169
  trained weight fell outside [-2, 2]. `model.safetensors` holds the identical
170
  weights in standard format with the full parameter breakdown in its header.
@@ -192,7 +198,7 @@ python main.py "a beach with palm trees" --res 256
192
  # eval: FID and CLIP
193
  python eval/run_eval.py --arch v3 --work ../pm-work --n 5000
194
 
195
- # regression: re score v1 through the identical pipeline for a fair comparison
196
  python eval/run_eval.py --arch precomputed --work ../pm-work \
197
  --images-dir v1_render/ --n 5000
198
  ```
 
22
  metrics:
23
  - name: fid
24
  type: fid
25
+ value: 383.91
26
  - task:
27
  type: text-to-image
28
  dataset:
 
31
  metrics:
32
  - name: clip
33
  type: clip
34
+ value: 20.73
35
  ---
36
 
37
  # PixelModel v3 🖼️
 
47
  else (its own card says "that's the only change"). v3 keeps the PNG idea and
48
  rebuilds the architecture: learned word embeddings instead of a fixed hash, sine
49
  activations instead of tanh, and FiLM conditioning instead of concatenation. It
50
+ is 919,427 parameters, and on the same eval pipeline it beats both v1 and v2 on
51
+ FID and on CLIP.
52
 
53
  ## What changed from v1
54
 
 
70
 
71
  ## Results
72
 
73
+ On the same eval pipeline, v3 posts the best FID and the best CLIP of the three.
 
74
 
75
+ ![v1 vs v2 vs v3 benchmark](pixelmodel-v3-benchmark.png)
76
 
77
+ | Metric | v1 | v2 | v3 |
78
+ |---|---|---|---|
79
+ | Parameters | 23,747 | ~200,000 | **919,427** |
80
+ | FID, published | 439.46 | not published | see below |
81
+ | CLIP, published | 20.02 | not published | see below |
82
+ | FID, this pipeline (lower better) | 420.75 | 390.68 | **383.91** |
83
+ | CLIP, this pipeline (higher better) | 20.10 | 20.48 | **20.73** |
84
+ | `model.png` | 160x149 | scaled | 959x959 (1.78 MB) |
85
+ | Native resolution | 64 | 64 | 128 |
86
 
87
+ Read this carefully, because the fair comparison is not v3 against a published
88
+ number. It is every model re run through the exact same code.
89
 
90
  - Re scoring v1 here gave FID **420.75**, not the published 439.46, and CLIP
91
  **20.10**, not 20.02. That roughly 19 point FID gap is library and subset drift,
92
+ not a quality change. Comparing v3 to a published number from a different setup
93
+ would have overstated the win, which is why the regression is not optional.
94
+ - v2 never published an FID or CLIP (its card lists both as "NOT DONE"), so it was
95
+ re scored here from its own weights through the identical pipeline: FID
96
+ **390.68**, CLIP **20.48**.
97
+ - On the same pipeline the ranking is unambiguous. FID: v3 **383.91** < v2 390.68
98
+ < v1 420.75. CLIP: v3 **20.73** > v2 20.48 > v1 20.10. v3 wins both, at about 39
99
+ times v1's parameter count and about 4.6 times v2's.
100
+
101
+ The thing that got v3 there was matching train and inference scale. v3 renders and
102
+ is scored on whole frames, so it is trained on whole frames at the same size,
103
+ rather than on zoomed in crops. Training on crops teaches the network patch
104
+ statistics that never appear when it paints a full image, and closing that gap is
105
+ what moved FID from 392 to 384 and CLIP from 19.9 to 20.7 without touching the
106
+ architecture.
107
 
108
  Everything labelled "this pipeline" is n=5000, MS COCO val2014 with a 256 center
109
  crop, scored with `torchmetrics` (`FrechetInceptionDistance` and `CLIPScore` with
 
114
 
115
  ![training loss](pixelmodel-v3-loss.png)
116
 
117
+ 80 epochs on a single RTX 3090, about 40 minutes, batch 32, 128x128 whole frames,
118
+ cosine learning rate from 2e-4, mixed precision. Training loss goes from 0.0720 to
119
+ 0.0540 and has flattened by the end.
120
 
121
  ### On scaling up
122
 
123
  I also trained a larger variant (about 3.4M parameters, wider and deeper, at
124
  256x256) to see whether the extra capacity helps. It does not. At the base
125
  learning rate it would not descend at all, and even after dropping the learning
126
+ rate its best loss (0.070) never reached base's 0.054 before it destabilised. A
127
  deeper SIREN with a large FiLM hypernetwork is simply harder to optimise here, and
128
  the extra parameters buy nothing for this task. So the released model is the
129
  919K base. That is the honest outcome, not a hidden one.
 
170
  - G channel is the low byte
171
  - B channel is reserved
172
 
173
+ At 919,427 weights the PNG is a roughly square 959x959 image of 1.78 MB that loads
174
  in about 30 ms. Round trip quantisation error is about 3e-5 per weight, and no
175
  trained weight fell outside [-2, 2]. `model.safetensors` holds the identical
176
  weights in standard format with the full parameter breakdown in its header.
 
198
  # eval: FID and CLIP
199
  python eval/run_eval.py --arch v3 --work ../pm-work --n 5000
200
 
201
+ # regression: re score v1 or v2 through the identical pipeline for a fair comparison
202
  python eval/run_eval.py --arch precomputed --work ../pm-work \
203
  --images-dir v1_render/ --n 5000
204
  ```