v3 whole-frame retrain: README.md (FID 383.91, CLIP 20.73)
Browse files
README.md
CHANGED
|
@@ -22,7 +22,7 @@ model-index:
|
|
| 22 |
metrics:
|
| 23 |
- name: fid
|
| 24 |
type: fid
|
| 25 |
-
value:
|
| 26 |
- task:
|
| 27 |
type: text-to-image
|
| 28 |
dataset:
|
|
@@ -31,7 +31,7 @@ model-index:
|
|
| 31 |
metrics:
|
| 32 |
- name: clip
|
| 33 |
type: clip
|
| 34 |
-
value:
|
| 35 |
---
|
| 36 |
|
| 37 |
# PixelModel v3 🖼️
|
|
@@ -47,7 +47,8 @@ The difference this time is the network. v2 was v1 with wider layers and nothing
|
|
| 47 |
else (its own card says "that's the only change"). v3 keeps the PNG idea and
|
| 48 |
rebuilds the architecture: learned word embeddings instead of a fixed hash, sine
|
| 49 |
activations instead of tanh, and FiLM conditioning instead of concatenation. It
|
| 50 |
-
is 919,427 parameters, and it beats v1
|
|
|
|
| 51 |
|
| 52 |
## What changed from v1
|
| 53 |
|
|
@@ -69,35 +70,40 @@ v1's real limits were three specific things, and v3 replaces each one.
|
|
| 69 |
|
| 70 |
## Results
|
| 71 |
|
| 72 |
-
|
| 73 |
-
hair short on CLIP.
|
| 74 |
|
| 75 |
-

|
| 76 |
|
| 77 |
-
| Metric |
|
| 78 |
-
|---|---|---|---|
|
| 79 |
-
| Parameters |
|
| 80 |
-
| FID, published |
|
| 81 |
-
| CLIP, published |
|
| 82 |
-
| FID, this pipeline (lower better) |
|
| 83 |
-
| CLIP, this pipeline (higher better) |
|
| 84 |
-
| `model.png` |
|
| 85 |
-
| Native resolution |
|
| 86 |
|
| 87 |
-
Read this carefully, because the fair comparison is not v3 against
|
| 88 |
-
number. It is
|
| 89 |
|
| 90 |
- Re scoring v1 here gave FID **420.75**, not the published 439.46, and CLIP
|
| 91 |
**20.10**, not 20.02. That roughly 19 point FID gap is library and subset drift,
|
| 92 |
-
not a quality change. Comparing v3
|
| 93 |
-
overstated the win
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
|
| 102 |
Everything labelled "this pipeline" is n=5000, MS COCO val2014 with a 256 center
|
| 103 |
crop, scored with `torchmetrics` (`FrechetInceptionDistance` and `CLIPScore` with
|
|
@@ -108,16 +114,16 @@ eval images are hash checked to be disjoint from training.
|
|
| 108 |
|
| 109 |

|
| 110 |
|
| 111 |
-
80 epochs on a single RTX 3090, about 40 minutes, batch 32, 128x128
|
| 112 |
-
learning rate from 2e-4, mixed precision. Training loss goes from 0.
|
| 113 |
-
0.
|
| 114 |
|
| 115 |
### On scaling up
|
| 116 |
|
| 117 |
I also trained a larger variant (about 3.4M parameters, wider and deeper, at
|
| 118 |
256x256) to see whether the extra capacity helps. It does not. At the base
|
| 119 |
learning rate it would not descend at all, and even after dropping the learning
|
| 120 |
-
rate its best loss (0.070) never reached base's 0.
|
| 121 |
deeper SIREN with a large FiLM hypernetwork is simply harder to optimise here, and
|
| 122 |
the extra parameters buy nothing for this task. So the released model is the
|
| 123 |
919K base. That is the honest outcome, not a hidden one.
|
|
@@ -164,7 +170,7 @@ mapped linearly from the range [-2, 2]:
|
|
| 164 |
- G channel is the low byte
|
| 165 |
- B channel is reserved
|
| 166 |
|
| 167 |
-
At 919,427 weights the PNG is a roughly square 959x959 image of 1.
|
| 168 |
in about 30 ms. Round trip quantisation error is about 3e-5 per weight, and no
|
| 169 |
trained weight fell outside [-2, 2]. `model.safetensors` holds the identical
|
| 170 |
weights in standard format with the full parameter breakdown in its header.
|
|
@@ -192,7 +198,7 @@ python main.py "a beach with palm trees" --res 256
|
|
| 192 |
# eval: FID and CLIP
|
| 193 |
python eval/run_eval.py --arch v3 --work ../pm-work --n 5000
|
| 194 |
|
| 195 |
-
# regression: re score v1 through the identical pipeline for a fair comparison
|
| 196 |
python eval/run_eval.py --arch precomputed --work ../pm-work \
|
| 197 |
--images-dir v1_render/ --n 5000
|
| 198 |
```
|
|
|
|
| 22 |
metrics:
|
| 23 |
- name: fid
|
| 24 |
type: fid
|
| 25 |
+
value: 383.91
|
| 26 |
- task:
|
| 27 |
type: text-to-image
|
| 28 |
dataset:
|
|
|
|
| 31 |
metrics:
|
| 32 |
- name: clip
|
| 33 |
type: clip
|
| 34 |
+
value: 20.73
|
| 35 |
---
|
| 36 |
|
| 37 |
# PixelModel v3 🖼️
|
|
|
|
| 47 |
else (its own card says "that's the only change"). v3 keeps the PNG idea and
|
| 48 |
rebuilds the architecture: learned word embeddings instead of a fixed hash, sine
|
| 49 |
activations instead of tanh, and FiLM conditioning instead of concatenation. It
|
| 50 |
+
is 919,427 parameters, and on the same eval pipeline it beats both v1 and v2 on
|
| 51 |
+
FID and on CLIP.
|
| 52 |
|
| 53 |
## What changed from v1
|
| 54 |
|
|
|
|
| 70 |
|
| 71 |
## Results
|
| 72 |
|
| 73 |
+
On the same eval pipeline, v3 posts the best FID and the best CLIP of the three.
|
|
|
|
| 74 |
|
| 75 |
+

|
| 76 |
|
| 77 |
+
| Metric | v1 | v2 | v3 |
|
| 78 |
+
|---|---|---|---|
|
| 79 |
+
| Parameters | 23,747 | ~200,000 | **919,427** |
|
| 80 |
+
| FID, published | 439.46 | not published | see below |
|
| 81 |
+
| CLIP, published | 20.02 | not published | see below |
|
| 82 |
+
| FID, this pipeline (lower better) | 420.75 | 390.68 | **383.91** |
|
| 83 |
+
| CLIP, this pipeline (higher better) | 20.10 | 20.48 | **20.73** |
|
| 84 |
+
| `model.png` | 160x149 | scaled | 959x959 (1.78 MB) |
|
| 85 |
+
| Native resolution | 64 | 64 | 128 |
|
| 86 |
|
| 87 |
+
Read this carefully, because the fair comparison is not v3 against a published
|
| 88 |
+
number. It is every model re run through the exact same code.
|
| 89 |
|
| 90 |
- Re scoring v1 here gave FID **420.75**, not the published 439.46, and CLIP
|
| 91 |
**20.10**, not 20.02. That roughly 19 point FID gap is library and subset drift,
|
| 92 |
+
not a quality change. Comparing v3 to a published number from a different setup
|
| 93 |
+
would have overstated the win, which is why the regression is not optional.
|
| 94 |
+
- v2 never published an FID or CLIP (its card lists both as "NOT DONE"), so it was
|
| 95 |
+
re scored here from its own weights through the identical pipeline: FID
|
| 96 |
+
**390.68**, CLIP **20.48**.
|
| 97 |
+
- On the same pipeline the ranking is unambiguous. FID: v3 **383.91** < v2 390.68
|
| 98 |
+
< v1 420.75. CLIP: v3 **20.73** > v2 20.48 > v1 20.10. v3 wins both, at about 39
|
| 99 |
+
times v1's parameter count and about 4.6 times v2's.
|
| 100 |
+
|
| 101 |
+
The thing that got v3 there was matching train and inference scale. v3 renders and
|
| 102 |
+
is scored on whole frames, so it is trained on whole frames at the same size,
|
| 103 |
+
rather than on zoomed in crops. Training on crops teaches the network patch
|
| 104 |
+
statistics that never appear when it paints a full image, and closing that gap is
|
| 105 |
+
what moved FID from 392 to 384 and CLIP from 19.9 to 20.7 without touching the
|
| 106 |
+
architecture.
|
| 107 |
|
| 108 |
Everything labelled "this pipeline" is n=5000, MS COCO val2014 with a 256 center
|
| 109 |
crop, scored with `torchmetrics` (`FrechetInceptionDistance` and `CLIPScore` with
|
|
|
|
| 114 |
|
| 115 |

|
| 116 |
|
| 117 |
+
80 epochs on a single RTX 3090, about 40 minutes, batch 32, 128x128 whole frames,
|
| 118 |
+
cosine learning rate from 2e-4, mixed precision. Training loss goes from 0.0720 to
|
| 119 |
+
0.0540 and has flattened by the end.
|
| 120 |
|
| 121 |
### On scaling up
|
| 122 |
|
| 123 |
I also trained a larger variant (about 3.4M parameters, wider and deeper, at
|
| 124 |
256x256) to see whether the extra capacity helps. It does not. At the base
|
| 125 |
learning rate it would not descend at all, and even after dropping the learning
|
| 126 |
+
rate its best loss (0.070) never reached base's 0.054 before it destabilised. A
|
| 127 |
deeper SIREN with a large FiLM hypernetwork is simply harder to optimise here, and
|
| 128 |
the extra parameters buy nothing for this task. So the released model is the
|
| 129 |
919K base. That is the honest outcome, not a hidden one.
|
|
|
|
| 170 |
- G channel is the low byte
|
| 171 |
- B channel is reserved
|
| 172 |
|
| 173 |
+
At 919,427 weights the PNG is a roughly square 959x959 image of 1.78 MB that loads
|
| 174 |
in about 30 ms. Round trip quantisation error is about 3e-5 per weight, and no
|
| 175 |
trained weight fell outside [-2, 2]. `model.safetensors` holds the identical
|
| 176 |
weights in standard format with the full parameter breakdown in its header.
|
|
|
|
| 198 |
# eval: FID and CLIP
|
| 199 |
python eval/run_eval.py --arch v3 --work ../pm-work --n 5000
|
| 200 |
|
| 201 |
+
# regression: re score v1 or v2 through the identical pipeline for a fair comparison
|
| 202 |
python eval/run_eval.py --arch precomputed --work ../pm-work \
|
| 203 |
--images-dir v1_render/ --n 5000
|
| 204 |
```
|