Upload 3 files
Browse files- eval/eval.md +78 -0
- eval/eval_results.json +6 -0
- eval/run_eval.py +6 -0
eval/eval.md
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# PixelModel v1 β Evaluation
|
| 2 |
+
|
| 3 |
+
Benchmark results for the [Tiny-T2I-Leaderboard](https://huggingface.co/spaces/FlameF0X/Tiny-T2I-Leaderboard),
|
| 4 |
+
measured with the leaderboard's preferred tooling (torchmetrics). Everything
|
| 5 |
+
needed to reproduce these numbers ships in this repo β see `EVAL_REPRODUCTION.md`.
|
| 6 |
+
|
| 7 |
+
## Parameter count
|
| 8 |
+
|
| 9 |
+
23,747 parameters in 10 tensors (5 weight matrices + 5 bias vectors), all
|
| 10 |
+
stored in `model.png` (160Γ149 px, 16-bit/weight codec) and exported to
|
| 11 |
+
`model.safetensors` with the count in the header metadata:
|
| 12 |
+
|
| 13 |
+
| Tensor | Shape | Params | Role |
|
| 14 |
+
|---|---|---|---|
|
| 15 |
+
| T1 + b1 | 80Γ64 + 80 | 5,200 | prompt embedding β trunk hidden |
|
| 16 |
+
| T2 + b2 | 64Γ80 + 64 | 5,184 | trunk hidden β latent |
|
| 17 |
+
| D1 + bd1 | 80Γ82 + 80 | 6,640 | latent + fourier(x,y) β decoder hidden |
|
| 18 |
+
| D2 + bd2 | 80Γ80 + 80 | 6,480 | decoder hidden β decoder hidden |
|
| 19 |
+
| D3 + bd3 | 3Γ80 + 3 | 243 | decoder hidden β RGB |
|
| 20 |
+
| **Total** | | **23,747 (~0.024M)** | |
|
| 21 |
+
|
| 22 |
+
The prompt embedding is a hashed function (FNV-1a over character trigrams and
|
| 23 |
+
words) with zero learned parameters; there is no text encoder and no VAE
|
| 24 |
+
(`text_encoder_parameters: 0`, `vae_parameters: 0` in both `config.json` and
|
| 25 |
+
the safetensors metadata). Well under the 1.5B ceiling β and 8.5Γ smaller than
|
| 26 |
+
v0's 202,752.
|
| 27 |
+
|
| 28 |
+
## Method
|
| 29 |
+
|
| 30 |
+
- **Real set**: 5,000 MS-COCO val2014 (image, caption) pairs β rows 0β4,999 of
|
| 31 |
+
the [`sayakpaul/coco-30-val-2014`](https://huggingface.co/datasets/sayakpaul/coco-30-val-2014)
|
| 32 |
+
stream β images center-cropped to 256Γ256.
|
| 33 |
+
- **Generated set**: `model.png` run on those same 5,000 captions at the
|
| 34 |
+
native 64Γ64 resolution (no post-upscaling).
|
| 35 |
+
- **FID**: `torchmetrics.image.fid.FrechetInceptionDistance` (`feature=2048`,
|
| 36 |
+
InceptionV3 pool3; the metric resizes both sets to 299Γ299 internally).
|
| 37 |
+
- **CLIP Score**: `torchmetrics.multimodal.CLIPScore` with
|
| 38 |
+
`openai/clip-vit-base-patch32` (the default), generated image vs the caption
|
| 39 |
+
that produced it, averaged over all 5,000.
|
| 40 |
+
- **Train/eval hygiene**: the model was trained on 20,000 pairs from rows
|
| 41 |
+
10,000+ of the same stream, content-hash-checked to be disjoint from the
|
| 42 |
+
5,000 eval images (0 overlaps found; the 30K rows are 30K unique images).
|
| 43 |
+
|
| 44 |
+
n=5000 is far past the singular-covariance regime that made v0's n=40 FID
|
| 45 |
+
directional-only (a 2048Γ2048 covariance needs n β« 2048 to be non-singular;
|
| 46 |
+
5,000 clears it, 40 did not).
|
| 47 |
+
|
| 48 |
+
## Results
|
| 49 |
+
|
| 50 |
+
| Metric | v0 (n=40) | **v1 (n=5000)** | Context |
|
| 51 |
+
|---|---|---|---|
|
| 52 |
+
| FID β | 566.84 | **439.46** | Competitive T2I models score ~10β30; v1's caption-conditioned color/layout statistics move it meaningfully toward the real-photo distribution, but 64Γ64 color washes remain far outside it. |
|
| 53 |
+
| CLIP Score β | 18.60 | **20.02** (cosine 0.2002) | ~15β20 is the no-semantic-correspondence floor; ~28β32 is competitive. v1 sits measurably above the floor β captions genuinely steer palette and layout β with 8.5Γ fewer parameters than v0. |
|
| 54 |
+
|
| 55 |
+
Training loss context: the model's final MSE (0.0627) is 9% below the
|
| 56 |
+
"always output the mean COCO image" baseline (0.0691) β everything below that
|
| 57 |
+
baseline is caption-conditioning, since the mean image is the best possible
|
| 58 |
+
caption-blind predictor.
|
| 59 |
+
|
| 60 |
+
## Leaderboard entry
|
| 61 |
+
|
| 62 |
+
```js
|
| 63 |
+
{
|
| 64 |
+
name: "PixelModel-v1",
|
| 65 |
+
org: "bench-labs",
|
| 66 |
+
params: "0.024M",
|
| 67 |
+
fid: 439.46,
|
| 68 |
+
clipScore: 0.2002,
|
| 69 |
+
resolution: "64x64",
|
| 70 |
+
releaseDate: "2026-07-17",
|
| 71 |
+
links: {
|
| 72 |
+
card: "https://huggingface.co/bench-labs/pixelmodel-v1"
|
| 73 |
+
}
|
| 74 |
+
},
|
| 75 |
+
```
|
| 76 |
+
|
| 77 |
+
*(`clipScore` given as the 0β1 cosine value per the template's field
|
| 78 |
+
description; the 0β100 form of the same number is 20.02.)*
|
eval/eval_results.json
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"n": 5000,
|
| 3 |
+
"native_resolution": "64x64",
|
| 4 |
+
"fid": 439.4560852050781,
|
| 5 |
+
"clip_score": 20.020164489746094
|
| 6 |
+
}
|
eval/run_eval.py
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Compatibility launcher for the root-level evaluation script."""
|
| 2 |
+
|
| 3 |
+
import runpy
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
|
| 6 |
+
runpy.run_path(Path(__file__).resolve().parents[1] / "run_eval.py", run_name="__main__")
|