wop commited on
Commit
aceb14e
Β·
verified Β·
1 Parent(s): e465a2f

Upload 3 files

Browse files
Files changed (3) hide show
  1. eval/eval.md +78 -0
  2. eval/eval_results.json +6 -0
  3. eval/run_eval.py +6 -0
eval/eval.md ADDED
@@ -0,0 +1,78 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # PixelModel v1 β€” Evaluation
2
+
3
+ Benchmark results for the [Tiny-T2I-Leaderboard](https://huggingface.co/spaces/FlameF0X/Tiny-T2I-Leaderboard),
4
+ measured with the leaderboard's preferred tooling (torchmetrics). Everything
5
+ needed to reproduce these numbers ships in this repo β€” see `EVAL_REPRODUCTION.md`.
6
+
7
+ ## Parameter count
8
+
9
+ 23,747 parameters in 10 tensors (5 weight matrices + 5 bias vectors), all
10
+ stored in `model.png` (160Γ—149 px, 16-bit/weight codec) and exported to
11
+ `model.safetensors` with the count in the header metadata:
12
+
13
+ | Tensor | Shape | Params | Role |
14
+ |---|---|---|---|
15
+ | T1 + b1 | 80Γ—64 + 80 | 5,200 | prompt embedding β†’ trunk hidden |
16
+ | T2 + b2 | 64Γ—80 + 64 | 5,184 | trunk hidden β†’ latent |
17
+ | D1 + bd1 | 80Γ—82 + 80 | 6,640 | latent + fourier(x,y) β†’ decoder hidden |
18
+ | D2 + bd2 | 80Γ—80 + 80 | 6,480 | decoder hidden β†’ decoder hidden |
19
+ | D3 + bd3 | 3Γ—80 + 3 | 243 | decoder hidden β†’ RGB |
20
+ | **Total** | | **23,747 (~0.024M)** | |
21
+
22
+ The prompt embedding is a hashed function (FNV-1a over character trigrams and
23
+ words) with zero learned parameters; there is no text encoder and no VAE
24
+ (`text_encoder_parameters: 0`, `vae_parameters: 0` in both `config.json` and
25
+ the safetensors metadata). Well under the 1.5B ceiling β€” and 8.5Γ— smaller than
26
+ v0's 202,752.
27
+
28
+ ## Method
29
+
30
+ - **Real set**: 5,000 MS-COCO val2014 (image, caption) pairs β€” rows 0–4,999 of
31
+ the [`sayakpaul/coco-30-val-2014`](https://huggingface.co/datasets/sayakpaul/coco-30-val-2014)
32
+ stream β€” images center-cropped to 256Γ—256.
33
+ - **Generated set**: `model.png` run on those same 5,000 captions at the
34
+ native 64Γ—64 resolution (no post-upscaling).
35
+ - **FID**: `torchmetrics.image.fid.FrechetInceptionDistance` (`feature=2048`,
36
+ InceptionV3 pool3; the metric resizes both sets to 299Γ—299 internally).
37
+ - **CLIP Score**: `torchmetrics.multimodal.CLIPScore` with
38
+ `openai/clip-vit-base-patch32` (the default), generated image vs the caption
39
+ that produced it, averaged over all 5,000.
40
+ - **Train/eval hygiene**: the model was trained on 20,000 pairs from rows
41
+ 10,000+ of the same stream, content-hash-checked to be disjoint from the
42
+ 5,000 eval images (0 overlaps found; the 30K rows are 30K unique images).
43
+
44
+ n=5000 is far past the singular-covariance regime that made v0's n=40 FID
45
+ directional-only (a 2048Γ—2048 covariance needs n ≫ 2048 to be non-singular;
46
+ 5,000 clears it, 40 did not).
47
+
48
+ ## Results
49
+
50
+ | Metric | v0 (n=40) | **v1 (n=5000)** | Context |
51
+ |---|---|---|---|
52
+ | FID ↓ | 566.84 | **439.46** | Competitive T2I models score ~10–30; v1's caption-conditioned color/layout statistics move it meaningfully toward the real-photo distribution, but 64Γ—64 color washes remain far outside it. |
53
+ | CLIP Score ↑ | 18.60 | **20.02** (cosine 0.2002) | ~15–20 is the no-semantic-correspondence floor; ~28–32 is competitive. v1 sits measurably above the floor β€” captions genuinely steer palette and layout β€” with 8.5Γ— fewer parameters than v0. |
54
+
55
+ Training loss context: the model's final MSE (0.0627) is 9% below the
56
+ "always output the mean COCO image" baseline (0.0691) β€” everything below that
57
+ baseline is caption-conditioning, since the mean image is the best possible
58
+ caption-blind predictor.
59
+
60
+ ## Leaderboard entry
61
+
62
+ ```js
63
+ {
64
+ name: "PixelModel-v1",
65
+ org: "bench-labs",
66
+ params: "0.024M",
67
+ fid: 439.46,
68
+ clipScore: 0.2002,
69
+ resolution: "64x64",
70
+ releaseDate: "2026-07-17",
71
+ links: {
72
+ card: "https://huggingface.co/bench-labs/pixelmodel-v1"
73
+ }
74
+ },
75
+ ```
76
+
77
+ *(`clipScore` given as the 0–1 cosine value per the template's field
78
+ description; the 0–100 form of the same number is 20.02.)*
eval/eval_results.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "n": 5000,
3
+ "native_resolution": "64x64",
4
+ "fid": 439.4560852050781,
5
+ "clip_score": 20.020164489746094
6
+ }
eval/run_eval.py ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ """Compatibility launcher for the root-level evaluation script."""
2
+
3
+ import runpy
4
+ from pathlib import Path
5
+
6
+ runpy.run_path(Path(__file__).resolve().parents[1] / "run_eval.py", run_name="__main__")