PixelModel — Evaluation
⚠️ Read this before the numbers
PixelModel is explicitly a toy (see README.md: "It's a toy. It's not useful. But it works."). It is a ~203K-parameter, 3-layer MLP with no text encoder, no VAE, and no diffusion/transformer backbone — the entire "model" is 3 weight matrices encoded as pixel values in model.png. It was trained on 6 solid-color 32×32 swatches (red/green/blue/white/yellow/dark) using a deterministic char-sum prompt embedding.
It cannot do general text-to-image generation. Prompts outside its 6 training targets don't produce anything resembling the caption — they produce noisy, mostly-yellow-biased 32×32 blocks (see gen_sample/ outputs below). The numbers in this file are real, measured results, not placeholders — and they reflect that reality: this model is far outside the distribution of real photos and does not respond meaningfully to prompt content.
This eval exists so the leaderboard entry reflects an honest, reproducible measurement rather than fabricated scores.
Parameter count
No bias terms; only 3 weight matrices, all stored in model.png:
| Layer | Shape | Params |
|---|---|---|
| W1 (prompt→hidden) | 64×32 | 2,048 |
| W2 (hidden→hidden) | 64×64 | 4,096 |
| W3 (hidden→output) | 3072×64 | 196,608 |
| Total | 202,752 (~0.2M) |
No separate text encoder or VAE — the char-sum embedding and the MLP above are the entire pipeline. Well under the 1.5B ceiling.
Method
- Real captions/images: 40 real (image, caption) pairs streamed from
sayakpaul/coco-30-val-2014(a public 30K-sample subset of MS-COCO val2014), used as-is — no cherry-picking, first 40 rows of the stream. - Generation: ran
model.forward()(same code asmain.py) on each of the 40 real captions to produce 40×32×32 RGB images. - FID: standard InceptionV3 pool3 (2048-dim) features via
pytorch-fid, comparing the 40 real COCO images against the 40 generated images. - CLIP Score:
openai/clip-vit-base-patch32, cosine similarity between each generated image and the caption used to generate it, scaled ×100, averaged over the 40 samples.
Full scripts are not bundled in this repo (they live outside it, in a throwaway eval env) — the method above is sufficient to reproduce: stream N real COCO pairs, run main.py-equivalent inference, feed both sets through InceptionV3 for FID and through CLIP for CLIP Score.
Important caveat on sample size
Standard COCO FID evaluations use ~30,000 samples because FID's covariance-matrix estimate needs that scale to be stable (it's a 2048×2048 covariance matrix). This run used n=40, chosen to keep the eval fast and disk-light given local constraints. At n=40 the covariance matrix is singular (rank ≤ 40 ≪ 2048); pytorch-fid raised a LinAlgWarning: Matrix is singular when computing it. The FID number below is therefore a rough, high-variance, directional signal, not a publication-grade FID — treat it as "wildly out of distribution" rather than a precise score. CLIP Score is a per-sample metric (no covariance estimation) so it doesn't have this specific problem, but n=40 is still a small sample for a stable mean.
Results
| Metric | Value | Context |
|---|---|---|
| FID (n=40) | 566.84 | Competitive T2I models score ~10–30 on full COCO FID; scores in the hundreds indicate the generated images are essentially out of the real-photo distribution entirely. |
| CLIP Score (n=40) | 18.60 (cosine 0.186) | Competitive T2I models score ~28–32; ~15–20 is roughly the floor for images with little-to-no real semantic correspondence to the prompt. |
Leaderboard entry
{
name: "PixelModel",
org: "bench-labs",
params: "0.2M",
fid: 566.84,
clipScore: 0.186,
resolution: "32x32",
releaseDate: "2026-07-14",
links: {
card: "https://huggingface.co/bench-labs/pixelmodel"
}
},
(clipScore given as the 0–1 cosine value per the template's field description; the 0–100 CLIP Score above is the more commonly reported form of the same number.)