eval.md

# PixelModel — Evaluation

## ⚠️ Read this before the numbers

PixelModel is explicitly a toy (see `README.md`: *"It's a toy. It's not useful. But it works."*). It is a **~203K-parameter, 3-layer MLP** with no text encoder, no VAE, and no diffusion/transformer backbone — the entire "model" is 3 weight matrices encoded as pixel values in `model.png`. It was trained on 6 solid-color 32×32 swatches (red/green/blue/white/yellow/dark) using a deterministic char-sum prompt embedding.

It **cannot do general text-to-image generation**. Prompts outside its 6 training targets don't produce anything resembling the caption — they produce noisy, mostly-yellow-biased 32×32 blocks (see `gen_sample/` outputs below). The numbers in this file are real, measured results, not placeholders — and they reflect that reality: this model is far outside the distribution of real photos and does not respond meaningfully to prompt content.

This eval exists so the leaderboard entry reflects an honest, reproducible measurement rather than fabricated scores.

## Parameter count

No bias terms; only 3 weight matrices, all stored in `model.png`:

| Layer | Shape | Params |
|---|---|---|
| W1 (prompt→hidden) | 64×32 | 2,048 |
| W2 (hidden→hidden) | 64×64 | 4,096 |
| W3 (hidden→output) | 3072×64 | 196,608 |
| **Total** | | **202,752 (~0.2M)** |

No separate text encoder or VAE — the char-sum embedding and the MLP above are the entire pipeline. Well under the 1.5B ceiling.

## Method

- **Real captions/images**: 40 real (image, caption) pairs streamed from [`sayakpaul/coco-30-val-2014`](https://huggingface.co/datasets/sayakpaul/coco-30-val-2014) (a public 30K-sample subset of MS-COCO val2014), used as-is — no cherry-picking, first 40 rows of the stream.
- **Generation**: ran `model.forward()` (same code as `main.py`) on each of the 40 real captions to produce 40×32×32 RGB images.
- **FID**: standard InceptionV3 pool3 (2048-dim) features via `pytorch-fid`, comparing the 40 real COCO images against the 40 generated images.
- **CLIP Score**: `openai/clip-vit-base-patch32`, cosine similarity between each generated image and the caption used to generate it, scaled ×100, averaged over the 40 samples.

Full scripts are not bundled in this repo (they live outside it, in a throwaway eval env) — the method above is sufficient to reproduce: stream N real COCO pairs, run `main.py`-equivalent inference, feed both sets through InceptionV3 for FID and through CLIP for CLIP Score.

### Important caveat on sample size

Standard COCO FID evaluations use ~30,000 samples because FID's covariance-matrix estimate needs that scale to be stable (it's a 2048×2048 covariance matrix). **This run used n=40**, chosen to keep the eval fast and disk-light given local constraints. At n=40 the covariance matrix is singular (rank ≤ 40 ≪ 2048); `pytorch-fid` raised a `LinAlgWarning: Matrix is singular` when computing it. The FID number below is therefore a **rough, high-variance, directional signal**, not a publication-grade FID — treat it as "wildly out of distribution" rather than a precise score. CLIP Score is a per-sample metric (no covariance estimation) so it doesn't have this specific problem, but n=40 is still a small sample for a stable mean.

## Results

| Metric | Value | Context |
|---|---|---|
| FID (n=40) | **566.84** | Competitive T2I models score ~10–30 on full COCO FID; scores in the hundreds indicate the generated images are essentially out of the real-photo distribution entirely. |
| CLIP Score (n=40) | **18.60** (cosine 0.186) | Competitive T2I models score ~28–32; ~15–20 is roughly the floor for images with little-to-no real semantic correspondence to the prompt. |

## Leaderboard entry

```js
{
  name:        "PixelModel",
  org:         "bench-labs",
  params:      "0.2M",
  fid:         566.84,
  clipScore:   0.186,
  resolution:  "32x32",
  releaseDate: "2026-07-14",
  links: {
    card: "https://huggingface.co/bench-labs/pixelmodel"
  }
},
```

*(`clipScore` given as the 0–1 cosine value per the template's field description; the 0–100 CLIP Score above is the more commonly reported form of the same number.)*