Add real eval results and reproduction guide

Runs an actual COCO-captions FID/CLIP Score eval against the model's
real generated outputs rather than filling the leaderboard template
with placeholder numbers, and documents the model's actual capability
(toy MLP, not general text-to-image) alongside the measurements.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

EVAL_REPRODUCTION.md

@@ -0,0 +1,155 @@
+# Reproducing the eval.md numbers
+
+Step-by-step guide to reproduce the FID / CLIP Score numbers in `eval.md` from scratch. Everything below was run in a throwaway virtualenv outside this repo; none of it is required to just *use* PixelModel (see `README.md` for that) — it's only needed if you want to re-run or extend the evaluation.
+
+Uses Python 3.13 (CPU-only; no GPU required, but slower). Total download size is roughly 1.5–2GB (torch, torchvision, CLIP weights, Inception weights) — pick an install location with that much free space.
+
+## 1. Create an isolated venv
+
+```bash
+python3.13 -m venv eval-venv
+# Windows:
+eval-venv\Scripts\activate
+# macOS/Linux:
+source eval-venv/bin/activate
+```
+
+## 2. Install dependencies
+
+CPU-only torch keeps the install small (~130MB vs several GB for a CUDA build):
+
+```bash
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
+pip install transformers scipy huggingface_hub "datasets<3" pyarrow pytorch-fid
+```
+
+`datasets<3` is pinned because `pytorch-fid==0.3.0`'s dependency chain and the CLIP scoring code below weren't tested against `datasets` 3.x.
+
+## 3. Fetch a real COCO caption/image sample
+
+This streams only the first N rows from a public 30K-pair COCO val2014 dataset — it does **not** download the full ~5GB dataset.
+
+```python
+# fetch_coco_sample.py
+import os, json, time
+from datasets import load_dataset
+
+N = 40
+OUT_DIR = "coco_sample"
+os.makedirs(OUT_DIR, exist_ok=True)
+
+ds = load_dataset("sayakpaul/coco-30-val-2014", split="train", streaming=True)
+
+meta = []
+for i, row in enumerate(ds):
+    if i >= N:
+        break
+    img = row["image"]
+    caption = row["caption"]
+    fname = f"real_{i:03d}.jpg"
+    img.convert("RGB").save(os.path.join(OUT_DIR, fname), format="JPEG", quality=90)
+    meta.append({"file": fname, "caption": caption})
+
+with open(os.path.join(OUT_DIR, "captions.json"), "w") as f:
+    json.dump(meta, f, indent=2)
+```
+
+Run it: `python fetch_coco_sample.py` — takes ~10 seconds, ~3.5MB on disk. Raise `N` for a less noisy (but slower, and eventually much larger) eval — real COCO FID evals typically use N=30,000.
+
+## 4. Generate PixelModel outputs for those captions
+
+Run from inside this repo (needs `model.py` and `model.png` on the path):
+
+```python
+# generate_outputs.py
+import os, json
+import numpy as np
+from PIL import Image
+import torch
+from model import load_model, forward  # from this repo
+
+META_PATH = "coco_sample/captions.json"
+OUT_DIR = "gen_sample"
+os.makedirs(OUT_DIR, exist_ok=True)
+
+with open(META_PATH) as f:
+    meta = json.load(f)
+
+pixels = load_model("model.png")
+
+for i, row in enumerate(meta):
+    with torch.no_grad():
+        result = forward(pixels, row["caption"])
+    arr = (result.numpy() * 255).clip(0, 255).astype(np.uint8)
+    Image.fromarray(arr, mode="RGB").save(os.path.join(OUT_DIR, f"gen_{i:03d}.png"))
+```
+
+## 5. Compute FID
+
+`pytorch-fid==0.3.0` calls `scipy.linalg.sqrtm(..., disp=False)`, which recent scipy versions (1.14+) no longer accept — this reimplements the Fréchet distance calculation without that removed kwarg instead of pinning an old scipy:
+
+```python
+# compute_fid.py
+import numpy as np
+from scipy import linalg
+from pytorch_fid.fid_score import compute_statistics_of_path
+from pytorch_fid.inception import InceptionV3
+
+def calculate_frechet_distance(mu1, sigma1, mu2, sigma2):
+    diff = mu1 - mu2
+    covmean = linalg.sqrtm(sigma1.dot(sigma2))
+    if np.iscomplexobj(covmean):
+        covmean = covmean.real
+    return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * np.trace(covmean)
+
+dims = 2048
+model = InceptionV3([InceptionV3.BLOCK_INDEX_BY_DIM[dims]]).to("cpu")
+
+# NOTE: real images are JPEGs of varying resolution; pytorch-fid's default
+# dataloader batches images without resizing first, which crashes on a batch
+# of mismatched sizes — use batch_size=1 to sidestep that.
+m1, s1 = compute_statistics_of_path("coco_sample", model, 1, dims, "cpu", 0)
+m2, s2 = compute_statistics_of_path("gen_sample", model, 1, dims, "cpu", 0)
+
+fid = calculate_frechet_distance(m1, s1, m2, s2)
+print(f"FID: {fid:.4f}")
+```
+
+Expect a `LinAlgWarning: Matrix is singular` at low sample counts (n < 2048) — this is expected, not a bug; see the caveat in `eval.md`.
+
+## 6. Compute CLIP Score
+
+```python
+# compute_clip_score.py
+import json
+import torch
+from PIL import Image
+from transformers import CLIPModel, CLIPProcessor
+
+with open("coco_sample/captions.json") as f:
+    meta = json.load(f)
+
+model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
+processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
+model.eval()
+
+scores = []
+for i, row in enumerate(meta):
+    img = Image.open(f"gen_sample/gen_{i:03d}.png").convert("RGB")
+    inputs = processor(text=[row["caption"]], images=img, return_tensors="pt", padding=True, truncation=True)
+    with torch.no_grad():
+        out = model(**inputs)
+    img_emb = out.image_embeds / out.image_embeds.norm(dim=-1, keepdim=True)
+    txt_emb = out.text_embeds / out.text_embeds.norm(dim=-1, keepdim=True)
+    cos_sim = (img_emb * txt_emb).sum(dim=-1).item()
+    scores.append(max(0.0, 100 * cos_sim))
+
+print(f"Mean CLIP Score (n={len(scores)}): {sum(scores)/len(scores):.4f}")
+```
+
+First run downloads `openai/clip-vit-base-patch32` (~600MB) from the Hugging Face Hub.
+
+## Notes / gotchas hit while building this
+
+- **Disk location matters.** All of the above (venv, pip cache, HF cache, downloaded datasets) should point at a drive with several GB free — `torch`+`torchvision`+`transformers`+CLIP+Inception weights add up to ~1.5–2GB. Set `PIP_CACHE_DIR`, `HF_HOME`, and `TORCH_HOME` env vars if your default drive is tight on space.
+- **Sample size is the biggest caveat.** n=40 is fast and cheap but gives a noisy FID (singular covariance matrix) and a noisy CLIP Score mean. Scaling toward the standard n=30,000 COCO FID protocol means re-running steps 3–6 with a bigger `N`, which will take proportionally longer and use proportionally more disk/bandwidth.

eval.md

@@ -0,0 +1,61 @@
+# PixelModel — Evaluation
+
+## ⚠️ Read this before the numbers
+
+PixelModel is explicitly a toy (see `README.md`: *"It's a toy. It's not useful. But it works."*). It is a **~203K-parameter, 3-layer MLP** with no text encoder, no VAE, and no diffusion/transformer backbone — the entire "model" is 3 weight matrices encoded as pixel values in `model.png`. It was trained on 6 solid-color 32×32 swatches (red/green/blue/white/yellow/dark) using a deterministic char-sum prompt embedding.
+
+It **cannot do general text-to-image generation**. Prompts outside its 6 training targets don't produce anything resembling the caption — they produce noisy, mostly-yellow-biased 32×32 blocks (see `gen_sample/` outputs below). The numbers in this file are real, measured results, not placeholders — and they reflect that reality: this model is far outside the distribution of real photos and does not respond meaningfully to prompt content.
+
+This eval exists so the leaderboard entry reflects an honest, reproducible measurement rather than fabricated scores.
+
+## Parameter count
+
+No bias terms; only 3 weight matrices, all stored in `model.png`:
+
+| Layer | Shape | Params |
+|---|---|---|
+| W1 (prompt→hidden) | 64×32 | 2,048 |
+| W2 (hidden→hidden) | 64×64 | 4,096 |
+| W3 (hidden→output) | 3072×64 | 196,608 |
+| **Total** | | **202,752 (~0.2M)** |
+
+No separate text encoder or VAE — the char-sum embedding and the MLP above are the entire pipeline. Well under the 1.5B ceiling.
+
+## Method
+
+- **Real captions/images**: 40 real (image, caption) pairs streamed from [`sayakpaul/coco-30-val-2014`](https://huggingface.co/datasets/sayakpaul/coco-30-val-2014) (a public 30K-sample subset of MS-COCO val2014), used as-is — no cherry-picking, first 40 rows of the stream.
+- **Generation**: ran `model.forward()` (same code as `main.py`) on each of the 40 real captions to produce 40×32×32 RGB images.
+- **FID**: standard InceptionV3 pool3 (2048-dim) features via `pytorch-fid`, comparing the 40 real COCO images against the 40 generated images.
+- **CLIP Score**: `openai/clip-vit-base-patch32`, cosine similarity between each generated image and the caption used to generate it, scaled ×100, averaged over the 40 samples.
+
+Full scripts are not bundled in this repo (they live outside it, in a throwaway eval env) — the method above is sufficient to reproduce: stream N real COCO pairs, run `main.py`-equivalent inference, feed both sets through InceptionV3 for FID and through CLIP for CLIP Score.
+
+### Important caveat on sample size
+
+Standard COCO FID evaluations use ~30,000 samples because FID's covariance-matrix estimate needs that scale to be stable (it's a 2048×2048 covariance matrix). **This run used n=40**, chosen to keep the eval fast and disk-light given local constraints. At n=40 the covariance matrix is singular (rank ≤ 40 ≪ 2048); `pytorch-fid` raised a `LinAlgWarning: Matrix is singular` when computing it. The FID number below is therefore a **rough, high-variance, directional signal**, not a publication-grade FID — treat it as "wildly out of distribution" rather than a precise score. CLIP Score is a per-sample metric (no covariance estimation) so it doesn't have this specific problem, but n=40 is still a small sample for a stable mean.
+
+## Results
+
+| Metric | Value | Context |
+|---|---|---|
+| FID (n=40) | **566.84** | Competitive T2I models score ~10–30 on full COCO FID; scores in the hundreds indicate the generated images are essentially out of the real-photo distribution entirely. |
+| CLIP Score (n=40) | **18.60** (cosine 0.186) | Competitive T2I models score ~28–32; ~15–20 is roughly the floor for images with little-to-no real semantic correspondence to the prompt. |
+
+## Leaderboard entry
+
+```js
+{
+  name:        "PixelModel",
+  org:         "bench-labs",
+  params:      "0.2M",
+  fid:         566.84,
+  clipScore:   0.186,
+  resolution:  "32x32",
+  releaseDate: "2026-07-14",
+  links: {
+    card: "https://huggingface.co/bench-labs/pixelmodel"
+  }
+},
+```
+
+*(`clipScore` given as the 0–1 cosine value per the template's field description; the 0–100 CLIP Score above is the more commonly reported form of the same number.)*