blog / posts /Pm5.html
TobiasLogic's picture
Add PixelModel v5 blog post
6047c78 verified
Raw History Blame Contribute Delete
20.8 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>PixelModel v5: The Bottleneck Was Data, Not Architecture</title>
<link rel="stylesheet" href="../style.css">
</head>
<body>
<script type="module" src="../summarize.js" defer></script>
<script src="../script.js" defer></script>
<div class="background"></div>
<main class="article">
<a class="back" href="../index.html">
← Back
</a>
<div class="meta">
July 2026
</div>
<h1>
PixelModel v5: The Bottleneck Was Data, Not Architecture
</h1>
<p>
<a href="./Pm4.html">PixelModel v4</a> gave the series real text understanding: a Transformer text
encoder, spatial cross-attention, native 256×256 output. It was a genuine architectural jump, and it
still scored FID 39.54. v5 changes none of that. Same DiT, same width, same depth, same VAE, same
text encoder, same rectified-flow objective, same 40,013,980 trainable parameters. The only thing
that moved is the data, and that alone was worth an FID drop to 34.06.
</p>
<p>
<img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/examples.png" alt="Nine 256px images generated by PixelModel v5" style="width: 100%; max-width: 860px;">
</p>
<h2>
v4 was starving, not undersized
</h2>
<p>
v4 trained on MS-COCO, about 83,000 images. At batch 256 for 80,000 steps that is roughly 247 passes
over the same photographs. A model that sees the same 83,000 images 247 times is not learning what
the world looks like, it is memorising a small album.
</p>
<p>
We looked at architecture changes first, the way you always do. Linear attention, deeper stacks, a
higher compression autoencoder, representation alignment through REPA. All real techniques, and none
of them address the actual problem, because the bottleneck was never the shape of the network.
</p>
<p>
<img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/pixelmodel-v5-data.png" alt="Training set size, v4 at 83k images versus v5 at 2.99M" style="width: 100%; max-width: 780px;">
</p>
<p>
v5 trains on 2,986,571 image and caption pairs, about 10 epochs instead of 247.
</p>
<h2>
The captions matter as much as the count
</h2>
<p>
The images come from CC12M. Raw CC12M alt text is close to unusable for this, because it is scraped
SEO copy — a typical caption reads like "This is the best recipe I have ever tried for Cuban bread.
Cuban Recipes, Bread Recipes, Cooking Recipes, Pan Cubano". None of that tells a diffusion model what
is actually in the frame.
</p>
<p>
v5 uses a recaptioned version where a vision model rewrote every caption in COCO style:
</p>
<blockquote>
A purple-lit subway train sits at a station platform.<br><br>
An old red tractor with gold fenders and red wheels stands in a snowy field.<br><br>
A compact kitchenette in a studio apartment features a stainless steel refrigerator, a microwave,
and a sink with a glass faucet.
</blockquote>
<p>
Those describe what is in the frame, which is what a text-to-image model actually needs. They also
match the caption distribution that FID and CLIP score are computed against, so the training signal
and the evaluation are speaking the same language.
</p>
<h2>
Training
</h2>
<p>
<img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/pixelmodel-v5-loss.png" alt="Training and held-out loss over 120,000 steps" style="width: 100%; max-width: 860px;">
</p>
<p>
120,000 steps at batch 256 on a single A100, about 7 hours. AdamW at 2e-4 with betas (0.9, 0.99), no
weight decay, gradient clipping at 1.0, cosine decay to a tenth of peak after a 1,000 step warmup, EMA
at 0.9999 starting after warmup, and 10% caption dropout for classifier-free guidance. It held 4.6
steps per second at 44.2 GiB the whole way.
</p>
<p>
The held-out curve climbs for the first 10,000 steps or so, which looks alarming and is not. EMA at
0.9999 has a time constant of roughly 10,000 steps, so early on those weights are still mostly
initialisation. The number only becomes meaningful once EMA has caught up, and from there it falls
without interruption.
</p>
<p>
<img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/pixelmodel-v5-val.png" alt="Held-out loss falling monotonically with no turn-up" style="width: 100%; max-width: 860px;">
</p>
<p>
<strong>v4 had no validation split at all</strong>, so it shipped whichever weights the last step
happened to produce. v5 holds out 4,096 pairs, evaluates every 2,000 steps, and writes the best
checkpoint to a separate file. Across the whole run the held-out loss improved at every single check
and never once turned upward — the clearest evidence that the data change worked, not just a
side effect of training longer. Our sibling model
<a href="https://huggingface.co/bench-labs/VoxelModel-v1">VoxelModel v1</a>, trained on 28,415 meshes
for 811 epochs, peaked at step 64,000 and got 3.08% worse by the end. Same recipe, same team,
different data regime, opposite outcome.
</p>
<h2>
Results
</h2>
<p>
Measured on 5,000 MS-COCO val2014 images at 256 center crop, using <code>torchmetrics</code> FID at
2048 features and <code>torchmetrics</code> CLIPScore with <code>openai/clip-vit-base-patch32</code>.
This is v4's protocol, run again rather than assumed.
</p>
<pre><code> v4 v5 (cfg 5.0) v5 (cfg 3.0) v5 (cfg 6.0)
FID ↓ 39.54 34.06 35.03 35.18
CLIP score ↑ 28.04 29.27 28.59 29.28
training images 83,000 2,986,571 2,986,571 2,986,571
epochs over data 247 ~10 ~10 ~10
trainable params 40,013,980 40,013,980 40,013,980 40,013,980
real photographs, CLIP — 30.05 (reference ceiling, all runs)</code></pre>
<p>
FID is 13.9% lower and CLIP score is 1.23 points higher, at an identical parameter count. Nothing in
the network changed between v4 and v5, so the entire difference is the training data. Feeding real
COCO photographs and their own captions through the same CLIPScore harness gives 30.05, which puts v5
at cfg 5.0 within 0.78 of the ceiling this metric can award.
</p>
<p>
Guidance scale was swept rather than assumed. FID bottoms out at cfg 5.0 and CLIP keeps climbing
slightly to cfg 6.0, the usual tension between the two. Sample count matters more than people expect
here too — the same cfg 5.0 checkpoint scores 74.35 on 1,000 images and 34.06 on 5,000, so FID numbers
are only comparable when the sample count matches.
</p>
<h2>
Specification
</h2>
<pre><code>trainable parameters 40,013,980
frozen VAE (sd-vae-ft-mse) 83,653,863
frozen text encoder (CLIP ViT-B/32) 63,165,952
combined total 186,833,795
architecture DiT, dim 384, depth 12, heads 6, patch 2
tokens 256, over a 32x32x4 latent
objective rectified flow, target x1 - x0, logit-normal timesteps
conditioning cross attention on 40 CLIP text tokens, plus adaLN-zero
resolution 256x256
sampling 50 steps, cfg 5.0 for best FID, cfg 6.0 for best CLIP</code></pre>
<p>
One correction worth flagging: v4's card reported 161.5M total parameters, but that count left out
the CLIP text encoder's 49,408 × 512 token embedding table — another 25.3M parameters that the encoder
cannot run without. The corrected total for both v4 and v5 is 186,833,795.
</p>
<h2>
What it does well and what it does not
</h2>
<p>
After a few hundred samples the boundary is consistent. It is good at material and light — food,
landscapes, skies, water, snow, foliage, stone, interiors, depth of field. A bowl of ramen, a wet
cobblestone street, a wheat field under storm light, a library of wooden shelves. These come back
looking like photographs.
</p>
<p>
<img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/hero.png" alt="Cobblestone street, library, wheat field, and snowy mountain generated by PixelModel v5" style="width: 100%; max-width: 640px;">
</p>
<p>
It is bad at structure. Faces and animal heads deform. Limbs merge. Wings on aircraft are stubby or
fused to the fuselage. Any text it renders, on a shop sign or an airliner livery, is convincing
looking gibberish. Prompts asking for one isolated object in an empty scene often return the scene
without the object — "a lighthouse on a cliff at sunset" gives a very nice sunset over a cliff and no
lighthouse.
</p>
<p>
Frequency in the training data matters more than complexity. Aircraft are structurally complicated
and come out fine because CC12M is full of them. Hot air balloons are simpler and fail, because they
are rare.
</p>
<p>
That failure profile is what a 40M model looks like once data is no longer the constraint. It has
learned appearance thoroughly and topology poorly, and topology is what capacity buys. v6 is where
that gets spent.
</p>
<h2>
Try it
</h2>
<pre><code>git clone https://huggingface.co/bench-labs/PixelModel-v5
python main.py "a bowl of ramen with a soft boiled egg" --out ramen.png --cfg 6.0 --steps 50</code></pre>
<p>
As with every model in this line, the weights are also stored as a PNG. <code>model.png</code> is not
a picture of the network, it is the network, with each parameter packed across the red and green
channels of one pixel. Explore the
<a href="https://huggingface.co/bench-labs/PixelModel-v5">PixelModel v5 repository</a> or the
<a href="https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard">text-to-image leaderboard</a>.
</p>
<blockquote>
v4 proved the network needed to understand text. v5 proves that once it does, the next several
points of FID are bought with data, not layers.
</blockquote>
</main>
<script>
/* ==========================================================
SYNTAX HIGHLIGHTING
========================================================== */
const PY_KW = new Set([
'def','class','return','if','elif','else','for','while','import','from','as',
'try','except','finally','with','yield','lambda','pass','break','continue',
'raise','and','or','not','in','is','global','nonlocal','assert','del',
'async','await'
]);
const JS_KW = new Set([
'const','let','var','function','return','if','else','for','while','do',
'switch','case','break','continue','new','class','extends','import','export',
'from','default','try','catch','finally','throw','typeof','instanceof',
'in','of','async','await','yield','void','delete','with'
]);
const PY_BI = new Set([
'print','len','range','int','str','float','list','dict','set','tuple','bool',
'type','isinstance','input','open','map','filter','zip','enumerate','sorted',
'reversed','sum','min','max','abs','round','any','all','super','property',
'staticmethod','classmethod','True','False','None','Exception','ValueError',
'TypeError','KeyError','IndexError','AttributeError','NotImplementedError',
'RuntimeError','StopIteration','FileNotFoundError','object'
]);
const JS_BI = new Set([
'console','Math','Array','Object','String','Number','Boolean','Date','RegExp',
'Error','TypeError','RangeError','SyntaxError','JSON','Promise','Map','Set',
'Symbol','Proxy','Reflect','parseInt','parseFloat','isNaN','isFinite',
'undefined','null','true','false','NaN','Infinity'
]);
function escapeHtml(s) {
return s.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
}
function detectLang(code) {
let py = 0, js = 0;
const pyP = /\b(def |class |import |from \w+ import|elif |print\(|True|False|None|self[\.\[])\b/g;
const jsP = /\b(const |let |var |function |=>|console\.|undefined|null|async |await )\b/g;
let m;
while ((m = pyP.exec(code)) !== null) py++;
while ((m = jsP.exec(code)) !== null) js++;
if (py > js) return 'python';
if (js > 0) return 'javascript';
return 'text';
}
function highlight(code) {
const lang = detectLang(code);
const kw = lang === 'python' ? PY_KW : lang === 'javascript' ? JS_KW : new Set();
const bi = lang === 'python' ? PY_BI : lang === 'javascript' ? JS_BI : new Set();
const tokens = [];
let i = 0;
const len = code.length;
while (i < len) {
if (lang === 'python') {
const t3 = code.slice(i, i + 3);
if (t3 === '"""' || t3 === "'''") {
const end = code.indexOf(t3, i + 3);
const span = end !== -1 ? code.slice(i, end + 3) : code.slice(i);
tokens.push({ type: 'string', text: span });
i += span.length;
continue;
}
}
{
const pm = code.slice(i).match(/^[frbFRB]{0,3}/);
if (pm) {
const pLen = pm[0].length;
const nc = code[i + pLen];
if (nc === '"' || nc === "'") {
const q = nc;
const isF = pm[0].toLowerCase().includes('f');
let depth = 0, k = i + pLen + 1;
let found = false;
while (k < len) {
const ch = code[k];
if (ch === '\\') { k += 2; continue; }
if (isF) {
if (ch === '{') { depth++; k++; continue; }
if (ch === '}') { depth = Math.max(0, depth - 1); k++; continue; }
}
if (ch === q && depth === 0) { found = true; break; }
k++;
}
if (found) {
tokens.push({ type: 'string', text: code.slice(i, k + 1) });
i = k + 1;
continue;
}
}
}
}
if (lang === 'python' && code[i] === '#') {
let end = code.indexOf('\n', i);
if (end === -1) end = len;
tokens.push({ type: 'comment', text: code.slice(i, end) });
i = end;
continue;
}
if (lang === 'javascript' && code.slice(i, i + 2) === '//') {
let end = code.indexOf('\n', i);
if (end === -1) end = len;
tokens.push({ type: 'comment', text: code.slice(i, end) });
i = end;
continue;
}
if (lang === 'javascript' && code.slice(i, i + 2) === '/*') {
let end = code.indexOf('*/', i + 2);
if (end === -1) end = len; else end += 2;
tokens.push({ type: 'comment', text: code.slice(i, end) });
i = end;
continue;
}
{
const m = code.slice(i).match(
/^(0x[0-9a-fA-F_]+|0b[01_]+|0o[0-7_]+|\d[\d_]*\.?\d*(?:e[+-]?\d+)?)/
);
if (m) {
tokens.push({ type: 'number', text: m[0] });
i += m[0].length;
continue;
}
}
if (lang === 'python' && code[i] === '@') {
const m = code.slice(i).match(/^@[\w.]+/);
if (m) {
tokens.push({ type: 'decorator', text: m[0] });
i += m[0].length;
continue;
}
}
{
const m = code.slice(i).match(/^\w+/);
if (m) {
const w = m[0];
let type = 'plain';
if (kw.has(w)) type = 'keyword';
else if (bi.has(w)) type = 'builtin';
else if (w === 'self' || w === 'cls' || w === 'this') type = 'self';
else {
let j = i + w.length;
while (j < len && code[j] === ' ') j++;
if (code[j] === '(') type = 'function';
}
tokens.push({ type, text: w });
i += w.length;
continue;
}
}
{
const m = code.slice(i).match(/^[+\-*/%=<>!&|^~:]+/);
if (m) {
tokens.push({ type: 'operator', text: m[0] });
i += m[0].length;
continue;
}
}
if ('(){}[],;.'.includes(code[i])) {
tokens.push({ type: 'punctuation', text: code[i] });
i++;
continue;
}
{
const m = code.slice(i).match(/^\s+/);
if (m) {
tokens.push({ type: 'plain', text: m[0] });
i += m[0].length;
} else {
tokens.push({ type: 'plain', text: code[i] });
i++;
}
}
}
return tokens.map(t => {
const esc = escapeHtml(t.text);
return t.type === 'plain' ? esc : `<span class="tok-${t.type}">${esc}</span>`;
}).join('');
}
document.querySelectorAll('pre code').forEach(block => {
block.innerHTML = highlight(block.textContent);
});
/* ==========================================================
LIGHTBOX
========================================================== */
class Lightbox {
constructor() {
this.images = [];
this.currentIndex = 0;
this.isOpen = false;
this._build();
this._bindImages();
this._bindKeys();
this._bindSwipe();
}
_build() {
this.el = document.createElement('div');
this.el.className = 'lightbox-overlay';
this.el.setAttribute('role', 'dialog');
this.el.setAttribute('aria-modal', 'true');
this.el.setAttribute('aria-label', 'Image viewer');
this.el.innerHTML = `
<button class="lightbox-btn lightbox-close" aria-label="Close">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2.5" stroke-linecap="round">
<line x1="18" y1="6" x2="6" y2="18"/>
<line x1="6" y1="6" x2="18" y2="18"/>
</svg>
</button>
<button class="lightbox-btn lightbox-arrow lightbox-prev" aria-label="Previous image">
<svg width="22" height="22" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round">
<polyline points="15 18 9 12 15 6"/>
</svg>
</button>
<button class="lightbox-btn lightbox-arrow lightbox-next" aria-label="Next image">
<svg width="22" height="22" viewBox="0 0 24 24" fill="none"
stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round">
<polyline points="9 6 15 12 9 18"/>
</svg>
</button>
<img class="lightbox-img" src="" alt="">
<div class="lightbox-counter"></div>
`;
document.body.appendChild(this.el);
this.imgEl = this.el.querySelector('.lightbox-img');
this.counterEl = this.el.querySelector('.lightbox-counter');
this.prevBtn = this.el.querySelector('.lightbox-prev');
this.nextBtn = this.el.querySelector('.lightbox-next');
this.closeBtn = this.el.querySelector('.lightbox-close');
this.closeBtn.addEventListener('click', () => this.close());
this.prevBtn.addEventListener('click', (e) => { e.stopPropagation(); this.prev(); });
this.nextBtn.addEventListener('click', (e) => { e.stopPropagation(); this.next(); });
this.el.addEventListener('click', (e) => {
if (e.target === this.el) this.close();
});
}
_bindImages() {
this.images = Array.from(document.querySelectorAll('.article img'));
this.images.forEach((img, idx) => {
img.addEventListener('click', () => this.open(idx));
});
}
_bindKeys() {
document.addEventListener('keydown', (e) => {
if (!this.isOpen) return;
if (e.key === 'Escape') this.close();
if (e.key === 'ArrowLeft') this.prev();
if (e.key === 'ArrowRight') this.next();
});
}
_bindSwipe() {
let startX = 0;
this.el.addEventListener('touchstart', (e) => {
startX = e.changedTouches[0].clientX;
}, { passive: true });
this.el.addEventListener('touchend', (e) => {
const dx = startX - e.changedTouches[0].clientX;
if (Math.abs(dx) > 60) {
dx > 0 ? this.next() : this.prev();
}
}, { passive: true });
}
open(idx) {
this.currentIndex = idx;
this.isOpen = true;
this._update();
requestAnimationFrame(() => {
this.el.classList.add('active');
});
document.body.style.overflow = 'hidden';
this.closeBtn.focus();
}
close() {
this.isOpen = false;
this.el.classList.remove('active');
document.body.style.overflow = '';
this.images[this.currentIndex]?.focus();
}
prev() {
this.currentIndex = (this.currentIndex - 1 + this.images.length) % this.images.length;
this._update();
}
next() {
this.currentIndex = (this.currentIndex + 1) % this.images.length;
this._update();
}
_update() {
const srcImg = this.images[this.currentIndex];
this.imgEl.style.opacity = '0';
this.imgEl.style.transform = 'scale(0.96)';
setTimeout(() => {
this.imgEl.src = srcImg.src;
this.imgEl.alt = srcImg.alt;
this.imgEl.removeAttribute('loading');
this.imgEl.style.opacity = '1';
this.imgEl.style.transform = 'scale(1)';
}, 120);
const total = this.images.length;
if (total > 1) {
this.counterEl.textContent = `${this.currentIndex + 1} / ${total}`;
this.prevBtn.style.display = '';
this.nextBtn.style.display = '';
} else {
this.counterEl.textContent = '';
this.prevBtn.style.display = 'none';
this.nextBtn.style.display = 'none';
}
}
}
new Lightbox();
</script>
</body>
</html>