Spaces:
Running
Running
Download posts/Pm5.html from bench-labs/blog: direct link, hf CLI and curl.
- Browser
- Download file 20.8 kB
-
https://huggingface.co/spaces/bench-labs/blog/resolve/main/posts/Pm5.html
- Command line
-
hf download hf://spaces/bench-labs/blog/posts/Pm5.html
-
curl -L -o Pm5.html https://huggingface.co/spaces/bench-labs/blog/resolve/main/posts/Pm5.html
20.8 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>PixelModel v5: The Bottleneck Was Data, Not Architecture</title> | |
| <link rel="stylesheet" href="../style.css"> | |
| </head> | |
| <body> | |
| <script type="module" src="../summarize.js" defer></script> | |
| <script src="../script.js" defer></script> | |
| <div class="background"></div> | |
| <main class="article"> | |
| <a class="back" href="../index.html"> | |
| ← Back | |
| </a> | |
| <div class="meta"> | |
| July 2026 | |
| </div> | |
| <h1> | |
| PixelModel v5: The Bottleneck Was Data, Not Architecture | |
| </h1> | |
| <p> | |
| <a href="./Pm4.html">PixelModel v4</a> gave the series real text understanding: a Transformer text | |
| encoder, spatial cross-attention, native 256×256 output. It was a genuine architectural jump, and it | |
| still scored FID 39.54. v5 changes none of that. Same DiT, same width, same depth, same VAE, same | |
| text encoder, same rectified-flow objective, same 40,013,980 trainable parameters. The only thing | |
| that moved is the data, and that alone was worth an FID drop to 34.06. | |
| </p> | |
| <p> | |
| <img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/examples.png" alt="Nine 256px images generated by PixelModel v5" style="width: 100%; max-width: 860px;"> | |
| </p> | |
| <h2> | |
| v4 was starving, not undersized | |
| </h2> | |
| <p> | |
| v4 trained on MS-COCO, about 83,000 images. At batch 256 for 80,000 steps that is roughly 247 passes | |
| over the same photographs. A model that sees the same 83,000 images 247 times is not learning what | |
| the world looks like, it is memorising a small album. | |
| </p> | |
| <p> | |
| We looked at architecture changes first, the way you always do. Linear attention, deeper stacks, a | |
| higher compression autoencoder, representation alignment through REPA. All real techniques, and none | |
| of them address the actual problem, because the bottleneck was never the shape of the network. | |
| </p> | |
| <p> | |
| <img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/pixelmodel-v5-data.png" alt="Training set size, v4 at 83k images versus v5 at 2.99M" style="width: 100%; max-width: 780px;"> | |
| </p> | |
| <p> | |
| v5 trains on 2,986,571 image and caption pairs, about 10 epochs instead of 247. | |
| </p> | |
| <h2> | |
| The captions matter as much as the count | |
| </h2> | |
| <p> | |
| The images come from CC12M. Raw CC12M alt text is close to unusable for this, because it is scraped | |
| SEO copy — a typical caption reads like "This is the best recipe I have ever tried for Cuban bread. | |
| Cuban Recipes, Bread Recipes, Cooking Recipes, Pan Cubano". None of that tells a diffusion model what | |
| is actually in the frame. | |
| </p> | |
| <p> | |
| v5 uses a recaptioned version where a vision model rewrote every caption in COCO style: | |
| </p> | |
| <blockquote> | |
| A purple-lit subway train sits at a station platform.<br><br> | |
| An old red tractor with gold fenders and red wheels stands in a snowy field.<br><br> | |
| A compact kitchenette in a studio apartment features a stainless steel refrigerator, a microwave, | |
| and a sink with a glass faucet. | |
| </blockquote> | |
| <p> | |
| Those describe what is in the frame, which is what a text-to-image model actually needs. They also | |
| match the caption distribution that FID and CLIP score are computed against, so the training signal | |
| and the evaluation are speaking the same language. | |
| </p> | |
| <h2> | |
| Training | |
| </h2> | |
| <p> | |
| <img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/pixelmodel-v5-loss.png" alt="Training and held-out loss over 120,000 steps" style="width: 100%; max-width: 860px;"> | |
| </p> | |
| <p> | |
| 120,000 steps at batch 256 on a single A100, about 7 hours. AdamW at 2e-4 with betas (0.9, 0.99), no | |
| weight decay, gradient clipping at 1.0, cosine decay to a tenth of peak after a 1,000 step warmup, EMA | |
| at 0.9999 starting after warmup, and 10% caption dropout for classifier-free guidance. It held 4.6 | |
| steps per second at 44.2 GiB the whole way. | |
| </p> | |
| <p> | |
| The held-out curve climbs for the first 10,000 steps or so, which looks alarming and is not. EMA at | |
| 0.9999 has a time constant of roughly 10,000 steps, so early on those weights are still mostly | |
| initialisation. The number only becomes meaningful once EMA has caught up, and from there it falls | |
| without interruption. | |
| </p> | |
| <p> | |
| <img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/pixelmodel-v5-val.png" alt="Held-out loss falling monotonically with no turn-up" style="width: 100%; max-width: 860px;"> | |
| </p> | |
| <p> | |
| <strong>v4 had no validation split at all</strong>, so it shipped whichever weights the last step | |
| happened to produce. v5 holds out 4,096 pairs, evaluates every 2,000 steps, and writes the best | |
| checkpoint to a separate file. Across the whole run the held-out loss improved at every single check | |
| and never once turned upward — the clearest evidence that the data change worked, not just a | |
| side effect of training longer. Our sibling model | |
| <a href="https://huggingface.co/bench-labs/VoxelModel-v1">VoxelModel v1</a>, trained on 28,415 meshes | |
| for 811 epochs, peaked at step 64,000 and got 3.08% worse by the end. Same recipe, same team, | |
| different data regime, opposite outcome. | |
| </p> | |
| <h2> | |
| Results | |
| </h2> | |
| <p> | |
| Measured on 5,000 MS-COCO val2014 images at 256 center crop, using <code>torchmetrics</code> FID at | |
| 2048 features and <code>torchmetrics</code> CLIPScore with <code>openai/clip-vit-base-patch32</code>. | |
| This is v4's protocol, run again rather than assumed. | |
| </p> | |
| <pre><code> v4 v5 (cfg 5.0) v5 (cfg 3.0) v5 (cfg 6.0) | |
| FID ↓ 39.54 34.06 35.03 35.18 | |
| CLIP score ↑ 28.04 29.27 28.59 29.28 | |
| training images 83,000 2,986,571 2,986,571 2,986,571 | |
| epochs over data 247 ~10 ~10 ~10 | |
| trainable params 40,013,980 40,013,980 40,013,980 40,013,980 | |
| real photographs, CLIP — 30.05 (reference ceiling, all runs)</code></pre> | |
| <p> | |
| FID is 13.9% lower and CLIP score is 1.23 points higher, at an identical parameter count. Nothing in | |
| the network changed between v4 and v5, so the entire difference is the training data. Feeding real | |
| COCO photographs and their own captions through the same CLIPScore harness gives 30.05, which puts v5 | |
| at cfg 5.0 within 0.78 of the ceiling this metric can award. | |
| </p> | |
| <p> | |
| Guidance scale was swept rather than assumed. FID bottoms out at cfg 5.0 and CLIP keeps climbing | |
| slightly to cfg 6.0, the usual tension between the two. Sample count matters more than people expect | |
| here too — the same cfg 5.0 checkpoint scores 74.35 on 1,000 images and 34.06 on 5,000, so FID numbers | |
| are only comparable when the sample count matches. | |
| </p> | |
| <h2> | |
| Specification | |
| </h2> | |
| <pre><code>trainable parameters 40,013,980 | |
| frozen VAE (sd-vae-ft-mse) 83,653,863 | |
| frozen text encoder (CLIP ViT-B/32) 63,165,952 | |
| combined total 186,833,795 | |
| architecture DiT, dim 384, depth 12, heads 6, patch 2 | |
| tokens 256, over a 32x32x4 latent | |
| objective rectified flow, target x1 - x0, logit-normal timesteps | |
| conditioning cross attention on 40 CLIP text tokens, plus adaLN-zero | |
| resolution 256x256 | |
| sampling 50 steps, cfg 5.0 for best FID, cfg 6.0 for best CLIP</code></pre> | |
| <p> | |
| One correction worth flagging: v4's card reported 161.5M total parameters, but that count left out | |
| the CLIP text encoder's 49,408 × 512 token embedding table — another 25.3M parameters that the encoder | |
| cannot run without. The corrected total for both v4 and v5 is 186,833,795. | |
| </p> | |
| <h2> | |
| What it does well and what it does not | |
| </h2> | |
| <p> | |
| After a few hundred samples the boundary is consistent. It is good at material and light — food, | |
| landscapes, skies, water, snow, foliage, stone, interiors, depth of field. A bowl of ramen, a wet | |
| cobblestone street, a wheat field under storm light, a library of wooden shelves. These come back | |
| looking like photographs. | |
| </p> | |
| <p> | |
| <img src="https://huggingface.co/bench-labs/PixelModel-v5/resolve/main/hero.png" alt="Cobblestone street, library, wheat field, and snowy mountain generated by PixelModel v5" style="width: 100%; max-width: 640px;"> | |
| </p> | |
| <p> | |
| It is bad at structure. Faces and animal heads deform. Limbs merge. Wings on aircraft are stubby or | |
| fused to the fuselage. Any text it renders, on a shop sign or an airliner livery, is convincing | |
| looking gibberish. Prompts asking for one isolated object in an empty scene often return the scene | |
| without the object — "a lighthouse on a cliff at sunset" gives a very nice sunset over a cliff and no | |
| lighthouse. | |
| </p> | |
| <p> | |
| Frequency in the training data matters more than complexity. Aircraft are structurally complicated | |
| and come out fine because CC12M is full of them. Hot air balloons are simpler and fail, because they | |
| are rare. | |
| </p> | |
| <p> | |
| That failure profile is what a 40M model looks like once data is no longer the constraint. It has | |
| learned appearance thoroughly and topology poorly, and topology is what capacity buys. v6 is where | |
| that gets spent. | |
| </p> | |
| <h2> | |
| Try it | |
| </h2> | |
| <pre><code>git clone https://huggingface.co/bench-labs/PixelModel-v5 | |
| python main.py "a bowl of ramen with a soft boiled egg" --out ramen.png --cfg 6.0 --steps 50</code></pre> | |
| <p> | |
| As with every model in this line, the weights are also stored as a PNG. <code>model.png</code> is not | |
| a picture of the network, it is the network, with each parameter packed across the red and green | |
| channels of one pixel. Explore the | |
| <a href="https://huggingface.co/bench-labs/PixelModel-v5">PixelModel v5 repository</a> or the | |
| <a href="https://huggingface.co/spaces/bench-labs/BenchLabs-Leaderboard">text-to-image leaderboard</a>. | |
| </p> | |
| <blockquote> | |
| v4 proved the network needed to understand text. v5 proves that once it does, the next several | |
| points of FID are bought with data, not layers. | |
| </blockquote> | |
| </main> | |
| <script> | |
| /* ========================================================== | |
| SYNTAX HIGHLIGHTING | |
| ========================================================== */ | |
| const PY_KW = new Set([ | |
| 'def','class','return','if','elif','else','for','while','import','from','as', | |
| 'try','except','finally','with','yield','lambda','pass','break','continue', | |
| 'raise','and','or','not','in','is','global','nonlocal','assert','del', | |
| 'async','await' | |
| ]); | |
| const JS_KW = new Set([ | |
| 'const','let','var','function','return','if','else','for','while','do', | |
| 'switch','case','break','continue','new','class','extends','import','export', | |
| 'from','default','try','catch','finally','throw','typeof','instanceof', | |
| 'in','of','async','await','yield','void','delete','with' | |
| ]); | |
| const PY_BI = new Set([ | |
| 'print','len','range','int','str','float','list','dict','set','tuple','bool', | |
| 'type','isinstance','input','open','map','filter','zip','enumerate','sorted', | |
| 'reversed','sum','min','max','abs','round','any','all','super','property', | |
| 'staticmethod','classmethod','True','False','None','Exception','ValueError', | |
| 'TypeError','KeyError','IndexError','AttributeError','NotImplementedError', | |
| 'RuntimeError','StopIteration','FileNotFoundError','object' | |
| ]); | |
| const JS_BI = new Set([ | |
| 'console','Math','Array','Object','String','Number','Boolean','Date','RegExp', | |
| 'Error','TypeError','RangeError','SyntaxError','JSON','Promise','Map','Set', | |
| 'Symbol','Proxy','Reflect','parseInt','parseFloat','isNaN','isFinite', | |
| 'undefined','null','true','false','NaN','Infinity' | |
| ]); | |
| function escapeHtml(s) { | |
| return s.replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>'); | |
| } | |
| function detectLang(code) { | |
| let py = 0, js = 0; | |
| const pyP = /\b(def |class |import |from \w+ import|elif |print\(|True|False|None|self[\.\[])\b/g; | |
| const jsP = /\b(const |let |var |function |=>|console\.|undefined|null|async |await )\b/g; | |
| let m; | |
| while ((m = pyP.exec(code)) !== null) py++; | |
| while ((m = jsP.exec(code)) !== null) js++; | |
| if (py > js) return 'python'; | |
| if (js > 0) return 'javascript'; | |
| return 'text'; | |
| } | |
| function highlight(code) { | |
| const lang = detectLang(code); | |
| const kw = lang === 'python' ? PY_KW : lang === 'javascript' ? JS_KW : new Set(); | |
| const bi = lang === 'python' ? PY_BI : lang === 'javascript' ? JS_BI : new Set(); | |
| const tokens = []; | |
| let i = 0; | |
| const len = code.length; | |
| while (i < len) { | |
| if (lang === 'python') { | |
| const t3 = code.slice(i, i + 3); | |
| if (t3 === '"""' || t3 === "'''") { | |
| const end = code.indexOf(t3, i + 3); | |
| const span = end !== -1 ? code.slice(i, end + 3) : code.slice(i); | |
| tokens.push({ type: 'string', text: span }); | |
| i += span.length; | |
| continue; | |
| } | |
| } | |
| { | |
| const pm = code.slice(i).match(/^[frbFRB]{0,3}/); | |
| if (pm) { | |
| const pLen = pm[0].length; | |
| const nc = code[i + pLen]; | |
| if (nc === '"' || nc === "'") { | |
| const q = nc; | |
| const isF = pm[0].toLowerCase().includes('f'); | |
| let depth = 0, k = i + pLen + 1; | |
| let found = false; | |
| while (k < len) { | |
| const ch = code[k]; | |
| if (ch === '\\') { k += 2; continue; } | |
| if (isF) { | |
| if (ch === '{') { depth++; k++; continue; } | |
| if (ch === '}') { depth = Math.max(0, depth - 1); k++; continue; } | |
| } | |
| if (ch === q && depth === 0) { found = true; break; } | |
| k++; | |
| } | |
| if (found) { | |
| tokens.push({ type: 'string', text: code.slice(i, k + 1) }); | |
| i = k + 1; | |
| continue; | |
| } | |
| } | |
| } | |
| } | |
| if (lang === 'python' && code[i] === '#') { | |
| let end = code.indexOf('\n', i); | |
| if (end === -1) end = len; | |
| tokens.push({ type: 'comment', text: code.slice(i, end) }); | |
| i = end; | |
| continue; | |
| } | |
| if (lang === 'javascript' && code.slice(i, i + 2) === '//') { | |
| let end = code.indexOf('\n', i); | |
| if (end === -1) end = len; | |
| tokens.push({ type: 'comment', text: code.slice(i, end) }); | |
| i = end; | |
| continue; | |
| } | |
| if (lang === 'javascript' && code.slice(i, i + 2) === '/*') { | |
| let end = code.indexOf('*/', i + 2); | |
| if (end === -1) end = len; else end += 2; | |
| tokens.push({ type: 'comment', text: code.slice(i, end) }); | |
| i = end; | |
| continue; | |
| } | |
| { | |
| const m = code.slice(i).match( | |
| /^(0x[0-9a-fA-F_]+|0b[01_]+|0o[0-7_]+|\d[\d_]*\.?\d*(?:e[+-]?\d+)?)/ | |
| ); | |
| if (m) { | |
| tokens.push({ type: 'number', text: m[0] }); | |
| i += m[0].length; | |
| continue; | |
| } | |
| } | |
| if (lang === 'python' && code[i] === '@') { | |
| const m = code.slice(i).match(/^@[\w.]+/); | |
| if (m) { | |
| tokens.push({ type: 'decorator', text: m[0] }); | |
| i += m[0].length; | |
| continue; | |
| } | |
| } | |
| { | |
| const m = code.slice(i).match(/^\w+/); | |
| if (m) { | |
| const w = m[0]; | |
| let type = 'plain'; | |
| if (kw.has(w)) type = 'keyword'; | |
| else if (bi.has(w)) type = 'builtin'; | |
| else if (w === 'self' || w === 'cls' || w === 'this') type = 'self'; | |
| else { | |
| let j = i + w.length; | |
| while (j < len && code[j] === ' ') j++; | |
| if (code[j] === '(') type = 'function'; | |
| } | |
| tokens.push({ type, text: w }); | |
| i += w.length; | |
| continue; | |
| } | |
| } | |
| { | |
| const m = code.slice(i).match(/^[+\-*/%=<>!&|^~:]+/); | |
| if (m) { | |
| tokens.push({ type: 'operator', text: m[0] }); | |
| i += m[0].length; | |
| continue; | |
| } | |
| } | |
| if ('(){}[],;.'.includes(code[i])) { | |
| tokens.push({ type: 'punctuation', text: code[i] }); | |
| i++; | |
| continue; | |
| } | |
| { | |
| const m = code.slice(i).match(/^\s+/); | |
| if (m) { | |
| tokens.push({ type: 'plain', text: m[0] }); | |
| i += m[0].length; | |
| } else { | |
| tokens.push({ type: 'plain', text: code[i] }); | |
| i++; | |
| } | |
| } | |
| } | |
| return tokens.map(t => { | |
| const esc = escapeHtml(t.text); | |
| return t.type === 'plain' ? esc : `<span class="tok-${t.type}">${esc}</span>`; | |
| }).join(''); | |
| } | |
| document.querySelectorAll('pre code').forEach(block => { | |
| block.innerHTML = highlight(block.textContent); | |
| }); | |
| /* ========================================================== | |
| LIGHTBOX | |
| ========================================================== */ | |
| class Lightbox { | |
| constructor() { | |
| this.images = []; | |
| this.currentIndex = 0; | |
| this.isOpen = false; | |
| this._build(); | |
| this._bindImages(); | |
| this._bindKeys(); | |
| this._bindSwipe(); | |
| } | |
| _build() { | |
| this.el = document.createElement('div'); | |
| this.el.className = 'lightbox-overlay'; | |
| this.el.setAttribute('role', 'dialog'); | |
| this.el.setAttribute('aria-modal', 'true'); | |
| this.el.setAttribute('aria-label', 'Image viewer'); | |
| this.el.innerHTML = ` | |
| <button class="lightbox-btn lightbox-close" aria-label="Close"> | |
| <svg width="18" height="18" viewBox="0 0 24 24" fill="none" | |
| stroke="currentColor" stroke-width="2.5" stroke-linecap="round"> | |
| <line x1="18" y1="6" x2="6" y2="18"/> | |
| <line x1="6" y1="6" x2="18" y2="18"/> | |
| </svg> | |
| </button> | |
| <button class="lightbox-btn lightbox-arrow lightbox-prev" aria-label="Previous image"> | |
| <svg width="22" height="22" viewBox="0 0 24 24" fill="none" | |
| stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"> | |
| <polyline points="15 18 9 12 15 6"/> | |
| </svg> | |
| </button> | |
| <button class="lightbox-btn lightbox-arrow lightbox-next" aria-label="Next image"> | |
| <svg width="22" height="22" viewBox="0 0 24 24" fill="none" | |
| stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"> | |
| <polyline points="9 6 15 12 9 18"/> | |
| </svg> | |
| </button> | |
| <img class="lightbox-img" src="" alt=""> | |
| <div class="lightbox-counter"></div> | |
| `; | |
| document.body.appendChild(this.el); | |
| this.imgEl = this.el.querySelector('.lightbox-img'); | |
| this.counterEl = this.el.querySelector('.lightbox-counter'); | |
| this.prevBtn = this.el.querySelector('.lightbox-prev'); | |
| this.nextBtn = this.el.querySelector('.lightbox-next'); | |
| this.closeBtn = this.el.querySelector('.lightbox-close'); | |
| this.closeBtn.addEventListener('click', () => this.close()); | |
| this.prevBtn.addEventListener('click', (e) => { e.stopPropagation(); this.prev(); }); | |
| this.nextBtn.addEventListener('click', (e) => { e.stopPropagation(); this.next(); }); | |
| this.el.addEventListener('click', (e) => { | |
| if (e.target === this.el) this.close(); | |
| }); | |
| } | |
| _bindImages() { | |
| this.images = Array.from(document.querySelectorAll('.article img')); | |
| this.images.forEach((img, idx) => { | |
| img.addEventListener('click', () => this.open(idx)); | |
| }); | |
| } | |
| _bindKeys() { | |
| document.addEventListener('keydown', (e) => { | |
| if (!this.isOpen) return; | |
| if (e.key === 'Escape') this.close(); | |
| if (e.key === 'ArrowLeft') this.prev(); | |
| if (e.key === 'ArrowRight') this.next(); | |
| }); | |
| } | |
| _bindSwipe() { | |
| let startX = 0; | |
| this.el.addEventListener('touchstart', (e) => { | |
| startX = e.changedTouches[0].clientX; | |
| }, { passive: true }); | |
| this.el.addEventListener('touchend', (e) => { | |
| const dx = startX - e.changedTouches[0].clientX; | |
| if (Math.abs(dx) > 60) { | |
| dx > 0 ? this.next() : this.prev(); | |
| } | |
| }, { passive: true }); | |
| } | |
| open(idx) { | |
| this.currentIndex = idx; | |
| this.isOpen = true; | |
| this._update(); | |
| requestAnimationFrame(() => { | |
| this.el.classList.add('active'); | |
| }); | |
| document.body.style.overflow = 'hidden'; | |
| this.closeBtn.focus(); | |
| } | |
| close() { | |
| this.isOpen = false; | |
| this.el.classList.remove('active'); | |
| document.body.style.overflow = ''; | |
| this.images[this.currentIndex]?.focus(); | |
| } | |
| prev() { | |
| this.currentIndex = (this.currentIndex - 1 + this.images.length) % this.images.length; | |
| this._update(); | |
| } | |
| next() { | |
| this.currentIndex = (this.currentIndex + 1) % this.images.length; | |
| this._update(); | |
| } | |
| _update() { | |
| const srcImg = this.images[this.currentIndex]; | |
| this.imgEl.style.opacity = '0'; | |
| this.imgEl.style.transform = 'scale(0.96)'; | |
| setTimeout(() => { | |
| this.imgEl.src = srcImg.src; | |
| this.imgEl.alt = srcImg.alt; | |
| this.imgEl.removeAttribute('loading'); | |
| this.imgEl.style.opacity = '1'; | |
| this.imgEl.style.transform = 'scale(1)'; | |
| }, 120); | |
| const total = this.images.length; | |
| if (total > 1) { | |
| this.counterEl.textContent = `${this.currentIndex + 1} / ${total}`; | |
| this.prevBtn.style.display = ''; | |
| this.nextBtn.style.display = ''; | |
| } else { | |
| this.counterEl.textContent = ''; | |
| this.prevBtn.style.display = 'none'; | |
| this.nextBtn.style.display = 'none'; | |
| } | |
| } | |
| } | |
| new Lightbox(); | |
| </script> | |
| </body> | |
| </html> | |