Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>PixelModel v4: Cross-Attention, Text Transformers, and Native 256px</title> | |
| <link rel="stylesheet" href="../style.css"> | |
| </head> | |
| <body> | |
| <script type="module" src="../summarize.js" defer></script> | |
| <script src="../script.js" defer></script> | |
| <div class="background"></div> | |
| <main class="article"> | |
| <a class="back" href="../index.html"> | |
| ← Back | |
| </a> | |
| <div class="meta"> | |
| July 2026 | |
| </div> | |
| <h1> | |
| PixelModel v4: Cross-Attention, Text Transformers, and Native 256px | |
| </h1> | |
| <p> | |
| <a href="./Pm2and3.html">PixelModel v3</a> proved that smart architectural changes—a SIREN implicit neural decoder, learned embeddings, and zero-initialized FiLM conditioning—could beat raw width while keeping the parameter budget under 1 million weights. But v3 had a glaring bottleneck that became obvious the second you gave it a complex prompt: it treated text as an unordered bag of words. | |
| </p> | |
| <p> | |
| PixelModel v4 fixes the text understanding path, replaces static Fourier coordinate encoding with learnable projections, and introduces spatial cross-attention conditioning at native 256×256 resolution. | |
| </p> | |
| <h2> | |
| Moving beyond the bag-of-words bottleneck | |
| </h2> | |
| <p> | |
| In v3, word vectors were mean-pooled across tokens into a single 192-dimensional latent vector z. This meant "a red dog on a green rug" and "a green dog on a red rug" produced almost identical global latents. The spatial coordinate decoder had no way of knowing <em>where</em> in the prompt an attribute belonged, nor could it handle syntax or negation. | |
| </p> | |
| <p> | |
| v4 scales parameters to 1,842,112—roughly 2× v3—and spends almost all of that new budget on solving text compositionality and high-frequency spatial detail. | |
| </p> | |
| <p> | |
| <img src="https://huggingface.co/bench-labs/PixelModel-v4/resolve/main/model.png" alt="The entire PixelModel v4 model weight PNG holding all 1,842,112 parameters" style="image-rendering: pixelated; width: 320px; max-width: 100%;"> | |
| </p> | |
| <h2> | |
| What actually changed in v4 | |
| </h2> | |
| <ul> | |
| <li><strong>2-Layer Transformer Text Encoder.</strong> We replaced mean-pooled lookup tables with Byte-Pair Encoding (BPE, 8,192 vocabulary) fed into a lightweight 2-layer, 4-head Transformer encoder (d_{\text{model}} = 128). The model now retains token sequence ordering and context.</li> | |
| <li><strong>Spatial Cross-Attention FiLM.</strong> Rather than modulating the entire decoder with a single global vector z, each SIREN layer performs multi-head cross-attention over the sequence of text token representations. Spatial coordinate locations query the text tokens directly to derive local scale and shift parameters.</li> | |
| <li><strong>Learnable Random Fourier Features (RFF).</strong> Fixed octave frequencies were prone to grid artifacts at higher resolutions. v4 initializes coordinate projections using a Gaussian distribution B \sim \mathcal{N}(0, \sigma^2) and allows B to train end-to-end alongside the decoder.</li> | |
| <li><strong>Native 256×256 Resolution.</strong> Training directly on whole 256×256 frames with a combined L1 and LPIPS perceptual loss allows the network to render high-frequency textures without losing global structure.</li> | |
| </ul> | |
| <h2> | |
| Re-benchmarking the series | |
| </h2> | |
| <p> | |
| To maintain strict evaluation parity with the previous releases, v4 was benchmarked on the exact same 5,000-image MS-COCO val2014 validation subset using identical <code>torchmetrics</code> evaluation code. | |
| </p> | |
| <pre><code> v1 v2 v3 v4 | |
| params 23,747 ~200,000 919,427 1,842,112 | |
| FID published 439.46 — — — | |
| CLIP published 20.02 — — — | |
| FID, this pipeline ↓ 420.75 390.68 383.91 314.12 | |
| CLIP, this pipeline ↑ 20.10 20.48 20.73 23.45 | |
| model.png 160x149 scaled 959x959 1357x1357 (3.52 MB) | |
| native resolution 64 64 128 256</code></pre> | |
| <p> | |
| The jump from v3 to v4 represents the largest single generational improvement in the series. FID dropped by nearly 70 points (383.91 → 314.12) while CLIP score jumped to 23.45. The Transformer text encoder and cross-attention mechanism directly account for the boost in text alignment. | |
| </p> | |
| <h2> | |
| Training & Optimization | |
| </h2> | |
| <p> | |
| v4 trained for 120 epochs on a single RTX 4090 in approximately 2.5 hours using batch size 64, AdamW (\beta_1=0.9, \beta_2=0.999, weight decay 0.01), and a cosine learning rate schedule peaking at 3e-4 with automatic mixed precision (AMP). | |
| </p> | |
| <p> | |
| Adding LPIPS perceptual loss (0.1 \times \text{LPIPS} + 0.9 \times \text{L1}) was essential: training SIREN networks on 256px frames with pure L_1 or MSE loss resulted in blurry high frequencies, whereas perceptual loss forced the learnable Fourier coordinate features to utilize their full bandwidth. | |
| </p> | |
| <h2> | |
| Architecture | |
| </h2> | |
| <pre><code>prompt string | |
| BPE Tokenizer -> token ids (vocab size: 8192, max tokens: 32) | |
| 2-Layer Transformer Encoder (d_model=128, nhead=4) [328,192 params] | |
| Sequence representations: T in R^{32 x 128} | |
| Cross-Attention FiLM Generator: | |
| Per-layer Multi-Head Cross-Attention (Query: coordinate state, Key/Value: T) | |
| Generates per-pixel modulation (scale, shift) for SIREN [612,352 params] | |
| for every pixel (x, y), decoded as one batched tensor: | |
| Learnable Fourier Projection: [x, y] x B (2 -> 64 dims) [128 params] | |
| SineLayer 0: Linear(64 -> 384) -> Cross-FiLM -> sin(30 * .) | |
| SineLayer 1: Linear(384 -> 384) -> Cross-FiLM -> sin(30 * .) | |
| SineLayer 2: Linear(384 -> 384) -> Cross-FiLM -> sin(30 * .) | |
| SineLayer 3: Linear(384 -> 384) -> Cross-FiLM -> sin(30 * .) | |
| head: Linear(384 -> 3) -> sigmoid = RGB [901,440 params]</code></pre> | |
| <h2> | |
| Try it | |
| </h2> | |
| <pre><code>git clone https://huggingface.co/bench-labs/PixelModel-v4 | |
| python main.py "a red dog sitting on a green rug" --out dog.png | |
| python main.py "a cyberpunk city street at sunset" --res 512</code></pre> | |
| <p> | |
| <code>model.png</code> carries the entire 1.84M parameter binary checkpoint inside a pixelated 1357×1357 PNG image, while <code>model.safetensors</code> provides standard weights and parameter keys. Explore the <a href="https://huggingface.co/bench-labs/PixelModel-v4">PixelModel v4 repository</a> or the complete <a href="https://huggingface.co/collections/bench-labs/text-to-image">text-to-image collection</a>. | |
| </p> | |
| <blockquote> | |
| v3 proved SIREN decoders work for coordinate generation. v4 proves that even tiny models need real sequence awareness to make prompt conditioning meaningful. | |
| </blockquote> | |
| </main> | |
| <script> | |
| /* ========================================================== | |
| SYNTAX HIGHLIGHTING | |
| ========================================================== */ | |
| const PY_KW = new Set([ | |
| 'def','class','return','if','elif','else','for','while','import','from','as', | |
| 'try','except','finally','with','yield','lambda','pass','break','continue', | |
| 'raise','and','or','not','in','is','global','nonlocal','assert','del', | |
| 'async','await' | |
| ]); | |
| const JS_KW = new Set([ | |
| 'const','let','var','function','return','if','else','for','while','do', | |
| 'switch','case','break','continue','new','class','extends','import','export', | |
| 'from','default','try','catch','finally','throw','typeof','instanceof', | |
| 'in','of','async','await','yield','void','delete','with' | |
| ]); | |
| const PY_BI = new Set([ | |
| 'print','len','range','int','str','float','list','dict','set','tuple','bool', | |
| 'type','isinstance','input','open','map','filter','zip','enumerate','sorted', | |
| 'reversed','sum','min','max','abs','round','any','all','super','property', | |
| 'staticmethod','classmethod','True','False','None','Exception','ValueError', | |
| 'TypeError','KeyError','IndexError','AttributeError','NotImplementedError', | |
| 'RuntimeError','StopIteration','FileNotFoundError','object' | |
| ]); | |
| const JS_BI = new Set([ | |
| 'console','Math','Array','Object','String','Number','Boolean','Date','RegExp', | |
| 'Error','TypeError','RangeError','SyntaxError','JSON','Promise','Map','Set', | |
| 'Symbol','Proxy','Reflect','parseInt','parseFloat','isNaN','isFinite', | |
| 'undefined','null','true','false','NaN','Infinity' | |
| ]); | |
| function escapeHtml(s) { | |
| return s.replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>'); | |
| } | |
| function detectLang(code) { | |
| let py = 0, js = 0; | |
| const pyP = /\b(def |class |import |from \w+ import|elif |print\(|True|False|None|self[\.\[])\b/g; | |
| const jsP = /\b(const |let |var |function |=>|console\.|undefined|null|async |await )\b/g; | |
| let m; | |
| while ((m = pyP.exec(code)) !== null) py++; | |
| while ((m = jsP.exec(code)) !== null) js++; | |
| if (py > js) return 'python'; | |
| if (js > 0) return 'javascript'; | |
| return 'text'; | |
| } | |
| function highlight(code) { | |
| const lang = detectLang(code); | |
| const kw = lang === 'python' ? PY_KW : lang === 'javascript' ? JS_KW : new Set(); | |
| const bi = lang === 'python' ? PY_BI : lang === 'javascript' ? JS_BI : new Set(); | |
| const tokens = []; | |
| let i = 0; | |
| const len = code.length; | |
| while (i < len) { | |
| if (lang === 'python') { | |
| const t3 = code.slice(i, i + 3); | |
| if (t3 === '"""' || t3 === "'''") { | |
| const end = code.indexOf(t3, i + 3); | |
| const span = end !== -1 ? code.slice(i, end + 3) : code.slice(i); | |
| tokens.push({ type: 'string', text: span }); | |
| i += span.length; | |
| continue; | |
| } | |
| } | |
| { | |
| const pm = code.slice(i).match(/^[frbFRB]{0,3}/); | |
| if (pm) { | |
| const pLen = pm[0].length; | |
| const nc = code[i + pLen]; | |
| if (nc === '"' || nc === "'") { | |
| const q = nc; | |
| const isF = pm[0].toLowerCase().includes('f'); | |
| let depth = 0, k = i + pLen + 1; | |
| let found = false; | |
| while (k < len) { | |
| const ch = code[k]; | |
| if (ch === '\\') { k += 2; continue; } | |
| if (isF) { | |
| if (ch === '{') { depth++; k++; continue; } | |
| if (ch === '}') { depth = Math.max(0, depth - 1); k++; continue; } | |
| } | |
| if (ch === q && depth === 0) { found = true; break; } | |
| k++; | |
| } | |
| if (found) { | |
| tokens.push({ type: 'string', text: code.slice(i, k + 1) }); | |
| i = k + 1; | |
| continue; | |
| } | |
| } | |
| } | |
| } | |
| if (lang === 'python' && code[i] === '#') { | |
| let end = code.indexOf('\n', i); | |
| if (end === -1) end = len; | |
| tokens.push({ type: 'comment', text: code.slice(i, end) }); | |
| i = end; | |
| continue; | |
| } | |
| if (lang === 'javascript' && code.slice(i, i + 2) === '//') { | |
| let end = code.indexOf('\n', i); | |
| if (end === -1) end = len; | |
| tokens.push({ type: 'comment', text: code.slice(i, end) }); | |
| i = end; | |
| continue; | |
| } | |
| if (lang === 'javascript' && code.slice(i, i + 2) === '/*') { | |
| let end = code.indexOf('*/', i + 2); | |
| if (end === -1) end = len; else end += 2; | |
| tokens.push({ type: 'comment', text: code.slice(i, end) }); | |
| i = end; | |
| continue; | |
| } | |
| { | |
| const m = code.slice(i).match( | |
| /^(0x[0-9a-fA-F_]+|0b[01_]+|0o[0-7_]+|\d[\d_]*\.?\d*(?:e[+-]?\d+)?)/ | |
| ); | |
| if (m) { | |
| tokens.push({ type: 'number', text: m[0] }); | |
| i += m[0].length; | |
| continue; | |
| } | |
| } | |
| if (lang === 'python' && code[i] === '@') { | |
| const m = code.slice(i).match(/^@[\w.]+/); | |
| if (m) { | |
| tokens.push({ type: 'decorator', text: m[0] }); | |
| i += m[0].length; | |
| continue; | |
| } | |
| } | |
| { | |
| const m = code.slice(i).match(/^\w+/); | |
| if (m) { | |
| const w = m[0]; | |
| let type = 'plain'; | |
| if (kw.has(w)) type = 'keyword'; | |
| else if (bi.has(w)) type = 'builtin'; | |
| else if (w === 'self' || w === 'cls' || w === 'this') type = 'self'; | |
| else { | |
| let j = i + w.length; | |
| while (j < len && code[j] === ' ') j++; | |
| if (code[j] === '(') type = 'function'; | |
| } | |
| tokens.push({ type, text: w }); | |
| i += w.length; | |
| continue; | |
| } | |
| } | |
| { | |
| const m = code.slice(i).match(/^[+\-*/%=<>!&|^~:]+/); | |
| if (m) { | |
| tokens.push({ type: 'operator', text: m[0] }); | |
| i += m[0].length; | |
| continue; | |
| } | |
| } | |
| if ('(){}[],;.'.includes(code[i])) { | |
| tokens.push({ type: 'punctuation', text: code[i] }); | |
| i++; | |
| continue; | |
| } | |
| { | |
| const m = code.slice(i).match(/^\s+/); | |
| if (m) { | |
| tokens.push({ type: 'plain', text: m[0] }); | |
| i += m[0].length; | |
| } else { | |
| tokens.push({ type: 'plain', text: code[i] }); | |
| i++; | |
| } | |
| } | |
| } | |
| return tokens.map(t => { | |
| const esc = escapeHtml(t.text); | |
| return t.type === 'plain' ? esc : `<span class="tok-${t.type}">${esc}</span>`; | |
| }).join(''); | |
| } | |
| document.querySelectorAll('pre code').forEach(block => { | |
| block.innerHTML = highlight(block.textContent); | |
| }); | |
| /* ========================================================== | |
| LIGHTBOX | |
| ========================================================== */ | |
| class Lightbox { | |
| constructor() { | |
| this.images = []; | |
| this.currentIndex = 0; | |
| this.isOpen = false; | |
| this._build(); | |
| this._bindImages(); | |
| this._bindKeys(); | |
| this._bindSwipe(); | |
| } | |
| _build() { | |
| this.el = document.createElement('div'); | |
| this.el.className = 'lightbox-overlay'; | |
| this.el.setAttribute('role', 'dialog'); | |
| this.el.setAttribute('aria-modal', 'true'); | |
| this.el.setAttribute('aria-label', 'Image viewer'); | |
| this.el.innerHTML = ` | |
| <button class="lightbox-btn lightbox-close" aria-label="Close"> | |
| <svg width="18" height="18" viewBox="0 0 24 24" fill="none" | |
| stroke="currentColor" stroke-width="2.5" stroke-linecap="round"> | |
| <line x1="18" y1="6" x2="6" y2="18"/> | |
| <line x1="6" y1="6" x2="18" y2="18"/> | |
| </svg> | |
| </button> | |
| <button class="lightbox-btn lightbox-arrow lightbox-prev" aria-label="Previous image"> | |
| <svg width="22" height="22" viewBox="0 0 24 24" fill="none" | |
| stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"> | |
| <polyline points="15 18 9 12 15 6"/> | |
| </svg> | |
| </button> | |
| <button class="lightbox-btn lightbox-arrow lightbox-next" aria-label="Next image"> | |
| <svg width="22" height="22" viewBox="0 0 24 24" fill="none" | |
| stroke="currentColor" stroke-width="2.5" stroke-linecap="round" stroke-linejoin="round"> | |
| <polyline points="9 6 15 12 9 18"/> | |
| </svg> | |
| </button> | |
| <img class="lightbox-img" src="" alt=""> | |
| <div class="lightbox-counter"></div> | |
| `; | |
| document.body.appendChild(this.el); | |
| this.imgEl = this.el.querySelector('.lightbox-img'); | |
| this.counterEl = this.el.querySelector('.lightbox-counter'); | |
| this.prevBtn = this.el.querySelector('.lightbox-prev'); | |
| this.nextBtn = this.el.querySelector('.lightbox-next'); | |
| this.closeBtn = this.el.querySelector('.lightbox-close'); | |
| this.closeBtn.addEventListener('click', () => this.close()); | |
| this.prevBtn.addEventListener('click', (e) => { e.stopPropagation(); this.prev(); }); | |
| this.nextBtn.addEventListener('click', (e) => { e.stopPropagation(); this.next(); }); | |
| this.el.addEventListener('click', (e) => { | |
| if (e.target === this.el) this.close(); | |
| }); | |
| } | |
| _bindImages() { | |
| this.images = Array.from(document.querySelectorAll('.article img')); | |
| this.images.forEach((img, idx) => { | |
| img.addEventListener('click', () => this.open(idx)); | |
| }); | |
| } | |
| _bindKeys() { | |
| document.addEventListener('keydown', (e) => { | |
| if (!this.isOpen) return; | |
| if (e.key === 'Escape') this.close(); | |
| if (e.key === 'ArrowLeft') this.prev(); | |
| if (e.key === 'ArrowRight') this.next(); | |
| }); | |
| } | |
| _bindSwipe() { | |
| let startX = 0; | |
| this.el.addEventListener('touchstart', (e) => { | |
| startX = e.changedTouches[0].clientX; | |
| }, { passive: true }); | |
| this.el.addEventListener('touchend', (e) => { | |
| const dx = startX - e.changedTouches[0].clientX; | |
| if (Math.abs(dx) > 60) { | |
| dx > 0 ? this.next() : this.prev(); | |
| } | |
| }, { passive: true }); | |
| } | |
| open(idx) { | |
| this.currentIndex = idx; | |
| this.isOpen = true; | |
| this._update(); | |
| requestAnimationFrame(() => { | |
| this.el.classList.add('active'); | |
| }); | |
| document.body.style.overflow = 'hidden'; | |
| this.closeBtn.focus(); | |
| } | |
| close() { | |
| this.isOpen = false; | |
| this.el.classList.remove('active'); | |
| document.body.style.overflow = ''; | |
| this.images[this.currentIndex]?.focus(); | |
| } | |
| prev() { | |
| this.currentIndex = (this.currentIndex - 1 + this.images.length) % this.images.length; | |
| this._update(); | |
| } | |
| next() { | |
| this.currentIndex = (this.currentIndex + 1) % this.images.length; | |
| this._update(); | |
| } | |
| _update() { | |
| const srcImg = this.images[this.currentIndex]; | |
| this.imgEl.style.opacity = '0'; | |
| this.imgEl.style.transform = 'scale(0.96)'; | |
| setTimeout(() => { | |
| this.imgEl.src = srcImg.src; | |
| this.imgEl.alt = srcImg.alt; | |
| this.imgEl.removeAttribute('loading'); | |
| this.imgEl.style.opacity = '1'; | |
| this.imgEl.style.transform = 'scale(1)'; | |
| }, 120); | |
| const total = this.images.length; | |
| if (total > 1) { | |
| this.counterEl.textContent = `${this.currentIndex + 1} / ${total}`; | |
| this.prevBtn.style.display = ''; | |
| this.nextBtn.style.display = ''; | |
| } else { | |
| this.counterEl.textContent = ''; | |
| this.prevBtn.style.display = 'none'; | |
| this.nextBtn.style.display = 'none'; | |
| } | |
| } | |
| } | |
| new Lightbox(); | |
| </script> | |
| </body> | |
| </html> | |