Spaces:
Running
Running
Download static/index.html from JetBrains-Research/layer-lab: direct link, hf CLI and curl.
- Browser
- Download file 16.3 kB
-
https://huggingface.co/spaces/JetBrains-Research/layer-lab/resolve/main/static/index.html
- Command line
-
hf download hf://spaces/JetBrains-Research/layer-lab/static/index.html
-
curl -L -o index.html https://huggingface.co/spaces/JetBrains-Research/layer-lab/resolve/main/static/index.html
16.3 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <meta name="theme-color" content="#ffffff"> | |
| <title>Layer lab — Qwen & Gemma</title> | |
| <link rel="icon" href="data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 64 64'%3E%3Crect x='6' y='4' width='52' height='56' rx='3' fill='white' stroke='%2337352f' stroke-width='3'/%3E%3Ctext x='32' y='46' text-anchor='middle' font-family='Georgia' font-size='42' font-weight='bold' fill='%2337352f'%3EL%3C/text%3E%3C/svg%3E"> | |
| <link rel="stylesheet" href="/styles.css"> | |
| <script type="module" src="/app.js"></script> | |
| </head> | |
| <body> | |
| <header class="site-header"> | |
| <a class="brand" href="/"><span class="brand-mark">L</span> Layer lab <span class="brand-light">/</span><span class="brand-caption">Layers & attention</span></a> | |
| <nav><a href="/grpo" class="grpo-nav" title="Open the GRPO sandbox">GRPO</a><a href="/ppo" class="grpo-nav" title="Open the PPO sandbox">PPO</a><a href="#lens-section">Layer explorer</a><a href="#attention-section">Attention</a><a href="#method">Method</a><span id="model-pill" class="model-pill">Qwen3-0.6B</span></nav> | |
| </header> | |
| <main> | |
| <div class="intro"> | |
| <div><p class="eyebrow">INTERACTIVE NOTEBOOK</p><h1>Inside the model</h1><p id="intro-description" class="intro-description">One context. 28 layers. 448 attention heads.<br>Trace how the prediction at each token changes.</p></div> | |
| <div class="intro-note"><span>MODEL <strong id="model-name">Qwen3-0.6B</strong></span><span id="model-shape">28 LAYERS <i> / </i>16 Q HEADS <i> / </i>8 KV HEADS</span><span id="runtime-note">Checkpoint · loading…</span></div> | |
| </div> | |
| <div class="workspace"> | |
| <aside class="controls" aria-label="Input context"> | |
| <div class="sidebar-heading"><h2>Context <span class="small-tag">INPUT</span></h2><button id="reset" class="text-button">Reset</button></div> | |
| <section class="control-section model-control"> | |
| <label class="field-label" for="model-select">Model</label> | |
| <select id="model-select"><option value="Qwen/Qwen3-0.6B">Qwen3-0.6B</option><option value="ThomasTheMaker/gm3-270m-TinyGSM-reasoning">Gemma 3 270M · TinyGSM reasoning</option></select> | |
| <p id="model-description" class="field-note">28 layers · 16 heads · native reasoning switch</p> | |
| </section> | |
| <section class="control-section"> | |
| <label class="field-label" for="template"><span class="step">01</span> Template</label> | |
| <select id="template"><option value="question">Question and answer</option><option value="math">Reasoning problem</option><option value="system">System instruction</option><option value="dialogue">Conversation with history</option><option value="completion">Text completion · Raw</option><option value="empty">Blank context</option></select> | |
| <div class="mode-row"><div class="segmented" aria-label="Context format"><button data-mode="chat" class="active" aria-pressed="true">Chat</button><button data-mode="raw" aria-pressed="false">Raw</button></div><span id="format-label" class="field-note">Official chat template</span></div> | |
| <div id="chat-editor"><div id="messages"></div><button id="add-message" class="text-button add-message">+ Add message</button></div> | |
| <div id="raw-editor" class="hidden"><label class="sr-only" for="raw-text">Exact model input</label><textarea id="raw-text" rows="9" spellcheck="false" placeholder="Exact text, including control tokens…"></textarea><p class="field-note">Text is passed through as entered, without a template or extra tokens.</p></div> | |
| <div class="thinking-row"><div><label for="thinking">Reasoning</label><p class="field-note" id="thinking-note">An empty <think> block precedes the answer</p></div><label class="switch" id="reasoning-switch"><input type="checkbox" id="thinking" aria-label="Enable reasoning"><span></span></label></div> | |
| <details id="prompt-details"><summary>Exact model input <span>⌄</span></summary><pre id="prompt-preview">Loading tokenizer…</pre><button id="edit-raw" class="text-button">Open in Raw ↗</button></details> | |
| </section> | |
| <section class="control-section"> | |
| <div class="section-label"><h2><span class="step">02</span> Control tokens</h2><span class="small-tag">CLICK TO INSERT</span></div> | |
| <div id="special-tokens" class="special-tokens"></div> | |
| <p class="field-note">Insert at the cursor. Hover to see the token ID and purpose.</p> | |
| <details class="extra-tokens"><summary>Other vocabulary tokens <span>+</span></summary><div id="extra-tokens" class="special-tokens"></div><p class="field-note">Additional markers belong to this tokenizer. Their presence does not give this text model image or audio support.</p></details> | |
| </section> | |
| <section class="control-section run-controls"> | |
| <button id="analyze" class="primary-button" disabled><span>Inspect context</span><span>↵</span></button> | |
| <div class="run-meta"><span id="token-count">— tokens</span><span>⌘ / Ctrl + Enter</span></div> | |
| <p id="model-status" class="load-status" role="status" aria-live="polite">Loading Qwen3-0.6B…</p> | |
| <p id="input-error" class="error hidden" role="alert"></p> | |
| </section> | |
| <p class="sidebar-footnote">Real forward pass · no API keys<br>Processed on this site's server.</p> | |
| </aside> | |
| <div class="results" id="results"> | |
| <div class="result-toolbar"><span id="run-label" class="eyebrow">CONTEXT → TRANSFORMER → NEXT TOKEN</span><button id="export" class="text-button" disabled>Export JSON ↓</button></div> | |
| <div class="metrics"><article><span>Context tokens</span><strong id="metric-tokens">—</strong><p id="metric-input">Exact model tokenization</p></article><article><span id="metric-next-label">Next token · layer 28</span><strong id="metric-next">—</strong><p id="metric-prob">Probability over the full vocabulary</p></article><article><span>Forward pass</span><strong id="metric-time">—</strong><p id="metric-runtime">28 layers × 16 heads</p></article></div> | |
| <div id="stale-notice" class="notice hidden">The context has changed. Run the analysis to update the results.</div> | |
| <div id="busy-notice" class="notice hidden" role="status"><span class="spinner"></span><span id="busy-text">Running the forward pass…</span><span id="busy-time"></span></div> | |
| <div id="empty-state" class="empty-state"><div class="empty-diagram"><span>Text</span><i>→</i><span class="diagram-tokens">tokens</span><i>→</i><span id="diagram-layers" class="diagram-layers">28 layers</span><i>→</i><span>predictions</span></div><h2>Every token under the microscope</h2><p>Choose a template or enter your own context.<br>Layer predictions and attention maps will appear here after the run.</p><span id="empty-model-note" class="empty-footnote">Computed from the actual Qwen3-0.6B weights</span></div> | |
| <div id="run-content" class="hidden"> | |
| <section class="panel" id="lens-section"> | |
| <div class="panel-heading"> | |
| <div><p class="eyebrow">01 / LAYER EXPLORER</p><h2>From input to prediction</h2></div> | |
| <div class="segmented" aria-label="Cell labels"><button data-lens="token" class="active" aria-pressed="true">Token + %</button><button data-lens="token-only" aria-pressed="false">Token only</button><button data-lens="probability" aria-pressed="false">% only</button></div> | |
| </div> | |
| <p id="lens-description" class="panel-description">Input tokens stay at the bottom. Read layers upward: layer 1 → final layer. Click a column to select a token. Use Attention to view its weights across layers.</p> | |
| <div class="lens-toolbar"> | |
| <div class="segmented lens-color-switch" aria-label="Color cells by"><button data-lens-color="probability" class="active" aria-pressed="true">Predictions</button><button data-lens-color="attention" aria-pressed="false">Attention</button></div> | |
| <label class="lens-head-control hidden" for="lens-head">Attention heads <select id="lens-head"><option value="mean">Mean of all heads</option></select></label> | |
| <label class="check-field lens-attention-setting hidden" title="Logarithmic colors relative to each layer's maximum reveal small weights. Percentages remain absolute."><input id="lens-relative" type="checkbox" checked>Boost weak weights</label> | |
| </div> | |
| <div class="lens-status-row"><p id="lens-status" role="status" aria-live="polite">Cell color and percentage show the next-token probability.</p><button id="lens-peak" class="text-button hidden" type="button">Show peak ↗</button><span class="flow-label">FINAL LAYER ↑ INPUT</span></div> | |
| <div class="heat-legend"><span id="lens-legend-label">Prediction probability</span><span id="lens-scale-min">0%</span><i></i><span id="lens-scale-max">100%</span><span class="scroll-hint">Scroll ↕ ↔ · drag the lower corner to resize</span></div> | |
| <div class="lens-scroll" tabindex="0" aria-label="Layers above input tokens. Scroll to explore predictions and attention."><table id="lens-table"></table></div> | |
| <form id="append-form" class="token-composer hidden" aria-label="Append your own token"> | |
| <div class="composer-heading"><label class="field-label" for="append-text">Write the next token</label><button id="append-cancel" class="text-button" type="button">Cancel ×</button></div> | |
| <textarea id="append-text" rows="2" spellcheck="false" placeholder="Type text exactly, including spaces…" aria-describedby="append-count"></textarea> | |
| <p id="append-count" class="field-note" role="status" aria-live="polite">Only the first tokenizer token will be appended. Spaces are preserved.</p> | |
| <div id="append-preview" class="append-preview" aria-label="First token to append"></div> | |
| <p id="append-error" class="error hidden" role="alert"></p> | |
| <button id="append-custom" class="secondary-button" type="submit" disabled>Append first token →</button> | |
| </form> | |
| <p id="lens-footnote" class="table-footnote">Input at the bottom · final layer at the top · positions start at 0</p> | |
| </section> | |
| <section class="panel token-panel" id="token-section"><div class="panel-heading"><div><p class="eyebrow">02 / TOKENIZATION</p><h2>The tokenized context</h2></div><span class="small-muted">Click a token to select its position</span></div><div class="token-legend"><span><i class="legend-square special"></i>Control token</span><span><i class="legend-square selected"></i>Selected position</span><span id="generated-legend" class="hidden"><i class="legend-square generated"></i>Added token</span></div><div id="context-tokens" class="context-tokens"></div><details id="generation-details" class="hidden"><summary>Appended text <span>⌄</span></summary><pre id="generated-text"></pre><p id="generation-note" class="field-note"></p></details></section> | |
| <section class="panel inspector-panel" id="inspector-section"><div class="panel-heading"><div><p class="eyebrow">03 / POSITION INSPECTOR</p><h2 id="inspector-heading">Position — / layer —</h2></div><div class="inspector-navigation"><button id="prev-token" aria-label="Previous token">←</button><button id="next-token" aria-label="Next token">→</button></div></div><div class="inspector-grid"><div><p class="mini-label">INPUT TOKEN</p><div id="selected-token" class="selected-token"></div><p id="selected-token-detail" class="field-note"></p><div id="causal-note" class="causal-note">This position can only attend to itself and preceding tokens.</div></div><div><p class="mini-label">TOP-5 NEXT TOKENS</p><div id="top-predictions"></div></div></div></section> | |
| <section class="panel attention-panel" id="attention-section"><div class="attention-toolbar"><div class="panel-heading"><div><p class="eyebrow">04 / ATTENTION</p><h2>Where does attention go?</h2></div><div class="attention-selectors"><label for="layer-select">Layer <select id="layer-select"></select></label><label for="head-select">Head <select id="head-select"></select></label></div></div><div class="attention-subbar"><span id="head-mapping" class="field-note"></span><label class="check-field"><input id="relative-scale" type="checkbox" checked>Amplify weak weights</label></div></div><p id="attention-description" class="panel-description">Color shows how much attention the selected position assigns to each token. Choose one head or the arithmetic mean of all query heads after softmax.</p><div class="attention-summary"><span id="attention-query"></span><span id="attention-peak"></span></div><div id="attention-tokens" class="attention-tokens"></div><div class="heat-legend attention-legend"><span id="attention-scale-label">Intensity</span><span>0</span><i></i><span id="attention-scale-max">100%</span><span class="masked-legend">▧ Future positions are hidden by the causal mask</span></div><div class="attention-view-heading"><h3 id="attention-matrix-heading">Attention matrix</h3><span class="field-note">Rows are queries · columns are keys · click a cell</span></div><div class="matrix-layout"><div class="matrix-wrap"><canvas id="attention-matrix" aria-label="Attention matrix. Use up/down arrows to select a query and left/right arrows to select a key." tabindex="0"></canvas></div><div class="matrix-detail"><p class="mini-label">QUERY → KEY</p><div id="edge-detail"></div><p class="field-note">Hover over a cell to see its weight.<br>Click to select the query position.</p></div></div><div class="heads-heading"><h3 id="heads-heading">All 16 heads in this layer</h3><span class="field-note">For the selected query position · click to switch</span></div><div id="heads-grid" class="heads-grid"></div></section> | |
| </div> | |
| <section class="method-panel" id="method"><p class="eyebrow">MODEL & METHOD</p><h2>How to read this experiment</h2><div class="method-grid"><article><span>01</span><h3>The actual model</h3><p id="method-model">Qwen3-0.6B has 28 decoder layers, 16 query heads, and 8 KV heads. A full causal forward pass includes attention and MLP blocks. With GQA, query head h uses KV head ⌊h / 2⌋.</p></article><article><span>02</span><h3>One LM head for every layer</h3><p id="method-lens">After each layer, apply the final RMSNorm and shared language model head, then softmax over all 151,936 tokens. This is a logit lens: early layers were not trained to predict text independently. The final layer gives the standard model prediction.</p></article><article><span>03</span><h3>Reasoning and the template</h3><p id="method-reasoning">Chat uses the official apply_chat_template with enable_thinking. Disabling reasoning adds an empty <think>…</think> block. In Raw mode, you supply the control tokens and prefix. <think> is an added token with special=false in the tokenizer configuration.</p></article><article><span>04</span><h3>Attention weights</h3><p>Each head has a separate map after softmax, with an attention mask. The mean averages these probabilities across all query heads in the selected layer. Each row sums to approximately 100%. Color amplification only affects the display. Attention is not a complete measure of influence: values, output projections, MLPs, and the residual stream also contribute.</p></article></div><div class="formula-strip">hₜ⁽ˡ⁾ → RMSNorm → LM head → softmax → argmax <span> / </span> <span id="attention-formula">A⁽ˡ,ʰ⁾ = softmax(QKᵀ / √128 + causal mask)</span></div><p class="method-bottom">Float32 computation from the BF16 checkpoint; maps are stored in float16. The inspector limit is <span id="limit-note">512</span> tokens, including the continuation; input is never truncated. <a id="checkpoint-link" href="https://huggingface.co/Qwen/Qwen3-0.6B" target="_blank" rel="noreferrer">Checkpoint ↗</a></p></section> | |
| <footer class="site-footer"><span>Layer lab <i>/</i> Layers & attention inspector</span><span id="footer-model">Model laboratory · Qwen3-0.6B</span></footer> | |
| </div> | |
| </div> | |
| </main> | |
| </body> | |
| </html> | |