--- title: Biegen emoji: 🪢 colorFrom: indigo colorTo: purple sdk: gradio sdk_version: 6.31.0 python_version: "3.12" app_file: app.py pinned: false suggested_hardware: l4x1 models: - stable-diffusion-v1-5/stable-diffusion-v1-5 short_description: Stable Diffusion 1.5 von innen biegen, auf einer Karte --- # Biegen Eighteen model-bending tools for Stable Diffusion 1.5, controlled from a map of the network. The tools come from the Neural_Bending lab (`~/ai/neural_nodes`); twelve of them used to be Tab 4 of the workshop Space "Erkunden, biegen, brechen", six more were ported for this Space. Here they get a Space of their own, built around one question: where in the network does a tool act, and what does it change there? ## The map The map (`bendmap.js`, a `gr.HTML` component with its own script) is the control surface. It reads top to bottom in the order SD 1.5 computes; the palette and the tool cards sit in a column beside it (above it on narrow screens). - **Text, top:** Prompt A, the token and position tables, CLIP's 12 layers, and the context: 77 vectors, shown as the prompt's actual CLIP tokens plus the end token and the padding. Prompt B runs through the same CLIP beside it, lit up only when a tool uses it. Arrows lead down into the UNet's cross-attention. - **UNet, middle:** the U shape with its real layout: Abwärts 1 to 4 (64 x 64 down to 8 x 8 positions), Mitte, Aufwärts 1 to 4, and the skip connections across. Each block shows what it contains: Res (convolutions), Selbst (self-attention) and Quer (cross-attention, the only place where text enters the image). Abwärts 4 and Aufwärts 1 contain convolutions only. The sampler loop runs around it: latent x_t on the left; on the right the noise prediction (with and without prompt), the guidance that combines the two, the sampler step, and the timestep t that goes back into every ResNet; the way back to x_t underneath. A tool that reads CLIP out early colours the CLIP layers it leaves out. - **VAE, bottom:** after the last step the latent drops into the VAE decoder; the first image of the last run and the measured change are shown next to it. - **Timeline, at the very bottom:** each tool's window from noise (0 %) to image (100 %), with handles you drag. For "Prompt weg ab …" and "Eingefroren" the knob itself is a moment, so the handle sets the knob. "Zittern" swells and fades inside its window; the band shows that curve. Picking a tool from the palette lights up its sites in the slot's colour (up to three tools: orange, blue, green). Hovering a tool in the palette shows every place it can act, as dashed outlines. The selected tool's other places are drawn dashed too; clicking one on the map moves the tool there. Every element explains itself in a tooltip and lists the tools acting on it. The sites come from `sites.py`. `tests/test_sites.py` installs every tool at every place on a tiny UNet with SD 1.5's layout, collects the hooks, attention processors, weight edits and sampler steps it really installs, maps the module names to sites, and checks the result against the table. A site on the map therefore names exactly the modules the code changes. ## What is measured Every run renders the same seeds without tools too (kept in memory, so changing only the tools reuses them). `measure.py` records, for the first image and every step, a strided sample of each UNet block's output (at most 320 channels on a 16 x 16 grid) and of the latent, in both runs, and compares them as a relative change |bent - plain| / |plain|. - **Block fill:** the change in the first step where anything changed. The latent going into that step is still identical, so only the block where the tool enters, and everything after it, light up. On the real model: Norm-Kollaps on Aufwärts 4 fills Aufwärts 4 alone, the second prompt on the up path starts at Aufwärts 2 (Aufwärts 1 has no cross-attention), the skip tool at the first up block it feeds. - **Line in each block:** the same change over all steps. - **Line above the timeline:** the change of the latent over the steps. - **Images:** each result is shown bent, with the unbent image under a wipe that follows the pointer, and the mean pixel change ("Δ 14 %"). The recorder is installed inside the engine's interventions, after the tools, so it sees what they leave behind (a frozen block's cached output, the mirrored latent). The map greys the measurement out as soon as the tools change after a run. ## The tools Each tool takes a Neural_Bending preset and reduces it to one knob, a place and a window. The defaults come from the calibration sheets of the workshop project (`python bench.py tools` in `~/ai/boden_workshop_space`); the last six were calibrated on the horse prompt plus a fox and a fisherman portrait, seeds 42 and 43. | Tool | Preset | Knob, default | Places | What the sheets showed | |---|---|---|---|---| | Wort herausschneiden | word-excision | Stärke 1 (negative emphasises) | word and padding, word only | the horse leaves the picture only when the padding is masked too | | Zweiter Prompt für die Oberfläche | layered-prompt | share of prompt B 1 | up path, down path, mid only | up path: same horse in marble; down path: a marble statue | | Wortreihenfolge vergessen | positional-amnesia | Stärke 1 | | painterly, colours move to other objects already at 0.25 | | Prompt weg ab … | prompt-blind-late | from 25 % | up path, everywhere | the astronaut becomes an ordinary rider in the same pose | | Spektrale Erosion | spectral-erosion | 12 % of the singular values | | generic people and objects; texture below 6 % | | Aufmerksamkeit: Nebel und Messer | attention-fog / attention-knife | temperature 4 from 20 % (coarse: 0.25, text: 3) | self-attention, coarse blocks, cross-attention | fog above 1, cut-paper collage below 1 on the coarse blocks, confetti below 1 everywhere | | Skip-Verbindungen | skip-flood / skip-severed | gain 2.5 | mid resolution, coarse | 0 invents texture, 2.5 over-resolves into colour blotches | | Symmetrie-Gefängnis | symmetry-prison | 0.3 (top/bottom 0.1, four-fold 0.2) | left/right, top/bottom, four-fold | full strength collapses into ornament on this sampler | | Eingefroren | frozen-in-thought | from 8 % | mid and up 2, mid only | early freezing leaves a soft fossil of the noise | | Kochen | boiling | noise 0.3 | | surfaces simmer, dissolves from 0.6 | | Norm-Kollaps | norm-collapse | Stärke 1 | up 4 (green), up 3 (orange) | woven high-contrast colour, subject readable | | VAE-Fieber | vae-fever | 3 bits | | invisible at 5 bits, fever colours at 3, abstract at 2 | | CLIP-Aphasie | clip-aphasia | read out 7 layers early | with the padding blanked, early read-out only | the fisherman becomes a generic man in front of a building; from 11 layers an unrelated photograph | | Guidance umkehren | guidance-inverted | 1 (0.5 = no guidance) | | 0.5 gives the same image for every prompt at a seed; 1 the anti-prompt (packaging, food, a cushion on a lawn); a window from 40 % keeps the layout | | Ohne Mitte | bottleneck-removed | 0.6 | mid and up 1-2, mid only | mid alone: a clean photo, at most a new pose; with up 1-2 the subject dissolves into fibres and fog, texture only from 0.8 | | Zittern | the-shakes | weight noise 1 std, 10-65 %, half a sine | self-attention, cross-attention | self: soft and hazy as behind fogged glass; cross: grainy like an etching; over the whole run no picture survives | | Zeitlüge | time-displaced | timestep +400 | | painterly mosaic at +400; -200 hard black and white, -400 pattern only | | Auflösungsgeist | resolution-ghost | factor 2 from 30 % | ResNets of down 1-2, the down path only | ResNets: a photo rebuilt from a thumbnail; the path only at the same factor stays almost clean, the skips carry the sharp version | Temperature, word masking and the second prompt run in one attention processor on PyTorch's fused attention, so no attention maps are kept in memory. The engine has no special code for the newer tools: "Guidance umkehren" rewrites the prompt half of the UNet's prediction so the engine's own guidance formula comes out reversed, "Zeitlüge" is a pre-hook on `time_proj`, "Zittern" adds `a * (x @ noise)` to the attention projections without touching their weights, and "CLIP-Aphasie" wraps the engine's text encoding for one call (encodings are cached under the layer they read). Every number may go past its slider (the knob's number box, steps, guidance), the seed may be negative. Windows are clamped to 0 to 100 % with a note. One job may cost at most a 4 x 4 grid at the default steps, rendered with and without tools; bigger requests are refused with a message, the outputs stay as they were. ## Files | File | | |---|---| | `app.py` | the Gradio app: prompts, map, settings, results, status with one recipe line per tool | | `bendmap.js`, `bendmap.css` | the map, the palette and the tool cards | | `compare.js` | the before/after wipe | | `sites.py` | where each tool acts, and module names to sites | | `measure.py` | the activation recorder and the effect | | `engine.py`, `engine_sd15.py`, `family.py`, `tools_sd15.py`, `safety.py` | symlinks to `~/ai/boden_workshop_space`, one engine for both Spaces | The NSFW output filter (`safety.py`) is on, as in the workshop Space. With the secret `WORKSHOP_PASSWORD` set, the Space asks for the user name `workshop` and that password. ## Development ``` .venv/bin/python -m pytest # CPU: sites, measurement, app (tiny model) BIEGEN_GPU_TESTS=1 .venv/bin/python -m pytest -m gpu # every tool at every place on the real model GRADIO_SERVER_PORT=7870 .venv/bin/python app.py python deploy_space.py Pixelkram/biegen --token-name claude_write ``` `.venv` is the workshop project's environment (Gradio 6.31, torch 2.10, diffusers 0.42.1).