biegen / README.md
Pixelkram's picture
Deploy app
2fd30a7 verified
|
Raw History Blame Contribute Delete
9.85 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade
metadata
title: Biegen
emoji: 🪢
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.31.0
python_version: '3.12'
app_file: app.py
pinned: false
suggested_hardware: l4x1
models:
  - stable-diffusion-v1-5/stable-diffusion-v1-5
short_description: Stable Diffusion 1.5 von innen biegen, auf einer Karte

Biegen

Eighteen model-bending tools for Stable Diffusion 1.5, controlled from a map of the network. The tools come from the Neural_Bending lab (~/ai/neural_nodes); twelve of them used to be Tab 4 of the workshop Space "Erkunden, biegen, brechen", six more were ported for this Space. Here they get a Space of their own, built around one question: where in the network does a tool act, and what does it change there?

The map

The map (bendmap.js, a gr.HTML component with its own script) is the control surface. It reads top to bottom in the order SD 1.5 computes; the palette and the tool cards sit in a column beside it (above it on narrow screens).

  • Text, top: Prompt A, the token and position tables, CLIP's 12 layers, and the context: 77 vectors, shown as the prompt's actual CLIP tokens plus the end token and the padding. Prompt B runs through the same CLIP beside it, lit up only when a tool uses it. Arrows lead down into the UNet's cross-attention.
  • UNet, middle: the U shape with its real layout: Abwärts 1 to 4 (64 x 64 down to 8 x 8 positions), Mitte, Aufwärts 1 to 4, and the skip connections across. Each block shows what it contains: Res (convolutions), Selbst (self-attention) and Quer (cross-attention, the only place where text enters the image). Abwärts 4 and Aufwärts 1 contain convolutions only. The sampler loop runs around it: latent x_t on the left; on the right the noise prediction (with and without prompt), the guidance that combines the two, the sampler step, and the timestep t that goes back into every ResNet; the way back to x_t underneath. A tool that reads CLIP out early colours the CLIP layers it leaves out.
  • VAE, bottom: after the last step the latent drops into the VAE decoder; the first image of the last run and the measured change are shown next to it.
  • Timeline, at the very bottom: each tool's window from noise (0 %) to image (100 %), with handles you drag. For "Prompt weg ab …" and "Eingefroren" the knob itself is a moment, so the handle sets the knob. "Zittern" swells and fades inside its window; the band shows that curve.

Picking a tool from the palette lights up its sites in the slot's colour (up to three tools: orange, blue, green). Hovering a tool in the palette shows every place it can act, as dashed outlines. The selected tool's other places are drawn dashed too; clicking one on the map moves the tool there. Every element explains itself in a tooltip and lists the tools acting on it.

The sites come from sites.py. tests/test_sites.py installs every tool at every place on a tiny UNet with SD 1.5's layout, collects the hooks, attention processors, weight edits and sampler steps it really installs, maps the module names to sites, and checks the result against the table. A site on the map therefore names exactly the modules the code changes.

What is measured

Every run renders the same seeds without tools too (kept in memory, so changing only the tools reuses them). measure.py records, for the first image and every step, a strided sample of each UNet block's output (at most 320 channels on a 16 x 16 grid) and of the latent, in both runs, and compares them as a relative change |bent - plain| / |plain|.

  • Block fill: the change in the first step where anything changed. The latent going into that step is still identical, so only the block where the tool enters, and everything after it, light up. On the real model: Norm-Kollaps on Aufwärts 4 fills Aufwärts 4 alone, the second prompt on the up path starts at Aufwärts 2 (Aufwärts 1 has no cross-attention), the skip tool at the first up block it feeds.
  • Line in each block: the same change over all steps.
  • Line above the timeline: the change of the latent over the steps.
  • Images: each result is shown bent, with the unbent image under a wipe that follows the pointer, and the mean pixel change ("Δ 14 %").

The recorder is installed inside the engine's interventions, after the tools, so it sees what they leave behind (a frozen block's cached output, the mirrored latent). The map greys the measurement out as soon as the tools change after a run.

The tools

Each tool takes a Neural_Bending preset and reduces it to one knob, a place and a window. The defaults come from the calibration sheets of the workshop project (python bench.py tools in ~/ai/boden_workshop_space); the last six were calibrated on the horse prompt plus a fox and a fisherman portrait, seeds 42 and 43.

Tool Preset Knob, default Places What the sheets showed
Wort herausschneiden word-excision Stärke 1 (negative emphasises) word and padding, word only the horse leaves the picture only when the padding is masked too
Zweiter Prompt für die Oberfläche layered-prompt share of prompt B 1 up path, down path, mid only up path: same horse in marble; down path: a marble statue
Wortreihenfolge vergessen positional-amnesia Stärke 1 painterly, colours move to other objects already at 0.25
Prompt weg ab … prompt-blind-late from 25 % up path, everywhere the astronaut becomes an ordinary rider in the same pose
Spektrale Erosion spectral-erosion 12 % of the singular values generic people and objects; texture below 6 %
Aufmerksamkeit: Nebel und Messer attention-fog / attention-knife temperature 4 from 20 % (coarse: 0.25, text: 3) self-attention, coarse blocks, cross-attention fog above 1, cut-paper collage below 1 on the coarse blocks, confetti below 1 everywhere
Skip-Verbindungen skip-flood / skip-severed gain 2.5 mid resolution, coarse 0 invents texture, 2.5 over-resolves into colour blotches
Symmetrie-Gefängnis symmetry-prison 0.3 (top/bottom 0.1, four-fold 0.2) left/right, top/bottom, four-fold full strength collapses into ornament on this sampler
Eingefroren frozen-in-thought from 8 % mid and up 2, mid only early freezing leaves a soft fossil of the noise
Kochen boiling noise 0.3 surfaces simmer, dissolves from 0.6
Norm-Kollaps norm-collapse Stärke 1 up 4 (green), up 3 (orange) woven high-contrast colour, subject readable
VAE-Fieber vae-fever 3 bits invisible at 5 bits, fever colours at 3, abstract at 2
CLIP-Aphasie clip-aphasia read out 7 layers early with the padding blanked, early read-out only the fisherman becomes a generic man in front of a building; from 11 layers an unrelated photograph
Guidance umkehren guidance-inverted 1 (0.5 = no guidance) 0.5 gives the same image for every prompt at a seed; 1 the anti-prompt (packaging, food, a cushion on a lawn); a window from 40 % keeps the layout
Ohne Mitte bottleneck-removed 0.6 mid and up 1-2, mid only mid alone: a clean photo, at most a new pose; with up 1-2 the subject dissolves into fibres and fog, texture only from 0.8
Zittern the-shakes weight noise 1 std, 10-65 %, half a sine self-attention, cross-attention self: soft and hazy as behind fogged glass; cross: grainy like an etching; over the whole run no picture survives
Zeitlüge time-displaced timestep +400 painterly mosaic at +400; -200 hard black and white, -400 pattern only
Auflösungsgeist resolution-ghost factor 2 from 30 % ResNets of down 1-2, the down path only ResNets: a photo rebuilt from a thumbnail; the path only at the same factor stays almost clean, the skips carry the sharp version

Temperature, word masking and the second prompt run in one attention processor on PyTorch's fused attention, so no attention maps are kept in memory. The engine has no special code for the newer tools: "Guidance umkehren" rewrites the prompt half of the UNet's prediction so the engine's own guidance formula comes out reversed, "Zeitlüge" is a pre-hook on time_proj, "Zittern" adds a * (x @ noise) to the attention projections without touching their weights, and "CLIP-Aphasie" wraps the engine's text encoding for one call (encodings are cached under the layer they read).

Every number may go past its slider (the knob's number box, steps, guidance), the seed may be negative. Windows are clamped to 0 to 100 % with a note. One job may cost at most a 4 x 4 grid at the default steps, rendered with and without tools; bigger requests are refused with a message, the outputs stay as they were.

Files

File
app.py the Gradio app: prompts, map, settings, results, status with one recipe line per tool
bendmap.js, bendmap.css the map, the palette and the tool cards
compare.js the before/after wipe
sites.py where each tool acts, and module names to sites
measure.py the activation recorder and the effect
engine.py, engine_sd15.py, family.py, tools_sd15.py, safety.py symlinks to ~/ai/boden_workshop_space, one engine for both Spaces

The NSFW output filter (safety.py) is on, as in the workshop Space. With the secret WORKSHOP_PASSWORD set, the Space asks for the user name workshop and that password.

Development

.venv/bin/python -m pytest                          # CPU: sites, measurement, app (tiny model)
BIEGEN_GPU_TESTS=1 .venv/bin/python -m pytest -m gpu   # every tool at every place on the real model
GRADIO_SERVER_PORT=7870 .venv/bin/python app.py
python deploy_space.py Pixelkram/biegen --token-name claude_write

.venv is the workshop project's environment (Gradio 6.31, torch 2.10, diffusers 0.42.1).