|
Download README.md from Pixelkram/biegen: direct link, hf CLI and curl.
- Browser
- Download file 9.85 kB
-
https://huggingface.co/spaces/Pixelkram/biegen/resolve/main/README.md
- Command line
-
hf download hf://spaces/Pixelkram/biegen/README.md
-
curl -L -o README.md https://huggingface.co/spaces/Pixelkram/biegen/resolve/main/README.md
9.85 kB
| title: Biegen | |
| emoji: 🪢 | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: gradio | |
| sdk_version: 6.31.0 | |
| python_version: "3.12" | |
| app_file: app.py | |
| pinned: false | |
| suggested_hardware: l4x1 | |
| models: | |
| - stable-diffusion-v1-5/stable-diffusion-v1-5 | |
| short_description: Stable Diffusion 1.5 von innen biegen, auf einer Karte | |
| # Biegen | |
| Eighteen model-bending tools for Stable Diffusion 1.5, controlled from a map of the network. The tools | |
| come from the Neural_Bending lab (`~/ai/neural_nodes`); twelve of them used to be Tab 4 of the workshop Space | |
| "Erkunden, biegen, brechen", six more were ported for this Space. Here they get a Space of their own, built around one question: where in | |
| the network does a tool act, and what does it change there? | |
| ## The map | |
| The map (`bendmap.js`, a `gr.HTML` component with its own script) is the control surface. It reads top to | |
| bottom in the order SD 1.5 computes; the palette and the tool cards sit in a column beside it (above it on | |
| narrow screens). | |
| - **Text, top:** Prompt A, the token and position tables, CLIP's 12 layers, and the context: 77 vectors, | |
| shown as the prompt's actual CLIP tokens plus the end token and the padding. Prompt B runs through the | |
| same CLIP beside it, lit up only when a tool uses it. Arrows lead down into the UNet's cross-attention. | |
| - **UNet, middle:** the U shape with its real layout: Abwärts 1 to 4 (64 x 64 down to 8 x 8 positions), | |
| Mitte, Aufwärts 1 to 4, and the skip connections across. Each block shows what it contains: Res | |
| (convolutions), Selbst (self-attention) and Quer (cross-attention, the only place where text enters the | |
| image). Abwärts 4 and Aufwärts 1 contain convolutions only. The sampler loop runs around it: latent x_t | |
| on the left; on the right the noise prediction (with and without prompt), the guidance that combines the | |
| two, the sampler step, and the timestep t that goes back into every ResNet; the way back to x_t underneath. | |
| A tool that reads CLIP out early colours the CLIP layers it leaves out. | |
| - **VAE, bottom:** after the last step the latent drops into the VAE decoder; the first image of the last | |
| run and the measured change are shown next to it. | |
| - **Timeline, at the very bottom:** each tool's window from noise (0 %) to image (100 %), with handles you | |
| drag. For "Prompt weg ab …" and "Eingefroren" the knob itself is a moment, so the handle sets the knob. | |
| "Zittern" swells and fades inside its window; the band shows that curve. | |
| Picking a tool from the palette lights up its sites in the slot's colour (up to three tools: orange, | |
| blue, green). Hovering a tool in the palette shows every place it can act, as dashed outlines. The selected | |
| tool's other places are drawn dashed too; clicking one on the map moves the tool there. Every element | |
| explains itself in a tooltip and lists the tools acting on it. | |
| The sites come from `sites.py`. `tests/test_sites.py` installs every tool at every place on a tiny UNet | |
| with SD 1.5's layout, collects the hooks, attention processors, weight edits and sampler steps it really | |
| installs, maps the module names to sites, and checks the result against the table. A site on the map | |
| therefore names exactly the modules the code changes. | |
| ## What is measured | |
| Every run renders the same seeds without tools too (kept in memory, so changing only the tools reuses | |
| them). `measure.py` records, for the first image and every step, a strided sample of each UNet block's | |
| output (at most 320 channels on a 16 x 16 grid) and of the latent, in both runs, and compares them as a | |
| relative change |bent - plain| / |plain|. | |
| - **Block fill:** the change in the first step where anything changed. The latent going into that step | |
| is still identical, so only the block where the tool enters, and everything after it, light up. On the | |
| real model: Norm-Kollaps on Aufwärts 4 fills Aufwärts 4 alone, the second prompt on the up path starts | |
| at Aufwärts 2 (Aufwärts 1 has no cross-attention), the skip tool at the first up block it feeds. | |
| - **Line in each block:** the same change over all steps. | |
| - **Line above the timeline:** the change of the latent over the steps. | |
| - **Images:** each result is shown bent, with the unbent image under a wipe that follows the pointer, | |
| and the mean pixel change ("Δ 14 %"). | |
| The recorder is installed inside the engine's interventions, after the tools, so it sees what they leave | |
| behind (a frozen block's cached output, the mirrored latent). The map greys the measurement out as soon as | |
| the tools change after a run. | |
| ## The tools | |
| Each tool takes a Neural_Bending preset and reduces it to one knob, a place and a window. The defaults come | |
| from the calibration sheets of the workshop project (`python bench.py tools` in `~/ai/boden_workshop_space`); | |
| the last six were calibrated on the horse prompt plus a fox and a fisherman portrait, seeds 42 and 43. | |
| | Tool | Preset | Knob, default | Places | What the sheets showed | | |
| |---|---|---|---|---| | |
| | Wort herausschneiden | word-excision | Stärke 1 (negative emphasises) | word and padding, word only | the horse leaves the picture only when the padding is masked too | | |
| | Zweiter Prompt für die Oberfläche | layered-prompt | share of prompt B 1 | up path, down path, mid only | up path: same horse in marble; down path: a marble statue | | |
| | Wortreihenfolge vergessen | positional-amnesia | Stärke 1 | | painterly, colours move to other objects already at 0.25 | | |
| | Prompt weg ab … | prompt-blind-late | from 25 % | up path, everywhere | the astronaut becomes an ordinary rider in the same pose | | |
| | Spektrale Erosion | spectral-erosion | 12 % of the singular values | | generic people and objects; texture below 6 % | | |
| | Aufmerksamkeit: Nebel und Messer | attention-fog / attention-knife | temperature 4 from 20 % (coarse: 0.25, text: 3) | self-attention, coarse blocks, cross-attention | fog above 1, cut-paper collage below 1 on the coarse blocks, confetti below 1 everywhere | | |
| | Skip-Verbindungen | skip-flood / skip-severed | gain 2.5 | mid resolution, coarse | 0 invents texture, 2.5 over-resolves into colour blotches | | |
| | Symmetrie-Gefängnis | symmetry-prison | 0.3 (top/bottom 0.1, four-fold 0.2) | left/right, top/bottom, four-fold | full strength collapses into ornament on this sampler | | |
| | Eingefroren | frozen-in-thought | from 8 % | mid and up 2, mid only | early freezing leaves a soft fossil of the noise | | |
| | Kochen | boiling | noise 0.3 | | surfaces simmer, dissolves from 0.6 | | |
| | Norm-Kollaps | norm-collapse | Stärke 1 | up 4 (green), up 3 (orange) | woven high-contrast colour, subject readable | | |
| | VAE-Fieber | vae-fever | 3 bits | | invisible at 5 bits, fever colours at 3, abstract at 2 | | |
| | CLIP-Aphasie | clip-aphasia | read out 7 layers early | with the padding blanked, early read-out only | the fisherman becomes a generic man in front of a building; from 11 layers an unrelated photograph | | |
| | Guidance umkehren | guidance-inverted | 1 (0.5 = no guidance) | | 0.5 gives the same image for every prompt at a seed; 1 the anti-prompt (packaging, food, a cushion on a lawn); a window from 40 % keeps the layout | | |
| | Ohne Mitte | bottleneck-removed | 0.6 | mid and up 1-2, mid only | mid alone: a clean photo, at most a new pose; with up 1-2 the subject dissolves into fibres and fog, texture only from 0.8 | | |
| | Zittern | the-shakes | weight noise 1 std, 10-65 %, half a sine | self-attention, cross-attention | self: soft and hazy as behind fogged glass; cross: grainy like an etching; over the whole run no picture survives | | |
| | Zeitlüge | time-displaced | timestep +400 | | painterly mosaic at +400; -200 hard black and white, -400 pattern only | | |
| | Auflösungsgeist | resolution-ghost | factor 2 from 30 % | ResNets of down 1-2, the down path only | ResNets: a photo rebuilt from a thumbnail; the path only at the same factor stays almost clean, the skips carry the sharp version | | |
| Temperature, word masking and the second prompt run in one attention processor on PyTorch's fused | |
| attention, so no attention maps are kept in memory. The engine has no special code for the newer tools: | |
| "Guidance umkehren" rewrites the prompt half of the UNet's prediction so the engine's own guidance formula | |
| comes out reversed, "Zeitlüge" is a pre-hook on `time_proj`, "Zittern" adds `a * (x @ noise)` to the | |
| attention projections without touching their weights, and "CLIP-Aphasie" wraps the engine's text encoding | |
| for one call (encodings are cached under the layer they read). | |
| Every number may go past its slider (the knob's number box, steps, guidance), the seed may be negative. | |
| Windows are clamped to 0 to 100 % with a note. One job may cost at most a 4 x 4 grid at the default steps, | |
| rendered with and without tools; bigger requests are refused with a message, the outputs stay as they were. | |
| ## Files | |
| | File | | | |
| |---|---| | |
| | `app.py` | the Gradio app: prompts, map, settings, results, status with one recipe line per tool | | |
| | `bendmap.js`, `bendmap.css` | the map, the palette and the tool cards | | |
| | `compare.js` | the before/after wipe | | |
| | `sites.py` | where each tool acts, and module names to sites | | |
| | `measure.py` | the activation recorder and the effect | | |
| | `engine.py`, `engine_sd15.py`, `family.py`, `tools_sd15.py`, `safety.py` | symlinks to `~/ai/boden_workshop_space`, one engine for both Spaces | | |
| The NSFW output filter (`safety.py`) is on, as in the workshop Space. With the secret `WORKSHOP_PASSWORD` | |
| set, the Space asks for the user name `workshop` and that password. | |
| ## Development | |
| ``` | |
| .venv/bin/python -m pytest # CPU: sites, measurement, app (tiny model) | |
| BIEGEN_GPU_TESTS=1 .venv/bin/python -m pytest -m gpu # every tool at every place on the real model | |
| GRADIO_SERVER_PORT=7870 .venv/bin/python app.py | |
| python deploy_space.py Pixelkram/biegen --token-name claude_write | |
| ``` | |
| `.venv` is the workshop project's environment (Gradio 6.31, torch 2.10, diffusers 0.42.1). | |