stable-diffusion-v1-1: zeromodels Keras 3 conversion of CompVis/stable-diffusion-v1-1

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,109 @@
+---
+pipeline_tag: text-to-image
+license: creativeml-openrail-m
+base_model: CompVis/stable-diffusion-v1-1
+library_name: zeromodels
+language:
+- en
+tags:
+- keras
+- zeromodels
+- stable-diffusion
+- stable-diffusion-diffusers
+- text-to-image
+- diffusion
+- latent-diffusion
+- arxiv:2112.10752
+- pytorch
+- jax
+- tf
+---
+*See [our collection](https://huggingface.co/zeromodels) for all Stable Diffusion 1.x checkpoints.*
+
+# Run Stable Diffusion 1.x with Keras 3: JAX, PyTorch, or TensorFlow
+
+[![GitHub](https://img.shields.io/badge/GitHub-ZeroModels-181717?logo=github)](https://github.com/IMvision12/ZeroModels) [![Docs](https://img.shields.io/badge/Docs-Stable_Diffusion_1.x-1f6feb)](https://imvision12.github.io/ZeroModels/stable_diffusion/) [![HuggingFace](https://img.shields.io/badge/HuggingFace-Stable_Diffusion_1.x-ffd21e?logo=huggingface&logoColor=black)](https://huggingface.co/zeromodels)
+
+# zeromodels/stable-diffusion-v1-1
+
+Paper: [High-Resolution Image Synthesis with Latent Diffusion Models (arXiv:2112.10752)](https://arxiv.org/abs/2112.10752) | [HF Papers](https://huggingface.co/papers/2112.10752)
+
+Pure-**Keras 3** conversion of [`CompVis/stable-diffusion-v1-1`](https://huggingface.co/CompVis/stable-diffusion-v1-1) for
+[zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on
+**TensorFlow / Torch / JAX**. The whole text-to-image model ships as **one container**:
+the UNet denoiser, the VAE and the CLIP ViT-L/14 text encoder in `model.weights.h5`
+(1.07B parameters, 3.97 GB), plus `zm_config.json` (the three
+component configs, the checkpoint's `PNDMScheduler` schedule with its `epsilon` objective
+and the default generation settings) and the tokenizer as `tokenizer.json`. Weights are stored in **float32**, exactly as released.
+This checkpoint generates **512x512** images (a 64x64 latent).
+
+For model details, intended use and limitations, see the upstream
+[model card](https://huggingface.co/CompVis/stable-diffusion-v1-1).
+
+## Architecture
+
+| Component | zeromodels class | Details |
+| --- | --- | --- |
+| Denoiser | `UNet2DConditionModel` | (320, 640, 1280, 1280) channels, 2 ResNet blocks per level, 8 attention heads on the 768-d text context, 1x1 conv token projection, 64x64x4 latent |
+| Autoencoder | `AutoencoderKL` | (128, 256, 512, 512) channels, x8 spatial compression to 4 latent channels, `scaling_factor` 0.18215 |
+| Text encoder | `CLIPTextModel` | CLIP ViT-L/14 text encoder: 768-d, 12 layers, 12 heads, 77 tokens, `quick_gelu` |
+| Scheduler | `PNDMScheduler` | scaled_linear betas 0.00085 to 0.012 over 1000 steps, `epsilon`; DDIM / PNDM / Euler / Euler-ancestral are drop-in |
+
+## Quick start
+
+```python
+import os
+os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"
+
+from PIL import Image
+from zeromodels.models.stable_diffusion import StableDiffusionTextToImage, StableDiffusionTokenizer
+
+model = StableDiffusionTextToImage.from_weights("zeromodels/stable-diffusion-v1-1")
+tokenizer = StableDiffusionTokenizer.from_weights("zeromodels/stable-diffusion-v1-1")
+
+inputs = tokenizer("a photograph of an astronaut riding a horse")
+images = model.generate(**inputs, num_inference_steps=50, guidance_scale=7.5, seed=0)
+Image.fromarray(images[0]).save("astronaut.png")  # (512, 512, 3) uint8
+```
+
+`generate` takes the tokenizer's `input_ids` (batch them for several prompts), an optional
+`negative_input_ids` (tokenize the negative prompt), `num_inference_steps`, `guidance_scale`,
+a `seed`, or explicit `latents` of shape `(batch, 64, 64, 4)` for results that are
+identical across backends.
+
+Load any Stable Diffusion 1.x checkpoint the same way with `from_weights("zeromodels/<variant>")`:
+
+| Variant | Hub | Training |
+| --- | --- | --- |
+| `stable-diffusion-v1-1` | [zeromodels/stable-diffusion-v1-1](https://huggingface.co/zeromodels/stable-diffusion-v1-1) | 237k steps at 256px on laion2B-en, then 194k steps at 512px on laion-high-resolution |
+| `stable-diffusion-v1-2` | [zeromodels/stable-diffusion-v1-2](https://huggingface.co/zeromodels/stable-diffusion-v1-2) | v1-1 + 515k steps at 512px on laion-aesthetics v2 5+ |
+| `stable-diffusion-v1-3` | [zeromodels/stable-diffusion-v1-3](https://huggingface.co/zeromodels/stable-diffusion-v1-3) | v1-2 + 195k steps at 512px, 10% text-conditioning dropout (classifier-free guidance) |
+| `stable-diffusion-v1-4` | [zeromodels/stable-diffusion-v1-4](https://huggingface.co/zeromodels/stable-diffusion-v1-4) | v1-2 + 225k steps at 512px, 10% text-conditioning dropout (classifier-free guidance) |
+| `stable-diffusion-v1-5` | [zeromodels/stable-diffusion-v1-5](https://huggingface.co/zeromodels/stable-diffusion-v1-5) | v1-2 + 595k steps at 512px, 10% text-conditioning dropout (classifier-free guidance) |
+
+## Tips
+
+- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
+- The graphs are built for 512px. Pass `unet_sample_size=<px / 8>, vae_sample_size=<px>` to
+  `from_weights` to build for another multiple of 64px (the weights are resolution-independent).
+- Swap the sampler any time: `model.scheduler = EulerDiscreteScheduler.from_config(model.config.scheduler_config)`
+  (`zeromodels.base.base_scheduler`).
+- `StableDiffusionModel.from_weights(...)` loads the same repo as the bare container
+  (UNet / VAE / text encoder as `.unet` / `.vae` / `.text_encoder`) without the generation loop.
+
+- Both `channels_last` and `channels_first` are supported (`keras.config.set_image_data_format`
+  before loading); `generate` always returns `(batch, H, W, 3)` uint8.
+- On-the-fly `hf:` conversion is not supported for diffusion models; the checkpoints are
+  hosted here, converted once.
+- See the [Stable Diffusion 1.x docs](https://imvision12.github.io/ZeroModels/stable_diffusion/).
+
+## License
+
+The weights are redistributed under the
+[CreativeML OpenRAIL-M](https://huggingface.co/spaces/CompVis/stable-diffusion-license) of the upstream checkpoint,
+including its use-based restrictions. By using them you agree to those terms.
+
+## Special Thanks
+
+Thank you to the CompVis group at LMU Munich, Runway and Stability AI for training and releasing Stable Diffusion, and to the
+Hugging Face diffusers team, whose implementation this port was verified against.

model.weights.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d98c9d89856688f47abf88c7ce92214d37f2fe5464def3cbde6eb7e19e8b99df
+size 4267129096

zm_config.json

@@ -0,0 +1,92 @@
+{
+  "library_name": "zeromodels",
+  "zeromodels_version": "1.3.3",
+  "model_module": "zeromodels.models.stable_diffusion",
+  "model_class": "StableDiffusionModel",
+  "variant": "stable-diffusion-v1-1",
+  "weights": "model.weights.h5",
+  "schema_version": 2,
+  "weight_dtype": "float32",
+  "model_type": "stable_diffusion",
+  "unet_config": {
+    "sample_size": 64,
+    "in_channels": 4,
+    "out_channels": 4,
+    "down_block_types": [
+      "CrossAttnDownBlock2D",
+      "CrossAttnDownBlock2D",
+      "CrossAttnDownBlock2D",
+      "DownBlock2D"
+    ],
+    "up_block_types": [
+      "UpBlock2D",
+      "CrossAttnUpBlock2D",
+      "CrossAttnUpBlock2D",
+      "CrossAttnUpBlock2D"
+    ],
+    "block_out_channels": [
+      320,
+      640,
+      1280,
+      1280
+    ],
+    "layers_per_block": 2,
+    "cross_attention_dim": 768,
+    "num_attention_heads": 8,
+    "norm_num_groups": 32,
+    "use_linear_projection": false,
+    "transformer_layers_per_block": 1,
+    "addition_embed_type": null,
+    "addition_time_embed_dim": 256,
+    "projection_class_embeddings_input_dim": null,
+    "num_time_ids": 6,
+    "text_seq_len": 77
+  },
+  "vae_config": {
+    "in_channels": 3,
+    "out_channels": 3,
+    "latent_channels": 4,
+    "block_out_channels": [
+      128,
+      256,
+      512,
+      512
+    ],
+    "layers_per_block": 2,
+    "norm_num_groups": 32,
+    "sample_size": 512,
+    "scaling_factor": 0.18215,
+    "force_upcast": false,
+    "shift_factor": 0.0,
+    "use_quant_conv": true,
+    "use_post_quant_conv": true
+  },
+  "text_config": {
+    "hidden_dim": 768,
+    "num_heads": 12,
+    "num_layers": 12,
+    "mlp_ratio": 4.0,
+    "vocab_size": 49408,
+    "max_seq_len": 77
+  },
+  "hidden_act": "quick_gelu",
+  "layer_norm_eps": 1e-05,
+  "bos_token_id": 49406,
+  "eos_token_id": 49407,
+  "pad_token_id": 49407,
+  "scheduler_config": {
+    "_class_name": "PNDMScheduler",
+    "beta_end": 0.012,
+    "beta_schedule": "scaled_linear",
+    "beta_start": 0.00085,
+    "num_train_timesteps": 1000,
+    "set_alpha_to_one": false,
+    "skip_prk_steps": true,
+    "steps_offset": 1,
+    "trained_betas": null
+  },
+  "generate_args": {
+    "num_inference_steps": 50,
+    "guidance_scale": 7.5
+  }
+}