utkucoban
/

NanoDiffuser

@@ -1,271 +0,0 @@
----
-license: openrail++
-pipeline_tag: text-to-image
-language:
-  - en
-tags:
-  - onnx
-  - onnxruntime
-  - webgpu
-  - stable-diffusion
-  - quantized
-  - browser
-inference: false
----
-# NanoDiffuser
-<p align="center">
-  <img src="./NDLGO2k.png" alt="NanoDiffuser logo" width="360" />
-</p>
-**One-step text-to-image diffusion, running locally in your browser.**
-NanoDiffuser uses a **custom-modified version of Stable Diffusion, adapted and quantized to fit a tight sub-400 MiB model budget**. Compact ONNX graphs, a JavaScript inference pipeline, and a minimal interface generate 512 × 512 images using the visitor's GPU through WebGPU. Hosting serves static files; image generation requires no inference server, API key, or server-side GPU.
-The modifications cover model conversion, mixed-precision quantization, browser compatibility, local inference, and static deployment. No additional training or fine-tuning was performed for this package. The approximately 382 MiB budget covers the inference graphs, weights, tokenizer, and configuration; JavaScript and WebAssembly assets are additional. In decimal units, the model assets are approximately 400.5 MB.
-## At a glance
-| Property | Configuration |
-| --- | --- |
-| Base | Custom-modified Stable Diffusion adapted for a sub-400 MiB model budget |
-| Output | One 512 × 512 RGB image, downloadable as PNG |
-| Sampling | One denoising step, classifier-free guidance disabled |
-| Scheduler | Exported one-step DEIS coefficients |
-| Text encoder | CLIP, asymmetric Q4 MatMul/Gather weights |
-| Denoiser | U-Net, Q4 matrix weights + per-channel INT8 convolution storage, with FP16 retained where needed |
-| Decoder | TAESD / `AutoencoderTiny`, FP16 |
-| Model assets | Approximately 382 MiB, excluding JavaScript and WebAssembly runtime assets |
-| Execution | ONNX Runtime Web with WebGPU; tokenization and orchestration in JavaScript |
-| Hosting | Static HTML, CSS, JavaScript, WebAssembly, and model files |
-## Getting started
-Open NanoDiffuser in a browser that exposes WebGPU and the `shader-f16` feature. The interface checks these capabilities before enabling generation. Chrome or Edge with hardware acceleration enabled are the suggested starting point; support also depends on the GPU, driver, operating system, and browser configuration.
-1. Enter a prompt or choose an example.
-2. Optionally open **Settings** to choose a fixed seed. Random seeds are enabled by default.
-3. Select **Generate image**. On the first run, the application downloads the model assets and loads the inference runtime. Progress shows the download and generation stages.
-4. Use **Download** to save the result as a PNG. The displayed seed belongs to that image.
-You can also use **Settings → Choose folder** to select an extracted local model package containing `manifest.json`, all three ONNX graphs, their external `.data` files, and the tokenizer files. Selecting a folder reads its files locally.
-The exported sequence length is **77 CLIP tokens**, including special tokens. Longer prompts are truncated during tokenization even though the text box accepts up to 1,000 characters. Resolution, batch size, sampling steps, and guidance are fixed; this interface does not provide negative prompts or image-to-image generation.
-## How generation works
-```text
-Prompt → CLIP tokenizer → Q4 CLIP encoder → FP16 text embeddings
-Seed → Gaussian noise → one-step U-Net → scheduler update
-                         ↓
-                   FP16 TAESD → pixels → canvas → PNG
-```
-The tokenizer runs locally through Transformers.js, pads or truncates the prompt to 77 tokens, and supplies an INT32 `[1, 77]` tensor. CLIP produces `[1, 77, 768]` text embeddings, which the browser converts to FP16 for the U-Net.
-A seeded JavaScript random generator and Box–Muller transform create Gaussian noise with shape `[1, 4, 64, 64]`. The U-Net receives the scaled noise, text embeddings, and timestep `999`. Its prediction is combined with the initial latent using coefficients saved from the one-step DEIS scheduler:
-```text
-initial_latent = noise × initNoiseSigma
-model_input   = initial_latent × modelInputScale
-denoised      = sampleCoefficient × initial_latent
-              + outputCoefficient × noise_prediction
-```
-The current manifest contains:
-```json
-{
-  "timestep": 999.0,
-  "initNoiseSigma": 1.0,
-  "modelInputScale": 1.0,
-  "sampleCoefficient": 14.642590522766113,
-  "outputCoefficient": -14.579278945922852
-}
-```
-These values are part of this export's sampling contract. The browser uses the recorded values directly. It does not reconstruct Diffusers' scheduler internals or apply an extra classifier-free guidance pass.
-The denoised latent is converted to FP16 and passed to TAESD. The decoder wrapper maps its output into `[0, 1]`; JavaScript converts the channel-first output into 8-bit RGBA pixels for the canvas and PNG export.
-Fixed seeds make the JavaScript noise sequence repeatable. They do not imply pixel-identical results to PyTorch, which uses a different random generator, or across different browser/GPU implementations.
-## Conversion and quantization
-The supplied `sdxs512_webgpu_colab.ipynb` exports three separate graphs using PyTorch's ONNX exporter, evaluation mode, constant folding, and fixed dimensions. Export starts with ONNX opset 17; the U-Net is subsequently set to opset 21 for its FP16 dequantization operations.
-| Component | Conversion | Graph + external weights |
-| --- | --- | ---: |
-| CLIP | FP32 export; asymmetric Q4 weight-only quantization of eligible MatMul and Gather operations | 63.36 MiB |
-| U-Net | FP16 export; mixed Q4 matrix weights and per-output-channel INT8 convolution storage | 312.71 MiB |
-| TAESD | FP16 decoder export | 2.36 MiB |
-Q4 quantization uses ONNX Runtime's weight-only quantizer with **block size 128**, asymmetric zero-points, `QOperator` format, and `accuracy_level=2`. Quantized matrix multiplication uses packed `MatMulNBits` weights. The graph checks require packed UINT8 zero-points.
-U-Net convolution weights are quantized per output channel to signed INT8, with FP16 scales and a `DequantizeLinear` node before convolution. **INT8 reduces weight storage; the convolution still executes in floating point.** Input/output convolutions and timestep-projection MatMuls are excluded from their respective quantization passes and retain FP16 weights.
-The U-Net uses Diffusers' explicit `AttnProcessor` during export. TAESD is loaded from the source checkpoint's `vae` subfolder; the export includes its decoder path.
-Compatibility handling includes:
-- Keeping CLIP export and LayerNormalization parameters in FP32, then converting final embeddings to FP16 at the U-Net boundary.
-- A repair utility for older exports with mixed FP32/FP16 LayerNormalization parameters: `test-site/scripts/repair_text_encoder_layernorm.py`.
-- Normalizing reduction axes into attributes or inputs according to the graph's opset.
-- Using U-Net opset 21 so FP16 dequantization scales produce weights compatible with FP16 convolution inputs.
-- Removing unreferenced initializers and consolidating external weights into one `.data` file per graph.
-- Saving the tokenizer, fixed tensor shapes, filenames, and measured scheduler coefficients alongside the weights.
-The Colab notebook is configured for a CUDA GPU, such as a T4. Its installation cell requests a session restart before export. Conversion can use several gigabytes of temporary disk space; the final release is much smaller. This conversion introduces no new training dataset.
-## Execution, memory, and caching
-The browser pipeline uses `onnxruntime-web/webgpu`, NCHW layout, and ONNX Runtime graph optimization set to `all`. The current JavaScript dependency versions are ONNX Runtime Web **1.29.0** and Transformers.js **4.2.0**. The source checkout's lockfile records the complete dependency versions.
-Both the capability check and inference initialization request `powerPreference: "high-performance"`. This asks the browser to prefer a discrete GPU, such as an NVIDIA RTX, on systems with integrated and discrete graphics. It is a preference; the browser and operating system retain the final selection. See the [WebGPU adapter selection reference](https://gpuweb.github.io/types/interfaces/GPURequestAdapterOptions.html).
-CLIP, U-Net, and TAESD sessions are created, run, and released sequentially to reduce peak GPU memory. Intermediate tensors are disposed when no longer needed. Model files can remain in browser memory, while GPU sessions are recreated on subsequent generations. This trades repeated initialization work for lower simultaneous GPU residency; the download size is not a VRAM requirement.
-WebAssembly threading is set to one. The application does not require cross-origin isolation headers for WASM threads. Serve it over HTTPS or localhost so the browser can expose WebGPU; opening `index.html` directly as a local file is not the supported launch method. See [ONNX Runtime's environment options](https://onnxruntime.ai/docs/tutorials/web/env-flags-and-session-options.html).
-The loader streams model downloads, checks file lengths, and stores assets in the browser's Cache API when available. A build-generated SHA-256 revision identifies the asset set and versions cache requests. Concurrent requests share the same pending model load; failed downloads can be retried, and truncated cached files are removed so retries can recover. Storage denial or quota errors do not prevent an uncached download.
-Cache availability depends on browser policy, private browsing, available disk space, and eviction. Cached model files can avoid repeated network transfers, but they do not remove session initialization costs. No service worker is registered, and a fully offline page reload is not guaranteed.
-## Files and integration
-The exported model package contains:
-```text
-README.md
-NDLGO2k.png
-manifest.json
-text_encoder_q4.onnx
-text_encoder_q4.onnx.data
-unet_mixed_q4q8.onnx
-unet_mixed_q4q8.onnx.data
-vae_decoder_fp16.onnx
-vae_decoder_fp16.onnx.data
-tokenizer/
-  tokenizer.json
-  tokenizer_config.json
-scheduler/
-  scheduler_config.json
-```
-Keep each `.onnx` file together with its matching `.onnx.data` file. The ONNX graphs contain external-weight references; uploading only the small graph files is insufficient. The runtime reads scheduler coefficients from `manifest.json`; the scheduler configuration is also retained in the exported package for reference.
-The static build places the inference assets under `models/sdxs/` and adds `assets.json` with filenames, byte counts, and a revision. The local-folder loader validates the manifest, required files, and fixed tensor shapes before accepting a package.
-The graphs require the accompanying browser orchestration or an equivalent implementation of tokenization, tensor conversion, sampling, and decoding. This ONNX package is not a standard Diffusers `from_pretrained()` directory, and no hosted Hugging Face Inference API endpoint is configured.
-## Run and host
-From the NanoDiffuser source checkout, use Node.js **22.13.0 or newer**:
-```bash
-cd test-site
-npm ci
-npm run dev:demo
-```
-The development address is `http://127.0.0.1:5173/`. To build and preview the static release:
-```bash
-npm run build:demo
-npm run preview:demo
-```
-The preview address is `http://127.0.0.1:4173/`. The output directory is `test-site/out`. The build copies the supplied `sdxs512_webgpu` weights, tokenizer, and manifest and packages the required browser runtime assets.
-To prepare an upload folder:
-```bash
-npm run package:hf
-```
-Create a Hugging Face **Static Space** and upload the **contents** of the generated `hf-space` folder to its root, retaining the directory structure:
-```text
-README.md
-index.html
-logo.png
-assets/
-models/
-```
-For this prebuilt upload, the Space's `README.md` only needs its deployment metadata; this model card contains the full documentation:
-```yaml
----
-title: NanoDiffuser
-emoji: ◻️
-colorFrom: gray
-colorTo: gray
-sdk: static
-app_file: index.html
-fullWidth: true
-header: mini
-short_description: Generate images locally with NanoDiffuser and WebGPU.
----
-```
-A prebuilt upload needs no build command or server dependencies. The included weights are served with the application, so this deployment does not need a separate remote model download integration. Upload real weight files, including large `.onnx.data` files, rather than unresolved Git LFS pointer text.
-If building on Hugging Face from source instead, upload the source tree and `sdxs512_webgpu` folder and use:
-```yaml
-sdk: static
-app_build_command: cd test-site && npm ci && npm run build:demo
-app_file: test-site/out/index.html
-```
-Exclude local dependencies and generated folders such as `node_modules`, `.next`, `.vinext`, `.wrangler`, `dist`, `out`, and `demo/public/models` from a source upload. Hugging Face documents both deployment approaches in its [Static Spaces guide](https://huggingface.co/docs/hub/spaces-sdks-static).
-## Example prompts
-The interface includes these five prompts. They are examples, not evidence of additional fine-tuning or measured benchmark performance.
-**Sunlit garden**
-> sunlit garden full of blooming flowers, stone path, small pond reflecting the sky, warm afternoon light, impressionist painting, oil on canvas, visible brushstrokes, soft broken color, luminous atmosphere, delicate painterly texture, elegant composition
-**Rainy Paris**
-> rainy Paris street, pedestrians with umbrellas, reflections on wet pavement, glowing shop windows, soft evening light, impressionist painting, loose brushwork, atmospheric perspective, oil on canvas, muted colors, romantic mood, painterly texture
-**Wildflower meadow**
-> woman walking through a meadow of wildflowers, flowing dress, summer breeze, bright sunlight, impressionist painting, oil on canvas, visible brushstrokes, luminous color, soft edges, natural light, poetic atmosphere
-**Flower bouquet**
-> bouquet of wildflowers in a ceramic vase, scattered petals on a wooden table, warm window light, impressionist still life painting, oil on canvas, visible brushstrokes, rich painterly texture, beautiful color harmony
-**Hillside village**
-> old European village on a hillside, stone cottages, narrow road, flowering trees, distant mountains, golden evening light, impressionist painting, oil on canvas, textured brushwork, warm colors, atmospheric perspective, timeless pastoral mood
-## Performance and validation
-One denoising step reduces the number of U-Net evaluations. It does not specify a universal generation time. Download speed, GPU/driver/browser support, session creation, shader compilation, quantized operators, and decoding all affect latency. The displayed elapsed time starts when inference is requested after the model-file loader completes and includes session setup; it excludes the preceding model-file download.
-No NanoDiffuser-specific FPS, minimum VRAM, or cross-device performance benchmark is claimed here. Performance measurements from other implementations do not establish the speed of this browser package.
-The exporter includes ONNX structural checks, quantization checks, finite-output checks, and reference comparisons with minimum cosine similarity gates of **0.96 for CLIP embeddings** and **0.90 for U-Net predictions**. Those thresholds are export acceptance criteria, not reported benchmark scores. The probes use a small reference case and do not establish general visual quality.
-The implementation work included successful builds, type/lint checks, and loader checks for shared downloads, cache reuse, retry recovery, corrupted cached files, and unavailable browser storage. An end-to-end GPU generation run and a broad device compatibility matrix were not performed as part of that verification.
-## Privacy, intended use, and limitations
-NanoDiffuser is intended for creative image generation, local experimentation, and browser inference development. Prompts, selected local model files, and generated pixels are processed on the user's device. The application has no prompt/image upload endpoint. Network requests fetch the application and model assets; normal hosting request logs can still exist. Browser model caching does not imply server-side storage of generated images.
-The interface states:
-> Images are AI-generated on your device and never uploaded. You’re responsible for what you create.
-Outputs can contain artifacts, inaccurate text, distorted objects, or imperfect prompt adherence. Quantization and the tiny decoder can change the output relative to higher-precision implementations. Inherited biases and limitations remain relevant, and this conversion does not add a content-safety classifier or moderation service. Review outputs before sharing or using them, and comply with the applicable licenses and rights of others.
-## License
-The converted weights retain their inherited **OpenRAIL++** licensing requirements. Preserve applicable license and attribution notices when redistributing the files. Software dependencies retain their respective licenses.