Jamesbass commited on
Commit
8163f7e
·
verified ·
1 Parent(s): 1239683

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +75 -0
README.md ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: stability-ai-non-commercial-research-community
4
+ license_link: https://huggingface.co/stabilityai/sdxl-turbo/blob/main/LICENSE.md
5
+ base_model:
6
+ - stabilityai/sdxl-turbo
7
+ - h94/IP-Adapter
8
+ tags:
9
+ - tensorrt
10
+ - sentinel
11
+ - streamdiffusion
12
+ - controlnet
13
+ - sdxl-turbo
14
+ - real-time
15
+ - optical-illusion
16
+ pipeline_tag: image-to-image
17
+ ---
18
+
19
+ # OpticalPattern ControlNet engine for Sentinel StreamDiff
20
+
21
+ Prebuilt TensorRT engine that fuses the SDXL-Turbo UNet, the **OpticalPattern** ControlNet
22
+ ([Civitai 161132](https://civitai.com/models/161132), v10e by nacholmo) and the SDXL
23
+ IP-Adapter into the single UNet engine that [Sentinel](https://sentinel.oodlabs.com)'s
24
+ StreamDiff node loads. It makes real-time optical illusions from a live camera at 896x512.
25
+
26
+ Code, the Sentinel project, the OP_Pattern control-image Module, build scripts and tuning
27
+ notes: **https://github.com/jhurlbut/sentinel-opticalpattern-controlnet**
28
+
29
+ ## Files
30
+
31
+ | File | Size | What |
32
+ |---|---|---|
33
+ | `profiles/sdxl/custom/opticalpattern/896x512/unet_controlnet_union_ipadapter_fp16.engine` | 8.4 GB | The fused engine. Must keep this filename and folder layout inside Sentinel's `engines/`. |
34
+ | `demos/op_diffusion_cloud_face.mp4` | 58 MB | Node output: a face illusion hidden in clouds |
35
+ | `demos/sentinel_window_spiral_interchange.mp4` | 45 MB | Sentinel window: highway interchange bent into a spiral |
36
+
37
+ ## Compatibility
38
+
39
+ This is a TensorRT **plan**, not portable weights. It only loads on the GPU architecture it
40
+ was built for:
41
+
42
+ - NVIDIA RTX 50-series (Blackwell, sm120). Built on an RTX 5090 Laptop GPU.
43
+ - TensorRT 10.15.1 runtime, which is what Sentinel 0.5.66 ships as `nvinfer_10.dll`.
44
+ - Sentinel 0.5.66 or newer, `engine_tier` = ControlNet + IP-Adapter, FP16, 896x512.
45
+ - About 9 GB of VRAM for this engine plus Sentinel's CLIP, VAE and IP-Adapter engines.
46
+
47
+ Any other GPU generation or TensorRT version: rebuild with the scripts in the GitHub repo.
48
+ The build takes 30 to 60 minutes.
49
+
50
+ ## Install
51
+
52
+ ```
53
+ git clone https://github.com/jhurlbut/sentinel-opticalpattern-controlnet
54
+ python sentinel-opticalpattern-controlnet/export/install_custom_pack.py --name opticalpattern --controlnet --engine <downloaded .engine> --display "SDXL OpticalPattern CN 896x512"
55
+ ```
56
+
57
+ Then open the Sentinel project from the repo, set the StreamDiff node's engine resolution to
58
+ `opticalpattern (896x512)` and relaunch. Full steps and the measured tuning recipe are in the
59
+ repo README.
60
+
61
+ ## Binding contract
62
+
63
+ Static batch 1. Inputs `sample` [1,4,64,112] f16, `timestep` [1] f32, `encoder_hidden_states`
64
+ [1,81,2048] f16 (77 text + 4 IP-Adapter tokens), `text_embeds` [1,1280] f16, `time_ids`
65
+ [1,6] f32, `controlnet_cond` [1,3,512,896] f16, `controlnet_scale` [1] f32,
66
+ `ipadapter_scale` [70] f32. Output `out_sample` [1,4,64,112] f16. Verified against the
67
+ PyTorch reference at correlation 0.99993.
68
+
69
+ ## License
70
+
71
+ The engine is a derivative of SDXL-Turbo (Stability AI Non-Commercial Research Community
72
+ License), the OpticalPattern ControlNet (CreativeML Open RAIL++-M with the author's addendum)
73
+ and IP-Adapter (Apache-2.0). It is provided for non-commercial research use only; commercial
74
+ use needs a Stability AI license. The ControlNet training weights themselves are not
75
+ redistributed here.