dn6 HF Staff commited on
Commit
e16c86b
·
verified ·
1 Parent(s): c68c5ae

Copy of akshan-main/tiny-qwenimage21-modular-pipe@5a780beb for diffusers modular tests (PR #14976)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ processor/tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: diffusers
3
+ tags:
4
+ - modular-diffusers
5
+ - diffusers
6
+ - qwenimage21
7
+ - text-to-image
8
+ ---
9
+ This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.
10
+
11
+ **Pipeline Type**: QwenImage21AutoBlocks
12
+
13
+ **Description**: Auto Modular pipeline for text-to-image and image-conditioned generation using Qwen-Image 2.1.
14
+ - for text-to-image generation, all you need to provide is `prompt`
15
+ - for image-conditioned generation, you need to provide `prompt` and `image` (one image or a list)
16
+
17
+ This pipeline uses a 4-block architecture that can be customized and extended.
18
+
19
+ ## Example Usage
20
+
21
+ [TODO]
22
+
23
+ ## Pipeline Architecture
24
+
25
+ This modular pipeline is composed of the following blocks:
26
+
27
+ 1. **text_encoder** (`QwenImage21AutoTextEncoderStep`)
28
+ - Text encoder step that encodes the prompt, together with the condition images when there are any.
29
+ 2. **vae_encoder** (`QwenImage21AutoVaeEncoderStep`)
30
+ - VAE encoder step that encodes the condition images into their latent representations.
31
+ 3. **denoise** (`QwenImage21AutoCoreDenoiseStep`)
32
+ - Auto core denoise step that performs the denoising process.
33
+ 4. **decode** (`QwenImage21DecodeStep`)
34
+ - Step that decodes the latents to RGBA images and postprocesses them
35
+
36
+ ## Model Components
37
+
38
+ 1. image_processor (`VaeImageProcessor`)
39
+ 2. text_encoder (`Qwen3VLForConditionalGeneration`)
40
+ 3. processor (`Qwen3VLProcessor`)
41
+ 4. guider (`ClassifierFreeGuidance`)
42
+ 5. vae (`AutoencoderKLQwenImage21`)
43
+ 6. scheduler (`FlowMatchEulerDiscreteScheduler`)
44
+ 7. transformer (`QwenImage21Transformer2DModel`)
45
+
46
+ ## Configuration Parameters
47
+
48
+ sample_sigmas (default: None): Default sampling grid of the checkpoint, used when `sigmas` is not passed.
49
+
50
+ ## Workflow Input Specification
51
+
52
+ <details>
53
+ <summary><strong>text2image</strong></summary>
54
+
55
+ - `prompt` (`str`): The prompt or prompts to guide image generation.
56
+
57
+ </details>
58
+
59
+ <details>
60
+ <summary><strong>image_conditioned</strong></summary>
61
+
62
+ - `image` (`Image | list`): Reference image(s) for denoising. Can be a single image or list of images.
63
+ - `prompt` (`str`): The prompt or prompts to guide image generation.
64
+
65
+ </details>
66
+
67
+
68
+ ## Input/Output Specification
69
+
70
+ **Inputs:**
71
+
72
+ - `image` (`Image | list`, *optional*): Reference image(s) for denoising. Can be a single image or list of images.
73
+ - `output_resolution` (`int`, *optional*, defaults to `1024`): Target side length used to derive the output size and to resize condition images.
74
+ - `prompt` (`str`): The prompt or prompts to guide image generation.
75
+ - `negative_prompt` (`str`, *optional*): The prompt or prompts not to guide the image generation.
76
+ - `generator` (`Generator`, *optional*): Torch generator for deterministic generation.
77
+ - `num_images_per_prompt` (`int`, *optional*, defaults to `1`): The number of images to generate per prompt.
78
+ - `height` (`int`, *optional*): The height in pixels of the generated image.
79
+ - `width` (`int`, *optional*): The width in pixels of the generated image.
80
+ - `image_latents` (`list`, *optional*): Normalized latents of each condition image. Can be generated from vae_encoder step.
81
+ - `latents` (`Tensor`): Pre-generated noisy latents for image generation.
82
+ - `num_inference_steps` (`int`): The number of denoising steps.
83
+ - `sigmas` (`list`, *optional*): Custom sigmas for the denoising process.
84
+ - `use_kv_cache` (`bool`, *optional*, defaults to `True`): Cache the text and condition-image keys and values after the first step. Valid because `causal_condition` modulates those tokens from `t = 0`, making their activations step-independent. Toggling it does not reproduce the same image bit-for-bit in reduced precision.
85
+ - `attention_kwargs` (`dict`, *optional*): Additional kwargs for attention processors.
86
+ - `**denoiser_input_fields` (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
87
+ - `output_type` (`str`, *optional*, defaults to `pil`): Output format: 'pil', 'np', 'pt'.
88
+
89
+ **Outputs:**
90
+
91
+ - `images` (`list`): Generated images.
modular_model_index.json ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_blocks_class_name": "QwenImage21AutoBlocks",
3
+ "_class_name": "QwenImage21ModularPipeline",
4
+ "_diffusers_version": "0.41.0.dev0",
5
+ "processor": [
6
+ "transformers",
7
+ "Qwen3VLProcessor",
8
+ {
9
+ "pretrained_model_name_or_path": "hf-internal-testing/tiny-qwenimage21-modular",
10
+ "revision": null,
11
+ "subfolder": "processor",
12
+ "type_hint": [
13
+ "transformers",
14
+ "Qwen3VLProcessor"
15
+ ],
16
+ "variant": null
17
+ }
18
+ ],
19
+ "sample_sigmas": null,
20
+ "scheduler": [
21
+ "diffusers",
22
+ "FlowMatchEulerDiscreteScheduler",
23
+ {
24
+ "pretrained_model_name_or_path": "hf-internal-testing/tiny-qwenimage21-modular",
25
+ "revision": null,
26
+ "subfolder": "scheduler",
27
+ "type_hint": [
28
+ "diffusers",
29
+ "FlowMatchEulerDiscreteScheduler"
30
+ ],
31
+ "variant": null
32
+ }
33
+ ],
34
+ "text_encoder": [
35
+ "transformers",
36
+ "Qwen3VLForConditionalGeneration",
37
+ {
38
+ "pretrained_model_name_or_path": "hf-internal-testing/tiny-qwenimage21-modular",
39
+ "revision": null,
40
+ "subfolder": "text_encoder",
41
+ "type_hint": [
42
+ "transformers",
43
+ "Qwen3VLForConditionalGeneration"
44
+ ],
45
+ "variant": null
46
+ }
47
+ ],
48
+ "transformer": [
49
+ "diffusers",
50
+ "QwenImage21Transformer2DModel",
51
+ {
52
+ "pretrained_model_name_or_path": "hf-internal-testing/tiny-qwenimage21-modular",
53
+ "revision": null,
54
+ "subfolder": "transformer",
55
+ "type_hint": [
56
+ "diffusers",
57
+ "QwenImage21Transformer2DModel"
58
+ ],
59
+ "variant": null
60
+ }
61
+ ],
62
+ "vae": [
63
+ "diffusers",
64
+ "AutoencoderKLQwenImage21",
65
+ {
66
+ "pretrained_model_name_or_path": "hf-internal-testing/tiny-qwenimage21-modular",
67
+ "revision": null,
68
+ "subfolder": "vae",
69
+ "type_hint": [
70
+ "diffusers",
71
+ "AutoencoderKLQwenImage21"
72
+ ],
73
+ "variant": null
74
+ }
75
+ ]
76
+ }
processor/chat_template.jinja ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system
2
+ You are a helpful assistant.<|im_end|>
3
+ {% endif %}<|im_start|>{{ message['role'] }}
4
+ {% if message['content'] is string %}{{ message['content'] }}<|im_end|>
5
+ {% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>
6
+ {% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
7
+ {% endif %}
processor/processor_config.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "do_convert_rgb": true,
4
+ "do_normalize": true,
5
+ "do_rescale": true,
6
+ "do_resize": true,
7
+ "image_mean": [
8
+ 0.48145466,
9
+ 0.4578275,
10
+ 0.40821073
11
+ ],
12
+ "image_processor_type": "Qwen2VLImageProcessor",
13
+ "image_std": [
14
+ 0.26862954,
15
+ 0.26130258,
16
+ 0.27577711
17
+ ],
18
+ "merge_size": 2,
19
+ "patch_size": 16,
20
+ "resample": 3,
21
+ "rescale_factor": 0.00392156862745098,
22
+ "size": {
23
+ "longest_edge": 4096,
24
+ "shortest_edge": 1024
25
+ },
26
+ "temporal_patch_size": 2
27
+ },
28
+ "processor_class": "Qwen3VLProcessor",
29
+ "video_processor": {
30
+ "do_convert_rgb": true,
31
+ "do_normalize": true,
32
+ "do_rescale": true,
33
+ "do_resize": true,
34
+ "do_sample_frames": true,
35
+ "fps": 2,
36
+ "image_mean": [
37
+ 0.5,
38
+ 0.5,
39
+ 0.5
40
+ ],
41
+ "image_std": [
42
+ 0.5,
43
+ 0.5,
44
+ 0.5
45
+ ],
46
+ "max_frames": 768,
47
+ "merge_size": 2,
48
+ "min_frames": 4,
49
+ "patch_size": 16,
50
+ "resample": 3,
51
+ "rescale_factor": 0.00392156862745098,
52
+ "return_metadata": false,
53
+ "size": {
54
+ "longest_edge": 786432,
55
+ "shortest_edge": 131072
56
+ },
57
+ "temporal_patch_size": 2,
58
+ "video_processor_type": "Qwen3VLVideoProcessor"
59
+ }
60
+ }
processor/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ff8cce547abc110590d19c6b5b6e0c6a7b4c8d1012d78b9c42131bae7f494a02
3
+ size 11420367
processor/tokenizer_config.json ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>",
11
+ "<|object_ref_start|>",
12
+ "<|object_ref_end|>",
13
+ "<|box_start|>",
14
+ "<|box_end|>",
15
+ "<|quad_start|>",
16
+ "<|quad_end|>",
17
+ "<|vision_start|>",
18
+ "<|vision_end|>",
19
+ "<|vision_pad|>",
20
+ "<|image_pad|>",
21
+ "<|video_pad|>"
22
+ ],
23
+ "is_local": true,
24
+ "local_files_only": false,
25
+ "max_length": null,
26
+ "model_max_length": 32768,
27
+ "pad_to_multiple_of": null,
28
+ "pad_token": "<|endoftext|>",
29
+ "pad_token_type_id": 0,
30
+ "padding_side": "left",
31
+ "processor_class": "Qwen3VLProcessor",
32
+ "split_special_tokens": false,
33
+ "tokenizer_class": "Qwen2Tokenizer",
34
+ "unk_token": null,
35
+ "variant": null
36
+ }
scheduler/scheduler_config.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "FlowMatchEulerDiscreteScheduler",
3
+ "_diffusers_version": "0.41.0.dev0",
4
+ "base_image_seq_len": 256,
5
+ "base_shift": 0.5,
6
+ "invert_sigmas": false,
7
+ "max_image_seq_len": 4096,
8
+ "max_shift": 1.15,
9
+ "num_train_timesteps": 1000,
10
+ "shift": 1.0,
11
+ "shift_terminal": null,
12
+ "stochastic_sampling": false,
13
+ "time_shift_type": "exponential",
14
+ "use_beta_sigmas": false,
15
+ "use_dynamic_shifting": false,
16
+ "use_exponential_sigmas": false,
17
+ "use_karras_sigmas": false
18
+ }
text_encoder/config.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3VLForConditionalGeneration"
4
+ ],
5
+ "dtype": "float32",
6
+ "image_token_id": 151655,
7
+ "model_type": "qwen3_vl",
8
+ "text_config": {
9
+ "attention_bias": false,
10
+ "attention_dropout": 0.0,
11
+ "dtype": "float32",
12
+ "head_dim": 8,
13
+ "hidden_act": "silu",
14
+ "hidden_size": 16,
15
+ "initializer_range": 0.02,
16
+ "intermediate_size": 16,
17
+ "max_position_embeddings": 128000,
18
+ "model_type": "qwen3_vl_text",
19
+ "num_attention_heads": 2,
20
+ "num_hidden_layers": 2,
21
+ "num_key_value_heads": 2,
22
+ "pad_token_id": null,
23
+ "rms_norm_eps": 1e-06,
24
+ "rope_parameters": {
25
+ "mrope_section": [
26
+ 1,
27
+ 1,
28
+ 2
29
+ ],
30
+ "rope_theta": 1000000.0,
31
+ "rope_type": "default"
32
+ },
33
+ "use_cache": true,
34
+ "vocab_size": 151936
35
+ },
36
+ "tie_word_embeddings": false,
37
+ "transformers_version": "5.15.0",
38
+ "video_token_id": 151656,
39
+ "vision_config": {
40
+ "deepstack_visual_indexes": [
41
+ 0
42
+ ],
43
+ "depth": 2,
44
+ "dtype": "float32",
45
+ "hidden_act": "gelu_pytorch_tanh",
46
+ "hidden_size": 16,
47
+ "in_channels": 3,
48
+ "initializer_range": 0.02,
49
+ "intermediate_size": 16,
50
+ "model_type": "qwen3_vl_vision",
51
+ "num_heads": 2,
52
+ "num_position_embeddings": 64,
53
+ "out_hidden_size": 16,
54
+ "patch_size": 16,
55
+ "spatial_merge_size": 2,
56
+ "temporal_patch_size": 2
57
+ },
58
+ "vision_end_token_id": 151653,
59
+ "vision_start_token_id": 151652
60
+ }
text_encoder/generation_config.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "output_attentions": false,
4
+ "output_hidden_states": false,
5
+ "transformers_version": "5.15.0",
6
+ "use_cache": true
7
+ }
text_encoder/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:65b388abe71f859ef783ff788d29836ea0a0513df3180f6f0ac4b476f6ea9d8b
3
+ size 19628032
transformer/config.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "QwenImage21Transformer2DModel",
3
+ "_diffusers_version": "0.41.0.dev0",
4
+ "_name_or_path": "/Volumes/T7 Shield/ai-lab/tiny/qwenimage21-modular/standard",
5
+ "attention_head_dim": 16,
6
+ "axes_dims_rope": [
7
+ 4,
8
+ 6,
9
+ 6
10
+ ],
11
+ "causal_condition": true,
12
+ "context_in_dim": 16,
13
+ "eps": 1e-06,
14
+ "in_channels": 8,
15
+ "mlp_ratio": 2,
16
+ "num_attention_heads": 2,
17
+ "num_layers": 2,
18
+ "out_channels": 8,
19
+ "patch_size": 1
20
+ }
transformer/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:96a310d836b05f743b29d1a1d5ea91d4b59f976dce7d44451b105b5bc4b569c9
3
+ size 150496
vae/config.json ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_class_name": "AutoencoderKLQwenImage21",
3
+ "_diffusers_version": "0.41.0.dev0",
4
+ "_name_or_path": "/Volumes/T7 Shield/ai-lab/tiny/qwenimage21-modular/standard",
5
+ "attn_scales": [],
6
+ "base_dim": 4,
7
+ "decoder_base_dim": 4,
8
+ "dim_mult": [
9
+ 1,
10
+ 1,
11
+ 1,
12
+ 1,
13
+ 1
14
+ ],
15
+ "dropout": 0.0,
16
+ "in_channels": 4,
17
+ "is_residual": true,
18
+ "latents_mean": [
19
+ 0.0,
20
+ 0.0,
21
+ 0.0,
22
+ 0.0,
23
+ 0.0,
24
+ 0.0,
25
+ 0.0,
26
+ 0.0
27
+ ],
28
+ "latents_std": [
29
+ 1.0,
30
+ 1.0,
31
+ 1.0,
32
+ 1.0,
33
+ 1.0,
34
+ 1.0,
35
+ 1.0,
36
+ 1.0
37
+ ],
38
+ "num_res_blocks": 1,
39
+ "out_channels": 4,
40
+ "patch_size": null,
41
+ "scale_factor_spatial": 16,
42
+ "scale_factor_temporal": 8,
43
+ "temperal_downsample": [
44
+ false,
45
+ true,
46
+ true,
47
+ true
48
+ ],
49
+ "z_dim": 8
50
+ }
vae/diffusion_pytorch_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:47b298bbf851ea4ec0b1b256c3a1480c3210485d7793bca35be919cbdfcfb393
3
+ size 52616