Image-Text-to-Text
Transformers
Safetensors
dots3_note
tiny-random
multimodal
test-fixture
conversational
Instructions to use miraclezqc/TestForDots3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use miraclezqc/TestForDots3 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="miraclezqc/TestForDots3") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("miraclezqc/TestForDots3") model = AutoModelForMultimodalLM.from_pretrained("miraclezqc/TestForDots3", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use miraclezqc/TestForDots3 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "miraclezqc/TestForDots3" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "miraclezqc/TestForDots3", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/miraclezqc/TestForDots3
- SGLang
How to use miraclezqc/TestForDots3 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "miraclezqc/TestForDots3" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "miraclezqc/TestForDots3", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "miraclezqc/TestForDots3" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "miraclezqc/TestForDots3", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use miraclezqc/TestForDots3 with Docker Model Runner:
docker model run hf.co/miraclezqc/TestForDots3
File size: 7,593 Bytes
6cd0083 14258a1 6cd0083 14258a1 6cd0083 14258a1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 | ---
library_name: transformers
license: apache-2.0
pipeline_tag: image-text-to-text
tags:
- tiny-random
- dots3_note
- multimodal
- test-fixture
---
# Tiny random Dots 3 Note — BF16 integration fixture
A fixed random checkpoint for Hub loading and generation regression tests of
[Transformers PR #47844](https://github.com/huggingface/transformers/pull/47844).
It has **12,058 parameters**, all saved floating tensors are **BF16**, and
model.safetensors is **41,846 bytes**. Initialization uses seed 42 in FP32,
followed by BF16 conversion. Generated tokens are regression values with no
expected semantic quality.
Architecture and preprocessing were compared with
[dots-studio/dots3-note-prev](https://huggingface.co/dots-studio/dots3-note-prev)
at revision **1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b**.
The current Dots implementation from PR #47844 is required. Loading uses the
standard Auto classes and does not require trust_remote_code.
## Coverage and size choices
| Component | Published model | Tiny fixture |
|---|---|---|
| Text depth / width | 46 / 5120 | 3 / 8 |
| Text layer 0 | DSA + dense MLP | DSA + dense MLP |
| Text layer 1 | DSA + routed/shared MoE | DSA + routed/shared MoE |
| Text layer 2 | SWA + routed/shared MoE | SWA + routed/shared MoE |
| DSA / SWA heads | 128 / 64 | 2 / 1 |
| DSA / SWA KV LoRA rank | 512 / 1024 | 2 / 4 |
| DSA index heads / dimension / top-k | 64 / 128 / 2048 | 2 / 8 / 4 |
| Sliding window | 513 | 4 |
| Text routed experts / selected / shared | 256 / 8 / 1 | 3 / 2 / 1 |
| Vision depth / width | 42 / 1536 | 3 / 8 |
| Vision MLPs | Dense and pyramid MoE | Dense, 3-expert MoE, 4-expert MoE |
| Vision selected experts | 2 | 2 |
| Vision patch / spatial merge | 14 / 2 | 2 / 2 |
| Audio depth / width | 32 / 1280 | 1 / 8 |
| Audio convolution stem channels | 480 | 2 |
| Audio mel bins | 128 | 8 |
| Vocabulary | 152064 | 341 |
| Saved precision | BF16 | BF16 |
Three text layers cover every attention/MLP combination present in the public
model. Three vision layers retain both dense computation and two different
expert counts. Top-2 routing selects a proper subset of the 3 or 4 experts.
Width 8 keeps two vision attention heads with nonzero rotary coordinates on
both spatial axes. The audio stem, encoder, adapter, text decoder, and LM head
are all present. Input embeddings and the LM head remain untied.
Headwise attention gates, MLA rescaling, shared experts, sigmoid routing,
DSA/SWA RoPE bases (80000000 / 50000), audio partial RoPE, spatial merge 2,
and temporal patch size 1 use the same implementation paths as the public model.
Long-context and production-size quantization kernels are outside this fixture's coverage.
## Changes beyond depth and width
- The config uses native nested text/vision/audio configs and per-layer overrides.
It uses Dots3NoteForConditionalGeneration for the multimodal architecture.
Legacy flattened dimensions and old processor-class references are omitted.
- The compact tokenizer contains all **85 public added-token strings** and the
full **256-byte alphabet**. Added-token flags, NFC normalization, pre-tokenizer,
post-processor, and byte decoder are preserved. BPE merges are removed and
token IDs are remapped, so ordinary-text segmentation differs from production.
- The public **chat_template.jinja is copied verbatim**. Image/video/audio,
role markers, thinking markers, and tool delimiters are represented.
- BOS, padding, media IDs, and both generation EOS IDs are remapped consistently.
Padding is on the left. Greedy generation is the fixture default.
- DSA top-k and the sliding window are shortened to 4 so small tests exercise
sparse selection and sliding-cache eviction.
- Image/video pixel limits are reduced to 16–64 pixels; both components use
the public CLIP normalization values. Video limits are 8 frames and 32 tokens.
- Audio retains **16 kHz**, **hop 160**, and **FFT 400**. Chunk duration is 1 second,
allowing short waveforms to exercise multiple chunks. The model and extractor
agree on sampling rate, hop length, mel bins, and tokens per sample.
- Standard processor_config.json explicitly selects Qwen2VLImageProcessor,
Qwen3VLVideoProcessor, and Dots3NoteFeatureExtractor.
## Loading
Replace MODEL_ID with this uploaded repository ID or a local folder path.
~~~python
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor
MODEL_ID = "YOUR_ORG/tiny-random-dots3-note"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
MODEL_ID,
dtype=torch.bfloat16,
attn_implementation="eager",
experts_implementation="eager",
).to("cpu").eval()
~~~
## Reproduce the four core fixed generation cases
CPU BF16 with eager attention is the reference configuration. This was validated
with Torch 2.10.0+cu128. The image/video arrays and waveform below are deterministic.
~~~python
import numpy as np
image = np.arange(8 * 8 * 3, dtype=np.uint8).reshape(8, 8, 3)
video = np.stack([
image, np.flip(image, axis=0).copy(),
image, np.flip(image, axis=1).copy(),
])
wave = (0.1 * np.sin(2 * np.pi * 440 * np.arange(2400) / 16000)).astype(np.float32)
media = {
"text": {},
"image": {"images": [image]},
"audio": {"audio": [wave], "sampling_rate": 16000},
"video": {
"videos": [video], "do_sample_frames": False,
"input_data_format": "channels_last", "cap_pixels_per_frame": False,
},
}
expected = {
"text": [247, 103, 217, 187, 59, 295, 13, 246],
"image": [247, 103, 217, 187, 59, 295, 13, 246],
"audio": [247, 103, 217, 187, 59, 92, 306, 158],
"video": [247, 103, 217, 187, 59, 295, 13, 246],
}
with torch.inference_mode():
for modality, kwargs in media.items():
content = [] if modality == "text" else [{"type": modality}]
content.append({"type": "text", "text": "Describe."})
prompt = processor.apply_chat_template(
[{"role": "user", "content": content}],
tokenize=False, add_generation_prompt=True, enable_thinking=False,
)
inputs = processor(text=prompt, return_tensors="pt", **kwargs).to(
"cpu", dtype=model.dtype
)
output = model.generate(**inputs, do_sample=False, max_new_tokens=8)
tokens = output[0, inputs.input_ids.shape[-1]:].tolist()
assert tokens == expected[modality], (modality, tokens)
~~~
## Validation scope
- CPU and SM90 CUDA, both eager and SDPA attention, all with BF16 weights and inference.
- Text, image, audio, video, multiple audio chunks, and mixed image/video/audio inputs.
- Finite logits, repeated greedy generation, batched generation, and beam search
with multiple audio chunks.
- Hooks verify both DSA indexers, SWA cache, dense MLPs, shared experts, and all
four MoE routers execute. Every routed expert is selected at least once across
the validation inputs.
- Perturbing each media tensor changes logits, confirming media enters the model.
- Standard Auto loaders and save/load round-trip, with no missing, unexpected,
or mismatched weights.
generation_expectations.json records the six cases for each device/backend.
CPU and CUDA BF16 differ for the short-audio generation; eager and SDPA match
on each tested device. The CPU reference above is the intended fixed-output
integration test. A proposed integration class uses MemoryCleanupMixin.
This fixture is ready for upload. Actual Hub download/revision validation must
be performed after publication. Pin the same Hub commit for processor and model
when adding it to CI.
|