| --- |
| library_name: transformers |
| license: apache-2.0 |
| pipeline_tag: image-text-to-text |
| tags: |
| - tiny-random |
| - dots3_note |
| - multimodal |
| - test-fixture |
| --- |
| |
| # Tiny random Dots 3 Note — BF16 integration fixture |
|
|
| A fixed random checkpoint for Hub loading and generation regression tests of |
| [Transformers PR #47844](https://github.com/huggingface/transformers/pull/47844). |
| It has **12,058 parameters**, all saved floating tensors are **BF16**, and |
| model.safetensors is **41,846 bytes**. Initialization uses seed 42 in FP32, |
| followed by BF16 conversion. Generated tokens are regression values with no |
| expected semantic quality. |
|
|
| Architecture and preprocessing were compared with |
| [dots-studio/dots3-note-prev](https://huggingface.co/dots-studio/dots3-note-prev) |
| at revision **1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b**. |
| The current Dots implementation from PR #47844 is required. Loading uses the |
| standard Auto classes and does not require trust_remote_code. |
|
|
| ## Coverage and size choices |
|
|
| | Component | Published model | Tiny fixture | |
| |---|---|---| |
| | Text depth / width | 46 / 5120 | 3 / 8 | |
| | Text layer 0 | DSA + dense MLP | DSA + dense MLP | |
| | Text layer 1 | DSA + routed/shared MoE | DSA + routed/shared MoE | |
| | Text layer 2 | SWA + routed/shared MoE | SWA + routed/shared MoE | |
| | DSA / SWA heads | 128 / 64 | 2 / 1 | |
| | DSA / SWA KV LoRA rank | 512 / 1024 | 2 / 4 | |
| | DSA index heads / dimension / top-k | 64 / 128 / 2048 | 2 / 8 / 4 | |
| | Sliding window | 513 | 4 | |
| | Text routed experts / selected / shared | 256 / 8 / 1 | 3 / 2 / 1 | |
| | Vision depth / width | 42 / 1536 | 3 / 8 | |
| | Vision MLPs | Dense and pyramid MoE | Dense, 3-expert MoE, 4-expert MoE | |
| | Vision selected experts | 2 | 2 | |
| | Vision patch / spatial merge | 14 / 2 | 2 / 2 | |
| | Audio depth / width | 32 / 1280 | 1 / 8 | |
| | Audio convolution stem channels | 480 | 2 | |
| | Audio mel bins | 128 | 8 | |
| | Vocabulary | 152064 | 341 | |
| | Saved precision | BF16 | BF16 | |
|
|
| Three text layers cover every attention/MLP combination present in the public |
| model. Three vision layers retain both dense computation and two different |
| expert counts. Top-2 routing selects a proper subset of the 3 or 4 experts. |
| Width 8 keeps two vision attention heads with nonzero rotary coordinates on |
| both spatial axes. The audio stem, encoder, adapter, text decoder, and LM head |
| are all present. Input embeddings and the LM head remain untied. |
|
|
| Headwise attention gates, MLA rescaling, shared experts, sigmoid routing, |
| DSA/SWA RoPE bases (80000000 / 50000), audio partial RoPE, spatial merge 2, |
| and temporal patch size 1 use the same implementation paths as the public model. |
| Long-context and production-size quantization kernels are outside this fixture's coverage. |
|
|
| ## Changes beyond depth and width |
|
|
| - The config uses native nested text/vision/audio configs and per-layer overrides. |
| It uses Dots3NoteForConditionalGeneration for the multimodal architecture. |
| Legacy flattened dimensions and old processor-class references are omitted. |
| - The compact tokenizer contains all **85 public added-token strings** and the |
| full **256-byte alphabet**. Added-token flags, NFC normalization, pre-tokenizer, |
| post-processor, and byte decoder are preserved. BPE merges are removed and |
| token IDs are remapped, so ordinary-text segmentation differs from production. |
| - The public **chat_template.jinja is copied verbatim**. Image/video/audio, |
| role markers, thinking markers, and tool delimiters are represented. |
| - BOS, padding, media IDs, and both generation EOS IDs are remapped consistently. |
| Padding is on the left. Greedy generation is the fixture default. |
| - DSA top-k and the sliding window are shortened to 4 so small tests exercise |
| sparse selection and sliding-cache eviction. |
| - Image/video pixel limits are reduced to 16–64 pixels; both components use |
| the public CLIP normalization values. Video limits are 8 frames and 32 tokens. |
| - Audio retains **16 kHz**, **hop 160**, and **FFT 400**. Chunk duration is 1 second, |
| allowing short waveforms to exercise multiple chunks. The model and extractor |
| agree on sampling rate, hop length, mel bins, and tokens per sample. |
| - Standard processor_config.json explicitly selects Qwen2VLImageProcessor, |
| Qwen3VLVideoProcessor, and Dots3NoteFeatureExtractor. |
| |
| ## Loading |
| |
| Replace MODEL_ID with this uploaded repository ID or a local folder path. |
| |
| ~~~python |
| import torch |
| from transformers import AutoModelForMultimodalLM, AutoProcessor |
| |
| MODEL_ID = "YOUR_ORG/tiny-random-dots3-note" |
| processor = AutoProcessor.from_pretrained(MODEL_ID) |
| model = AutoModelForMultimodalLM.from_pretrained( |
| MODEL_ID, |
| dtype=torch.bfloat16, |
| attn_implementation="eager", |
| experts_implementation="eager", |
| ).to("cpu").eval() |
| ~~~ |
| |
| ## Reproduce the four core fixed generation cases |
| |
| CPU BF16 with eager attention is the reference configuration. This was validated |
| with Torch 2.10.0+cu128. The image/video arrays and waveform below are deterministic. |
| |
| ~~~python |
| import numpy as np |
| |
| image = np.arange(8 * 8 * 3, dtype=np.uint8).reshape(8, 8, 3) |
| video = np.stack([ |
| image, np.flip(image, axis=0).copy(), |
| image, np.flip(image, axis=1).copy(), |
| ]) |
| wave = (0.1 * np.sin(2 * np.pi * 440 * np.arange(2400) / 16000)).astype(np.float32) |
| media = { |
| "text": {}, |
| "image": {"images": [image]}, |
| "audio": {"audio": [wave], "sampling_rate": 16000}, |
| "video": { |
| "videos": [video], "do_sample_frames": False, |
| "input_data_format": "channels_last", "cap_pixels_per_frame": False, |
| }, |
| } |
| expected = { |
| "text": [247, 103, 217, 187, 59, 295, 13, 246], |
| "image": [247, 103, 217, 187, 59, 295, 13, 246], |
| "audio": [247, 103, 217, 187, 59, 92, 306, 158], |
| "video": [247, 103, 217, 187, 59, 295, 13, 246], |
| } |
| with torch.inference_mode(): |
| for modality, kwargs in media.items(): |
| content = [] if modality == "text" else [{"type": modality}] |
| content.append({"type": "text", "text": "Describe."}) |
| prompt = processor.apply_chat_template( |
| [{"role": "user", "content": content}], |
| tokenize=False, add_generation_prompt=True, enable_thinking=False, |
| ) |
| inputs = processor(text=prompt, return_tensors="pt", **kwargs).to( |
| "cpu", dtype=model.dtype |
| ) |
| output = model.generate(**inputs, do_sample=False, max_new_tokens=8) |
| tokens = output[0, inputs.input_ids.shape[-1]:].tolist() |
| assert tokens == expected[modality], (modality, tokens) |
| ~~~ |
| |
| ## Validation scope |
| |
| - CPU and SM90 CUDA, both eager and SDPA attention, all with BF16 weights and inference. |
| - Text, image, audio, video, multiple audio chunks, and mixed image/video/audio inputs. |
| - Finite logits, repeated greedy generation, batched generation, and beam search |
| with multiple audio chunks. |
| - Hooks verify both DSA indexers, SWA cache, dense MLPs, shared experts, and all |
| four MoE routers execute. Every routed expert is selected at least once across |
| the validation inputs. |
| - Perturbing each media tensor changes logits, confirming media enters the model. |
| - Standard Auto loaders and save/load round-trip, with no missing, unexpected, |
| or mismatched weights. |
| |
| generation_expectations.json records the six cases for each device/backend. |
| CPU and CUDA BF16 differ for the short-audio generation; eager and SDPA match |
| on each tested device. The CPU reference above is the intended fixed-output |
| integration test. A proposed integration class uses MemoryCleanupMixin. |
| |
| This fixture is ready for upload. Actual Hub download/revision validation must |
| be performed after publication. Pin the same Hub commit for processor and model |
| when adding it to CI. |
| |
