---
library_name: transformers
license: apache-2.0
pipeline_tag: image-text-to-text
tags:
  - tiny-random
  - dots3_note
  - multimodal
  - test-fixture
---

# Tiny random Dots 3 Note — BF16 integration fixture

A fixed random checkpoint for Hub loading and generation regression tests of
[Transformers PR #47844](https://github.com/huggingface/transformers/pull/47844).
It has **12,058 parameters**, all saved floating tensors are **BF16**, and
model.safetensors is **41,846 bytes**. Initialization uses seed 42 in FP32,
followed by BF16 conversion. Generated tokens are regression values with no
expected semantic quality.

Architecture and preprocessing were compared with
[dots-studio/dots3-note-prev](https://huggingface.co/dots-studio/dots3-note-prev)
at revision **1e1e7b0cd37a3a48a6c8d7fa55d5f9d14377006b**.
The current Dots implementation from PR #47844 is required. Loading uses the
standard Auto classes and does not require trust_remote_code.

## Coverage and size choices

| Component | Published model | Tiny fixture |
|---|---|---|
| Text depth / width | 46 / 5120 | 3 / 8 |
| Text layer 0 | DSA + dense MLP | DSA + dense MLP |
| Text layer 1 | DSA + routed/shared MoE | DSA + routed/shared MoE |
| Text layer 2 | SWA + routed/shared MoE | SWA + routed/shared MoE |
| DSA / SWA heads | 128 / 64 | 2 / 1 |
| DSA / SWA KV LoRA rank | 512 / 1024 | 2 / 4 |
| DSA index heads / dimension / top-k | 64 / 128 / 2048 | 2 / 8 / 4 |
| Sliding window | 513 | 4 |
| Text routed experts / selected / shared | 256 / 8 / 1 | 3 / 2 / 1 |
| Vision depth / width | 42 / 1536 | 3 / 8 |
| Vision MLPs | Dense and pyramid MoE | Dense, 3-expert MoE, 4-expert MoE |
| Vision selected experts | 2 | 2 |
| Vision patch / spatial merge | 14 / 2 | 2 / 2 |
| Audio depth / width | 32 / 1280 | 1 / 8 |
| Audio convolution stem channels | 480 | 2 |
| Audio mel bins | 128 | 8 |
| Vocabulary | 152064 | 341 |
| Saved precision | BF16 | BF16 |

Three text layers cover every attention/MLP combination present in the public
model. Three vision layers retain both dense computation and two different
expert counts. Top-2 routing selects a proper subset of the 3 or 4 experts.
Width 8 keeps two vision attention heads with nonzero rotary coordinates on
both spatial axes. The audio stem, encoder, adapter, text decoder, and LM head
are all present. Input embeddings and the LM head remain untied.

Headwise attention gates, MLA rescaling, shared experts, sigmoid routing,
DSA/SWA RoPE bases (80000000 / 50000), audio partial RoPE, spatial merge 2,
and temporal patch size 1 use the same implementation paths as the public model.
Long-context and production-size quantization kernels are outside this fixture's coverage.

## Changes beyond depth and width

- The config uses native nested text/vision/audio configs and per-layer overrides.
  It uses Dots3NoteForConditionalGeneration for the multimodal architecture.
  Legacy flattened dimensions and old processor-class references are omitted.
- The compact tokenizer contains all **85 public added-token strings** and the
  full **256-byte alphabet**. Added-token flags, NFC normalization, pre-tokenizer,
  post-processor, and byte decoder are preserved. BPE merges are removed and
  token IDs are remapped, so ordinary-text segmentation differs from production.
- The public **chat_template.jinja is copied verbatim**. Image/video/audio,
  role markers, thinking markers, and tool delimiters are represented.
- BOS, padding, media IDs, and both generation EOS IDs are remapped consistently.
  Padding is on the left. Greedy generation is the fixture default.
- DSA top-k and the sliding window are shortened to 4 so small tests exercise
  sparse selection and sliding-cache eviction.
- Image/video pixel limits are reduced to 16–64 pixels; both components use
  the public CLIP normalization values. Video limits are 8 frames and 32 tokens.
- Audio retains **16 kHz**, **hop 160**, and **FFT 400**. Chunk duration is 1 second,
  allowing short waveforms to exercise multiple chunks. The model and extractor
  agree on sampling rate, hop length, mel bins, and tokens per sample.
- Standard processor_config.json explicitly selects Qwen2VLImageProcessor,
  Qwen3VLVideoProcessor, and Dots3NoteFeatureExtractor.

## Loading

Replace MODEL_ID with this uploaded repository ID or a local folder path.

~~~python
import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

MODEL_ID = "YOUR_ORG/tiny-random-dots3-note"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForMultimodalLM.from_pretrained(
    MODEL_ID,
    dtype=torch.bfloat16,
    attn_implementation="eager",
    experts_implementation="eager",
).to("cpu").eval()
~~~

## Reproduce the four core fixed generation cases

CPU BF16 with eager attention is the reference configuration. This was validated
with Torch 2.10.0+cu128. The image/video arrays and waveform below are deterministic.

~~~python
import numpy as np

image = np.arange(8 * 8 * 3, dtype=np.uint8).reshape(8, 8, 3)
video = np.stack([
    image, np.flip(image, axis=0).copy(),
    image, np.flip(image, axis=1).copy(),
])
wave = (0.1 * np.sin(2 * np.pi * 440 * np.arange(2400) / 16000)).astype(np.float32)
media = {
    "text": {},
    "image": {"images": [image]},
    "audio": {"audio": [wave], "sampling_rate": 16000},
    "video": {
        "videos": [video], "do_sample_frames": False,
        "input_data_format": "channels_last", "cap_pixels_per_frame": False,
    },
}
expected = {
    "text":  [247, 103, 217, 187, 59, 295, 13, 246],
    "image": [247, 103, 217, 187, 59, 295, 13, 246],
    "audio": [247, 103, 217, 187, 59, 92, 306, 158],
    "video": [247, 103, 217, 187, 59, 295, 13, 246],
}
with torch.inference_mode():
    for modality, kwargs in media.items():
        content = [] if modality == "text" else [{"type": modality}]
        content.append({"type": "text", "text": "Describe."})
        prompt = processor.apply_chat_template(
            [{"role": "user", "content": content}],
            tokenize=False, add_generation_prompt=True, enable_thinking=False,
        )
        inputs = processor(text=prompt, return_tensors="pt", **kwargs).to(
            "cpu", dtype=model.dtype
        )
        output = model.generate(**inputs, do_sample=False, max_new_tokens=8)
        tokens = output[0, inputs.input_ids.shape[-1]:].tolist()
        assert tokens == expected[modality], (modality, tokens)
~~~

## Validation scope

- CPU and SM90 CUDA, both eager and SDPA attention, all with BF16 weights and inference.
- Text, image, audio, video, multiple audio chunks, and mixed image/video/audio inputs.
- Finite logits, repeated greedy generation, batched generation, and beam search
  with multiple audio chunks.
- Hooks verify both DSA indexers, SWA cache, dense MLPs, shared experts, and all
  four MoE routers execute. Every routed expert is selected at least once across
  the validation inputs.
- Perturbing each media tensor changes logits, confirming media enters the model.
- Standard Auto loaders and save/load round-trip, with no missing, unexpected,
  or mismatched weights.

generation_expectations.json records the six cases for each device/backend.
CPU and CUDA BF16 differ for the short-audio generation; eager and SDPA match
on each tested device. The CPU reference above is the intended fixed-output
integration test. A proposed integration class uses MemoryCleanupMixin.

This fixture is ready for upload. Actual Hub download/revision validation must
be performed after publication. Pin the same Hub commit for processor and model
when adding it to CI.