---
library_name: transformers
pipeline_tag: image-text-to-text
base_model:
- mistralai/Mistral-Large-4-1T-A52B
---

This is a randomly initialized tiny model based on [mistralai/Mistral-Large-4-1T-A52B](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B) for testing and debugging.

| Feature | Source | This repo |
|---|---|---|
| Checkpoint tensor elements | 7.13B BF16, 63.84M F32, 1.05T F8_E4M3 | 54.84M BF16, 320 F32, 1.31M F8_E4M3 |
| Safetensors weight files | 1.06 TB (54 files) | 111.13 MB (consolidated.safetensors) |
| Architecture / input | Multimodal MoE; text and image | Same |
| Vocabulary rows / tied embeddings | 204,800 / false | 204,800 / true |
| Backbone layers | 72 | 12: three complete RoPE/RoPE/RoPE/NoPE cycles |
| Text hidden size / attention head dimension | 6,144 / 64 | 128 / 32 |
| Routed / selected / shared experts | 128 / 4 / 1 | 32 / 4 / 1 |
| DSA indexer | 32 heads; 128-wide; top-2,048 | 4 heads; 128-wide; top-128 |
| Cross-layer KV sharing | Source pattern | Two complete three-own/one-shared NoPE cycles |
| Vision tower | 30 layers; 2,048 width | 4 layers; 64 width |
| DSpark | 3 dense MLA stages; rank 512; depth 3 | 3 dense MLA stages; rank 8; depth 3 |
| Checkpoint precision | E4M3 block-FP8 + FP32 scales; BF16 exclusions | Same format; 64×64 blocks |

TP=2 vLLM uses a 64-wide FP8 activation group per rank, which fixes the residual width at 128. The checkpoint retains 64×64 FP8 blocks, a 512-wide MLA latent KV required by H20 sparse MLA, and four 32-wide attention heads. The input/output BF16 token table is tied, saving one 52.43 MB matrix; the optimized vLLM kernel constraints still place the checkpoint above the 30 MB soft target.

| File path | Size |
|------|------|
| consolidated.safetensors | 111.1MB |

### Example usage

#### Transformers (✅ verified)

```python
def run_transformers(model_id: str):
    import torch
    from PIL import Image
    from transformers import AutoModelForImageTextToText, AutoProcessor
    from transformers.utils.quantization_config import FineGrainedFP8Config

    model = AutoModelForImageTextToText.from_pretrained(
        str(model_id),
        dtype=torch.bfloat16,
        device_map="cuda",
        attn_implementation="sdpa",
        quantization_config=FineGrainedFP8Config(dequantize=True),
    )
    processor = AutoProcessor.from_pretrained(str(model_id))
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": Image.new("RGB", (56, 56), (30, 120, 220))},
                {"type": "text", "text": "Describe this image in one word."},
            ],
        }
    ]
    inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        reasoning_effort="low",
    ).to(model.device)
    output = model.generate(**inputs, do_sample=False, max_new_tokens=10)
    print(processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True))


if __name__ == "__main__":
    run_transformers(model_id="tiny-random/mistral-large-4")
```

#### vLLM offline (✅ verified)

Install the snapshot package with `pip install --no-deps <model directory>` before starting vLLM.

```python
def run_vllm(model_id: str):
    import os

    os.environ.setdefault("VLLM_USE_FLASHINFER_SAMPLER", "0")
    os.environ.setdefault("VLLM_ALLREDUCE_USE_FLASHINFER", "0")
    os.environ.setdefault("VLLM_ALLREDUCE_USE_SYMM_MEM", "0")
    from PIL import Image
    from transformers import AutoProcessor
    from vllm import LLM, SamplingParams

    model_id = str(model_id)
    processor = AutoProcessor.from_pretrained(model_id)
    llm = LLM(
        model=model_id,
        trust_remote_code=True,
        tensor_parallel_size=2,
        speculative_config={"method": "dspark", "num_speculative_tokens": 3},
        kv_cache_dtype="bfloat16",
        max_model_len=512,
        max_num_batched_tokens=1024,
        max_num_seqs=2,
        gpu_memory_utilization=0.1,
        disable_custom_all_reduce=True,
        kernel_config={"linear_backend": "triton"},
        attention_config={"backend": "FLASHINFER_MLA_SPARSE_SM90"},
    )
    sampling = SamplingParams(temperature=0, max_tokens=10, ignore_eos=True)
    try:
        print(llm.generate(["Hello"], sampling)[0].outputs[0].text)
        image = Image.new("RGB", (56, 56), "blue")
        inputs = processor.apply_chat_template(
            [
                {
                    "role": "user",
                    "content": [
                        {"type": "image", "image": image},
                        {"type": "text", "text": "Describe this image in one word."},
                    ],
                }
            ],
            add_generation_prompt=True,
            tokenize=True,
            return_dict=True,
            return_tensors="pt",
            reasoning_effort="low",
        )
        result = llm.generate(
            {"prompt_token_ids": inputs["input_ids"][0].tolist(), "multi_modal_data": {"image": image}},
            sampling,
        )
        print(result[0].outputs[0].text)
    finally:
        llm.llm_engine.engine_core.shutdown(timeout=30)


if __name__ == "__main__":
    run_vllm(model_id="tiny-random/mistral-large-4")
```

#### vLLM serving (unverified)

Use the official Mistral Large 4 vLLM image.

```bash
docker run --rm --runtime nvidia --gpus all --ipc=host -p 8000:8000 --entrypoint bash \
  -v "$(pwd)/mistral-large-4:/model:ro" vllm/vllm-openai:mistral-large-4 \
  -lc 'pip install --no-deps /model && vllm serve /model \
  --tensor-parallel-size 2 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}' \
  --kv-cache-dtype bfloat16 --disable-custom-all-reduce \
  --kernel-config '{"linear_backend":"triton"}' \
  --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_SM90"}' \
  --limit-mm-per-prompt '{"image":1}' --max-model-len 2048 \
  --max-num-batched-tokens 256 --max-num-seqs 4 --gpu-memory-utilization 0.2'
```

### Creating this repo

- Code: [.meta/create_model.py](.meta/create_model.py)
- Model structure: [.meta/model.log](.meta/model.log)
- Environment: [.meta/env.json](.meta/env.json)

*This repo is generated by GPT-6.1-Sol.*