tiny-random/mistral-large-4

This is a randomly initialized tiny model based on mistralai/Mistral-Large-4-1T-A52B for testing and debugging.

Feature	Source	This repo
Checkpoint tensor elements	7.13B BF16, 63.84M F32, 1.05T F8_E4M3	54.84M BF16, 320 F32, 1.31M F8_E4M3
Safetensors weight files	1.06 TB (54 files)	111.13 MB (consolidated.safetensors)
Architecture / input	Multimodal MoE; text and image	Same
Vocabulary rows / tied embeddings	204,800 / false	204,800 / true
Backbone layers	72	12: three complete RoPE/RoPE/RoPE/NoPE cycles
Text hidden size / attention head dimension	6,144 / 64	128 / 32
Routed / selected / shared experts	128 / 4 / 1	32 / 4 / 1
DSA indexer	32 heads; 128-wide; top-2,048	4 heads; 128-wide; top-128
Cross-layer KV sharing	Source pattern	Two complete three-own/one-shared NoPE cycles
Vision tower	30 layers; 2,048 width	4 layers; 64 width
DSpark	3 dense MLA stages; rank 512; depth 3	3 dense MLA stages; rank 8; depth 3
Checkpoint precision	E4M3 block-FP8 + FP32 scales; BF16 exclusions	Same format; 64×64 blocks

TP=2 vLLM uses a 64-wide FP8 activation group per rank, which fixes the residual width at 128. The checkpoint retains 64×64 FP8 blocks, a 512-wide MLA latent KV required by H20 sparse MLA, and four 32-wide attention heads. The input/output BF16 token table is tied, saving one 52.43 MB matrix; the optimized vLLM kernel constraints still place the checkpoint above the 30 MB soft target.

File path	Size
consolidated.safetensors	111.1MB

Example usage

Transformers (✅ verified)

def run_transformers(model_id: str):
    import torch
    from PIL import Image
    from transformers import AutoModelForImageTextToText, AutoProcessor
    from transformers.utils.quantization_config import FineGrainedFP8Config

    model = AutoModelForImageTextToText.from_pretrained(
        str(model_id),
        dtype=torch.bfloat16,
        device_map="cuda",
        attn_implementation="sdpa",
        quantization_config=FineGrainedFP8Config(dequantize=True),
    )
    processor = AutoProcessor.from_pretrained(str(model_id))
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": Image.new("RGB", (56, 56), (30, 120, 220))},
                {"type": "text", "text": "Describe this image in one word."},
            ],
        }
    ]
    inputs = processor.apply_chat_template(
        messages,
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
        reasoning_effort="low",
    ).to(model.device)
    output = model.generate(**inputs, do_sample=False, max_new_tokens=10)
    print(processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True))


if __name__ == "__main__":
    run_transformers(model_id="tiny-random/mistral-large-4")

vLLM offline (✅ verified)

Install the snapshot package with pip install --no-deps <model directory> before starting vLLM.

def run_vllm(model_id: str):
    import os

    os.environ.setdefault("VLLM_USE_FLASHINFER_SAMPLER", "0")
    os.environ.setdefault("VLLM_ALLREDUCE_USE_FLASHINFER", "0")
    os.environ.setdefault("VLLM_ALLREDUCE_USE_SYMM_MEM", "0")
    from PIL import Image
    from transformers import AutoProcessor
    from vllm import LLM, SamplingParams

    model_id = str(model_id)
    processor = AutoProcessor.from_pretrained(model_id)
    llm = LLM(
        model=model_id,
        trust_remote_code=True,
        tensor_parallel_size=2,
        speculative_config={"method": "dspark", "num_speculative_tokens": 3},
        kv_cache_dtype="bfloat16",
        max_model_len=512,
        max_num_batched_tokens=1024,
        max_num_seqs=2,
        gpu_memory_utilization=0.1,
        disable_custom_all_reduce=True,
        kernel_config={"linear_backend": "triton"},
        attention_config={"backend": "FLASHINFER_MLA_SPARSE_SM90"},
    )
    sampling = SamplingParams(temperature=0, max_tokens=10, ignore_eos=True)
    try:
        print(llm.generate(["Hello"], sampling)[0].outputs[0].text)
        image = Image.new("RGB", (56, 56), "blue")
        inputs = processor.apply_chat_template(
            [
                {
                    "role": "user",
                    "content": [
                        {"type": "image", "image": image},
                        {"type": "text", "text": "Describe this image in one word."},
                    ],
                }
            ],
            add_generation_prompt=True,
            tokenize=True,
            return_dict=True,
            return_tensors="pt",
            reasoning_effort="low",
        )
        result = llm.generate(
            {"prompt_token_ids": inputs["input_ids"][0].tolist(), "multi_modal_data": {"image": image}},
            sampling,
        )
        print(result[0].outputs[0].text)
    finally:
        llm.llm_engine.engine_core.shutdown(timeout=30)


if __name__ == "__main__":
    run_vllm(model_id="tiny-random/mistral-large-4")

vLLM serving (unverified)

Use the official Mistral Large 4 vLLM image.

docker run --rm --runtime nvidia --gpus all --ipc=host -p 8000:8000 --entrypoint bash \
  -v "$(pwd)/mistral-large-4:/model:ro" vllm/vllm-openai:mistral-large-4 \
  -lc 'pip install --no-deps /model && vllm serve /model \
  --tensor-parallel-size 2 \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}' \
  --kv-cache-dtype bfloat16 --disable-custom-all-reduce \
  --kernel-config '{"linear_backend":"triton"}' \
  --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_SM90"}' \
  --limit-mm-per-prompt '{"image":1}' --max-model-len 2048 \
  --max-num-batched-tokens 256 --max-num-seqs 4 --gpu-memory-utilization 0.2'

Creating this repo

• Code: .meta/create_model.py
• Model structure: .meta/model.log
• Environment: .meta/env.json

This repo is generated by GPT-6.1-Sol.