| --- |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| base_model: |
| - mistralai/Mistral-Large-4-1T-A52B |
| --- |
| |
| This is a randomly initialized tiny model based on [mistralai/Mistral-Large-4-1T-A52B](https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B) for testing and debugging. |
|
|
| | Feature | Source | This repo | |
| |---|---|---| |
| | Checkpoint tensor elements | 7.13B BF16, 63.84M F32, 1.05T F8_E4M3 | 54.84M BF16, 320 F32, 1.31M F8_E4M3 | |
| | Safetensors weight files | 1.06 TB (54 files) | 111.13 MB (consolidated.safetensors) | |
| | Architecture / input | Multimodal MoE; text and image | Same | |
| | Vocabulary rows / tied embeddings | 204,800 / false | 204,800 / true | |
| | Backbone layers | 72 | 12: three complete RoPE/RoPE/RoPE/NoPE cycles | |
| | Text hidden size / attention head dimension | 6,144 / 64 | 128 / 32 | |
| | Routed / selected / shared experts | 128 / 4 / 1 | 32 / 4 / 1 | |
| | DSA indexer | 32 heads; 128-wide; top-2,048 | 4 heads; 128-wide; top-128 | |
| | Cross-layer KV sharing | Source pattern | Two complete three-own/one-shared NoPE cycles | |
| | Vision tower | 30 layers; 2,048 width | 4 layers; 64 width | |
| | DSpark | 3 dense MLA stages; rank 512; depth 3 | 3 dense MLA stages; rank 8; depth 3 | |
| | Checkpoint precision | E4M3 block-FP8 + FP32 scales; BF16 exclusions | Same format; 64×64 blocks | |
|
|
| TP=2 vLLM uses a 64-wide FP8 activation group per rank, which fixes the residual width at 128. The checkpoint retains 64×64 FP8 blocks, a 512-wide MLA latent KV required by H20 sparse MLA, and four 32-wide attention heads. The input/output BF16 token table is tied, saving one 52.43 MB matrix; the optimized vLLM kernel constraints still place the checkpoint above the 30 MB soft target. |
|
|
| | File path | Size | |
| |------|------| |
| | consolidated.safetensors | 111.1MB | |
|
|
| ### Example usage |
|
|
| #### Transformers (✅ verified) |
|
|
| ```python |
| def run_transformers(model_id: str): |
| import torch |
| from PIL import Image |
| from transformers import AutoModelForImageTextToText, AutoProcessor |
| from transformers.utils.quantization_config import FineGrainedFP8Config |
| |
| model = AutoModelForImageTextToText.from_pretrained( |
| str(model_id), |
| dtype=torch.bfloat16, |
| device_map="cuda", |
| attn_implementation="sdpa", |
| quantization_config=FineGrainedFP8Config(dequantize=True), |
| ) |
| processor = AutoProcessor.from_pretrained(str(model_id)) |
| messages = [ |
| { |
| "role": "user", |
| "content": [ |
| {"type": "image", "image": Image.new("RGB", (56, 56), (30, 120, 220))}, |
| {"type": "text", "text": "Describe this image in one word."}, |
| ], |
| } |
| ] |
| inputs = processor.apply_chat_template( |
| messages, |
| add_generation_prompt=True, |
| tokenize=True, |
| return_dict=True, |
| return_tensors="pt", |
| reasoning_effort="low", |
| ).to(model.device) |
| output = model.generate(**inputs, do_sample=False, max_new_tokens=10) |
| print(processor.decode(output[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) |
| |
| |
| if __name__ == "__main__": |
| run_transformers(model_id="tiny-random/mistral-large-4") |
| ``` |
|
|
| #### vLLM offline (✅ verified) |
|
|
| Install the snapshot package with `pip install --no-deps <model directory>` before starting vLLM. |
|
|
| ```python |
| def run_vllm(model_id: str): |
| import os |
| |
| os.environ.setdefault("VLLM_USE_FLASHINFER_SAMPLER", "0") |
| os.environ.setdefault("VLLM_ALLREDUCE_USE_FLASHINFER", "0") |
| os.environ.setdefault("VLLM_ALLREDUCE_USE_SYMM_MEM", "0") |
| from PIL import Image |
| from transformers import AutoProcessor |
| from vllm import LLM, SamplingParams |
| |
| model_id = str(model_id) |
| processor = AutoProcessor.from_pretrained(model_id) |
| llm = LLM( |
| model=model_id, |
| trust_remote_code=True, |
| tensor_parallel_size=2, |
| speculative_config={"method": "dspark", "num_speculative_tokens": 3}, |
| kv_cache_dtype="bfloat16", |
| max_model_len=512, |
| max_num_batched_tokens=1024, |
| max_num_seqs=2, |
| gpu_memory_utilization=0.1, |
| disable_custom_all_reduce=True, |
| kernel_config={"linear_backend": "triton"}, |
| attention_config={"backend": "FLASHINFER_MLA_SPARSE_SM90"}, |
| ) |
| sampling = SamplingParams(temperature=0, max_tokens=10, ignore_eos=True) |
| try: |
| print(llm.generate(["Hello"], sampling)[0].outputs[0].text) |
| image = Image.new("RGB", (56, 56), "blue") |
| inputs = processor.apply_chat_template( |
| [ |
| { |
| "role": "user", |
| "content": [ |
| {"type": "image", "image": image}, |
| {"type": "text", "text": "Describe this image in one word."}, |
| ], |
| } |
| ], |
| add_generation_prompt=True, |
| tokenize=True, |
| return_dict=True, |
| return_tensors="pt", |
| reasoning_effort="low", |
| ) |
| result = llm.generate( |
| {"prompt_token_ids": inputs["input_ids"][0].tolist(), "multi_modal_data": {"image": image}}, |
| sampling, |
| ) |
| print(result[0].outputs[0].text) |
| finally: |
| llm.llm_engine.engine_core.shutdown(timeout=30) |
| |
| |
| if __name__ == "__main__": |
| run_vllm(model_id="tiny-random/mistral-large-4") |
| ``` |
|
|
| #### vLLM serving (unverified) |
|
|
| Use the official Mistral Large 4 vLLM image. |
|
|
| ```bash |
| docker run --rm --runtime nvidia --gpus all --ipc=host -p 8000:8000 --entrypoint bash \ |
| -v "$(pwd)/mistral-large-4:/model:ro" vllm/vllm-openai:mistral-large-4 \ |
| -lc 'pip install --no-deps /model && vllm serve /model \ |
| --tensor-parallel-size 2 \ |
| --speculative-config '{"method":"dspark","num_speculative_tokens":3}' \ |
| --kv-cache-dtype bfloat16 --disable-custom-all-reduce \ |
| --kernel-config '{"linear_backend":"triton"}' \ |
| --attention-config '{"backend":"FLASHINFER_MLA_SPARSE_SM90"}' \ |
| --limit-mm-per-prompt '{"image":1}' --max-model-len 2048 \ |
| --max-num-batched-tokens 256 --max-num-seqs 4 --gpu-memory-utilization 0.2' |
| ``` |
|
|
| ### Creating this repo |
|
|
| - Code: [.meta/create_model.py](.meta/create_model.py) |
| - Model structure: [.meta/model.log](.meta/model.log) |
| - Environment: [.meta/env.json](.meta/env.json) |
|
|
| *This repo is generated by GPT-6.1-Sol.* |
