Jundot's picture
Jundot HF Staff
Upload Inkling-2-Small-4bit
b87fc01 verified
|
Raw History Blame Contribute Delete
1.74 kB
metadata
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
library_name: mlx
pipeline_tag: image-text-to-text
base_model: thinkingmachines/Inkling-2-Small
tags:
  - moe
  - mlx

mlx-community/Inkling-2-Small-4bit

This model mlx-community/Inkling-2-Small-4bit was converted to MLX format from thinkingmachines/Inkling-2-Small using mlx-lm with ml-explore/mlx-lm#2061. Refer to the original model card for more details on the model.

It takes text and image input. The vision tower is included. The DSpark draft is not included.

The routed experts keep the original quantization. They are NVFP4 with a per-expert global scale in layers 0 to 39 and MXFP8 in layers 40 to 47. The attention, embedding and output head weights are 4-bit affine with group size 64. The vision tower stays in bfloat16 and the router in float32.

Tested with the branch of that pull request.

Use with mlx

pip install -U git+https://github.com/ml-explore/mlx-lm.git@refs/pull/2061/head
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Inkling-2-Small-4bit")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_dict=False,
    )

response = generate(model, tokenizer, prompt=prompt, verbose=True)

For image input, run mlx_lm.server and send the images as image_url parts of a chat message.