Image-Text-to-Text
MLX
English
apple-silicon
Mixture of Experts
mixture-of-experts
vision-language
gemma
falcon-perception
inference
Instructions to use waltgrace/mlx-expert-sniper with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use waltgrace/mlx-expert-sniper with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("waltgrace/mlx-expert-sniper") config = load_config("waltgrace/mlx-expert-sniper") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Commit History
-
Add Gemma 4-26B-A4B support: 4.15 tok/s on M4 Mac Mini
Nico +1: Claude Opus 4.6 (1M context)Replace placeholder Gemma 4 engine with working implementation: - Custom forward pass for Gemma 4 architecture (sliding/full attention, layer scalars, dual layernorms, dense MLP + MoE in parallel) - Mixed quantization handling (experts 4-bit, dense MLP 8-bit) - Cache-aware routing bias=1.5 (steers router toward cached experts) - Gemma 4 chat template encoder (turn_start/turn_end tokens 105/106) - gelu_approx activation in expert FFN Replace preprocess_gemma4.py with SwitchLinear unstack: - Loads mlx-community/gemma-4-26b-a4b-it-4bit (15.6 GB) instead of bf16 source - Unstacks (128, out, in) experts into per-expert bin blocks - Preserves bfloat16 bytes via uint16 view (no precision loss) Wire up auto-dispatch in: - sniper.py (SniperEngine.from_dir auto-detects gemma4 model_type) - generate.py (generate_stream forks to _gemma4_generate_stream) - calibrate.py (_build_engine handles gemma4 path) Update download registry: gemma4-26b now points to mlx-community 4-bit version (15.6 GB)…3f56a7b
-
Add Gemma 4 MLX model class + preprocess
Verified signaturecc1d5e2 -
Add Gemma 4 MLX model class + preprocess
Verified signature4a30158 -
Add Gemma 4 MLX model class + preprocess
Verified signature337dcd8 -
Add Gemma 4 MLX model class + preprocess
Verified signaturee2bb666 -
Add Gemma 4 MLX model class + preprocess
Verified signature7409761 -
Add Coder model + multi-model fixes
Verified signature46f93d5 -
Add Coder model + multi-model fixes
Verified signature3a81687 -
Add Coder model + multi-model fixes
Verified signature62e0ebd -
Add Coder model + multi-model fixes
Verified signature7878baa -
Add Coder model + multi-model fixes
Verified signaturebded42e -
Add chat command + shared generate module
Verified signature3e350f6 -
Add chat command + shared generate module
Verified signature8f6ad09 -
Add 30B support + bias sweep results
Verified signature517be8a -
Add 30B support + bias sweep results
Verified signature8829376 -
Add 30B support + bias sweep results
Verified signature22d3dc7 -
Add Ollama-compatible serve command
Verified signaturef7efda4 -
Add Ollama-compatible serve command
Verified signature55fe21b -
Add download command: mlx-sniper download qwen3.5-35b
Verified signaturee1498f9 -
Add download command: mlx-sniper download qwen3.5-35b
Verified signatured694ca1
-
Update src/mlx_expert_sniper/preprocess.py
Verified signature05209e2 -
Update src/mlx_expert_sniper/expert_io.py
Verified signaturebc2c2b1 -
Update src/mlx_expert_sniper/engine.py
Verified signature7e0ed9b -
Update src/mlx_expert_sniper/coactivation.py
Verified signature482f969 -
Update src/mlx_expert_sniper/calibrate.py
Verified signature9110344 -
Update src/mlx_expert_sniper/cli.py
Verified signature1a7f86e -
Update src/mlx_expert_sniper/__init__.py
Verified signature0535634
-
v0.2.0: Add Qwen3.5-35B-A3B support (5.78 tok/s, 19.5 GB on 16 GB RAM)
Verified signatured14a3c2
-
v0.1.0: MoE expert sniping for MLX — run models larger than your RAM
Verified signature2d8a9bb