Instructions to use mlx-community/Qwen2-Math-72B-Instruct-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use mlx-community/Qwen2-Math-72B-Instruct-8bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("mlx-community/Qwen2-Math-72B-Instruct-8bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- MLX LM
How to use mlx-community/Qwen2-Math-72B-Instruct-8bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "mlx-community/Qwen2-Math-72B-Instruct-8bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "mlx-community/Qwen2-Math-72B-Instruct-8bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mlx-community/Qwen2-Math-72B-Instruct-8bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Atomic Chat
Commit History
-
Upload folder using huggingface_hub (#1)
Verified signature- 67d3b889ec1cb557b2972d1c0075fd1602a7b75dba2ba90f5108b2d714639886 (61c0e14c646382c1f918852960a805f2dedc997c) - edfec6ed588944c71450f7b524a0768f75e70be6216da936a7500a43d0509af7 (b47189566b9500c098823cb2eb9af593d4f8c489) - f134b89daea17165299704d1f44b6fec5d7c3a93fcc163c72a402acdae915afa (123cb60b65a3c663bd858d7f3e103473efbe83ce) - fc85e5fd4808c432af7bd75ae8e72fc7c97b024a5d22a2e932ed7bf7be0d8b7f (c56638fc80cbec80087f70ae874e1e38b9a84931) - b490dcd25219f9899dac5b26d4f124b13e7abafa0e83f5126a8e742ba08333f8 (fb5a25955c7c696182e5a0a8467c7f22b3f572a7) - 5a57103480713e8a889e4ea485ffdb1cf7ac5c2845add3077296bbde5e4c0709 (cc862072af59d0573b122df170c3d17ef28a3e71) - 1ec52663ba922c6eb8602b914219a67d02a9099d7ff52f01528f0d4f80e5e0f5 (22d41602d2ebb9a7943eefb6f0770c401d0780a1) - b69cf8822739a60f961499bf52d3ec4d9bb6ed8cba9cc27d324942ff71eef3ad (38e9b1ba8184b978b016048fe7109182779eea92) - 182970dd5abe4fe350cb750846cf43694dad335a8306136c4b69820569b7a18b (e813a3771c2a33d1b2c832bdce8e653d4122ac7a) - 74b15acf…9a6b50f