Instructions to use mlx-community/Qwen2-72B-Instruct-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use mlx-community/Qwen2-72B-Instruct-4bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("mlx-community/Qwen2-72B-Instruct-4bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- MLX LM
How to use mlx-community/Qwen2-72B-Instruct-4bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "mlx-community/Qwen2-72B-Instruct-4bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "mlx-community/Qwen2-72B-Instruct-4bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mlx-community/Qwen2-72B-Instruct-4bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Atomic Chat
Commit History
-
Upload folder using huggingface_hub (#1)
Verified signature- 71bcd15742157ffec7bb6334aa33af58b660b89d9cde51e632527711fe5a2673 (ab1419d2375b72e7089718c7cd6da37c5366a8ff) - 196c33adde134044a0480eb441261681736405fed71963abc0886e5e4d94a178 (7a2e94771d43c30cda68299fc82e5c7fedd4329e) - 4836ff7ddbd1f93dca2fd5ea1d095afd3ab54938e13bbba4bd107da1deb9d11f (50e52a1bedb3d7a01675be37b0960020b907ea59) - 77a7de0bfaa2c0b62f1054d3f346a05fdce1f8076067f285bc3fc129c3b79084 (1a4f0829550d09d122e72296095c13e9dc49d004) - 932238af79d2f705f7552b49e9d475ca3c947be60fe20b759f82ce34bdba2aca (9c5f5e2dbc4eaf8c97b6180d5f34c17124cc0bfd) - 74cac052ce9ed02fd76de7b106c960b7fac1688c36d50e6ee069b73fb33f1513 (093e42b1a0e4151d83eec002ca74be4d5b75dfdb) - 078a825dbe95bb3cf12f5e089f73982c2fd4a125744020a4b630bd001131432b (aed605f6fc06015cca74889d9397761603513889) - 6b5c03f95e92018af4ef8d94cd11183cd6ab565ee103257eba78e547f9f2365e (9e41ce54b1af22ee025357efadd33d46ad2b1714) - 79b8ae89550590b095ee03081b9408c7237e582376f72841485653c6e794de01 (92721a37e972880a0acc442106aedb636dd3121c)d410561