Instructions to use mlx-community/Qwen2-72B-Instruct-8bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use mlx-community/Qwen2-72B-Instruct-8bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("mlx-community/Qwen2-72B-Instruct-8bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- MLX LM
How to use mlx-community/Qwen2-72B-Instruct-8bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "mlx-community/Qwen2-72B-Instruct-8bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "mlx-community/Qwen2-72B-Instruct-8bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "mlx-community/Qwen2-72B-Instruct-8bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Atomic Chat
Commit History
-
Upload folder using huggingface_hub (#1)
Verified signature- 3ad7b429e38a8197b5944da14227ccc07ba47cd961c15401b976d3853bc83ac3 (8153cda67382b036d767887dc0da500ed46ba77f) - 685b1737f3ef6de93267d01bfbc7ef9a3ca265b84bfdbd383988b6af0bb4453c (521f7957b5bf89641ae017f1f37a7248758b946c) - 316fbfe33bc0de2710b740416508cdce1627e8bf07f22ddf481890a69826ae52 (ddb134229216b5116bddf84f457425712a7797ce) - 604822d9d92dfd51e06ff691e30d290b2d3adef3f32d9ed8b61460d5f0ab33a5 (ac2a35eb767cc404c72b8e40452496d46ada5c1e) - b5af0887219e8de9ed3f2002e84a89fff626bba37aca7b563422bac0acbbfa9e (fa961613d75604b123686e30f99cb901bddd0f97) - 510863c7c6ae61b56ecd800c679fb10c88b2806d6a604b3d9b963cbc2e55bd89 (17f51560ca9b77ef8a4bc3248990119cda3598bc) - fd023a8a4c85968af51224b8a0136ff128554efb43b962b62c37d549854d8ee0 (0b2bf0711bd28af3373b7071a52b8e3ffe1e18c0) - 3c24b8241aaf9eadbdcbecafd288605b1db79619fc2765b2768871b52922f1e8 (25cc9fe93e83d7428b58dfc0d323eca84a678e4d) - fcb0500df151cdad2fde9230b7521f780b6fc6053af090969a9666eec1a91006 (c3136caa0a088ec02787e1e3f2134afd1edcbdd3) - c96a4a67…6f7d783