Instructions to use vikhyatk/moondream2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use vikhyatk/moondream2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="vikhyatk/moondream2", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("vikhyatk/moondream2", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use vikhyatk/moondream2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "vikhyatk/moondream2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vikhyatk/moondream2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/vikhyatk/moondream2
- SGLang
How to use vikhyatk/moondream2 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "vikhyatk/moondream2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vikhyatk/moondream2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "vikhyatk/moondream2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "vikhyatk/moondream2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use vikhyatk/moondream2 with Docker Model Runner:
docker model run hf.co/vikhyatk/moondream2
Commit History
-
Update handler.py
Verified signature43b595d
-
Creating a handler.py file to support HF dedicated inference endpoints
Verified signatureHi all, Made a quick handler.py file for people to use. It works for me pretty well. Feel free to test it out :) Here is an example for when making a request to the endpoint: import requests import base64 import json # Endpoint URL api_url = 'your API URL' # Path to the image file you want to test with image_path = '' # Replace with the path to your image file # The question to send to the API question = 'what is going on in this picture?' # Open the image file in binary mode and encode it in base64 with open(image_path, 'rb') as img: encoded_image = base64.b64encode(img.read()).decode('utf-8') # Prepare the JSON payload payload = { 'inputs': { 'image': encoded_image, 'question': question } } # Set the headers headers = { "Accept" : "application/json", "Authorization": "Bearer HUGGINGFACE_TOKEN", "Content-Type": "application/json" } # Send the POST request response = requests.post(api_url, headers=headers, data=json.dumps(payload)) # Print the re…aa98b0a
-
Update README.md
Verified signature5d82fdf
-
update version
6275e43 -
Upload Moondream
Verified signature30c7cdf
-
Upload moondream2-text-model-f16.gguf
Verified signature4cb9e48 -
Upload moondream2-mmproj-f16.gguf
Verified signaturee67286f
-
Create versions.txt
Verified signature4774575
-
Upload Moondream
Verified signatureab90666 -
Update config.json
Verified signatured49116c -
Update config.json
Verified signaturefec13c3 -
Upload Moondream
Verified signature848799b -
Update config.json
Verified signature169f11a -
Upload Moondream
Verified signaturef416500 -
Upload Moondream
Verified signatured8914b8 -
update gguf files
ea7d26c
-
Update README.md
Verified signature11d40d7
-
Update README.md
Verified signature46d1e55 -
Upload Moondream
Verified signature9ba2958
-
Upload 2 files
Verified signature0691a81
-
Upload Moondream
Verified signature2b705ee -
Update README.md
Verified signature010e389 -
Update README.md
Verified signaturea51ee63 -
Upload Moondream
Verified signature66847d8 -
Update README.md
Verified signature2a4b8c9
-
Add new type for vllm models (#4)
Verified signature- Add new type for vllm models (62e51d9447f45644098de3f33b2a36e3e4692bf8) Co-authored-by: Omar Sanseviero <osanseviero@users.noreply.huggingface.co>66b7ffa
-
Update README.md
Verified signatureeead139 -
Upload Moondream
Verified signaturec064b4c
-
Update README.md
Verified signaturefc49662 -
Upload Moondream
Verified signature4a8fa31
-
Update README.md
Verified signature24fcff5 -
Update README.md
Verified signaturedffbe5f -
Upload Moondream
Verified signatureb3d72d1 -
Upload Moondream
Verified signaturee04241e -
Upload Moondream
Verified signaturee041484 -
Upload tokenizer
Verified signature3a1302b -
Upload Moondream
Verified signaturef5943d1 -
initial commit
Verified signaturede8fafd