Image-Text-to-Text
Transformers
Safetensors
multilingual
phi3_v
text-generation
nlp
code
vision
conversational
custom_code
Instructions to use rfpaul/Phi-3.5-vision-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use rfpaul/Phi-3.5-vision-instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="rfpaul/Phi-3.5-vision-instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("rfpaul/Phi-3.5-vision-instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use rfpaul/Phi-3.5-vision-instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "rfpaul/Phi-3.5-vision-instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rfpaul/Phi-3.5-vision-instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/rfpaul/Phi-3.5-vision-instruct
- SGLang
How to use rfpaul/Phi-3.5-vision-instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "rfpaul/Phi-3.5-vision-instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rfpaul/Phi-3.5-vision-instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "rfpaul/Phi-3.5-vision-instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "rfpaul/Phi-3.5-vision-instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use rfpaul/Phi-3.5-vision-instruct with Docker Model Runner:
docker model run hf.co/rfpaul/Phi-3.5-vision-instruct
Commit History
-
Bug fix for Transformers v4.49.0 per https://huggingface.co/microsoft/Phi-3.5-vision-instruct/discussions/39/files; added a note to README.md to indicate this repo as a fork.
fcbbbe1
-
fix_rope_scaling (#28)
Verified signature- fix longrope scaling (c9cf54d18c8e2eb89d78edd1995d73cdb8eb8b74)4a0d683
-
Add proper library name (#23)
Verified signature- Add proper library name (8c0cf74f939e34e4efe78f180232f9f92438f093) Co-authored-by: Omar Sanseviero <osanseviero@users.noreply.huggingface.co>c69b58c
-
Fix task tag (#16)
Verified signature- Fix task tag (23aac7404718a0afe2121ea60ef303c821216b10) Co-authored-by: Merve Noyan <merve@users.noreply.huggingface.co>c881ad5
-
Fix spelling error in README (#10)
Verified signature- Fix spelling error in README (996b6a9a9e04588d504c3037e04dd5637842d744) Co-authored-by: pluto <fromplutowithlove@users.noreply.huggingface.co>c68f852
-
Fix AutoImageProcessor mapping (#4)
Verified signature- Fix AutoImageProcessor mapping (4153d2b2bdd603d889b354d229661a16ad3734f2) Co-authored-by: Zifei Tong <zifeitong@users.noreply.huggingface.co>ee47981 -
Fixing Scores for Gemini 1.5 Flash and GPT-4o (#2)
Verified signature- Fixing Scores for Gemini 1.5 Flash and GPT-4o (e09f40e8500bdfbe7df35a1be98144944b92fabb) Co-authored-by: Victor Fragoso <vfragoso@users.noreply.huggingface.co>1c4d695 -
Update README.md
Verified signature0eb7016 -
Update README.md
Verified signaturebea9545 -
fix_import (#1)
Verified signature- fix import (b330978e4027db0529fd11c72c013fba5145c917) - fix import (6ff8b355df43fecb64be8fb9c1ad700765b38fd3) Co-authored-by: Haiping Wu <haipingwu@users.noreply.huggingface.co>64f88b6 -
upload model card
7da0ee8
-
upload initial files
249c1c2
-
initial commit
Verified signature40a03b9