Instructions to use OEvortex/HelpingAI-Vision with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OEvortex/HelpingAI-Vision with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="OEvortex/HelpingAI-Vision", trust_remote_code=True)# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("OEvortex/HelpingAI-Vision", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use OEvortex/HelpingAI-Vision with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "OEvortex/HelpingAI-Vision" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OEvortex/HelpingAI-Vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/OEvortex/HelpingAI-Vision
- SGLang
How to use OEvortex/HelpingAI-Vision with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "OEvortex/HelpingAI-Vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OEvortex/HelpingAI-Vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "OEvortex/HelpingAI-Vision" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "OEvortex/HelpingAI-Vision", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use OEvortex/HelpingAI-Vision with Docker Model Runner:
docker model run hf.co/OEvortex/HelpingAI-Vision
Update README.md
Browse files
README.md
CHANGED
|
@@ -24,19 +24,13 @@ widget:
|
|
| 24 |
|
| 25 |
## Model details
|
| 26 |
|
| 27 |
-
The
|
| 28 |
-
Having high-quality embeddings for smaller parts of the image helps to extract more details and understand the scene better.
|
| 29 |
|
| 30 |
-
For every crop of the image,
|
| 31 |
-
gives the token embedding of size [N, 2560]. Right now, the tokens do not contain explicit information about their position in the original image. I plan to add it later.
|
| 32 |
|
| 33 |
-
|
| 34 |
-
[SigLIP 400M](https://huggingface.co/timm/ViT-SO400M-14-SigLIP-384).
|
| 35 |
-
|
| 36 |
-
The context length during training was 1200 tokens, as the L4 GPUs I used didn't allow me to get more.
|
| 37 |
-
|
| 38 |
-
As Dolphin 2.6 Phi, LLaVA-3b uses ChatML prompt format:
|
| 39 |
|
|
|
|
| 40 |
```
|
| 41 |
<|im_start|>system
|
| 42 |
You are Dolphin, a helpful AI assistant.<|im_end|>
|
|
|
|
| 24 |
|
| 25 |
## Model details
|
| 26 |
|
| 27 |
+
The fundamental concept behind HelpingAI-Vision is to generate one token embedding per N parts of an image, as opposed to producing N visual token embeddings for the entire image. This approach, based on the Dolphin 2.6 Phi model and incorporating the LLaVA adapter, aims to enhance scene understanding by capturing more detailed information.
|
|
|
|
| 28 |
|
| 29 |
+
For every crop of the image, an embedding is generated using the full SigLIP encoder (size [1, 1152]). Subsequently, all N embeddings undergo processing through the LLaVA adapter, resulting in a token embedding of size [N, 2560]. Currently, these tokens lack explicit information about their position in the original image, with plans to incorporate positional information in a later update.
|
|
|
|
| 30 |
|
| 31 |
+
HelpingAI-Vision was fine-tuned from Dolphin 2.6 Phi, leveraging the vision tower from SigLIP 400M. The training process had a context length of 1200 tokens, determined by the limitations of the L4 GPUs used.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
+
The model adopts the ChatML prompt format, suggesting its potential application in chat-based scenarios. If you have specific queries or would like further details, feel free
|
| 34 |
```
|
| 35 |
<|im_start|>system
|
| 36 |
You are Dolphin, a helpful AI assistant.<|im_end|>
|