Image-Text-to-Text
Transformers
Safetensors
inkling_mm_model
fp8
Mixture of Experts
vllm
llm-compressor
compressed-tensors
conversational
Instructions to use RedHatAI/Inkling-FP8-dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use RedHatAI/Inkling-FP8-dynamic with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="RedHatAI/Inkling-FP8-dynamic") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("RedHatAI/Inkling-FP8-dynamic") model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/Inkling-FP8-dynamic", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use RedHatAI/Inkling-FP8-dynamic with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "RedHatAI/Inkling-FP8-dynamic" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/Inkling-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/RedHatAI/Inkling-FP8-dynamic
- SGLang
How to use RedHatAI/Inkling-FP8-dynamic with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "RedHatAI/Inkling-FP8-dynamic" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/Inkling-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "RedHatAI/Inkling-FP8-dynamic" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/Inkling-FP8-dynamic", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use RedHatAI/Inkling-FP8-dynamic with Docker Model Runner:
docker model run hf.co/RedHatAI/Inkling-FP8-dynamic
Add files using upload-large-folder tool
Browse files- model-00088-of-00108.safetensors +3 -0
- model-00091-of-00108.safetensors +3 -0
- model-00092-of-00108.safetensors +3 -0
- model-00093-of-00108.safetensors +3 -0
- model-00094-of-00108.safetensors +3 -0
- model-00095-of-00108.safetensors +3 -0
- model-00096-of-00108.safetensors +3 -0
- model-00097-of-00108.safetensors +3 -0
- model-00098-of-00108.safetensors +3 -0
- model-00099-of-00108.safetensors +3 -0
- model-00100-of-00108.safetensors +3 -0
- model-00101-of-00108.safetensors +3 -0
- model-00102-of-00108.safetensors +3 -0
- model-00103-of-00108.safetensors +3 -0
- model-00104-of-00108.safetensors +3 -0
- model-00105-of-00108.safetensors +3 -0
- model-00106-of-00108.safetensors +3 -0
- model-00107-of-00108.safetensors +3 -0
- mtp.safetensors +3 -0
model-00088-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2d927e2f6db1b2c32bc37893398f39799d5f78250a03ea947c786851d27315e6
|
| 3 |
+
size 9968878892
|
model-00091-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6fd0dd634deccfec7b5eca8542514e82b10909f88e05c91ec572cbb3f43d42ca
|
| 3 |
+
size 9833575112
|
model-00092-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2e1dc42a9dba50cb59f21df80ce1873e05b91b17b03679ef0af0f80d743ab8e0
|
| 3 |
+
size 10327503640
|
model-00093-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d5c2e5dff4ad498075ed78ef48475318b6cdbba9e4921583558d83b81e61e3d6
|
| 3 |
+
size 9691988336
|
model-00094-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c86cbdfee576a3f3f0128d2ad7ae68b49ab3a857bdbb22f5d721633b39c91104
|
| 3 |
+
size 10214460232
|
model-00095-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fc596e78aa11fa4a46e18cf32229589017b258ba3a9132672d12d57ad0e44b63
|
| 3 |
+
size 9868198480
|
model-00096-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2dad7593f57da099851075b176c918b42e50cfcffde7acd3c6d5a75d689b9fb9
|
| 3 |
+
size 9924946848
|
model-00097-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cd1f249b34af2476e366cc338b708cb11e24122f1d438508fe01f2f81953c9f9
|
| 3 |
+
size 9842995808
|
model-00098-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:38056e8eeaf6524a961e77e93af163dcf10ff5f2d036870d4da032a347192cf7
|
| 3 |
+
size 9928028524
|
model-00099-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2e32f4eb72c776aa2ef7efbcd83daa283bcf1dbe712f9b611b1a465c22b038a5
|
| 3 |
+
size 9921861536
|
model-00100-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fa4d24dcfa2327aba2a45dd579b8f4e81dcb184f4239cd637606ca824df8ceed
|
| 3 |
+
size 5262939100
|
model-00101-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bdc0efed27ceed9d5ad5560564ce3fe3e8a645232f06c24f60d83e89a7bc41bc
|
| 3 |
+
size 9666822392
|
model-00102-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:03e13ba965223617b9380ed61bb2956ae81c462938bfb94c840fba025f86272e
|
| 3 |
+
size 9833563420
|
model-00103-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:507b0022ac22167735bef1eda17a0b8007c42241d699f32a33cd458910c162c5
|
| 3 |
+
size 4910482496
|
model-00104-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ea8a900042601fc6b21321f8880a9a856721614f0125420b1d8453d3997da260
|
| 3 |
+
size 10230400522
|
model-00105-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5ee3ea8d6f0e450e32011384fccd3d33f7190b2ea9935d0ef7f38825de0eb2a0
|
| 3 |
+
size 10318093648
|
model-00106-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b3108fa3b8f4e28b937bed7d1f75f21e394505c5029d703003bbf41994e75cc4
|
| 3 |
+
size 9855612312
|
model-00107-of-00108.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:da5b40c932aa9ddd5ce3b1f8082ce80dc36acbcdac3f963dae804976c50f371c
|
| 3 |
+
size 19428016376
|
mtp.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1f545cd9cb67833fa92a39ae5316e2dabf3e48ca4571b37ec54d7b6b80f70191
|
| 3 |
+
size 6294039568
|