Instructions to use nvidia/Flora-Captioner with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/Flora-Captioner with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="nvidia/Flora-Captioner")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("nvidia/Flora-Captioner")
model = AutoModelForMultimodalLM.from_pretrained("nvidia/Flora-Captioner", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/Flora-Captioner with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/Flora-Captioner"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Flora-Captioner",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/Flora-Captioner

SGLang

How to use nvidia/Flora-Captioner with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/Flora-Captioner" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Flora-Captioner",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/Flora-Captioner" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Flora-Captioner",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use nvidia/Flora-Captioner with Docker Model Runner:
```
docker model run hf.co/nvidia/Flora-Captioner
```

Omar L commited on 21 days ago

Commit

ce5e7aa

verified ·

1 Parent(s): 6b2302c

Correct checkpoint parameter-count metadata

Browse files

Set total model parameters to 27356728560; retain all checkpoint tensors.

Files changed (3) hide show

MODEL_PROVENANCE.json +12 -1
SOURCE_MANIFEST.sha256 +1 -1
model.safetensors.index.json +1 -1

MODEL_PROVENANCE.json CHANGED Viewed

@@ -9,5 +9,16 @@
   "schema_version": "nvember_data_engine_model_provenance_v1",
   "source_full_sha256_verified": true,
   "source_mutated": false,
-  "training_lineage": "mecka_v3_redelivered_t2"
 }

   "schema_version": "nvember_data_engine_model_provenance_v1",
   "source_full_sha256_verified": true,
   "source_mutated": false,
+  "training_lineage": "mecka_v3_redelivered_t2",
+  "packaging_adjustments": [
+    {
+      "file": "model.safetensors.index.json",
+      "change": {
+        "metadata.total_parameters": 27356728560
+      },
+      "original_sha256": "54adabdb313601233d9cbdd43a5f6ed8594f94bb95ad50459bdfe23fae901428",
+      "packaged_sha256": "0424520f63f04816eee4d229a4487dc9744866ceb46513e591dd987e401b1663",
+      "weights_changed": false
+    }
+  ]
 }

SOURCE_MANIFEST.sha256 CHANGED Viewed

@@ -4,7 +4,7 @@ db4e25f7f009c05b70fc2982a4d1fe3b62c63a000351bf4a0316ae133695af13  generation_con
 a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d  merges.txt
 779c177b0712b3382ccbfb88f50cf2bcc08193c187f148f73c3bd6d6504ef52f  model-00001-of-00002.safetensors
 e2821612a17cf57b21ee075dad71c6a29f4f400cb234516b0be3ce53405b7351  model-00002-of-00002.safetensors
-54adabdb313601233d9cbdd43a5f6ed8594f94bb95ad50459bdfe23fae901428  model.safetensors.index.json
 27225450ac9c6529872ee1924fcb0962ff5634834f817040f444118116f4e516  preprocessor_config.json
 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42  tokenizer.json
 316230d6a809701f4db5ea8f8fc862bc3a6f3229c937c174e674ff3ca0a64ac8  tokenizer_config.json

 a9d356d7bdf1ef4949e3e748e95b8e10ad9d4e2e838eddc38a0a7b6b94d1db8d  merges.txt
 779c177b0712b3382ccbfb88f50cf2bcc08193c187f148f73c3bd6d6504ef52f  model-00001-of-00002.safetensors
 e2821612a17cf57b21ee075dad71c6a29f4f400cb234516b0be3ce53405b7351  model-00002-of-00002.safetensors
+0424520f63f04816eee4d229a4487dc9744866ceb46513e591dd987e401b1663  model.safetensors.index.json
 27225450ac9c6529872ee1924fcb0962ff5634834f817040f444118116f4e516  preprocessor_config.json
 5f9e4d4901a92b997e463c1f46055088b6cca5ca61a6522d1b9f64c4bb81cb42  tokenizer.json
 316230d6a809701f4db5ea8f8fc862bc3a6f3229c937c174e674ff3ca0a64ac8  tokenizer_config.json

model.safetensors.index.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
   "metadata": {
-    "total_parameters": 3054832,
     "total_size": 54713457120
   },
   "weight_map": {

 {
   "metadata": {
+    "total_parameters": 27356728560,
     "total_size": 54713457120
   },
   "weight_map": {