# CLI de Servir

O CLI `transformers serve` é uma opção leve para servidores locais ou auto-hospedados. Ele evita o overhead operacional e de tempo de execução adicional de motores de inferência dedicados, como o vLLM. Use-o para avaliação, experimentação e implantações com carga moderada. Recursos como [continuous batching](../continuous_batching) aumentam a taxa de transferência e reduzem a latência.

> [!TIP]
> Para implantações de produção em larga escala, use vLLM ou SGLang com um modelo Transformer como backend. Saiba mais no guia [Backends de inferência](../community_integrations/transformers_as_backend).

O comando `transformers serve` inicia um servidor local compatível com o [OpenAI SDK](https://platform.openai.com/docs/overview). O servidor funciona com muitos aplicativos de terceiros e suporta as APIs REST abaixo.

- `/v1/chat/completions` para solicitações de texto, imagem, áudio e vídeo
- `/v1/completions` para completions de texto legados a partir de um prompt livre
- `/v1/responses` suporta a [Responses API](https://platform.openai.com/docs/api-reference/responses)
- `/v1/audio/transcriptions` para transcrições de áudio
- `/v1/systemone` responde a perguntas de resposta fechada (múltipla escolha, pontuação, sim/não) com uma probabilidade para cada resposta permitida
- `/v1/models` lista os modelos disponíveis para integrações de terceiros
- `/load_model` transmite o progresso do carregamento do modelo via SSE

Instale as dependências de serving.

```bash
pip install transformers[serving]
```

Execute `transformers serve` para iniciar um servidor. O endereço padrão do servidor é http://localhost:8000.

```shell
transformers serve
```

## v1/chat/completions

A API `v1/chat/completions` é baseada na [Chat Completions API](https://platform.openai.com/docs/api-reference/chat). Ela suporta solicitações de texto, imagem, áudio e vídeo para LLMs, VLMs e modelos multimodais. Use-a com `curl`, o [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) ou o cliente [OpenAI](https://platform.openai.com/docs/quickstart).

### Completions baseados em texto

```python
from huggingface_hub import InferenceClient

messages = [{"role": "user", "content": "What is the Transformers library known for?"}]
client = InferenceClient("http://localhost:8000")

result = client.chat_completion(messages, model="Qwen/Qwen2.5-0.5B-Instruct", max_tokens=256)
print(result.choices[0].message.content)
```

O [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) retorna uma string impressa.

```shell
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
```

```python
from huggingface_hub import InferenceClient

messages = [{"role": "user", "content": "What is the Transformers library known for?"}]
client = InferenceClient("http://localhost:8000")

stream = client.chat_completion(messages, model="Qwen/Qwen2.5-0.5B-Instruct", max_tokens=256, stream=True)
for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

O [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) retorna uma string impressa.

```shell
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What is the Transformers library known for?"
        }
    ],
)
print(completion.choices[0].message.content)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```shell
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "What is the Transformers library known for?"
        }
    ],
    stream=True,
)

for chunk in completion:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```shell
The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]
```

```shell
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "What is the Transformers library known for?"}],
    "temperature": 0.9,
    "max_tokens": 256
  }'
```

O comando retorna uma string JSON.

```json
{
  "id": "ade3578d-e2c3-47ed-b651-c014991a92f6",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "content": "The Transformers library is primarily known for its ability to create and manipulate large-scale language models [...]",
        "role": "assistant"
      }
    }
  ],
  "created": 1776184004,
  "model": "Qwen/Qwen2.5-0.5B-Instruct@main",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 256,
    "prompt_tokens": 37,
    "total_tokens": 293
  }
}
```

```shell
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "What is the Transformers library known for?"}],
    "temperature": 0.9,
    "max_tokens": 256,
    "stream": true
  }'
```

O comando retorna um fluxo de blocos.

```shell
data: {"id":"596d3941-754d-43b4-8773-9bf9940049f5","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776183469,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"596d3941-754d-43b4-8773-9bf9940049f5","choices":[{"delta":{"content":"The"},"index":0}],"created":1776183469,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}

(...)
```

### Completions baseados em imagens

```python
from huggingface_hub import InferenceClient

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
                }
            },
        ],
    }
]
client = InferenceClient("http://localhost:8000")

result = client.chat_completion(messages, model="Qwen/Qwen2.5-VL-7B-Instruct", max_tokens=256)
print(result.choices[0].message.content)
```

O [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) retorna uma string impressa.

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```python
from huggingface_hub import InferenceClient

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
                }
            },
        ],
    }
]
client = InferenceClient("http://localhost:8000")

stream = client.chat_completion(messages, model="Qwen/Qwen2.5-VL-7B-Instruct", max_tokens=256, stream=True)
for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

O [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) retorna uma string impressa.

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
                    }
                },
            ],
        }
    ],
)
print(completion.choices[0].message.content)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What's in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
                    }
                },
            ],
        }
    ],
    stream=True,
)

for chunk in completion:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What's in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 256
  }'
```

O comando retorna uma string JSON.

```json
{
  "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon [...]",
        "role": "assistant"
      }
    }
  ],
  "created": 1753366665,
  "model": "Qwen/Qwen2.5-VL-7B-Instruct@main",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 50,
    "prompt_tokens": 120,
    "total_tokens": 170
  }
}
```

```shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What's in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 256
  }'
```

O comando retorna um fluxo de blocos.

```shell
data: {"id":"f47ac10b-58cc-4372-a567-0e02b2c3d479","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1753366665,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"f47ac10b-58cc-4372-a567-0e02b2c3d479","choices":[{"delta":{"content":"The "},"index":0}],"created":1753366701,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"chat.completion.chunk","system_fingerprint":""}

(...)
```

### Completions baseados em áudio

Modelos multimodais como [Gemma 4](https://huggingface.co/google/gemma-4-E2B-it) e [Qwen2.5-Omni](https://huggingface.co/Qwen/Qwen2.5-Omni-3B) aceitam entrada de áudio por meio do tipo de conteúdo `input_audio` da OpenAI. Codifique o áudio em Base64 e especifique o formato (`mp3` ou `wav`).

```python
import base64
from huggingface_hub.utils import httpx
from huggingface_hub import InferenceClient

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe this audio."},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
        ],
    }
]
client = InferenceClient("http://localhost:8000")

result = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256)
print(result.choices[0].message.content)
```

O [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) retorna uma string impressa.

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```python
import base64
from huggingface_hub.utils import httpx
from huggingface_hub import InferenceClient

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe this audio."},
            {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
        ],
    }
]
client = InferenceClient("http://localhost:8000")

stream = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256, stream=True)
for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

O [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) retorna uma string impressa.

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```python
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe this audio."},
                {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
            ],
        }
    ],
)
print(completion.choices[0].message.content)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```python
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe this audio."},
                {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
            ],
        }
    ],
    stream=True,
)

for chunk in completion:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```shell
# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)

cat <<EOF > /tmp/audio_request.json
{
  "model": "google/gemma-4-E2B-it",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Transcribe this audio."},
        {"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
      ]
    }
  ]
}
EOF

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d @/tmp/audio_request.json
```

O comando retorna uma string JSON.

```json
{
  "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "This week, I traveled to Chicago to deliver my final farewell address to the nation [...]",
        "role": "assistant"
      }
    }
  ],
  "created": 1753366665,
  "model": "google/gemma-4-E2B-it@main",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 85,
    "prompt_tokens": 200,
    "total_tokens": 285
  }
}
```

```shell
# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)

cat <<EOF > /tmp/audio_request.json
{
  "model": "google/gemma-4-E2B-it",
  "stream": true,
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "Transcribe this audio."},
        {"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
      ]
    }
  ]
}
EOF

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d @/tmp/audio_request.json
```

O comando retorna um fluxo de blocos.

```shell
data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"content":"This"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}

(...)
```

> [!WARNING]
> O tipo de conteúdo `audio_url` é uma extensão que não faz parte do padrão da OpenAI e pode ser alterado em versões futuras.

Você também pode passar áudio por URL com o tipo de conteúdo `audio_url` para evitar a codificação base64.

```python
completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Transcribe this audio."},
                {"type": "audio_url", "audio_url": {"url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"}},
            ],
        }
    ],
)
```

### Completions baseados em vídeo

> [!WARNING]
> O tipo de conteúdo `video_url` é uma extensão que não faz parte do padrão da OpenAI e pode ser alterado em versões futuras.

Use o tipo de conteúdo `video_url` para entrada de vídeo. Se o modelo suportar áudio (por exemplo, Gemma 4, Qwen2.5-Omni), o servidor extrai a trilha de áudio do vídeo e a processa junto com os quadros visuais.

> [!TIP]
> O processamento de vídeo requer [torchcodec](https://github.com/pytorch/torchcodec). Instale-o com `pip install torchcodec`.

```python
from huggingface_hub import InferenceClient

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
            {"type": "text", "text": "What is happening in the video and what is the song about?"},
        ],
    }
]
client = InferenceClient("http://localhost:8000")

result = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256)
print(result.choices[0].message.content)
```

O [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) retorna uma string impressa.

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```python
from huggingface_hub import InferenceClient

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
            {"type": "text", "text": "What is happening in the video and what is the song about?"},
        ],
    }
]
client = InferenceClient("http://localhost:8000")

stream = client.chat_completion(messages, model="google/gemma-4-E2B-it", max_tokens=256, stream=True)
for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

O [InferenceClient](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/inference_client#huggingface_hub.InferenceClient) retorna uma string impressa.

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
                {"type": "text", "text": "What is happening in the video and what is the song about?"},
            ],
        }
    ],
    max_tokens=256,
)
print(completion.choices[0].message.content)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
                {"type": "text", "text": "What is happening in the video and what is the song about?"},
            ],
        }
    ],
    max_tokens=256,
    stream=True,
)

for chunk in completion:
    token = chunk.choices[0].delta.content
    if token:
        print(token, end="")
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
          {"type": "text", "text": "What is happening in the video and what is the song about?"}
        ]
      }
    ],
    "max_tokens": 256
  }'
```

O comando retorna uma string JSON.

```json
{
  "id": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": {
        "content": "The video captures a live music performance at a music festival or a large concert [...]",
        "role": "assistant"
      }
    }
  ],
  "created": 1753366665,
  "model": "google/gemma-4-E2B-it@main",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 256,
    "prompt_tokens": 350,
    "total_tokens": 606
  }
}
```

```shell
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "stream": true,
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
          {"type": "text", "text": "What is happening in the video and what is the song about?"}
        ]
      }
    ],
    "max_tokens": 256
  }'
```

O comando retorna um fluxo de blocos.

```shell
data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"role":"assistant"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}

data: {"id":"cb997e1d-98b9-414a-be89-1880288610ef","choices":[{"delta":{"content":"Based"},"index":0}],"created":1776254856,"model":"google/gemma-4-E2B-it@main","object":"chat.completion.chunk","system_fingerprint":""}

(...)
```

### Conversas de múltiplas rodadas[[completions]]

Para ter uma conversa de múltiplas rodadas, inclua o histórico completo da conversa na lista `messages` com papéis alternados de `user` e `assistant`. Como todos os servidores compatíveis com a OpenAI, a API é sem estado, portanto, cada solicitação deve conter o histórico completo da conversa.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    messages=[
        {"role": "user", "content": "What is the capital of France?"},
        {"role": "assistant", "content": "The capital of France is Paris."},
        {"role": "user", "content": "How many people live there?"},
    ],
)
print(completion.choices[0].message.content)
```

A pergunta subsequente "Quantas pessoas moram lá?" depende do contexto anterior, então o modelo responde sobre Paris.

```
As of 2021, the population of Paris is approximately 2.2 million people.
```

## v1/completions

A API `v1/completions` é baseada na [API legada de Completions](https://platform.openai.com/docs/api-reference/completions). Diferentemente de `/v1/chat/completions`, ela recebe um `prompt` de texto livre em vez de mensagens de chat e retorna o texto gerado em `choices[].text`. Isso é útil para modelos base (não instruct) e para tarefas de conclusão de texto em que um modelo de chat não é necessário. Ela também suporta `suffix` para inserção de texto no meio (fill-in-the-middle).

```shell
curl -X POST http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B",
    "prompt": "The capital of France is",
    "max_tokens": 20
  }'
```

O comando retorna a seguinte resposta.

```json
{
  "id": "chatcmpl-abc123",
  "object": "text_completion",
  "created": 1234567890,
  "model": "Qwen/Qwen2.5-0.5B@main",
  "choices": [
    {
      "text": " Paris, and the capital of the United States is Washington, D.C.",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 16,
    "total_tokens": 21
  }
}
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

# Non-streaming
completion = client.completions.create(
    model="Qwen/Qwen2.5-0.5B",
    prompt="The capital of France is",
    max_tokens=20,
)
print(completion.choices[0].text)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna o seguinte.

```shell
 Paris, and the capital of the United States is Washington, D.C.
```

O streaming também é suportado.

```python
stream = client.completions.create(
    model="Qwen/Qwen2.5-0.5B",
    prompt="The capital of France is",
    max_tokens=20,
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].text, end="")
```

## v1/responses

A [Responses API](https://platform.openai.com/docs/api-reference/responses) é o endpoint de API mais recente da OpenAI para geração. Ela suporta interações com estado e integra ferramentas embutidas para estender as capacidades de um modelo. A OpenAI [recomenda](https://platform.openai.com/docs/guides/migrate-to-responses) o uso da Responses API em vez da Chat Completions API para novos projetos.

A API `v1/responses` suporta solicitações de texto, imagem, áudio e vídeo por meio do comando `curl` e do cliente [OpenAI](https://platform.openai.com/docs/quickstart).

### Respostas baseadas em texto

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```shell
Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    input="Tell me a three sentence bedtime story about a unicorn.",
    max_output_tokens=256,
    stream=True,
)

for event in response:
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="")
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```shell
Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "input": "Tell me a three sentence bedtime story about a unicorn.",
    "max_output_tokens": 256,
    "stream": false
  }'
```

O comando retorna uma string JSON.

```json
{
  "id": "resp_8693af9b-b561-4267-bd1a-22d1b43284cb",
  "created_at": 1776258580.7114341,
  "model": "Qwen/Qwen2.5-0.5B-Instruct@main",
  "object": "response",
  "output": [
    {
      "id": "msg_8693af9b-b561-4267-bd1a-22d1b43284cb",
      "content": [
        {
          "annotations": [],
          "text": "Once upon a time, in a faraway land, there lived a beautiful unicorn named Luna [...]",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "status": "completed"
}
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "stream": true,
    "input": "Tell me a three sentence bedtime story about a unicorn.",
    "max_output_tokens": 256
  }'
```

O comando retorna um fluxo de blocos.

```shell
data: {"response":{"id":"resp_req_0","created_at":1754059817.783648,"model":"Qwen/Qwen2.5-0.5B-Instruct@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}

data: {"content_index":0,"delta":"Once ","item_id":"msg_req_0","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}

data: {"content_index":0,"delta":"upon ","item_id":"msg_req_0","output_index":0,"sequence_number":6,"type":"response.output_text.delta"}

(...)
```

### Respostas baseadas em imagens

A API de Respostas também suporta entradas de imagem, áudio e vídeo.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    input=[
        {"type": "input_text", "text": "What's in this image?"},
        {
            "type": "input_image",
            "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
        },
    ],
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    input=[
        {"type": "input_text", "text": "What's in this image?"},
        {
            "type": "input_image",
            "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg",
        },
    ],
    max_output_tokens=256,
    stream=True,
)

for event in response:
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="")
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```xmp
The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon, given the barren, rocky landscape and the dark sky in the background. The astronaut is holding a large egg that has cracked open, revealing a small creature inside. The scene is imaginative and playful, combining elements of space exploration with a whimsical twist involving the egg and the creature.
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "input": [
      {"type": "input_text", "text": "What'\''s in this image?"},
      {"type": "input_image", "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"}
    ],
    "max_output_tokens": 256,
    "stream": false
  }'
```

O comando retorna uma string JSON.

```json
{
  "id": "resp_f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "created_at": 1776258580.7114341,
  "model": "Qwen/Qwen2.5-VL-7B-Instruct@main",
  "object": "response",
  "output": [
    {
      "id": "msg_f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "content": [
        {
          "annotations": [],
          "text": "The image depicts an astronaut in a space suit standing on what appears to be the surface of the moon [...]",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "status": "completed"
}
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-VL-7B-Instruct",
    "stream": true,
    "input": [
      {"type": "input_text", "text": "What'\''s in this image?"},
      {"type": "input_image", "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg"}
    ],
    "max_output_tokens": 256
  }'
```

O comando retorna um fluxo de blocos.

```shell
data: {"response":{"id":"resp_f47ac10b-58cc-4372-a567-0e02b2c3d479","created_at":1776258580.7114341,"model":"Qwen/Qwen2.5-VL-7B-Instruct@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}

data: {"content_index":0,"delta":"The ","item_id":"msg_f47ac10b","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}

(...)
```

### Respostas baseadas em áudio

```python
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "Transcribe this audio."},
        {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
    ],
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```python
import base64
from huggingface_hub.utils import httpx
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

audio_url = "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"
audio_b64 = base64.b64encode(httpx.get(audio_url, follow_redirects=True).content).decode()

response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "Transcribe this audio."},
        {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
            ],
        }
    ],
    max_output_tokens=256,
    stream=True,
)

for event in response:
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="")
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```
This week, I traveled to Chicago to deliver my final farewell address to the nation, following in the tradition of presidents before me. It was an opportunity to say thank you. Whether we've seen eye to eye or rarely agreed at all, my conversations with you, the American people, in living rooms and schools, at farms and on factory floors, at diners, and on distant military outposts, all these conversations are what have kept me honest.
```

```shell
# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)

cat <<EOF > /tmp/audio_request.json
{
  "model": "google/gemma-4-E2B-it",
  "input": [
    {"type": "input_text", "text": "Transcribe this audio."},
    {"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
  ],
  "max_output_tokens": 256,
  "stream": false
}
EOF

curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d @/tmp/audio_request.json
```

O comando retorna uma string JSON.

```json
{
  "id": "resp_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "created_at": 1776258580.7114341,
  "model": "google/gemma-4-E2B-it@main",
  "object": "response",
  "output": [
    {
      "id": "msg_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
      "content": [
        {
          "annotations": [],
          "text": "This week, I traveled to Chicago to deliver my final farewell address to the nation [...]",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "status": "completed"
}
```

```shell
# First, base64-encode the audio file and build the JSON payload
AUDIO_B64=$(curl -sL https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3 | base64 -w 0)

cat <<EOF > /tmp/audio_request.json
{
  "model": "google/gemma-4-E2B-it",
  "stream": true,
  "input": [
    {"type": "input_text", "text": "Transcribe this audio."},
    {"type": "input_audio", "input_audio": {"data": "$AUDIO_B64", "format": "mp3"}}
  ],
  "max_output_tokens": 256
}
EOF

curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d @/tmp/audio_request.json
```

O comando retorna um fluxo de blocos.

```shell
data: {"response":{"id":"resp_a1b2c3d4-e5f6-7890-abcd-ef1234567890","created_at":1776258580.7114341,"model":"google/gemma-4-E2B-it@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}

data: {"content_index":0,"delta":"This ","item_id":"msg_a1b2c3d4","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}

(...)
```

> [!WARNING]
> O tipo de conteúdo `audio_url` é uma extensão que não faz parte do padrão da OpenAI e pode ser alterado em versões futuras.

Você também pode passar áudio por URL com o tipo de conteúdo `audio_url` para evitar a codificação base64.

```python
response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "Transcribe this audio."},
        {"type": "audio_url", "audio_url": {"url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama_first_45_secs.mp3"}},
    ],
    max_output_tokens=256,
    stream=False,
)
```

### Respostas baseadas em vídeo

> [!WARNING]
> O tipo de conteúdo `video_url` é uma extensão que não faz parte do padrão da OpenAI e pode ser alterado em versões futuras.

> [!TIP]
> O processamento de vídeo requer [torchcodec](https://github.com/pytorch/torchcodec). Instale-o com `pip install torchcodec`.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "What is happening in the video and what is the song about?"},
        {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
    ],
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="google/gemma-4-E2B-it",
    input=[
        {"type": "input_text", "text": "What is happening in the video and what is the song about?"},
        {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}},
    ],
    max_output_tokens=256,
    stream=True,
)

for event in response:
    if hasattr(event, "delta") and event.delta:
        print(event.delta, end="")
```

O cliente [OpenAI](https://platform.openai.com/docs/quickstart) retorna uma string impressa.

```
The video captures a live music performance at a music festival or a large concert. There are several musicians on stage, including a central figure playing an acoustic guitar and singing. The foreground is filled with the backs of the audience, indicating a large crowd watching the show. The stage is dramatically lit with bright spotlights and blue and white stage lighting, with haze and smoke creating an immersive atmosphere.

The lyrics of the song are: "I don't care 'bout street, from that fresh street, 'cause there's no problem, another one I want to be, in the storm..."
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "input": [
      {"type": "input_text", "text": "What is happening in the video and what is the song about?"},
      {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}}
    ],
    "max_output_tokens": 256,
    "stream": false
  }'
```

O comando retorna uma string JSON.

```json
{
  "id": "resp_b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "created_at": 1776258580.7114341,
  "model": "google/gemma-4-E2B-it@main",
  "object": "response",
  "output": [
    {
      "id": "msg_b2c3d4e5-f6a7-8901-bcde-f12345678901",
      "content": [
        {
          "annotations": [],
          "text": "The video captures a live music performance at a music festival or a large concert [...]",
          "type": "output_text"
        }
      ],
      "role": "assistant",
      "status": "completed",
      "type": "message"
    }
  ],
  "status": "completed"
}
```

```shell
curl http://localhost:8000/v1/responses \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "stream": true,
    "input": [
      {"type": "input_text", "text": "What is happening in the video and what is the song about?"},
      {"type": "video_url", "video_url": {"url": "https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/concert.mp4"}}
    ],
    "max_output_tokens": 256
  }'
```

O comando retorna um fluxo de blocos.

```shell
data: {"response":{"id":"resp_b2c3d4e5-f6a7-8901-bcde-f12345678901","created_at":1776258580.7114341,"model":"google/gemma-4-E2B-it@main","object":"response","output":[],"status":"queued"},"sequence_number":0,"type":"response.created"}

data: {"content_index":0,"delta":"Based ","item_id":"msg_b2c3d4e5","output_index":0,"sequence_number":5,"type":"response.output_text.delta"}

(...)
```

### Conversas de múltiplas rodadas[[responses]]

Para conversas de múltiplas rodadas, passe uma lista de mensagens com as chaves `role` no campo `input`. Como todos os servidores compatíveis com a OpenAI, a API é sem estado, portanto, cada solicitação deve conter o histórico completo da conversa.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<random_string>")

response = client.responses.create(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    input=[
        {"role": "user", "content": "What is the capital of France?"},
        {"role": "assistant", "content": "The capital of France is Paris."},
        {"role": "user", "content": "How many people live there?"},
    ],
    max_output_tokens=256,
    stream=False,
)
print(response.output[0].content[0].text)
```

A pergunta subsequente "Quantas pessoas moram lá?" depende do contexto anterior, então o modelo responde sobre Paris.

```
As of 2021, Paris has a population of approximately 2.8 million people.
```

## v1/systemone

O endpoint `v1/systemone` responde a perguntas tipadas sobre um `state` compartilhado, seguindo a [System One API da TypeSafe](https://api.typesafe.ai/openapi.json). Use-o para decisões como rotear um ticket de suporte, classificar a urgência ou escolher uma ação.

O modelo pontua as respostas permitidas em uma única passagem de inferência em lote, sem gerar texto. Este endpoint suporta modelos autoregressivos com um modelo de chat;

Envie um modelo, o estado a ser avaliado e as perguntas indexadas por nome:

```shell
curl http://localhost:8000/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-1.5B-Instruct",
    "state": "Our integration has been failing for 3 days and we are losing sales!",
    "questions": {
      "team": {
        "type": "choice",
        "instructions": "Which team handles this?",
        "criteria": {"billing": null, "technical": "Bugs or integration problems"}
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is this ticket?",
        "criteria": ["low", "medium", "high"]
      },
      "upset": {"type": "noul", "instructions": "Is the customer upset?"}
    }
  }'
```

Instale o [TypeSafe SDK](https://github.com/typesafe-ai/typesafe-sdk-js) para Node.js 20 ou mais recente:

```shell
npm install @typesafe-ai/sdk
```

Defina `baseURL` para a raiz do servidor; o SDK adiciona `/v1/systemone`. O SDK requer uma chave de API, mas um valor de preenchimento funciona com o servidor local.

```typescript
import { choice, noul, score, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient({
  baseURL: "http://localhost:8000",
  apiKey: "<random_string>",
});

const response = await client.systemOne({
  model: "Qwen/Qwen2.5-1.5B-Instruct",
  state: "Our integration has been failing for 3 days and we are losing sales!",
  questions: {
    team: choice("Which team handles this?", {
      billing: null,
      technical: "Bugs or integration problems",
    }),
    urgency: score("How urgent is this ticket?", ["low", "medium", "high"]),
    upset: noul("Is the customer upset?"),
  },
});

console.log(response.answers.team.choice);
console.log(response.answers.urgency.score);
console.log(response.answers.upset.noul);
```

O cliente exibe a equipe selecionada, a pontuação de urgência e a probabilidade de o cliente estar irritado. Por exemplo:

```shell
technical
1.6739138662815094
0.851952850818634
```

As respostas usam os mesmos nomes das perguntas. Por exemplo:

```json
{
  "model": "Qwen/Qwen2.5-1.5B-Instruct",
  "answers": {
    "team": {"type": "choice", "choice": "technical", "probabilities": {"billing": 0.000006962256520637311, "technical": 0.999993085861206}, "confidence": 0.9999861717224121},
    "urgency": {"type": "score", "score": 1.6739138662815094, "legend": {"0": "low", "1": "medium", "2": "high"}, "probabilities": {"0": 0.0031353600788861513, "1": 0.31981542706489563, "2": 0.6770492196083069}, "confidence": 0.5108707791659981},
    "upset": {"type": "noul", "noul": 0.851952850818634}
  },
  "usage": {"input_tokens": 213, "output_tokens": 0}
}
```

| Tipo de pergunta | `criteria` | Resposta |
| :--- | :--- | :--- |
| `choice` | Nomes de opções mapeados para descrições, que podem ser `null`. | A opção mais provável, as probabilidades das opções e a confiança. |
| `score` | Descrições dos níveis ordenadas do menor para o maior. | O índice de nível ponderado pela probabilidade, as probabilidades, uma legenda e a confiança. |
| `noul` | Descrições opcionais para `true` e `false`. | A probabilidade de `true`. |

`state`, `instructions` opcionais e descrições podem conter texto ou objetos e arrays JSON. Cada solicitação aceita até 64 perguntas, com 1–26 opções ou níveis de pontuação por pergunta.

Para modelos com capacidade de visão, adicione um array `images` com até quatro URLs `data:image/...;base64,...`. Cada pergunta recebe as mesmas imagens. O processador e o modelo de chat do modelo devem suportar imagens; URLs de imagens remotas não são aceitas.

### Configuração de decisão

Os modelos de decisão ainda não compartilham um padrão para armazenar prompts, rótulos de resposta e configurações de calibração. Esses elementos variam entre as implementações dos modelos, portanto `--decision-config` permite fornecê-los explicitamente. Modelos com cabeças de decisão personalizadas ainda exigem uma implementação compatível; esse arquivo apenas configura o formato do prompt e a pontuação do próximo token.

Por padrão, as perguntas usam um prompt de fallback e o modelo de chat do modelo com o raciocínio desativado. Os rótulos de resposta são `A`–`Z` para escolhas e pontuações, e `Yes`/`No` para booleanos. As probabilidades são o softmax sobre esses rótulos em temperatura 1.

Passe `--decision-config` para substituir esses padrões para um modelo:

```shell
transformers serve openjev/openjev --decision-config decision.json
```

O arquivo JSON suporta três campos opcionais:

| Campo | Significado |
| :--- | :--- |
| `labels` | Rótulos de resposta por tipo de pergunta, na ordem das opções (`true`, depois `false` para `noul`). Cada um deve ser um token único e conhecido. |
| `temperature` | Temperaturas positivas por tipo de pergunta, aplicadas como `softmax(label_logits / temperature)`. |
| `chat_template` | Um modelo Jinja inline que substitui o prompt completo, incluindo o formato de chat e o prefixo do assistente. |

Por exemplo, para alterar os rótulos e a calibração mantendo o prompt padrão:

```json
{
  "labels": {"noul": ["A", "B"]},
  "temperature": {"choice": 0.85, "score": 0.85, "noul": 1.5547129}
}
```

Um template personalizado recebe os `messages` habituais, além dos campos brutos `state`, `images` e `question`. A pergunta expõe `name`, `type`, `instructions`, `options`, `labels`, `descriptions` e `legend`; opções, rótulos e descrições compartilham a mesma ordem. Use esses campos para substituir a redação de fallback e `tojson` para renderizar valores estruturados. Templates para entradas de imagem devem incluir os placeholders de imagem do modelo.

O arquivo é carregado na inicialização e se aplica apenas a `/v1/systemone`. Configurações omitidas mantêm seus valores padrão; nenhuma configuração de decisão é lida automaticamente do checkpoint.

## v1/audio/transcriptions

O endpoint `v1/audio/transcriptions` transcreve áudio usando modelos de speech-to-text. Ele segue o formato da [API de transcrição de áudio](https://platform.openai.com/docs/api-reference/audio/createTranscription).

```shell
curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F "file=@/path/to/audio.wav" \
  -F "model=openai/whisper-large-v3"
```

O comando retorna a seguinte resposta.

```shell
{
  "text": "Transcribed text from the audio file",
}
```

## v1/models

O endpoint `v1/models` varre o cache local do Hugging Face e retorna uma lista dos modelos baixados no formato compatível com o OpenAI. Ferramentas de terceiros usam esse endpoint para descobrir os modelos disponíveis.

Baixe um modelo antes de executar `transformers serve`.

```bash
transformers download Qwen/Qwen2.5-0.5B-Instruct
```

Uma vez baixado, o modelo aparece nas respostas de `/v1/models`.

```shell
curl http://localhost:8000/v1/models
```

O endpoint retorna um objeto JSON com os modelos disponíveis.

## Carregando modelos

O endpoint `/load_model` pré-carrega um modelo e transmite o progresso via [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events) (SSE). O CLI `transformers chat` o utiliza automaticamente para que os usuários vejam o progresso do download e do carregamento, em vez de um prompt que fica travado. Use-o para aquecer um modelo antes de enviar solicitações de inferência.

### Solicitação

```shell
curl -N -X POST http://localhost:8000/load_model \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct"}'
```

O campo `model` é um identificador de modelo do Hugging Face, opcionalmente com um sufixo `@revision` (`Qwen/Qwen2.5-0.5B-Instruct@main`). Se a revisão for omitida, o padrão é `main`.

### Resposta

A resposta é um stream SSE (`Content-Type: text/event-stream`). Cada frame é um objeto JSON em uma linha `data:`.

```
data: {"status": "loading", "model": "Qwen/Qwen2.5-0.5B-Instruct@main", "stage": "processor"}
```

Todo evento contém, no mínimo, os campos `status` e `model`. Os campos adicionais dependem do status.

| Campo | Presente quando | Descrição |
|-------|-------------|-------------|
| `status` | Sempre | `loading`, `ready` ou `error` |
| `model` | Sempre | Canonical `model_id@revision` |
| `stage` | `status == "loading"` | Um de `processor`, `config`, `download`, `weights` (veja as etapas abaixo) |
| `progress` | etapas `download` e `weights` | Objeto com `current` e `total` (inteiro ou null) |
| `cached` | `status == "ready"` | `true` se o modelo já estava na memória |
| `message` | `status == "error"` | Descrição do erro |

### Etapas

O carregamento passa por essas etapas em ordem. Algumas podem ser ignoradas (`download` é ignorado quando os arquivos já estão em cache local).

| Etapa | Há progresso? | Descrição |
|-------|---------------|-------------|
| `processor` | Não | Carregando o tokenizer/processor |
| `config` | Não | Carregando a configuração do modelo |
| `download` | Sim (bytes) | Baixando arquivos de modelo |
| `weights` | Sim (itens) | Carregando tensores de pesos na memória |

O stream termina com exatamente um evento terminal, `ready` (sucesso) ou `error` (falha).

## Timeout

`transformers serve` processa solicitações para qualquer modelo. Cada modelo é carregado sob demanda e permanece na memória da GPU. Os modelos são descarregados automaticamente após 300 segundos de inatividade para liberar a memória da GPU. Defina `--model-timeout` para um valor diferente em segundos, ou `-1` para desativar o descarregamento.

```shell
transformers serve --model-timeout 400
```

### Carregando exemplos

Os exemplos abaixo mostram as respostas para um modelo recém-baixado, um modelo carregado do seu cache local (que ignora a etapa de download) e um modelo já em memória.

```
data: {"status": "loading", "model": "org/model@main", "stage": "processor"}
data: {"status": "loading", "model": "org/model@main", "stage": "config"}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 0, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 134600000, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "download", "progress": {"current": 269100000, "total": 269100000}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 1, "total": 272}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 272, "total": 272}}
data: {"status": "ready", "model": "org/model@main", "cached": false}
```

```
data: {"status": "loading", "model": "org/model@main", "stage": "processor"}
data: {"status": "loading", "model": "org/model@main", "stage": "config"}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 1, "total": 272}}
data: {"status": "loading", "model": "org/model@main", "stage": "weights", "progress": {"current": 272, "total": 272}}
data: {"status": "ready", "model": "org/model@main", "cached": false}
```

```
data: {"status": "ready", "model": "org/model@main", "cached": true}
```

## Chamada de ferramentas

O servidor `transformers serve` suporta chamadas de função no estilo OpenAI. Modelos treinados para uso de ferramentas geram chamadas de função estruturadas que o seu aplicativo executa.

> [!NOTE]
> O tool calling funciona com qualquer modelo cujo tokenizer declare tokens de tool call. Qwen e Gemma 4 funcionam sem configuração adicional. Abra uma [issue](https://github.com/huggingface/transformers/issues/new/choose) para solicitar suporte a um modelo específico.

Defina as ferramentas como uma lista de especificações de funções seguindo o formato da OpenAI.

```py
import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<KEY>")

tools = [
  {
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get the current weather in a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city name, e.g. San Francisco"
          },
          "unit": {
            "type": "string",
            "enum": ["celsius", "fahrenheit"],
            "description": "temperature unit"
          }
        },
        "required": ["location"]
      }
    }
  }
]
```

Personalize a geração passando os parâmetros de [GenerationConfig](/docs/transformers/main/pt/main_classes/text_generation#transformers.GenerationConfig) para o argumento `extra_body` em [create](https://platform.openai.com/docs/api-reference/responses/create).

```py
generation_config = {
  "max_new_tokens": 512,
  "temperature": 0.7,
  "top_p": 0.9,
  "top_k": 50,
  "do_sample": True,
  "repetition_penalty": 1.1,
  "no_repeat_ngram_size": 3,
}

response = client.responses.create(
  model="Qwen/Qwen2.5-7B-Instruct",
  instructions="You are a helpful weather assistant. Use the get_weather tool to answer questions.",
  input="What's the weather like in San Francisco?",
  tools=tools,
  stream=True,
  extra_body={"generation_config": json.dumps(generation_config)}
)

for event in response:
  print(event)
```

### Chamadas de ferramentas em múltiplas rodadas

Após o modelo retornar uma chamada de ferramenta, execute a função localmente e, em seguida, envie o resultado de volta em uma solicitação de acompanhamento para obter a resposta final do modelo. O padrão difere ligeiramente entre as duas APIs. Consulte o [guia de chamada de funções da OpenAI](https://developers.openai.com/api/docs/guides/function-calling?api-mode=chat) para a especificação completa.

Os exemplos abaixo reutilizam a lista `tools` definida acima.

Passe o resultado da ferramenta como uma mensagem `role: "tool"` com o `tool_call_id` correspondente.

```py
# Model returns a tool call
messages = [{"role": "user", "content": "What's the weather like in San Francisco?"}]
response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=messages,
    tools=tools,
)
assistant_message = response.choices[0].message

# Execute the tool locally
tool_call = assistant_message.tool_calls[0]
result = {"temperature": 22, "condition": "sunny"}  # your actual function call here

# Send the tool result back
messages.append(assistant_message)
messages.append({
    "role": "tool",
    "tool_call_id": tool_call.id,
    "content": json.dumps(result),
})
final_response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=messages,
    tools=tools,
)
print(final_response.choices[0].message.content)
```

Passe o resultado da ferramenta como um item `function_call_output` na lista `input` da solicitação de acompanhamento.

```py
user_message = {"role": "user", "content": "What's the weather like in San Francisco?"}
response = client.responses.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    input=[user_message],
    tools=tools,
    stream=False,
)
tool_call = next(item for item in response.output if item.type == "function_call")

result = {"temperature": 22, "condition": "sunny"}

final_response = client.responses.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    input=[
        user_message,
        tool_call.model_dump(exclude_none=True),
        {"type": "function_call_output", "call_id": tool_call.call_id, "output": json.dumps(result)},
    ],
    tools=tools,
    stream=False,
)
print(final_response.output_text)
```

## Raciocínio

Modelos de raciocínio emitem uma cadeia de raciocínio oculta antes da resposta final. O servidor detecta esses trechos de raciocínio, remove os delimitadores da resposta visível e expõe o texto de raciocínio em um campo dedicado, permitindo que os clientes o renderizem separadamente.

- Chat Completions retorna o raciocínio como `reasoning_content` na mensagem do assistente.
- A API de Respostas retorna o raciocínio como um item de saída `reasoning` que precede o item `message`.

A detecção de raciocínio depende de um [modelo de resposta](../chat_response_parsing), que informa ao servidor onde uma seção de raciocínio começa e termina. O servidor procura esse modelo em dois locais, na seguinte ordem.

1. O atributo `response_template` do tokenizer, se o modelo o incluir.
2. Um fallback embutido correspondente à arquitetura do modelo, para tokenizers que ainda não incluem um template.

Modelos que não possuem nenhum dos dois são servidos sem análise de resposta. O texto decodificado bruto é retornado como `content` e nenhum campo de raciocínio é exibido, mesmo que o modelo emita as tags `<think>...</think>`.

### Ativar o raciocínio no servidor

Use `--reasoning` para controlar se o modelo de chat emite tokens de raciocínio.

| Valor | Comportamento |
|---|---|
| `auto` (padrão) | Deixe para o padrão do modelo de chat. |
| `on` | Force o modo de raciocínio ativado definindo `enable_thinking=True` nos kwargs do modelo de chat. |
| `off` | Force o desligamento do raciocínio definindo `enable_thinking=False`. |

```sh
transformers serve Qwen/Qwen3-1.7B --reasoning on
```

> [!WARNING]
> Nem todos os modelos suportam raciocínio. `--reasoning` só funciona quando o modelo de chat do modelo trata a variável `enable_thinking` (ou quando seu tokenizer declara delimitadores de raciocínio). Para outros modelos, a flag é ignorada silenciosamente.

`--reasoning on` e `--reasoning off` funcionam definindo a variável de chat template `enable_thinking`. Alguns chat templates usam um nome de variável diferente para alternar o modo de raciocínio ou aceitam variáveis extras que alteram a forma como o prompt é renderizado. Passe qualquer variável de chat template diretamente com `--chat-template-kwargs` como um objeto JSON.

```sh
transformers serve Qwen/Qwen3-1.7B \
  --chat-template-kwargs '{"enable_thinking": true}'
```

Os clientes também podem enviar `chat_template_kwargs` no corpo da solicitação para substituir os padrões do servidor em uma única solicitação, sem reiniciar o servidor.

### Chat Completions

As respostas não em streaming incluem `reasoning_content` junto com `content` na mensagem do assistente.

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="<KEY>")

completion = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    messages=[{"role": "user", "content": "What is 17 * 23?"}],
)
message = completion.choices[0].message
print("reasoning:", message.reasoning_content)
print("answer:", message.content)
```

O streaming emite deltas de `reasoning_content` antes dos deltas de `content`. Concatene cada campo separadamente.

```python
chunks = list(
    client.chat.completions.create(
        model="Qwen/Qwen3-1.7B",
        messages=[{"role": "user", "content": "What is 17 * 23?"}],
        stream=True,
    )
)
reasoning = "".join(getattr(c.choices[0].delta, "reasoning_content", None) or "" for c in chunks)
content = "".join(c.choices[0].delta.content or "" for c in chunks)
```

```shell
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-1.7B",
    "messages": [{"role": "user", "content": "What is 17 * 23?"}],
    "chat_template_kwargs": {"enable_thinking": true}
  }'
```

A resposta inclui `reasoning_content` na mensagem do assistente.

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "17 * 23 = 391.",
        "reasoning_content": "Let me compute 17 * 23. 17 * 20 = 340, 17 * 3 = 51, so 340 + 51 = 391."
      },
      "finish_reason": "stop",
      "index": 0
    }
  ]
}
```

### API de Respostas

A API de Responses retorna o raciocínio como um item de saída `reasoning` separado, que precede o item `message`.

```python
response = client.responses.create(
    model="Qwen/Qwen3-1.7B",
    input="What is 17 * 23?",
    stream=False,
)
reasoning_item = next(i for i in response.output if i.type == "reasoning")
message_item = next(i for i in response.output if i.type == "message")
print("reasoning:", reasoning_item.content[0].text)
print("answer:", message_item.content[0].text)
```

O streaming abre primeiro o item de raciocínio e emite eventos `response.reasoning_text.delta`, em seguida o fecha e abre o item de mensagem para emitir eventos `response.output_text.delta`.

```python
stream = client.responses.create(
    model="Qwen/Qwen3-1.7B",
    input="What is 17 * 23?",
    stream=True,
)

for event in stream:
    if event.type == "response.reasoning_text.delta":
        print(event.delta, end="")
    elif event.type == "response.output_text.delta":
        print(event.delta, end="")
```

### Viagem de ida e volta em várias rodadas

Para conversas de múltiplas rodadas, inclua o texto de raciocínio da resposta anterior na próxima solicitação. O servidor o passa para o modelo de chat, que pode renderizá-lo como parte da rodada anterior do assistente, para que o modelo veja seu próprio processo de pensamento anterior.

Anexe `reasoning_content` à mensagem anterior do assistente.

```python
first = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    messages=[{"role": "user", "content": "What is 17 * 23?"}],
).choices[0].message

second = client.chat.completions.create(
    model="Qwen/Qwen3-1.7B",
    messages=[
        {"role": "user", "content": "What is 17 * 23?"},
        {
            "role": "assistant",
            "content": first.content,
            "reasoning_content": first.reasoning_content,
        },
        {"role": "user", "content": "Now multiply that result by 2."},
    ],
)
print(second.choices[0].message.content)
```

Passe o item `reasoning` anterior novamente como um item de entrada antes da mensagem do assistente.

```python
first = client.responses.create(
    model="Qwen/Qwen3-1.7B",
    input="What is 17 * 23?",
    stream=False,
)
reasoning_item = next(i for i in first.output if i.type == "reasoning")
message_item = next(i for i in first.output if i.type == "message")

second = client.responses.create(
    model="Qwen/Qwen3-1.7B",
    input=[
        {"role": "user", "content": "What is 17 * 23?"},
        reasoning_item.model_dump(exclude_none=True),
        {"role": "assistant", "content": message_item.content[0].text},
        {"role": "user", "content": "Now multiply that result by 2."},
    ],
    stream=False,
)
print(second.output_text)
```

## Encaminhamento de portas

O encaminhamento de portas permite servir modelos a partir de um servidor remoto. Certifique-se de que você tem acesso SSH ao servidor e, em seguida, execute este comando em sua máquina local.

```bash
ssh -N -f -L 8000:localhost:8000 your_server_account@your_server_IP -p port_to_ssh_into_your_server
```

## Reprodutibilidade

Passe um modelo como argumento posicional para evitar dicas de modelo por solicitação e produzir execuções estáveis e repetíveis.

```sh
transformers serve Qwen/Qwen2.5-0.5B-Instruct \
  --continuous-batching \
  --dtype "bfloat16"
```