Deploying on AWS documentation

Build a reasoning trip-planning agent on Amazon SageMaker AI with Hugging Face vLLM

Deploying on AWS

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Build a reasoning trip-planning agent on Amazon SageMaker AI with Hugging Face vLLM

Written by Dario Salvati
Last updated 2026-07-29

In this notebook, we’ll deploy Qwen/Qwen3-8B with the Hugging Face vLLM Deep Learning Container (DLC) on Amazon SageMaker, and use it to build a trip-planning agent: an assistant that reasons through multi-constraint travel requests and calls tools to check the weather, convert currencies, and find places to visit.

We’ll walk through the following steps:

Select a reasoning and tool-calling model and a SageMaker DLC for your use case
Deploy the model to SageMaker with the SageMaker Python SDK ModelBuilder
Configure vLLM for reasoning and tool calling
Build an agent with Strands Agents and give it custom tools
Hold a multi-turn conversation where the agent reasons, calls tools, and remembers context
Optionally connect a Hugging Face MCP server and launch a Gradio app
Clean up the endpoint resources to avoid ongoing charges

For this example, you’ll need AWS credentials and a SageMaker execution role. Qwen3-8B is a public model, so a Hugging Face token is optional here. Authenticated requests get more generous rate limits and faster downloads, and a token is required if you switch to a gated model.

How an agent works

An agent is a loop around a language model. The model does not just answer in one shot; it decides, step by step, what information it still needs and how to get it.

Two capabilities make this possible:

Reasoning: the model thinks through the problem before answering, which helps with multi-step requests such as planning a trip under several constraints.
Tool calling: instead of guessing a fact, the model can call a function you provide (check the weather, convert a currency, look up places) and continue with the real result.

Put together, the loop looks like this: the user asks a question, the model reasons about it and requests a tool, your code runs the tool and returns the result, and the model reasons again with that new information. This repeats until the model has everything it needs to give a final answer.

Qwen/Qwen3-8B supports both a thinking mode and native tool calling, so it can drive this loop on its own. We’ll deploy it, then let Strands Agents run the loop for us.

Hugging Face vLLM and Strands Agents

vLLM is a high-throughput inference engine for large language models. The Hugging Face vLLM DLC packages it for SageMaker with current transformers and huggingface_hub, and exposes an OpenAI-compatible API (including tool calling and reasoning parsing) through simple environment variables. Its multimodal sibling, vLLM-Omni, serves tasks like text-to-speech or image generation; here we serve a text model, so we use the plain vLLM image.

Strands Agents is a lightweight agent framework with a native SageMaker integration. Instead of hand-writing the tool-calling loop, we point Strands at our endpoint and it orchestrates the reasoning, tool calls, and results for us.

Setup

To run this example, we’ll install the SageMaker Python SDK for model deployment, huggingface_hub for authentication, strands-agents (with its SageMaker integration) for the agent, and gradio for the optional interactive app.

%pip install -q "sagemaker>=3" huggingface_hub "strands-agents[sagemaker]>=1.48.0" "gradio>=5"

We are going to need:

An HF_TOKEN: used to download the model from Hugging Face. Optional for the public Qwen3-8B; authenticated requests get more generous rate limits and faster downloads (and a token is required for gated models).
A SageMaker execution role: used to pull the DLC from ECR and deploy the model to SageMaker.

Let’s start by setting up the token and the execution role.

from huggingface_hub import get_token

HF_TOKEN = get_token()

if HF_TOKEN:
    print("HF_TOKEN loaded")
else:
    print(
        "No HF_TOKEN found. Public Qwen3-8B downloads still work. Run "
        "huggingface_hub.notebook_login() to authenticate for higher rate "
        "limits or to access gated models."
    )

Production tip: avoid pasting long-lived tokens into notebooks or committing them. For production, keep the token in a vault such as AWS Secrets Manager and load it at deploy time:

import json
import boto3

secret = boto3.client("secretsmanager").get_secret_value(SecretId="hf-token")
HF_TOKEN = json.loads(secret["SecretString"])["HF_TOKEN"]

To avoid handing the token to the endpoint at all, pre-stage the model weights in S3 and point the container at them, so no Hub token is needed at runtime.

import os

import boto3
from sagemaker.core.helper.session_helper import Session, get_execution_role

REGION = boto3.Session().region_name or os.environ.get("AWS_REGION", "us-east-1")
boto_sess = boto3.Session(region_name=REGION)
sess = Session(boto_session=boto_sess)

try:
    role = get_execution_role(sagemaker_session=sess)
    print(f"Role extracted from execution role: {role}")
except Exception:
    role_name = "sagemaker_execution_role"
    iam_client = boto_sess.client("iam")
    role = iam_client.get_role(RoleName=role_name)["Role"]["Arn"]
    print(f"Role extracted from iam client: {role}")

Choosing a model and a DLC

There are many open LLMs on the Hugging Face Hub. For an agent we want strong instruction-following, reasoning, and tool-calling, while staying small enough to run on a single GPU.

We’ll use Qwen/Qwen3-8B and serve it with the Hugging Face vLLM DLC. Rather than hardcoding the container URI, we resolve it with the SageMaker SDK’s image_uris.retrieve helper, which returns the right image for the chosen region and instance. You can also browse the available images on the AWS Available Images page.

We’ll target an ml.g5.xlarge instance (a single NVIDIA A10G 24GB GPU). For higher concurrency or larger models, consider larger GPU instance types such as ml.g5.2xlarge or ml.g6e.*.

from time import strftime

from sagemaker.core.image_uris import retrieve

MODEL_ID = "Qwen/Qwen3-8B"
INSTANCE_TYPE = "ml.g5.xlarge"
IMAGE_URI = retrieve(
    "huggingface-vllm",
    region=REGION,
    image_scope="inference",
    instance_type=INSTANCE_TYPE,
)

RESOURCE_SUFFIX = strftime("%Y%m%d-%H%M%S")
MODEL_NAME = f"trip-planner-agent-model-{RESOURCE_SUFFIX}"
ENDPOINT_NAME = f"trip-planner-agent-endpoint-{RESOURCE_SUFFIX}"

print(IMAGE_URI)

Configuring vLLM for reasoning and tool calling

The vLLM container is configured entirely through environment variables. Any vLLM server flag can be passed by uppercasing it, replacing dashes with underscores, and prefixing it with SM_VLLM_ (for example --max-model-len becomes SM_VLLM_MAX_MODEL_LEN).

Three of these variables are what turn a plain text model into an agent backend:

SM_VLLM_ENABLE_AUTO_TOOL_CHOICE=true lets the model decide when to call a tool.
SM_VLLM_TOOL_CALL_PARSER=hermes parses Qwen3’s tool calls into the OpenAI format.
SM_VLLM_REASONING_PARSER=qwen3 separates the model’s thinking from its final answer.

We also set SM_VLLM_HOST=0.0.0.0, which is required so the container passes the SageMaker health check. ModelBuilder sets HF_MODEL_ID for us, so we don’t repeat the model id here.

env_vars = {
    "SM_VLLM_HOST": "0.0.0.0",  # Bind to all interfaces so the health check passes
    "SM_VLLM_MAX_MODEL_LEN": "16384",  # Context length; bounds KV cache on a single A10G
    "SM_VLLM_GPU_MEMORY_UTILIZATION": "0.9",
    "SM_VLLM_ENABLE_AUTO_TOOL_CHOICE": "true",  # Let the model call tools
    "SM_VLLM_TOOL_CALL_PARSER": "hermes",  # Parse Qwen3 tool calls
    "SM_VLLM_REASONING_PARSER": "qwen3",  # Separate reasoning from the answer
}

# Only pass the token when present; SageMaker rejects non-string env values.
if HF_TOKEN:
    env_vars["HF_TOKEN"] = HF_TOKEN

Deploy the model

The SageMaker Python SDK v3 deploys models with ModelBuilder: we describe the model, the serving container, and the instance, then call build and deploy. There is no need to assemble model, endpoint-config, and endpoint resources by hand.

One detail worth calling out: this image is built on CUDA 13 (cu130 in the tag). vLLM images with CUDA 13 or newer require the al2-ami-sagemaker-inference-gpu-3-1 inference AMI, which we pass to deploy. Without it the container fails to start before any logs are produced.

from sagemaker.serve import ModelBuilder, ModelServer

model_builder = ModelBuilder(
    model=MODEL_ID,
    role_arn=role,
    sagemaker_session=sess,
    instance_type=INSTANCE_TYPE,
    image_uri=IMAGE_URI,
    model_server=ModelServer.VLLM,
    env_vars=env_vars,
)

built_model = model_builder.build(model_name=MODEL_NAME)

model_builder.deploy(
    endpoint_name=ENDPOINT_NAME,
    initial_instance_count=1,
    instance_type=INSTANCE_TYPE,
    inference_ami_version="al2-ami-sagemaker-inference-gpu-3-1",
    container_timeout_in_seconds=900,
    wait=True,
)

A first look: reasoning and tool calls

Before bringing in the agent framework, let’s send a single request straight to the endpoint to see what the model returns. SageMaker requests go through the /invocations route; we use CustomAttributes to forward them to vLLM’s OpenAI-compatible Chat Completions API.

We pass a get_weather tool definition and let the model decide whether to call it (tool_choice="auto"). The response separates the model’s reasoning (reasoning_content) from any tool calls it wants to make (tool_calls).

import json

runtime = boto_sess.client("sagemaker-runtime")

weather_tool = {
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the weather forecast for a city on a given date.",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name."},
                "date": {"type": "string", "description": "Date in YYYY-MM-DD format."},
            },
            "required": ["city", "date"],
        },
    },
}

response = runtime.invoke_endpoint(
    EndpointName=ENDPOINT_NAME,
    ContentType="application/json",
    Body=json.dumps({
        "model": MODEL_ID,
        "messages": [
            {"role": "user", "content": "What should I pack for Lisbon on 2026-10-12?"}
        ],
        "tools": [weather_tool],
        "tool_choice": "auto",
    }),
    CustomAttributes="route=/v1/chat/completions",
)

message = json.loads(response["Body"].read())["choices"][0]["message"]
print("Reasoning:\n", message.get("reasoning_content"), "\n")
print("Tool calls:\n", json.dumps(message.get("tool_calls"), indent=2))

Building the agent

We describe the endpoint once with SageMakerAIModel, hand the agent our tools, and from then on a single call runs the full reason-and-act loop until the model produces an answer.

Two payload_config choices matter here:

stream=True: a reasoning model can generate for a while. Streaming returns tokens as they are produced instead of waiting for the whole answer, which keeps the response flowing and avoids the real-time endpoint timing out on longer generations.
chat_template_kwargs={"enable_thinking": True}: keeps Qwen3 in thinking mode.

The provider logs the full request and response (and every streamed chunk) at INFO, so we first raise the log level to WARNING to keep the agent’s output readable.

import logging

# The SageMaker model provider logs the full request, every streamed chunk, and
# the final response at INFO level. Raise the level to keep the output readable.
logging.getLogger("strands").setLevel(logging.WARNING)

from strands.models.sagemaker import SageMakerAIModel

sagemaker_model = SageMakerAIModel(
    endpoint_config={"endpoint_name": ENDPOINT_NAME, "region_name": REGION},
    payload_config={
        "max_tokens": 8192,
        "temperature": 0.7,
        "top_p": 0.95,
        "stream": True,
        "additional_args": {"chat_template_kwargs": {"enable_thinking": True}},
    },
)

Giving the agent tools

Strands turns any Python function into a tool with the @tool decorator: it reads the type hints and docstring to build the schema the model sees. Our concierge gets three tools it can combine to plan a trip.

To keep the notebook self-contained and reproducible, these are deterministic mocks. Swap in real APIs (or a built-in like strands_tools.http_request) when you adapt this to production; the agent code does not change.

from strands import tool


@tool
def get_weather(city: str, date: str) -> str:
    """Get the weather forecast for a city on a given date.

    Args:
        city: City to look up.
        date: Date in YYYY-MM-DD format.
    """
    return f"{city} on {date}: sunny, 22°C during the day and 14°C at night."


@tool
def convert_currency(amount: float, from_currency: str, to_currency: str) -> str:
    """Convert an amount between two currencies.

    Args:
        amount: Amount to convert.
        from_currency: ISO code to convert from, e.g. EUR.
        to_currency: ISO code to convert to, e.g. USD.
    """
    per_eur = {"EUR": 1.0, "USD": 1.08, "GBP": 0.85, "JPY": 170.0}
    in_eur = amount / per_eur[from_currency.upper()]
    converted = in_eur * per_eur[to_currency.upper()]
    return f"{amount:.2f} {from_currency.upper()} = {converted:.2f} {to_currency.upper()}"


@tool
def search_places(city: str, category: str) -> str:
    """Find points of interest in a city by category.

    Args:
        city: City to search in.
        category: Kind of place, e.g. "museum", "restaurant", or "park".
    """
    catalog = {
        "museum": ["National Tile Museum", "Berardo Collection", "MAAT"],
        "restaurant": ["Time Out Market", "Cervejaria Ramiro", "A Cevicheria"],
        "park": ["Eduardo VII Park", "Monsanto Forest Park", "Jardim da Estrela"],
    }
    found = catalog.get(category.lower(), ["City-center walking tour"])
    return f"{category.title()} options in {city}: " + ", ".join(found)

Running the agent

Now we send a request that no single tool can answer on its own. The agent has to reason about what it needs, call several tools, and combine the results into an itinerary.

The run_agent helper below sends a prompt and then prints two things: the tool calls the agent chose to make (so we can see it acting), followed by the final answer. It reads them from agent.messages, the running conversation history, so the printout stays clean while the verbose provider logs remain off.

import json

from strands import Agent

agent = Agent(
    model=sagemaker_model,
    tools=[get_weather, convert_currency, search_places],
    callback_handler=None,
)


def run_agent(prompt: str):
    """Run the agent, print the tool calls it made, then the final answer."""
    start = len(agent.messages)
    result = agent(prompt)

    for message in agent.messages[start:]:
        for block in message.get("content", []):
            if "toolUse" in block:
                call = block["toolUse"]
                print(f"[tool call] {call['name']}({json.dumps(call['input'])})")

    print()
    print(result)
    return result


result = run_agent(
    "Plan a 3-day trip to Lisbon starting 2026-10-12. My budget is 800 EUR, "
    "so tell me what that is in USD. Suggest a few museums and restaurants, "
    "and tell me what to pack based on the weather."
)

Following up in the same conversation

A Strands Agent keeps the conversation history, so it behaves like a real assistant across turns. We can ask a follow-up and it reuses everything it already worked out, calling tools again only where the change requires it.

follow_up = run_agent(
    "Actually, change the destination to Porto but keep the same budget and dates. "
    "Update the packing tips if the weather is different."
)

Optional: connect the Hugging Face MCP server

Model Context Protocol (MCP) servers expose ready-made tools an agent can call. Strands connects to them with MCPClient, so you can, for example, give the concierge access to the Hugging Face Hub without writing any new tools.

This section is optional. Set ENABLE_MCP = True to try it. The Hugging Face MCP server requires authentication, so make sure HF_TOKEN is set first. MCP tools must be used inside the client’s context manager.

ENABLE_MCP = False

if ENABLE_MCP:
    if not HF_TOKEN:
        raise ValueError(
            "The Hugging Face MCP server requires authentication. Set a token with "
            "huggingface_hub.notebook_login() before enabling MCP."
        )

    from mcp.client.streamable_http import streamablehttp_client
    from strands.tools.mcp import MCPClient

    hf_mcp = MCPClient(
        lambda: streamablehttp_client(
            "https://huggingface.co/mcp",
            headers={"Authorization": f"Bearer {HF_TOKEN}"},
        )
    )
    with hf_mcp:
        mcp_agent = Agent(
            model=sagemaker_model,
            tools=[get_weather, convert_currency, search_places, *hf_mcp.list_tools_sync()],
            callback_handler=None,
        )
        print(mcp_agent("Find a small Qwen model with tool-calling support and include the Hub links."))

Optional: an interactive chat app

For a nicer experience than a notebook cell, we can wrap the agent in a small Gradio chat app. It streams the agent’s events as they happen, rendering the reasoning, each tool call, its result, and the final answer as separate, collapsible blocks.

This section is optional. Set ENABLE_GRADIO = True to launch the app.

Security and cost warning: share=True creates a public Gradio URL. Anyone with that URL can chat with the app, which invokes the already-deployed billable SageMaker endpoint using your AWS credentials. Share the URL only with trusted users, and disable the app or delete the endpoint when you are finished.

import gradio as gr

# A dedicated agent for the app, so the chat starts fresh instead of inheriting
# the conversation from the demo cells above.
chat_agent = Agent(
    model=sagemaker_model,
    tools=[get_weather, convert_currency, search_places],
    callback_handler=None,
)


async def chat(message, history):
    # Gradio passes an empty history at the start of a conversation (including
    # after the "Clear" button), which is our cue to reset the agent's memory.
    if not history:
        chat_agent.messages.clear()

    turn = []
    kind = None  # "reasoning" | "tool" | "text" for the current block
    tool_name = "tool"

    async for event in chat_agent.stream_async(message):
        if "event" in event:
            raw = event["event"]

            # A tool block announces its name at the start.
            tool_use = raw.get("contentBlockStart", {}).get("start", {}).get("toolUse")
            if tool_use:
                tool_name = tool_use.get("name", "tool")
                kind = None  # force a fresh block for this tool call

            delta = raw.get("contentBlockDelta", {}).get("delta")
            if not delta:
                continue

            # Classify the chunk: reasoning, tool arguments, or final answer.
            if "reasoningContent" in delta:
                piece, this, meta = delta["reasoningContent"].get("text", ""), "reasoning", {"title": "💭 Thinking"}
            elif "toolUse" in delta:
                piece, this, meta = delta["toolUse"].get("input", ""), "tool", {"title": f"🛠️ Using tool `{tool_name}`"}
            else:
                piece, this, meta = delta.get("text", ""), "text", None

            if this != kind:
                kind = this
                turn.append(gr.ChatMessage(role="assistant", content="", metadata=meta))
            turn[-1].content += piece
            yield turn

        elif "message" in event:
            msg = event["message"]
            if msg.get("role") != "user":
                continue
            for block in msg.get("content", []):
                result = block.get("toolResult")
                if not result:
                    continue
                text = "\n".join(
                    item.get("text", "") for item in result.get("content", []) if "text" in item
                ).strip()
                if text:
                    status = result.get("status", "success").upper()
                    turn.append(gr.ChatMessage(
                        role="assistant",
                        content=text,
                        metadata={"title": f"✅ Tool result [{status}]"},
                    ))
                    kind = None
                    yield turn

    yield turn


ENABLE_GRADIO = False

if ENABLE_GRADIO:
    demo = gr.ChatInterface(
        fn=chat,
        title="Trip-planning concierge",
        description="Ask for help planning a trip. The agent reasons, calls tools, and answers.",
        examples=[
            "Plan a 2-day trip to Lisbon on 2026-10-12 with a 500 EUR budget in USD.",
            "What museums should I visit in Lisbon, and what's the weather on 2026-10-13?",
        ],
    )
    demo.launch(share=True, server_name="0.0.0.0", server_port=7860, show_error=True)

Cleanup

SageMaker endpoints are billed while they are InService, so delete the endpoint and its associated resources when you’re done.

sm = boto_sess.client("sagemaker")
endpoint_config_name = sm.describe_endpoint(EndpointName=ENDPOINT_NAME)["EndpointConfigName"]

# delete_endpoint is asynchronous; wait for it to finish before removing the
# config and model, which stay in use while the endpoint is still deleting.
sm.delete_endpoint(EndpointName=ENDPOINT_NAME)
sm.get_waiter("endpoint_deleted").wait(EndpointName=ENDPOINT_NAME)

sm.delete_endpoint_config(EndpointConfigName=endpoint_config_name)
sm.delete_model(ModelName=built_model.model_name)

Conclusion and references

We deployed Qwen/Qwen3-8B on the Hugging Face vLLM DLC with the SageMaker SDK ModelBuilder, configured vLLM for reasoning and tool calling, and built a trip-planning agent with Strands Agents that reasons, calls tools, and holds a multi-turn conversation. The same pattern works for other tool-calling models on the Hub: swap the model id, adjust the instance, and register your own @tool functions.

References:

📍 Find the complete example on GitHub here!

Update on GitHub

←Deploy an AI Agent on AWS Inferentia2 with SageMaker Deploy Qwen3-TTS for ad voiceovers on Amazon SageMaker AI→