diff --git a/asian_ai/performance/vLLM/SECURITY.md b/asian_ai/performance/vLLM/SECURITY.md
new file mode 100644
index 0000000000000000000000000000000000000000..df2425b86c8023cbca7f83fac066e2f8892e77ff
--- /dev/null
+++ b/asian_ai/performance/vLLM/SECURITY.md
@@ -0,0 +1,59 @@
+# Security Policy
+
+## Reporting security issues
+
+Please report security issues privately using [the vulnerability submission form](https://github.com/vllm-project/vllm/security/advisories/new).
+
+## Issue triage
+
+Reports will then be triaged by the [vulnerability management team](https://docs.vllm.ai/en/latest/contributing/vulnerability_management.html).
+
+## Threat model
+
+Please see the [Security Guide in the vLLM documentation](https://docs.vllm.ai/en/latest/usage/security.html) for more information on vLLM's security assumptions and recommendations.
+
+Please see [PyTorch's Security Policy](https://github.com/pytorch/pytorch/blob/main/SECURITY.md) for more information and recommendations on how to securely interact with models.
+
+## Issue severity
+
+We will determine the risk of each issue, taking into account our experience dealing with past issues, versions affected, common defaults, and use cases. We use the following severity categories:
+
+### CRITICAL Severity
+
+Vulnerabilities that allow remote attackers to execute arbitrary code, take full control of the system, or significantly compromise confidentiality, integrity, or availability without any interaction or privileges needed, examples include remote code execution via network, deserialization issues that allow exploit chains. Generally those issues which are rated as CVSS ≥ 9.0.
+
+### HIGH Severity
+
+Serious security flaws that allow elevated impact—like RCE in specific, limited contexts or significant data loss—but require advanced conditions or some trust, examples include RCE in advanced deployment modes (e.g. multi-node), or high impact issues where some sort of privileged network access is required. These issues typically have CVSS scores between 7.0 and 8.9
+
+### MODERATE Severity
+
+Vulnerabilities that cause denial of service or partial disruption, but do not allow arbitrary code execution or data breach and have limited impact. These issues have a CVSS rating between 4.0 and 6.9
+
+### LOW Severity
+
+Minor issues such as informational disclosures, logging errors, non-exploitable flaws, or weaknesses that require local or high-privilege access and offer negligible impact. Examples include side channel attacks or hash collisions. These issues often have CVSS scores less than 4.0
+
+## Fix disclosure policy
+
+When a security report is accepted, the fix process depends on the severity:
+
+* **CRITICAL and HIGH severity**: Fixes are developed in a private security fork and coordinated with the prenotification group before public disclosure.
+* **MODERATE and LOW severity**: Fixes are developed and submitted as public pull requests. These issues do not require embargo since they do not enable arbitrary code execution or significant data breach, and public visibility accelerates community review and adoption of the fix.
+
+The vulnerability management team reserves the right to adjust the disclosure approach on a case-by-case basis, taking into account factors such as active exploitation, unusual attack surface, or coordination requirements with downstream vendors.
+
+## Prenotification policy
+
+For certain security issues of CRITICAL, HIGH, or MODERATE severity level, we may prenotify certain organizations or vendors that ship vLLM. The purpose of this prenotification is to allow for a coordinated release of fixes for severe issues.
+
+* This prenotification will be in the form of a private email notification. It may also include adding security contacts to the GitHub security advisory, typically a few days before release.
+
+* If you wish to be added to the prenotification group, please send an email copying all the members of the [vulnerability management team](https://docs.vllm.ai/en/latest/contributing/vulnerability_management.html). Each vendor contact will be analyzed on a case-by-case basis.
+
+* Organizations and vendors who either ship or use vLLM, are eligible to join the prenotification group if they meet at least one of the following qualifications
+ * Substantial internal deployment leveraging the upstream vLLM project.
+ * Established internal security teams and comprehensive compliance measures.
+ * Active and consistent contributions to the upstream vLLM project.
+
+* We may withdraw organizations from receiving future prenotifications if they release fixes or any other information about issues before they are public. Group membership may also change based on policy refinements for who may be included.
diff --git a/asian_ai/performance/vLLM/rust-toolchain.toml b/asian_ai/performance/vLLM/rust-toolchain.toml
new file mode 100644
index 0000000000000000000000000000000000000000..50c7b070fe87e55b12a623fd7c29242ff756a52c
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust-toolchain.toml
@@ -0,0 +1,2 @@
+[toolchain]
+channel = "1.95"
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hermes.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hermes.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..1c399aed01b2c3d8ca1782e971f02197fa304250
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hermes.jinja
@@ -0,0 +1,130 @@
+{%- macro json_to_python_type(json_spec) %}
+ {%- set basic_type_map = {
+ "string": "str",
+ "number": "float",
+ "integer": "int",
+ "boolean": "bool"
+} %}
+
+ {%- if basic_type_map[json_spec.type] is defined %}
+ {{- basic_type_map[json_spec.type] }}
+ {%- elif json_spec.type == "array" %}
+ {{- "list[" + json_to_python_type(json_spec|items) + "]" }}
+ {%- elif json_spec.type == "object" %}
+ {%- if json_spec.additionalProperties is defined %}
+ {{- "dict[str, " + json_to_python_type(json_spec.additionalProperties) + ']' }}
+ {%- else %}
+ {{- "dict" }}
+ {%- endif %}
+ {%- elif json_spec.type is iterable %}
+ {{- "Union[" }}
+ {%- for t in json_spec.type %}
+ {{- json_to_python_type({"type": t}) }}
+ {%- if not loop.last %}
+ {{- "," }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "]" }}
+ {%- else %}
+ {{- "Any" }}
+ {%- endif %}
+{%- endmacro %}
+
+
+{{- bos_token }}
+{{- "<|im_start|>system\nYou are a function calling AI model. You are provided with function signatures within XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: " }}
+{%- if tools is iterable and tools | length > 0 %}
+ {%- for tool in tools %}
+ {%- if tool.function is defined %}
+ {%- set tool = tool.function %}
+ {%- endif %}
+ {{- '{"type": "function", "function": ' }}
+ {{- '{"name": "' + tool.name + '", ' }}
+ {{- '"description": "' + tool.name + '(' }}
+ {%- for param_name, param_fields in tool.parameters.properties|items %}
+ {{- param_name + ": " + json_to_python_type(param_fields) }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- endif %}
+ {%- endfor %}
+ {{- ")" }}
+ {%- if tool.return is defined %}
+ {{- " -> " + json_to_python_type(tool.return) }}
+ {%- endif %}
+ {{- " - " + tool.description + "\n\n" }}
+ {%- for param_name, param_fields in tool.parameters.properties|items %}
+ {%- if loop.first %}
+ {{- " Args:\n" }}
+ {%- endif %}
+ {{- " " + param_name + "(" + json_to_python_type(param_fields) + "): " + param_fields.description|trim }}
+ {%- endfor %}
+ {%- if tool.return is defined and tool.return.description is defined %}
+ {{- "\n Returns:\n " + tool.return.description }}
+ {%- endif %}
+ {{- '"' }}
+ {{- ', "parameters": ' }}
+ {%- if tool.parameters.properties | length == 0 %}
+ {{- "{}" }}
+ {%- else %}
+ {{- tool.parameters|tojson }}
+ {%- endif %}
+ {{- "}" }}
+ {%- if not loop.last %}
+ {{- "\n" }}
+ {%- endif %}
+ {%- endfor %}
+{%- endif %}
+{{- " " }}
+{{- 'Use the following pydantic model json schema for each tool call you will make: {"properties": {"name": {"title": "Name", "type": "string"}, "arguments": {"title": "Arguments", "type": "object"}}, "required": ["name", "arguments"], "title": "FunctionCall", "type": "object"}}
+' }}
+{{- "For each function call return a json object with function name and arguments within XML tags as follows:
+" }}
+{{- "
+" }}
+{{- '{"name": , "arguments": }
+' }}
+{{- '<|im_end|>' }}
+{%- for message in messages %}
+ {%- if message.role == "user" or message.role == "system" or (message.role == "assistant" and message.tool_calls is not defined) %}
+ {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+ {%- elif message.role == "assistant" and message.tool_calls is defined %}
+ {{- '<|im_start|>' + message.role }}
+ {%- for tool_call in message.tool_calls %}
+ {{- '\n\n' }}
+ {%- if tool_call.function is defined %}
+ {%- set tool_call = tool_call.function %}
+ {%- endif %}
+ {{- '{' }}
+ {{- '"name": "' }}
+ {{- tool_call.name }}
+ {{- '"' }}
+ {%- if tool_call.arguments is defined %}
+ {{- ', ' }}
+ {{- '"arguments": ' }}
+ {{- tool_call.arguments|tojson }}
+ {%- endif %}
+ {{- '}' }}
+ {{- '\n' }}
+ {%- endfor %}
+ {{- '<|im_end|>\n' }}
+ {%- elif message.role == "tool" %}
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
+ {{- '<|im_start|>tool\n' }}
+ {%- endif %}
+ {{- '\n' }}
+ {{- message.content }}
+ {%- if not loop.last %}
+ {{- '\n\n' }}
+ {%- else %}
+ {{- '\n' }}
+ {%- endif %}
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
+ {{- '<|im_end|>' }}
+ {%- elif loop.last %}
+ {{- '<|im_end|>' }}
+ {%- endif %}
+ {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+ {{- '<|im_start|>assistant\n' }}
+{%- endif %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hunyuan_a13b.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hunyuan_a13b.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..8e3fd1fe6915f0b3f3dcc9901268b28708144ccc
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_hunyuan_a13b.jinja
@@ -0,0 +1,113 @@
+{% set loop_messages = messages %}
+{% if tools %}
+ {% set weekday_map = {'Monday': '星期一', 'Tuesday': '星期二', 'Wednesday': '星期三', 'Thursday': '星期四', 'Friday': '星期五', 'Saturday': '星期六', 'Sunday': '星期日'} %}
+ {% set weekday_cn = weekday_map[strftime_now('%A')] %}
+ {% set datetime_str = strftime_now('%Y-%m-%d %H:%M:%S') %}
+ {% set datetime_str = datetime_str + ' ' + weekday_cn %}
+ {% for message in loop_messages %}
+ {% if 'content' in message %}
+ {% set content = message['content'] %}
+ {% else %}
+ {% set content = '' %}
+ {% endif %}
+ {% if loop.index0 == 0 %}
+ {% set content_tmp = '你是一位函数组合专家。你会得到一个问题和一组可能的函数。根据问题,你需要进行一个或多个函数/工具调用以实现目的。
+如果没有一个函数可以使用,请直接使用自然语言回复用户,以助手:开头。
+如果给定的问题缺少函数所需的参数,请使用自然语言进行提问,向用户询问必要信息,以助手:开头。
+如果调用结果已经足够回答用户问题,请对历史结果进行总结,使用自然语言回复用户,以助手:开头。
+你应该只在工具调用部分返回函数调用。如果你决定调用任何函数,你必须将其格式化为[{"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},...]。你不应该在回复中包含任何其他文本。以下是你可以调用的函数列表,格式为JSON。
+' %}
+ {% set content_tmp = content_tmp + '
+' + tools | tojson + '
+' %}
+ {% if message['role'] == 'system' %}
+ {% set content_tmp = content_tmp + '
+额外要求:
+' + content + '
+
+如果你决定返回函数调用,请将其格式化为[{"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},...],不得包含其他文本。如果额外要求里有格式要求,请忽略,以此处为准。
+否则,请参考开头说的三种情况,以助手:开头进行回复。
+
+如果额外要求里有时间信息,就以额外要求里的时间为准,否则,参考当前时间:' + datetime_str %}
+ {% set content = '<|startoftext|>' + content_tmp + '<|extra_4|>' %}
+ {% elif message['role'] == 'user' %}
+ {% set content_tmp = content_tmp + '
+如果你决定返回函数调用,请将其格式化为[{"name": "func_name1", "arguments": {"argument1": "value1", "argument2": "value2"}},...],不得包含其他文本。
+否则,请参考开头说的三种情况,以助手:开头进行回复。
+
+当前时间:' + datetime_str %}
+ {% set content_tmp = '<|startoftext|>' + content_tmp + '<|extra_4|>'%}
+ {% set content = content_tmp + '用户:' + content + '<|extra_0|>' %}
+ {% endif %}
+ {% else %}
+ {% if message['role'] == 'user' %}
+ {% set content = '用户:' + content + '<|extra_0|>' %}
+ {% elif message['role'] == 'assistant' %}
+ {% if 'tool_calls' in message %}
+ {% set tool_calls = message['tool_calls'] %}
+ {% set ns = namespace(tool_calls="[") %}
+ {% for tool_call in tool_calls %}
+ {% set function = tool_call['function'] %}
+ {% set name = function['name'] %}
+ {% set ns.tool_calls = ns.tool_calls + '{"name": "' + name + '", '%}
+ {% set arguments = function['arguments'] %}
+ {% if arguments is not string %}
+ {% set arguments = arguments | tojson %}
+ {% endif %}
+ {% set ns.tool_calls = ns.tool_calls + '"arguments": ' + arguments + '}' %}
+ {% if not loop.last %}
+ {% set ns.tool_calls = ns.tool_calls + ', '%}
+ {% endif %}
+ {% endfor %}
+ {% set ns.tool_calls = ns.tool_calls + ']' %}
+ {% set content = content + '' + ns.tool_calls + '' %}
+ {% else %}
+ {% set content = '助手:' + content %}
+ {% endif %}
+ {% set content = content + '<|eos|>' %}
+ {% elif message['role'] == 'tool' %}
+ {% if content is not string %}
+ {set content = content | tojson }
+ {% endif %}
+ {% set content = '' + content + '' %}
+ {% set content = content + '<|extra_0|>' %}
+ {% endif %}
+ {% endif %}
+ {{- content -}}
+ {% endfor %}
+{% else %}
+ {% set context = {'has_head': true} %}
+ {% for message in loop_messages %}
+ {% if 'content' in message %}
+ {% set content = message['content'] %}
+ {% else %}
+ {% set content = '' %}
+ {% endif %}
+ {% if loop.index0 == 0 %}
+ {% if content == '' %}
+ {% set _ = context.update({'has_head': false}) %}
+ {% elif message['role'] == 'system' %}
+ {% set content = '<|startoftext|>' + content + '<|extra_4|>' %}
+ {% endif %}
+ {% endif %}
+ {% if message['role'] == 'user' %}
+ {% if loop.index0 == 1 and not context.has_head %}
+ {% set content = '<|startoftext|>' + content %}
+ {% endif %}
+ {% if loop.index0 == 1 and context.has_head %}
+ {% set content = content + '<|extra_0|>' %}
+ {% else %}
+ {% set content = '<|startoftext|>' + content + '<|extra_0|>' %}
+ {% endif %}
+ {% elif message['role'] == 'assistant' %}
+ {% set content = content + '<|eos|>' %}
+ {% elif message['role'] == 'tool' %}
+ {% set content = content + '<|extra_0|>' %}
+ {% endif %}
+ {{- content -}}
+ {% endfor %}
+{% endif %}
+{%- if enable_thinking is defined and enable_thinking is false %}
+ {{- '\n\n\n' }}
+{%- endif %}
+
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_internlm2_tool.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_internlm2_tool.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..9ad38a1593761bab38d1709c94c601553b55580c
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_internlm2_tool.jinja
@@ -0,0 +1,60 @@
+{%- if messages[0]["role"] == "system" %}
+ {%- set system_message = messages[0]["content"] %}
+ {%- set loop_messages = messages[1:] %}
+{%- else %}
+ {%- set loop_messages = messages %}
+{%- endif %}
+
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- endif %}
+
+{{- bos_token }}
+{%- if system_message is defined %}
+{{- "<|im_start|>system\n" + system_message + "<|im_end|>\n" }}
+{%- endif %}
+
+{%- if tools is not none %}
+ {{- "<|im_start|>system name=<|plugin|>\n[" }}
+ {%- for tool in tools %}
+ {{- tool.function|tojson }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- else %}
+ {{- "]" }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "<|im_end|>\n" }}
+{%- endif %}
+
+{%- for message in loop_messages %}
+ {%- if message["role"] == "user" %}
+ {{- "<|im_start|>user\n" + message["content"] + "<|im_end|>\n"}}
+ {%- elif message.tool_calls is defined and message.tool_calls is not none %}
+ {%- set content = message["content"] if message["content"] else "" %}
+ {{- "<|im_start|>assistant\n" + content }}
+ {%- for tool_call in message.tool_calls %}
+ {%- set function=tool_call.function %}
+ {{- "<|action_start|><|plugin|>\n" }}
+ {{- '{"name": "' + function.name + '", '}}
+ {{- '"arguments": ' + function.arguments|tojson + '}' }}
+ {{- "<|action_end|>" }}
+ {%- endfor %}
+ {{- "<|im_end|>\n" }}
+ {%- elif message["role"] == "assistant" %}
+ {{- "<|im_start|>assistant\n" + message["content"] + "<|im_end|>\n"}}
+ {%- elif message["role"] == "tool_results" or message["role"] == "tool" or message["role"] == "function" %}
+ {%- if message.content is defined and message.content.content is defined %}
+ {%- set content = message.content.content %}
+ {%- else %}
+ {%- set content = message.content %}
+ {%- endif %}
+ {{- "<|im_start|>environment name=<|plugin|>\n" + content|string + "<|im_end|>\n" }}
+ {%- else %}
+ {{- raise_exception("Only user and assistant and tool_results and tool and function roles are supported, with the exception of an initial optional system message!") }}
+ {%- endif %}
+{%- endfor %}
+
+{%- if add_generation_prompt %}
+{{- '<|im_start|>assistant\n' }}
+{%- endif %}
\ No newline at end of file
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.1_json.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.1_json.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..ea245f4dbc446b347f7b371ae73112bb6be09837
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.1_json.jinja
@@ -0,0 +1,120 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+ {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+ {#- Llama 3.1 doesn't pass all tests if the tools are in the system prompt #}
+ {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not date_string is defined %}
+ {%- if strftime_now is defined %}
+ {%- set date_string = strftime_now("%d %b %Y") %}
+ {%- else %}
+ {%- set date_string = "26 Jul 2024" %}
+ {%- endif %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+ {%- if messages[0]['content'] is string %}
+ {%- set system_message = messages[0]['content']|trim %}
+ {%- else %}
+ {%- set system_message = messages[0]['content'][0]['text']|trim %}
+ {%- endif %}
+ {%- set messages = messages[1:] %}
+{%- else %}
+ {%- if tools is not none %}
+ {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language. When you receive a tool call response, use the output to format an answer to the original user question." %}
+ {%- else %}
+ {%- set system_message = "" %}
+ {%- endif %}
+{%- endif %}
+
+{#- System message #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if tools is not none %}
+ {{- "Environment: ipython\n" }}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+ {{- "You have access to the following functions. To call a function, please respond with JSON for a function call. " }}
+ {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. ' }}
+ {{- "Do not use variables.\n\n" }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+ {#- Extract the first user message so we can plug it in here #}
+ {%- if messages | length != 0 %}
+ {%- if messages[0]['content'] is string %}
+ {%- set first_user_message = messages[0]['content']|trim %}
+ {%- else %}
+ {%- set first_user_message = messages[0]['content'] | selectattr('type', 'equalto', 'text') | map(attribute='text') | map('trim') | join('\n') %}
+ {%- endif %}
+ {%- set messages = messages[1:] %}
+ {%- else %}
+ {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+ {%- endif %}
+ {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+ {{- "Given the following functions, please respond with a JSON for a function call " }}
+ {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+ {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. ' }}
+ {{- "Do not use variables.\n\n" }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+ {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+ {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' }}
+ {%- if message['content'] is string %}
+ {{- message['content'] | trim}}
+ {%- else %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'text' %}
+ {{- content['text'] | trim }}
+ {%- endif %}
+ {%- endfor %}
+ {%- endif %}
+ {{- '<|eot_id|>' }}
+ {%- elif 'tool_calls' in message %}
+ {%- if not message.tool_calls|length == 1 %}
+ {{- raise_exception("This model only supports single tool-calls at once!") }}
+ {%- endif %}
+ {%- set tool_call = message.tool_calls[0].function %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+ {{- '{"name": "' + tool_call.name + '", ' }}
+ {{- '"parameters": ' }}
+ {{- tool_call.arguments | tojson }}
+ {{- "}" }}
+ {{- "<|eot_id|>" }}
+ {%- elif message.role == "tool" or message.role == "ipython" %}
+ {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+ {%- if message.content is string %}
+ {{- { "output": message.content } | tojson }}
+ {%- else %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'text' %}
+ {{- { "output": content['text'] } | tojson }}
+ {%- endif %}
+ {%- endfor %}
+ {%- endif %}
+ {{- "<|eot_id|>" }}
+ {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_json.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_json.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..5950c6773b8f1a4abcb12dd3c457b08c609cd1db
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_json.jinja
@@ -0,0 +1,133 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+ {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+ {%- set tools_in_user_message = false %}
+{%- endif %}
+{%- if not date_string is defined %}
+ {%- if strftime_now is defined %}
+ {%- set date_string = strftime_now("%d %b %Y") %}
+ {%- else %}
+ {%- set date_string = "26 Jul 2024" %}
+ {%- endif %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- endif %}
+
+{#- Find out if there are any images #}
+{% set image_ns = namespace(has_images=false) %}
+{%- for message in messages %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'image' %}
+ {%- set image_ns.has_images = true %}
+ {%- endif %}
+ {%- endfor %}
+{%- endfor %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+ {%- if messages[0]['content'] is string %}
+ {%- set system_message = messages[0]['content']|trim %}
+ {%- else %}
+ {%- set system_message = messages[0]['content'][0]['text']|trim %}
+ {%- endif %}
+ {%- set messages = messages[1:] %}
+{%- else %}
+ {%- if tools is not none %}
+ {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language. When you receive a tool call response, use the output to format an answer to the original user question." %}
+ {%- else %}
+ {%- set system_message = "" %}
+ {%- endif %}
+{%- endif %}
+
+{#- System message if there are no images, if the user supplied one, or if tools are used (default tool system message) #}
+{%- if system_message or not image_ns.has_images %}
+ {{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+ {%- if tools is not none %}
+ {{- "Environment: ipython\n" }}
+ {%- endif %}
+ {{- "Cutting Knowledge Date: December 2023\n" }}
+ {{- "Today Date: " + date_string + "\n\n" }}
+ {%- if tools is not none and not tools_in_user_message %}
+ {{- "You have access to the following functions. To call a function, please respond with JSON for a function call. " }}
+ {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. ' }}
+ {{- "Do not use variables.\n\n" }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+ {%- endif %}
+ {{- system_message }}
+ {{- "<|eot_id|>" }}
+{%- endif %}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+ {#- Extract the first user message so we can plug it in here #}
+ {%- if messages | length != 0 %}
+ {%- if messages[0]['content'] is string %}
+ {%- set first_user_message = messages[0]['content']|trim %}
+ {%- else %}
+ {%- set first_user_message = messages[0]['content'] | selectattr('type', 'equalto', 'text') | map(attribute='text') | map('trim') | join('\n') %}
+ {%- endif %}
+ {%- set messages = messages[1:] %}
+ {%- else %}
+ {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+ {%- endif %}
+ {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+ {{- "Given the following functions, please respond with a JSON for a function call " }}
+ {{- "with its proper arguments that best answers the given prompt.\n\n" }}
+ {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}. ' }}
+ {{- "Do not use variables.\n\n" }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+ {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+ {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n' }}
+ {%- if message['content'] is string %}
+ {{- message['content'] | trim}}
+ {%- else %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'image' %}
+ {{- '<|image|>' }}
+ {%- elif content['type'] == 'text' %}
+ {{- content['text'] | trim }}
+ {%- endif %}
+ {%- endfor %}
+ {%- endif %}
+ {{- '<|eot_id|>' }}
+ {%- elif 'tool_calls' in message %}
+ {%- if not message.tool_calls|length == 1 %}
+ {{- raise_exception("This model only supports single tool-calls at once!") }}
+ {%- endif %}
+ {%- set tool_call = message.tool_calls[0].function %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+ {{- '{"name": "' + tool_call.name + '", ' }}
+ {{- '"parameters": ' }}
+ {{- tool_call.arguments | tojson }}
+ {{- "}" }}
+ {{- "<|eot_id|>" }}
+ {%- elif message.role == "tool" or message.role == "ipython" %}
+ {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+ {%- if message.content is string %}
+ {{- { "output": message.content } | tojson }}
+ {%- else %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'text' %}
+ {{- { "output": content['text'] } | tojson }}
+ {%- endif %}
+ {%- endfor %}
+ {%- endif %}
+ {{- "<|eot_id|>" }}
+ {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_pythonic.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_pythonic.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..1abdd0898c0f278aa640a219a41c9eb3732991e4
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama3.2_pythonic.jinja
@@ -0,0 +1,98 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+ {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+ {%- set tools_in_user_message = false %}
+{%- endif %}
+{%- if not date_string is defined %}
+ {%- if strftime_now is defined %}
+ {%- set date_string = strftime_now("%d %b %Y") %}
+ {%- else %}
+ {%- set date_string = "26 Jul 2024" %}
+ {%- endif %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+ {%- set system_message = messages[0]['content']|trim %}
+ {%- set messages = messages[1:] %}
+{%- else %}
+ {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language. When you receive a tool call response, use the output to format an answer to the original user question." %}
+{%- endif %}
+
+{#- System message #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if tools is not none %}
+ {{- "Environment: ipython\n" }}
+{%- endif %}
+{{- "Cutting Knowledge Date: December 2023\n" }}
+{{- "Today Date: " + date_string + "\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+ {{- "You have access to the following functions. To call functions, please respond with a python list of the calls. " }}
+ {{- 'Respond in the format [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] ' }}
+ {{- "Do not use variables.\n\n" }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+ {#- Extract the first user message so we can plug it in here #}
+ {%- if messages | length != 0 %}
+ {%- set first_user_message = messages[0]['content']|trim %}
+ {%- set messages = messages[1:] %}
+ {%- else %}
+ {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+ {%- endif %}
+ {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
+ {{- "Given the following functions, please respond with a python list for function calls " }}
+ {{- "with their proper arguments to best answer the given prompt.\n\n" }}
+ {{- 'Respond in the format [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)] ' }}
+ {{- "Do not use variables.\n\n" }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+ {{- first_user_message + "<|eot_id|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+ {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+ {%- elif 'tool_calls' in message %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n[' -}}
+ {%- for tool_call in message.tool_calls %}
+ {%- if tool_call.function is defined %}
+ {%- set tool_call = tool_call.function %}
+ {%- endif %}
+ {{- tool_call.name + '(' -}}
+ {%- for param in tool_call.arguments %}
+ {{- param + '=' -}}
+ {{- "%s" | format(tool_call.arguments[param]) -}}
+ {% if not loop.last %}, {% endif %}
+ {%- endfor %}
+ {{- ')' -}}
+ {% if not loop.last %}, {% endif %}
+ {%- endfor %}
+ {{- ']<|eot_id|>' -}}
+ {%- elif message.role == "tool" or message.role == "ipython" %}
+ {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+ {%- if message.content is mapping %}
+ {{- message.content | tojson }}
+ {%- else %}
+ {{- { "output": message.content } | tojson }}
+ {%- endif %}
+ {{- "<|eot_id|>" }}
+ {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_json.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_json.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..d3202b8977969f5e66c6b5d63883b330edba8613
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_json.jinja
@@ -0,0 +1,116 @@
+{%- macro is_array_of_type_objects(var) -%}
+ {%- if var is iterable and var is not string -%}
+ {%- set valid = true -%}
+ {%- for item in var -%}
+ {%- if 'type' not in item -%}
+ {%- set valid = false -%}
+ {%- break -%}
+ {%- endif -%}
+ {%- endfor -%}
+ {{ valid }}
+ {%- else -%}
+ {{ false }}
+ {%- endif -%}
+{%- endmacro %}
+
+{%- macro render_message(message) %}
+ {%- if message['content'] is string %}
+ {{- message['content']|trim }}
+ {%- elif is_array_of_type_objects(data) == 'True' %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'image' %}
+ {{- '<|image|>' }}
+ {%- elif content['type'] == 'text' %}
+ {{- content['text']|trim }}
+ {%- endif %}
+ {%- endfor %}
+ {%- else %}
+ {{- message['content']|tojson }}
+ {%- endif %}
+{%- endmacro %}
+
+{{- bos_token }}
+{%- if custom_tools is defined %}
+ {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+ {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+ {%- set system_message = messages[0] %}
+ {%- set messages = messages[1:] %}
+{%- else %}
+ {%- set system_message = ({ "content": "You are a helpful assistant with tool calling "
+ "capabilities. Only reply with a tool call if the function exists in the "
+ "library provided by the user. If it doesn't exist, just reply directly in "
+ "natural language. When you receive a tool call response, use the output to "
+ "format an answer to the original user question."}) %}
+{%- endif %}
+
+{%- set tool_lib_preamble = 'Tools: You have access to the following tools. You might need to use one '
+ 'or more function/tool calls to fulfill the task. \n'
+ 'If none are needed, then proceed to the response.\n\n'
+ 'Tool Call Syntax: You can call tools using the following syntax:\n'
+ '{"name": function name, "parameters": dictionary of argument name and its value}.\n'
+ 'Separate multiple function calls by "; ". Do not use variables.\n'
+ 'Do not include anything else when calling the tools with the syntax above.\n\n'
+ 'Here is a list of functions in JSON format that you can invoke.\n' %}
+
+{{- "<|header_start|>system<|header_end|>\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+ {{- tool_lib_preamble }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+{%- endif %}
+{{- render_message(system_message) }}
+{{ "<|eot|>\n" }}
+
+{#- Custom tools are passed in a user message with some extra guidance #}
+{%- if tools_in_user_message and not tools is none %}
+ {#- Extract the first user message so we can plug it in here #}
+ {%- if messages | length != 0 %}
+ {%- set first_user_message = messages[0] %}
+ {%- set messages = messages[1:] %}
+ {%- else %}
+ {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
+ {%- endif %}
+ {{- '<|header_start|>user<|header_end|>\n\n' }}
+ {{- tool_lib_preamble }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+ {{- render_message(first_user_message) + "\n<|eot|>"}}
+{%- endif %}
+
+{%- for message in messages %}
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+ {{- '<|header_start|>' + message['role'] + '<|header_end|>\n\n' }}
+ {{- render_message(message) }}
+ {{- "\n<|eot|>" }}
+ {%- elif 'tool_calls' in message and message.tool_calls|length > 0 %}
+ {{- '\n<|header_start|>assistant<|header_end|>\n\n' -}}
+ {{- render_message(message) }}
+ {%- for tool_call in message.tool_calls %}
+ {{- '{"name": "' + tool_call.function.name + '", ' }}
+ {{- '"parameters": ' }}
+ {{- tool_call.function.arguments | tojson }}
+ {{- "}" }}
+ {%- endfor %}
+ {{- "\n<|eot|>" }}
+ {%- elif message.role == "tool" or message.role == "ipython" %}
+ {{- "\n<|header_start|>ipython<|header_end|>\n\n" }}
+ {{- render_message(message) }}
+ {{- "\n<|eom|>" }}
+ {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+ {{- '\n<|header_start|>assistant<|header_end|>\n\n' }}
+{%- endif %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_pythonic.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_pythonic.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..55c1472aecb201dc80e8643ab07006d116a76583
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_llama4_pythonic.jinja
@@ -0,0 +1,111 @@
+{{- bos_token }}
+{%- if custom_tools is defined and custom_tools%}
+ {%- set tools = custom_tools %}
+{%- endif %}
+{%- if tools is defined and tools %}
+ {%- set tool_definition = tool_definition ~ (tools | tojson(indent=4)) %}
+{%- else %}
+ {%- set tools = none %}
+{%- endif %}
+
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+ {%- set user_provided_system_message = true %}
+ {%- if messages[0]['content'] is string %}
+ {%- set system_message = messages[0]['content']|trim %}
+ {%- else %}
+ {%- set system_message = messages[0]['content'][0]['text']|trim %}
+ {%- endif %}
+ {%- set messages = messages[1:] %}
+{%- else %}
+ {%- if tools is not none %}
+ {#- Since not system_message was provided by user, if tool is provided, system_message is now default tool system message #}
+ {#- This system message is from llama website:https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/ #}
+ {%- set system_message = "You are a helpful assistant and an expert in function composition. You can answer general questions using your internal knowledge OR invoke functions when necessary. Follow these strict guidelines:\n\n1. FUNCTION CALLS:\n- ONLY use functions that are EXPLICITLY listed in the function list below\n- If NO functions are listed (empty function list []), respond ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\"\n- If a function is not in the list, respond ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\"\n- If ALL required parameters are present AND the query EXACTLY matches a listed function's purpose: output ONLY the function call(s)\n- Use exact format: [func_name1(param1=value1, param2=value2), func_name2(...)]\nExamples:\nCORRECT: [get_weather(location=\"Vancouver\"), calculate_route(start=\"Boston\", end=\"New York\")] <- Only if get_weather and calculate_route are in function list\nINCORRECT: get_weather(location=\"New York\")\nINCORRECT: Let me check the weather: [get_weather(location=\"New York\")]\nINCORRECT: [get_events(location=\"Singapore\")] <- If function not in list\n\n2. RESPONSE RULES:\n- For pure function requests matching a listed function: ONLY output the function call(s)\n- For knowledge questions: ONLY output text\n- For missing parameters: ONLY request the specific missing parameters\n- For unavailable services (not in function list): output ONLY with internal knowledge or \"I don't have access to [Unavailable service] information\". Do NOT execute a function call.\n- If the query asks for information beyond what a listed function provides: output ONLY with internal knowledge about your limitations\n- NEVER combine text and function calls in the same response\n- NEVER suggest alternative functions when the requested service is unavailable\n- NEVER create or invent new functions not listed below\n\n3. STRICT BOUNDARIES:\n- ONLY use functions from the list below - no exceptions\n- NEVER use a function as an alternative to unavailable information\n- NEVER call functions not present in the function list\n- NEVER add explanatory text to function calls\n- NEVER respond with empty brackets\n- Use proper Python/JSON syntax for function calls\n- Check the function list carefully before responding\n\n4. TOOL RESPONSE HANDLING:\n- When receiving tool responses: provide concise, natural language responses\n- Don't repeat tool response verbatim\n- Don't add supplementary information\n\nHere is a list of functions in JSON format that you can invoke:\n" %}
+ {%- else %}
+ {%- set system_message = "" %}
+ {%- endif %}
+{%- endif %}
+{#- Now writing the system message: use the user provided system message if user_provided_system_message, else default tool system message if tools presented #}
+{%- if system_message %}
+ {#- always use user provided system message to override default tool system message #}
+ {{- "<|header_start|>system<|header_end|>\n\n" }}
+ {{- system_message }}
+ {%- if user_provided_system_message and tools %}
+ {{- "\nHere is a list of functions in JSON format that you can invoke. Use exact format: [func_name1(param1=value1, param2=value2), func_name2(...)]\n" }}
+ {{- tool_definition -}}
+ {%- elif tool_definition %}
+ {{- tool_definition -}}
+ {%- endif %}
+ {{- "<|eot|>" }}
+{%- endif %}
+
+{#- Now deal with all other messages #}
+{%- for message in messages %}
+ {#- Base case: messages that are not from tool role and has empty tool_call list #}
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or ('tool_calls' in message and message.tool_calls|length != 0 )) %}
+ {{- '<|header_start|>' + message['role'] + '<|header_end|>\n\n' }}
+ {%- if message['content'] is string %}
+ {{- message['content'] }}
+ {%- else %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'image' %}
+ {{- '<|image|>' }}
+ {%- elif content['type'] == 'text' %}
+ {{- content['text'] | trim }}
+ {%- endif %}
+ {%- endfor %}
+ {%- endif %}
+ {{- "<|eot|>" }}
+ {#- Tool case: messages has non-empty tool_call list, must from assistant #}
+ {%- elif 'tool_calls' in message %}
+ {#- assume tool_calls are always coming from assistant #}
+ {%- if message.role == 'assistant' %}
+ {{- '<|header_start|>assistant<|header_end|>\n\n' -}}
+ {%- if message['content'] is string %}
+ {{- message['content'] }}
+ {%- else %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'image' %}
+ {{- '<|image|>' }}
+ {%- elif content['type'] == 'text' %}
+ {{- content['text'] }}
+ {%- endif %}
+ {%- endfor %}
+ {%- endif %}
+ {{- "[" }}
+ {%- for tool_call in message.tool_calls %}
+ {%- if tool_call.function is defined %}
+ {%- set tool_call = tool_call.function %}
+ {%- endif %}
+ {{- tool_call.name + '(' -}}
+ {%- for param in tool_call.arguments %}
+ {{- param + '="' -}}
+ {{- "%s" | format(tool_call.arguments[param]) -}}
+ {{- '"' -}}
+ {% if not loop.last %}, {% endif %}
+ {%- endfor %}
+ {{- ')' -}}
+ {% if not loop.last %}, {% endif %}
+ {%- endfor %}
+ {{- "]<|eot|>" }}
+{%- endif %}
+{#- Tool_response case: messages are from tool_response #}
+ {%- elif message.role == "tool" or message.role == "ipython" %}
+ {{- "<|header_start|>ipython<|header_end|>\n\n" }}
+ {%- if message.content is string %}
+ {{- message.content | tojson }}
+ {%- else %}
+ {%- for content in message['content'] %}
+ {%- if content['type'] == 'text' %}
+ {{- content['text'] | tojson }}
+ {%- endif %}
+ {%- endfor %}
+ {%- endif %}
+ {{- "<|eot|>" }}
+ {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+ {{- '<|header_start|>assistant<|header_end|>\n\n' }}
+{%- endif %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..013ff60c1255d1a85efbe6ae66245a9c8621557f
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral.jinja
@@ -0,0 +1,86 @@
+{%- if messages[0]["role"] == "system" %}
+ {%- set system_message = messages[0]["content"] %}
+ {%- set loop_messages = messages[1:] %}
+{%- else %}
+ {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- endif %}
+{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
+
+{%- for message in loop_messages | rejectattr("role", "equalto", "tool") | rejectattr("role", "equalto", "tool_results") | selectattr("tool_calls", "undefined") %}
+ {%- if (message["role"] == "user") != (loop.index0 % 2 == 0) %}
+ {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
+ {%- endif %}
+{%- endfor %}
+
+{{- bos_token }}
+{%- for message in loop_messages %}
+ {%- if message["role"] == "user" %}
+ {%- if tools is not none and (message == user_messages[-1]) %}
+ {{- "[AVAILABLE_TOOLS] [" }}
+ {%- for tool in tools %}
+ {%- set tool = tool.function %}
+ {{- '{"type": "function", "function": {' }}
+ {%- for key, val in tool.items() if key != "return" %}
+ {%- if val is string %}
+ {{- '"' + key + '": "' + val + '"' }}
+ {%- else %}
+ {{- '"' + key + '": ' + val|tojson }}
+ {%- endif %}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "}}" }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- else %}
+ {{- "]" }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "[/AVAILABLE_TOOLS]" }}
+ {%- endif %}
+ {%- if loop.last and system_message is defined %}
+ {{- "[INST] " + system_message + "\n\n" + message["content"] + "[/INST]" }}
+ {%- else %}
+ {{- "[INST] " + message["content"] + "[/INST]" }}
+ {%- endif %}
+ {%- elif message["role"] == "tool_calls" or message.tool_calls is defined %}
+ {%- if message.tool_calls is defined %}
+ {%- set tool_calls = message.tool_calls %}
+ {%- else %}
+ {%- set tool_calls = message.content %}
+ {%- endif %}
+ {{- "[TOOL_CALLS] [" }}
+ {%- for tool_call in tool_calls %}
+ {%- set out = tool_call.function|tojson %}
+ {{- out[:-1] }}
+ {%- if not tool_call.id is defined or tool_call.id|length < 9 %}
+ {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (1)" + tool_call.id) }}
+ {%- endif %}
+ {{- ', "id": "' + tool_call.id[-9:] + '"}' }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- else %}
+ {{- "]" + eos_token }}
+ {%- endif %}
+ {%- endfor %}
+ {%- elif message["role"] == "assistant" %}
+ {{- " " + message["content"] + eos_token }}
+ {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
+ {%- if message.content is defined and message.content.content is defined %}
+ {%- set content = message.content.content %}
+ {%- else %}
+ {%- set content = message.content %}
+ {%- endif %}
+ {{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }}
+ {%- if not message.tool_call_id is defined or message.tool_call_id|length < 9 %}
+ {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (2)" + message.tool_call_id) }}
+ {%- endif %}
+ {{- '"call_id": "' + message.tool_call_id[-9:] + '"}[/TOOL_RESULTS]' }}
+ {%- else %}
+ {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
+ {%- endif %}
+{%- endfor %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral3.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral3.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..e9325f8b533dd6a6c7436d7da6f0ca73c1d8a2e1
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral3.jinja
@@ -0,0 +1,126 @@
+{%- set today = strftime_now("%Y-%m-%d") %}
+{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information, you say that you don't have the information and don't make up anything.\nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \"What are some good restaurants around me?\" => \"Where are you?\" or \"When is the next flight to Tokyo\" => \"Where do you travel from?\")" %}
+
+{{- bos_token }}
+
+{%- if messages[0]['role'] == 'system' %}
+ {%- if messages[0]['content'] is string %}
+ {%- set system_message = messages[0]['content'] %}
+ {%- set loop_messages = messages[1:] %}
+ {%- else %}
+ {%- set system_message = messages[0]['content'][0]['text'] %}
+ {%- set loop_messages = messages[1:] %}
+ {%- endif %}
+{%- else %}
+ {%- set system_message = default_system_message %}
+ {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- elif tools is not none %}
+ {%- set parallel_tool_prompt = "You are a helpful assistant that can call tools. If you call one or more tools, format them in a single JSON array or objects, where each object is a tool call, not as separate objects outside of an array or multiple arrays. Use the format [{\"name\": tool call name, \"arguments\": tool call arguments}, additional tool calls] if you call more than one tool. If you call tools, do not attempt to interpret them or otherwise provide a response until you receive a tool call result that you can interpret for the user." %}
+ {%- if system_message is defined %}
+ {%- set system_message = parallel_tool_prompt + "\n\n" + system_message %}
+ {%- else %}
+ {%- set system_message = parallel_tool_prompt %}
+ {%- endif %}
+{%- endif %}
+{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }}
+
+{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
+
+{%- set filtered_messages = [] %}
+{%- for message in loop_messages %}
+ {%- if message["role"] not in ["tool", "tool_results"] and not message.get("tool_calls") %}
+ {%- set filtered_messages = filtered_messages + [message] %}
+ {%- endif %}
+{%- endfor %}
+
+{%- for message in filtered_messages %}
+ {%- if (message["role"] == "user") != (loop.index0 % 2 == 0) %}
+ {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
+ {%- endif %}
+{%- endfor %}
+
+{%- for message in loop_messages %}
+ {%- if message["role"] == "user" %}
+ {%- if tools is not none and (message == user_messages[-1]) %}
+ {{- "[AVAILABLE_TOOLS] [" }}
+ {%- for tool in tools %}
+ {%- set tool = tool.function %}
+ {{- '{"type": "function", "function": {' }}
+ {%- for key, val in tool.items() if key != "return" %}
+ {%- if val is string %}
+ {{- '"' + key + '": "' + val + '"' }}
+ {%- else %}
+ {{- '"' + key + '": ' + val|tojson }}
+ {%- endif %}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "}}" }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- else %}
+ {{- "]" }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "[/AVAILABLE_TOOLS]" }}
+ {%- endif %}
+ {%- if message['content'] is string %}
+ {{- '[INST]' + message['content'] + '[/INST]' }}
+ {%- else %}
+ {{- '[INST]' }}
+ {%- for block in message['content'] %}
+ {%- if block['type'] == 'text' %}
+ {{- block['text'] }}
+ {%- elif block['type'] == 'image' or block['type'] == 'image_url' %}
+ {{- '[IMG]' }}
+ {%- else %}
+ {{- raise_exception('Only text and image blocks are supported in message content!') }}
+ {%- endif %}
+ {%- endfor %}
+ {{- '[/INST]' }}
+ {%- endif %}
+ {%- elif message["role"] == "tool_calls" or message.tool_calls is defined %}
+ {%- if message.tool_calls is defined %}
+ {%- set tool_calls = message.tool_calls %}
+ {%- else %}
+ {%- set tool_calls = message.content %}
+ {%- endif %}
+ {{- "[TOOL_CALLS] [" }}
+ {%- for tool_call in tool_calls %}
+ {%- set out = tool_call.function|tojson %}
+ {{- out[:-1] }}
+ {%- if not tool_call.id is defined or tool_call.id|length < 9 %}
+ {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (1)" + tool_call.id) }}
+ {%- endif %}
+ {{- ', "id": "' + tool_call.id[-9:] + '"}' }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- else %}
+ {{- "]" + eos_token }}
+ {%- endif %}
+ {%- endfor %}
+ {%- elif message['role'] == 'assistant' %}
+ {%- if message['content'] is string %}
+ {{- message['content'] + eos_token }}
+ {%- else %}
+ {{- message['content'][0]['text'] + eos_token }}
+ {%- endif %}
+ {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
+ {%- if message.content is defined and message.content.content is defined %}
+ {%- set content = message.content.content %}
+ {%- else %}
+ {%- set content = message.content %}
+ {%- endif %}
+ {{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }}
+ {%- if not message.tool_call_id is defined or message.tool_call_id|length < 9 %}
+ {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (2)" + message.tool_call_id) }}
+ {%- endif %}
+ {{- '"call_id": "' + message.tool_call_id[-9:] + '"}[/TOOL_RESULTS]' }}
+ {%- else %}
+ {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
+ {%- endif %}
+{%- endfor %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral_parallel.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral_parallel.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..97d1ad55e3e0e418b4b415a3fedebfdef983ab1a
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_mistral_parallel.jinja
@@ -0,0 +1,93 @@
+{%- if messages[0]["role"] == "system" %}
+ {%- set system_message = messages[0]["content"] %}
+ {%- set loop_messages = messages[1:] %}
+{%- else %}
+ {%- set loop_messages = messages %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- elif tools is not none %}
+ {%- set parallel_tool_prompt = "You are a helpful assistant that can call tools. If you call one or more tools, format them in a single JSON array or objects, where each object is a tool call, not as separate objects outside of an array or multiple arrays. Use the format [{\"name\": tool call name, \"arguments\": tool call arguments}, additional tool calls] if you call more than one tool. If you call tools, do not attempt to interpret them or otherwise provide a response until you receive a tool call result that you can interpret for the user." %}
+ {%- if system_message is defined %}
+ {%- set system_message = parallel_tool_prompt + "\n\n" + system_message %}
+ {%- else %}
+ {%- set system_message = parallel_tool_prompt %}
+ {%- endif %}
+{%- endif %}
+{%- set user_messages = loop_messages | selectattr("role", "equalto", "user") | list %}
+
+{%- for message in loop_messages | rejectattr("role", "equalto", "tool") | rejectattr("role", "equalto", "tool_results") | selectattr("tool_calls", "undefined") %}
+ {%- if (message["role"] == "user") != (loop.index0 % 2 == 0) %}
+ {{- raise_exception("After the optional system message, conversation roles must alternate user/assistant/user/assistant/...") }}
+ {%- endif %}
+{%- endfor %}
+
+{{- bos_token }}
+{%- for message in loop_messages %}
+ {%- if message["role"] == "user" %}
+ {%- if tools is not none and (message == user_messages[-1]) %}
+ {{- "[AVAILABLE_TOOLS] [" }}
+ {%- for tool in tools %}
+ {%- set tool = tool.function %}
+ {{- '{"type": "function", "function": {' }}
+ {%- for key, val in tool.items() if key != "return" %}
+ {%- if val is string %}
+ {{- '"' + key + '": "' + val + '"' }}
+ {%- else %}
+ {{- '"' + key + '": ' + val|tojson }}
+ {%- endif %}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "}}" }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- else %}
+ {{- "]" }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "[/AVAILABLE_TOOLS]" }}
+ {%- endif %}
+ {%- if loop.last and system_message is defined %}
+ {{- "[INST] " + system_message + "\n\n" + message["content"] + "[/INST]" }}
+ {%- else %}
+ {{- "[INST] " + message["content"] + "[/INST]" }}
+ {%- endif %}
+ {%- elif message["role"] == "tool_calls" or message.tool_calls is defined %}
+ {%- if message.tool_calls is defined %}
+ {%- set tool_calls = message.tool_calls %}
+ {%- else %}
+ {%- set tool_calls = message.content %}
+ {%- endif %}
+ {{- "[TOOL_CALLS] [" }}
+ {%- for tool_call in tool_calls %}
+ {%- set out = tool_call.function|tojson %}
+ {{- out[:-1] }}
+ {%- if not tool_call.id is defined or tool_call.id|length < 9 %}
+ {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (1)" + tool_call.id) }}
+ {%- endif %}
+ {{- ', "id": "' + tool_call.id[-9:] + '"}' }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- else %}
+ {{- "]" + eos_token }}
+ {%- endif %}
+ {%- endfor %}
+ {%- elif message["role"] == "assistant" %}
+ {{- " " + message["content"] + eos_token }}
+ {%- elif message["role"] == "tool_results" or message["role"] == "tool" %}
+ {%- if message.content is defined and message.content.content is defined %}
+ {%- set content = message.content.content %}
+ {%- else %}
+ {%- set content = message.content %}
+ {%- endif %}
+ {{- '[TOOL_RESULTS] {"content": ' + content|string + ", " }}
+ {%- if not message.tool_call_id is defined or message.tool_call_id|length < 9 %}
+ {{- raise_exception("Tool call IDs should be alphanumeric strings with length >= 9! (2)" + message.tool_call_id) }}
+ {%- endif %}
+ {{- '"call_id": "' + message.tool_call_id[-9:] + '"}[/TOOL_RESULTS]' }}
+ {%- else %}
+ {{- raise_exception("Only user and assistant roles are supported, with the exception of an initial optional system message!") }}
+ {%- endif %}
+{%- endfor %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_phi4_mini.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_phi4_mini.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..1c351476affdfe8ddf8c7a49b3318d922b8efa29
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_phi4_mini.jinja
@@ -0,0 +1,62 @@
+{%- if messages and messages[0]['role'] == 'system' %}
+ {%- set system_message = messages[0]['content']|trim %}
+ {%- set messages = messages[1:] %}
+{%- else %}
+ {%- set system_message = "You are a helpful assistant." %}
+{%- endif %}
+
+{%- if messages %}
+<|system|>
+{{ system_message }}
+{%- if tools %}
+In addition to plain text responses, you can choose to call one or more of the provided functions.
+
+Use the following rule to decide when to call a function:
+ * if the response can be generated from your internal knowledge (e.g., as in the case of queries like "What is the capital of Poland?"), do so
+ * if you need external information that can be obtained by calling one or more of the provided functions, generate a function calls
+
+If you decide to call functions:
+ * prefix function calls with functools marker (no closing marker required)
+ * all function calls should be generated in a single JSON list formatted as functools[{"name": [function name], "arguments": [function arguments as JSON]}, ...]
+ * follow the provided JSON schema. Do not hallucinate arguments or values. Do to blindly copy values from the provided samples
+ * respect the argument type formatting. E.g., if the type is number and format is float, write value 7 as 7.0
+ * make sure you pick the right functions that match the user intent
+
+
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+{%- endif %}<|end|>
+
+ {%- for message in messages %}
+ {%- if message.role != "system" %}
+<|{{ message.role }}|>
+ {%- if message.content and message.role == "tools" %}
+{"result": {{ message.content }}}
+ {%- elif message.content %}
+{{ message.content }}
+ {%- elif message.tool_calls %}
+ {%- for call in message.tool_calls %}
+{"name": "{{ call.function.name }}", "arguments": {{ call.function.arguments }}}
+ {%- if not loop.last %},{% endif %}
+ {%- endfor %}
+ {%- endif %}<|end|>
+ {%- endif %}
+ {%- endfor %}<|assistant|>
+
+{%- else %}
+ {%- if system_message %}
+<|system|>
+
+{{ system_message }}<|end|>
+ {%- endif %}
+ {%- if prompt %}
+<|user|>
+
+{{ prompt }}<|end|>
+ {%- endif %}<|assistant|>
+
+{%- endif %}
+{{ response }}
+{%- if response %}<|user|>{% endif %}
\ No newline at end of file
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_qwen3coder.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_qwen3coder.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..389b27415b7baffd6eecbadd6b15d03b74be98c2
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_qwen3coder.jinja
@@ -0,0 +1,117 @@
+{% macro render_extra_keys(json_dict, handled_keys) %}
+ {%- if json_dict is mapping %}
+ {%- for json_key in json_dict if json_key not in handled_keys %}
+ {%- if json_dict[json_key] is mapping or (json_dict[json_key] is sequence and json_dict[json_key] is not string) %}
+ {{- '\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | tojson | safe) ~ '' ~ json_key ~ '>' }}
+ {%- else %}
+ {{-'\n<' ~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '' ~ json_key ~ '>' }}
+ {%- endif %}
+ {%- endfor %}
+ {%- endif %}
+{% endmacro %}
+
+{%- if messages[0]["role"] == "system" %}
+ {%- set system_message = messages[0]["content"] %}
+ {%- set loop_messages = messages[1:] %}
+{%- else %}
+ {%- set loop_messages = messages %}
+{%- endif %}
+
+{%- if not tools is defined %}
+ {%- set tools = [] %}
+{%- endif %}
+
+{%- if system_message is defined %}
+ {{- "<|im_start|>system\n" + system_message }}
+{%- else %}
+ {%- if tools is iterable and tools | length > 0 %}
+ {{- "<|im_start|>system\nYou are Qwen, a helpful AI assistant that can interact with a computer to solve tasks." }}
+ {%- endif %}
+{%- endif %}
+{%- if tools is iterable and tools | length > 0 %}
+ {{- "\n\n# Tools\n\nYou have access to the following functions:\n\n" }}
+ {{- "" }}
+ {%- for tool in tools %}
+ {%- if tool.function is defined %}
+ {%- set tool = tool.function %}
+ {%- endif %}
+ {{- "\n\n" ~ tool.name ~ "" }}
+ {%- if tool.description is defined %}
+ {{- '\n' ~ (tool.description | trim) ~ '' }}
+ {%- endif %}
+ {{- '\n' }}
+ {%- if tool.parameters is defined and tool.parameters is mapping and tool.parameters.properties is defined and tool.parameters.properties is mapping %}
+ {%- for param_name, param_fields in tool.parameters.properties|items %}
+ {{- '\n' }}
+ {{- '\n' ~ param_name ~ '' }}
+ {%- if param_fields.type is defined %}
+ {{- '\n' ~ (param_fields.type | string) ~ '' }}
+ {%- endif %}
+ {%- if param_fields.description is defined %}
+ {{- '\n' ~ (param_fields.description | trim) ~ '' }}
+ {%- endif %}
+ {%- set handled_keys = ['name', 'type', 'description'] %}
+ {{- render_extra_keys(param_fields, handled_keys) }}
+ {{- '\n' }}
+ {%- endfor %}
+ {%- endif %}
+ {% set handled_keys = ['type', 'properties'] %}
+ {{- render_extra_keys(tool.parameters, handled_keys) }}
+ {{- '\n' }}
+ {%- set handled_keys = ['type', 'name', 'description', 'parameters'] %}
+ {{- render_extra_keys(tool, handled_keys) }}
+ {{- '\n' }}
+ {%- endfor %}
+ {{- "\n" }}
+ {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n\n\n\nvalue_1\n\n\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n- Function calls MUST follow the specified format: an inner block must be nested within XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n' }}
+{%- endif %}
+{%- if system_message is defined %}
+ {{- '<|im_end|>\n' }}
+{%- else %}
+ {%- if tools is iterable and tools | length > 0 %}
+ {{- '<|im_end|>\n' }}
+ {%- endif %}
+{%- endif %}
+{%- for message in loop_messages %}
+ {%- if message.role == "assistant" and message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 %}
+ {{- '<|im_start|>' + message.role }}
+ {%- if message.content is defined and message.content is string and message.content | trim | length > 0 %}
+ {{- '\n' + message.content | trim + '\n' }}
+ {%- endif %}
+ {%- for tool_call in message.tool_calls %}
+ {%- if tool_call.function is defined %}
+ {%- set tool_call = tool_call.function %}
+ {%- endif %}
+ {{- '\n\n\n' }}
+ {%- if tool_call.arguments is defined %}
+ {%- for args_name, args_value in tool_call.arguments|items %}
+ {{- '\n' }}
+ {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+ {{- args_value }}
+ {{- '\n\n' }}
+ {%- endfor %}
+ {%- endif %}
+ {{- '\n' }}
+ {%- endfor %}
+ {{- '<|im_end|>\n' }}
+ {%- elif message.role == "user" or message.role == "system" or message.role == "assistant" %}
+ {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+ {%- elif message.role == "tool" %}
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
+ {{- '<|im_start|>user\n' }}
+ {%- endif %}
+ {{- '\n' }}
+ {{- message.content }}
+ {{- '\n\n' }}
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
+ {{- '<|im_end|>\n' }}
+ {%- elif loop.last %}
+ {{- '<|im_end|>\n' }}
+ {%- endif %}
+ {%- else %}
+ {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>\n' }}
+ {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+ {{- '<|im_start|>assistant\n' }}
+{%- endif %}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_toolace.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_toolace.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..78d0426b9bfe551925b0d1da1360b702320df817
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_toolace.jinja
@@ -0,0 +1,65 @@
+{{- bos_token }}
+
+{%- if custom_tools is defined %}
+ {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- endif %}
+
+{#- This block extracts the system message, so we can slot it into the right place. #}
+{%- if messages[0]['role'] == 'system' %}
+ {%- set system_message = messages[0]['content']|trim %}
+ {%- set messages = messages[1:] %}
+{%- else %}
+ {%- set system_message = "You are a helpful assistant with tool calling capabilities. Only reply with a tool call if the function exists in the library provided by the user. If it doesn't exist, just reply directly in natural language." %}
+{%- endif %}
+
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if tools is not none and not tools_in_user_message %}
+ {{- "You are an expert in composing functions. You are given a question and a set of possible functions. Based on the question, you will need to make one or more function/tool calls to achieve the purpose.\n" }}
+ {{- "If none of the function can be used, point it out. If the given question lacks the parameters required by the function, also point it out.\n" }}
+ {{- "You should only return the function call in tools call sections.\n\n" }}
+ {{- "If you decide to invoke any of the function(s), you MUST put it in the format of [func_name1(params_name1=params_value1, params_name2=params_value2...), func_name2(params)]\n" }}
+ {{- "You SHOULD NOT include any other text in the response.\n" }}
+ {{- "Here is a list of functions in JSON format that you can invoke.\n" }}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+ {{- "\n" }}
+{%- endif %}
+{{- system_message }}
+{{- "<|eot_id|>" }}
+
+{%- for message in messages %}
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+ {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+ {%- elif 'tool_calls' in message %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n[' -}}
+ {%- for tool_call in message.tool_calls %}
+ {%- if tool_call.function is defined %}
+ {%- set tool_call = tool_call.function %}
+ {%- endif %}
+ {{- tool_call.name + '(' -}}
+ {%- for param in tool_call.arguments %}
+ {{- param + '=' -}}
+ {{- "%s" | format(tool_call.arguments[param]) -}}
+ {% if not loop.last %}, {% endif %}
+ {%- endfor %}
+ {{- ')' -}}
+ {% if not loop.last %}, {% endif %}
+ {%- endfor %}
+ {{- ']<|eot_id|>' -}}
+ {%- elif message.role == "tool" or message.role == "ipython" %}
+ {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
+ {%- if message.content is mapping %}
+ {{- message.content | tojson }}
+ {%- else %}
+ {{- { "output": message.content } | tojson }}
+ {%- endif %}
+ {{- "<|eot_id|>" }}
+ {%- endif %}
+{%- endfor %}
+
+{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_llama.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_llama.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..596af47fdde664f300071f11a6e7c6b4c414e128
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_llama.jinja
@@ -0,0 +1,77 @@
+{{- bos_token }}
+{%- if custom_tools is defined %}
+ {%- set tools = custom_tools %}
+{%- endif %}
+{%- if not tools_in_user_message is defined %}
+ {%- set tools_in_user_message = true %}
+{%- endif %}
+{%- if not tools is defined %}
+ {%- set tools = none %}
+{%- endif %}
+
+{#- Extract system message #}
+{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
+{%- if messages[0]['role'] == 'system' %}
+ {%- set system_message = messages[0]['content'] | trim %}
+ {%- set messages = messages[1:] %}
+ {{- system_message + "\n" }}
+{%- else %}
+ {%- set system_message = "You are a helpful assistant. You are developed by Salesforce xLAM team." %}
+ {% set format_instruction %}You have access to a set of tools. When using tools, make calls in a single JSON array:
+
+[{"name": "tool_call_name", "arguments": {"arg1": "value1", "arg2": "value2"}}, ... (additional parallel tool calls as needed)]
+
+If no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:{% endset %}
+ {{- system_message + "\n" }}
+ {%- if tools is not none %}
+ {{- format_instruction + "\n\n" }}
+ {%- endif %}
+{%- endif %}
+
+
+{%- if tools is not none %}
+ {%- for t in tools %}
+ {{- t | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+{%- endif %}
+{{- "<|eot_id|>" }}
+
+{%- for message in messages %}
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
+ {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
+ {%- elif 'tool_calls' in message %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
+ {%- if message['tool_calls'] %}
+ {{- "[" }}
+ {%- for tool_call_function in message.tool_calls %}
+ {%- set tool_call = tool_call_function.function %}
+ {{- '{"name": "' + tool_call.name + '", ' }}
+ {{- '"arguments": ' }}
+ {{- tool_call.arguments | tojson }}
+ {{- "}" }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "]" }}
+ {{- "<|eot_id|>" }}
+ {%- elif message['content'] %}
+ {{- message['content'] | trim + '<|eot_id|>' }}
+ {%- else %}
+ {{- "[]\n" + '<|eot_id|>' }}
+ {%- endif %}
+ {%- elif message.role == "tool" or message.role == "ipython" %}
+ {{- "<|start_header_id|>" + "ipython" + "<|end_header_id|>\n\n" }}
+ {%- set content = message["content"] %}
+ {%- if content is mapping or (content is iterable and content is not string) %}
+ {{- content | tojson }}
+ {%- else %}
+ {{- content }}
+ {%- endif %}
+ {{- "<|eot_id|>" }}
+ {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
+{%- endif %}
\ No newline at end of file
diff --git a/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_qwen.jinja b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_qwen.jinja
new file mode 100644
index 0000000000000000000000000000000000000000..830f55c8a3017eda321d726d993ed95b55dab847
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/chat/tests/templates/vllm_examples/tool_chat_template_xlam_qwen.jinja
@@ -0,0 +1,66 @@
+{# System message #}
+{{- "<|im_start|>system\n" }}
+{%- if messages[0]['role'] == 'system' %}
+ {%- set system_message = messages[0]['content'] | trim %}
+ {%- set messages = messages[1:] %}
+ {{- system_message + "\n" }}
+{%- else %}
+ {%- set system_message = "You are a helpful assistant. You are developed by Salesforce xLAM team." %}
+ {% set format_instruction %}You have access to a set of tools. When using tools, make calls in a single JSON array:
+
+[{"name": "tool_call_name", "arguments": {"arg1": "value1", "arg2": "value2"}}, ... (additional parallel tool calls as needed)]
+
+If no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:{% endset %}
+ {{- system_message + "\n" }}
+ {%- if tools is not none %}
+ {{- format_instruction + "\n\n" }}
+ {%- endif %}
+{%- endif %}
+
+{%- if tools is not none %}
+ {%- for func in tools %}
+ {{- func | tojson(indent=4) }}
+ {{- "\n\n" }}
+ {%- endfor %}
+{%- endif %}
+{{- "<|im_end|>\n" }}
+{%- for message in messages %}
+ {%- if message['role'] == 'tool' %}
+ {{- "<|im_start|>tool\n" }}
+ {%- if message.content is defined and message.content.content is defined %}
+ {%- set content = message.content.content %}
+ {%- else %}
+ {%- set content = message.content %}
+ {%- endif %}
+ {%- if content is mapping or content is iterable and content is not string %}
+ {{- content | tojson }}
+ {%- else %}
+ {{- content }}
+ {%- endif %}
+ {{- "<|im_end|>\n" }}
+ {%- elif 'tool_calls' in message %}
+ {{- "<|im_start|>assistant\n" }}
+ {%- if message['tool_calls'] %}
+ {{- "[" }}
+ {%- for tool_call in message.tool_calls %}
+ {%- set out = tool_call.function | tojson %}
+ {{- out }}
+ {%- if not loop.last %}
+ {{- ", " }}
+ {%- endif %}
+ {%- endfor %}
+ {{- "]"}}
+ {%- elif message['content'] %}
+ {{- message['content'] | trim }}
+ {%- else %}
+ {{- "[]\n" }}
+ {%- endif %}
+ {{- "<|im_end|>\n" }}
+ {%- else %}
+ {{- "<|im_start|>" + message['role'] + "\n" + message['content'] | trim + "<|im_end|>\n" }}
+ {%- endif %}
+{%- endfor %}
+
+{%- if add_generation_prompt %}
+ {{- "<|im_start|>assistant\n" }}
+{%- endif %}
diff --git a/asian_ai/performance/vLLM/rust/src/cmd/Cargo.toml b/asian_ai/performance/vLLM/rust/src/cmd/Cargo.toml
new file mode 100644
index 0000000000000000000000000000000000000000..4048c664e0ed6a7c849c65c77f7b2f0fe2521aee
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/cmd/Cargo.toml
@@ -0,0 +1,41 @@
+[package]
+name = "vllm-cmd"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[[bin]]
+name = "vllm-rs"
+path = "src/main.rs"
+
+[features]
+default = []
+native-tls-vendored = ["dep:native-tls-vendored"]
+
+[dependencies]
+anyhow.workspace = true
+clap.workspace = true
+educe.workspace = true
+itertools.workspace = true
+mimalloc.workspace = true
+native-tls-vendored = { workspace = true, optional = true }
+serde.workspace = true
+serde_json.workspace = true
+serde_with.workspace = true
+thiserror-ext.workspace = true
+time.workspace = true
+tokio = { workspace = true, features = ["signal"] }
+tokio-util.workspace = true
+tracing.workspace = true
+tracing-subscriber.workspace = true
+uuid.workspace = true
+vllm-chat.workspace = true
+vllm-engine-core-client.workspace = true
+vllm-managed-engine.workspace = true
+vllm-server.workspace = true
+
+[dev-dependencies]
+expect-test.workspace = true
+
+[lints]
+workspace = true
diff --git a/asian_ai/performance/vLLM/rust/src/cmd/examples/README.md b/asian_ai/performance/vLLM/rust/src/cmd/examples/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..ba8ef545a473675b448766a769a908de238e5042
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/cmd/examples/README.md
@@ -0,0 +1,44 @@
+# `vllm-rs` CLI Quick Start
+
+Start Qwen3 with one managed `vllm-rs serve` command from the repo root:
+
+```bash
+HF_HUB_OFFLINE=1 \
+VLLM_CPU_KVCACHE_SPACE=2 \
+VLLM_HOST_IP=127.0.0.1 \
+VLLM_LOOPBACK_IP=127.0.0.1 \
+cargo run --bin vllm-rs -- serve \
+ Qwen/Qwen3-0.6B \
+ --python ../vllm/.venv/bin/python \
+ --max-model-len 512 \
+ -- \
+ --dtype float16
+```
+
+This launches:
+
+- a managed headless Python `vllm` engine
+- the Rust OpenAI-compatible frontend on `127.0.0.1:8000`
+
+All Python engine arguments must be placed after `--`. Arguments before `--` are parsed by the Rust
+frontend itself.
+
+You can then send OpenAI-style requests to the Rust frontend:
+
+```bash
+curl http://127.0.0.1:8000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "Qwen/Qwen3-0.6B",
+ "messages": [{"role": "user", "content": "What is the capital of France?"}],
+ "stream": true
+ }'
+```
+
+If you already started headless `vllm` yourself, use `frontend` instead:
+
+```bash
+cargo run --bin vllm-rs -- frontend \
+ --handshake-address tcp://127.0.0.1:62100 \
+ Qwen/Qwen3-0.6B
+```
diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/cli.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/cli.rs
new file mode 100644
index 0000000000000000000000000000000000000000..5b697f7e142387c7eacf99a7fb86b4ed78bf5558
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/cmd/src/cli.rs
@@ -0,0 +1,694 @@
+//! CLI argument definitions for the `vllm-rs` binary.
+//!
+//! Python vLLM references:
+//! - Engine args:
+//! - Environment variables:
+
+mod unsupported;
+
+use std::collections::HashMap;
+use std::ffi::{OsStr, OsString};
+use std::path::PathBuf;
+use std::time::Duration;
+
+use clap::{Args, Parser, Subcommand};
+use educe::Educe;
+use serde::de::DeserializeOwned;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+use serde_with::{DefaultOnNull, OneOrMany, serde_as};
+use thiserror_ext::AsReport as _;
+use uuid::Uuid;
+use vllm_chat::ReasoningParserFactory;
+use vllm_engine_core_client::TransportMode;
+use vllm_managed_engine::ManagedEngineConfig;
+use vllm_managed_engine::cli::{ManagedEngineArgs, repartition_managed_engine_args};
+use vllm_server::{
+ ApiServerOptions, ChatTemplateContentFormatOption, Config, CoordinatorMode, CorsConfig,
+ DEFAULT_KEEP_ALIVE_TIMEOUT, HttpListenerMode, ParserSelection, RendererSelection, TlsConfig,
+};
+
+use crate::cli::unsupported::UnsupportedArgs;
+
+/// Top-level parser for the `vllm-rs` binary.
+#[derive(Debug, Parser)]
+#[command(
+ name = "vllm-rs",
+ about = "Rust frontend and managed-engine CLI for vLLM."
+)]
+pub struct Cli {
+ #[command(subcommand)]
+ pub command: Command,
+}
+
+impl Cli {
+ pub fn parse() -> Self {
+ Self::try_parse_from(std::env::args_os()).unwrap_or_else(|error| error.exit())
+ }
+
+ pub fn try_parse_from(itr: I) -> Result
+ where
+ I: IntoIterator- ,
+ T: Into,
+ {
+ let args: Vec = itr.into_iter().map(Into::into).collect();
+ let repartitioned_args = repartition_managed_engine_args::(&args, Some("serve"))?;
+ ::try_parse_from(&repartitioned_args).inspect(|cli| {
+ if let Command::Serve(serve) = &cli.command
+ && serve.debug_cli
+ {
+ println!(
+ "Original CLI args: {}\n",
+ args.join(OsStr::new(" ")).display()
+ );
+ println!(
+ "Repartitioned CLI args: {}\n",
+ repartitioned_args.join(OsStr::new(" ")).display()
+ );
+ println!(
+ "Passthrough Python args: {}",
+ serve.managed_engine.python_args.join(" ")
+ );
+ std::process::exit(0);
+ }
+ })
+ }
+}
+
+/// Supported top-level CLI commands.
+#[derive(Debug, Subcommand, PartialEq, Eq)]
+pub enum Command {
+ /// Run the Rust OpenAI frontend as a Python-supervised worker.
+ Frontend(FrontendArgs),
+ /// Launch a managed Python headless engine, then run the Rust OpenAI
+ /// frontend.
+ Serve(ServeArgs),
+}
+
+/// A JSON-encoded list of strings, matching Python's `json.loads` CLI type for
+/// the CORS list arguments (e.g. `--allowed-origins '["*"]'`). Parsing the whole
+/// value as one item keeps clap from treating the field as a repeated flag.
+#[derive(Clone, Debug, PartialEq, Eq, Deserialize)]
+#[serde(transparent)]
+pub struct JsonStringList(pub Vec);
+
+/// Runtime arguments shared by both paths of the Rust frontend:
+///
+/// - External-engine mode: Python-supervised bootstrap, `vllm serve` -> `vllm-rs frontend`.
+/// Arguments are deserialized from a single JSON object and defaults follow `serde` attrs.
+/// - Managed-engine mode: Rust-managed Python engine, `vllm-rs serve`.
+/// Arguments are parsed from CLI flags and defaults follow `clap` attrs.
+#[serde_as]
+#[derive(Educe, Clone, Args, PartialEq, Eq, Deserialize)]
+#[educe(Debug)]
+pub struct SharedRuntimeArgs {
+ #[serde(rename = "model_tag")]
+ /// Model identifier or local model directory used for backend loading and
+ /// public model ID.
+ pub model: String,
+
+ /// Maximum time to wait for the expected engines to register on the
+ /// frontend transport.
+ #[arg(
+ long = "engine-ready-timeout-secs",
+ env = "VLLM_ENGINE_READY_TIMEOUT_S",
+ default_value_t = default_engine_ready_timeout_secs()
+ )]
+ #[serde(default = "default_engine_ready_timeout_secs")]
+ pub engine_ready_timeout_secs: u64,
+
+ /// Select the tool call parser depending on the model that you're using.
+ /// Use `auto` to infer from the model or `none` to disable parsing.
+ #[arg(long, default_value_t)]
+ #[serde(default = "default_py_bootstrap_parser_selection")]
+ pub tool_call_parser: ParserSelection,
+ /// Select the reasoning parser depending on the model that you're using.
+ /// Use `auto` to infer from the model or `none` to disable parsing.
+ #[arg(long, default_value_t)]
+ #[serde(default = "default_py_bootstrap_parser_selection")]
+ pub reasoning_parser: ParserSelection,
+ /// Select the chat renderer implementation.
+ #[arg(long = "tokenizer-mode", default_value_t)]
+ #[serde(default, rename = "tokenizer_mode")]
+ pub renderer: RendererSelection,
+ /// Disable multimodal inputs and treat the model as language-only.
+ #[arg(long)]
+ #[serde(default)]
+ pub language_model_only: bool,
+ /// Override the maximum model context length. When set, the frontend uses
+ /// this value instead of the model's `max_position_embeddings` from
+ /// `config.json`.
+ #[arg(long)]
+ pub max_model_len: Option,
+ /// Maximum number of log probabilities to return when `logprobs` is
+ /// specified in sampling parameters. `-1` means no cap.
+ #[arg(long, value_parser = clap::value_parser!(i32).range(-1..), allow_negative_numbers = true)]
+ #[serde(default)]
+ pub max_logprobs: Option,
+ /// TCP port for the gRPC Generate service. When not set, no gRPC server is
+ /// started.
+ #[arg(long)]
+ #[serde(default)]
+ pub grpc_port: Option,
+ /// Maximum time to wait for active requests to drain during shutdown.
+ #[arg(long, default_value_t = 0)]
+ #[serde(default)]
+ pub shutdown_timeout: u64,
+ /// Maximum idle time (seconds) on a keep-alive HTTP connection before the
+ /// server closes it (default 5).
+ #[arg(long = "http-timeout-keep-alive", env = "VLLM_HTTP_TIMEOUT_KEEP_ALIVE")]
+ #[serde(default)]
+ pub http_timeout_keep_alive: Option,
+
+ /// The file path to the chat template, or the template in single-line form
+ /// for the specified model.
+ #[arg(long)]
+ #[serde(default)]
+ pub chat_template: Option,
+
+ /// Default keyword arguments to pass to the chat template renderer.
+ ///
+ /// These will be merged with request-level chat_template_kwargs, with
+ /// request values taking precedence. Useful for setting default
+ /// behavior for reasoning models.
+ ///
+ /// Example: `{"enable_thinking": false}` to disable thinking mode by
+ /// default for Qwen3/DeepSeek models.
+ #[arg(long, value_parser = parse_json::>, value_name = "JSON")]
+ #[serde(default)]
+ pub default_chat_template_kwargs: Option>,
+
+ /// The format to render message content within a chat template.
+ ///
+ /// * "auto" detects the format from the template
+ /// * "string" renders content as a string. Example: `"Hello World"`
+ /// * "openai" renders content as a list of dictionaries, similar to OpenAI schema. Example:
+ /// `[{"type": "text", "text": "Hello world!"}]`
+ #[arg(long, default_value_t)]
+ #[serde(default)]
+ pub chat_template_content_format: ChatTemplateContentFormatOption,
+
+ /// Log a summary line for each completed request, including prompt/output
+ /// token counts and finish reason.
+ #[arg(long)]
+ #[serde(default)]
+ pub enable_log_requests: bool,
+
+ /// Include prompt_tokens_details in usage when cached prompt tokens are
+ /// present.
+ #[arg(
+ long,
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ #[serde(default)]
+ pub enable_prompt_tokens_details: bool,
+
+ /// If specified, API server will add X-Request-Id header to responses.
+ #[arg(
+ long,
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ #[serde(default)]
+ pub enable_request_id_headers: bool,
+
+ /// If provided, the server will require one of these keys to be presented
+ /// in the Authorization header.
+ #[educe(Debug(ignore))]
+ #[arg(long, env = "VLLM_API_KEY", value_delimiter = ' ')]
+ #[serde_as(as = "DefaultOnNull>")]
+ #[serde(default)]
+ pub api_key: Vec,
+
+ /// Disable periodic logging of engine statistics (throughput, queue depth,
+ /// cache usage).
+ #[arg(long)]
+ #[serde(default)]
+ pub disable_log_stats: bool,
+
+ /// The model name(s) used in the API. If multiple names are provided, the
+ /// server will respond to any of the provided names. The model name in the
+ /// model field of a response will be the first name in this list. If not
+ /// specified, the model name will be the same as the `--model` argument.
+ /// Noted that this name(s) will also be used in `model_name` tag
+ /// content of prometheus metrics, if multiple names provided, metrics
+ /// tag will take the first one.
+ #[arg(long, num_args = 0..)]
+ #[serde(default)]
+ pub served_model_name: Vec,
+
+ /// CORS allowed origins as a JSON list. `["*"]` allows any origin.
+ #[arg(long, value_parser = parse_json::, value_name = "JSON", default_value = r#"["*"]"#)]
+ #[serde(default = "default_cors_wildcard")]
+ pub allowed_origins: JsonStringList,
+
+ /// CORS allowed methods as a JSON list. `["*"]` allows the standard set.
+ #[arg(long, value_parser = parse_json::, value_name = "JSON", default_value = r#"["*"]"#)]
+ #[serde(default = "default_cors_wildcard")]
+ pub allowed_methods: JsonStringList,
+
+ /// CORS allowed request headers as a JSON list. `["*"]` mirrors the request.
+ #[arg(long, value_parser = parse_json::, value_name = "JSON", default_value = r#"["*"]"#)]
+ #[serde(default = "default_cors_wildcard")]
+ pub allowed_headers: JsonStringList,
+
+ /// Allow CORS credentials (cookies, authorization headers).
+ #[arg(
+ long,
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ #[serde(default)]
+ pub allow_credentials: bool,
+
+ /// The file path to the SSL key file. When omitted, the key is read from
+ /// `--ssl-certfile` (combined PEM).
+ #[arg(long)]
+ #[serde(default)]
+ pub ssl_keyfile: Option,
+
+ /// The file path to the SSL cert file. Enables TLS when set.
+ #[arg(long)]
+ #[serde(default)]
+ pub ssl_certfile: Option,
+
+ /// The CA certificates file used to verify client certificates (mTLS).
+ #[arg(long)]
+ #[serde(default)]
+ pub ssl_ca_certs: Option,
+
+ /// Whether a client certificate is required: 0 = none, 1 = optional,
+ /// 2 = required (mirrors Python's `ssl.CERT_*`).
+ #[arg(long, default_value_t = 0, value_parser = clap::value_parser!(i32).range(0..=2))]
+ #[serde(default)]
+ pub ssl_cert_reqs: i32,
+
+ /// OpenSSL cipher string for HTTPS (TLS 1.2 and below).
+ /// When unset, the linked OpenSSL's default suites are used.
+ #[arg(long)]
+ #[serde(default)]
+ pub ssl_ciphers: Option,
+
+ /// Profiler configuration forwarded by the Python supervisor.
+ ///
+ /// When set with a non-null `profiler` type, the Rust frontend registers
+ /// the `/start_profile` and `/stop_profile` routes and forwards calls to
+ /// the engine via the `"profile"` utility RPC.
+ #[arg(long, value_parser = parse_json::, value_name = "JSON")]
+ #[serde(default)]
+ pub profiler_config: Option,
+
+ /// Unsupported Python vLLM frontend arguments recognized but not yet
+ /// implemented in Rust.
+ #[educe(Debug(ignore))]
+ #[command(flatten)]
+ #[serde(default, flatten)]
+ pub unsupported: UnsupportedArgs,
+}
+
+impl SharedRuntimeArgs {
+ /// Maximum time to wait for the expected engines to register on the
+ /// frontend transport.
+ pub fn ready_timeout(&self) -> Duration {
+ Duration::from_secs(self.engine_ready_timeout_secs)
+ }
+
+ /// Maximum time to wait for active requests to drain during shutdown.
+ pub fn shutdown_timeout(&self) -> Duration {
+ Duration::from_secs(self.shutdown_timeout)
+ }
+
+ /// Maximum idle time on a keep-alive HTTP connection before the server
+ /// closes it.
+ pub fn keep_alive_timeout(&self) -> Duration {
+ self.http_timeout_keep_alive
+ .map_or(DEFAULT_KEEP_ALIVE_TIMEOUT, Duration::from_secs)
+ }
+
+ /// Return the configured profiler mode, when profiling is enabled.
+ pub fn profiler(&self) -> Option {
+ self.profiler_config.as_ref().and_then(|c| c.profiler.clone())
+ }
+
+ /// Return the profiler config JSON for managed Python engine forwarding.
+ pub fn profiler_config_json(&self) -> Option {
+ self.profiler_config
+ .as_ref()
+ .map(serde_json::to_string)
+ .transpose()
+ .expect("profiler config serialization should not fail")
+ }
+
+ /// Apply fallback logic for API key configuration from env variables.
+ fn apply_env_api_key_fallback(&mut self) {
+ if self.api_key.is_empty()
+ && let Ok(api_key) = std::env::var("VLLM_API_KEY")
+ {
+ self.api_key.push(api_key);
+ }
+ }
+
+ /// Build the OpenAI-server config for the Python-bootstrap worker contract.
+ ///
+ /// The resulting config binds the Python-supplied transport addresses and
+ /// inherits an already open HTTP listener from the supervisor process.
+ fn into_bootstrapped_config(
+ self,
+ listen_fd: i32,
+ input_address: String,
+ output_address: String,
+ coordinator_address: Option,
+ engine_start_index: u32,
+ engine_count: usize,
+ ) -> Config {
+ let ready_timeout = self.ready_timeout();
+ let shutdown_timeout = self.shutdown_timeout();
+ let keep_alive_timeout = self.keep_alive_timeout();
+ let api_server_options = self.api_server_options();
+ let cors = self.cors_config();
+ let tls = self.tls_config();
+ let profiler = self.profiler();
+
+ Config {
+ transport_mode: TransportMode::Bootstrapped {
+ input_address,
+ output_address,
+ engine_start_index,
+ engine_count,
+ ready_timeout,
+ },
+ coordinator_mode: match coordinator_address {
+ Some(address) => CoordinatorMode::External { address },
+ None => CoordinatorMode::None,
+ },
+ model: self.model,
+ served_model_name: self.served_model_name,
+ listener_mode: HttpListenerMode::InheritedFd { fd: listen_fd },
+ tool_call_parser: self.tool_call_parser,
+ reasoning_parser: self.reasoning_parser,
+ renderer: self.renderer,
+ language_model_only: self.language_model_only,
+ chat_template: self.chat_template,
+ default_chat_template_kwargs: self.default_chat_template_kwargs,
+ chat_template_content_format: self.chat_template_content_format,
+ max_logprobs: self.max_logprobs,
+ api_server_options,
+ cors,
+ tls,
+ api_keys: self.api_key,
+ disable_log_stats: self.disable_log_stats,
+ grpc_port: self.grpc_port,
+ shutdown_timeout,
+ keep_alive_timeout,
+ profiler,
+ }
+ }
+
+ /// Build the OpenAI-server config for the managed `serve` path that still
+ /// owns the startup handshake and binds its own HTTP listener.
+ fn into_managed_config(
+ self,
+ listener_mode: HttpListenerMode,
+ handshake_address: String,
+ advertised_host: String,
+ engine_count: usize,
+ local_input_address: Option,
+ local_output_address: Option,
+ ) -> Config {
+ let ready_timeout = self.ready_timeout();
+ let shutdown_timeout = self.shutdown_timeout();
+ let keep_alive_timeout = self.keep_alive_timeout();
+ let api_server_options = self.api_server_options();
+ let cors = self.cors_config();
+ let tls = self.tls_config();
+ let profiler = self.profiler();
+
+ Config {
+ transport_mode: TransportMode::HandshakeOwner {
+ handshake_address,
+ advertised_host,
+ engine_count,
+ ready_timeout,
+ local_input_address,
+ local_output_address,
+ },
+ coordinator_mode: CoordinatorMode::MaybeInProc,
+ model: self.model,
+ served_model_name: self.served_model_name,
+ listener_mode,
+ tool_call_parser: self.tool_call_parser,
+ reasoning_parser: self.reasoning_parser,
+ renderer: self.renderer,
+ language_model_only: self.language_model_only,
+ chat_template: self.chat_template,
+ default_chat_template_kwargs: self.default_chat_template_kwargs,
+ chat_template_content_format: self.chat_template_content_format,
+ max_logprobs: self.max_logprobs,
+ api_server_options,
+ cors,
+ tls,
+ api_keys: self.api_key,
+ disable_log_stats: self.disable_log_stats,
+ grpc_port: self.grpc_port,
+ shutdown_timeout,
+ keep_alive_timeout,
+ profiler,
+ }
+ }
+
+ fn api_server_options(&self) -> ApiServerOptions {
+ ApiServerOptions {
+ enable_log_requests: self.enable_log_requests,
+ enable_prompt_tokens_details: self.enable_prompt_tokens_details,
+ enable_request_id_headers: self.enable_request_id_headers,
+ }
+ }
+
+ fn cors_config(&self) -> CorsConfig {
+ CorsConfig {
+ allow_origins: self.allowed_origins.0.clone(),
+ allow_methods: self.allowed_methods.0.clone(),
+ allow_headers: self.allowed_headers.0.clone(),
+ allow_credentials: self.allow_credentials,
+ }
+ }
+
+ /// Build the TLS config: `Some` when any `ssl_*` argument is set, else
+ /// `None` (plaintext). The combination is validated in [`Config::validate`].
+ fn tls_config(&self) -> Option {
+ let tls_requested = self.ssl_certfile.is_some()
+ || self.ssl_keyfile.is_some()
+ || self.ssl_ca_certs.is_some()
+ || self.ssl_cert_reqs != 0
+ || self.ssl_ciphers.is_some();
+ tls_requested.then(|| TlsConfig {
+ cert_file: self.ssl_certfile.clone(),
+ key_file: self.ssl_keyfile.clone(),
+ ca_certs: self.ssl_ca_certs.clone(),
+ cert_reqs: self.ssl_cert_reqs,
+ ciphers: self.ssl_ciphers.clone(),
+ })
+ }
+}
+
+fn default_engine_ready_timeout_secs() -> u64 {
+ 600
+}
+
+fn default_cors_wildcard() -> JsonStringList {
+ JsonStringList(vec!["*".to_string()])
+}
+
+fn default_py_bootstrap_parser_selection() -> ParserSelection {
+ ParserSelection::None
+}
+
+/// Minimal profiler configuration parsed from `--profiler-config`.
+///
+/// Only the `profiler` field is inspected by the Rust frontend to decide
+/// whether to register the `/start_profile` and `/stop_profile` routes.
+/// All other fields are accepted but ignored — they are consumed by the
+/// Python engine layer.
+#[derive(Clone, Debug, Default, PartialEq, Eq, Deserialize, Serialize)]
+pub struct ProfilerConfig {
+ /// Profiler backend type (e.g. `"torch"`, `"cuda"`). When `null` or
+ /// absent, profiling is disabled.
+ #[serde(default)]
+ pub profiler: Option,
+ /// Additional Python profiler config fields consumed by the engine layer.
+ #[serde(flatten)]
+ pub extra: serde_json::Map,
+}
+
+fn parse_json(value: &str) -> Result {
+ serde_json::from_str(value).map_err(|e| format!("invalid JSON object: {}", e.as_report()))
+}
+
+fn parse_runtime_args_json(value: &str) -> Result {
+ let mut args: SharedRuntimeArgs = serde_json::from_str(value)
+ .map_err(|e| format!("invalid JSON arguments: {}", e.as_report()))?;
+ // --args-json is parsed with serde, so clap's env support does not run for
+ // the Python-supervised frontend path.
+ args.apply_env_api_key_fallback();
+ args.unsupported.check()?;
+ Ok(args)
+}
+
+/// Arguments for running the Rust frontend as a Python-bootstrapped worker.
+#[derive(Educe, Clone, Args, PartialEq, Eq)]
+#[educe(Debug)]
+pub struct FrontendArgs {
+ /// Inherited listening socket file descriptor passed by the Python
+ /// supervisor.
+ #[arg(long)]
+ pub listen_fd: i32,
+ /// Frontend input ROUTER socket address that the Python engines will
+ /// connect to.
+ #[arg(long)]
+ pub input_address: String,
+ /// Frontend output PULL socket address that the Python engines will push
+ /// responses to.
+ #[arg(long)]
+ pub output_address: String,
+ /// Optional Python-owned frontend-side DP coordinator socket address for
+ /// external coordinator mode in the bootstrapped frontend path, i.e.,
+ /// `stats_update_address`.
+ #[arg(long)]
+ pub coordinator_address: Option,
+ /// First data-parallel engine rank expected to register with this
+ /// bootstrapped frontend.
+ #[arg(long, default_value_t = 0)]
+ pub engine_start_index: u32,
+ /// Total number of data-parallel engines expected for this frontend.
+ #[arg(long, default_value_t = 1)]
+ pub engine_count: usize,
+
+ /// Shared frontend arguments as one JSON object.
+ #[arg(long = "args-json", value_parser = parse_runtime_args_json, value_name = "JSON")]
+ pub runtime: SharedRuntimeArgs,
+}
+
+impl FrontendArgs {
+ /// Convert the CLI arguments into the OpenAI server's runtime config.
+ pub fn into_config(self) -> Config {
+ self.runtime.into_bootstrapped_config(
+ self.listen_fd,
+ self.input_address,
+ self.output_address,
+ self.coordinator_address,
+ self.engine_start_index,
+ self.engine_count,
+ )
+ }
+}
+
+/// Arguments for the managed-engine mode that spawns Python on behalf of the
+/// user.
+#[derive(Educe, Clone, Args, PartialEq, Eq)]
+#[educe(Debug)]
+#[command(override_usage = "vllm-rs serve [OPTIONS] [-- ...]")]
+pub struct ServeArgs {
+ /// Only launch the managed Python headless engine and do not start the Rust
+ /// frontend.
+ #[arg(long)]
+ pub headless: bool,
+ /// HTTP bind host for the OpenAI-compatible server.
+ #[arg(long, default_value = "127.0.0.1")]
+ pub host: String,
+ /// HTTP bind port for the OpenAI-compatible server.
+ #[arg(long, default_value_t = 8000)]
+ pub port: u16,
+ /// Unix domain socket path. If set, host and port arguments are ignored.
+ #[arg(long)]
+ pub uds: Option,
+
+ /// Flag to print debug information about CLI argument parsing and exit.
+ #[educe(Debug(ignore))]
+ #[arg(long, hide = true, env = "VLLM_RS_DEBUG_CLI")]
+ pub debug_cli: bool,
+
+ /// Shared frontend arguments.
+ #[command(flatten)]
+ pub runtime: SharedRuntimeArgs,
+
+ /// Managed Python headless-engine arguments.
+ #[command(flatten)]
+ pub managed_engine: ManagedEngineArgs,
+}
+
+impl ServeArgs {
+ /// Build the OpenAI-server runtime config used after the managed Python
+ /// engine starts.
+ pub fn to_frontend_config(&self, handshake_address: String) -> Config {
+ // Prefer IPC sockets for local engine input/output.
+ let (local_input_address, local_output_address) =
+ self.managed_engine.frontend_local_only().then(frontend_ipc_addresses).unzip();
+ let listener_mode = match &self.uds {
+ Some(path) => HttpListenerMode::BindUnix { path: path.clone() },
+ None => HttpListenerMode::BindTcp {
+ host: self.host.clone(),
+ port: self.port,
+ },
+ };
+
+ self.runtime.clone().into_managed_config(
+ listener_mode,
+ handshake_address,
+ self.managed_engine.handshake_host.clone(),
+ self.managed_engine.data_parallel_size,
+ local_input_address,
+ local_output_address,
+ )
+ }
+
+ /// Build the managed Python-engine spawn configuration with the given
+ /// handshake port.
+ pub fn to_managed_engine_config(&self, handshake_port: u16) -> ManagedEngineConfig {
+ let reasoning_parser =
+ effective_engine_reasoning_parser(&self.runtime.reasoning_parser, &self.runtime.model);
+ let profiler_config = self.runtime.profiler_config_json();
+
+ self.managed_engine.clone().into_config(
+ self.runtime.model.clone(),
+ self.runtime.max_model_len,
+ self.runtime.max_logprobs,
+ profiler_config,
+ reasoning_parser.as_deref(),
+ self.runtime.language_model_only,
+ self.runtime.disable_log_stats,
+ self.runtime.shutdown_timeout,
+ handshake_port,
+ )
+ }
+}
+
+fn effective_engine_reasoning_parser(selection: &ParserSelection, model: &str) -> Option {
+ match selection {
+ ParserSelection::Auto => ReasoningParserFactory::global()
+ .resolve_name_for_model(model)
+ .map(str::to_string),
+ ParserSelection::None => None,
+ ParserSelection::Explicit(name) => Some(name.clone()),
+ }
+}
+
+/// Allocate fresh IPC endpoints for one managed frontend instance.
+fn frontend_ipc_addresses() -> (String, String) {
+ let preferred_base_path = std::env::var_os("VLLM_RPC_BASE_PATH")
+ .map(PathBuf::from)
+ .unwrap_or_else(std::env::temp_dir);
+ let input_name = format!("vllm-rs-i-{}", Uuid::new_v4().simple());
+ let output_name = format!("vllm-rs-o-{}", Uuid::new_v4().simple());
+
+ let input = preferred_base_path.join(input_name);
+ let output = preferred_base_path.join(output_name);
+
+ (
+ format!("ipc://{}", input.to_string_lossy()),
+ format!("ipc://{}", output.to_string_lossy()),
+ )
+}
+
+#[cfg(test)]
+mod tests;
diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/cli/tests.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/cli/tests.rs
new file mode 100644
index 0000000000000000000000000000000000000000..b1c384c3544aa3b3e6461c1c3ac8d7208cd0a208
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/cmd/src/cli/tests.rs
@@ -0,0 +1,1718 @@
+use expect_test::expect;
+use vllm_engine_core_client::TransportMode;
+use vllm_server::{Config, HttpListenerMode, ParserSelection, RendererSelection};
+
+use super::{Cli, Command};
+
+#[test]
+fn serve_args_forward_python_flags_with_separator() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--python",
+ "../vllm/.venv/bin/python",
+ "--max-model-len",
+ "512",
+ "--",
+ "--dtype",
+ "float16",
+ ])
+ .unwrap();
+
+ expect![[r#"
+ Cli {
+ command: Serve(
+ ServeArgs {
+ headless: false,
+ host: "127.0.0.1",
+ port: 8000,
+ uds: None,
+ runtime: SharedRuntimeArgs {
+ model: "Qwen/Qwen3-0.6B",
+ engine_ready_timeout_secs: 600,
+ tool_call_parser: Auto,
+ reasoning_parser: Auto,
+ renderer: Auto,
+ language_model_only: false,
+ max_model_len: Some(
+ 512,
+ ),
+ max_logprobs: None,
+ grpc_port: None,
+ shutdown_timeout: 0,
+ http_timeout_keep_alive: None,
+ chat_template: None,
+ default_chat_template_kwargs: None,
+ chat_template_content_format: Auto,
+ enable_log_requests: false,
+ enable_prompt_tokens_details: false,
+ enable_request_id_headers: false,
+ disable_log_stats: false,
+ served_model_name: [],
+ allowed_origins: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allowed_methods: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allowed_headers: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allow_credentials: false,
+ ssl_keyfile: None,
+ ssl_certfile: None,
+ ssl_ca_certs: None,
+ ssl_cert_reqs: 0,
+ ssl_ciphers: None,
+ profiler_config: None,
+ },
+ managed_engine: ManagedEngineArgs {
+ python: "../vllm/.venv/bin/python",
+ handshake_host: "127.0.0.1",
+ handshake_port: None,
+ data_parallel_size: 1,
+ data_parallel_size_local: None,
+ python_args: [
+ "--dtype",
+ "float16",
+ ],
+ },
+ },
+ ),
+ }
+ "#]]
+ .assert_debug_eq(&cli);
+}
+
+#[test]
+fn serve_args_auto_forward_python_flags_without_separator() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--python",
+ "python3",
+ "--quantization",
+ "awq",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "--quantization",
+ "awq",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_auto_forward_enable_lora_to_python() {
+ let cli =
+ Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "--enable-lora"]).unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "--enable-lora",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_forward_shutdown_timeout_to_managed_engine() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--shutdown-timeout",
+ "60",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert_eq!(args.runtime.shutdown_timeout, 60);
+
+ let config = args.to_managed_engine_config(5555);
+ expect![[r#"
+ [
+ "--reasoning-parser",
+ "qwen3",
+ "--shutdown-timeout",
+ "60",
+ ]
+ "#]]
+ .assert_debug_eq(&config.python_args);
+}
+
+#[test]
+fn serve_args_forward_disable_log_stats_to_managed_engine() {
+ let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "--disable-log-stats"])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert!(args.runtime.disable_log_stats);
+
+ let config = args.to_managed_engine_config(5555);
+ expect![[r#"
+ [
+ "--reasoning-parser",
+ "qwen3",
+ "--disable-log-stats",
+ ]
+ "#]]
+ .assert_debug_eq(&config.python_args);
+}
+
+#[test]
+fn serve_args_forward_profiler_config_to_managed_engine() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--profiler-config",
+ r#"{"profiler":"torch","torch_profiler_dir":"/tmp/profile"}"#,
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert_eq!(args.runtime.profiler().as_deref(), Some("torch"));
+
+ let config = args.to_managed_engine_config(5555);
+ let profiler_flag_index = config
+ .python_args
+ .iter()
+ .position(|arg| arg == "--profiler-config")
+ .expect("profiler config flag");
+ let profiler_config: serde_json::Value =
+ serde_json::from_str(&config.python_args[profiler_flag_index + 1])
+ .expect("profiler config json");
+ assert_eq!(
+ profiler_config,
+ serde_json::json!({
+ "profiler": "torch",
+ "torch_profiler_dir": "/tmp/profile",
+ })
+ );
+}
+
+#[test]
+fn serve_args_forward_max_logprobs_to_frontend_and_managed_engine() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--max-logprobs",
+ "-1",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert_eq!(args.runtime.max_logprobs, Some(-1));
+
+ let frontend_config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string());
+ assert_eq!(frontend_config.max_logprobs, Some(-1));
+
+ let engine_config = args.to_managed_engine_config(5555);
+ expect![[r#"
+ [
+ "--max-logprobs",
+ "-1",
+ "--reasoning-parser",
+ "qwen3",
+ ]
+ "#]]
+ .assert_debug_eq(&engine_config.python_args);
+}
+
+#[test]
+fn serve_args_resolve_auto_reasoning_parser_for_managed_engine() {
+ let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B"]).unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert_eq!(args.runtime.reasoning_parser, ParserSelection::Auto);
+
+ let config = args.to_managed_engine_config(5555);
+ expect![[r#"
+ [
+ "--reasoning-parser",
+ "qwen3",
+ ]
+ "#]]
+ .assert_debug_eq(&config.python_args);
+}
+
+#[test]
+fn serve_args_forward_explicit_reasoning_parser_to_managed_engine() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Unknown/Model",
+ "--reasoning-parser",
+ "deepseek_r1",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+
+ let config = args.to_managed_engine_config(5555);
+ expect![[r#"
+ [
+ "--reasoning-parser",
+ "deepseek_r1",
+ ]
+ "#]]
+ .assert_debug_eq(&config.python_args);
+}
+
+#[test]
+fn serve_args_do_not_forward_disabled_reasoning_parser_to_managed_engine() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--reasoning-parser",
+ "none",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+
+ let config = args.to_managed_engine_config(5555);
+ assert!(config.python_args.is_empty());
+}
+
+#[test]
+fn serve_args_forward_reasoning_parser_even_with_passthrough_reasoning_parser() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--",
+ "--reasoning-parser",
+ "deepseek_r1",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+
+ let config = args.to_managed_engine_config(5555);
+ expect![[r#"
+ [
+ "--reasoning-parser",
+ "deepseek_r1",
+ "--reasoning-parser",
+ "qwen3",
+ ]
+ "#]]
+ .assert_debug_eq(&config.python_args);
+}
+
+#[test]
+fn serve_args_auto_forward_python_multi_char_alias_without_separator() {
+ let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "-tp", "2"]).unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "--tensor-parallel-size",
+ "2",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_accept_explicit_deepseek_v32_renderer() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--tokenizer-mode",
+ "deepseek_v32",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert_eq!(args.runtime.renderer, RendererSelection::DeepSeekV32);
+}
+
+#[test]
+fn serve_passes_enable_request_id_headers_into_config() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--enable-request-id-headers",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string());
+ assert!(config.api_server_options.enable_request_id_headers);
+}
+
+#[test]
+fn serve_passes_enable_prompt_tokens_details_into_config() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--enable-prompt-tokens-details",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string());
+ assert!(config.api_server_options.enable_prompt_tokens_details);
+}
+
+#[test]
+fn serve_passes_tls_into_config() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--ssl-certfile",
+ "/tmp/cert.pem",
+ "--ssl-keyfile",
+ "/tmp/key.pem",
+ "--ssl-ca-certs",
+ "/tmp/ca.pem",
+ "--ssl-cert-reqs",
+ "2",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string());
+ let tls = config.tls.expect("tls configured");
+ assert_eq!(tls.cert_file.as_deref(), Some("/tmp/cert.pem"));
+ assert_eq!(tls.key_file.as_deref(), Some("/tmp/key.pem"));
+ assert_eq!(tls.ca_certs.as_deref(), Some("/tmp/ca.pem"));
+ assert_eq!(tls.cert_reqs, 2);
+}
+
+#[test]
+fn serve_without_ssl_flags_has_no_tls() {
+ let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B"]).unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string());
+ assert!(config.tls.is_none());
+}
+
+#[test]
+fn serve_ssl_keyfile_without_certfile_fails_validation() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--ssl-keyfile",
+ "/tmp/key.pem",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string());
+ // TLS is requested (a key was given) but there is no certificate, so
+ // validation fails loud rather than silently serving plaintext.
+ assert_eq!(config.tls.as_ref().expect("tls requested").cert_file, None);
+ let err = config.validate().unwrap_err().to_string();
+ assert!(err.contains("--ssl-certfile is required"), "{err}");
+}
+
+#[test]
+fn serve_mtls_without_ca_certs_fails_validation() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--ssl-certfile",
+ "/tmp/cert.pem",
+ "--ssl-cert-reqs",
+ "2",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string());
+ // Client-cert verification without a CA bundle has nothing to verify
+ // against, so it fails loud at startup.
+ let err = config.validate().unwrap_err().to_string();
+ assert!(err.contains("--ssl-ca-certs is required"), "{err}");
+}
+
+#[test]
+fn frontend_args_json_passes_tls_into_config() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","ssl_certfile":"/tmp/cert.pem","ssl_keyfile":"/tmp/key.pem"}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ let config = args.into_config();
+ let tls = config.tls.expect("tls configured");
+ assert_eq!(tls.cert_file.as_deref(), Some("/tmp/cert.pem"));
+ assert_eq!(tls.key_file.as_deref(), Some("/tmp/key.pem"));
+}
+
+#[test]
+fn frontend_args_json_rejects_out_of_range_cert_reqs() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","ssl_certfile":"/tmp/cert.pem","ssl_cert_reqs":5}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ // The JSON path bypasses clap's range check, so validate() is the only guard.
+ let config = args.into_config();
+ let err = config.validate().unwrap_err().to_string();
+ assert!(err.contains("--ssl-cert-reqs"), "{err}");
+}
+
+#[test]
+fn frontend_args_json_passes_enable_request_id_headers_into_config() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","enable_request_id_headers":true}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ let config = args.into_config();
+ assert!(config.api_server_options.enable_request_id_headers);
+}
+
+#[test]
+fn serve_passes_api_keys_into_config() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--api-key",
+ "secret-a",
+ "--api-key",
+ "secret-b",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://127.0.0.1:62100".to_string());
+ assert_eq!(config.api_keys, vec!["secret-a", "secret-b"]);
+ let debug = format!("{config:#?}");
+ assert!(debug.contains("api_keys: [; 2]"));
+ assert!(!debug.contains("secret-a"));
+ assert!(!debug.contains("secret-b"));
+}
+
+#[test]
+fn frontend_args_json_accepts_api_key_string() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","api_key":"secret"}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ let config = args.into_config();
+ assert_eq!(config.api_keys, vec!["secret"]);
+}
+
+#[test]
+fn frontend_args_json_accepts_api_key_list() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","api_key":["secret-a","secret-b"]}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ let config = args.into_config();
+ assert_eq!(config.api_keys, vec!["secret-a", "secret-b"]);
+}
+
+#[test]
+fn serve_args_reject_unknown_renderer_value() {
+ let error = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--tokenizer-mode",
+ "definitely_missing",
+ ])
+ .unwrap_err();
+
+ expect![[r#"
+ error: invalid value 'definitely_missing' for '--tokenizer-mode ': unknown renderer `definitely_missing` (expected one of: auto, hf, deepseek_v32, deepseek_v4, harmony)
+
+ For more information, try '--help'.
+ "#]]
+ .assert_eq(&error.to_string());
+}
+
+#[test]
+fn serve_args_reject_unsupported_flag_arg() {
+ let error = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--root-path",
+ "/prefix",
+ ])
+ .unwrap_err();
+
+ expect![[r#"
+ error: invalid value '/prefix' for '--root-path ': argument is not implemented in Rust frontend yet
+
+ Remove this unsupported argument to continue.
+
+ Alternatively, if you intend to pass it only to the Python engine, put it after `--` (e.g., `-- `).
+ This may lead to unexpected behavior as the Rust frontend will completely ignore that argument.
+
+ For more information, try '--help'.
+ "#]].assert_eq(&error.to_string());
+}
+
+#[test]
+fn serve_args_reject_unsupported_no_flag_alias() {
+ let error = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--no-enable-log-deltas",
+ ])
+ .unwrap_err();
+
+ expect![[r#"
+ error: invalid value 'true' for '--enable-log-deltas []': argument is not implemented in Rust frontend yet
+
+ Remove this unsupported argument to continue.
+
+ Alternatively, if you intend to pass it only to the Python engine, put it after `--` (e.g., `-- `).
+ This may lead to unexpected behavior as the Rust frontend will completely ignore that argument.
+
+ For more information, try '--help'.
+ "#]]
+ .assert_eq(&error.to_string());
+}
+
+#[test]
+fn frontend_args_accept_json() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--coordinator-address",
+ "tcp://127.0.0.1:7000",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","engine_count":2}"#,
+ ])
+ .unwrap();
+
+ expect![[r#"
+ Cli {
+ command: Frontend(
+ FrontendArgs {
+ listen_fd: 3,
+ input_address: "ipc:///tmp/input.sock",
+ output_address: "ipc:///tmp/output.sock",
+ coordinator_address: Some(
+ "tcp://127.0.0.1:7000",
+ ),
+ engine_start_index: 0,
+ engine_count: 1,
+ runtime: SharedRuntimeArgs {
+ model: "Qwen/Qwen3-0.6B",
+ engine_ready_timeout_secs: 600,
+ tool_call_parser: None,
+ reasoning_parser: None,
+ renderer: Auto,
+ language_model_only: false,
+ max_model_len: None,
+ max_logprobs: None,
+ grpc_port: None,
+ shutdown_timeout: 0,
+ http_timeout_keep_alive: None,
+ chat_template: None,
+ default_chat_template_kwargs: None,
+ chat_template_content_format: Auto,
+ enable_log_requests: false,
+ enable_prompt_tokens_details: false,
+ enable_request_id_headers: false,
+ disable_log_stats: false,
+ served_model_name: [],
+ allowed_origins: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allowed_methods: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allowed_headers: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allow_credentials: false,
+ ssl_keyfile: None,
+ ssl_certfile: None,
+ ssl_ca_certs: None,
+ ssl_cert_reqs: 0,
+ ssl_ciphers: None,
+ profiler_config: None,
+ },
+ },
+ ),
+ }
+ "#]]
+ .assert_debug_eq(&cli);
+}
+
+#[test]
+fn frontend_args_json_applies_defaults() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ assert_eq!(args.runtime.model, "Qwen/Qwen3-0.6B");
+ assert_eq!(args.runtime.engine_ready_timeout_secs, 600);
+ assert_eq!(args.runtime.tool_call_parser, ParserSelection::None);
+ assert_eq!(args.runtime.reasoning_parser, ParserSelection::None);
+ assert_eq!(args.runtime.renderer, RendererSelection::Auto);
+ assert_eq!(args.runtime.max_model_len, None);
+ assert_eq!(args.runtime.max_logprobs, None);
+ assert_eq!(args.runtime.shutdown_timeout, 0);
+}
+
+#[test]
+fn frontend_args_json_accepts_supported_non_default_fields() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","engine_ready_timeout_secs":42,"tool_call_parser":"hermes","reasoning_parser":"qwen3_thinking","tokenizer_mode":"deepseek_v32","language_model_only":true,"max_model_len":8192,"max_logprobs":-1,"shutdown_timeout":3}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ assert_eq!(args.runtime.engine_ready_timeout_secs, 42);
+ assert_eq!(
+ args.runtime.tool_call_parser,
+ ParserSelection::Explicit("hermes".to_string())
+ );
+ assert_eq!(
+ args.runtime.reasoning_parser,
+ ParserSelection::Explicit("qwen3_thinking".to_string())
+ );
+ assert_eq!(args.runtime.renderer, RendererSelection::DeepSeekV32);
+ assert!(args.runtime.language_model_only);
+ assert_eq!(args.runtime.max_model_len, Some(8192));
+ assert_eq!(args.runtime.max_logprobs, Some(-1));
+ assert_eq!(args.runtime.shutdown_timeout, 3);
+}
+
+#[test]
+fn serve_args_accept_none_reasoning_parser() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--reasoning-parser",
+ "none",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert_eq!(args.runtime.reasoning_parser, ParserSelection::None);
+ assert_eq!(args.runtime.tool_call_parser, ParserSelection::Auto);
+}
+
+#[test]
+fn frontend_args_json_ignores_unknown_fields() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","uds":"/tmp/vllm.sock","nested_unknown":{"x":1}}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ assert_eq!(args.runtime.model, "Qwen/Qwen3-0.6B");
+}
+
+#[test]
+fn frontend_args_json_sets_prompt_tokens_details_flag() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","api_server_count":2,"enable_prompt_tokens_details":true}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ assert_eq!(args.runtime.model, "Qwen/Qwen3-0.6B");
+ assert!(args.runtime.enable_prompt_tokens_details);
+}
+
+#[test]
+fn serve_args_parse_cors_flags() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--allowed-origins",
+ r#"["http://a.com","http://b.com"]"#,
+ "--allowed-methods",
+ r#"["GET","POST"]"#,
+ "--allow-credentials",
+ ])
+ .unwrap();
+
+ let Command::Serve(serve) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert_eq!(
+ serve.runtime.allowed_origins.0,
+ ["http://a.com", "http://b.com"]
+ );
+ assert_eq!(serve.runtime.allowed_methods.0, ["GET", "POST"]);
+ assert!(serve.runtime.allow_credentials);
+ // Unspecified lists keep the permissive default.
+ assert_eq!(serve.runtime.allowed_headers.0, ["*"]);
+}
+
+#[test]
+fn serve_args_cors_defaults_are_permissive() {
+ let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B"]).unwrap();
+
+ let Command::Serve(serve) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert_eq!(serve.runtime.allowed_origins.0, ["*"]);
+ assert_eq!(serve.runtime.allowed_methods.0, ["*"]);
+ assert_eq!(serve.runtime.allowed_headers.0, ["*"]);
+ assert!(!serve.runtime.allow_credentials);
+}
+
+#[test]
+fn frontend_args_json_parses_cors_fields() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","allowed_origins":["http://a.com"],"allow_credentials":true}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ assert_eq!(args.runtime.allowed_origins.0, ["http://a.com"]);
+ assert!(args.runtime.allow_credentials);
+ // Unspecified lists fall back to the permissive default via serde.
+ assert_eq!(args.runtime.allowed_methods.0, ["*"]);
+}
+
+#[test]
+fn frontend_args_json_rejects_unsupported_fields() {
+ let error = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","root_path":"/prefix"}"#,
+ ])
+ .unwrap_err();
+
+ expect![[r#"
+ error: invalid value '{"model_tag":"Qwen/Qwen3-0.6B","root_path":"/prefix"}' for '--args-json ':
+ The following arguments are not implemented in Rust frontend yet:
+ - root_path
+
+ Remove these arguments to continue.
+
+ For more information, try '--help'.
+ "#]].assert_eq(&error.to_string());
+}
+
+#[test]
+fn frontend_args_json_aggregates_multiple_unsupported_fields() {
+ let error = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","response_role":"assistant","root_path":"/prefix"}"#,
+ ])
+ .unwrap_err();
+
+ let actual = error.to_string().replace(": \n", ":\n");
+ expect![[r#"
+ error: invalid value '{"model_tag":"Qwen/Qwen3-0.6B","response_role":"assistant","root_path":"/prefix"}' for '--args-json ':
+ The following arguments are not implemented in Rust frontend yet:
+ - response_role
+ - root_path
+
+ Remove these arguments to continue.
+
+ For more information, try '--help'.
+ "#]].assert_eq(&actual);
+}
+
+#[test]
+fn frontend_args_json_rejects_malformed_json() {
+ let error = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B""#,
+ ])
+ .unwrap_err();
+
+ expect![[r#"
+ error: invalid value '{"model_tag":"Qwen/Qwen3-0.6B"' for '--args-json ': invalid JSON arguments: EOF while parsing an object at line 1 column 30
+
+ For more information, try '--help'.
+ "#]].assert_eq(&error.to_string());
+}
+
+#[test]
+fn serve_args_reject_flags_before_model() {
+ let error = Cli::try_parse_from(["vllm-rs", "serve", "--python", "python3", "Qwen/Qwen3-0.6B"])
+ .unwrap_err();
+
+ expect![[r#"
+ error: the model must appear immediately after the command
+
+ Usage: vllm-rs serve [OPTIONS] [-- ...]
+
+ For more information, try '--help'.
+ "#]]
+ .assert_eq(&error.to_string());
+}
+
+#[test]
+fn serve_args_accept_headless_mode() {
+ let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "--headless"]).unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert!(args.headless);
+}
+
+#[test]
+fn serve_args_keep_python_passthrough_flags_after_separator() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--python",
+ "python3",
+ "--",
+ "--tensor-parallel-size",
+ "2",
+ "--dtype",
+ "float16",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "--tensor-parallel-size",
+ "2",
+ "--dtype",
+ "float16",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_keep_python_multi_char_alias_after_separator() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--python",
+ "python3",
+ "--",
+ "-tp",
+ "2",
+ "--dtype",
+ "float16",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "-tp",
+ "2",
+ "--dtype",
+ "float16",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_keep_frontend_arg_after_separator() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--",
+ "--uds",
+ "/tmp/vllm.sock",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "--uds",
+ "/tmp/vllm.sock",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_keep_python_multi_char_engine_aliases_after_separator() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--",
+ "-dpr",
+ "1",
+ "-dpl",
+ "2",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "-dpr",
+ "1",
+ "-dpl",
+ "2",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_auto_forward_unknown_flags_without_separator() {
+ let cli = Cli::try_parse_from(["vllm-rs", "serve", "Qwen/Qwen3-0.6B", "--foo", "bar"]).unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "--foo",
+ "bar",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_auto_forward_negative_value_without_separator() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--num-gpu-blocks-override",
+ "-1",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ expect![[r#"
+ [
+ "--num-gpu-blocks-override",
+ "-1",
+ ]
+ "#]]
+ .assert_debug_eq(&args.managed_engine.python_args);
+}
+
+#[test]
+fn serve_args_accept_handshake_aliases() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--python",
+ "python3",
+ "--handshake-host",
+ "10.99.48.128",
+ "--handshake-port",
+ "13345",
+ "--data-parallel-size",
+ "4",
+ ])
+ .unwrap();
+
+ expect![[r#"
+ Cli {
+ command: Serve(
+ ServeArgs {
+ headless: false,
+ host: "127.0.0.1",
+ port: 8000,
+ uds: None,
+ runtime: SharedRuntimeArgs {
+ model: "Qwen/Qwen3-0.6B",
+ engine_ready_timeout_secs: 600,
+ tool_call_parser: Auto,
+ reasoning_parser: Auto,
+ renderer: Auto,
+ language_model_only: false,
+ max_model_len: None,
+ max_logprobs: None,
+ grpc_port: None,
+ shutdown_timeout: 0,
+ http_timeout_keep_alive: None,
+ chat_template: None,
+ default_chat_template_kwargs: None,
+ chat_template_content_format: Auto,
+ enable_log_requests: false,
+ enable_prompt_tokens_details: false,
+ enable_request_id_headers: false,
+ disable_log_stats: false,
+ served_model_name: [],
+ allowed_origins: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allowed_methods: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allowed_headers: JsonStringList(
+ [
+ "*",
+ ],
+ ),
+ allow_credentials: false,
+ ssl_keyfile: None,
+ ssl_certfile: None,
+ ssl_ca_certs: None,
+ ssl_cert_reqs: 0,
+ ssl_ciphers: None,
+ profiler_config: None,
+ },
+ managed_engine: ManagedEngineArgs {
+ python: "python3",
+ handshake_host: "10.99.48.128",
+ handshake_port: Some(
+ 13345,
+ ),
+ data_parallel_size: 4,
+ data_parallel_size_local: None,
+ python_args: [],
+ },
+ },
+ ),
+ }
+ "#]]
+ .assert_debug_eq(&cli);
+}
+
+#[test]
+fn serve_args_accept_data_parallel_primary_flags() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--data-parallel-address",
+ "10.99.48.128",
+ "--data-parallel-rpc-port",
+ "13345",
+ "--data-parallel-size",
+ "4",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ assert!(!args.headless);
+ assert_eq!(args.managed_engine.handshake_host, "10.99.48.128");
+ assert_eq!(args.managed_engine.handshake_port, Some(13345));
+ assert_eq!(args.managed_engine.data_parallel_size, 4);
+}
+
+#[test]
+fn serve_frontend_config_uses_dp_address_as_advertised_host() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--handshake-host",
+ "10.99.48.128",
+ "--data-parallel-size",
+ "4",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://10.99.48.128:29550".to_string());
+
+ let TransportMode::HandshakeOwner {
+ handshake_address,
+ advertised_host,
+ engine_count,
+ ready_timeout,
+ local_input_address,
+ local_output_address,
+ } = &config.transport_mode
+ else {
+ panic!("expected handshake-owned transport");
+ };
+
+ assert_eq!(handshake_address, "tcp://10.99.48.128:29550");
+ assert_eq!(advertised_host, "10.99.48.128");
+ assert_eq!(*engine_count, 4);
+ assert!(
+ local_input_address
+ .as_deref()
+ .is_some_and(|address| address.starts_with("ipc://"))
+ );
+ assert!(
+ local_output_address
+ .as_deref()
+ .is_some_and(|address| address.starts_with("ipc://"))
+ );
+ assert_ne!(local_input_address, local_output_address);
+
+ expect![[r#"
+ Config {
+ transport_mode: HandshakeOwner {
+ handshake_address: "tcp://10.99.48.128:29550",
+ advertised_host: "10.99.48.128",
+ engine_count: 4,
+ ready_timeout: 600s,
+ local_input_address: Some(
+ "",
+ ),
+ local_output_address: Some(
+ "",
+ ),
+ },
+ coordinator_mode: MaybeInProc,
+ model: "Qwen/Qwen3-0.6B",
+ served_model_name: [],
+ listener_mode: BindTcp {
+ host: "127.0.0.1",
+ port: 8000,
+ },
+ tool_call_parser: Auto,
+ reasoning_parser: Auto,
+ renderer: Auto,
+ language_model_only: false,
+ chat_template: None,
+ default_chat_template_kwargs: None,
+ chat_template_content_format: Auto,
+ max_logprobs: None,
+ api_server_options: ApiServerOptions {
+ enable_log_requests: false,
+ enable_prompt_tokens_details: false,
+ enable_request_id_headers: false,
+ },
+ cors: CorsConfig {
+ allow_origins: [
+ "*",
+ ],
+ allow_methods: [
+ "*",
+ ],
+ allow_headers: [
+ "*",
+ ],
+ allow_credentials: false,
+ },
+ tls: None,
+ api_keys: [],
+ disable_log_stats: false,
+ grpc_port: None,
+ shutdown_timeout: 0ns,
+ keep_alive_timeout: 5s,
+ profiler: None,
+ }
+ "#]]
+ .assert_debug_eq(&Config {
+ transport_mode: TransportMode::HandshakeOwner {
+ handshake_address: handshake_address.clone(),
+ advertised_host: advertised_host.clone(),
+ engine_count: *engine_count,
+ ready_timeout: *ready_timeout,
+ local_input_address: Some("".to_string()),
+ local_output_address: Some("".to_string()),
+ },
+ ..config.clone()
+ });
+}
+
+#[test]
+fn serve_frontend_config_keeps_tcp_transport_for_non_local_only_topology() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--data-parallel-address",
+ "10.99.48.128",
+ "--data-parallel-size",
+ "4",
+ "--data-parallel-size-local",
+ "2",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://10.99.48.128:29550".to_string());
+
+ expect![[r#"
+ Config {
+ transport_mode: HandshakeOwner {
+ handshake_address: "tcp://10.99.48.128:29550",
+ advertised_host: "10.99.48.128",
+ engine_count: 4,
+ ready_timeout: 600s,
+ local_input_address: None,
+ local_output_address: None,
+ },
+ coordinator_mode: MaybeInProc,
+ model: "Qwen/Qwen3-0.6B",
+ served_model_name: [],
+ listener_mode: BindTcp {
+ host: "127.0.0.1",
+ port: 8000,
+ },
+ tool_call_parser: Auto,
+ reasoning_parser: Auto,
+ renderer: Auto,
+ language_model_only: false,
+ chat_template: None,
+ default_chat_template_kwargs: None,
+ chat_template_content_format: Auto,
+ max_logprobs: None,
+ api_server_options: ApiServerOptions {
+ enable_log_requests: false,
+ enable_prompt_tokens_details: false,
+ enable_request_id_headers: false,
+ },
+ cors: CorsConfig {
+ allow_origins: [
+ "*",
+ ],
+ allow_methods: [
+ "*",
+ ],
+ allow_headers: [
+ "*",
+ ],
+ allow_credentials: false,
+ },
+ tls: None,
+ api_keys: [],
+ disable_log_stats: false,
+ grpc_port: None,
+ shutdown_timeout: 0ns,
+ keep_alive_timeout: 5s,
+ profiler: None,
+ }
+ "#]]
+ .assert_debug_eq(&config);
+}
+
+#[test]
+fn frontend_args_reject_legacy_handshake_flags() {
+ let error = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#,
+ "--handshake-address",
+ "tcp://127.0.0.1:62100",
+ ])
+ .unwrap_err();
+
+ assert!(error.to_string().contains("--handshake-address"));
+}
+
+#[test]
+fn frontend_config_uses_external_coordinator_when_coordinator_address_is_present() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--coordinator-address",
+ "tcp://127.0.0.1:7000",
+ "--engine-start-index",
+ "3",
+ "--engine-count",
+ "1",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ let config = args.into_config();
+
+ expect![[r#"
+ Config {
+ transport_mode: Bootstrapped {
+ input_address: "ipc:///tmp/input.sock",
+ output_address: "ipc:///tmp/output.sock",
+ engine_start_index: 3,
+ engine_count: 1,
+ ready_timeout: 600s,
+ },
+ coordinator_mode: External {
+ address: "tcp://127.0.0.1:7000",
+ },
+ model: "Qwen/Qwen3-0.6B",
+ served_model_name: [],
+ listener_mode: InheritedFd {
+ fd: 3,
+ },
+ tool_call_parser: None,
+ reasoning_parser: None,
+ renderer: Auto,
+ language_model_only: false,
+ chat_template: None,
+ default_chat_template_kwargs: None,
+ chat_template_content_format: Auto,
+ max_logprobs: None,
+ api_server_options: ApiServerOptions {
+ enable_log_requests: false,
+ enable_prompt_tokens_details: false,
+ enable_request_id_headers: false,
+ },
+ cors: CorsConfig {
+ allow_origins: [
+ "*",
+ ],
+ allow_methods: [
+ "*",
+ ],
+ allow_headers: [
+ "*",
+ ],
+ allow_credentials: false,
+ },
+ tls: None,
+ api_keys: [],
+ disable_log_stats: false,
+ grpc_port: None,
+ shutdown_timeout: 0ns,
+ keep_alive_timeout: 5s,
+ profiler: None,
+ }
+ "#]]
+ .assert_debug_eq(&config);
+}
+
+#[test]
+fn serve_frontend_config_uses_unix_listener_when_uds_is_present() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "serve",
+ "Qwen/Qwen3-0.6B",
+ "--uds",
+ "/tmp/vllm.sock",
+ ])
+ .unwrap();
+
+ let Command::Serve(args) = cli.command else {
+ panic!("expected serve args");
+ };
+ let config = args.to_frontend_config("tcp://127.0.0.1:29550".to_string());
+
+ assert_eq!(
+ config.listener_mode,
+ HttpListenerMode::BindUnix {
+ path: "/tmp/vllm.sock".to_string(),
+ }
+ );
+}
+
+#[test]
+fn frontend_args_json_enables_profiling_when_profiler_config_set() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","profiler_config":{"profiler":"torch","torch_profiler_dir":"/tmp/profile"}}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ assert_eq!(args.runtime.profiler().as_deref(), Some("torch"));
+ let config = args.into_config();
+ assert_eq!(config.profiler.as_deref(), Some("torch"));
+}
+
+#[test]
+fn frontend_args_json_disables_profiling_when_profiler_config_absent() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B"}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ assert_eq!(args.runtime.profiler(), None);
+ let config = args.into_config();
+ assert_eq!(config.profiler, None);
+}
+
+#[test]
+fn frontend_args_json_disables_profiling_when_profiler_type_is_null() {
+ let cli = Cli::try_parse_from([
+ "vllm-rs",
+ "frontend",
+ "--listen-fd",
+ "3",
+ "--input-address",
+ "ipc:///tmp/input.sock",
+ "--output-address",
+ "ipc:///tmp/output.sock",
+ "--args-json",
+ r#"{"model_tag":"Qwen/Qwen3-0.6B","profiler_config":{"profiler":null}}"#,
+ ])
+ .unwrap();
+
+ let Command::Frontend(args) = cli.command else {
+ panic!("expected frontend args");
+ };
+ assert_eq!(args.runtime.profiler(), None);
+ let config = args.into_config();
+ assert_eq!(config.profiler, None);
+}
diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/cli/unsupported.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/cli/unsupported.rs
new file mode 100644
index 0000000000000000000000000000000000000000..4f05aefcf4b901b5bf6b1eed1c4083e7efb0b47e
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/cmd/src/cli/unsupported.rs
@@ -0,0 +1,578 @@
+#![allow(clippy::doc_lazy_continuation)]
+
+use std::fmt::Display;
+use std::str::FromStr;
+
+use clap::Args;
+use clap::builder::{TypedValueParser, ValueParserFactory};
+use itertools::Itertools;
+use serde::{Deserialize, Deserializer, Serialize};
+
+/// Marker type for frontend-owned `serve` arguments that `vllm-rs` recognizes
+/// but does not support yet.
+///
+/// When passed as JSON args, it can be deserialized from any value, and
+/// serializes back to the original value.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(transparent)]
+pub struct Unsupported(pub serde_json::Value);
+
+impl FromStr for Unsupported {
+ type Err = String;
+
+ fn from_str(_s: &str) -> Result {
+ Err("argument is not implemented in Rust frontend yet
+
+Remove this unsupported argument to continue.
+
+Alternatively, if you intend to pass it only to the Python engine, put it after `--` (e.g., `-- `).
+This may lead to unexpected behavior as the Rust frontend will completely ignore that argument."
+ .to_string())
+ }
+}
+
+/// Marker type for no-op arguments that are accepted by the Rust frontend but
+/// have no effect.
+///
+/// When passed as JSON args, it can be deserialized from any value, but always
+/// serializes back to `null`.
+#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
+pub struct Noop;
+
+impl<'de> Deserialize<'de> for Noop {
+ fn deserialize(_deserializer: D) -> Result
+ where
+ D: Deserializer<'de>,
+ {
+ Ok(Noop)
+ }
+}
+
+impl ValueParserFactory for Noop {
+ type Parser = NoopValueParser;
+
+ fn value_parser() -> Self::Parser {
+ NoopValueParser
+ }
+}
+
+#[derive(Copy, Clone, Debug)]
+pub struct NoopValueParser;
+
+#[track_caller]
+fn noop_warn(arg: impl Display) {
+ tracing::warn!("argument '{arg}' currently has no effect in Rust frontend, ignoring");
+}
+
+impl TypedValueParser for NoopValueParser {
+ type Value = Noop;
+
+ fn parse_ref(
+ &self,
+ _cmd: &clap::Command,
+ arg: Option<&clap::Arg>,
+ _value: &std::ffi::OsStr,
+ ) -> Result {
+ if let Some(arg) = arg {
+ noop_warn(arg);
+ }
+ Ok(Noop)
+ }
+}
+
+/// Frontend-owned Python `serve` arguments that `vllm-rs` recognizes but does
+/// not support yet.
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Clone, PartialEq, Eq, Default, Args, Serialize, Deserialize)]
+#[command(next_help_heading = "Options not implemented in Rust frontend yet")]
+pub struct UnsupportedArgs {
+ #[command(flatten)]
+ #[serde(default, flatten)]
+ top_level: TopLevelUnsupportedArgs,
+ #[command(flatten)]
+ #[serde(default, flatten)]
+ engine: EngineUnsupportedArgs,
+ #[command(flatten)]
+ #[serde(default, flatten)]
+ server: ServerUnsupportedArgs,
+}
+
+impl UnsupportedArgs {
+ /// Check whether any unsupported arguments are set, and if so, return an
+ /// error listing them. Also warn about any no-op arguments that are set
+ /// but will be ignored.
+ pub(crate) fn check(&self) -> Result<(), String> {
+ let value = serde_json::to_value(self).unwrap();
+ let map = value.as_object().unwrap();
+ let mut unsupported = Vec::new();
+
+ for (key, value) in map {
+ if value.is_null() {
+ noop_warn(key);
+ } else {
+ unsupported.push(key.as_str());
+ }
+ }
+
+ if !unsupported.is_empty() {
+ unsupported.sort_unstable();
+ let bullets = unsupported.into_iter().map(|key| format!("- {key}")).join("\n");
+ return Err(format!(
+ "
+The following arguments are not implemented in Rust frontend yet:
+{bullets}
+
+Remove these arguments to continue."
+ ));
+ }
+
+ Ok(())
+ }
+}
+
+/// Frontend-owned Python `vllm serve` top-level arguments that `vllm-rs`
+/// recognizes but does not support yet.
+///
+/// Source of truth in Python vLLM:
+/// - `vllm.entrypoints.openai.cli_args.make_arg_parser(...)`
+/// - `vllm.entrypoints.cli.serve.ServeSubcommand.subparser_init(...)`
+///
+/// These are not part of `EngineArgs`, `AsyncEngineArgs`, `BaseFrontendArgs`,
+/// or `FrontendArgs`. They live on the `serve` command itself and control
+/// managed-engine / multi-process orchestration rather than the shared frontend
+/// runtime config.
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Clone, PartialEq, Eq, Default, Args, Serialize, Deserialize)]
+pub struct TopLevelUnsupportedArgs {
+ /// How many API server processes to run. Defaults to data_parallel_size if
+ /// not specified.
+ #[arg(long, hide = true)]
+ pub api_server_count: Option,
+
+ /// Read CLI options from a config file. Must be a YAML with the following
+ /// options: https://docs.vllm.ai/en/latest/configuration/serve_args.html
+ #[arg(long)]
+ pub config: Option,
+
+ /// Launch a gRPC server instead of the HTTP OpenAI-compatible server.
+ /// Requires: pip install vllm[grpc].
+ #[arg(long, default_missing_value = "true", num_args = 0..=1)]
+ pub grpc: Option,
+}
+
+/// Frontend-owned Python engine arguments that `vllm-rs` recognizes but does
+/// not support yet.
+///
+/// Source of truth in Python vLLM:
+/// - `vllm.engine.arg_utils.EngineArgs.add_cli_args(...)`
+/// - `vllm.engine.arg_utils.AsyncEngineArgs.add_cli_args(...)`
+///
+/// These arguments are declared through the Python engine-args surface, but
+/// they are still frontend-owned: the API server / AsyncLLM layer reads them
+/// for tokenizer setup, request validation, routing, logging, and other
+/// frontend behavior, so Rust must recognize them rather than treating them as
+/// pure engine passthrough.
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Clone, PartialEq, Eq, Default, Args, Serialize, Deserialize)]
+pub struct EngineUnsupportedArgs {
+ /// Name or path of the Hugging Face tokenizer to use. If unspecified, model
+ /// name or path will be used.
+ #[arg(long)]
+ pub tokenizer: Option,
+
+ /// Name or path of the Hugging Face config to use. If unspecified, model
+ /// name or path will be used.
+ #[arg(long)]
+ pub hf_config_path: Option,
+
+ /// Allowing API requests to read local images or videos from directories
+ /// specified by the server file system. This is a security risk. Should
+ /// only be enabled in trusted environments.
+ #[arg(long)]
+ pub allowed_local_media_path: Option,
+
+ /// If set, only media URLs that belong to this domain can be used for
+ /// multi-modal inputs.
+ #[arg(long)]
+ pub allowed_media_domains: Option,
+
+ /// The specific revision to use for the tokenizer on the Hugging Face Hub.
+ /// It can be a branch name, a tag name, or a commit id. If unspecified,
+ /// will use the default version.
+ #[arg(long)]
+ pub tokenizer_revision: Option,
+
+ /// Skip initialization of tokenizer and detokenizer. Expects valid
+ /// `prompt_token_ids` and `None` for prompt from the input. The generated
+ /// output will contain token ids.
+ #[arg(
+ long,
+ visible_alias = "no-skip-tokenizer-init",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub skip_tokenizer_init: Option,
+
+ /// If `True`, enables passing text embeddings as inputs via the
+ /// `prompt_embeds` key.
+ ///
+ /// WARNING: The vLLM engine may crash if incorrect shape of embeddings is
+ /// passed. Only enable this flag for trusted users!
+ #[arg(
+ long,
+ visible_alias = "no-enable-prompt-embeds",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub enable_prompt_embeds: Option,
+
+ /// The token to use as HTTP bearer authorization for remote files. If
+ /// `True`, will use the token generated when running `hf auth login`
+ /// (stored in `~/.cache/huggingface/token`).
+ #[arg(long, default_missing_value = "true", num_args = 0..=1)]
+ pub hf_token: Option,
+
+ /// If a dictionary, contains arguments to be forwarded to the Hugging Face
+ /// config. If a callable, it is called to update the HuggingFace config.
+ #[arg(long)]
+ pub hf_overrides: Option,
+
+ /// The folder path to the generation config. Defaults to `"auto"`, the
+ /// generation config will be loaded from model path. If set to `"vllm"`, no
+ /// generation config is loaded, vLLM defaults will be used. If set to a
+ /// folder path, the generation config will be loaded from the specified
+ /// folder path. If `max_new_tokens` is specified in generation config,
+ /// then it sets a server-wide limit on the number of output tokens for
+ /// all requests.
+ #[arg(long)]
+ pub generation_config: Option,
+
+ /// IOProcessor plugin name to load at model startup
+ #[arg(long)]
+ pub io_processor_plugin: Option,
+
+ /// Path to a dynamically reasoning parser plugin that can be dynamically
+ /// loaded and registered.
+ #[arg(long)]
+ pub reasoning_parser_plugin: Option,
+
+ /// Rank of the data parallel group.
+ #[arg(long, env = "VLLM_DP_RANK")]
+ pub data_parallel_rank: Option,
+
+ /// Whether to use "hybrid" DP LB mode. Applies only to online serving
+ /// and when data_parallel_size > 0. Enables running an AsyncLLM
+ /// and API server on a "per-node" basis where vLLM load balances
+ /// between local data parallel ranks, but an external LB balances
+ /// between vLLM nodes/replicas. Set explicitly in conjunction with
+ /// --data-parallel-start-rank.
+ #[arg(
+ long,
+ visible_alias = "no-data-parallel-hybrid-lb",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub data_parallel_hybrid_lb: Option,
+
+ /// Whether to use "external" DP LB mode. Applies only to online serving
+ /// and when data_parallel_size > 0. This is useful for a "one-pod-per-rank"
+ /// wide-EP setup in Kubernetes. Set implicitly when --data-parallel-rank
+ /// is provided explicitly to vllm serve.
+ #[arg(
+ long,
+ visible_alias = "no-data-parallel-external-lb",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub data_parallel_external_lb: Option,
+
+ /// This feature is work in progress and no prefill optimization takes place
+ /// with this flag enabled currently.
+ #[arg(
+ long,
+ visible_alias = "no-kv-sharing-fast-prefill",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub kv_sharing_fast_prefill: Option,
+
+ /// The maximum number of input items and options allowed per
+ /// prompt for each modality.
+ #[arg(long)]
+ pub limit_mm_per_prompt: Option,
+
+ /// Additional args passed to process media inputs, keyed by modalities.
+ #[arg(long)]
+ pub media_io_kwargs: Option,
+
+ /// Arguments to be forwarded to the model's processor for multi-modal data,
+ /// e.g., image processor.
+ #[arg(long)]
+ pub mm_processor_kwargs: Option,
+
+ /// The size (in GiB) of the multi-modal processor cache.
+ #[arg(long)]
+ pub mm_processor_cache_gb: Option,
+
+ /// Type of cache to use for the multi-modal preprocessor/mapper.
+ #[arg(long)]
+ pub mm_processor_cache_type: Option,
+
+ /// Dictionary mapping specific modalities to LoRA model paths.
+ #[arg(long)]
+ pub default_mm_loras: Option,
+
+ /// Target URL to which OpenTelemetry traces will be sent.
+ #[arg(long)]
+ pub otlp_traces_endpoint: Option,
+
+ /// It makes sense to set this only if `--otlp-traces-endpoint` is set.
+ #[arg(long)]
+ pub collect_detailed_traces: Option,
+
+ /// The interval (or buffer size) for streaming in terms of token length.
+ #[arg(long)]
+ pub stream_interval: Option,
+
+ /// Structured outputs configuration.
+ #[arg(long)]
+ pub structured_outputs_config: Option,
+
+ /// Log aggregate rather than per-engine statistics when using data
+ /// parallelism.
+ #[arg(long, default_missing_value = "true", num_args = 0..=1)]
+ pub aggregate_engine_logging: Option,
+}
+
+/// Frontend-owned Python OpenAI server arguments that `vllm-rs` recognizes but
+/// does not support yet.
+///
+/// Source of truth in Python vLLM:
+/// - `vllm.entrypoints.openai.cli_args.BaseFrontendArgs`
+/// - `vllm.entrypoints.openai.cli_args.FrontendArgs`
+///
+/// These are not engine args. They belong to the Python OpenAI-compatible
+/// frontend / API-server layer itself, for example chat-template configuration,
+/// tool/frontend behavior, TLS / CORS / HTTP server settings, and other
+/// northbound server knobs.
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Clone, PartialEq, Eq, Default, Args, Serialize, Deserialize)]
+pub struct ServerUnsupportedArgs {
+ /// LoRA modules configurations in either 'name=path' format or JSON format
+ /// or JSON list format. Example (old format): `'name=path'` Example (new
+ /// format): `{"name": "name", "path": "lora_path",
+ /// "base_model_name": "id"}`
+ #[arg(long)]
+ pub lora_modules: Option,
+
+ /// Whether to trust the chat template provided in the request. If False,
+ /// the server will always use the chat template specified by
+ /// `--chat-template` or the ones from tokenizer.
+ #[arg(
+ long,
+ visible_alias = "no-trust-request-chat-template",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub trust_request_chat_template: Option,
+
+ /// The role name to return if `request.add_generation_prompt=true`.
+ #[arg(long)]
+ pub response_role: Option,
+
+ /// When `--max-logprobs` is specified, represents single tokens as
+ /// strings of the form 'token_id:{token_id}' so that tokens that are not
+ /// JSON-encodable can be identified.
+ #[arg(
+ long,
+ visible_alias = "no-return-tokens-as-token-ids",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub return_tokens_as_token_ids: Option,
+
+ /// Enable auto tool choice for supported models. Use `--tool-call-parser`
+ /// to specify which parser to use.
+ #[arg(
+ long,
+ visible_alias = "no-enable-auto-tool-choice",
+ default_missing_value = "true",
+ num_args = 0..=1,
+ hide = true
+ )]
+ pub enable_auto_tool_choice: Option,
+
+ /// If specified, exclude tool definitions in prompts when
+ /// tool_choice='none'.
+ #[arg(
+ long,
+ visible_alias = "no-exclude-tools-when-tool-choice-none",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub exclude_tools_when_tool_choice_none: Option,
+
+ /// Special the tool parser plugin write to parse the model-generated tool
+ /// into OpenAI API format, the name register in this plugin can be used in
+ /// `--tool-call-parser`.
+ #[arg(long)]
+ pub tool_parser_plugin: Option,
+
+ /// Comma-separated list of host:port pairs (IPv4, IPv6, or hostname).
+ /// Examples: 127.0.0.1:8000, [::1]:8000, localhost:1234. Or `demo` for
+ /// built-in demo tools (browser and Python code interpreter). WARNING:
+ /// The `demo` Python tool executes model-generated code in Docker without
+ /// network isolation by default. See the security guide for more
+ /// information.
+ #[arg(long)]
+ pub tool_server: Option,
+
+ /// Path to logging config JSON file for both vllm and uvicorn
+ #[arg(long, /* env = "VLLM_LOGGING_CONFIG_PATH" */)]
+ pub log_config_file: Option,
+
+ /// Max number of prompt characters or prompt ID numbers being printed in
+ /// log. The default of None means unlimited.
+ #[arg(long)]
+ pub max_log_len: Option,
+
+ /// If set to True, enable tracking server_load_metrics in the app state.
+ #[arg(
+ long,
+ visible_alias = "no-enable-server-load-tracking",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub enable_server_load_tracking: Option,
+
+ /// If set to True, including usage on every request.
+ #[arg(
+ long,
+ visible_alias = "no-enable-force-include-usage",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub enable_force_include_usage: Option,
+
+ /// Enable the `/tokenizer_info` endpoint. May expose chat
+ /// templates and other tokenizer configuration.
+ #[arg(
+ long,
+ visible_alias = "no-enable-tokenizer-info-endpoint",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub enable_tokenizer_info_endpoint: Option,
+
+ /// If set to True, log model outputs (generations).
+ /// Requires `--enable-log-requests`. As with `--enable-log-requests`,
+ /// information is only logged at INFO level at maximum.
+ #[arg(
+ long,
+ visible_alias = "no-enable-log-outputs",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub enable_log_outputs: Option,
+
+ /// If set to False, output deltas will not be logged. Relevant only if
+ /// --enable-log-outputs is set.
+ #[arg(
+ long,
+ visible_alias = "no-enable-log-deltas",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub enable_log_deltas: Option,
+
+ /// If set to True, log the stack trace of error responses
+ #[arg(
+ long,
+ // env = "VLLM_SERVER_DEV_MODE",
+ visible_alias = "no-log-error-stack",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub log_error_stack: Option,
+
+ /// If set to True, only enable the Tokens In<>Out endpoint.
+ /// This is intended for use in a Disaggregated Everything setup.
+ #[arg(
+ long,
+ visible_alias = "no-tokens-only",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub tokens_only: Option,
+
+ /// Log level for uvicorn.
+ #[arg(long)]
+ pub uvicorn_log_level: Option,
+
+ /// Disable uvicorn access log.
+ #[arg(
+ long,
+ visible_alias = "no-disable-uvicorn-access-log",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub disable_uvicorn_access_log: Option,
+
+ /// Comma-separated list of endpoint paths to exclude from uvicorn access
+ /// logs. This is useful to reduce log noise from high-frequency endpoints
+ /// like health checks. Example: "/health,/metrics,/ping".
+ /// When set, access logs for requests to these paths will be suppressed
+ /// while keeping logs for other endpoints.
+ #[arg(long)]
+ pub disable_access_log_for_endpoints: Option,
+
+ /// Refresh SSL Context when SSL certificate files change
+ #[arg(
+ long,
+ visible_alias = "no-enable-ssl-refresh",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub enable_ssl_refresh: Option,
+
+ /// FastAPI root_path when app is behind a path based routing proxy.
+ #[arg(long)]
+ pub root_path: Option,
+
+ /// Additional ASGI middleware to apply to the app. We accept multiple
+ /// --middleware arguments. The value should be an import path. If a
+ /// function is provided, vLLM will add it to the server using
+ /// `@app.middleware('http')`. If a class is provided, vLLM will
+ /// add it to the server using `app.add_middleware()`.
+ #[arg(long)]
+ pub middleware: Option,
+
+ /// Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint.
+ #[arg(
+ long,
+ visible_alias = "no-disable-fastapi-docs",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub disable_fastapi_docs: Option,
+
+ /// Maximum size (bytes) of an incomplete HTTP event (header or body) for
+ /// h11 parser. Helps mitigate header abuse. Default: 4194304 (4 MB).
+ #[arg(long)]
+ pub h11_max_incomplete_event_size: Option,
+
+ /// Maximum number of HTTP headers allowed in a request for h11 parser.
+ /// Helps mitigate header abuse. Default: 256.
+ #[arg(long)]
+ pub h11_max_header_count: Option,
+
+ /// Enable offline FastAPI documentation for air-gapped environments.
+ /// Uses vendored static assets bundled with vLLM.
+ #[arg(
+ long,
+ visible_alias = "no-enable-offline-docs",
+ default_missing_value = "true",
+ num_args = 0..=1
+ )]
+ pub enable_offline_docs: Option,
+}
diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/logging.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/logging.rs
new file mode 100644
index 0000000000000000000000000000000000000000..1338ba67f745fd00caa6c3c009d7339c3680c12c
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/cmd/src/logging.rs
@@ -0,0 +1,328 @@
+use std::{env, fmt, process};
+
+use time::UtcOffset;
+use time::macros::format_description;
+use tracing::level_filters::LevelFilter;
+use tracing::{Event, Level, Subscriber};
+use tracing_subscriber::Layer as _;
+use tracing_subscriber::filter::Targets;
+use tracing_subscriber::fmt::format::{FormatEvent, FormatFields, Writer};
+use tracing_subscriber::fmt::time::FormatTime;
+use tracing_subscriber::fmt::{FmtContext, FormattedFields};
+use tracing_subscriber::layer::SubscriberExt as _;
+use tracing_subscriber::registry::LookupSpan;
+use tracing_subscriber::util::SubscriberInitExt as _;
+
+const CYAN: &str = "\x1b[0;36m";
+const GREY: &str = "\x1b[90m";
+const GREEN: &str = "\x1b[32m";
+const YELLOW: &str = "\x1b[33m";
+const RED: &str = "\x1b[31m";
+const WHITE: &str = "\x1b[37m";
+const RESET: &str = "\x1b[0m";
+const VLLM_TIME_FORMAT: &[time::format_description::FormatItem<'static>] =
+ format_description!("[month]-[day] [hour]:[minute]:[second]");
+
+const PROCESS_LABEL: &str = "RustFrontend";
+
+/// Install the process-wide vLLM-style tracing subscriber for the CLI binary.
+pub(crate) fn init_tracing() {
+ let filter = build_targets_filter(
+ env::var("VLLM_LOGGING_LEVEL").ok().as_deref(),
+ env::var("RUST_LOG").ok().as_deref(),
+ );
+ let formatter = VllmEventFormatter::new();
+
+ let _ = tracing_subscriber::registry()
+ .with(tracing_subscriber::fmt::layer().event_format(formatter).with_filter(filter))
+ .try_init();
+}
+
+/// Build the CLI log filter by merging the vLLM-style default level with
+/// Rust-style target overrides.
+///
+/// Precedence:
+/// - Start from `VLLM_LOGGING_LEVEL` as the default level for all targets.
+/// - If `RUST_LOG` contains a global default level such as `warn`, it overrides
+/// `VLLM_LOGGING_LEVEL`.
+/// - Any explicit target directives in `RUST_LOG`, such as `hyper=info`, override whichever default
+/// level is active for those targets only.
+fn build_targets_filter(vllm_logging_level: Option<&str>, rust_log: Option<&str>) -> Targets {
+ let mut filter =
+ Targets::new().with_default(map_python_log_level(vllm_logging_level.unwrap_or("INFO")));
+
+ if let Some(rust_log) = rust_log
+ && !rust_log.is_empty()
+ {
+ let rust_log_targets: Targets = rust_log.parse().expect("failed to parse `RUST_LOG`");
+ if let Some(default_level) = rust_log_targets.default_level() {
+ filter = filter.with_default(default_level);
+ }
+ filter = filter.with_targets(rust_log_targets);
+ }
+
+ filter
+}
+
+#[derive(Debug, Clone, Copy)]
+struct VllmLocalTimer {
+ local_offset: UtcOffset,
+}
+
+impl Default for VllmLocalTimer {
+ fn default() -> Self {
+ let local_offset = UtcOffset::current_local_offset().unwrap_or(UtcOffset::UTC);
+ Self { local_offset }
+ }
+}
+
+impl FormatTime for VllmLocalTimer {
+ fn format_time(&self, w: &mut Writer<'_>) -> fmt::Result {
+ let now = time::OffsetDateTime::now_utc().to_offset(self.local_offset);
+ let formatted = now.format(VLLM_TIME_FORMAT).map_err(|_| fmt::Error)?;
+ w.write_str(&formatted)
+ }
+}
+
+#[derive(Debug, Clone)]
+struct VllmEventFormatter {
+ prefix: String,
+ timer: VllmLocalTimer,
+}
+
+impl VllmEventFormatter {
+ fn new() -> Self {
+ Self {
+ prefix: format!("({} pid={})", PROCESS_LABEL, process::id()),
+ timer: VllmLocalTimer::default(),
+ }
+ }
+
+ fn write_process_prefix(&self, writer: &mut Writer<'_>, ansi: bool) -> fmt::Result {
+ write_colored(writer, ansi, Some(CYAN), &self.prefix)?;
+ writer.write_char(' ')
+ }
+
+ fn write_level(&self, writer: &mut Writer<'_>, level: &Level, ansi: bool) -> fmt::Result {
+ let (text, color) = match *level {
+ Level::TRACE => ("TRACE", WHITE),
+ Level::DEBUG => ("DEBUG", WHITE),
+ Level::INFO => ("INFO", GREEN),
+ Level::WARN => ("WARNING", YELLOW),
+ Level::ERROR => ("ERROR", RED),
+ };
+ write_colored(writer, ansi, Some(color), text)
+ }
+
+ fn write_timestamp(&self, writer: &mut Writer<'_>, ansi: bool) -> fmt::Result {
+ if ansi {
+ writer.write_str(GREY)?;
+ }
+ if self.timer.format_time(writer).is_err() {
+ writer.write_str("")?;
+ }
+ if ansi {
+ writer.write_str(RESET)?;
+ }
+ Ok(())
+ }
+
+ fn write_location(
+ &self,
+ writer: &mut Writer<'_>,
+ file: Option<&str>,
+ line: Option,
+ full_path: bool,
+ ansi: bool,
+ ) -> fmt::Result {
+ let Some(file) = file else {
+ return Ok(());
+ };
+ let file = if full_path {
+ file
+ } else {
+ shorten_file_path(file)
+ };
+ if ansi {
+ writer.write_str(GREY)?;
+ }
+ match line {
+ Some(line) => write!(writer, "[{file}:{line}]")?,
+ None => write!(writer, "[{file}]")?,
+ }
+ if ansi {
+ writer.write_str(RESET)?;
+ }
+ Ok(())
+ }
+
+ fn write_scope
(&self, ctx: &FmtContext<'_, S, N>, writer: &mut Writer<'_>) -> fmt::Result
+ where
+ S: Subscriber + for<'lookup> LookupSpan<'lookup>,
+ N: for<'writer> FormatFields<'writer> + 'static,
+ {
+ let Some(scope) = ctx.event_scope() else {
+ return Ok(());
+ };
+
+ let mut seen = false;
+ for span in scope.from_root() {
+ if seen {
+ writer.write_str(":")?;
+ }
+ seen = true;
+ writer.write_str(span.metadata().name())?;
+
+ let ext = span.extensions();
+ if let Some(fields) = ext.get::>()
+ && !fields.is_empty()
+ {
+ write!(writer, "{{{fields}}}")?;
+ }
+ }
+
+ if seen {
+ writer.write_str(": ")?;
+ }
+
+ Ok(())
+ }
+}
+
+impl FormatEvent for VllmEventFormatter
+where
+ S: Subscriber + for<'lookup> LookupSpan<'lookup>,
+ N: for<'writer> FormatFields<'writer> + 'static,
+{
+ fn format_event(
+ &self,
+ ctx: &FmtContext<'_, S, N>,
+ mut writer: Writer<'_>,
+ event: &Event<'_>,
+ ) -> fmt::Result {
+ let meta = event.metadata();
+ let ansi = writer.has_ansi_escapes();
+
+ self.write_process_prefix(&mut writer, ansi)?;
+ self.write_level(&mut writer, meta.level(), ansi)?;
+ writer.write_char(' ')?;
+ self.write_timestamp(&mut writer, ansi)?;
+ writer.write_char(' ')?;
+ // Use the full file path only when DEBUG (or more verbose) is enabled anywhere,
+ // independent of the level of this particular event. Filenames alone are often
+ // ambiguous, but full paths are too noisy for normal INFO-level operation.
+ let full_path = LevelFilter::current() >= LevelFilter::DEBUG;
+ self.write_location(&mut writer, meta.file(), meta.line(), full_path, ansi)?;
+ writer.write_char(' ')?;
+ self.write_scope(ctx, &mut writer)?;
+ ctx.format_fields(writer.by_ref(), event)?;
+ writer.write_char('\n')
+ }
+}
+
+/// Shorten a source file path for log output while preserving enough context
+/// for common Rust entrypoint and module filenames.
+///
+/// - For `mod.rs`, keep the parent directory as `parent/mod.rs`.
+/// - For `src/lib.rs` and `src/main.rs`, keep one additional component as `crate/src/lib.rs` or
+/// `crate/src/main.rs` when available.
+/// - Other files are displayed as just the basename.
+fn shorten_file_path(file: &str) -> &str {
+ let mut parts = file.rsplit('/');
+ let name = parts.next().unwrap_or(file);
+ let parent = parts.next();
+ let grandparent = parts.next();
+
+ let Some(parent) = parent else {
+ return file;
+ };
+
+ if name == "mod.rs" {
+ return &file[file.len() - parent.len() - 1 - name.len()..];
+ }
+
+ if !matches!(name, "lib.rs" | "main.rs") || parent != "src" {
+ return name;
+ }
+ let Some(grandparent) = grandparent else {
+ return file;
+ };
+
+ &file[file.len() - grandparent.len() - 1 - parent.len() - 1 - name.len()..]
+}
+
+fn write_colored(
+ writer: &mut Writer<'_>,
+ ansi: bool,
+ color: Option<&str>,
+ text: &str,
+) -> fmt::Result {
+ if ansi {
+ if let Some(color) = color {
+ writer.write_str(color)?;
+ }
+ writer.write_str(text)?;
+ if color.is_some() {
+ writer.write_str(RESET)?;
+ }
+ return Ok(());
+ }
+
+ writer.write_str(text)
+}
+
+/// Map a Python logging level name to the corresponding Rust tracing level.
+fn map_python_log_level(level: &str) -> LevelFilter {
+ match level.to_ascii_uppercase().as_str() {
+ "CRITICAL" | "FATAL" => LevelFilter::ERROR,
+ "ERROR" => LevelFilter::ERROR,
+ "WARNING" | "WARN" => LevelFilter::WARN,
+ "INFO" => LevelFilter::INFO,
+ "DEBUG" => LevelFilter::DEBUG,
+ "NOTSET" => LevelFilter::TRACE,
+ _ => LevelFilter::INFO,
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn rust_log_target_overrides_are_merged_with_vllm_default_level() {
+ let filter = build_targets_filter(Some("DEBUG"), Some("hyper=warn,tower=error"));
+
+ assert_eq!(filter.to_string(), "tower=error,hyper=warn,debug");
+ }
+
+ #[test]
+ fn rust_log_default_level_overrides_vllm_default_level() {
+ let filter = build_targets_filter(Some("DEBUG"), Some("warn,hyper=info"));
+
+ assert_eq!(filter.to_string(), "hyper=info,warn");
+ }
+
+ #[test]
+ fn invalid_vllm_level_falls_back_to_info() {
+ let filter = build_targets_filter(Some("bogus"), None);
+
+ assert_eq!(filter.to_string(), "info");
+ }
+
+ #[test]
+ fn location_path_uses_filename_for_non_ambiguous_files() {
+ assert_eq!(shorten_file_path("src/cmd/src/logging.rs"), "logging.rs");
+ assert_eq!(shorten_file_path("src/chat/lib.rs"), "lib.rs");
+ assert_eq!(shorten_file_path("src/chat/main.rs"), "main.rs");
+ assert_eq!(shorten_file_path("src/chat/src/xmod.rs"), "xmod.rs");
+ }
+
+ #[test]
+ fn location_path_keeps_more_context_for_common_entrypoint_filenames() {
+ assert_eq!(shorten_file_path("src/lib.rs"), "src/lib.rs");
+ assert_eq!(shorten_file_path("src/chat/src/lib.rs"), "chat/src/lib.rs");
+ assert_eq!(shorten_file_path("src/cmd/src/main.rs"), "cmd/src/main.rs");
+ assert_eq!(shorten_file_path("mod.rs"), "mod.rs");
+ assert_eq!(shorten_file_path("src/chat/src/tool/mod.rs"), "tool/mod.rs");
+ }
+}
diff --git a/asian_ai/performance/vLLM/rust/src/cmd/src/main.rs b/asian_ai/performance/vLLM/rust/src/cmd/src/main.rs
new file mode 100644
index 0000000000000000000000000000000000000000..d24c10fead1c98242315b8f5ac8f5c1ce0401213
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/cmd/src/main.rs
@@ -0,0 +1,192 @@
+mod cli;
+mod logging;
+
+use std::env;
+use std::process::ExitStatus;
+
+use anyhow::{Context, Result, anyhow, bail};
+use tokio_util::sync::CancellationToken;
+use tracing::{info, warn};
+use vllm_managed_engine::ManagedEngineHandle;
+
+use crate::cli::{Cli, Command};
+
+#[global_allocator]
+static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
+
+const TOKIO_WORKER_THREADS_ENV: &str = "TOKIO_WORKER_THREADS";
+const DEFAULT_MAX_TOKIO_WORKER_THREADS: usize = 32;
+
+/// Cap the default number of Tokio worker threads if the user did not
+/// explicitly set `TOKIO_WORKER_THREADS` to avoid spawning too many threads on
+/// machines with a large number of CPUs, which may lead to excessive context
+/// switching and degraded performance.
+fn tokio_worker_threads() -> Option {
+ if env::var_os(TOKIO_WORKER_THREADS_ENV).is_some() {
+ return None;
+ }
+
+ std::thread::available_parallelism()
+ .map(|parallelism| {
+ let available = parallelism.get();
+ let worker_threads = available.min(DEFAULT_MAX_TOKIO_WORKER_THREADS);
+ if worker_threads < available {
+ info!(
+ available_parallelism = available,
+ capped_worker_threads = worker_threads,
+ "capping tokio worker threads, set {TOKIO_WORKER_THREADS_ENV} to override"
+ );
+ }
+ worker_threads
+ })
+ .ok()
+}
+
+/// Reason that caused a managed `serve` session to stop.
+#[derive(Debug)]
+enum ShutdownReason {
+ Signal,
+ Server(anyhow::Error),
+ EngineExited(ExitStatus),
+}
+
+/// Cancellation token tripped by Ctrl-C or SIGTERM.
+fn shutdown_signal() -> CancellationToken {
+ let token = CancellationToken::new();
+ let shutdown = token.clone();
+
+ tokio::spawn(async move {
+ let ctrl_c = async {
+ tokio::signal::ctrl_c().await.expect("failed to install Ctrl-C signal handler");
+ };
+
+ let sigterm = async {
+ tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
+ .expect("failed to install SIGTERM signal handler")
+ .recv()
+ .await;
+ };
+
+ tokio::select! {
+ _ = ctrl_c => info!("received shutdown signal (Ctrl-C), shutting down..."),
+ _ = sigterm => info!("received shutdown signal (SIGTERM), shutting down..."),
+ }
+
+ shutdown.cancel();
+ });
+
+ token
+}
+
+fn main() -> Result<()> {
+ logging::init_tracing();
+ let cli = Cli::parse();
+
+ let mut runtime = tokio::runtime::Builder::new_multi_thread();
+ runtime.enable_all();
+ if let Some(worker_threads) = tokio_worker_threads() {
+ runtime.worker_threads(worker_threads);
+ }
+
+ runtime
+ .build()
+ .context("failed to build Tokio runtime")?
+ .block_on(async_main(cli))
+}
+
+async fn async_main(cli: Cli) -> Result<()> {
+ match cli.command {
+ Command::Frontend(args) => vllm_server::serve(args.into_config(), shutdown_signal()).await,
+ Command::Serve(args) => {
+ let handshake_port = args.managed_engine.resolve_handshake_port()?;
+
+ if args.managed_engine.data_parallel_size_local == Some(0) {
+ if args.headless {
+ bail!("cannot combine `--headless` with `--data-parallel-size-local 0`");
+ }
+
+ let handshake_address = args.managed_engine.handshake_address(handshake_port);
+ info!(
+ %handshake_address,
+ engine_count = args.managed_engine.data_parallel_size,
+ "running Rust frontend without a managed local Python engine"
+ );
+ let config = args.to_frontend_config(handshake_address);
+ return vllm_server::serve(config, shutdown_signal()).await;
+ }
+
+ let shutdown_timeout = args.runtime.shutdown_timeout();
+ let engine_config = args.to_managed_engine_config(handshake_port);
+ let handshake_address = engine_config.handshake_address();
+
+ let engine = ManagedEngineHandle::spawn(engine_config)
+ .await
+ .context("failed to start managed Python headless engine")?;
+
+ let shutdown = shutdown_signal();
+
+ let mut serve_task = if args.headless {
+ info!("running managed Python headless engine without Rust frontend");
+ let shutdown = shutdown.clone();
+ tokio::spawn(async move {
+ shutdown.cancelled().await;
+ Ok(())
+ })
+ } else {
+ let config = args.to_frontend_config(handshake_address);
+ let shutdown = shutdown.clone();
+ tokio::spawn(async move {
+ let result = vllm_server::serve(config, shutdown).await;
+ if result.is_ok() {
+ info!("OpenAI server shut down gracefully");
+ }
+ result
+ })
+ };
+
+ let shutdown_reason = tokio::select! {
+ biased;
+
+ // Received shutdown signal via Ctrl-C or SIGTERM.
+ _ = shutdown.cancelled() => ShutdownReason::Signal,
+
+ // Engine process exited unexpectedly.
+ status = engine.wait_for_exit() => {
+ warn!(%status, "managed Python headless engine exited, shutting down...");
+ ShutdownReason::EngineExited(status)
+ }
+
+ // Serve task exited unexpectedly.
+ serve_result = &mut serve_task => {
+ let serve_result = serve_result.context("serve task join failed")?;
+ match serve_result {
+ Ok(()) => ShutdownReason::Server(anyhow!("OpenAI server shut down unexpectedly without error")),
+ Err(error) => ShutdownReason::Server(error),
+ }
+ }
+ };
+ // Regardless of the shutdown reason, broadcast shutdown signal here to ensure
+ // that all serving tasks are notified.
+ shutdown.cancel();
+
+ // Shutdown begins. Terminate the managed engine first.
+ engine.shutdown(shutdown_timeout).await?;
+ info!("managed engine shut down gracefully");
+ // Wait for the API server to shut down gracefully by draining in-flight
+ // requests.
+ if !matches!(shutdown_reason, ShutdownReason::Server(_)) {
+ serve_task.await.context("serve task join failed")??;
+ }
+
+ match shutdown_reason {
+ ShutdownReason::Signal => Ok(()),
+ ShutdownReason::Server(error) => {
+ Err(error.context("OpenAI server shut down unexpectedly"))
+ }
+ ShutdownReason::EngineExited(status) => Err(anyhow!(
+ "managed Python headless engine exited unexpectedly with status {status}"
+ )),
+ }
+ }
+ }
+}
diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/Cargo.toml b/asian_ai/performance/vLLM/rust/src/engine-core-client/Cargo.toml
new file mode 100644
index 0000000000000000000000000000000000000000..61f2e62b16775f05439f9fc9b82a5872bcaa8432
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/Cargo.toml
@@ -0,0 +1,49 @@
+[package]
+name = "vllm-engine-core-client"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+
+[features]
+test-util = ["dep:tempfile"]
+
+[dependencies]
+arc-swap.workspace = true
+bytemuck.workspace = true
+byteorder.workspace = true
+bytes.workspace = true
+easy-ext.workspace = true
+enum-as-inner.workspace = true
+futures.workspace = true
+half.workspace = true
+hex.workspace = true
+itertools.workspace = true
+parking_lot.workspace = true
+rmp-serde.workspace = true
+rmpv.workspace = true
+serde.workspace = true
+serde_default.workspace = true
+serde_json.workspace = true
+serde_repr.workspace = true
+serde_tuple.workspace = true
+serde_with.workspace = true
+task-local.workspace = true
+tempfile = { workspace = true, optional = true }
+thiserror.workspace = true
+thiserror-ext.workspace = true
+tokio.workspace = true
+tokio-util.workspace = true
+tracing.workspace = true
+vllm-metrics.workspace = true
+zeromq.workspace = true
+
+[dev-dependencies]
+anyhow.workspace = true
+clap.workspace = true
+expect-test.workspace = true
+hex.workspace = true
+tempfile.workspace = true
+tracing-subscriber.workspace = true
+
+[lints]
+workspace = true
diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/README.md b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..aa1fc1ca675e13e5c47866ffd14161ff552d82ba
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/README.md
@@ -0,0 +1,53 @@
+# Engine-Core Smoke Tests
+
+Start headless `vllm`:
+
+```bash
+source ../vllm/.venv/bin/activate
+HF_HUB_OFFLINE=1 \
+VLLM_LOGGING_LEVEL=DEBUG \
+VLLM_CPU_KVCACHE_SPACE=2 \
+VLLM_HOST_IP=127.0.0.1 \
+VLLM_LOOPBACK_IP=127.0.0.1 \
+python3 -m vllm.entrypoints.cli.main serve Qwen/Qwen3-0.6B \
+ --headless \
+ --enable-sleep-mode \
+ --data-parallel-address 127.0.0.1 \
+ --data-parallel-rpc-port 62100 \
+ --data-parallel-size-local 1 \
+ --max-model-len 512 \
+ --dtype float16
+```
+
+Run the Rust smoke test through the `vllm-engine-core-client` utility interface:
+
+```bash
+cargo run -p vllm-engine-core-client --example external_engine_utility_call -- \
+ --handshake-address tcp://127.0.0.1:62100 \
+ --host 127.0.0.1
+```
+
+If your current engine setup does not support sleep mode, skip the `sleep` / `wake_up` part of the
+smoke:
+
+```bash
+cargo run -p vllm-engine-core-client --example external_engine_utility_call -- \
+ --handshake-address tcp://127.0.0.1:62100 \
+ --host 127.0.0.1 \
+ --skip-sleep-wake
+```
+
+Run the Rust smoke test for sample logprobs decoding through the raw engine-core request path:
+
+```bash
+cargo run -p vllm-engine-core-client --example external_engine_logprobs -- \
+ --handshake-address tcp://127.0.0.1:62100 \
+ --host 127.0.0.1
+```
+
+This smoke requests a small generated-token `logprobs` payload plus prompt logprobs over a much
+longer prompt, so it exercises both the inline and aux-frame decode paths against a real engine.
+The Rust client decodes those payloads into semantic per-position records rather than exposing the
+raw ndarray/tensor wire shape.
+
+IMPORTANT: You must restart `vllm` each time you run the smoke test, as the vLLM engine cannot manage frontend closures and subsequent reconnects. In other words, do not reuse existing `vllm` instances, if any.
diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_logprobs.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_logprobs.rs
new file mode 100644
index 0000000000000000000000000000000000000000..7c9da201062c69c2319b4f9b60501a32c300d3f7
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_logprobs.rs
@@ -0,0 +1,190 @@
+use std::time::Duration;
+
+use anyhow::{Context, Result, bail};
+use clap::Parser;
+use futures::StreamExt as _;
+use tokio::time::timeout;
+use tracing_subscriber::EnvFilter;
+use vllm_engine_core_client::protocol::output::EngineCoreFinishReason;
+use vllm_engine_core_client::protocol::request::EngineCoreRequest;
+use vllm_engine_core_client::protocol::sampling::EngineCoreSamplingParams;
+use vllm_engine_core_client::{
+ EngineCoreClient, EngineCoreClientConfig, EngineCoreStreamOutput, TransportMode,
+};
+
+const BASE_PROMPT_TOKEN_IDS: &[u32] = &[20841, 448, 6896, 25, 23811];
+
+#[derive(Debug, Parser)]
+#[command(about = "Smoke-test engine-core sample logprobs against an external vLLM engine.")]
+struct Args {
+ #[arg(long)]
+ handshake_address: String,
+ #[arg(long, default_value_t = 1)]
+ engine_count: usize,
+ #[arg(long, default_value = "Qwen/Qwen3-0.6B")]
+ model: String,
+ #[arg(long, default_value = "127.0.0.1")]
+ host: String,
+ #[arg(long, default_value_t = 0)]
+ client_index: u32,
+ #[arg(long, default_value_t = 30)]
+ ready_timeout_secs: u64,
+ #[arg(long, default_value_t = 120)]
+ output_timeout_secs: u64,
+ #[arg(long, default_value_t = 1)]
+ max_tokens: u32,
+ #[arg(long, default_value_t = 2)]
+ logprobs: i32,
+ #[arg(long, default_value_t = 1)]
+ prompt_logprobs: i32,
+ #[arg(long, default_value_t = 96)]
+ prompt_repeats: usize,
+}
+
+fn init_tracing() {
+ let filter = EnvFilter::try_from_default_env()
+ .unwrap_or_else(|_| EnvFilter::new("vllm_engine_core_client=debug"));
+ let _ = tracing_subscriber::fmt().with_env_filter(filter).try_init();
+}
+
+fn unique_request_id() -> String {
+ let nanos = std::time::SystemTime::now()
+ .duration_since(std::time::UNIX_EPOCH)
+ .expect("system clock should be after unix epoch")
+ .as_nanos();
+ format!("rust-engine-core-logprobs-{nanos}")
+}
+
+fn build_prompt_token_ids(prompt_repeats: usize) -> Vec {
+ let repeats = prompt_repeats.max(1);
+ BASE_PROMPT_TOKEN_IDS.repeat(repeats)
+}
+
+fn build_request(
+ request_id: String,
+ prompt_token_ids: Vec,
+ max_tokens: u32,
+ logprobs: i32,
+ prompt_logprobs: i32,
+ client_index: u32,
+) -> EngineCoreRequest {
+ EngineCoreRequest {
+ request_id,
+ prompt_token_ids: Some(prompt_token_ids),
+ sampling_params: Some(EngineCoreSamplingParams {
+ max_tokens,
+ logprobs: Some(logprobs),
+ prompt_logprobs: Some(prompt_logprobs),
+ ..EngineCoreSamplingParams::for_test()
+ }),
+ arrival_time: 0.0,
+ client_index,
+ ..EngineCoreRequest::default()
+ }
+}
+
+async fn wait_for_final_output(
+ mut stream: vllm_engine_core_client::EngineCoreOutputStream,
+) -> Result {
+ while let Some(output) = stream.next().await {
+ let output = output.context("failed to receive engine-core output")?;
+ if output.finished() {
+ return Ok(output);
+ }
+ }
+ bail!("request stream ended without a final output")
+}
+
+#[tokio::main(flavor = "multi_thread")]
+async fn main() -> Result<()> {
+ init_tracing();
+ let args = Args::parse();
+ let ready_timeout = Duration::from_secs(args.ready_timeout_secs);
+ let output_timeout = Duration::from_secs(args.output_timeout_secs);
+ let request_id = unique_request_id();
+ let prompt_token_ids = build_prompt_token_ids(args.prompt_repeats);
+ let client = EngineCoreClient::connect(EngineCoreClientConfig {
+ transport_mode: TransportMode::HandshakeOwner {
+ handshake_address: args.handshake_address.clone(),
+ advertised_host: args.host.clone(),
+ engine_count: args.engine_count,
+ ready_timeout,
+ local_input_address: None,
+ local_output_address: None,
+ },
+ coordinator_mode: None,
+ model_name: args.model.clone(),
+ client_index: args.client_index,
+ })
+ .await
+ .context("failed to connect to external vLLM engine")?;
+
+ println!("model={}", args.model);
+ println!("handshake_address={}", args.handshake_address);
+ println!("engine_count={}", args.engine_count);
+ println!("input_address={}", client.input_address());
+ println!("output_address={}", client.output_address());
+ println!("engine_identities={:x?}", client.engine_identities());
+
+ let request = build_request(
+ request_id.clone(),
+ prompt_token_ids.clone(),
+ args.max_tokens,
+ args.logprobs,
+ args.prompt_logprobs,
+ args.client_index,
+ );
+ println!("request_id={request_id}");
+ println!("prompt_len={}", prompt_token_ids.len());
+ println!("base_prompt_len={}", BASE_PROMPT_TOKEN_IDS.len());
+ println!("prompt_repeats={}", args.prompt_repeats);
+ println!("requested_logprobs={}", args.logprobs);
+ println!("requested_prompt_logprobs={}", args.prompt_logprobs);
+
+ let stream = client.call(request).await.context("failed to submit engine-core request")?;
+ let output = timeout(output_timeout, wait_for_final_output(stream))
+ .await
+ .context("timed out waiting for final output")??;
+
+ let finish_reason = output.finish_reason;
+ let token_ids = output.new_token_ids.clone();
+ let logprobs = output
+ .new_logprobs
+ .as_ref()
+ .and_then(|value| value.as_direct())
+ .context("engine output did not include decoded sample logprobs")?;
+ let prompt_logprobs = output
+ .new_prompt_logprobs_tensors
+ .as_ref()
+ .and_then(|value| value.as_direct())
+ .context("engine output did not include decoded prompt logprobs")?;
+
+ println!("token_ids={token_ids:?}");
+ println!("finish_reason={finish_reason:?}");
+ println!("new_logprobs={logprobs:#?}");
+ println!("new_prompt_logprobs_tensors={prompt_logprobs:#?}");
+
+ client.shutdown().await.context("failed to shut down engine-core client")?;
+
+ if finish_reason != Some(EngineCoreFinishReason::Length) {
+ bail!("unexpected finish_reason: expected Length, got {finish_reason:?}");
+ }
+ if token_ids.is_empty() {
+ bail!("engine returned no generated token ids");
+ }
+ if logprobs.is_empty() {
+ bail!("decoded logprobs payload is unexpectedly empty");
+ }
+ if prompt_logprobs.is_empty() {
+ bail!("decoded prompt logprobs payload is unexpectedly empty");
+ }
+ if prompt_logprobs.len() + 1 < prompt_token_ids.len() {
+ bail!(
+ "prompt logprobs rows look too short: prompt_len={}, rows={}",
+ prompt_token_ids.len(),
+ prompt_logprobs.len()
+ );
+ }
+
+ Ok(())
+}
diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_utility_call.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_utility_call.rs
new file mode 100644
index 0000000000000000000000000000000000000000..fd3f62933ddcafd2d0ecfdf01649388c1cdaab22
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/examples/external_engine_utility_call.rs
@@ -0,0 +1,144 @@
+use std::time::Duration;
+
+use anyhow::{Context, Result, bail};
+use clap::Parser;
+use tracing_subscriber::EnvFilter;
+use vllm_engine_core_client::protocol::utility::PauseMode;
+use vllm_engine_core_client::{EngineCoreClient, EngineCoreClientConfig, TransportMode};
+
+#[derive(Debug, Parser)]
+#[command(about = "Smoke-test EngineCoreClient utility calls against an external vLLM engine.")]
+struct Args {
+ #[arg(long)]
+ handshake_address: String,
+ #[arg(long, default_value_t = 1)]
+ engine_count: usize,
+ #[arg(long, default_value = "Qwen/Qwen3-0.6B")]
+ model: String,
+ #[arg(long, default_value = "127.0.0.1")]
+ host: String,
+ #[arg(long, default_value_t = 0)]
+ client_index: u32,
+ #[arg(long, default_value_t = 30)]
+ ready_timeout_secs: u64,
+ #[arg(
+ long,
+ default_value_t = false,
+ help = "Expected initial result of is_sleeping() before running the smoke steps."
+ )]
+ expected_is_sleeping: bool,
+ #[arg(long, default_value_t = false)]
+ reset_running_requests: bool,
+ #[arg(long, default_value_t = false)]
+ reset_external: bool,
+ #[arg(long, default_value_t = 1)]
+ sleep_level: u32,
+ #[arg(long, default_value_t = PauseMode::Abort)]
+ sleep_mode: PauseMode,
+ #[arg(
+ long,
+ default_value_t = false,
+ help = "Skip sleep/wake_up calls when the engine was not started with sleep-mode support."
+ )]
+ skip_sleep_wake: bool,
+}
+
+fn init_tracing() {
+ let filter = EnvFilter::try_from_default_env()
+ .unwrap_or_else(|_| EnvFilter::new("vllm_engine_core_client=debug"));
+ let _ = tracing_subscriber::fmt().with_env_filter(filter).try_init();
+}
+
+#[tokio::main(flavor = "multi_thread")]
+async fn main() -> Result<()> {
+ init_tracing();
+ let args = Args::parse();
+ let client = EngineCoreClient::connect(EngineCoreClientConfig {
+ transport_mode: TransportMode::HandshakeOwner {
+ handshake_address: args.handshake_address.clone(),
+ advertised_host: args.host.clone(),
+ engine_count: args.engine_count,
+ ready_timeout: Duration::from_secs(args.ready_timeout_secs),
+ local_input_address: None,
+ local_output_address: None,
+ },
+ coordinator_mode: None,
+ model_name: args.model.clone(),
+ client_index: args.client_index,
+ })
+ .await
+ .context("failed to connect to external vLLM engine")?;
+
+ println!("model={}", args.model);
+ println!("handshake_address={}", args.handshake_address);
+ println!("engine_count={}", args.engine_count);
+ println!("input_address={}", client.input_address());
+ println!("output_address={}", client.output_address());
+ println!("engine_identities={:x?}", client.engine_identities());
+
+ let initial_is_sleeping =
+ client.is_sleeping().await.context("failed to call is_sleeping utility")?;
+
+ println!("is_sleeping(initial)={initial_is_sleeping}");
+
+ if initial_is_sleeping != args.expected_is_sleeping {
+ bail!(
+ "unexpected initial is_sleeping state: expected {}, got {}",
+ args.expected_is_sleeping,
+ initial_is_sleeping
+ );
+ }
+
+ let reset_prefix_cache = client
+ .reset_prefix_cache(args.reset_running_requests, args.reset_external)
+ .await
+ .context("failed to call reset_prefix_cache utility")?;
+ println!("reset_prefix_cache={reset_prefix_cache}");
+
+ client.reset_mm_cache().await.context("failed to call reset_mm_cache utility")?;
+ println!("reset_mm_cache=ok");
+
+ client
+ .reset_encoder_cache()
+ .await
+ .context("failed to call reset_encoder_cache utility")?;
+ println!("reset_encoder_cache=ok");
+
+ if args.skip_sleep_wake {
+ println!("sleep_wake=skipped");
+ } else {
+ client.sleep(args.sleep_level, args.sleep_mode).await.with_context(|| {
+ format!(
+ "failed to call sleep utility with level={} mode={}",
+ args.sleep_level, args.sleep_mode
+ )
+ })?;
+ println!(
+ "sleep=ok level={} mode={}",
+ args.sleep_level, args.sleep_mode
+ );
+
+ let sleeping_after_sleep =
+ client.is_sleeping().await.context("failed to call is_sleeping after sleep")?;
+ println!("is_sleeping(after_sleep)={sleeping_after_sleep}");
+
+ if !sleeping_after_sleep {
+ bail!("engine should report sleeping=true after sleep()");
+ }
+
+ client.wake_up(None).await.context("failed to call wake_up utility")?;
+ println!("wake_up=ok");
+
+ let sleeping_after_wake =
+ client.is_sleeping().await.context("failed to call is_sleeping after wake_up")?;
+ println!("is_sleeping(after_wake)={sleeping_after_wake}");
+
+ if sleeping_after_wake {
+ bail!("engine should report sleeping=false after wake_up()");
+ }
+ }
+
+ client.shutdown().await.context("failed to shut down engine-core client")?;
+
+ Ok(())
+}
diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client.rs
new file mode 100644
index 0000000000000000000000000000000000000000..9c481362d43c62729666ddb20a8fabdad07b17ea
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client.rs
@@ -0,0 +1,793 @@
+use std::sync::Arc;
+use std::time::Duration;
+
+use futures::future::{join_all, try_join_all};
+use itertools::Itertools;
+use serde::Serialize;
+use tokio::sync::mpsc;
+use tokio_util::task::AbortOnDropHandle;
+use tracing::{debug, info, trace};
+
+use crate::client::imp::{ClientInner, run_abort_loop, run_output_dispatcher_loop};
+use crate::coordinator::CoordinatorHandle;
+use crate::error::{Error, Result};
+use crate::protocol::dtype::ModelDtype;
+use crate::protocol::handshake::EngineCoreReadyResponse;
+use crate::protocol::lora::LoraRequest;
+use crate::protocol::request::{EngineCoreRequest, EngineCoreRequestType};
+use crate::protocol::utility::{EngineCoreUtilityRequest, PauseMode};
+use crate::runtime::{BackgroundShutdownRuntime, build_zmq_runtime};
+use crate::transport::{self, ConnectedEngine};
+
+pub(crate) mod imp;
+mod state;
+mod stream;
+
+pub use stream::{EngineCoreOutputStream, EngineCoreStreamOutput};
+
+/// How the frontend acquires its request/response transport with Python
+/// `EngineCoreProc`s.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
+pub enum TransportMode {
+ /// The Rust process owns the startup handshake and allocates or binds the
+ /// frontend transport addresses itself before replying to engine
+ /// `HELLO` messages.
+ HandshakeOwner {
+ /// Shared handshake endpoint that engines dial during startup.
+ handshake_address: String,
+ /// Host/IP that engines should use to connect back to the frontend
+ /// transport sockets.
+ advertised_host: String,
+ /// Total number of engines expected to join this transport.
+ engine_count: usize,
+ /// Maximum time to wait for each startup phase to complete.
+ ready_timeout: Duration,
+ /// Optional explicit bind address for the input ROUTER socket.
+ local_input_address: Option,
+ /// Optional explicit bind address for the output PULL socket.
+ local_output_address: Option,
+ },
+
+ /// The Python supervisor has already chosen the frontend transport
+ /// addresses, and the Rust process only needs to bind them and wait for
+ /// engine registration frames.
+ Bootstrapped {
+ /// Input ROUTER socket address that engines will connect to for
+ /// requests.
+ input_address: String,
+ /// Output PULL socket address that engines will connect to for
+ /// responses.
+ output_address: String,
+ /// First data-parallel engine rank expected to register on this
+ /// transport.
+ engine_start_index: u32,
+ /// Total number of engines expected to register on this transport.
+ engine_count: usize,
+ /// Maximum time to wait for all expected engines to register.
+ ready_timeout: Duration,
+ },
+}
+
+/// Which coordinator implementation should be active when one is present for a
+/// frontend client.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum CoordinatorMode {
+ /// Run the Rust in-process coordinator for managed `serve` deployments.
+ InProc,
+ /// Connect to an external coordinator owned by another process.
+ External { address: String },
+}
+
+/// Configuration for connecting a Rust frontend client to an already running
+/// Python `EngineCoreProc`.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct EngineCoreClientConfig {
+ /// Frontend-to-engine transport setup.
+ pub transport_mode: TransportMode,
+ /// Frontend-side coordinator behavior, or `None` when requests should flow
+ /// directly to engines without any coordinator involvement.
+ pub coordinator_mode: Option,
+ /// Model name used for frontend-side metrics labels.
+ pub model_name: String,
+ /// Frontend client index stamped onto every request.
+ pub client_index: u32,
+}
+
+impl EngineCoreClientConfig {
+ /// Create a new client config with the given handshake address, expecting a
+ /// single engine, and default values for all other fields.
+ pub fn new_single(handshake_address: impl Into) -> Self {
+ Self {
+ transport_mode: TransportMode::HandshakeOwner {
+ handshake_address: handshake_address.into(),
+ advertised_host: "127.0.0.1".to_string(),
+ engine_count: 1,
+ ready_timeout: Duration::from_secs(30),
+ local_input_address: None,
+ local_output_address: None,
+ },
+ coordinator_mode: None,
+ model_name: String::new(),
+ client_index: 0,
+ }
+ }
+
+ /// Set the model name used by frontend-side metrics and diagnostics.
+ pub fn with_model_name(mut self, model_name: impl Into) -> Self {
+ self.model_name = model_name.into();
+ self
+ }
+
+ /// Override the client index stamped onto every outgoing request.
+ pub fn with_client_index(mut self, client_index: u32) -> Self {
+ self.client_index = client_index;
+ self
+ }
+
+ /// Override the optional coordinator mode for this client config.
+ pub fn with_coordinator_mode(mut self, coordinator_mode: Option) -> Self {
+ self.coordinator_mode = coordinator_mode;
+ self
+ }
+
+ /// Override the locally bound input/output addresses for handshake-owned
+ /// transport mode.
+ ///
+ /// This is primarily used by tests that want deterministic IPC endpoints
+ /// while still exercising the handshake-owned startup path.
+ pub fn with_local_input_output_addresses(
+ mut self,
+ local_input_address: Option,
+ local_output_address: Option,
+ ) -> Self {
+ let TransportMode::HandshakeOwner {
+ local_input_address: current_input,
+ local_output_address: current_output,
+ ..
+ } = &mut self.transport_mode
+ else {
+ panic!("local input/output overrides are only valid in handshake-owned mode");
+ };
+ *current_input = local_input_address;
+ *current_output = local_output_address;
+ self
+ }
+}
+
+/// The reason a request stream is being aborted when its output stream is
+/// dropped.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum AbortCause {
+ /// The consumer dropped the stream before the request reached a terminal
+ /// engine output.
+ #[default]
+ DroppedStream,
+ /// The frontend matched a stop string locally and intentionally stopped
+ /// consuming the stream.
+ StopStringMatched,
+}
+
+task_local::task_local! {
+ static ABORT_CAUSE: AbortCause;
+}
+
+impl AbortCause {
+ /// Return the abort cause currently associated with this task, or
+ /// [`AbortCause::DroppedStream`] by default.
+ pub fn current() -> Self {
+ ABORT_CAUSE.try_get().unwrap_or_default()
+ }
+
+ /// Drop one value while marking the drop as happening for this abort cause.
+ pub fn drop_as(self, value: T) {
+ ABORT_CAUSE.sync_scope(self, move || drop(value));
+ }
+}
+
+/// Internal auto-abort work item sent from stream `Drop` handlers to the abort
+/// worker.
+#[derive(Debug, Clone)]
+pub(crate) struct AbortRequest {
+ request_id: String,
+ cause: AbortCause,
+}
+
+/// Default ZMQ-based implementation that talks directly to a Python
+/// `EngineCoreProc`.
+pub struct EngineCoreClient {
+ config: EngineCoreClientConfig,
+ input_address: String,
+ output_address: String,
+ engines: Vec,
+ inner: Arc,
+ coordinator: Option,
+ abort_tx: mpsc::UnboundedSender,
+
+ /// Runtime used to send messages to the engine and drive all background tasks.
+ runtime: BackgroundShutdownRuntime,
+ // Background tasks
+ output_task: AbortOnDropHandle<()>,
+ dispatcher_task: AbortOnDropHandle<()>,
+ abort_task: AbortOnDropHandle<()>,
+ coordinator_output_task: Option>,
+ coordinator_task: Option>,
+}
+
+impl EngineCoreClient {
+ /// Connect to Python `EngineCoreProc`s using the configured
+ /// transport/coordinator modes.
+ ///
+ /// In handshake-owned mode this method drives the full engine startup
+ /// handshake. In bootstrapped mode it binds the provided frontend
+ /// sockets and waits for the expected engine registration frames.
+ pub async fn connect(config: EngineCoreClientConfig) -> Result {
+ let connected = match &config.transport_mode {
+ TransportMode::HandshakeOwner {
+ handshake_address,
+ advertised_host,
+ engine_count,
+ ready_timeout,
+ local_input_address,
+ local_output_address,
+ } => {
+ let enable_inproc_coordinator = match config.coordinator_mode {
+ None => false,
+ Some(CoordinatorMode::InProc) => true,
+ Some(CoordinatorMode::External { .. }) => {
+ return Err(Error::UnsupportedExternalCoordinator);
+ }
+ };
+
+ transport::connect_handshake(
+ handshake_address,
+ *engine_count,
+ advertised_host,
+ local_input_address.as_deref(),
+ local_output_address.as_deref(),
+ enable_inproc_coordinator,
+ *ready_timeout,
+ )
+ .await?
+ }
+
+ TransportMode::Bootstrapped {
+ input_address,
+ output_address,
+ engine_start_index,
+ engine_count,
+ ready_timeout,
+ } => {
+ if let Some(CoordinatorMode::InProc) = config.coordinator_mode {
+ panic!("cannot use in-process coordinator with bootstrapped transport mode")
+ }
+
+ transport::connect_bootstrapped(
+ input_address,
+ output_address,
+ *engine_start_index,
+ *engine_count,
+ *ready_timeout,
+ )
+ .await?
+ }
+ };
+
+ Self::from_connected(config, connected).await
+ }
+
+ /// Create a new client instance from the connected transport state after
+ /// the startup handshake completes.
+ async fn from_connected(
+ config: EngineCoreClientConfig,
+ connected: transport::ConnectedTransport,
+ ) -> Result {
+ let (output_tx, output_rx) = mpsc::channel(64);
+ let (abort_tx, abort_rx) = mpsc::unbounded_channel();
+ let engines = connected.engines;
+ let runtime = build_zmq_runtime();
+ let inner = Arc::new(ClientInner::new(
+ connected.input_send,
+ runtime.handle().clone(),
+ config.model_name.clone(),
+ &engines,
+ ));
+ let output_task = AbortOnDropHandle::new(runtime.spawn(transport::run_output_loop(
+ connected.output_socket,
+ output_tx,
+ )));
+ let dispatcher_task = AbortOnDropHandle::new(
+ runtime.spawn(run_output_dispatcher_loop(inner.clone(), output_rx)),
+ );
+ let abort_task =
+ AbortOnDropHandle::new(runtime.spawn(run_abort_loop(inner.clone(), abort_rx)));
+
+ // If any engine reported a dp_stats_address in its ready response, use it
+ // as the external coordinator address.
+ let dp_stats_address: Option =
+ engines.iter().find_map(|engine| engine.ready_response.dp_stats_address.clone());
+
+ let (coordinator, coordinator_output_task, coordinator_task) =
+ if let Some(coordinator_transport) = connected.coordinator {
+ let (handle, runner) =
+ CoordinatorHandle::new_inproc(coordinator_transport.input_socket);
+ let (coordinator_output_tx, coordinator_output_rx) = mpsc::channel(64);
+ let coordinator_output_task =
+ AbortOnDropHandle::new(runtime.spawn(transport::run_output_loop(
+ coordinator_transport.output_socket,
+ coordinator_output_tx,
+ )));
+ let coordinator_task = AbortOnDropHandle::new(
+ runtime.spawn(runner.run(coordinator_output_rx, inner.clone())),
+ );
+ (
+ Some(handle),
+ Some(coordinator_output_task),
+ Some(coordinator_task),
+ )
+ } else if let Some(address) =
+ dp_stats_address.as_deref().or(match config.coordinator_mode.as_ref() {
+ Some(CoordinatorMode::External { address }) => Some(address.as_str()),
+ _ => None,
+ })
+ {
+ let (handle, service) = CoordinatorHandle::connect_external(address).await?;
+ let coordinator_task =
+ AbortOnDropHandle::new(runtime.spawn(service.run(inner.clone())));
+ (Some(handle), None, Some(coordinator_task))
+ } else {
+ (None, None, None)
+ };
+
+ Ok(Self {
+ config,
+ input_address: connected.input_address,
+ output_address: connected.output_address,
+ engines,
+ inner,
+ coordinator,
+ abort_tx,
+ runtime,
+ output_task,
+ dispatcher_task,
+ abort_task,
+ coordinator_output_task,
+ coordinator_task,
+ })
+ }
+
+ /// Return the address of the input socket that the client uses to send
+ /// requests to the engine.
+ pub fn input_address(&self) -> &str {
+ &self.input_address
+ }
+
+ /// Return the address of the output socket that the client listens on for
+ /// engine responses.
+ pub fn output_address(&self) -> &str {
+ &self.output_address
+ }
+
+ /// Return the number of engines connected to this client.
+ pub fn engine_count(&self) -> usize {
+ self.engines.len()
+ }
+
+ /// Return the engine-side indices connected to this client.
+ pub fn engine_indices(&self) -> Vec {
+ self.engines
+ .iter()
+ .map(|engine| engine.engine_id.engine_index().expect("engine id must encode as u16"))
+ .collect()
+ }
+
+ /// Return the engine identities of all engines connected to this client.
+ pub fn engine_identities(&self) -> Vec<&[u8]> {
+ self.engines.iter().map(|engine| &*engine.engine_id).collect()
+ }
+
+ /// Return the ready responses received from all engines on the input
+ /// socket.
+ pub fn ready_responses(&self) -> Vec<&EngineCoreReadyResponse> {
+ self.engines.iter().map(|engine| &engine.ready_response).collect()
+ }
+
+ /// Return the engine-reported effective model dtype.
+ pub fn model_dtype(&self) -> ModelDtype {
+ self.engines
+ .first()
+ .expect("engine core client requires at least one engine")
+ .ready_response
+ .dtype
+ }
+
+ /// Return the engine-reported Python vLLM version.
+ pub fn vllm_version(&self) -> &str {
+ self.engines
+ .first()
+ .expect("engine core client requires at least one engine")
+ .ready_response
+ .vllm_version
+ .as_str()
+ }
+
+ /// Return the total number of GPU blocks summed across all connected
+ /// engines.
+ pub fn total_num_gpu_blocks(&self) -> u64 {
+ self.engines.iter().map(|engine| engine.ready_response.num_gpu_blocks).sum()
+ }
+
+ /// Return the minimum engine-reported `max_model_len` across all engines.
+ ///
+ /// This is the auto-fitted value after KV cache profiling and may differ
+ /// from the originally configured value.
+ pub fn max_model_len(&self) -> u32 {
+ self.engines
+ .iter()
+ .map(|engine| engine.ready_response.max_model_len as u32)
+ .min()
+ .expect("engine core client requires at least one engine")
+ }
+
+ /// Return the world size (TP * PP) from the parallel config, if available.
+ pub fn world_size(&self) -> u64 {
+ self.engines
+ .first()
+ .expect("engine core client requires at least one engine")
+ .ready_response
+ .world_size
+ }
+
+ /// Return the data parallel size from the parallel config, if available.
+ pub fn data_parallel_size(&self) -> u64 {
+ self.engines
+ .first()
+ .expect("engine core client requires at least one engine")
+ .ready_response
+ .data_parallel_size
+ }
+
+ /// Get the model name associated with this client used for metrics
+ /// labeling.
+ pub fn model_name(&self) -> &str {
+ self.inner.model_name()
+ }
+
+ /// Return whether the client still considers the engine healthy.
+ pub fn is_healthy(&self) -> bool {
+ self.inner.is_healthy()
+ }
+
+ /// Return the first persistent health error observed by the client, if any.
+ pub fn health_error(&self) -> Option> {
+ self.inner.health_error()
+ }
+}
+
+// Client API implementation.
+impl EngineCoreClient {
+ /// Add a new request to the engine and return a per-request raw output
+ /// stream.
+ pub async fn call(&self, mut req: EngineCoreRequest) -> Result {
+ req.client_index = self.config.client_index;
+ req.validate()?;
+ trace!(
+ request_id = %req.request_id,
+ client_index = req.client_index,
+ current_wave = req.current_wave,
+ request = ?req,
+ "sending add request"
+ );
+
+ let request_id = req.request_id.clone();
+ let lora_name = req.lora_request.as_ref().map(|lora| lora.lora_name.clone());
+ let data_parallel_rank = req.data_parallel_rank;
+ let (engine_id, rx) =
+ self.inner.register_request(request_id.clone(), lora_name, data_parallel_rank)?;
+
+ let result: Result<()> = async {
+ if let Some(coordinator) = self.coordinator.as_ref() {
+ let snapshot = coordinator.snapshot();
+ req.current_wave = snapshot.current_wave;
+ if !snapshot.engines_running {
+ coordinator.notify_first_request(engine_id.clone())?;
+ }
+ }
+
+ debug!(
+ request_id = req.request_id,
+ ?engine_id,
+ "registered request to engine"
+ );
+
+ self.inner.send_to_engine(&engine_id, EngineCoreRequestType::Add, &req).await?;
+ Ok(())
+ }
+ .await;
+
+ // Failed to send the request to the engine, roll back the registration.
+ if let Err(error) = result {
+ self.inner.rollback_request(&request_id);
+ return Err(error);
+ }
+
+ Ok(EngineCoreOutputStream::new(
+ request_id,
+ self.abort_tx.clone(),
+ rx,
+ ))
+ }
+
+ /// Abort currently in-flight requests by request ID.
+ pub async fn abort(&self, ids: &[String]) -> Result<()> {
+ let abortable = self.inner.abortable_request_ids(ids)?;
+
+ trace!(request_ids = ?ids, abortable_request_ids = ?abortable, "sending abort request ids");
+
+ if abortable.is_empty() {
+ return Ok(());
+ }
+
+ // Finalize the consumer streams first, before the engine round-trip.
+ let all_request_ids: Vec = abortable.values().flatten().cloned().collect();
+ self.inner.abort_requests_locally(&all_request_ids);
+
+ for (engine_id, request_ids) in abortable {
+ self.inner.do_abort_requests(&engine_id, &request_ids).await?;
+ }
+ Ok(())
+ }
+
+ /// Call a typed utility method on all connected engines, returning one
+ /// decoded result per connected engine if all calls succeed or an error
+ /// if any call fails.
+ ///
+ /// Callers should pass utility arguments using Rust tuple semantics so the
+ /// encoded payload matches Python's `(client_index, call_id,
+ /// method_name, args)` contract: `()`, `(arg,)`, `(arg1, arg2)`, etc.
+ pub async fn call_utility(&self, method: &str, args: A) -> Result>
+ where
+ T: serde::de::DeserializeOwned,
+ A: serde::Serialize + std::fmt::Debug,
+ {
+ trace!(
+ method,
+ client_index = self.config.client_index,
+ engine_count = self.engines.len(),
+ "sending utility request"
+ );
+
+ // Phase 1: allocate one call id per engine and build the per-engine
+ // request payloads up-front. Any failure here (registry closed, encode
+ // error) must roll back the call ids already allocated so they do not
+ // leak in the utility registry until shutdown.
+ let mut pending_calls = Vec::with_capacity(self.engines.len());
+ let mut prepared_sends = Vec::with_capacity(self.engines.len());
+ for engine in &self.engines {
+ let (call_id, rx) = match self.inner.allocate_and_register_utility_call() {
+ Ok(pair) => pair,
+ Err(err) => {
+ self.inner.unregister_utility_calls(pending_calls.iter().map(|(id, _)| *id));
+ return Err(err);
+ }
+ };
+ let request = match EngineCoreUtilityRequest::new(
+ self.config.client_index,
+ call_id,
+ method,
+ &args,
+ ) {
+ Ok(request) => request,
+ Err(err) => {
+ self.inner.unregister_utility_calls(
+ pending_calls.iter().map(|(id, _)| *id).chain(std::iter::once(call_id)),
+ );
+ return Err(err);
+ }
+ };
+ pending_calls.push((call_id, rx));
+ prepared_sends.push((&engine.engine_id, request));
+ }
+
+ // Phase 2: dispatch every utility request concurrently. `try_join_all`
+ // fails fast on the first transport error and drops the remaining send
+ // futures; any engines that already received the request will reply,
+ // but those replies are simply dropped because we roll back the call
+ // ids below.
+ let send_futures = prepared_sends.iter().map(|(engine_id, request)| {
+ self.inner.send_to_engine(engine_id, EngineCoreRequestType::Utility, request)
+ });
+ if let Err(err) = try_join_all(send_futures).await {
+ self.inner.unregister_utility_calls(pending_calls.iter().map(|(id, _)| *id));
+ return Err(err);
+ }
+
+ // Phase 3: wait for all engines to respond and preserve the per-engine
+ // result list.
+ let futures = pending_calls.into_iter().map(|(call_id, rx)| async move {
+ rx.await
+ .map_err(|_| Error::UtilityCallClosed {
+ method: method.to_string(),
+ call_id,
+ })??
+ .into_typed_result(method)
+ });
+ try_join_all(futures).await
+ }
+
+ /// Call a utility method on all connected engines and return the shared
+ /// result if every engine agrees.
+ pub async fn call_utility_consensus(&self, method: &str, args: A) -> Result
+ where
+ T: serde::de::DeserializeOwned + std::fmt::Debug + PartialEq,
+ A: serde::Serialize + std::fmt::Debug,
+ {
+ let results: Vec = self.call_utility(method, args).await?;
+
+ if results.iter().all_equal() {
+ // `engine_count >= 1` is enforced during startup handshake so `results` must be
+ // non-empty.
+ Ok(results.into_iter().next().unwrap())
+ } else {
+ Err(Error::InconsistentUtilityResults {
+ method: method.to_string(),
+ values: format!("{results:?}"),
+ })
+ }
+ }
+
+ /// Execute `collective_rpc` on all engines and flatten all engine results
+ /// into one list.
+ pub async fn collective_rpc(
+ &self,
+ method: &str,
+ timeout: Option,
+ args: A,
+ kwargs: K,
+ ) -> Result>
+ where
+ A: serde::Serialize + std::fmt::Debug,
+ K: serde::Serialize + std::fmt::Debug,
+ {
+ let results = self
+ .call_utility::("collective_rpc", (method, timeout, args, kwargs))
+ .await?;
+
+ Ok(results
+ .into_iter()
+ .flat_map(|result| match result {
+ // Each engine's `collective_rpc` result is itself the worker-level result list.
+ rmpv::Value::Array(results) => results,
+ other => vec![other],
+ })
+ .collect())
+ }
+
+ /// Return whether the engine is currently sleeping at any level.
+ pub async fn is_sleeping(&self) -> Result {
+ self.call_utility_consensus("is_sleeping", ()).await
+ }
+
+ /// Reset the multi-modal cache.
+ pub async fn reset_mm_cache(&self) -> Result<()> {
+ self.call_utility::<(), _>("reset_mm_cache", ()).await?;
+ Ok(())
+ }
+
+ /// Reset the encoder cache.
+ pub async fn reset_encoder_cache(&self) -> Result<()> {
+ self.call_utility::<(), _>("reset_encoder_cache", ()).await?;
+ Ok(())
+ }
+
+ /// Reset the prefix cache and optionally the external connector cache.
+ ///
+ /// Under data parallel, returns `true` only when every engine confirms the
+ /// reset (AND aggregation).
+ pub async fn reset_prefix_cache(
+ &self,
+ reset_running_requests: bool,
+ reset_connector: bool,
+ ) -> Result {
+ Ok(self
+ .call_utility(
+ "reset_prefix_cache",
+ (reset_running_requests, reset_connector),
+ )
+ .await?
+ .into_iter()
+ .all(|reset| reset))
+ }
+
+ /// Load or refresh one LoRA adapter on every connected engine.
+ pub async fn add_lora(&self, lora_request: &LoraRequest) -> Result {
+ Ok(self
+ .call_utility::("add_lora", (lora_request,))
+ .await?
+ .into_iter()
+ .all(|loaded| loaded))
+ }
+
+ /// Remove one LoRA adapter from every connected engine.
+ pub async fn remove_lora(&self, lora_id: u64) -> Result {
+ Ok(self
+ .call_utility::("remove_lora", (lora_id,))
+ .await?
+ .into_iter()
+ .all(|removed| removed))
+ }
+
+ /// Put the engine to sleep.
+ pub async fn sleep(&self, level: u32, mode: PauseMode) -> Result<()> {
+ self.call_utility::<(), _>("sleep", (level, mode)).await?;
+ Ok(())
+ }
+
+ /// Wake the engine from sleep, optionally limiting the wake-up to specific
+ /// tags.
+ pub async fn wake_up(&self, tags: Option>) -> Result<()> {
+ self.call_utility::<(), _>("wake_up", (tags,)).await?;
+ Ok(())
+ }
+
+ /// Pause the scheduler so generation can be halted
+ pub async fn pause_scheduler(&self, mode: PauseMode, clear_cache: bool) -> Result<()> {
+ self.call_utility::<(), _>("pause_scheduler", (mode, clear_cache)).await?;
+ Ok(())
+ }
+
+ /// Resume the scheduler after a pause
+ pub async fn resume_scheduler(&self) -> Result<()> {
+ self.call_utility::<(), _>("resume_scheduler", ()).await?;
+ Ok(())
+ }
+
+ /// Return whether the scheduler is currently in any pause state.
+ pub async fn is_scheduler_paused(&self) -> Result {
+ self.call_utility_consensus("is_scheduler_paused", ()).await
+ }
+
+ /// Start profiling the engine.
+ pub async fn start_profile(&self, profile_prefix: Option<&str>) -> Result<()> {
+ self.call_utility::<(), _>("profile", (true, profile_prefix)).await?;
+ Ok(())
+ }
+
+ /// Stop profiling the engine.
+ pub async fn stop_profile(&self, profile_prefix: Option<&str>) -> Result<()> {
+ self.call_utility::<(), _>("profile", (false, profile_prefix)).await?;
+ Ok(())
+ }
+
+ /// Shut down local client tasks and close transport state.
+ pub async fn shutdown(self) -> Result<()> {
+ let Self {
+ inner,
+ abort_tx,
+ runtime,
+ output_task,
+ dispatcher_task,
+ abort_task,
+ coordinator_output_task,
+ coordinator_task,
+ ..
+ } = self;
+
+ info!("shutting down engine-core client");
+ inner.shutdown();
+ drop(abort_tx);
+
+ // Abort all client tasks first, then await them.
+ // Note the aborting orders here.
+ let mut tasks = vec![abort_task, dispatcher_task, output_task];
+ tasks.extend(coordinator_task);
+ tasks.extend(coordinator_output_task);
+
+ tasks.iter().for_each(|t| t.abort());
+ join_all(tasks).await;
+ drop(inner);
+ drop(runtime);
+
+ info!("engine-core client shut down");
+ Ok(())
+ }
+}
diff --git a/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/imp.rs b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/imp.rs
new file mode 100644
index 0000000000000000000000000000000000000000..9975471c7472f6c92c39b1c3faf3651fde404a0d
--- /dev/null
+++ b/asian_ai/performance/vLLM/rust/src/engine-core-client/src/client/imp.rs
@@ -0,0 +1,476 @@
+use std::collections::{BTreeMap, BTreeSet};
+use std::sync::Arc;
+use std::time::{SystemTime, UNIX_EPOCH};
+
+use arc_swap::ArcSwapOption;
+use parking_lot::Mutex;
+use thiserror_ext::AsReport as _;
+use tokio::runtime::Handle;
+use tokio::sync::mpsc;
+use tracing::{debug, info, trace, warn};
+use vllm_metrics::METRICS;
+use zeromq::RouterSendHalf;
+
+use crate::client::state::{OutputReceiver, RequestRegistry, UtilityReceiver, UtilityRegistry};
+use crate::client::stream::EngineCoreStreamOutput;
+use crate::client::{AbortCause, AbortRequest};
+use crate::error::{client_closed, dispatcher_closed, unexpected_dispatcher_output};
+use crate::metrics::{LoraInfoExporter, record_scheduler_stats};
+use crate::protocol::encode_msgpack;
+use crate::protocol::output::{EngineCoreOutput, EngineCoreOutputs};
+use crate::protocol::request::EngineCoreRequestType;
+use crate::protocol::stats::SchedulerStats;
+use crate::protocol::utility::UtilityOutput;
+use crate::transport::{ConnectedEngine, EngineId};
+use crate::{Error, Result, transport};
+
+pub(crate) struct ClientInner {
+ input_send: RouterSendHalf,
+ /// The runtime handle used for sending messages to the engine.
+ handle: Handle,
+ model_name: String,
+ request_reg: Mutex,
+ utility_reg: Mutex,
+ health_error: ArcSwapOption,
+}
+
+impl ClientInner {
+ /// Create a new instance with the given input send half after the startup
+ /// handshake completes.
+ pub fn new(
+ input_send: RouterSendHalf,
+ handle: Handle,
+ model_name: String,
+ engines: &[ConnectedEngine],
+ ) -> Self {
+ Self {
+ input_send,
+ handle,
+ model_name,
+ request_reg: Mutex::new(RequestRegistry::new(engines)),
+ utility_reg: Mutex::new(UtilityRegistry::default()),
+ health_error: ArcSwapOption::empty(),
+ }
+ }
+
+ /// Get the model name associated with this client used for metrics
+ /// labeling.
+ pub fn model_name(&self) -> &str {
+ &self.model_name
+ }
+
+ /// Register a newly added request. Return the selected engine id and the
+ /// per-request output channel bound to its `request_id`.
+ ///
+ /// When `data_parallel_rank` is provided, the request is routed to that
+ /// specific engine rank, bypassing load balancing. `lora_name` is the
+ /// request's LoRA adapter, tracked for `vllm:lora_requests_info`.
+ pub fn register_request(
+ &self,
+ request_id: String,
+ lora_name: Option,
+ data_parallel_rank: Option,
+ ) -> Result<(EngineId, OutputReceiver)> {
+ let mut registry = self.request_reg.lock();
+ if registry.is_closed() {
+ return Err(self.closed_error());
+ }
+ registry.register(request_id, lora_name, data_parallel_rank)
+ }
+
+ /// Allocate the next utility `call_id` and register its waiting receiver.
+ pub fn allocate_and_register_utility_call(&self) -> Result<(u64, UtilityReceiver)> {
+ let mut registry = self.utility_reg.lock();
+ if registry.is_closed() {
+ return Err(self.closed_error());
+ }
+ Ok(registry.allocate_and_register())
+ }
+
+ /// Undo a batch of utility call allocations when the fan-out send fails
+ /// partway through. Silently ignores unknown call ids so callers can pass
+ /// the full set without first filtering successful sends.
+ pub fn unregister_utility_calls(&self, call_ids: impl IntoIterator- ) {
+ self.utility_reg.lock().unregister_many(call_ids);
+ }
+
+ /// Undo a request registration when `add_request()` fails.
+ pub fn rollback_request(&self, request_id: &str) {
+ let _ = self.request_reg.lock().remove(request_id);
+ }
+
+ /// Filter the given request IDs to the subset that are still tracked as
+ /// active and can be aborted, grouped by the engine that originally
+ /// accepted them.
+ pub fn abortable_request_ids(
+ &self,
+ request_ids: &[String],
+ ) -> Result>> {
+ let registry = self.request_reg.lock();
+ if registry.is_closed() {
+ return Err(self.closed_error());
+ }
+ Ok(registry.abortable_request_ids(request_ids))
+ }
+
+ /// Obtain stream senders for a whole engine output batch with one registry
+ /// lock acquisition.
+ pub fn take_senders_for_outputs<'a>(
+ &self,
+ outputs: impl IntoIterator
- ,
+ ) -> Vec